The Internet of Things (IoT) is rapidly growing, and offers many economical and societal potentials and benefits. Nevertheless, the IoT also introduces new threats to our Security, Privacy and Safety (SPS). The existing work on mitigating these SPS threats often fails to address the fundamental challenges behind the mitigation measures proposed, and fails to make the relations between different mitigation measures explicit. This paper, therefore, offers a conceptual framework for understanding and approaching the challenges and obstacles that arise in addressing the SPS threats of the IoT. This contribution aims to help policymakers in adopting policies and strategies that stimulate others to develop, deploy and use IoT devices, applications and services in secure, privacy-friendly and safe ways.