Now that the application of Artificial Intelligence (Al) is becoming more mainstream, it is applied in many different fields, and consequently, it is starting to play a more prominent role in design processes. In current mainstream HCI (Human-Computer Interaction) design frameworks, the human (or user) is seen as the main focus, and stakeholders' perspectives are taken into account throughout the whole problem-solving process to design thoughtful solutions. The increased complexity of design processes caused by the rise of Al, however, poses new challenges to these existing approaches, particularly for involving the human in the design process. Five challenges that can be of influence on accounting for the human in design processes involving Al are identified and elaborated upon: 1) insufficient Al literacy of designers and users, 2) the black-box nature of neural networks, 3) where to start: design vs data, 4) customized solutions for narrow user segments, and 5) thinking ahead: an extended collaborative design process. These challenges arise across the exploration, design, implementation, evaluation, and deployment phases. This extended abstract discusses possible approaches per challenge on how to warrant integration of the human perspective.