Generative Artificial Intelligence (GenAI) is rapidly being integrated across industries, promising transformative practices in areas such as software development, marketing, healthcare, finance, manufacturing, and customer service, and enhancing efficiency through Human-Artificial Intelligence (HAI) interaction and collaboration. However, one of the major obstacles for fluent HAI interaction is the lack of common context and understanding. Without these, the AI and the user are not guaranteed to be aligned, resulting in inefficient and frustrating collaboration. In this position paper, we propose to integrate model-based understanding of the user into GenAI unlike current use of information retrieval augmented generation in copilots. The model implements hypotheses about human cognition and how human goals, beliefs, and abilities impact behavior and adaptation in interactive environments. This permits the GenAI to infer the latent mental states of the user and predict how the user would adapt to different actions taken by the AI. We emphasize the importance of cognitively informed HAI interaction, advocating for future interdisciplinary research in the intersection of cognitive science and AI, focusing on finding empirical evidence about how GenAIs' ability to infer the latent cognitive states of the user benefits human-GenAI interaction. To facilitate responsible development, implementation and widespread acceptance of cognitively informed AI agents we propose and recommend the CRAFTS AI Framework. Ignoring the principles laid out in the framework could significantly hinder the progress and potential benefits of HAI collaboration.