Collaborating with AI for Creative Work: How Large Language Models (LLMs) Are Transforming Creativity
- Yuan Ren
- Mar 25
- 3 min read
Since its emergence in December 2022, ChatGPT has sparked widespread discussion, leading to the rapid adoption of Large Language Models (LLMs) across various industries. These models now assist users in handling a broad spectrum of open-ended tasks, many of which demand a high level of creativity.
Last year, Zenan Chen and Jason Chan published a groundbreaking study in the top journal Management Science, titled "Large Language Models in Creative Work: The Role of Collaboration Modality and User Expertise." This study, grounded in a rigorously designed randomized experiment, investigated how different modes of human-LLM collaboration impact the creative process in ad copywriting. It focused on comparing two primary ways of using LLMs: (a) as ghostwriters, where the model takes the lead in content generation, and (b) as sounding boards, providing feedback and evaluation on human-created content. The experiment invited both experts and non-experts to participate, requiring them to complete ad copywriting tasks both with and without the assistance of LLMs, and then assessing the quality of the advertisements.
Interesting research finding: how does AI collaborate with humans?
A key finding of the study is that the impact of LLM usage is not uniform; rather, it is shaped by the interaction between the collaboration model and the user's level of expertise. With LLM feedback, non-experts were able to produce ad content that was more semantically aligned with expert-level content, effectively bridging the gap between them and domain experts. However, for experts, using LLMs as ghostwriters not only failed to provide significant advantages but also negatively impacted their performance. This occurs because the ghostwriter model is especially prone to the anchoring effect, where the initial content generated by LLMs subtly constrains experts' creative thinking, ultimately resulting in lower-quality advertisements. Furthermore, experts did not experience substantial benefits when using LLMs as sounding boards. This may be explained by the ceiling effect, wherein the general-purpose LLM’s suggestions fail to surpass the depth of expertise and practical experience that experts already possess.
AI in Advertising Creativity: A Double-Edged Sword
Beyond those insights, the study presents additional thought-provoking findings. Generally, advertisements that are objective, moderately lengthy, and highly readable tend to achieve higher engagement rates. However, the initial drafts generated by LLMs in ghostwriter mode exhibited lower readability.
Moreover, in the ghostwriter mode, LLMs tend to incorporate more emojis and hashtags into their initial drafts, which may undermine the credibility of the message and the perceived reliability of the source (Koch et al., 2023). Although participants made partial adjustments to the use of emojis after working with the ghostwriter model, these elements initially introduced by LLMs still appeared to exert an anchoring effect on the final advertisements.
Insights for Businesses and Practitioners: The Vast Potential of AI Adoption
This study provides highly valuable insights for practitioners, highlighting the importance of selecting the optimal LLM collaboration model based on the practitioner’s level of expertise. For novice professionals with limited experience, the sounding board model is more effective in enhancing the quality of their creative work. Organizations may consider leveraging LLMs in this capacity to support new hires or help them navigate unfamiliar operational challenges, thereby accelerating their learning and growth. Simultaneously, organizations should implement clear guidelines for LLM usage, emphasizing potential cognitive biases that employees may face when engaging with the ghostwriter model. This approach should be employed with caution, particularly when relying on experts for high-level creative output, as it carries the risk of triggering the anchoring effect and constraining their creative thinking. While general-purpose LLMs may not directly enhance expert-level work, developing specialized models trained on domain-specific data could offer fresh perspectives and valuable insights to experienced professionals.
As LLM technology continues to advance and become more widely adopted, individuals and organizations must carefully evaluate both its advantages and inherent limitations. By strategically leveraging LLM strengths and mitigating its shortcomings, businesses will be better positioned to harness the power of generative AI, unlocking human creativity and driving superior outcomes across various creative fields.
References：
Chen, Z., & Chan, J. (2024). Large Language Model in Creative Work: The Role of Collaboration Modality and User Expertise. Management Science, 70(12), 9101–9117. https://doi.org/10.1287/mnsc.2023.03014
Koch T, Denner N, Crispin M, Hohagen T (2023) Funny but not credible? Why using (many) emojis decreases message credibil ity and source trustworthiness. Soc. Media Soc., ePub ahead of print September 4, https://doi.org/10.1177/20563051231194584.