Date: June 20, 2024
Speakers: Joshua Cova and Luuk Schmitz, Max Planck Institute for the Study of Societies
This workshop, led by Joshua Cova and Luuk Schmitz from the Max Planck Institute for the Study of Societies, provided an insightful introduction to the use of Large Language Models (LLMs) in social science research. The session covered several key areas:
Four main uses of LLMs in social sciences:
a) Modeling human behavior computationally (Aher, Arriaga, and Kalai, 2022; Dillion et al., 2023; Horton, 2023; Tjuatja et al., 2023)
b) Simulating social relationships (Park et al., 2023; Wang et al., 2024; Akata et al., 2023)
c) Interacting with human agents (Chopra and Haaland, 2023)
d) Text annotation (Törnberg, 2023; Gilardi, Alizadeh, and Kubli, 2023; Leek, Bischl, and Freier, 2024; Laurer, 2024)
Validation and Performance Metrics: Cova emphasized the importance of validation in using LLMs, referencing principles from previous research in quantitative text analysis. He highlighted the need for iterative processes in prompt design and the use of various performance metrics such as accuracy, precision, and recall.
Model Selection:Schmitz discussed the importance of choosing the right model, considering factors such as reproducibility, ethics, transparency, language, scalability, and complexity (Törnberg, 2024). He also presented a decision tree for using generative LLMs in text annotation (Weber and Reichardt, 2024).
Practical Considerations:The speakers shared their experiences with different LLM platforms and discussed the trade-offs between performance and open science considerations. They emphasized that the choice between LLMs and classifier models should be based on the specific research needs and available resources.
Ethical Considerations:While not extensively covered in this session, the speakers mentioned that research ethics and the debate between closed and open-sourced models would be discussed in a future session.
The workshop provided a comprehensive overview of LLMs in social science research, emphasizing their potential while cautioning against indiscriminate use. The speakers advocated for a thoughtful approach, considering alternatives and the specific value LLMs can add to research projects. For the slide, please refer to Luuk's twitter post: https://x.com/Luukschmitz/status/1808083990529683510
Comments