« Back

Can LLMs conduct interviews and if so how should they do so?

Author(s): Hjalmar Carlsen, Tobias Gårdhus

Friday 16  |   11:40-12:00

Room: TP45

Session: Pretrained models and sociology: ethical, methodological and theoretical considerations

Recently it has been argued that generative LLMs can conduct large-scale qualitative interviews, the so-called AI-interview. While generative LLMs have demonstrated their ability to produce human-like content and engage in human-like conversations, they are also known to be overconfident and presumptions in their understanding of their interlocutor and for their lack of transparency in their operation. These are major issues for producing high-quality interview data. Therefore, it is important to discuss which type of interview a LLM system can and should conduct. In this paper, we start by addressing this question. For most applications, we argue for a medium to low degree of freedom for the AI-interview, where the LLM system is mainly responsible for probing for further information under predefined questions or for translating existing questions into a certain conversational context. We then present our system architecture that allows for conducting LLM interviews in transparent and controlled manner. Finally, we demonstrate a working AI-interviewer based on a local LLM, which solves the legal and ethical issues with using ChatGTP model.