Learning to Learn with Large Language Models

by Isabel Schorr

With the release of ChatGPT in November 2022, educational practices in schools and universities have undergone a rapid transformation. Much like the introduction of Google's search engine, ChatGPT has altered how learners access and engage with information, solve problems, and regulate their learning processes, introducing both new possibilities and challenges. Large Language Models (LLMs) such as ChatGPT open up new educational opportunities, from personalized feedback to new ways of exploratory learning and timely feedback - offering support in all phases of self-regulated learning (SRL). Their potential applications are not limited to specific subjects, as good LLMs can guide learners through multi-step calculation problems, explain complicated chemical reaction mechanisms, decipher the poetic depth of Friedrich Schiller, and even write entire essays on request. Yet, these benefits are accompanied by concerns from both educators and learners about overdependence, misinformation, and the kind of cognitive outsourcing that risks undermining essential learning processes.

Established learning theories about self-regulation and collaborative learning were originally established with human-human or individual learning in mind. Despite the widespread use of LLMs by learners, our understanding of what constitutes effective and ineffective learning in chatbot interactions remains limited. A major obstacle in addressing this problem is the lack of suitable data. High-quality interaction data between learners and LLMs is scarce, and labeled data is even harder to obtain. Creating such labels requires a substantial amount of time and financial resources. Additionally, we lack established annotation schemes that capture meaningful dynamics in chatbot-mediated learning, in contrast, for example, to the well-developed frameworks available for traditional classroom discourse.

From a machine learning perspective, LLM-based learning offers a clear opportunity: the interaction traces are text-structured, dense, and theoretically analyzable with computational methods. ML has long been used to identify inefficiencies in online learning behavior and evaluate teaching processes. Yet, important questions remain open for LLM-based learning:

• What kinds of representations or labels meaningfully capture learning processes in human-LLM dialogues?
• What scalable approaches can help overcome data scarcity?
• How can we detect unproductive or counterproductive interactions in real time?
• How can learning-theoretic insights guide the development and refinement of AI systems?

These questions define the core of my PhD, where I work on developing computational methods to better represent and improve learning in human-LLM interactions.