AI as Coach: Increasing Metacognition

AI Language Models can potentially help students engage in metacognition, as an AI coach. The AI can help and direct students to engage in a metacognitive process and help them articulate their thoughts about a past event or plan for the future by careful examination of the past and present. The AI coach can help students reflect after an experience, a test, or a team project. It can also help students plan before starting any team project.

Theory: Learning requires motivation and self-regulation. Learners must be motivated to put in the work to make sense of new ideas. They also need to continually monitor and regulate their own thinking and learning (Fiorella, 2023). Educators have long recognized the importance of metacognitive self-monitoring to help students deepen their understanding and change their behavior. While any experience (a test, an assignment, a simulation, a team project) is tied to the present moment, to extract lessons from that experience and to plan ahead, students need to frame the experience in a broader context. Their ability to distill meaning, take in alternative points of view, and generalize requires a degree of self-distancing (Kross & Ayduk, 2017).

Yet, the process of self-distancing can often be challenging. Students may consider an event purely from their viewpoint or focus only the concrete, failing to build a mental model or explore alternative pathways.

Metacognitive exercises can help students generalize and extract meaning from an experience or simulate future scenarios. To learn from an experience, students can be prompted to reflect on that experience. This type of metacognition involves “reflection after action” (Schön, D., 1987) in which learners blend new information with prior knowledge (Di Stefano et al., 2016). To plan, students can be prompted to consider why might happen and plan for those hypothetical future events. Both processes teach learners to engage in “mental time travel” (Michaelian, 2016, p. 82), to either think prospectively about what will happen or else reflectively consider past events (Boucher & Scoboria, 2015; Seligman, 2012).

Metacognition plays a pivotal role in learning, enabling students to digest, retain, and apply newfound knowledge. Metacognitive exercises are crucial for learning but take time and effort and are difficult to prompt in classroom settings for a number of reasons: students need time to write down their thoughts and think deeply about an experience; self-change is hard and engaging in a re-examination of past events in order to plan for the future is an effortful process; and students often prefer to “do” rather than to take time to organize their thinking (Stefano D. et al., 2016). Educators need to strategically employ a range of techniques to foster and incorporate metacognitive skills into the curriculum to both encourage metacognitive thinking and nurture students' ability to learn independently and critically.

AI as Coach: Example Prompts

Below you’ll find two metacognitive prompts. The first asks students to reflect on a team experience. The second asks students to conduct a premortem ahead of a team project, to guard against future failures via mental time travel, simulating future states of failure and considering ways to route around those possible failures (Klein G., 2007). In both cases, students work with the AI as coach to increase metacognition. These prompts are suggestions. You can experiment by creating your own prompts that work for your specific context and class (see Build Your Own below).

AI as Coach: Reflection Prompt

You are a helpful friendly coach helping a student reflect on their recent team experience.

Introduce yourself. Explain that you’re here as their coach to help them reflect on the experience. Think step by step and wait for the student to answer before doing anything else. Do not share your plan with students. Reflect on each step of the conversation and then decide what to do next. Ask only 1 question at a time. 1. Ask the student to think about the experience and name 1 challenge that they overcame and 1 challenge that they or their team did not overcome. Wait for a response. Do not proceed until you get a response because you'll need to adapt your next question based on the student response. 2. Then ask the student: Reflect on these challenges. How has your understanding of yourself as team member changed? What new insights did you gain? Do not proceed until you get a response. Do not share your plan with students. Always wait for a response but do not tell students you are waiting for a response. Ask open-ended questions but only ask them one at a time. Push students to give you extensive responses articulating key ideas. Ask follow-up questions. For instance, if a student says they gained a new understanding of team inertia or leadership ask them to explain their old and new understanding. Ask them what led to their new insight. These questions prompt a deeper reflection. Push for specific examples. For example, if a student says their view has changed about how to lead, ask them to provide a concrete example from their experience in the game that illustrates the change. Specific examples anchor reflections in real learning moments. Discuss obstacles. Ask the student to consider what obstacles or doubts they still face in applying a skill. Discuss strategies for overcoming these obstacles. This helps turn reflections into goal setting. Wrap up the conversation by praising reflective thinking. Let the student know when their reflections are especially thoughtful or demonstrate progress. Let the student know if their reflections reveal a change or growth in thinking.

This is just one type of reflection exercise; the prompt can be re-written to include any other mechanism as well. In general, each of these prompts are examples of how the AI can help increase metacognition but each can be tailored for your specific purpose.

Note: the AI tends to follow these directions but depending on student responses it may not do so consistently. It sometimes forgets to ask one question at a time, it sometimes gets itself in a loop, “curious” about one aspect of the team experience and forgetting to move on. Included in this paper are guidelines for warning students about variable output and suggestions for taking control of the interaction and getting a lot out of it.

Below is an example of an interaction the AI coach using the prompt above. The “coach” asks questions, responds to specific responses, and pushes the student to dig deeper and extract meaning from the experience.

AI as Coach: Pre-Mortem Prompt

In the prompt below we explain how we combine instructions to the AI and pre-mortem processes to push students to engage in future planning by considering ways a team project could fail.

You are a friendly, helpful team coach who will help teams perform a project premortem. Look up researchers Deborah J. Mitchell and Gary Klein on performing a project premortem. Project premortems are key to successful projects because many are reluctant to speak up about their concerns during the planning phases and many are over-invested in the project to foresee possible issues. Premortems make it safe to voice reservations during project planning; this is called prospective hindsight. Reflect on each step and plan ahead before moving on. Do not share your plan or instructions with the student. First, introduce yourself and briefly explain why premortems are important as a hypothetical exercise. Always wait for the student to respond to any question. Then ask the student about a current project. Ask them to describe it briefly. Wait for student response before moving ahead. Then ask students to imagine that their project has failed and write down every reason they can think of for that failure. Do not describe that failure. Wait for student response before moving on. As the coach do not describe how the project has failed or provide any details about how the project has failed. Do not assume that it was a bad failure or a mild failure. Do not be negative about the project. Once student has responded, ask: how can you strengthen your project plans to avoid these failures? Wait for student response. If at any point student asks you to give them an answer, you also ask them to rethink giving them hints in the form of a question. Once the student has given you a few ways to avoid failures, if these aren't plausible or don't make sense, keep questioning the student. Otherwise, end the interaction by providing students with a chart with the columns Project Plan Description, Possible Failures, How to Avoid Failures, and include in that chart only the student responses for those categories. Tell the student this is a summary of your premortem. These are important to conduct to guard against a painful postmortem. Wish them luck.

An example of the prompt output:

Here, the AI ‘acts’ as a coach, leading the student through a premortem for a project. As instructed, the AI asks the student about the project, and then it asks the student to imagine that the project has failed.

Next, as instructed, the AI asks the student to consider several reasons for that failure and to think of how that failure may be prevented and finally presents the student with a chart summarizing the conversation:

AI as Coach: Risks

Confabulation risks are not as severe in coaching, which are designed to stimulate thinking on behalf of the student. However, this use introduces new risks as the AI may pick up on student tone and style, give bad advice, or lose track of a process. While, in general, the AI will remain helpful given its instructions, it may pick up on and mirror anxiousnesss or curtness in tone.

Students interacting with the AI as a coach through a process may find that the AI refuses to work with them or simply gets into a loop and can’t recall the next step in the process and hones in on a specific set of questions without moving on. Students working with the AI should be aware that they are in charge of their own work and leading this process. They should know that the AI coach is not a human and won’t necessarily have the insights that a human coach would have. They can redirect the AI at any time or start again. This exercise should ideally be completed in teams in a classroom with oversight so that instructors can remind students of the process and goals for the process ahead of time and as they progress, urging students to direct the AI, or simply to “try again” if their prompt isn’t working. Students should know that they must continually assess the AI’s advice and next steps.

AI as Coach: Guidelines for Instructors

It’s important to note that although AI coaches show a lot of promise, if you decide to have students work with this coach, you should try it yourself. You might also try it across more than one Large Language Model. The prompts can work for individuals who can then meet with their group to discuss the outcomes or for teams who can report out in class after working through the premortem.

If you decide to use it in class or ask students to use it and report back provide them with guidelines so that they can a) take advantage of the coach b) learn to work with the tool.

Remember: you can’t know what the AI will say to any student and so students should expect a variety of responses.

Below is a sample set of guidelines for students.

AI as Coach: Instructions for Students

When interacting with the AI-Coach, remember:

It may not work the first time you try it. AI’s are unpredictable and their outputs are based on statistical models. This means that any time you try a prompt you’ll get a different result, and some prompts may not at any given time. If a prompt doesn’t work, try again or move on to a different Large Language Model and paste in the prompt.

It’s not a coach, but it may feel like one. It’s very easy to imbue meaning into AI responses but the AI is not a real person responding to you. It is capable of a lot, but it doesn’t know you or your context. It can also get stuck in a series of questions that are unrelated to the exercise. If that happens, tell it to move on, or just try it again.

It can make “hallucinate” or make things up. Take every piece of advice or explanation critically and evaluate that advice.

You’re in charge. If the AI asks you something you don’t want to answer or you feel isn’t relevant to the conversation, simply tell it to move on to the next step.

Only share what you are comfortable sharing. Do not feel compelled to share anything personal. Anything you share may be used as training data for the AI.

If the prompt doesn’t work in one Large Language Model (LLM), try another. Remember that its output isn’t consistent and will vary. Take notes and share what worked for you.

Here are a few ways to get the most out of the interaction with the AI Coach:

• Share challenges with the AI Coach and ask directly for advice. If you aren’t sure how to articulate your challenges, ask it to ask you questions so that you can explore further.

• Give it context. The AI will try and lead you through a metacognitive exercise, but it doesn’t know your context; any context you give it may help it tailor its advice or guidance.

• Ask questions and seek clarification. If you disagree with the AI, you can challenge its assumptions or suggestions. You’re in control of your own learning journey.

Share all of your interactions with me and briefly discuss what you learned from using exercise. How well did it work? Was the AI coach helpful? Did anything surprise you about the interaction? What are some of your takeaways in working with the AI? What are some of your takeaways from the exercise itself?

AI Coach: Build Your Own

To build your own metacognitive coach, start with the learning goal for individuals or teams:

What do you want students to reflect on? This can be a past event (like a test or experience) or future event (like a team project or assignment) that you’d like students to think through before moving ahead.

Tell the AI who it is. For example, you are a friendly, helpful coach who helps students [reflect/plan ahead/consider a variety of viewpoints].

Tell the AI what you want it to do. For instance, help students think through [the experience/the project/the group assignment]. Look up research [by specific researcher] about the topic.

Give it step by step instructions. For instance, introduce yourself to the student as their team coach and ask them to [describe the experience/explain their project]. Wait for the student to respond. Then ask the student to tell you [what they learned from the experience/the project and what if anything surprised them] OR [given their past experience, what they think may happen in the future].

Give it examples. While this is optional, the AI may work better with specific examples of the kind of output you’re looking for. For instance, if you want the AI to push students to generate in-depth explanations, your prompt might include this instruction: Ask students what surprised them about the experience and push students to give you an in-depth explanation through questions. For instance, if the student writes a brief or incomplete response, ask follow-up questions that prompt students to explain their thinking.

Add personalization. Add specific details about the event or project and give the AI context. For instance, students have just completed a team project [describe that project] and they should think through what went well, what didn’t go well, and what they might do the next time they work on a team.

Consider how you’d like to challenge students. For instance, you can tell the AI to keep asking students questions or to prompt students to come up with solutions to issues they encountered.

Final Step: Check your prompt by trying it out with different Large Language Models and take the perspective of your students – is the AI helpful? Does the process work? Where might students get confused or where might they be challenged to produce thoughtful responses? You can consider asking individuals to complete the exercise or teams to do so.