PROBLEM & PROPOSAL

Large Language Models present opportunities and challenges for the teaching of writing. Based on pattern-recognition, a LLM’s ability to query, interact, and present information conversationally, in a natural language interface, means essential features of the writing experience – developing one’s voice and understanding through a set of active thinking and drafting exercises – can be nearly instantly modeled and represented, seemingly co-opted, by a machine learning system that’s remarkably human-like and solicitous in its word choice, tone and etiquette.

This emergent expression of AI suggests the promise in Bloom’s Two-Sigma Problem[1] may soon be attainable in zero marginal cost mentors-by-our-side. At the same time, black box platforms and large foundation models are being released by tech companies able to deploy systems broadly via existing services, device relationships, and installed customer bases. For now (and by design), this enormous software release functions like a top-down experiment in requirements gathering, standards definition, and market development on terms set out by technology companies and their programmers. In addition to discussions in the CS community about AGI risk, questions of social impact and value, related concerns of algorithmic bias, and a considered understanding of user stories are beginning to surface in response to the rollout.

Understandably, many of the use cases and objectives appearing in the education sector focus on AI co-pilots or tutors able to provide targeted, personalized support and assessment, helping students “level up” skills and productivity. Meantime, K12 stakeholders, from administrators to teachers, students[2], and their families, navigate institutional discussions about plagiarism policies and AI compliance while carving out time to engage with the tools, testing, refining, and hacking routines they might use to elevate themselves as a thinkers, creators, and writers. For a network and PLC with unique assets aligned to the production of text, story, and argument, namely teachers of writing and their students, this situation presents a set of high-value propositions:

      How might this community of teacher leaders use these tools on a regular basis to a) develop their own practice as educators (in professional development settings and in their classrooms) and b) contribute meaningfully to the improvement of generative AI systems that support the developing the writer versus primarily predicting, suggesting, correcting, or enhancing the text?

      What would be the best technical approach[3] – continued prompt experimentation and work through large foundational models and their APIs, and/or data curation, selected fine-tuning, and the creation of unique, contextual reward mechanics running on smaller scale, open source LMs? – for accomplishing this in an accessible user experience and intentional order of operations that non-technical educators can easily engage with and advocate for?

      Might this unfold by design, with a first principle being: in addition to super-charged efficiency for remixing structure, voice and interpretation, bots or agents will function explicitly as empathetic, active listeners focused on helping to develop mastery of the writing process with and for their learner partners?

PROJECT OVERVIEW

Given the expense and scale in LLMs, related industry concerns about the role of foundational models relative to their market position[4], and a committed teaching and learning interest in documenting, curating, and elevating the domain expertise writing instructors bring to their practice, support a cohort of skilled educators focused on a fine-tuning demonstration with a publicly available language model[5] that can support teachers and students with instructional resources and coaching, respectively. Iterative work in this setting, with prompt sets, chatrolls, and fine-tuning,[6] would involve:

      Requisite inventory to assemble resources, sample text, data cleanup, and prompt models to gather and organize professional research and curricular material so it can be usefully queried and formatted for knowledge base extraction.

      The creation of similarly aligned taxonomies in chatroll dialogues and conversational datasets organized for specific learning situations and assignment types, with unique frameworks and protocols for bot feedback for learners – eg: coaching advice, modeled on expert writing instruction.

This will include explorations of syntax and response protocols targeted to differentiation and the several ways learners explore and acquire knowledge and understanding; additional insight may be gleaned from CBT-aligned wellness applications[7] (systems such as Woebot, Wsya) and research projects such as “Emphathetic Dialogues”[8] which seek to capture parameters for underlying emotions in 1:1 conversational settings, a relevant consideration for mentor/student feedback related to the development of ideas and the sharing of work in progress.

      A stipended multi-year program for participating educators – curriculum fellows -- minimally running for 18 months, across two cycles of summer workshops, with a set number of school year convenings for training and testing, focused on defining criteria, sorting prompt models and conversational datasets, curating a collection of instructor-created exercises, activities, and related pacing guides the community can use to pressure test AI practices in writing settings.

      Funding a small engineering and design team, including NLP/ML advisorial and research roles, to manage and promote this work, including presentation at selected conferences, and the release of a dedicated chatbot to support the teaching of writing.

[1] On Bloom's two sigma problem: A systematic review of the effectiveness of mastery learning, tutoring, and direct instruction

[2] High School Senior: Why aren’t more teachers embracing AI?

[3] “Understanding Size Tradeoffs with Generative Models - How to Select the Right Model (GPT4 vs LLAMA2)?

[4] Google: “We Have No Moat, and Neither does OpenAI”

[5] Hugging Face and Hugging Face, Github and more unite to defend Open Source EU AI legislation

[6] Fine-tuning work assumes that by initializing a LM with a set of pre-trained weights one can then train it on task- and domain-specific data further optimizing the unique LM implementation.   As ChatGPT puts it: “...‘weights’ in a language model architecture are learned parameters that represent the model’s knowledge. During pre-training, the model learns the initial set of weights by exposure to large amounts of text data [as with the development of foundational LLMs], and during fine-tuning [what the working group would accomplish­­­­­­­­], these weights are further adjusted to make the model more effective for specific tasks.”

[7] WYSA and Woebot. A Survey of Mental Health Chatbots Using NLP. 

[8] EmphatheticDialogues. The EmpatheticDialogues dataset is a large-scale multi-turn empathetic dialogue dataset collected on the Amazon Mechanical Turk, containing 24,850 one-to-one open-domain conversations. Each conversation was obtained by pairing two crowd-workers: a speaker and a listener. The speaker is asked to talk about the personal emotional feelings. The listener infers the underlying emotion through what the speaker says and responds empathetically. The dataset provides 32 evenly distributed emotion labels.