Metacognition and Skill Discovery in LLM Math Reasoning

The May 20, 2024 academic paper explores the metacognitive capabilities of Large Language Models (LLMs), specifically focusing on mathematical problem-solving. The core approach involves developing a method for a powerful LLM, such as GPT-4, to identify and label mathematical questions with specific skills, which are then organized into broader, interpretable categories. This process creates a Skill Exemplar Repository containing skill names matched with question-answer pairs. Experiments validate that providing an LLM with these skill labels and associated examples as in-context prompts significantly improves accuracy on challenging math datasets like MATH and GSM8K, outperforming baseline prompting techniques like Chain-of-Thought. Furthermore, the skill knowledge transferred effectively to other, less powerful LLMs and different math datasets, demonstrating the utility of this LLM-generated metacognitive framework. Source: https://arxiv.org/pdf/2405.12205