FREIA: Unsupervised RL Enhances LLM Reasoning with Adaptive Rewards

What changed

The core innovation with FREIA, introduced on May 7, 2026, is its ability to enable unsupervised reinforcement learning (RL) for large language models (LLMs) that can adapt to the model’s evolving reasoning capabilities during training. Existing unsupervised RL methods often struggle with this adaptability, potentially misdirecting policy optimization when ground-truth supervision is absent. Reinforcement learning is a standard post-training mechanism for improving LLM reasoning, but its performance remains sensitive to the design of the reward function that drives policy optimization, as noted in recent research.

FREIA addresses this by integrating two key components: Free Energy-Driven Reward (FER) and Adaptive Advantage Shaping (AAS). FER dynamically adjusts rewards to balance the need for the model to reach consensus in its outputs with the need to explore new solution paths. This is a crucial distinction from static reward functions that might not account for the model’s current state of understanding or its capacity for novel problem-solving. AAS further refines this by adaptively adjusting the learning signals based on the statistical characteristics of the rewards sampled during training. This means the algorithm isn’t just reacting to rewards but is learning how to interpret and use those rewards more effectively over time.

How it works

FREIA operates on the principle that an LLM’s reasoning capabilities are not static but evolve throughout training. To leverage this, it employs a two-pronged adaptive mechanism within an unsupervised reinforcement learning framework. Reinforcement learning, in general, focuses on an agent learning to make decisions by maximizing future rewards, even if immediate rewards are negative, making it suitable for problems with long-term consequences.

First, the Free Energy-Driven Reward (FER) component is inspired by the Free Energy Principle, a concept from neuroscience and physics that suggests biological systems tend to minimize “free energy” or surprise. In FREIA’s context, FER adapts the reward signal to balance two competing objectives: achieving consensus in the model’s outputs and encouraging exploration. When an LLM is highly uncertain or consistently producing diverse, non-convergent answers, FER might prioritize exploration, rewarding actions that lead to novel or less common solutions. Conversely, as the model’s reasoning improves and it starts to converge on consistent, high-quality answers, FER might shift to reward consensus, reinforcing the most probable correct paths. This dynamic adjustment prevents the model from getting stuck in local optima or endlessly exploring when a stable solution space has been found.

Second, Adaptive Advantage Shaping (AAS) refines the learning process by adjusting the “advantage” signal, which is a core concept in actor-critic RL algorithms. The advantage function estimates how much better an action is than the average action in a given state. AAS dynamically modifies this signal based on the statistical properties of the sampled rewards. For instance, if rewards are highly variable or noisy, AAS might temper the learning signal to prevent over-correction. If rewards are consistently high or low, it might amplify the signal to accelerate learning. This adaptive scaling ensures that the model’s policy optimization is robust and efficient, even in the absence of explicit ground-truth labels, which is particularly challenging in unsupervised settings. By continuously re-calibrating how the model interprets and reacts to its own generated rewards, FREIA enables a more stable and effective self-improvement loop for LLM reasoning.

Why it matters for operators

For operators building or deploying LLM-powered applications, FREIA represents a significant step towards more robust and self-improving AI systems, particularly where human supervision for fine-tuning is costly or impractical. The ability of an LLM to adapt its learning signals and reward functions without explicit human feedback means that models can potentially continue to improve their reasoning capabilities in deployment, or with minimal human intervention. This is crucial for applications requiring high levels of logical consistency, such as code generation, complex data analysis, or scientific discovery tools, where the “correct” answer might not be immediately obvious or easily labeled by humans.

The immediate implication is that operators should start considering unsupervised RL techniques as a viable path for post-deployment model refinement, moving beyond traditional supervised fine-tuning or even RLHF, which still relies on human judgment. While the paper focuses on mathematical reasoning, the underlying principles of adaptive reward shaping and balancing exploration with consensus are broadly applicable. Operators should investigate how these concepts could be applied to their specific domains, especially those with sparse or ambiguous ground truth. This could lead to more resilient LLMs that are less prone to “catastrophic forgetting” during updates and more capable of handling novel, out-of-distribution problems. We believe this trend points towards a future where foundation models are not just pre-trained and fine-tuned, but are continuously learning and self-optimizing in production environments, reducing the operational overhead associated with constant re-training and human data labeling.

Benchmarks and evidence

FREIA’s effectiveness was empirically evaluated across nine datasets spanning three distinct reasoning tasks. The research highlights its superior performance compared to other unsupervised RL-based baselines. A notable area of improvement was in mathematical reasoning tasks.

• Mathematical Reasoning: FREIA achieved an average improvement of 0.5 to 3.5 points in Pass@1 scores when using the DeepSeek-R1-Distill-Qwen-1.5B model. This metric indicates the percentage of problems solved correctly on the first attempt without human intervention.
• General Performance: Across the tested reasoning tasks, FREIA consistently outperformed other unsupervised RL methods, demonstrating its robust adaptability.

These results indicate that FREIA’s adaptive reward and advantage shaping mechanisms are effective in guiding LLMs toward better reasoning capabilities, particularly in domains requiring precise logical inference.

Risks and open questions

• Generalizability Beyond Reasoning: While FREIA shows promise in reasoning tasks, its effectiveness in other LLM applications (e.g., creative writing, summarization, conversational AI) remains an open question. The “consensus” aspect might need redefinition for more subjective tasks.
• Computational Overhead: Adaptive reward functions and advantage shaping can introduce additional computational complexity. The practical implications for training time and resource consumption, especially for larger models or continuous learning scenarios, need further investigation.
• Interpretability of Adaptive Rewards: Understanding precisely how FER and AAS influence the model’s learning trajectory can be complex. The dynamic nature of these mechanisms might make it harder to diagnose specific failure modes or biases compared to static reward functions.
• Defining “Consensus” in Unsupervised Settings: The Free Energy-Driven Reward balances consensus and exploration. Defining what constitutes “consensus” in a truly unsupervised setting, especially for nuanced or open-ended problems, could still be a challenge. The risk is that the model might converge on a suboptimal “consensus” if the exploration phase isn’t sufficiently robust.