📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large Italian-language LLM from scratch, but scored near chance on academic benchmarks, raising questions about the scale of native-language investment needed for country-specific AI. This challenges assumptions in the European sovereign-LLM movement.
Italy’s Minerva-3B, a large sovereign language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, a result that questions the effectiveness of current scaling strategies for country-specific AI models.
Developed by Sapienza University of Rome’s NLP group led by Roberto Navigli, Minerva was built using Italy’s national supercomputing infrastructure, CINECA, and funded through Italy’s PNRR initiative. The project aimed to produce a high-performing Italian-language model with open weights, data, and code, contrasting with European efforts like Portugal’s AMÁLIA, which layered specialization onto multilingual foundations.
Despite the significant investment and impressive technical performance on some benchmarks, Minerva-3B’s score of 4.9% on the INVALSI Italian school exams marks a near-chance result, indicating a disconnect between model size, training data, and language-specific knowledge depth. Researchers emphasize that larger datasets and more parameters are crucial for handling complex language tasks, even at the national level.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Building A large language model with Ai: A Practical Guide to Structuring LLM Systems from Scratch Using Reverse-Engineering Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Minerva’s Benchmark Performance for European AI Strategies
This development suggests that simply scaling up native-language data and parameters may not be sufficient for achieving country-specific AI expertise at the academic or practical level. It challenges the assumption that larger investments automatically produce deeper language understanding, raising questions about the optimal approach for sovereign-language models in Europe.
For policymakers and AI developers, Minerva’s results highlight the need to reconsider resource allocations and technical strategies, emphasizing quality and targeted data over sheer size. The findings also suggest that European efforts might need to confront the reality of scaling limitations more directly, shaping future investments and research priorities.
European Sovereign-LLM Approaches and the Scaling Dilemma
The European sovereign-LLM debate has centered on whether to build models from scratch or adapt multilingual models through continuation training. Portugal’s AMÁLIA, for example, layered European Portuguese onto a multilingual foundation, but its public weights and actual performance on complex tasks remain limited.
Italy’s Minerva, by contrast, trained from scratch on a massive dataset with a high proportion of Italian content, aiming for country-specific expertise. While the project produced models outperforming comparable multilingual models on some benchmarks, its low score on the INVALSI tests reveals a fundamental challenge: size and data alone may not suffice for deep language understanding at the academic level.
“While the pre-training dataset composition is important, the overall size of the dataset and the number of parameters are more crucial for handling complex language tasks.”
— Research team
Unresolved Questions About Scale and Language Knowledge Depth
It remains unclear whether further increasing model size, data quality, or specialized training will significantly improve performance on complex, country-specific tasks. The results are based on one benchmark, and broader evaluations are ongoing.
Additionally, the generalizability of these findings to other languages and models at different scales is still being studied, leaving open questions about the most effective strategies for sovereign AI development.
Next Steps in Evaluating and Scaling European Sovereign Models
The Minerva team is continuing to iterate on their models, including ongoing experiments with continual training and different data compositions. Future evaluations will test whether increased scale or refined methodologies can bridge the knowledge gap identified in current benchmarks.
Policymakers and researchers will likely reassess investment strategies, emphasizing targeted data collection and model scaling at the national level, informed by these empirical findings. Broader benchmarking and cross-model comparisons are expected to follow.
Key Questions
Why did Minerva perform poorly on the Italian school-exam benchmark?
The evaluation suggests that, despite large-scale training, the model lacks the depth of country-specific knowledge needed for complex academic tasks, highlighting limitations of current scaling strategies.
Does this mean smaller models are better for country-specific tasks?
Not necessarily. The findings indicate that size alone isn’t sufficient; targeted data, training methodology, and model architecture are also critical for achieving country-specific expertise.
What does this mean for Europe’s AI sovereignty efforts?
It suggests that European projects may need to re-evaluate their strategies, focusing on data quality and scale, and understanding the limits of current approaches to truly develop deep country-specific AI models.
Will increasing model size improve performance?
Future research will determine if larger models trained with more or better data can overcome current limitations; ongoing experiments aim to clarify this.
Source: ThorstenMeyerAI.com
