🔍 Read the full analysis: A News Pipeline Meets 71 AI Configurations: The Results on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Thorsten Meyer says he tested 71 AI configurations on news-writing tasks between September 8 and October 13, 2026, using English and German editions and two AI judges. GPT-6 Astra had the highest reported scores, but Meyer kept GPT-6 Luna in production because Astra cost about 100 times more. The test also found that two fallback models produced fabricated quotes or unusable output, and they were removed from the route.
Thorsten Meyer says a test of 71 AI model configurations for news writing found that GPT-6 Astra produced the highest-quality articles, while lower-cost GPT-6 Luna remained his production choice. The results, published October 13, compare English and German articles generated from real news sources and also led Meyer to remove two fallback models after they produced fabricated quotes or invalid output.
Meyer reports that the test ran from September 8 to October 13, 2026, using the production prompt from his news pipeline. The configurations covered API and subscription models, free-tier models accessed through OpenRouter, and open-weight models running on his own hardware. Each source was turned into an English article and a German article for readers in Germany, Austria and Switzerland.
The standard test used two sources: a 6,340-character product filing and a 619-character political news item. Meyer says the shorter source better reflected much of his regular work. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, evaluated the articles for source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German articles were also assessed for language quality.
In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages, with perfect source-fidelity scores. In a further run of 12 articles, it averaged 3.93. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test, and 3.67 in the robustness run. Meyer says Luna’s source-fidelity score in that run was 4.71 and all of its articles produced valid output under the pipeline’s production repair step.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Quality Versus Per-Article Cost
The reported results show why a model’s overall reputation or benchmark ranking may not predict how well it handles a particular publishing task. For a pipeline producing many articles, quality, reliability and cost per article all affect the choice. Meyer estimates Astra at about $0.12 per article, compared with roughly $0.0012 for Luna—about a 100-fold difference—while Astra took around 50 seconds and Luna around 17 seconds.
Meyer says he judged Astra too expensive for routine news production and retained Luna, which scored lower but cleared his operational requirements. That is a decision for this pipeline, not proof that Luna is the best choice for other publishers. Different source material, prompts, languages, review procedures and quality thresholds could produce different results.
The fallback findings also matter because a system can fail even when its primary model performs well. According to Meyer, GPT-4.1 nano generated invented quotations, while gpt-oss-120b failed to produce valid articles in the tested cases. In news publishing, such failures can create factual and editorial risks if output is not caught before publication. The report does not say that any fabricated material reached readers.
As an affiliate, we earn on qualifying purchases.
How the Pipeline Was Tested
Meyer’s report is a task-specific evaluation, not a general ranking of AI systems. The models received the same production prompt and source material, and their articles were assessed against the source by two AI judges. Meyer says the strongest configurations also completed a robustness run producing 10 additional articles from real sources, mostly short ones.
The report gives 95% bootstrap intervals for the robustness results and says that, with only two articles in the standard test, differences smaller than about 0.3 points should be treated as noise. Meyer cites non-overlapping intervals for Astra and Luna in the robustness run as evidence that their quality gap was larger than ordinary variation in this test. These methods and results are described by Meyer; the source material provided does not include an independent replication.
The test also compared reasoning-effort settings for GPT-6 Luna. Meyer reports that increasing effort did not produce a reliable quality improvement. The highest setting cost about 1.43 cents per article and took 211 seconds, compared with 0.12 cents and 17 seconds at the no-effort setting. He says the small apparent advantage at high effort fell within the robustness-run intervals, so production remained on the no-effort setting.
“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer, in the report
Limits of the Reported Results
The figures come from Meyer’s own test and production pipeline. The supplied report does not provide the full article outputs, judge instructions, model configuration details for every entry, or an independent review of the scoring. It is not clear how results would change with other prompts, topics, source lengths or editorial standards.
The standard test used two sources, and Meyer himself cautions that small score differences in that stage may be noise. The robustness run included more articles but was limited to the strongest configurations. Its reported results offer additional evidence for those comparisons, not a broad evaluation of every model on every type of news assignment.
The report says the two fallback systems generated fabricated quotations or invalid output, but does not establish how often those problems would occur outside the tested examples. It also does not say whether any flawed output was published. Meyer’s account reports the system changes, but no external audit or publisher-side review is included in the supplied material.
Luna Remains the Production Model
Meyer says GPT-6 Luna remains the primary model for the news route, with the two tested fallback models removed on October 7. The supplied report does not specify what replacement fallback, if any, was put in place, or describe a new failover plan. That leaves the resilience of the revised route unclear from the available details.
The next useful evidence would be continued production monitoring and testing across a larger range of sources, including short items like those that dominate Meyer’s workflow. Readers should also look for details on how invalid or unsupported claims are detected before publication, and whether the model scores hold when topics, languages and article formats vary.
For now, the reported outcome is a practical trade-off rather than a universal model verdict: Astra led this test on writing quality, while Meyer selected Luna for its lower cost and acceptable performance. The report’s results remain specific to the tested task and period.
Key Questions
Which model scored highest in the test?
GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the robustness run, according to Meyer.
Why did Meyer keep GPT-6 Luna in production?
Meyer says Luna’s article quality was consistent enough for his production requirements at a much lower reported cost. He estimates Luna at about $0.0012 per article and Astra at about $0.12.
What did the test find about reasoning effort?
Meyer reports that higher reasoning-effort settings did not produce a reliable quality gain for GPT-6 Luna in this test. The maximum setting was slower and more expensive than the no-effort setting.
What happened to the fallback models?
Meyer says GPT-4.1 nano and gpt-oss-120b were removed from the news route on October 7 after tests found fabricated quotations from the former and no valid articles from the latter in four attempts.
Do the results show which AI model is best for all newsrooms?
No. The report evaluates models on one publisher’s prompt, pipeline and test sources. Results may differ for other newsrooms, languages, article formats or editorial checks.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
