🔍 Read the full analysis: How One GPT-6 Astra Matrix Edit Improved Qwen’s JSON Tab Escaping Score on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
An internal experiment reported by ThorstenMeyerAI.com found that a GPT-6 Astra agent’s edit to one Qwen3-4B-Instruct-2507 matrix raised exact answers on a frozen JSON tab-escaping test from 49 of 120 to 70 of 120. The synthetic test showed no losses among the cases that changed, but 50 answers remained wrong, table-row extraction stayed at zero of 30, and the result has not received an independent review.
An OpenAI GPT-6 Astra agent improved Qwen3-4B-Instruct-2507’s exact score on a synthetic JSON escaping test from 49 of 120 cases to 70 of 120 by editing one model matrix, according to an experiment report published by ThorstenMeyerAI.com on November 5. The result is a limited gain on a frozen test: 50 answers were still wrong, and the model did not improve on a separate table-row extraction task.
The agent, running as a Codex session, inspected the model, selected the edit, prepared and froze the final test, applied the change and measured the outcome. The report says it calculated a rank-one update to the output projection matrix in the model’s last multilayer perceptron (MLP) layer. It did not use gradient training. The other 397 stored tensors were bit-identical to the original model.
The task tested whether the model could copy short text containing tabs and other special characters into JSON without changing the text. A literal tab inside a JSON string must be represented by the escape sequence \t; the answer counted as correct only if parsing the JSON recovered exactly the original text. On the frozen test, the edited model got 21 additional cases right, with no cases reported as newly broken, raising the score by 17.5 percentage points.
The report says a more explicit prompt did not fix the initial problem: the model scored 16 of 32 screening cases both before and after prompting. The source describes the final test cases as synthetic. It also reports no gain on extracting a table row, where both versions scored 0 of 30. The experiment was performed on Qwen3-4B-Instruct-2507, which the source identifies as an open model under the Apache-2.0 license.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in Exact JSON Output
The result matters as a small, measurable example of changing a model’s weights to address a specific failure, rather than relying on a prompt alone. Exact character preservation is relevant to software that passes text through JSON: a raw tab where an escaped tab is required can produce invalid output or alter data during parsing. The test therefore measures more than whether a response looks plausible; it checks whether the encoded text survives a round trip unchanged.
But the result does not establish that this approach broadly improves Qwen or other models. The gain applies to one model, one matrix and one narrowly defined task. Half of the final cases remained incorrect, and the separate extraction task did not improve. The source also says a second AI has not yet independently reviewed this experiment. Readers should treat the score as a reported internal test result, not as evidence of general reliability in production.
Products Worth Considering
Apple MacBook Pro MPHF3LL/A 14.2-Inch Laptop | M2 Pro 12-Core Chip 16GB Unified Memory 1TB SSD | 19-Core GPU Liquid Retina XDR Display Space
FLEXISPOT 60-Inch Executive Standing Desk with Drawers and CPU Cabinet
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
As an affiliate, we earn on qualifying purchases.
The Earlier Test Exposed Tab Errors
This was the third experiment in a week of work on modifying model weights, according to ThorstenMeyerAI.com. The first involved Claude editing a local model called Ornith. The site reported that Ornith completed more legal answers after the edit, but its legal score remained zero. In a later review, GPT-6 Astra identified issues in that work, including a resume bug, a command-line problem and side effects from weight conversion.
Before the JSON escaping experiment, GPT-6 Astra had also tried an edit to Qwen for a different task: extracting five fields from synthetic business messages. That earlier frozen test produced 97 correct answers out of 200 for both the edited and original versions. The report says both versions also produced 47 invalid answers, mostly by placing literal tab characters in JSON strings rather than escaping them. The new experiment used that observed failure as a lead for a more targeted test; it was not a broad search across all possible model weaknesses.
“The edited model got 70 of 120 test cases exactly right instead of 49.”
— ThorstenMeyerAI.com experiment report
Independent Checks Are Still Pending
The reported score has not yet been checked by a second AI, according to the source. That leaves open whether an independent review would confirm the experiment’s setup, edit and interpretation; the article does not report a replication by another researcher or team. The source says the final tests were frozen before the result was observed, but it does not provide a detailed account in the supplied material of the test-generation process, the full case distribution or performance on real-world JSON traffic.
It is also unclear whether the same edit would help on different prompts, other kinds of text, other model versions or broader structured-output tasks. The reported 21-case gain is specific to the described 120-case test and should not be read as a general error-rate reduction. The model’s remaining 50 incorrect answers and unchanged table-row score place clear limits on what this experiment establishes.
Replication and Broader Tests Needed
The next useful step is an independent review of the saved edit and frozen test results, followed by replication on additional cases that were not used during the experiment. Tests on varied prompts and real applications could show whether the gain extends beyond the synthetic examples. Separate measurements would also be needed for other structured-output tasks, including extraction and handling of different special characters.
ThorstenMeyerAI.com’s report does not state a schedule for those checks or identify a second reviewer. Until they are completed, the finding remains a single reported experiment: a one-matrix edit improved one narrow score, while broader effectiveness and reliability remain unconfirmed.
Key Questions
What changed in the Qwen model?
The GPT-6 Astra Codex agent applied a rank-one update to the output projection matrix in Qwen3-4B-Instruct-2507’s last MLP layer. The report says the other 397 stored tensors were unchanged.
How much did the JSON escaping score improve?
On the frozen test, the reported exact-answer count rose from 49 of 120 to 70 of 120, a gain of 21 cases, or 17.5 percentage points. The cases were synthetic.
Did the edit improve other tasks?
Not in the separate table-row extraction test described in the report: both the original and edited versions scored 0 of 30. The experiment does not establish improvement on other tasks.
Has the result been independently verified?
No independent review of this experiment is reported. ThorstenMeyerAI.com says the work has not yet been checked by a second AI, and the supplied material does not describe an outside replication.
Source: ThorstenMeyerAI.com
