🔍 Read the full analysis: Did A Matrix Edit Help Qwen Escape Tabs In JSON? The Score Went 49 To 70 on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
An OpenAI GPT-6 Astra agent made a rank-one edit to one matrix in Qwen3-4B-Instruct-2507. On a frozen 120-case synthetic test of preserving special characters in JSON, exact answers rose from 49 to 70, with 21 cases fixed and none regressed; a separate table-extraction task remained at zero of 30. The result has not yet received an independent review.
An OpenAI GPT-6 Astra agent edited one matrix in the small open-weight model Qwen3-4B-Instruct-2507, raising its exact-answer score from 49 of 120 to 70 of 120 on a frozen synthetic test of copying special characters into JSON. The reported gain was 21 cases, with no test cases that had been correct becoming wrong; the experiment remains narrow and has not yet been independently reviewed.
The experiment targeted a specific failure: copying short text containing tabs, line breaks, quotation marks or backslashes into JSON without changing any characters. A real tab must be represented inside JSON as the escape sequence \t, and parsing the output must recover the original text exactly. Responses that were valid JSON but altered or lost the source text counted as incorrect.
According to the experiment report, the agent calculated a rank-one update to the output projection of the model’s final MLP layer. It did not use gradient training. The report says the other 397 stored tensors were bit-identical to the original model. The final test was frozen before the result was measured, and the runs are publicly available on Lexassay, the report says.
The gain did not extend to every task described. On a separate test involving extraction of a table row, the model scored zero of 30. On the 120-case JSON test, 50 answers remained wrong after the edit. Earlier screening results also showed 16 correct answers out of 32 both before and after a more explicit escaping prompt, according to the source report.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in JSON Fidelity
The experiment offers a measurable result for a targeted edit to a language model’s weights: the edited version recovered the exact source text in 21 additional test cases on the frozen JSON benchmark. That matters because small serialization errors can make structured outputs unreliable. A tab copied as a raw character, omitted, or changed to spaces may cause downstream software to misread data even when a response otherwise looks plausible.
The finding is not evidence that the edit makes Qwen generally more capable or reliably handles JSON in real applications. The benchmark used synthetic cases, covered one defined behavior and left 50 of 120 answers incorrect. The unchanged result on table-row extraction also shows that this edit did not solve an unrelated task. The score is best read as a limited result for a known failure mode, not a broad model upgrade.
Products Worth Considering
FLEXISPOT EN1 Electric Standing Desk, 48 x 24 in, Black
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, White Silver
Dell UltraSharp U4025QW 40 5K2K WUHD Curved LED Monitor (21:9)
As an affiliate, we earn on qualifying purchases.
A Follow-Up to an Earlier Failure
This was the third experiment in a series of AI-assisted weight-editing work described by ThorstenMeyerAI.com. In an earlier attempt, GPT-6 Astra tried to improve Qwen’s extraction of fields from synthetic business messages. On that experiment’s frozen final test, both the original and edited versions scored 97 of 200. The report says both also produced 47 invalid answers, often because raw tab characters appeared inside JSON strings.
That shared error prompted the next experiment’s narrower focus on JSON escaping. The reported 49-to-70 comparison therefore followed an identified failure rather than a broad search of model weaknesses. The source also describes an earlier, separate edit to a local model called Ornith; it says that edit increased the number of published benchmark tasks answered but did not change the legal score. These prior trials provide context, but do not independently verify the Qwen result.
The source says GPT-6 Astra, operating as a Codex agent with extra-high reasoning effort, planned and ran this experiment, including inspecting the weights, creating tests and producing the report. Anthropic’s Claude was not involved in this experiment, according to the account, although a Claude agent later drafted the article from experiment records and checked its figures. The source identifies the work as an internal AI experiment, not legal advice.
“The starting hint for experiment 3 came from the previous attempt.”
— The experiment report
Independent Checks Are Still Pending
The reported comparison is specific to the published synthetic test set. The source does not establish whether the gain will hold on naturally occurring text, other JSON tasks, different prompts or other model versions. It also does not provide an independent evaluation of the edited weights: the account explicitly says the experiment has not yet been checked by a second AI, unlike an earlier trial in the series.
The figures describe exact answers on 120 final test cases, but they do not show performance across repeated runs or establish why the matrix update produced the change. The report says the test was frozen before measurement and that results are public, but the provided account does not describe a replication by an outside researcher. The separate zero-of-30 table-row result is a distinct task and should not be treated as part of the 120-case score.
Replication Would Test the Result
The immediate next step is independent scrutiny of the public runs and edited model, followed by replication on a separate set of cases. A useful check would test whether the 21-case improvement persists when examples differ from the synthetic test data and whether it affects other kinds of structured output.
Until such checks are reported, the confirmed scope is limited to the source’s frozen 120-case benchmark and its stated before-and-after scores. No further experiment or review date is specified in the material provided. Readers should treat the result as an early, targeted measurement rather than evidence of a general improvement in Qwen’s JSON handling.
Key Questions
What changed in the Qwen experiment?
A GPT-6 Astra agent made a rank-one update to one matrix in Qwen3-4B-Instruct-2507. The report says the other 397 stored tensors remained bit-identical to the original.
What does the score of 49 to 70 measure?
It is the number of exact answers on a 120-case synthetic test asking the model to preserve text with special characters when placing it in JSON. The reported change is 21 more correct cases, or 17.5 percentage points.
Did the edit fix all of the model’s errors?
No. The edited model still missed 50 of 120 cases on the JSON test, and it scored zero of 30 on a separate table-row extraction task.
Has another evaluator verified the result?
Not according to the source account. It says the experiment has not yet received an independent review, although its runs are described as public on Lexassay.
Source: ThorstenMeyerAI.com
