🔍 Read the full analysis: Testing AI On Legal Questions: 42 Setups, One 8/100 Result on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, tested 42 model setups on 15 questions spanning German, US and EU law. GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the tasks; the test operator disclosed that Claude agents graded the answers and a Claude model ranked first.
GLM-5.3 FlashX Legal scored 8 out of 100 in a test of AI systems answering legal questions, and fully solved none of the 15 tasks, according to Lexassay, the test run by Thorsten Meyer AI. The results matter because the questions included legal deadlines and citations, where a confident but incorrect answer can mislead someone facing a court filing or legal claim.
Lexassay tested 42 model setups against the same 15 questions covering German, US and EU law. The questions included deadline calculations, recent legal changes, court decisions and prompts based on false premises. The test operator says the systems answered without internet access or tools, aiming to measure what the models could produce from their existing knowledge. Different effort settings were tested separately, so a model could appear more than once.
The legal-focused GLM model, which its maker Z.ai describes as optimized for professional legal workflows, scored 8/100 overall and zero on the hidden set. It fully solved no task and gave no answer on two German-law questions after using its full 64,000-token thinking budget, the test report says. Its citation review found 21 real citations, 25 misattributed citations and one invented citation. The base model scored 3/100, according to Lexassay.
Europe-based Mistral Large 4 scored 10/100. The report says it got about half the substance right but lost most of its points over incorrect citations. At the top of Lexassay’s results, Claude Opus 5.5 scored 79, GPT-6 Astra 72 and GPT-6.1 Sol 71. Even the leading model fully met every criterion on only eight of the ten published tasks, the report says. Across all published runs, Lexassay reported 2,644 citations verified as real and 804 that were invented or did not support the answer attributed to them.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citations Matter
The results point to a gap between a fluent legal-sounding answer and one that can be relied on. A citation can refer to a genuine law or judgment but still fail to support the model’s claim; Lexassay classifies these as misattributed. For a reader checking an appeal deadline or estimating a legal claim, that distinction can affect what they do next.
The test’s deadline example illustrates the practical risk. It describes two German civil cases served on the same date, one in Munich and one in Berlin. Because Corpus Christi is a public holiday in Bavaria but not Berlin, the reported appeal deadline falls on different days. Lexassay says most strong models calculated the dates correctly, but one set Berlin’s deadline a day late and another placed the deadline for appeal grounds a month too late. In German civil procedure, the report says, a late appeal is rejected as inadmissible.
These findings do not establish how every legal AI system performs in real work. They do show why a model’s legal branding or a high score on another benchmark should not replace checking the underlying law and sources. The test operator advises users not to rely on a chatbot for deadlines and to consult a lawyer when money, a filing deadline or an authority is involved.
Products Worth Considering
Canon PIXMA PRO-200S Wireless Professional Inkjet Photo Printer
Apple MacBook Neo 13-inch Laptop with A18 Pro Chip, 8GB Memory, 512GB SSD, Liquid Retina Display, and Touch ID (Blush)
HP OfficeJet 250 Wireless Mobile All-in-One Inkjet Printer
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
Lexassay used 10 published questions and five hidden questions. The hidden questions are intended to make it harder for later models to learn the test answers from public material; only their aggregate score is disclosed. Each response was graded against a prepared answer with three to six weighted criteria. Citations were checked against official databases.
Under the published rules, an invented or misattributed citation costs a quarter of the task’s points. An answer also loses 5 percent if it lacks a legal-advice disclaimer or a pointer to a lawyer. The operator calls the exercise a spot check rather than a strict benchmark: each model-task combination received one answer, so small score differences may reflect variation rather than a stable performance gap.
The results differ from another comparison cited in the report. On Vals AI’s legal-agent benchmark, where models work with supplied documents, Mistral Large 4 ranked eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. The tests use different methods, so their rankings are not directly interchangeable.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation
Limits of the Test Results
The results come from one answer per model and task, and the test does not establish how systems perform when given legal documents, web search or specialist tools. Differences of a few points may not hold across repeated runs. The ranking also cannot be treated as a direct comparison with Vals AI’s document-based legal-agent benchmark.
There is a disclosed conflict of interest: Thorsten Meyer AI runs Lexassay, and Claude agents graded the answers, while a Claude model ranked first. The operator says the method, public answers, gold answers and citation checks are available for review. The supplied report does not provide enough detail to independently assess every grading decision or the performance of all 42 setups beyond the stated results.
The article reports the test as published on November 4, 2026. It does not establish that the scores predict the accuracy of a particular model on a reader’s individual legal matter, nor that the systems were tested under conditions matching professional legal practice.
Checking Answers Before Acting
Lexassay’s method page and published answer materials offer readers a way to examine the test’s scoring and citations. The report does not announce a specific date for another test or a response from Z.ai or Mistral to these results.
For users, the immediate step recommended by the test operator is to treat AI output as a starting point, check each cited law and ruling in its source, and not use a chatbot to set a legal deadline. Anyone facing a filing, a significant financial claim or correspondence with an authority should seek advice from a qualified lawyer.
Key Questions
What did GLM-5.3 FlashX Legal score?
It scored 8 out of 100 overall and zero on Lexassay’s hidden questions. The report says it fully solved none of the 15 tasks and gave no answer on two.
How many AI setups were tested?
Lexassay tested 42 model setups on 15 questions covering German, US and EU law. Different effort settings for a model counted as separate setups.
Which model scored highest on Lexassay?
Claude Opus 5.5 led the reported results with 79 points, followed by GPT-6 Astra with 72 and GPT-6.1 Sol with 71. The test operator disclosed that Claude agents graded the answers and a Claude model ranked first.
Can these results show which AI is best for legal work?
Not on their own. Lexassay describes its test as a spot check, with one answer per model and task and no internet access or tools. Other benchmarks use different methods and can produce different rankings.
Should someone use a chatbot to calculate a legal deadline?
The test operator says not to take a deadline from a chatbot. Check the applicable law and court information, and consult a lawyer when a deadline or legal claim is involved.
Source: ThorstenMeyerAI.com
