🔍 Read the full analysis: Why Z.ai’s Legal AI Results Deserve A Closer Look: Pass Rate And Rank on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s version of Harvey’s Legal Agent Benchmark, ranking seventh among 81 models. The score means every criterion was met on about one in five tasks, while the model passed 92.5% of individual criteria in the same run. The result is a benchmark measure, not evidence that the model can handle legal work without professional review.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s version of Harvey’s Legal Agent Benchmark, placing it seventh among 81 models on the archived leaderboard. The figure counts a task as passed only when every required criterion is met, so it indicates full completion on roughly one in five benchmark tasks—not that the model got only one-fifth of its individual answers right.
Z.ai describes the model as built for legal research, contract review and drafting, including preparing initial drafts. Its developer documentation says the model underwent legal-domain mid-training and post-training, with reinforcement learning in what the company describes as tens of thousands of task environments modeled on legal workflows. Z.ai recommends using it as an agent connected to services providing laws, cases and company information.
Vals AI’s archived Version 1 leaderboard lists GLM Legal at 19.17% of tasks and $0.15 per test. In the same table, the base GLM-5.3-Flash model scored 6.67%. The difference is 12.5 percentage points, matching Z.ai’s stated improvement over the base model. Vals also reports a 92.5% criteria pass rate for the legal model; its Version 1 methodology counted a criterion only when both judges passed it.
The distinction between the task and criterion scores is substantial. A task may require a legal work product to satisfy many separate checks—such as factual conclusions, citations, deadlines and recommendations. Under Harvey’s all-pass rule, one missed criterion makes the entire task a failure, even when many other parts are correct.
Why the Two Pass Rates Differ
The paired scores show why a single benchmark percentage can be easy to misread. A 19.17% all-pass rate measures complete performance across complex tasks, while the 92.5% figure measures performance on individual criteria. The first is closer to whether a work product cleared every check; the second indicates that many individual requirements were met across the test. Neither number, on its own, establishes that the system is safe to use without review.
For law firms and legal teams, the leaderboard is useful as a structured comparison, but it does not settle procurement or deployment questions. Firms still need to examine whether the model cites real and relevant authorities, how it handles sensitive client information, and whether its outputs can be checked within existing workflows. Z.ai’s documentation presents the model’s work as drafts for legal professionals, making human review part of the described use rather than an optional detail.
The benchmark also shows that GLM Legal’s result came at a listed test cost below several higher-ranked models. That price is a leaderboard figure for the benchmark run, not a full estimate of operating costs: connecting external services, reviewing outputs and handling data may add expenses. The result therefore offers one comparison point, not a complete measure of value for a legal practice.
Products Worth Considering
Apple Mac mini Desktop Computer with M6 Chip (2026)
FLEXISPOT 40-Inch Height-Adjustable Standing Desk Converter with Keyboard Tray, Black
LG PuriCare AeroHit Smart Air Purifier for Large Rooms up to 1000 sq ft, True HEPA Filter with 360° Airflow, PM1.0 Sensor & LED Air Quality
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas. The tasks ask an agent to use matter materials to produce work products, which are assessed against more than 75,000 expert-written, binary pass-or-fail rubric criteria. Harvey says a task passes only if every criterion passes; its example of a change-of-control task includes 57 criteria.
The public leaderboard uses a private holdout set of 120 tasks, according to Vals AI, which operates the listed leaderboard using Harvey’s grading protocol. Internet access is disabled during testing. Two language-model judges, GPT 5.5 and Claude Sonnet 4.6, assess submissions, and the reported task pass rate is the average of their scores. Z.ai says its results were independently evaluated by Vals; the company’s October 13 blog post names Vals, and Vals lists the release as October 12.
On the archived Version 1 table, GLM Legal ranked seventh, behind models including Muse Spark 1.2 and Gemini 4 Argon. Its base model appeared at 33rd with a 6.67% task pass rate. These positions and scores describe that leaderboard version, not necessarily later revisions or every possible legal task.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai
Limits of the Published Results
The leaderboard result does not establish how the model performs across all legal matters, jurisdictions or live client workflows. The public benchmark describes a 120-task private holdout, but the available material does not provide the full task-level results or explain how the score might vary by practice area. The documentation page is undated, and the leaderboard figures cited here are from its archived Version 1.
Z.ai calls the model’s performance “state-of-the-art” among open-source models, but the source material does not identify the comparison set behind that phrase, and no published model weights for the legal model are identified there. The claim should not be read as a verified ranking across every open-source legal system. The benchmark’s stated evaluation supports the specific Vals leaderboard placement and its test methodology, not a broader conclusion about legal reliability.
A separate small spot check described by ThorstenMeyerAI.com involved four tasks and was not a benchmark. The writer reported finding Z.ai model citations to rulings that could not be located, while reporting no invented rulings in the tested OpenAI and Claude outputs; two Claude models cited a real ruling for the opposite of its holding. Those observations are limited to that small check and do not establish comparative error rates. It also remains unclear from the cited Z.ai pages which rules govern client-data processing in a firm’s particular deployment. The pages describe draft work for legal professionals, but the source review found no “not legal advice” notice.
What Firms Should Verify Next
The next useful evidence would be updated leaderboard results, fuller information about the model’s evaluation and clearer deployment details from Z.ai. Firms considering the system will need to ask how client information is processed, which external services receive it, and what contractual or technical controls apply. Z.ai recommends connecting the agent to services for laws, cases and company data, so those connections are also part of the data and security review.
Any pilot should test the model against the firm’s own matters and require reviewers to check citations, legal conclusions, deadlines and other high-impact details before drafts leave the organization. The current benchmark gives readers a defined result and comparison, but it does not answer those operational questions or replace case-specific review.
Key Questions
What does GLM Legal’s 19.17% score mean?
It means the model passed every required criterion on 19.17% of tasks in the Vals AI leaderboard evaluation. Under Harvey’s all-pass grading rule, a task fails if even one criterion is missed.
Why is the criteria pass rate 92.5%?
The 92.5% figure measures individual rubric criteria rather than entire tasks. A model can meet most individual checks across a set of tasks while still failing many complete tasks because each task requires every criterion to pass.
Did GLM Legal rank first on the benchmark?
No. Vals AI’s archived Version 1 leaderboard placed it seventh of 81 models. Z.ai’s “state-of-the-art among open-source models” wording refers to a comparison set that is not specified in the cited material.
Does the score show the model is ready to give legal advice?
No. The benchmark measures performance on a defined set of legal-agent tasks; it does not establish reliability across all matters or remove the need for professional review. Z.ai describes the output as draft work for legal professionals.
How does GLM Legal compare with its base model?
On the same archived leaderboard, GLM Legal scored 19.17% and GLM-5.3-Flash scored 6.67%, a difference of 12.5 percentage points. These are results from that benchmark version, not a guarantee of performance in other settings.
Source: ThorstenMeyerAI.com
