🔍 Read the full analysis: ChatGPT Helped Consultants Perform Data Science Tasks on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
In a randomized trial at Boston Consulting Group, consultants using ChatGPT performed better than a control group on coding, statistics and prediction tasks. The advantage did not show up on a follow-up technical test without AI, and the researchers describe the boost as temporary.
A randomized trial involving Boston Consulting Group consultants found that participants using ChatGPT performed better than a control group on coding, statistics and prediction tasks outside many participants’ existing skills. But on a later technical test taken without ChatGPT, researchers found no evidence that the AI group had gained an advantage, suggesting the improvement helped people complete work with the tool rather than learn to do it independently.
The study, published in Nature Human Behaviour on October 26, 2026, compared consultants who received 15 to 20 minutes of training on prompting ChatGPT with a control group trained to use Google and resources such as Stack Overflow and Khan Academy. The ChatGPT group used GPT-4. Participants completed two of three 90-minute tasks: writing Python code to clean and combine datasets, evaluating a statistical model, or building a prediction model for international football matches.
In the researchers’ reported comparisons, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. The study also compared participants with 44 BCG data scientists who completed the tasks without AI. On average, consultants using ChatGPT remained below that expert benchmark on all three tasks. The authors report that the AI group finished coding and prediction faster, while the time difference on statistics was not significant.
After the work tasks, participants answered technical questions on related topics without access to ChatGPT. The authors found no evidence that the treatment group did better than the control group on that test. They also report that ChatGPT users became more confident that the tool could solve problems it could not; the paper describes that finding as exploratory, not a definitive result.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
A Work Boost, Not a Skills Test
The findings speak to a practical distinction for employers: better output with AI is not the same as lasting employee skill. ChatGPT helped consultants produce stronger work during the tasks, including on technical problems that many did not know how to handle. Yet the follow-up did not show that the experience left them better able to answer related questions without the tool.
For companies using AI in technical work, that distinction affects how they evaluate training and performance. A task completed successfully with a chatbot may reflect the combined capability of the worker and tool, rather than independent expertise. Employers may need to assess work quality, verify AI-generated output and test skills separately when those skills matter without AI. These are implications of the study, not evidence that every workplace or AI system will produce the same result.
The paper’s evidence also has limits: it concerns particular tasks, a short training session and consultants at one firm. It does not establish how durable learning might be after longer instruction, repeated practice or other forms of AI use. The results do, however, caution against treating immediate productivity gains as proof that employees have been upskilled.
Products Worth Considering
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, White Silver
FlexiSpot E6 Electric Standing Desk, 55 × 28 in, Bamboo Top, White Frame
Dell UltraSharp U2424H 24-Inch FHD Monitor
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Set Up
The experiment was preregistered before data collection, which took place in late March and early April 2024. Researchers allocated 986 consultants equally between the ChatGPT and control groups. Of those, 573 began the survey and 487 submitted work on both assigned tasks; those 487 make up the study’s main analysis sample. All participants had at least a bachelor’s degree, and the authors report that most said they had no coding experience.
The tasks were developed by BCG data scientists alongside OpenAI’s Economic Impacts Research team. The design aimed to make ChatGPT useful while preventing it from correctly answering every question on its own, so simply copying a response would not suffice. The researchers used the performance of 44 BCG data scientists working without AI as an expert benchmark, then tested participants on related technical questions after removing access to ChatGPT.
One subgroup result was stronger: consultants the paper classified as “competent coders” scored about five percentage points below data scientists on coding and statistics while using ChatGPT. The authors found no evidence of a performance difference from experts on those two tasks for that subgroup. On prediction, the subgroup remained 20 points below. Those comparisons apply to the specified subgroup and tasks, not to the full group of consultants.
““Generative AI and the temporary upskilling of knowledge workers””
— Wiles, Krayer and co-authors, in the study
What the Trial Cannot Establish
The results do not show whether a longer training program, sustained use or different teaching methods could produce skills that persist without AI. Nor does a test of five technical questions establish how well participants might perform on every real-world task after the trial. The study tests a specific setup at one consulting firm, using particular tasks and GPT-4; it cannot establish that the same effects apply to all workers, employers or AI tools.
The authors’ confidence finding also needs care: they label it exploratory. The supplied study summary does not give further detail on how that confidence changed or whether it affected later work. It is also not clear from the reported results whether the performance gains would hold over a longer period or in normal workplace projects with different levels of oversight.
How Employers May Use the Findings
The paper reports results from a completed trial, not a forthcoming experiment or policy change. The next practical step for organizations is to distinguish AI-assisted task performance from unaided capability when evaluating training and technical work. They can also monitor whether outputs are accurate, rather than assuming that an employee’s successful use of a chatbot proves independent expertise.
Further research would be needed to test whether longer or more structured AI training leads to lasting learning, and whether similar effects appear across different jobs and tools. Until then, the BCG trial supports a narrower conclusion: in these tasks, ChatGPT improved performance while available, while the researchers detected no corresponding advantage on the later test without it.
Key Questions
What did the BCG study find?
Consultants using ChatGPT performed better than a control group on coding, statistics and prediction tasks. On a later technical test without ChatGPT, the authors found no evidence of an advantage for the AI group.
How much higher were the ChatGPT group’s scores?
The study reported scores 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. These are comparisons between the two study groups on the specified tasks.
Did the consultants reach the level of BCG data scientists?
Not on average. The ChatGPT group remained below the 44 data scientists’ benchmark on all three tasks. A subgroup of consultants with coding experience came close on coding and statistics, but remained behind on prediction.
Does the study prove ChatGPT cannot teach lasting skills?
No. It found no evidence of an advantage on one follow-up test taken without ChatGPT after a short training session and task period. The study does not establish what longer training or repeated practice might achieve.
Source: ThorstenMeyerAI.com
