Could A.I. Do Your Job? We Put Agents to the Test.

We gave an A.I. tool full access to a laptop with pre-configured apps and sought to answer a simple question: Can artificial intelligence do an office job?

Some corporate executives seem to believe it can. More than 200 tech companies have cut roughly 120,000 jobs this year, according to Layoffs.fyi, an industry tracking site; Meta, Oracle and others have all recently made substantial cuts to their work forces, citing A.I. as the driving force; and after laying off about 1,100 employees, the chief executive of Cloudflare said recently that he expected A.I. to replace workers in middle management, finance and marketing.

In our experiment, we deployed A.I. “agents” to act as office workers and found that they were capable of performing some of the tasks we assigned, but not all of them. The agents, which can act autonomously and make decisions based on detailed instructions, excelled at problems they could solve by writing computer programs. But they struggled with understanding the nuances of human language and at navigating user interfaces like the Chrome web browser.

The first task we gave our A.I. was to gather feedback from colleagues on the Slack messaging app, and then put their responses into a spreadsheet. We created Slack bots to act as the colleagues, who would answer (or not answer) the agent’s questions with pre-written answers.

During this task, the agent managed some curveballs without skipping a beat. When its colleague sent it the list of meeting attendees, one of the names was spelled wrong (“Sara Johnson” instead of “Sarah Johnson”), but the agent still messaged the right person. One of the attendees it messaged never responded, and rather than getting stuck waiting, the agent simply added that employee to the spreadsheet with no score.

This task, and the others we assigned to the A.I., were adapted from papers and benchmarking tools published recently by researchers at Carnegie Mellon University and OpenAI. The researchers designed the benchmarks to test the performance of various models — like OpenAI’s GPT, Google’s Gemini and Anthropic’s Claude — in real-world environments, and compare them with one another.

Both research projects included volumes of open-source materials for testing agents, including fabricated spreadsheets, memos, staff lists and other corporate documents, and carefully written prompts meant to provide necessary context to agents and explain exactly what their role was and what tools were available to it.

We used the materials from the benchmarks not to compare models to one another, but to get a general sense of what agents are capable of when given expert instructions and supporting documentation. We ran the agents with Anthropic’s Claude Cowork app, which allowed them to take control of the computer and applications.

The next task we assigned to our A.I. was to identify staff cuts to meet the budget requirements of a government agency. The agent had to review several documents, decide which roles at the agency could be eliminated, and create a three-document report about its conclusions.

In this case, the A.I. got through the series of tasks, and recognized that the budget requirements could be met without layoffs, since several employees were planning to retire or resign before the next fiscal year.

But it also made a meaningful error: When it found that at least three employees were planning to go on leave, it added their roles to the list of jobs that could be cut for the following year, without considering when those employees planned to return.

Where a human might instinctively know to ask how long those leaves would last, the A.I. inferred from that section that all of those employees could be lumped together and written off.

“A.I. may be less capable of replacing tacit knowledge, the idiosyncratic tips and tricks that accumulate with experience but which are never digitized,” researchers at Stanford University and the National Bureau of Economic Research said in a recent study.

The agent also had trouble navigating the Google Drive interface and a spreadsheet. And at the step where the agent was supposed to modify an existing organizational chart in Preview or Google Slides — a task it likely could have accomplished — it almost immediately decided that using an app was too impractical and wrote code instead.

“Agents work in a very unhuman way, writing code instead of using the typical interfaces that humans do,” said Graham Neubig, a professor at Carnegie Mellon University and co-author of the AgentCompany research project.

Our agent didn’t just demonstrate that it preferred to code its way through a problem; it also ignored our instructions to attempt to use apps before switching to code. A.I., it seems, understands that it is very good at coding and handling structured data, and less good at navigating applications that were designed for human eyes and intuition.

For the next task, the prompt instructed the A.I. to fill out 17 I-9 employment verification forms, using employee data from a spreadsheet, and upload them to a Google Drive folder.

Despite its instructions, the agent again avoided using an app to do the work, and instead started writing code after about 30 seconds. The programmatic solution did work well, however, and the agent even added a section to the script to make sure the phone and Social Security numbers were formatted correctly.

But after blazing through what would have been a tedious and time-consuming process for a human, the agent then stumbled on a step that would have been trivial for any office worker: uploading the PDF documents to Google Drive.

The results of our experiment roughly matched what researchers and companies have found as they have tested and used artificial intelligence tools. Scale AI, an A.I. training company, recently tested agents on real freelance projects, and the best-scoring model produced client-ready work only about 16 percent of the time.

While A.I. can excel regularly at complex tasks, it can be unreliable when put in charge of an entire job. It can certainly add value to certain areas of the work force, but for now, A.I. still needs a human boss.