2,500% ROI from Microsoft Office AI Copilot

Nov 03, 2023

Summary: An advertising agency experienced huge gains relative to monthly costs from deploying the AI Copilot product from Microsoft. However, the productivity gains were much lower than those seen in other studies of business professionals using AI.

The Wall Street Journal ran an article on the AI Copilot for Microsoft Office (sorry, Microsoft 365 🙄). The following data nugget was buried deep within: The marketing and advertising agency network Dentsu bought 300 licenses and discovered that each user saves 5 to 10 hours per month from using the new AI features.

Let’s do the math:

Take the average to assume that the average employee saves 7.5 hours per month.
Estimate loaded hourly cost for employees at a leading marketing and advertising agency in the U.S. to be $100. (This doesn’t mean staff get paid a $100/hr salary. Instead, this is the agency’s cost of having them around, including all the many overhead costs. While I don’t know Dentsu’s billing rates, a typical hourly billing rate for such agencies is for clients to pay $250/hr, which corresponds well with a loaded staff cost of $100/hr. The remaining $150 is spent on sales costs, management overhead, additional overhead, and a small profit.)
Multiply the two numbers to arrive at savings of $750 per user per month.
Microsoft charges $30 monthly to use the AI Copilot for MS 365.
Divide these two numbers to see that the estimated ROI from AI is 2,500%.

Pay $30/month to gain a gold bar’s worth of return on investment from deploying the Microsoft 365 AI Copilot? A no-brainer. (image by Dall-E.)

(I have no financial interest in recommending this product. I don’t know whether it’s any good since Microsoft still reserves the AI Copilot for its big enterprise customers. I’m just reporting the data. By way of disclosure, I did consult for MS Office in 2007, and a graduate student of mine from 1990 is now a Principal Designer for MS 365. Neither of these facts influenced my decision today to report on exciting data.)

Here’s another way of looking at the numbers:

Assume a 40-hour work week for these advertising folks.
This means that employees work 173 hours per month.
Saving 7.5 hours per month is 4% of those working hours.
Assuming that staff use the saved time to perform more productive work, their productivity has increased by 5%.

Mammoth ROI, Modest Productivity

ROI is extremely high because of the cheap price of the Microsoft 365 AI Copilot and the ease of integrating it into employees’ existing workflow. (Assuming that people use Office already.) At the same time, a 5% productivity gain is meager if we compare it with all other research on the productivity impact of AI. For example, the Boston Consulting Group measured a 33% productivity increase from AI. (And most other productivity research has found much higher gains.)

Why this massive gap between 5% at an advertising agency and 33% at a consulting agency? Let's dispense with the clichéd notion that ad agency folks lack gray matter. Even if you believe that Dentsu has a lower-IQ staff than BCG (one of the world’s most elite companies), that should increase the gains at Dentsu because AI narrows skill gaps and serves as a forklift for the mind.

Are all these advertising creatives and account execs airheads? I don’t think that’s the explanation for the data. (Advertising agency staff by Midjourney.)

I have 3 possible explanations for the low productivity gains from the Microsoft AI Copilot:

Microsoft’s AI Copilot is less advanced than ChatGPT, which was used in other productivity studies. This doesn’t sound plausible since Microsoft is a significant investor in OpenAI and reportedly uses GPT as its LLM. I don’t know the details of Microsoft’s product, but they may use GPT 3.5 to save on inference compute. If this is true, it could explain the low performance compared to GPT 4, which is known to be vastly superior to 3.5 but eats copious inference compute.
Dentsu doesn’t use AI well. They have a learning curve to climb, which is precisely what they are doing with this pilot project, so I only applaud them for being early. If this explanation is correct, Dentsu will enjoy much higher gains later, after adjusting their work processes to account for the new AI capabilities.
People only use MS Office for a small part of the workday. Thus, productivity gains might be 33% while people use Office, but the company only enjoys a much smaller percentage gain across the entire workday. This latter explanation seems highly plausible.

Workflow Must Change for Full AI Benefits

Note that if explanation 3 is true, people need to change their work habits to use Office more now that it’s a better productivity tool. I am sure that Microsoft would love this conclusion to be accurate. However, I only mention this as a possibility, not as a fact. More research is needed. (Sorry for the trite conclusion.)

Introducing AI into the workplace must lead to workflow changes. This is similar to how changing from steam power to electricity made it necessary to rethink the structure of factories. (Image by Dall-E.)

Two or even all three explanations likely bear some of the blame. Explanations 2 and 3 share one crucial insight: To get full benefits from AI, companies must change their workflow to use the new capabilities better. It’s a seductive aspect of embedding AI into MS Office that you sprinkle some AI magic on top of what you do anyway, but this is the wrong approach. The task-artifact cycle dictates that when we get new tools (or AI-infused tools), we also should change the task.

The task-artifact cycle is an idea initially developed by John M. Carroll that explains how tasks and digital tools evolve together over time. It’s a never-ending cycle where introducing a new tool changes how a task is done, and performing the (revised) task leads to changes in the tool. The cycle also highlights that tools must adapt to shifts in the workplace or community, which are constantly evolving with human input.

The task-artifact cycle is an endless chase after better performance, where improvements in the tools lead to improvements in how work is done, which lead to further improvements in the tools, and so on. (Image by Dall-E.)

More on AI UX

This article is part of a more extensive series I’m writing about the user experience of modern AI tools. Suggested reading order:

About the Author

Jakob Nielsen, Ph.D., is a usability pioneer with 40 years experience in UX. He founded the discount usability movement for fast and cheap iterative design, including heuristic evaluation and the 10 usability heuristics. He formulated the eponymous Jakob’s Law of the Internet User Experience. Named “the king of usability” by Internet Magazine, “the guru of Web page usability" by The New York Times, and “the next best thing to a true time machine” by USA Today. Previously, Dr. Nielsen was a Sun Microsystems Distinguished Engineer and a Member of Research Staff at Bell Communications Research, the branch of Bell Labs owned by the Regional Bell Operating Companies. He is the author of 8 books, including the best-selling Designing Web Usability: The Practice of Simplicity (published in 22 languages), Usability Engineering (26,276 citations in Google Scholar), and the pioneering Hypertext and Hypermedia. Dr. Nielsen holds 79 United States patents, mainly on making the Internet easier to use. He received the Lifetime Achievement Award for Human–Computer Interaction Practice from ACM SIGCHI.

· Follow Jakob on LinkedIn.

· Subscribe to Jakob’s newsletter to get the full text of new articles emailed to you as soon as they are published.

Jakob Nielsen on UX

Discussion about this post