Companies Begin To Ration AI For Employees As Token Costs Skyrocket

As the financial and tech worlds contemplate their first trillionaire, cost seems like such a relative issue. 

But even OpenAI's CEO Sam Altman, no shrinking violet when it comes to extolling AI's virtues, is admitting that exponentially exploding token costs are becoming a problem. No one is yet willing to state that such eye-watering expenditures will slow demand for AI (Heaven Forefend!) but the implication is there, along with lingering concerns about measuring how token use contributes to business results. Expect to see an initial burst of explanatory bombast about 'efficiencies' followed by more serious attempts to somehow mitigate the cost impact without denying AI firms their massive payday. JL

Bradley Olson reports in the Wall Street Journal:

Costs are skyrocketing for tokens, the basic unit of measurement for AI computing, as AI model providers seek to balance supply, demand and manage their costs. Some enterprises report their AI spending double or triple. The shift to usage-based pricing has forced enterprise customers to reckon with consumption.  Corporate leaders are scrambling to bring down expenses by finding ways to ration AI use in their organizations and increase computing efficiency.  Microsoft limited access to an Anthropic program for employees who can use an internal coding assistant instead. Companies using advanced AI coding tools have found only 18% of spending on tokens translating into shipped coding products that reach real users so Salesforce introduced a system for tracking how token use contributes to positive business outcomes. 

Use of artificial intelligence by big companies is exploding—and the soaring cost has some of them pumping the brakes in a way that could complicate AI’s triumphal march across the economy.

Executives across industries this year have urged employees to integrate AI tools into their work, spending freely to encourage experimentation and seeking to send a message to Wall Street that their companies won’t be left behind in a coming wave of disruption.

All that enthusiasm has resulted in skyrocketing costs for so-called tokens, the basic unit of measurement for AI computing, as AI model providers seek to balance supply and demand and manage their own costs. Some enterprises have hit their annual budget in just three months or reported seeing their AI spending bills double or triple. 

Now corporate leaders are scrambling to bring down expenses by finding ways to ration AI use in their organizations, steer workers toward cheaper, homegrown tools and help them hone their skills to improve returns. 

Top technical executives at Uber Technologies, Meta Platforms, Microsoft, Salesforce, DoorDash and other companies have all talked about new efforts to ensure AI use contributes to productivity or have taken steps to reduce the availability of some tools for certain employees.

AI critics have pointed to efforts to direct AI spending more carefully as evidence of a warning sign that the ultrafast pace of AI growth could slow. That would potentially hurt Anthropic or ChatGPT maker OpenAI as they take steps toward public listings this year. Anthropic on Thursday closed a $65 billion fundraising round that values the startup at $965 billion.

But a number of investors and tech executives cautioned against betting on a pullback, noting that sales and usage by corporate AI customers have climbed far faster than forecasts.  

“We’re still in the pretty early innings” for AI adoption, said Will McGough, chief investment officer at wealth manager Prime Capital Financial, which is invested in a number of tech companies and is closely evaluating the coming IPOs of giant AI startups. “Even massive companies are still figuring things out.”

Just a few months ago, the prevailing sentiment around AI use at many big companies was the more, the better. All-you-can-eat subscriptions amounted to a subsidy by the model-makers, which often lost money on the intensive activity of power users. Exhorted to embrace the wave of change, employees at some companies engaged in tokenmaxxing, or using as much computing as possible in order to be seen as AI-forward—a practice that continued even as the model companies shifted to usage-based pricing. 

Matan Grinberg, chief executive of coding automator Factory, said one executive at a top financial institution told him his employees were burning hundreds of thousands of dollars a month on tokens. Some, the executive said, were using powerful premium-tier models to answer the simplest of questions, or just engage in small talk. 

“If your daughter needs tutoring in algebra, you can probably find someone cheaper than Albert Einstein,” he said.

Matan Grinberg, CEO of coding automator Factory, said some companies’ employees are using expensive models to answer simple questions. Factory

Higher costs may eventually steer users toward cheaper models that cost a fraction of the price, but many companies remain wary of such AI systems because several of the cheapest options were developed in China, according to executives. Anthropic, OpenAI, Google and others also offer cheaper versions of their flagship models, and Factory and others have developed systems to help companies triage queries and steer some tasks to cheaper options.

Token use continues to grow immensely. Google said at a recent event that it now processes over 3.2 quadrillion tokens a month, seven times as much as a year ago. The company and others are seeking to reduce the cost of AI use in a variety of ways, including increasing computing efficiency. 

That shift to usage-based pricing has forced enterprise customers to reckon with their consumption. An Uber executive said by March, the company had blown through its annual budget for agentic, or autonomous, AI use. Microsoft limited access to an Anthropic program for some employees who can use an internal coding assistant instead. Salesforce introduced a system for tracking how token use ultimately contributes to positive business outcomes. 

“It has been great to let people experiment but now we have too many overlapping tools,” Meta Chief Technology Officer Andrew Bosworth said in an April memo to employees. “Nobody should be using AI tools just for the sake of using them. All motion is not progress and token usage alone is not a measure of impact of any kind.”

Andrew Bosworth, Meta’s chief technology officer, says employees shouldn’t be using AI tools ‘just for the sake of using them.’ David Paul Morris/Bloomberg News

A Microsoft spokesman said the company’s decision to reduce access to Anthropic’s Claude Code program wasn’t rooted in cost but stemmed from a desire to standardize what employees use across its organization. 

An Anthropic spokeswoman said the company’s models help customers achieve greater productivity, such as completing complex tasks in less than two weeks that would have taken more than seven months in the past.

“As with any new technology and way of working, teams are still discovering where the biggest gains are and how best to measure them,” she said. “We’re working with customers to give them the tools to make sure the return is something they can see, not just feel.”

Software engineers and startup executives warn that even though it is possible to complete tasks far more quickly, spending on debugging, reviewing and rewriting AI-generated code remains high, indicating that the models still need to be improved.

For companies using advanced AI coding tools, only 18% of spending on tokens is translating into shipped coding products that reach real users, according to EntelligenceAI, a startup that aggregated data on more than 2,000 companies using advanced AI tools for coding.