That ‘cheap’ open-source AI model is actually burning through your compute budget

The research examined 19 different AI models across three categories of tasks: basic knowledge questions, mathematical problems, and logic puzzles. The team measured “token efficiency” — how many computational units models use relative to the complexity of their solutions—a metric that has received little systematic study despite its significant cost implications.

“Token efficiency is a critical metric for several practical reasons,” the researchers noted. “While hosting open weight models may be cheaper, this cost advantage could be easily offset if they require more tokens to reason about a given problem.”

The inefficiency is particularly pronounced for Large Reasoning Models (LRMs), which use extended “chains of thought” to solve complex problems. These models, designed to think through problems step-by-step, can consume thousands of tokens pondering simple questions that should require minimal computation.

For basic knowledge questions like “What is the capital of Australia?” the study found that reasoning models spend “hundreds of tokens pondering simple knowledge questions” that could be answered in a single word.

Which AI models actually deliver bang for your buck

The research revealed stark differences between model providers. OpenAI’s models, particularly its o4-mini and newly released open-source gpt-oss variants, demonstrated exceptional token efficiency, especially for mathematical problems. The study found OpenAI models “stand out for extreme token efficiency in math problems,” using up to three times fewer tokens than other commercial models.

Among open-source options, Nvidia’s llama-3.3-nemotron-super-49b-v1 emerged as “the most token efficient open weight model across all domains,” while newer models from companies like Magistral showed “exceptionally high token usage” as outliers.

The efficiency gap varied significantly by task type. While open models used roughly twice as many tokens for mathematical and logic problems, the difference ballooned for simple knowledge questions where efficient reasoning should be unnecessary.

What enterprise leaders need to know about AI computing costs

The findings have immediate implications for enterprise AI adoption, where computing costs can scale rapidly with usage. Companies evaluating AI models often focus on accuracy benchmarks and per-token pricing, but may overlook the total computational requirements for real-world tasks.

“The better token efficiency of closed weight models often compensates for the higher API pricing of those models,” the researchers found when analyzing total inference costs.

The study also revealed that closed-source model providers appear to be actively optimizing for efficiency. “Closed weight models have been iteratively optimized to use fewer tokens to reduce inference cost,” while open-source models have “increased their token usage for newer versions, possibly reflecting a priority toward better reasoning performance.”

How researchers cracked the code on AI efficiency measurement

The research team faced unique challenges in measuring efficiency across different model architectures. Many closed-source models don’t reveal their raw reasoning processes, instead providing compressed summaries of their internal computations to prevent competitors from copying their techniques.

To address this, researchers used completion tokens — the total computational units billed for each query — as a proxy for reasoning effort. They discovered that “most recent closed source models will not share their raw reasoning traces” and instead “use smaller language models to transcribe the chain of thought into summaries or compressed representations.”

The study’s methodology included testing with modified versions of well-known problems to minimize the influence of memorized solutions, such as altering variables in mathematical competition problems from the American Invitational Mathematics Examination (AIME).

The future of AI efficiency: What’s coming next

The researchers suggest that token efficiency should become a primary optimization target alongside accuracy for future model development. “A more densified CoT will also allow for more efficient context usage and may counter context degradation during challenging reasoning tasks,” they wrote.

The release of OpenAI’s open-source gpt-oss models, which demonstrate state-of-the-art efficiency with “freely accessible CoT,” could serve as a reference point for optimizing other open-source models.

The complete research dataset and evaluation code are available on GitHub, allowing other researchers to validate and extend the findings. As the AI industry races toward more powerful reasoning capabilities, this study suggests that the real competition may not be about who can build the smartest AI — but who can build the most efficient one.

After all, in a world where every token counts, the most wasteful models may find themselves priced out of the market, regardless of how well they can think.

Daily insights on business use cases with VB Daily

If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.

Read our Privacy Policy

Thanks for subscribing. Check out more VB newsletters here.

An error occured.