There are so many LLMs available that many of you prefer using the most powerful model from the leader (OpenAI) without questioning whether it’s necessary. This is a very expensive mistake: using OpenAI’s O1 model when the Qwen 235B model would suffice can multiply your costs by 200! It’s a bit like using only Ferrari cars in your company fleet instead of adjusting the specs and cost to meet your needs.
The chart below compares today’s top LLMs (those with an intelligence index above 50% of the best one) in terms of both intelligence and costs. The green curve is the « best choice curve » – it is where you will always achieve the lowest cost for a given level of intelligence, while the red curve shows where the costs are highest. Of course the « intelligence index » can be challenged depending on the use case. But paying 200x the cost is not a great strategy.
Thanks to Gilles Babinet for the dataset and the comment regarding EU models (which are no more on the « best choice curve »).
#MLOPS