Sorting Data with LLMs: Algorithms Beat Naive Prompting

Many people who have tried using large language models (LLMs) to sort data, for example to pick the best item from a list, have run into disappointing results. The problem is not always the model itself, but the sorting approach. I recently ran an experiment comparing five different sorting methods on 164 posts from my Telegram channel, and the results were quite revealing.

The naive approach, where you simply ask an LLM to rate each item in a list and sort by those scores, often turns out to be ineffective. LLMs are prone to systematic errors and are not always consistent in their ratings. They can also be influenced by the order of items in the list. Put simply, LLMs are not designed for direct sorting.

One interesting alternative I explored is the TrueSkill algorithm, originally developed for the player-matching system in Xbox Live. TrueSkill estimates players' skill from the results of their matches and uses those estimates to predict the probability of winning future games. For data sorting, TrueSkill can be used to compare list items against each other and build a ranking from those comparisons.

TrueSkill models the skill of each item as a normal distribution. When two items are compared, the algorithm updates their skill distributions based on the outcome. This process is repeated across all pairs of items in the list until the skill distributions stabilize. The resulting mean values of the distributions are then used to rank the items.

In my experiment, TrueSkill performed significantly better than the naive approaches. It showed a higher correlation with real data and was less prone to systematic errors. However, it is important to note that TrueSkill requires a large number of comparisons to reach good accuracy. This can be a problem for large data lists.

So what conclusions can be drawn from this experiment? First, you should not rely on naive approaches to sorting data with LLMs. Second, alternative algorithms such as TrueSkill exist and can significantly improve results. Third, choosing the right algorithm depends on the specific task and the size of the data list. In the future, even more efficient algorithms for sorting data with LLMs may appear, designed specifically for this purpose. That would open up new possibilities for using LLMs in tasks that require precise ranking and selection.