Opper releases Jevman: language models play 100 games of Pac-Man each

Startup Opper has open-sourced Jevman, a benchmark in which six low-latency models play Pac-Man against ghost bots. Each model played 100 games, and the results are collected in a public leaderboard. The GitHub repository lets you add your own model to the ranking. One game costs a human player about 2 cents. The project appeared on Hacker News under Show HN and, at the time of publication, had 52 points and 10 comments.

What was tested and how

The authors cite two recent releases as the trigger for the experiment. OpenAI launched a decisions endpoint, and Cloudflare released clef a week earlier. There are plenty of other alternative models for fast decisions on the market, and the authors wanted to compare them on a single task. Pac-Man works as a simple, fast decision-making benchmark: you have to pick a move immediately, and the game does not pause while you think.

The ranking includes:

• jev 1.13
• kev
• clef
• clef flash
• GPT-6 Luna
• Laya

Run parameters from the project description:

• 100 games per model
• opponents: ghost bots
• mode: real time, made possible by the models' low latency
• results: public leaderboard
• code: open repository opper-ai/jevman-benchmark
• cost per game for a human player: about 2 cents

The post itself contains no numerical results: placements and average scores are published only in the leaderboard. The authors only mention that beating jev's record is surprisingly hard.

Why a game instead of another test

Static benchmarks measure answer quality. Here something else matters more: the model receives the state of the board, quickly picks an action and immediately sees the consequences. A mistake costs a life, and a slow answer means Pac-Man is already no longer where it needs to be. For models sold as fast real-time decision makers, this kind of check is closer to practice than long reasoning tests. That is our interpretation; the authors put it more modestly: the game is simply a good, simple test.

The approach has obvious limitations. The opponents are scripted bots, not other models, and the final ranking depends on their behavior. A hundred games per model gives a decent sample, but the gap between close participants, such as clef and clef flash, may fall within statistical error. Read the methodology in the repository, not on the leaderboard.

The author's interest also deserves a separate note. All models run through Opper, the post author's own startup. It is an aggregator platform and, at the same time, the organizer of the comparison. The open code partly addresses the concern, because the results can be rechecked.

How to try it yourself

The project has two entry points. The first is for developers: the CONTRIBUTING file has a "Benchmark your own model" section that explains how to run your model and get onto the leaderboard. The second is for everyone else: you can sit down and play Pac-Man yourself, with models as the ghosts. You can mix them or assign the role entirely to one model family, whether jev, kev or clef. Free credits have been added for new users.

The practical takeaway for anyone choosing a model for agents or game scenarios: first run your candidate through this benchmark and compare it with the ranking. Then measure real latency and per-request cost on your own workload. At 2 cents a game, doing so is almost free. And if your model is not on the list, add it to the repository and check whether it keeps pace with jev, whose record, according to the authors, is not easy to beat.