Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents: 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseResearch Paper · For R&DAWS Machine Learning Blog→ original

AWS adds a leaderboard of 70+ models to SageMaker: quality, speed and price in one table

Choosing a model in SageMaker JumpStart used to mean reading model cards and comparing benchmarks run under different conditions. Now AWS has run 70+ models through three reasoning benchmarks (BBH, GPQA Diamond, MMLU-Pro) itself and measured response speed and instance price. In the help-desk bot example, the mid-size model nearly matches the large one but costs $5.75 per hour instead of $37.66. Treat it as a basis for a shortlist, and verify the final choice on your own data.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
AWS adds a leaderboard of 70+ models to SageMaker: quality, speed and price in one table
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1BBH, GPQA Diamond and MMLU-Pro scores are shown separately, with no overall score
Key metric #2Price covers only the on-demand SageMaker instance, without traffic, storage or discounts
Key metric #3Time to first token matters for chat, tokens per second for batch processing

Imagine you are building an internal bot for a support team and open the SageMaker JumpStart catalog, which holds hundreds of open models. Where do you start? Until now, you had to read model cards one by one, hunt for benchmark results that different teams had measured with different prompts, settings and hardware, and guess by eye what a model would cost you in particular.

Amazon has now fixed that: SageMaker Studio has a new Model Leaderboard, a table that compares 70+ models (from Meta, Google, Qwen, IBM, Arcee and others) on quality, speed and price. AWS produced all the figures itself and with a single method, so they can be compared side by side. For a developer, this saves time at the first stage of selection: a shortlist of candidates takes minutes rather than an evening of manual cross-checking.

What the table shows

The leaderboard is a new tab on the JumpStart foundation models page, next to "Categories" and "All models". Search and filters are unchanged, and the page opens directly on the ranking of large language models.

The data comes in two types. The first is quality on three public benchmarks, meaning standard sets of questions with known answers. BIG-Bench Hard tests multi-step reasoning: logical deduction, object tracking, arithmetic. GPQA Diamond covers graduate-level science questions that are hard to look up. MMLU-Pro is a set of difficult multiple-choice tests across a broad range of topics, including science, law and engineering. On all three, higher is better. The scores are not merged into a single number, so you can look at the test closest to your task.

The second type is speed and cost. AWS sends a standard test workload to an endpoint deployed with each JumpStart configuration and measures time to first token (how long a user waits for the answer to start, which matters for chat), generation speed in tokens per second (which matters for batch processing) and the instance price per hour.

Workload profile and optimization goal

There are two menus above the table. The use case profile (Max Context by default) changes only the speed and price figures; quality scores stay the same. The optimization goal selects the deployment configuration:

  • Balanced — the lowest estimated price per token;
  • Cost optimized — the cheapest instance per hour, but latency may be higher;
  • Latency optimized — the shortest time to first token;
  • Throughput optimized — maximum throughput, possibly with speculative decoding.

For a customer-facing chatbot, AWS advises starting with Latency optimized; for overnight document processing, Throughput or Cost optimized fit better. Columns are highlighted in blue: the darker the shade, the better the result. You can sort the table and hide unneeded columns, and the page has its own link you can share with your team.

Example: a help-desk bot

AWS gives a hypothetical example: you need a model with a commercial license and a predictable budget. After filtering and choosing a chat profile with Latency optimized, the picture looks like this. Model A (large): BBH 95.9, MMLU-Pro 87.2, first token in 2.33 s, 37 tokens/s, $37.66 per hour. Model B (mid-size): 94.2 and 84.9, 0.98 s, 44 tokens/s, $5.75 per hour. Model C (small): 77.4 and 66.1, 0.27 s, 61 tokens/s, $2.61 per hour. This is an illustration, not data from real models.

The takeaway: Model B trails the large model only slightly on quality, responds faster and costs roughly 6.5 times less. For a help-desk bot, where reasoning and answering quickly matter more than squeezing out the last points of general knowledge, this is the best starting point. Model C is worth keeping as a cheap fallback.

Limitations

Cost reflects only the price of the SageMaker instance at the on-demand rate. Data transfer, storage, inference components and discounts such as Reserved Instances and Savings Plans are not included, so your actual bill will differ.

Benchmarks measure general capabilities, not your product. Run the two or three best candidates on your own data and prompts. Treat close scores as a tie: the difference may fall within the normal variation between runs. Reasoning and non-reasoning models are evaluated with different generation parameters, and GPQA Diamond for reasoning models is averaged over 10 passes, so comparisons between these two types are approximate. The metrics apply to the configuration they were measured on, so deploy a model with the same profile and goal.

Coverage is still incomplete: not every JumpStart model has scores, and a missing score does not mean the model is weak, only that it has not been run. At launch, the table covers only LLMs.

What's next

AWS plans to support private JumpStart hubs, so that organizations see a ranking of only their approved models, as well as separate leaderboards for object detection, text embeddings and image understanding.

The practical order is this: first filter by hard requirements (license, provider, modalities), then sort by the benchmark you need, choose a profile and optimization goal, and finally deploy two or three models and test them on real queries.

Story

Key Questions Answered

What the table shows?

The leaderboard is a new tab on the JumpStart foundation models page, next to "Categories" and "All models". Search and filters are unchanged, and the page opens directly on the ranking of large language models. The data comes in two types. The first is quality on three public benchmarks, meaning standard sets of questions with known answers. BIG-Bench Hard tests multi-step reasoning: logical deduction, object tracking, arithmetic. GPQA Diamond covers graduate-level science questions that are hard to look up.

What are the key details on Workload profile and optimization goal?

There are two menus above the table. The use case profile (Max Context by default) changes only the speed and price figures; quality scores stay the same. The optimization goal selects the deployment configuration: Balanced — the lowest estimated price per token; Cost optimized — the cheapest instance per hour, but latency may be higher; Latency optimized — the shortest time to first token; Throughput optimized — maximum throughput, possibly with speculative decoding.

What are the key details on Example: a help-desk bot?

AWS gives a hypothetical example: you need a model with a commercial license and a predictable budget. After filtering and choosing a chat profile with Latency optimized, the picture looks like this. Model A (large): BBH 95.9, MMLU-Pro 87.2, first token in 2.33 s, 37 tokens/s, $37.66 per hour. Model B (mid-size): 94.2 and 84.9, 0.98 s, 44 tokens/s, $5.75 per hour. Model C (small): 77.4 and 66.1, 0.27 s, 61 tokens/s, $2.61 per hour. This is an illustration, not data from real models.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…