Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents: 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseInsight · On RadarAWS Machine Learning Blog→ original

Exa improves embedding speed and cost by 47% on AWS Inferentia2 chips

Search engine Exa turns web pages into vector embeddings, and with tens of billions of documents, every fraction of a cent per page adds up. After rewriting its model kernels for AWS Inferentia2 chips, the team improved speed and cost per million documents by 47%. AI agents wrote the code, and an automated accuracy test stopped them from degrading quality. For large embedding workloads, measure cost per million documents.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Exa improves embedding speed and cost by 47% on AWS Inferentia2 chips
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
BaselineFP16 on inf2.xlarge; the end result is roughly 1.5 times more documents per second
Automated testcosine similarity to the reference of at least 0.999 after every change
Key metric #3Re-indexing 30 billion documents doubled the load and prompted a hardware review

Search company Exa has described how it made document processing cheaper: after optimizing for AWS Inferentia2 chips, throughput and cost per million documents improved by 47% compared with the first version. The work concerns embeddings: for a page to appear in semantic search results, it must first be converted into a vector of numbers. Imagine you swap the model in a RAG system and have to recompute the entire knowledge base.

With tens of billions of pages, even half a cent of extra cost per thousand documents turns into a huge bill. The practical takeaway: for large embedding workloads, it pays to optimize code for specific hardware and to count the price per million documents rather than the hourly rental price of a server.

Why the optimization was needed

Exa is building its own web index, in which both pages and user queries land in the same vector space. That makes the cost of an embedding a driver of the whole product's economics. The trigger was a migration. The team was moving to a stronger document model and recomputing embeddings for 30 billion priority pages. While the re-indexing ran, the load temporarily doubled: the old model served the live index while the new one filled the new index. For the engineers, it was a convenient moment to review their hardware and look at AWS Neuron chips: Inferentia2 and Trainium2.

The starting point was simple: a model in FP16 format, running in parallel on several cores. All figures in the article refer to the smallest instance, inf2.xlarge, with one chip and two NeuronCores. One process occupies one core, so Kubernetes pods pack tightly onto instances of any size and the fleet stays homogeneous.

How the 47% was achieved

The optimization went step by step, and almost every step contributed.

  • Weight layout. It was aligned with the scheduler and with the structure of the rest of the model. This eliminated redundant matrix transpositions.
  • Autocast. The compiler lowers precision in matrix multiplications. The team checked the result against reference data: cosine similarity to the original model had to be at least 0.999.
  • Ready-made NKI kernels. The Neuron Kernel Interface library provides templates for common layers. The out-of-the-box attention kernel delivered the next gain.
  • A custom SwiGLU MLP. The Neuron Explorer profiler showed that data was constantly being spilled to HBM memory and read back. As a result, the tensor engine sat idle. The engineers wrote their own fused kernel with stationary weights, and the idle time disappeared. They wrote it in high-level NKI and used the low-level ISA only for individual operations. This makes the code easier to read and debug.
  • An attention kernel tailored to their model. General-purpose kernels are built for any data shape and waste compute on padding. Exa rewrote attention for the model's actual head architecture and folded several steps into a single matrix multiplication. This cut HBM accesses and produced the last notable speedup.

In total, the result was roughly 1.5 times more documents per second. By Exa's calculation, that is a 47% improvement in throughput and cost per million documents relative to the baseline version. The original headline rounds the result to "50%", but 47% is the more accurate figure.

Agents as kernel engineers

AI agents wrote all the optimizations inside a simple harness on the Flyte platform. Each agent runs its own workflow:

  1. Compiles an executable NEFF file.
  2. Checks the forward pass against a reference dataset (threshold 0.999).
  3. Profiles a single pass.
  4. Runs a 30-minute load test on real hardware.

The agents were matched to the task and the budget. Lightweight models iterate over compiler parameters. Reasoning models test hypotheses and look for bottlenecks. The strongest code generators write the custom kernels. The automated check keeps agents from breaking accuracy in pursuit of speed. Skills for Neuron development also proved useful separately, because many models do not know the latest NKI syntax.

Flexibility instead of lock-in to one type of hardware

So far, Exa has optimized Inferentia2 in the us-west-2 region. Availability of spot and on-demand instances varies by region, and tying yourself to one hardware type makes it easy to run into capacity shortages. So Exa schedules pods across several regions from a single managing Kubernetes cluster. The scheduler compares price per unit of performance and places the load on the cheapest nodes. The Inferentia2 work carried over to Trainium2 almost unchanged.

What to take away

If you have a steady stream of embeddings, look at the price per million documents. General-purpose kernels leave headroom on the table if you know the exact shapes of your model. You can use agents for low-level optimization, but only with a strict automated accuracy test and a benchmark on real hardware.

There are limitations. Exa wrote the article itself on an AWS platform, so it is more a case study than an independent benchmark. There are no absolute dollar prices, the model name is not disclosed, and the move to Trainium2 is described without numbers.

Story

Key Questions Answered

Why the optimization was needed?

Exa is building its own web index, in which both pages and user queries land in the same vector space. That makes the cost of an embedding a driver of the whole product's economics. The trigger was a migration. The team was moving to a stronger document model and recomputing embeddings for 30 billion priority pages. While the re-indexing ran, the load temporarily doubled: the old model served the live index while the new one filled the new index.

How the 47% was achieved?

The optimization went step by step, and almost every step contributed. Weight layout. It was aligned with the scheduler and with the structure of the rest of the model. This eliminated redundant matrix transpositions. Autocast. The compiler lowers precision in matrix multiplications. The team checked the result against reference data: cosine similarity to the original model had to be at least 0.999. Ready-made NKI kernels. The Neuron Kernel Interface library provides templates for common layers.

What are the key details on Agents as kernel engineers?

AI agents wrote all the optimizations inside a simple harness on the Flyte platform. Each agent runs its own workflow: Compiles an executable NEFF file. Checks the forward pass against a reference dataset (threshold 0.999). Profiles a single pass. Runs a 30-minute load test on real hardware. The agents were matched to the task and the budget. Lightweight models iterate over compiler parameters. Reasoning models test hypotheses and look for bottlenecks. The strongest code generators write the custom kernels.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…