Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureAPI / Product · ActionableAWS Machine Learning Blog→ original

Amazon SageMaker HyperPod Now Saves Live LLM Requests to S3 at Three Levels

A model in production meets things that were not in its tests: new products and slang. Amazon SageMaker HyperPod, AWS's service for running models on GPU clusters, now saves live requests and responses to S3 at three levels: the endpoint, the load balancer, and the model pod. In a demo, a ticket router did not know about new air conditioners. The gap was found in Athena and closed through fine-tuning. That is why it is better to turn on traffic recording before the first drift.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Amazon SageMaker HyperPod Now Saves Live LLM Requests to S3 at Three Levels
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1Levels 1 and 3 store full texts; if both are enabled, records are duplicated
Demo900 requests in 15 minutes, evaluation set of 280 examples
Key metric #3Records may contain names and addresses, so KMS encryption and masking are needed

Imagine a support bot for a home appliance store. Yesterday it sorted requests about refrigerators and washing machines perfectly. Today the store released air conditioners, and the bot does not know what to do with them.

Pre-assembled test suites will not reveal this problem: they were made before the new product launched. The problem can only be seen in real user requests, and those requests need to be stored somewhere. To make this possible, AWS has added a Data Capture feature to Amazon SageMaker HyperPod.

HyperPod is AWS's service for running and fine-tuning models on GPU clusters. Data Capture writes incoming requests and responses of a running model to S3, that is, to Amazon's cloud storage. According to AWS, latency stays minimal and model availability is not affected.

Three Recording Levels: What Exactly Gets Saved

The feature is enabled with the dataCapture block in the configuration. Recording can happen at three independent points along the request path, and any of them can be combined.

Level one is the SageMaker endpoint, the model's entry point. Full requests and responses are saved here through SageMaker's built-in mechanism. It is intended for compliance, auditing, and the Model Monitor service.

Level two is the ALB load balancer. It writes only access log fields: the request string, response codes, timings, client IP, and user agent. Dialogue texts are not stored there. However, it is a cheap way to monitor traffic and error rates.

Level three is the model pod, a container group in Kubernetes. The HyperPod inference operator adds three auxiliary containers to it: an NGINX proxy, an OpenTelemetry collector, and an S3 uploader. Full requests and responses, plus event metadata, end up in the bucket. This level is chosen for debugging, analytics, and collecting training data.

An important caveat: levels one and three both store full texts. If you enable both, each record ends up in S3 twice. For this reason, AWS recommends choosing the level for the task: the first for compliance, the second for traffic metrics, and the third for working with content.

How It Works: The Request Router Example

The scenario in the article is invented, and the data is synthetic. The Qwen2.5-3B-Instruct model works as a ticket router: it assigns a request to one of the categories: refrigerator, washing machine, oven, dishwasher, microwave, or "Other". Then the store launches the Arctis air conditioner line, and the router knows nothing about it and assigns the closest familiar category.

Next comes a five-step chain on a single cluster, which AWS calls a model improvement "flywheel":

  1. Write all requests and responses to S3.
  2. Analyze the records with SQL queries in Athena, a service that reads files directly from S3.
  3. Label the records with a SQL policy.
  4. Fine-tune the model on a second GPU using the training operator.
  5. Deploy the new version alongside the old one, keeping recording enabled.

In the demo, 900 requests are run through the router in 15 minutes. The evaluation set consists of 280 labeled examples. Accuracy is measured before and after fine-tuning on data that could not have existed at the time of the first deployment.

The demo requires two GPUs: one serves the model, and the second fine-tunes it. The authors used ml.g6e.8xlarge and ml.g6e.xlarge, each with one NVIDIA L40S with 48 GB. The serving pod holds its GPU for the entire lifetime of the endpoint. Therefore, with only one GPU, the training pod will hang in the Pending status.

What to Watch For During Deployment

  • Send requests to port 8081, where the recording proxy runs. The model itself listens on port 8000, and calls made directly to it will not reach S3.
  • By default, 100% of requests are recorded. This can be reduced through sampling, the payload size can be limited, and a KMS encryption key can be set. The default buffer of 10 records is flushed every 60 seconds.
  • If the endpoint is healthy but there are no files in S3, first check the IAM role of the Mountpoint-S3 CSI driver. It is the driver that writes the files, not the pod's service account.
  • Request and response bodies are stored in the SageMaker data capture format, base64-encoded. They must be decoded before parsing.

Privacy and Limitations

The third level stores full texts, and those texts contain names, addresses, and order numbers. Protect the records before turning them into training data. AWS suggests sampling, encryption through KMS (the key management service), S3 lifecycle rules, and personal data masking, for example through Amazon Comprehend.

The example itself is built on synthetic data and a small model with 3 billion parameters. On live traffic and large models, the results will depend on labeling quality and data volume.

The practical takeaway for teams on HyperPod: it is better to enable traffic recording in advance. When users bring up something new, you will immediately have both proof of the problem and material for fine-tuning.

Story

Key Questions Answered

What are the key details on Three Recording Levels: What Exactly Gets Saved?

The feature is enabled with the dataCapture block in the configuration. Recording can happen at three independent points along the request path, and any of them can be combined. Level one is the SageMaker endpoint, the model's entry point. Full requests and responses are saved here through SageMaker's built-in mechanism. It is intended for compliance, auditing, and the Model Monitor service. Level two is the ALB load balancer.

How It Works: The Request Router Example?

The scenario in the article is invented, and the data is synthetic. The Qwen2.5-3B-Instruct model works as a ticket router: it assigns a request to one of the categories: refrigerator, washing machine, oven, dishwasher, microwave, or "Other". Then the store launches the Arctis air conditioner line, and the router knows nothing about it and assigns the closest familiar category. Next comes a five-step chain on a single cluster, which AWS calls a model improvement "flywheel": Write all requests and responses to S3.

What to Watch For During Deployment?

Send requests to port 8081, where the recording proxy runs. The model itself listens on port 8000, and calls made directly to it will not reach S3. By default, 100% of requests are recorded. This can be reduced through sampling, the payload size can be limited, and a KMS encryption key can be set. The default buffer of 10 records is flushed every 60 seconds. If the endpoint is healthy but there are no files in S3, first check the IAM role of the Mountpoint-S3 CSI driver.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…