Hugging Face and Lambda: how to parse and fine-tune agent reasoning traces

MarkTechPost released a practical breakdown of the lambda/hermes-agent-reasoning-traces dataset, which helps study how AI agents think, invoke tools, and respond in multi-step dialogues. This is not a new model release, but a ready-made working pipeline: from parsing raw traces to analytics, visualization, and data preparation for fine-tuning.

What's Inside the Dataset

The focus of the material is a dataset on Hugging Face from Lambda with two configurations: kimi and glm-5.1. The first contains 7646 examples, the second 7055. Each example contains a list of messages, a description of available tools, a task category, a subcategory, and the original user query. The format is close to ShareGPT: the dialogue contains system, user, agent, and tool messages, so from a single record you can reconstruct almost the entire flow of the agent's work.

"Each example is a real agent dialogue with step-by-step reasoning and

actual tool invocation results."

The main value of the dataset is that it contains not just the final answer, but intermediate steps. For the kimi configuration, an average length of 24.3 turns per example and 13.9 tool invocations is reported; for glm-5.1 — 19.1 turns and 9.7 invocations. The dataset has nine categories in total, including Terminal & Coding, Agent Tools, Repository Tasks, Browser Automation, and File Operations. In other words, this collection contains not toy prompts, but real-world scenarios where an agent writes code, browses the web, works with files, and calls external functions.

How the Breakdown Works

The authors start with basic inspection of the train split using the datasets library: they examine fields, categories, and individual examples. Then they build parsers using regular expressions to separately extract reasoning blocks, function calls, and tool responses. This step is necessary to break down a single agent trajectory into understandable parts and separately analyze the internal reasoning, actions, and final response.

• Thoughts, tool invocations, and final text are extracted from agent messages
• JSON parsing errors are flagged separately to avoid breaking the pipeline
• Average metrics are calculated across the sample: dialogue length, number of invocations, and error frequency
• Graphs are built for popular tools, parallel invocations, and category distribution

On a sample of 3000 trajectories, the guide calculates average metrics and visualizes them through matplotlib. It also shows how to output a single complete trace in a readable format: where the user query was, where the agent reasoned, which tool it called, and what it returned. For teams evaluating agents, this is especially useful: instead of a single final score, you can see actual behavior patterns, unnecessary invocations, empty thoughts, and recurring errors in tool responses.

Preparation for Training

In the second half, the material transitions from analytics to ML practice. Dialogues are converted to a message format compatible with chat models and typical training pipelines, and tool responses are repackaged as input context for the next step. Then tokenization and label masking follow: only assistant message tokens go into the loss, while everything else is masked.

This is an important foundation for supervised fine-tuning, if the goal is to train the model to respond and act based on a trajectory already traversed. The authors also add a small trace replayer, which allows step-by-step playback of agent behavior, and a demonstration training loop via TRL. The example uses a tokenizer from Qwen2.

5 and a small train subset, so it's more of a template than a ready-made recipe for production. But that's actually a plus of the material: you can quickly run it, replace the config, add your own metrics, and get a basic laboratory for analyzing agent traces without lengthy setup and unnecessary infrastructure.

What This Means

The market is gradually shifting from evaluating only the final answer to analyzing the complete behavior of AI agents. Such datasets and guides give teams a practical way to look not only at what the model answered, but also at how exactly it thought, made mistakes, invoked tools, and what it should be fine-tuned on next.