DuET cuts piano-to-score transcription error from 6.56% to 3.64%

Transcribing expressive piano performances into sheet music remains a hard problem, and the new DuET paper shows where it can be improved. The authors propose a Dual-Encoder Transformer that solves two tasks jointly: converting MIDI from live playing into a score (PM2S) and determining beats and downbeats frame by frame. On Gangqinpu, a dataset the authors built, the average error on the MUSTER metric dropped from 6.56% to 3.64% compared with the best baseline. That is a reduction of almost 45% in relative terms. Code, weights and a demo are on GitHub.

Why sheet music is harder than it looks

A performer does not play to a metronome. They slow down before a climax, play slightly ahead of the beat, stretch a phrase. To turn such a stream of events into a score, a model has to recover the metrical structure: where the bars fall, how the beats are divided, and what duration to write for each note. Score attributes are tied to notes, but beats are tied to time. A beat can land on a rest or in the middle of a long note, and then there is no note onset nearby to anchor to. Models that look only at the sequence of notes lose this information.

How DuET works

The authors' answer is dual encoding. One encoder works at the note level, the other at the level of time frames. Bidirectional cross-attention sits between them, so the note representations get context about the beat and bar, while the frame representations know which notes are sounding around them. An autoregressive decoder predicts structured score attributes and alternates between the two representations at every step.

A separate part of the work concerns positional encodings. TimeRoPE provides a shared time reference frame for both encoders, tied to performance time rather than to the index in the sequence. TPRoPE (Time-Pitch Rotary Positional Encoding) explicitly encodes relative time and pitch relationships between notes.

Key details of the work:

• architecture: two encoders (note-level and frame-level), bidirectional cross-attention, an autoregressive decoder;
• tasks: PM2S transcription and frame-level tracking of beats and downbeats;
• new components: TimeRoPE and TPRoPE;
• datasets: ASAP (classical piano) and Gangqinpu (multiple genres, with an emphasis on pop music);
• metric: MUSTER, average error;
• result on Gangqinpu: 6.56% → 3.64%;
• result on ASAP: 11.30% → 11.22%;
• publicly available: code, weights, demo.

What the experiments showed

The main gain comes on Gangqinpu: a difference of 2.92 percentage points. On ASAP there is a formal improvement, but a minimal one: 0.08 points, about 0.7% relative to the previous result. Classical repertoire, with its complex agogics and dense texture, still produces an error above 11%. That figure shows the task is far from solved. Ablations confirm the contribution of three elements: frame-level context, time-based positional encoding and masking of the decoder history. Each of them individually contributes to the quality of tempo and duration estimates.

It is also worth noting that Gangqinpu was compiled by the authors themselves. A comparison on a self-built dataset always calls for caution until other groups reproduce the result on independent data. The open weights make that possible.

Who this is for

For developers of notation editors and learning apps, DuET offers a ready-made foundation for importing MIDI keyboard playing into a readable score, especially for pop arrangements. For music AI researchers, the ideas of a shared time axis for two modalities and TPRoPE are useful: they can be transferred to other tasks with events distributed over time. Anyone planning to deploy the model should first run it on their own recordings and check the results on repertoire close to classical. There, the quality margin is noticeably smaller for now.