Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

DJ transitions internals

This page is a short overview of how Listen mixes one song into the next. The measurements, benchmark tables, prototype results and the end-to-end test report are in the full design notes: docs/dj-transitions.md on GitHub. For the user side, see DJ transitions and Song analysis.

The pipeline

 song --> analysis --> store --> planner --> two mpv decks
          (helper)     (shared)  (listen-mix)  (phase lock, EQ, speed, volume)

1. Analysis

crates/listen-analysis decodes each song through mpv to 22,050 Hz audio, so the beat grid is in the same time base as playback (mpv removes MP3 encoder delay exactly). From the samples it computes:

  • Beats and downbeats with the neural beat tracker beat-this (small model, MIT-licensed, run with the pure-Rust rten). The model files are downloaded once on first use and checked against a pinned SHA-256. An own dynamic-programming tracker is the fallback when the model is not available; a BPM tag can correct its half-tempo errors.
  • Key, with an own detector based on Temperley profiles over spectral-peak chroma, reported as Camelot keys with a confidence.
  • Structure: intro and outro length in bars, vocal and breakdown sections, an energy curve.
  • Timbre with the bliss-audio vector, loudness (EBU R128) and a waveform.

Analysis runs in a short-lived helper process (listen --analysis-worker) and results go to a shard-based store that can be shared between computers. YouTube Music songs are analysed on demand only, within a request budget; see YouTube Music internals.

2. Planning

crates/listen-mix contains no DSP and no mpv code, which makes it cheap to test.

  • score rates how well one song flows into another (tempo, key, energy, timbre, mixability, repetition) and orders DJ picks with a beam search.
  • plan decides which pairs may be mixed at all (mix_eligible) and builds a TransitionPlan: the kind of transition (a beat-matched blend or a plain fade), where the outgoing song stops and the incoming one starts, envelopes for volume, bass and filters, and speed ramps. Songs more than a few percent apart in tempo get an echo-out or a cut (no overlap of the dry signals, each deck at its own tempo; best_handover) instead of a beat match, and an echo-out or cut also beats a filter-out over clashing vocals; a fade is left for songs without a beat grid and for AI DJ talk-overs. Each blend candidate fits its tempo to the overlapping bars (local_match): the octave from the overlap’s length, then a least-squares speed ratio over every beat of the overlap, anchored at the shared first downbeat, and a linear glide (TransitionPlan::in_glide, read through in_speed_at by the desktop mixer, the preview renderer and the Android mixer) when that takes more than 2 ms of drift off and changes the speed by at most 4 %. When a grid slows down and picks up again inside the blend (a ritardando, a live drummer), a speed curve with one knot per outgoing bar (fit_curve: the speed linear between knots, its integral fitted to the incoming beats by least squares, second differences penalised more until the curve stays within 4 %) replaces the ramp if it takes another 2 ms off, or if the ramp leaves more than 4 ms and the curve gets any closer (curve_wins); it is TransitionPlan::in_curve (wall seconds after the mix point, speed), in_glide keeps its ends for older readers, and in_speed_at interpolates it. Beats past the last one the analyser found (the grid extrapolates at the nominal tempo) are left out of the fits. On the bug-129 pairs: DAUM → Being Freezed (8-bar BeatBlend) 25.6 → 1.3 ms, Falling Leaves are Beautiful → Mutuals (4-bar ShortBlend) 8.5 → 2.8 ms (4.1 ms with its 3.9 % ramp alone; measured on every grid beat between the mix point and the stop, in wall time). The drift that is left costs DRIFT_PER_S beyond 4 ms. On 427 real history pairs the plans played without phase nudges keep 193 of 210 beat-mixed pairs within 4 ms on every beat (98 of 222 with the songs’ average tempos; worst 53 ms instead of 1.3 s); 12 pairs whose overlapping bars can’t be matched become echo-outs or cuts. PlanCtx::no_echo (set by the app when the outgoing song itself came in through an echo-out, MixState::entered_echo) drops the echo candidates, so two echo-outs never follow each other; bridge::gap counts an echo-out or cut as “no beat-matched join”, bridge::always_try exempts such pairs from the bridge chance and bridge::choose then asks no flow gain of the legs over the direct pair (only that they flow no worse). The echo is four adelay taps behind a send volume mixed back by amix in the deck’s lavfi chain (live delays/weights/volume commands, listen_core::settings::DECK_ECHO_*). amix and volume step in one sample, so the send and the dry signal are switched by afade gates whose start_time is set to the planned file time 1.5 s ahead (deck_echo_commands): each switch is a 5 ms ramp at exactly that sample, as in listen_mix::echo, the same effect on samples for the preview and the Android mixer.
  • loudness (contract C5) also holds the blend gain: plan stores BlendLevels in each overlapping plan (both decks’ local loudness every 0.5 s across the blend after the decks’ filters, plus the 4 s edge levels before and after), and blend_trim(plan, t, g_out, g_in, deck_gain_out_db, deck_gain_in_db) is the trim both engines and render apply to both decks: the summed loudness follows the line between the edge levels, weighted by how much the decks overlap, clamped to −6..+3 dB. It replaced the fixed 0.85 trim. LocalLevel is Features::lufs anchored on the per-bar curves and power-averaged over 3 s: the energy curve is plain power, so each bar is corrected towards K-weighting from its bass share (the low-band curve against the energy curve, scaled by BASS_SHARE_SCALE of the bound the curves allow) and its brightness (centroid curve, up to BRIGHT_DB); against the audio’s 3 s short-term loudness that is 0.75 dB sd on 78 bench songs (1.04 before) and 0.72 on 47 others (0.96). The same bass share gives what the bass swap’s shelf takes (LocalLevel::bass_loss_lu, 1.0 LU sd against the song played through the shelf; the fixed 2 LU it replaced missed by more than 3 LU in 18 % of windows), and plan_levels takes every filter’s loss off the deck’s level, so the trim makes up for a bass swap instead of cutting both decks as if the outgoing were still whole (bug 148). --check-mix --levels prints the model, the measured level and the level through the −24 dB shelf (measured_no_bass). A boost may take a deck above unity: the desktop sends mpv volumes up to its default volume-max of 130 % for it (listen_player::MAX_VOLUME), so the app plays what render and verify model. Handovers get none.
  • render makes an offline preview of a plan, available through listen --check-mix --render.
  • verify is render-and-verify: plan_verified takes the planner’s options in order (plan::alternatives: the chosen plan, then the other candidates and handovers by score, then the plain fade) and plays each one through on the analysis curves every 0.1 s. It uses both decks’ local loudness (loudness::LocalLevel), the vocal curve with lyrics merged in, the gain envelope, the equal-loudness trim, the loudness the EQ moves take (the bass shelf from the song’s own bass share) and their rough vocal losses, and the echo repeats. It arms the first plan that passes the gates: a hole or bump over 1.5 LU (against both the songs’ own levelled loudness and the edge line; a modelled hole counts 0.6 LU deeper, HOLE_MARGIN_LU, the curves’ error against rendered blends), more than 1 s of audible vocals on both decks, or a mid-line exit (lyric cues only). The desktop plans before it fetches the incoming file, so the check runs on curves rather than audio. audio_levels and audio_dip_bump measure a rendered preview the same way (BS.1770 K-weighting over left and right, 3 s short-term) for --check-mix --render. The reference for each deck is that deck alone on the preview’s own timeline (render::render_decks): a tempo ramp before the blend moves the outgoing up to a second away from its file, and measuring against the file turned level steps at the mix point into holes the mix doesn’t have. With the filter losses in the trim the planner’s first choice passes on all 427 real history pairs (it failed in 18 before, mostly a hole within the margin), and the check takes 0.08 ms on average. On the 104-pair render bench (96 blends) the largest rendered hole went from 2.98 to 0.30 LU (5 over 1.5 before), the largest bump stayed 1.11.
  • lyric merges a VocalMap (from listen_lyrics::vocal_map: sung stretches, line ends, chorus occurrences of synced lyrics) into bars.vocal and sets bars.cue flags (LINE_END, ENTRY, CHORUS, CHORUS_END). It drops lyrics that are anti-correlated with the audio vocal curve or fit clearly better shifted by up to 6 bars. The planner then adds cue-outs after a sung phrase or at a breakdown in the last 40 % (cut cost 0.02/s), cue-ins 1, 2 or 4 bars before a verse or chorus entry (skip cost 0.01/s), +0.5/+0.3 when a blend ends on the incoming chorus/entry, and −0.5 for stopping the outgoing mid-line. Without lyrics the plans are unchanged. The desktop looks lyrics up for the playing and next song (app/src/app/mix/vocal.rs) and re-plans when they arrive before the preload.

3. Two mpv decks

app/src/app/mix/ and decks.rs run a plan on two mpv players:

  1. About 20 seconds before the mix point the incoming song is loaded paused on the idle deck, with its start position set.
  2. It is unpaused at the planned moment. A learned latency compensates for the start-up delay.
  3. Phase lock: during the silent pre-roll bar both decks’ audio-pts are read back to back, and the incoming deck’s speed is nudged by about 0.5 percent (up to 2 percent while it is still silent) until the beats line up. The loop keeps running through the blend to catch underruns. Seeks are not used to correct the phase.
  4. Volume, bass, high-pass and low-pass changes are sent slightly early (about 240 ms with the 0.1 s audio buffer) and as ramps of at least one beat, so the jitter of mpv’s filter commands is inaudible. Tempo matching uses the rubberband filter, with scaletempo2 as a fallback.
  5. Halfway through the blend the incoming deck becomes the active deck.

Anything unexpected (a late deck, a skip, a seek, a pause) cancels the blend and falls back to mpv’s gapless playback.

Measured accuracy

In the prototype, the two decks stayed aligned with a median error below 0.1 ms and a 95th percentile below 1 ms once the lock converged. The design target was a median of 10 ms or less. The end-to-end test (tools/dj-e2e.sh, described in Tests) checks real blends for gaps, level and doubled beats.

Tempo matching (heart-rate mode)

The heart-rate mode of both apps uses listen_mix::tempo: a reading (heart beats or steps per minute from a Wear OS watch, a Bluetooth strap or the phone’s step detector; on the desktop a Bluetooth sensor) is smoothed with a 30-second time constant and turned into a tempo target clamped to 70–180 BPM, ± 8 BPM, with half and double time counting (85 BPM fits 170). When the target has moved more than 5 BPM since the last time, the upcoming automatic queue items are re-ranked: songs within 4 BPM stay, closest first, the others make room for tempo-targeted Auto-DJ picks. Songs added by hand and the playing song are never touched. Music does not measurably change heart rate during exercise; the mode picks music that fits the effort, it is not training guidance.

The fit is judged at the faster pulse (tempo::fit_distance = min(|b − t|, |b − 2t|, |2b − t|)), so a 70 BPM target fits 62–78, 132–148 and 31–39 BPM and a 130 BPM song does not fit 70. TempoTarget::distance halves a double-time song instead (130 counts as 65) and is not used for the fit. Fills go through tempo::tempo_fill: the fitting fetched songs, topped up from the analysis store (closest to the current song’s energy and timbre, never excluded, queued or played songs), and the nearest misfits only when the store is short too (TempoFill::padded_note says so). The AI DJ’s prompt gets listen_dj::tempo::rule and a listen_dj::tempo::fit_note per candidate, the same on both apps.

Where to read more

The full notes explain why each choice was made, with results for the analyser benchmark, key detection, the mpv deck prototype and the end-to-end run: github.com/leks2205/Listen/blob/main/docs/dj-transitions.md.

Bridge scenes

listen_core::scene keeps bridges in the pair’s scene. A Scene is a language (ISO 639-1, or instrumental) plus genre Families bits (pop, hip hop, r&b, rock, electronic, jazz, folk, trot, latin, classical, ambient, blues, country, reggae), read from the describe-it tags_en (Scene::from_tags), the song’s genres (from_genres), the artist’s other tagged songs (SceneIndex, language held by ≥ 60 % of them, families by ≥ 1/3) and the title script (Hangul, kana; language only), strongest first. compare fails on a different language or disjoint families (pop~r&b, hip hop~r&b, pop~electronic, rock~folk, jazz~latin, r&b~blues, jazz~blues count half). Without comparable tags the CLaMP 3 audio cosine decides: ≥ AUDIO_SAME (0.60) is one scene. On the index of 2026-11-06 (305 songs) K-pop↔K-pop pairs have audio cosines p5/p25/p50/p95 0.39/0.49/0.59/0.82 and K-pop↔US hip hop 0.28/0.35/0.41/0.56 (48 % vs 2 % at ≥ 0.60); the tag-embedding cosines separate completely (K-pop pairs p5 0.49, K-pop↔US hip hop p95 0.44). The same artist is always one scene; artists heard within 20 minutes of each other (Neighbours::from_history) fill an unknown. Gate::judge: a pair in one scene needs a bridge in it, two scenes need a bridge from one of them, an unknown candidate is rejected when the pair is known, and no gate applies when neither song is known. listen_analysis::bridge_scene_gate filters the candidates and gives each a score (0..1) that bridge::choose_scored takes off the cost (×0.3). The desktop loads the index from Listen/embeddings/v1/clamp3 next to the analysis store (analysis::keep_scenes, reloaded when the files change); the phone mirrors listen-server’s GET /v1/scenes?since= (SceneRecord: tags and the audio embedding as bytes). listen --check-bridges <analysis copy> <embeddings dir> <history.jsonl> prints the bridge chosen without and with the gate for history pairs.

Eras. Scene::era is a decade start (1970 = before 1980): an explicit decade or year in the tags (“2020s”, “90s ballad”, “1999”), else a K-pop generation (1st → 1990s, 2nd → 2000s, 3rd → 2010s, 4th/5th → 2020s), else the song’s year (SceneIndex::profile_dated, listen_analysis::scene_profile passes Song::year), else the artist’s (common_era: the median when ≥ 60 % are within a decade of it), else the up to 5 closest-sounding dated songs (cosine ≥ AUDIO_SAME, at least 3, agreeing as above). compare: both known and more than ERA_SPAN (10) apart is Other("era"); a decade apart scores ×0.9; one side unknown needs language and genre both matched (×0.85), else Unknown. History neighbours never fill across a known/unknown era. Reject::OtherEra and Tally::other_era count those.

Heart-rate tempo picks. scene::Session (built by listen_analysis::tempo_session from the current song, the songs before it and the queued ones; the first SESSION_LEN = 5 known) finds the member most others share a scene with; with a majority its merged profile (most common language and era, families at least half have) is the session scene, otherwise every member’s scene counts (mixed). judge: the session’s artists pass, a candidate must compare as one scene with the majority (or fit any member when mixed), an unknown one is rejected, no gate when nothing is known. listen_analysis::tempo_scene_filter wraps it with the heard set (played history songs, favourites) as a SceneFilter for listen_mix::tempo::tempo_fill, which gates only the store top-up (the DJ’s pool and its padding stay), puts heard songs first and takes 0.3 × the scene score off the sound cost. TempoFill::scene_note is the log line. listen --check-tempo <analysis copy> <embeddings dir> <history.jsonl> --bpm 146 --probe "Artist – Title" replays a top-up without and with the gate.