# Delayed Streams Modeling > Convert streaming X-to-Y tasks (speech-to-text, text-to-speech, simultaneous translation, full-duplex dialogue) into a single decoder-only autoregr... - **Category**: Agentic AI - **Subcategory**: Streaming & UX - **Canonical URL**: https://designpattern.fyi/patterns/delayed_streams_modeling/ --- ## Description **Intent**: Convert streaming X-to-Y tasks (speech-to-text, text-to-speech, simultaneous translation, full-duplex dialogue) into a single decoder-only autoregressive problem by time-aligning the parallel streams with a fixed offset in preprocessing, eliminating the learned read/write policy required by cascade systems. **Context**: A team is building a low-latency speech system — a real-time translator, a voice assistant that has to hold a conversation, or a full-duplex dialogue agent where the human and the agent can talk over each other. The conventional architecture is a cascade: a speech-to-text (STT) model transcribes the user's audio, a language model reasons about the text, and a text-to-speech (TTS) model produces the reply audio. Simultaneous-translation systems usually add a separate "read/write policy" that decides at each moment whether to wait for more input or emit the next chunk of output. **Solution**: In preprocessing, represent each training example as parallel token streams (source and target) interleaved on a shared time axis, with the target stream offset by a fixed delay (the chosen latency budget, e.g. 1-3 seconds for translation, ~80ms for full-duplex dialogue). Train a standard decoder-only transformer to autoregressively predict the next interleaved token. At inference, feed source tokens as they arrive and read off target tokens at the offset position — no learned policy decides when to emit, the offset structure does. The same architecture handles speech-to-text (text stream offset behind audio), text-to-speech (audio stream offset behind text), simultaneous translation (target language offset behind source), and full-duplex dialogue (each speaker's stream offset behind the joint conversation). ## Use Cases - Latency budget is tight (sub-second to few-second). - Task is naturally a stream-to-stream transduction (speech, translation, dialogue). - Time-aligned paired data is available or can be synthesized. - Cascade complexity (STT+LLM+TTS) is dominating engineering cost or latency. ## Trade-offs ### Advantages - Single model replaces a cascade; one training pipeline, one deployment target. - Latency is a preprocessing knob, not a learned behaviour — easy to tune. - Naturally supports full-duplex (both sides as parallel offset streams). - Eliminates learned read/write policy and its failure modes. - Stream alignment is interpretable: the offset is the latency. ### Considerations & Drawbacks - Requires time-aligned paired data, which is hard to obtain for some language pairs and modalities. - Fixed offset means latency cannot adapt to easy vs hard segments — a learned policy could. - Single model couples STT, LLM, and TTS quality; weakness in one role is hard to isolate. - Long-context behavioural shaping (instruction-following, refusals) is less clean than in a separate LLM stage. - Architecture commits to streaming use; batch tasks gain little from the offset structure. --- **Reference**: [Original Source](https://www.agentpatternscatalog.org/patterns/delayed-streams-modeling/)