How It Works

This page describes the signal-processing pipelines the components configure and the audio data flow through their wrappers. See Hardware & Audio Design for the physical signals these pipelines consume.

System diagram showing capture, echo-cancellation processing, Home Assistant, playback, and the AEC feedback paths

The capture and playback paths run together: the reference lets the echo canceller identify the device’s own loudspeaker audio in the microphone signals.

The SpeexDSP pipeline

The pipeline uses the open-source Xiph SpeexDSP library, vendored inside the component. Per processed microphone channel it runs:

  1. Capture synchronized signals. One to four microphones and the playback hardware reference are read from the shared TDM clock domain.
  2. Linear acoustic echo cancellation. speex_echo_cancellation() adapts a frequency-domain filter — up to 16,384 samples (~1 s) of echo tail — between the reference and each microphone, and subtracts the estimated echo. The long filter is this component’s main advantage: the tail must cover speaker-plus-room decay plus the reference offset, or residual echo remains after convergence.
  3. Channel handling. Without beamforming, first or second runs one canceller and preprocessor for that microphone only. With beamforming, each microphone keeps its own adaptive echo filter inside one Speex multichannel state; an allocation-free fixed-point localizer (normalized cross-correlation with Q15 sub-sample interpolation) estimates the inter-microphone delay, and an adaptive delay-and-sum beamformer produces one mono stream before a single shared preprocessor. Weak or ambiguous correlations retain the previous stable direction.
  4. Noise suppression, residual-echo suppression, AGC, and VAD. speex_preprocess_run() consumes the canceller state, applies separate echo suppression targets for far-end-only audio and double-talk, reduces stationary noise, normalizes the level toward agc.target_level, and reports voice activity via get_vad_state() / get_vad_probability(). The optional reference-aware AGC gate limits amplification during playback: ineligible audio loses boost and stops updating the loudness estimate, but samples are never muted and the AEC keeps adapting.
  5. Publish enhanced mono audio. The cleaned 16-bit, 16 kHz stream feeds the same pre-roll and live microphone ring buffers described below.

Processing runs on a dedicated task pinned to core 0 at priority 4. The audio task logs its own load every five seconds at INFO level:

DSP load: 313 frames in 5002 ms: processing avg 2210 us/frame (13.8% of real time), peak 4980 us (31.1%), frame budget 16000 us

The percentage is the share of one core used to keep up with 16 kHz. Load scales with filter_length, frame_size, channel count, and the meters; values near 100% mean the canceller can no longer keep up and frames will be dropped. At startup the component also logs where the canceller state was allocated:

SpeexDSP state memory: 118 KB internal, 0 KB PSRAM

Rolling pre-buffer and wake-word hand-off

Wake-word detection and voice-assistant capture are separate microphone consumers. When the detector recognizes a wake word, it has already consumed audio containing the beginning of the request. Stopping that listener and starting the voice assistant also takes time. Without a hand-off buffer, the speech-to-text stream can therefore begin late and lose a short command or the first word after the wake phrase.

Timeline showing the rolling history, wake-word hand-off, preserved request prefix, live audio, and excluded response tail

The microphone wrapper avoids that gap with two bounded PSRAM buffers:

  • a continuously overwritten one-second history buffer, independent of whether a consumer is currently listening; and
  • a four-second utterance queue used to deliver the preserved prefix and subsequent live audio in order.

When wake-word detection fires, request_pre_roll() snapshots audio beginning 100 ms before the last sample delivered to the detector. It also includes any newer samples already captured but not yet delivered. The snapshot remains armed while the wake-word listener stops and the voice-assistant listener starts. begin_pre_roll_replay() then releases the queued audio before normal live delivery continues, so the downstream stream is a single chronological utterance rather than a separate replay followed by live audio.

The one-second history is a maximum retention window, not one second of audio blindly prepended to every request. The detector’s recorded stream position is used to choose the hand-off point, with only 100 ms of extra headroom. Storage is fixed-size: history overwrites its oldest samples, while an active utterance preserves FIFO order and reports dropped bytes if its four-second queue is exhausted.

When the integration calls set_response_playing(true), response audio is excluded from both live delivery and rolling history. Calling set_response_playing(false) starts a further 300 ms exclusion for the acoustic tail. This reduces the chance that residual synthesized speech is replayed as the start of the user’s next request. The integration must call the hand-off methods at the correct wake-word and voice-assistant lifecycle points; the microphone platform cannot infer those transitions on its own. Full-duplex barge-in uses the separate capture-gate policy described in the GRU guide.

In the unreleased implementation, enabled neural speech protection reads this same capture-gate state before Speex preprocessing. Confirmation holds full protection through the existing one-second hold-over, while playback restart after at least 300 ms of reference silence cuts both policies short on the same frame. There is no second timer. Onset uses neural probability before confirmation; outside the armed gate and enabled neural assistant/playback scope, the extra probability is zero and ordinary Speex processing resumes. AEC remains active throughout.

Playback, automatic rate matching, and reference

The ESPHome speaker accepts signed 16-bit, 16 kHz mono or stereo PCM and holds it in a one-second playback buffer. playback_gain_db attenuates each sample before both the TDM output and playback-reference tap, so a mono reference uses the same samples sent to the DAC while preserving headroom in the analogue amplifier/loopback path. If automatic rate matching is enabled, the component makes a small timing correction before buffering the audio. Mono is duplicated into the two configured TX slots; stereo uses one slot for each channel. The four-slot TDM transmitter then carries those samples to the DAC, amplifier, and loudspeaker.

AEC uses the hardware reference from the configured ADC slot. AEC is very sensitive to timing alignment between the reference and the microphones. It will only work effectively with a hardware reference captured in sync with the microphone channels.

Why rate matching is needed

Requesting 16,000 Hz from the I2S peripheral does not guarantee an actual TDM sample clock of exactly 16,000 Hz. The hardware derives that clock from its source clock using dividers with finite precision; divider rounding and source clock tolerance can leave the physical sample rate slightly above or below the requested rate. Rate matching compensates for this difference between the nominal PCM sample rate and the satellite’s actual playback rate.

Without correction, the same number of input samples takes slightly more or less time to play than the PCM format specifies. During continuously paced playback this can gradually fill or drain the playback buffer. Home Assistant can also deliver audio in bursts, but packet arrival speed does not define the PCM sample rate and is not the correction target.

Comparison of playback-buffer drift with and without automatic I2S clock rate matching

With resampler: true, the component automatically resamples nominally 16 kHz PCM to match the satellite’s estimated hardware playback rate:

  1. Playback starts at the nominal 16,000 samples per second.
  2. The component estimates the hardware consumption rate from the duration of successful blocking I2S writes containing a full frame of playback audio. This is an estimate from write timing, not a direct measurement of the clock signal: DMA buffering and task scheduling can affect it. It is the only correction target; network throughput is not the PCM sample rate.
  3. The measured rate is clamped to 15,000–17,000 Hz, and the active correction moves toward it one hertz at a time instead of changing abruptly.
  4. A continuous fractional-phase linear interpolator produces slightly more or fewer samples for the actual hardware output rate. Phase and the preceding sample carry across packet boundaries, avoiding a discontinuity at each packet.

The learned playback rate is retained between streams so a later response can begin with the previous correction. Stopping or clearing playback resets interpolation state so samples from two unrelated streams are never blended. Short-term delivery variation — bursts and gaps from the network — is absorbed by the one-second playback buffer rather than by changing voice pitch.

This is automatic hardware sample-rate compensation. It does not decode compressed audio, accept a genuinely different sample rate, repair severe network starvation, or change the hardware clock. Input must still be signed 16-bit, nominally 16 kHz PCM. Mono and stereo are both supported.

Task scheduling, CPU affinity, and priority

The DSP processing task runs on core 0 at priority 4. On dual-core targets the playback/I2S TX task runs on core 1 at priority 20; the code selects core 0 for single-core builds. Playback normally blocks in the I2S driver. This scheduling fallback does not establish support for targets lacking TDM. If a frame exceeds its real-time budget (frame_size / 16 kHz), the processing task yields so system work is not starved. Diagnostics report the processing peak; if backlog reaches the transport, a later read may also record an rx_error.

FreeRTOS schedules larger priority numbers first. These priorities are intentionally asymmetric. Priority 20 protects continuous I2S transmission from ordinary application work, while priority 4 does not mean that audio processing is unimportant. Per-frame buffers use internal RAM; playback and reference queues use ESPHome ring buffers.

Wake-word cue

The example requests a short confirmation beep after retaining microphone pre-roll. The playback task includes the cue in the speaker/reference path while capture, AEC and voice-pipeline startup continue.