How It Works
This page describes the signal-processing pipelines the components configure and the audio data flow through their wrappers. See Hardware & Audio Design for the physical signals these pipelines consume.
The capture and playback paths run together: the reference lets the echo canceller identify the device’s own loudspeaker audio in the microphone signals.
The SpeexDSP pipeline
The pipeline uses the open-source Xiph SpeexDSP library, vendored inside the component. Per processed microphone channel it runs:
- Capture synchronized signals. One to four microphones and the playback hardware reference are read from the shared TDM clock domain.
- Linear acoustic echo cancellation.
speex_echo_cancellation()adapts a frequency-domain filter — up to 16,384 samples (~1 s) of echo tail — between the reference and each microphone, and subtracts the estimated echo. The long filter is this component’s main advantage: the tail must cover speaker-plus-room decay plus the reference offset, or residual echo remains after convergence. - Channel handling. Without beamforming,
firstorsecondruns one canceller and preprocessor for that microphone only. Withbeamforming, each microphone keeps its own adaptive echo filter inside one Speex multichannel state; an allocation-free fixed-point localizer (normalized cross-correlation with Q15 sub-sample interpolation) estimates the inter-microphone delay, and an adaptive delay-and-sum beamformer produces one mono stream before a single shared preprocessor. Weak or ambiguous correlations retain the previous stable direction. - Noise suppression, residual-echo suppression, AGC, and VAD.
speex_preprocess_run()consumes the canceller state, applies separate echo suppression targets for far-end-only audio and double-talk, reduces stationary noise, normalizes the level towardagc.target_level, and reports voice activity viaget_vad_state()/get_vad_probability(). The optional reference-aware AGC gate limits amplification during playback: ineligible audio loses boost and stops updating the loudness estimate, but samples are never muted and the AEC keeps adapting. - Publish enhanced mono audio. The cleaned 16-bit, 16 kHz stream feeds the same pre-roll and live microphone ring buffers described below.
Processing runs on a dedicated task pinned to core 0 at priority 4. The audio task logs its own load every five seconds at INFO level:
DSP load: 313 frames in 5002 ms: processing avg 2210 us/frame (13.8% of real time), peak 4980 us (31.1%), frame budget 16000 us
The percentage is the share of one core used to keep up with 16 kHz. Load
scales with filter_length, frame_size, channel count, and the meters;
values near 100% mean the canceller can no longer keep up and frames will be
dropped. At startup the component also logs where the canceller state was
allocated:
SpeexDSP state memory: 118 KB internal, 0 KB PSRAM
Rolling pre-buffer and wake-word hand-off
Wake-word detection and voice-assistant capture are separate microphone consumers. When the detector recognizes a wake word, it has already consumed audio containing the beginning of the request. Stopping that listener and starting the voice assistant also takes time. Without a hand-off buffer, the speech-to-text stream can therefore begin late and lose a short command or the first word after the wake phrase.
The microphone wrapper avoids that gap with two bounded PSRAM buffers:
- a continuously overwritten one-second history buffer, independent of whether a consumer is currently listening; and
- a four-second utterance queue used to deliver the preserved prefix and subsequent live audio in order.
When wake-word detection fires, request_pre_roll() snapshots audio beginning
100 ms before the last sample delivered to the detector. It also includes any
newer samples already captured but not yet delivered. The snapshot remains
armed while the wake-word listener stops and the voice-assistant listener
starts. begin_pre_roll_replay() then releases the queued audio before normal
live delivery continues, so the downstream stream is a single chronological
utterance rather than a separate replay followed by live audio.
The one-second history is a maximum retention window, not one second of audio blindly prepended to every request. The detector’s recorded stream position is used to choose the hand-off point, with only 100 ms of extra headroom. Storage is fixed-size: history overwrites its oldest samples, while an active utterance preserves FIFO order and reports dropped bytes if its four-second queue is exhausted.
When the integration calls set_response_playing(true), response audio is
excluded from both live delivery and rolling history. Calling
set_response_playing(false) starts a further 300 ms exclusion for the acoustic tail.
This reduces the chance that residual synthesized speech is replayed as the
start of the user’s next request. The integration must call the hand-off methods
at the correct wake-word and voice-assistant lifecycle points; the microphone
platform cannot infer those transitions on its own. Full-duplex barge-in uses
the separate capture-gate policy described in the
GRU guide.
In the unreleased implementation, enabled neural speech protection reads this same capture-gate state before Speex preprocessing. Confirmation holds full protection through the existing one-second hold-over, while playback restart after at least 300 ms of reference silence cuts both policies short on the same frame. There is no second timer. Onset uses neural probability before confirmation; outside the armed gate and enabled neural assistant/playback scope, the extra probability is zero and ordinary Speex processing resumes. AEC remains active throughout.
Playback, automatic rate matching, and reference
The ESPHome speaker accepts signed 16-bit, 16 kHz mono or stereo PCM and holds
it in a one-second playback buffer. playback_gain_db attenuates each sample
before both the TDM output and playback-reference tap, so a mono reference uses
the same samples sent to the DAC while preserving headroom in the analogue
amplifier/loopback path. If automatic rate matching is enabled, the
component makes a small timing correction before buffering the audio. Mono is
duplicated into the two configured TX slots; stereo uses one slot for each
channel. The four-slot TDM transmitter then carries those samples to the DAC,
amplifier, and loudspeaker.
AEC uses the hardware reference from the configured ADC slot. AEC is very sensitive to timing alignment between the reference and the microphones. It will only work effectively with a hardware reference captured in sync with the microphone channels.
Why rate matching is needed
Requesting 16,000 Hz from the I2S peripheral does not guarantee an actual TDM sample clock of exactly 16,000 Hz. The hardware derives that clock from its source clock using dividers with finite precision; divider rounding and source clock tolerance can leave the physical sample rate slightly above or below the requested rate. Rate matching compensates for this difference between the nominal PCM sample rate and the satellite’s actual playback rate.
Without correction, the same number of input samples takes slightly more or less time to play than the PCM format specifies. During continuously paced playback this can gradually fill or drain the playback buffer. Home Assistant can also deliver audio in bursts, but packet arrival speed does not define the PCM sample rate and is not the correction target.
With resampler: true, the component automatically resamples nominally 16 kHz
PCM to match the satellite’s estimated hardware playback rate:
- Playback starts at the nominal 16,000 samples per second.
- The component estimates the hardware consumption rate from the duration of successful blocking I2S writes containing a full frame of playback audio. This is an estimate from write timing, not a direct measurement of the clock signal: DMA buffering and task scheduling can affect it. It is the only correction target; network throughput is not the PCM sample rate.
- The measured rate is clamped to 15,000–17,000 Hz, and the active correction moves toward it one hertz at a time instead of changing abruptly.
- A continuous fractional-phase linear interpolator produces slightly more or fewer samples for the actual hardware output rate. Phase and the preceding sample carry across packet boundaries, avoiding a discontinuity at each packet.
The learned playback rate is retained between streams so a later response can begin with the previous correction. Stopping or clearing playback resets interpolation state so samples from two unrelated streams are never blended. Short-term delivery variation — bursts and gaps from the network — is absorbed by the one-second playback buffer rather than by changing voice pitch.
This is automatic hardware sample-rate compensation. It does not decode compressed audio, accept a genuinely different sample rate, repair severe network starvation, or change the hardware clock. Input must still be signed 16-bit, nominally 16 kHz PCM. Mono and stereo are both supported.
Task scheduling, CPU affinity, and priority
The DSP processing task runs on core 0 at priority 4. On dual-core targets the
playback/I2S TX task runs on core 1 at priority 20; the code selects core 0 for
single-core builds. Playback normally blocks in the I2S driver. This scheduling
fallback does not establish support for targets lacking TDM. If a frame exceeds
its real-time budget (frame_size / 16 kHz), the processing task yields so
system work is not starved. Diagnostics report the processing peak; if backlog
reaches the transport, a later read may also record an rx_error.
FreeRTOS schedules larger priority numbers first. These priorities are intentionally asymmetric. Priority 20 protects continuous I2S transmission from ordinary application work, while priority 4 does not mean that audio processing is unimportant. Per-frame buffers use internal RAM; playback and reference queues use ESPHome ring buffers.
Wake-word cue
The example requests a short confirmation beep after retaining microphone pre-roll. The playback task includes the cue in the speaker/reference path while capture, AEC and voice-pipeline startup continue.