Optional GRU barge-in detection
The latest component includes an optional learned detector for independent near-end speech during playback. Use it when a full-duplex voice-assistant integration can accept microphone audio during its spoken response and handle an interruption. Ordinary wake-word recognition during music or alarms does not require this detector. AEC continues running in every mode.
The supplied voice-assistant example does not enable barge-in. Adding a detector cannot make a standard assistant session interruptible by itself: the assistant/provider must implement that response lifecycle. The optional bundled assistant engine requires separate configuration, as described below.
The detector is included in release v1.01. The shared speech-protection
hold-over described below is an unreleased development change; it is not part
of the existing v1.01 tag.
How VAD is selected
| Device reference playback | Voice assistant active | VAD source |
|---|---|---|
| No | Either | SpeexDSP |
| Yes | No | SpeexDSP; processed audio goes directly to the wake-word listener |
| Yes | Yes | GRU, after consecutive-frame confirmation |
The component exposes one speech decision through get_vad_state() and the
microphone’s is_near_end_speech(). get_vad_probability() follows the selected
detector. Playback is determined on the capture timeline from the reference,
including its configured tail, rather than the speaker queue alone.
The assistant-active lambda is evaluated on the main loop and its atomic
snapshot is read by the audio task. Keep that lambda inexpensive.
Feature normalization and the 16-frame context update in all valid states, matching training. Dense/GRU inference runs only when playback and assistant activity coincide. Leaving this scope clears confirmation without restarting the context. Explicit disable/reset and invalid input reset causal state. There is no shadow option, duck stage or detector cooldown.
Configuration
Merge this fragment into a compatible full-duplex assistant configuration:
aec_speexdsp:
id: voice_audio
# Keep the board's existing ADC, pins and reference slot mapping.
frame_size: 256
microphone_slots: [0, 2] # Waveshare 7B; verify other boards.
output_channel: first
vad: true
neural_barge_in:
enabled: true
voice_assistant_active: !lambda return id(voice_assistant_engine).is_running();
consecutive_frames: 3
speech_protection: true # Preserve speech sent during full-duplex playback.
log_interval_ms: 5000
# Omit threshold to use the checkpoint's calibrated 0.9995.
Requirements: 16 kHz capture, exactly two microphone slots, frame_size: 256,
vad: true, a synchronized usable reference, and a single processed output
or beamforming. capture_only is incompatible. The model is omitted from
ESP32 firmware unless enabled: true is configured. Explicitly configuring
enabled: false also omits it; the runtime enable setter cannot enable a
feature that was not compiled in.
neural_barge_in option |
Default | Meaning |
|---|---|---|
enabled |
false |
Compile and enable the detector. |
voice_assistant_active |
Required when enabled | Boolean lambda defining the active assistant session. |
threshold |
Exported model value, 0.9995 |
Probability cutoff (0–1); a YAML override replaces calibration. |
consecutive_frames |
3 |
Consecutive qualifying frames, range 1–32, before confirmation. |
speech_protection |
false |
Enable neural residual-echo protection. In the development version, onset uses GRU probability and confirmed speech follows the capture gate’s hold-over and early cut-off. Requires an armed barge-in gate and active neural assistant/playback scope. |
log_interval_ms |
5000 |
Sampled diagnostic logs; minimum 250 ms. |
Hardware reference samples reach the detector directly from their TDM slot. Validate reference integrity and assistant-only false triggers before tuning the threshold. A high threshold can reject normal speech, and lowering it can accept residual echo. Threshold tuning alone cannot separate overlapping speech/echo scores. The normalizer adapts continuously, but threshold and confirmation settings do not auto-tune.
Capture gate and assistant integration
Enabling GRU changes VAD selection. It does not automatically enable microphone
upload gating or interrupt speaker playback. A compatible assistant can call
set_barge_in_active(true) on the microphone when entering its full-duplex
response and set_barge_in_active(false) on listening, response completion,
idle and error. These calls are idempotent; repeat flush notifications must
not truncate an utterance already in progress.
While the capture gate is armed, confirmed VAD opens delivery with retained
512 ms pre-roll, followed by continuous cleaned PCM through a one-second
hangover renewed by each confirmed frame. Rejected positions before opening
are masked; weak words during the open hangover are retained. Playback resuming
after at least 300 ms of reference silence clears the inherited hangover before
processing the new frame. Already queued user audio survives; fresh confirmed
speech can reopen delivery. Short playback gaps do not cancel it. AEC continues
adapting. Residual echo can still pass during the bounded open interval, so test
assistant-only playback as well as complete interruption questions.
An assistant integration must also manage upload capacity, response cancellation,
queue flushes, completion and error cleanup. The component offers
set_delivery_condition(...), request_pre_roll(), begin_pre_roll_replay()
and discard_pending_audio() for that integration; these are not a substitute
for an assistant/provider interruption protocol.
For an ordinary non-barge-in session, use set_response_playing(true) during
TTS and release it when the response completes. This suppresses response audio
and its acoustic tail from delivery/history. Do not combine that muting policy
with a full-duplex response that needs to hear interruptions.
Speech protection (unreleased)
For a full-duplex response, enable neural_barge_in.speech_protection: true
and arm the microphone with set_barge_in_active(true). The setting remains
opt-in (false by default). Opening delivery alone does not protect the audio:
the GRU examines raw microphones, but the provider hears post-Speex PCM.
Before confirmation, GRU probability relaxes residual echo suppression to preserve onset candidates in history. On the confirming frame, protection reaches full strength. It stays at full strength for the capture gate’s one-second trailing window even when probability falls during weak words or pauses. The preprocessor reads the gate’s capture position before processing each frame; it does not maintain a separate wall-clock timer.
When playback resumes after at least 300 ms of reference silence, both protection and delivery cancel the inherited hold-over on that same frame. Fresh confirmation can reopen it. Short gaps and repeated flush/start calls do not truncate an ongoing utterance, and queued user PCM is retained. Candidate protection does not retroactively reconstruct audio already suppressed before the detector found evidence of speech.
The extra probability is zero outside the armed barge-in gate during enabled neural assistant/playback scope, including neural disable, inactive assistant, and no active playback reference. Speex then uses its normal speech probability and suppression settings. Echo cancellation, noise suppression and AGC remain in operation; full neural protection selects the configured active-speech residual echo suppression, rather than bypassing DSP or sending raw microphones. Check intelligibility and assistant-only false triggers at your intended volume.
This fix is currently available in the local development checkout only. The
ref: v1.01 snippet below loads the released implementation, without the shared
hold-over fix. To test the local checkout, use a local external component
source pointing to src/esphome; after publication, pin the commit or release
containing the fix.
Independent probabilities for testing
sensor:
- platform: aec_speexdsp
aec_speexdsp_id: voice_audio
speex_vad_probability:
id: speex_vad_probability
name: "SpeexDSP VAD Probability"
disabled_by_default: true
gru_probability:
id: gru_barge_probability
name: "GRU Barge-In Probability"
disabled_by_default: true
update_interval: 500ms
These optional entities report 0–100 percent. Either may be omitted. Displays
can read id(speex_vad_probability).state and id(gru_barge_probability).state.
Speex reports independently while GRU supplies VAD. GRU reports zero when
inference is inactive, disabled or not compiled, so zero outside the scope
means no current inference. Polling samples the latest frame, not a peak over
the interval. Direct getters get_speex_vad_probability() and
get_neural_barge_probability() return 0–1.
Model provenance and validation
The bundled raw84 model uses 72 band-major Bark log powers (microphone 1, microphone 2, reference) and 12 compact spatial delta values. A Dense32 layer, GRU24 over 16 causal frames and one output contain 6,921 learned parameters. These inputs are raw channels, not echo-cancelled residual features.
The model was trained on captured data from the Waveshare 7B board, using corrected acoustic labels and separate fitting, calibration and held-out evaluation sessions. Validate it on your own board, room and playback volume.
Checkpoint SHA-256:
d0a8ce88f4021c72923ac7ac862066d70e4e374976d22d7ad5b27ab76b7dd41e.
The header exports weights, training-only initial normalization/spatial
statistics and threshold together. src/esphome/aec_speexdsp/barge_in_model.json
retains capture-session split IDs and dataset, label and training-source hashes.
The training captures and checkpoint are not shipped in this component repository.
Use host routing/preprocessor tests and C++/PyTorch parity before changing
model semantics. On hardware, test playback-only segments, quiet speech over
TTS, early/repeated interruptions, wake-word handoff and error recovery at the
intended volume. NEURAL_BARGE logs active scope, probability, confirmation,
events and CPU budget; NEURAL_FEATURES logs context, resets and normalization.
The current model was exercised on ESP32-P4; measure resource usage before
enabling it on other targets.
Optional bundled voice-assistant engine
Load both components from the same v1.01 release when using the custom
full-duplex engine:
external_components:
- source:
type: git
url: https://github.com/matt123p/esphome-aec
ref: v1.01
path: src/esphome
components: [aec_speexdsp, voice_assistant]
refresh: 0s
The ordinary example keeps its standard one-request lifecycle. Loading the
optional engine alone does not enable barge-in or install capture callbacks.
For full duplex, set voice_assistant.barge_in: true and connect the capture gate:
esphome:
on_boot:
priority: -90
then:
- lambda: |-
id(voice_assistant_engine).set_barge_in_capture_gate(true);
id(voice_assistant_engine).set_barge_in_audio_condition([]() {
return id(voice_microphone).is_near_end_speech();
});
id(voice_microphone).set_delivery_condition([](size_t bytes) {
return id(voice_assistant_engine).can_accept_microphone_bytes(bytes);
});
voice_assistant:
barge_in: true
on_tts_start:
- lambda: |-
if (id(voice_microphone).set_barge_in_active(true))
id(voice_assistant_engine).reset_microphone_upload();
on_tts_stream_start:
- lambda: |-
if (id(voice_microphone).set_barge_in_active(true))
id(voice_assistant_engine).reset_microphone_upload();
on_listening:
- lambda: id(voice_microphone).set_barge_in_active(false);
on_tts_stream_end:
- lambda: id(voice_microphone).set_barge_in_active(false);
on_error:
- lambda: |-
id(voice_microphone).set_barge_in_active(false);
id(voice_microphone).discard_pending_audio();
on_idle:
- lambda: |-
id(voice_microphone).set_barge_in_active(false);
id(voice_microphone).discard_pending_audio();
Merge these callbacks into your lifecycle rather than enabling the example’s response-muting policy at the same time. Use continuous conversation startup and a state-aware idle/response watchdog when the provider supports it. Keep wake-word pre-roll handoff and listening release in your integration.
Gemini/provider interruption is authoritative: the detector releases microphone audio, and Core’s flush-marked TTS stream start purges queued playback while preserving user capture. Assist RUN_END can precede TTS playback completion; the engine waits for stream end and speaker drain before starting the next pipeline with the same conversation ID.
The engine retains and retries refused microphone packets. After two seconds of sustained backpressure it stops and retires the API connection. A failed stop packet also retires it. Startup acknowledgements have a ten-second limit; failure stops capture and reconnects rather than waiting indefinitely. Late PCM and TTS events during idle/startup/stopping are ignored. These are recovery measures, not evidence that network or provider stalls have been eliminated.
NEURAL_BARGE reports candidate_frames and confirmed_frames alongside
infer_frames for each diagnostic window. Compare acceptance duration, not just
event counts. A flush while the user is silent is a false interruption; neither
packet delivery nor a successful host test proves intelligible acoustic input.