sdr-wideband-suite/docs/telemetry-debug-runbook.md

Debug Telemetry Runbook

This project now includes structured server-side telemetry for the audio/DSP pipeline.

Endpoints

• GET /api/debug/telemetry/live
  • Current counters/gauges/distributions and recent events.
• GET /api/debug/telemetry/history
  • Historical metric samples.
  • Query params:
    • since, until: unix seconds/ms or RFC3339
    • limit
    • name, prefix
    • signal_id, session_id, stage, trace_id, component
    • tag_<key>=<value> for arbitrary tag filters
    • include_persisted=true|false
• GET /api/debug/telemetry/events
  • Historical events/anomalies.
  • Same filters as history plus level.
• GET /api/debug/telemetry/config
  • Active telemetry config from runtime + collector.
• POST /api/debug/telemetry/config
  • Runtime config update (also saved to autosave config).

Config knobs

debug.telemetry in config:

• enabled
• heavy_enabled
• heavy_sample_every
• metric_sample_every
• metric_history_max
• event_history_max
• retention_seconds
• persist_enabled
• persist_dir
• rotate_mb
• keep_files

Persisted JSONL files rotate in persist_dir (default: debug/telemetry).

5-10 minute debug flow

1. Keep enabled=true, heavy_enabled=false, persist_enabled=true.
2. Run workload for 5-10 minutes.
3. Pull live state:
   • GET /api/debug/telemetry/live
4. Pull anomalies:
   • GET /api/debug/telemetry/events?since=<start>&level=warn
5. Pull pipeline timing and queue/backpressure:
   • GET /api/debug/telemetry/history?since=<start>&prefix=stage.
   • GET /api/debug/telemetry/history?since=<start>&prefix=streamer.
6. If IQ boundary issues persist, temporarily set heavy_enabled=true (keep sampling coarse with heavy_sample_every > 1), rerun, then inspect iq.* metrics and audio.* anomalies by signal_id/session_id.