Troubleshooting Common Issues with RTSP/RTMP/HTTP/URL DirectShow Source Filters

Troubleshooting Common Issues with RTSP/RTMP/HTTP/URL DirectShow Source FiltersDirectShow source filters that accept RTSP, RTMP, HTTP and plain URL streams are powerful components for building media playback, recording, and processing applications on Windows. Because they must handle network variability, protocol differences, container/codec heterogeneity, and Windows multimedia pipelines, they can fail in many subtle ways. This article walks through common problems, diagnostic techniques, and practical fixes — from connection failures and format negotiation to latency, buffering, and codec mismatches.

---

1. Understand the pipeline and roles

A DirectShow graph typically includes a source filter, optional transform filters (decoders, parsers, render-time processors), and a renderer (audio/video sink). The source filter’s responsibilities include:

• Opening network connections (RTSP/RTMP/HTTP).
• Demultiplexing containers (e.g., MP4, FLV, MPEG-TS).
• Exposing media streams and media types to downstream filters (AM_MEDIA_TYPE).
• Providing timestamps, seeking information, and other sample properties.

Many issues stem from mismatches between what the source exposes and what downstream filters expect. Before troubleshooting, map your graph: what source filter is used, which parsers/decoders are connected, and what renderers are present.

---

2. Common connection and protocol issues

Symptoms: immediate connection failure, long delays before playback, intermittent disconnects.

Possible causes and fixes:

• Network/firewall/port blocking:

  • Ensure required ports are open (RTSP typically uses TCP 554; RTMP uses TCP 1935; HTTP uses ⁄₄₄₃ or custom).
  • Check NAT and firewall rules on client, server, and intermediate routers. Use telnet or curl to confirm connectivity.

• Authentication/authorization problems:

  • Confirm credentials and supported auth schemes (Basic, Digest, NTLM).
  • Some RTSP servers require separate RTSP and RTP authentication or HTTP tunneling — consult server docs.

• Protocol mismatch (RTSP over TCP vs UDP):

  • RTSP can transport RTP over UDP or interleaved TCP. If UDP packets are blocked, force TCP interleaving in the source filter settings.
  • If jitter or packet loss occurs over UDP, prefer TCP/RTP-over-RTSP.

• Server-side limits and session handling:

  • Servers may limit concurrent streams, require session keepalive, or drop idle connections. Ensure your filter sends periodic keepalives (RTSP OPTIONS or GET_PARAMETER) and handles ⁄₄₀₇ responses.

• Incorrect URL syntax:

  • RTSP/RTMP/HTTP URLs often have stream identifiers, query parameters, or tokens. Confirm exact URL as used by working clients (VLC, ffplay).

Diagnostics:

• Use Wireshark to capture RTSP/RTMP/HTTP exchanges; inspect RTSP responses (401, 454, 461, 459, etc.).
• Test stream with VLC, ffplay, or cURL to isolate whether the problem is server or filter side.

---

3. Container and demuxing problems

Symptoms: audio only, video only, no stream enumeration, or downstream decoders refuse media types.

Causes and fixes:

• Unsupported container format:

  • Ensure the source filter recognizes the container (FLV for RTMP, MPEG-TS for many RTSP streams, MP4 for HTTP progressive). If not, add or enable an appropriate demuxer.

• Corrupt or fragmented headers:

  • Some streams publish codec headers in the stream (SPS/PPS for H.264) later than the first frames. Make the source filter buffer until codec extradata is available and expose correct AM_MEDIA_TYPE with proper subtype and format block (e.g., H264: add AVC1/Annex B / extradata).

• Incorrect timestamps and timebase:

  • RTP streams use RTP timestamps; convert to 100-nanosecond reference time (REFERENCE_TIME) properly. Handle PTS/DTS for container formats. Wrong timestamp mapping causes audio/video drift or refusal to render.

• Stream enumeration errors:

  • Ensure the filter enumerates all available streams (audio, video, subtitles) and sets correct pins. If the server changes stream count mid-session (e.g., switches codecs), implement dynamic reconnection or pin management.

Diagnostics:

• Inspect initial bytes with tcpdump/Wireshark to see container signature.
• Compare media type blocks from the filter with those from working players (MediaInfo, ffprobe).

---

4. Codec negotiation and decoder compatibility

Symptoms: “unsupported format” errors, green frames, artifacts, or audio-only playback.

Causes and fixes:

• Wrong FourCC or subtype:

  • Map codec identifiers (e.g., H.264 can be MEDIASUBTYPE_H264, MEDIASUBTYPE_avc1) correctly. Provide correct format blocks: for H.264 include SPS/PPS or AVCDecoderConfigurationRecord when needed.

• Missing format block (pbFormat):

  • For many decoders the BITMAPINFOHEADER/VIDEOINFOHEADER2 or codec-specific headers are required. Populate avg time per frame, bmiHeader fields, rcSource, and format-specific blobs.

• Hardware decoder issues:

  • Hardware acceleration (DXVA) may require strict format alignment or specific profile/level. If frames fail on hardware, fall back to software decoder as a test.

• Audio codec mismatches:

  • AAC may appear in ADTS or LATM; specify correct format (ADTS vs LATM) and include decoder-specific config (AudioSpecificConfig). For AAC in MP4 streams, provide extradata from the ‘esds’ box.

Diagnostics:

• Use GraphEdit/GraphStudioNext to build a graph manually and see which decoder pins connect.
• Log AM_MEDIA_TYPE contents when connections are made.
• Test decoding with ffplay to confirm codec presence and valid headers.

---

5. Buffering, latency, and jitter

Symptoms: long startup times, stuttering, audio/video out of sync, or sudden jumps.

Causes and fixes:

• Insufficient buffer sizes:

  • Increase socket/read buffers and internal packet queues. For high-latency networks, raise initial buffering (prefetch N seconds of media) to avoid underflows.

• Inadequate jitter buffer for RTP:

  • Implement an adaptive jitter buffer that grows on packet loss and network variance. Keep a small target (100–500 ms) but allow expansion.

• Improper clock synchronization:

  • Synchronize stream timestamps to the system reference clock. Implement or use DirectShow’s reference clock correctly to maintain AV sync. If streams use PTS gaps, smooth timestamp jumps.

• TCP vs UDP tradeoffs:

  • TCP reduces packet loss but increases latency due to retransmission; UDP allows low-latency at the risk of loss. Provide configurable transport and let users choose.

Diagnostics:

• Plot buffer fill level and packet arrival intervals.
• Log dropped packets, reorders, and late samples.
• Compare real-time vs stream time to measure drift.

---

6. Seeking and trick play

Symptoms: seeking not supported, slow seeks, or incorrect position reporting.

Causes and fixes:

• No index or sparse indexing:

  • For HTTP progressive files, use range requests and implement byte-range seeking. For live streams, seeking may be impossible unless server supports DVR features.

• RTP/RTSP server limitations:

  • RTSP often supports PLAY with range NPT/time parameters; ensure the server exposes NPT or absolute time ranges. Implement RTSP PLAY range parsing and reposition logic.

• Incorrect duration or start time reporting:

  • Calculate duration from container metadata when available. For live streams, report unknown duration and disable seeking UI.

Diagnostics:

• Test seeking behavior in VLC/ffplay and compare request patterns.
• Use server logs to see whether range/seek requests reach the server.

---

7. Threading, concurrency, and resource leaks

Symptoms: crashes, deadlocks, memory growth, file descriptor exhaustion, or slow performance.

Causes and fixes:

• Incorrect COM threading model:

  • Ensure DirectShow filters are built with a compatible COM apartment model. Use proper synchronization when accessing shared data across threads.

• Blocking operations on worker threads that serve callbacks:

  • Avoid long synchronous network calls in callbacks that block filter graph threads. Use producer/consumer queues and dedicated IO threads.

• Not releasing media samples or COM interfaces:

  • Balance AddRef/Release on IMediaSample, AM_MEDIA_TYPE, and downstream interfaces. Use smart pointers (CComPtr) to reduce leaks.

• Socket/resource leaks:

  • Close sockets on Stop/Teardown and handle abrupt disconnects gracefully. Reuse or pool sockets where appropriate.

Diagnostics:

• Run with Application Verifier, DebugDiag, or Visual Studio memory/profiler.
• Monitor handle counts, memory usage, and thread counts over time.

---

8. Error handling and recovery strategies

• Graceful reconnection: implement exponential backoff, but allow manual retry.
• Fallback transports: try TCP interleaved RTSP if UDP fails; fall back to HTTP or HLS if server supports it.
• Notify application/UI with actionable error codes rather than raw HRESULTs. Map common RTSP/HTTP status to user-friendly messages.

---

9. Testing and debugging tools

• GraphStudioNext / GraphEdit — build and inspect DirectShow graphs.
• Wireshark/tcpdump — capture and analyze network traffic.
• ffprobe/ffplay — validate stream format and headers.
• MediaInfo — quick overview of container and codec metadata.
• Process Explorer/VMMap — inspect handles, DLLs, and memory.
• Visual Studio debugger and sanitizers — find race conditions and leaks.

---

10. Practical checklist for resolving a problem

1. Reproduce the issue with a known-good player (VLC/ffplay).
2. Capture network traffic and server responses.
3. Inspect AM_MEDIA_TYPE and format blocks when pins connect.
4. Test with software decoder to rule out hardware acceleration bugs.
5. Increase buffers/jitter buffer settings and observe behavior.
6. Check for threading and resource leaks in long-running sessions.
7. Implement graceful retry and transport fallbacks.

---

Troubleshooting networked DirectShow source filters is often a process of isolating layers: network, protocol, container, codec, graph negotiation, and runtime resources. Methodical logging, targeted tests with third-party players, and careful handling of timestamps, headers, and threading will resolve most issues.