I’ve been hunting down some audio/video drift issues over the last few days with OBS-NDI. Here’s what I’ve been able to find.
Note this is based on my understanding of reading through the code for both OBS and obs-ndi. It may be inaccurate.
-
Each OBS source maintains a “last timestamp” and “next expected timestamp” for audio. If the source sends audio to OBS that is later than the expected timestamp, buffers get adjusted.
-
If the source’s audio buffer gets “too large,” audio gets dropped instead of getting added to the buffer.
-
OBS will resample audio it receives from a source to match the internal sample rate OBS is running at. So does NDI, depending on how you set up that source. In OBS-NDI, the source’s sample rate is used and passed to OBS.
-
The latest version of OBS-NDI allows use of the framesync APIs. This will do some resampling/stretching as well. Note this receive and decode code path is separate from the typical “receiver” path.
-
Sources aren’t compelled to send video/audio frames at regular intervals. OBS seems to not like it when it doesn’t receive an audio frame for some time, then gets a frame. (You can try this with Screen Capture HX. Make no sound, then make some sounds, then make no sound.)
-
Network vs. Source timing doesn’t change much - just which frame property OBS looks at to determine a frame’s timestamp. Each NDI frame has a timecode and a timestamp. Timecode is supposed to be nanoseconds since Unix epoch, but it’s not guaranteed (My Mevos transmit time since last power on).
-
Timestamp is time frame was submitted to the NDI SDK - which could be an issue because it’s when a frame is passed to NDI by an NDI source for transmission, not when it was received by the receiver.
In NDI, timestamps are based on when the sender gave NDI a frame. Not when the receiver got the frame. Timecodes can be synthesized, but the source can also generate them themselves.
It seems like if OBS gets an audio frame that’s “too old” in it’s opinion, it could try extending the buffer or resampling audio to make it fit. And in the audio processing for placing audio in the circular buffer in OBS, if the buffer would become too large, it short-circuits out of the method. Meaning the audio never gets placed in the buffer.
tl;dr I believe there is a fundamental incompatibility between OBS’ internal sync and the way OBS-NDI handles sync. OBS-NDI passes timecode/timestamp direct to OBS, which I think could cause issues with sync.
UPDATE - 2024-01-10: I forked obs-ndi and calculated timestamps a bit different. I’ve been testing for a few minutes in my worst-case scenario - sending my local feeds via bridge to cloud over a garbage cable connection with a 150ms delay on bridge.
UPDATE - 2024-01-11: UPDATE: after chatting with the folks behind OBS, and taking a look at a similar situation (the DeckLink driver), one of the solutions that seems to work is decoupling the audio/video for NDI, and synthesizing some timestamps for audio. So far, so good!
So far, sync has been just about perfect. No audio glitches either. The odd frameskip, but that can be sorted later.