How can video retrieval handle multiple modalities at different times?

Video RAG inherits a problem text RAG does not have: the same content appears in multiple modalities (visual, audio, subtitles) at related but offset timestamps, and naive retrieval treats them as independent chunks. TV-RAG adds time awareness in two places. Retrieved text is ranked using temporal offsets — passages closer in time to other relevant matches score higher — and key frames are selected using entropy-based sampling rather than uniform stride, which concentrates attention on moments where the visual signal carries information rather than redundant near-duplicate frames.

The combined effect is cross-modal alignment. By jointly conditioning text retrieval on temporal proximity and visual sampling on visual entropy, TV-RAG produces a packet of evidence where the subtitles, frames, and audio refer to the same moment in the video rather than to drifting time windows. This matters because reasoning about long video — the kind a video LLM is supposed to do — frequently requires combining what was said with what was shown, and this works only if the retrieved evidence is actually synchronized.

The result is also training-free. The temporal ranking and entropy sampling are imposed at retrieval time without modifying the underlying video LLM, which makes the technique deployable on top of existing systems. The general principle is that for any retrieval over a temporally-extended source, time should be a first-class ranking signal rather than a byproduct of which chunk happened to be cut where. Can byte-level models match tokenized performance with better efficiency? uses entropy in the analogous role at the input-encoding layer — concentrating representational effort where information density is highest.