HNHacker News
TopNewBestAskShowJobs

smacke

466 karma · joined February 9, 2019

https://github.com/smacke
submissionscomments
smacke··on Show HN: Automatically synchronize subtitles with video
Actually, it unfortunately doesn't work if the drift gets worse over time -- so far, it only works with constant drift. Maybe fixing constant 1st derivative drift is the next step!
smacke··on Show HN: Automatically synchronize subtitles with video
It's actually fairly annoying for subtitles, but it's useful for getting videos in unfamiliar languages to the right ballpark. Afterwards, my go-to so far has been to just perform a manual one-off adjustment in VLC after the video starts (you can offset subtitles in increments / decrements of 50ms with hotkeys h and j).
smacke··on Show HN: Automatically synchronize subtitles with video
Precisely. If I could more easily get at the raw classifier output of webrtcvad, it should be possible to be even smarter (we could have square waves with any amplitude between -1 and +1, not just either -1 and +1, which should take into account the classifier uncertainty).

EDIT: err, I'm actually not sure about the pitch shift part, that's a bit of vocabulary I'm not familiar with. If you've seen the fast polynomial multiplication algo from CLRS, it's basically that. E.g. if we have strings 1101 and 0101, we can find the best alignment by looking at the exponent of the largest coefficient after multiplying

polynomial(1101)*polynomial(reverse(0101))

where polynomial(1101) = x^3 + x^2 - x + 1

and polynomial(reverse(0101)) = polynomial(1010) = x^3 - x^2 + x - 1

smacke··on Show HN: Automatically synchronize subtitles with video
Audio extraction is actually the most expensive part. It depends on the length of the video, but my experience is that it finishes in 20 to 30 seconds. It's possible one might be able to sample different parts of the video to bring the runtime down further; I plan to experiment with this when I get the chance.
smacke··on Show HN: Automatically synchronize subtitles with video
Pretty much! Given the anecdotal success I've personally had with this approach, I'm hoping it could get picked up by VLC so that the algo can be run in there directly.
smacke··on Show HN: Automatically synchronize subtitles with video
The approach taken here is to not worry about those. :)

These will introduce some noise, but the synchronization algorithm seems to be robust enough that noise doesn't matter (whether it's from the voice activity detector or from non speech-related subtitles).

smacke··on Show HN: Automatically synchronize subtitles with video
Really appreciate the feedback -- will definitely work on improving the clarity in the README. Glad you like the project!
smacke··on Show HN: Automatically synchronize subtitles with video
Totally agreed. I'm not sure how to do the alignment scoring step in this case, perhaps some kind of DTW / FFT hybrid... definitely worth further consideration.
smacke··on Show HN: Automatically synchronize subtitles with video
Interesting! I was not aware of that project before. It looks like it could be worthwhile to incorporate their neural net-based VAD, if it's not too slow. At a first glance, it looks like the main difference is the postprocessing step -- they use some heuristics to avoid trying all possible alignments, while subsync uses FFT as its secret sauce to get away with trying out all of them. :)

I should include in the readme that for >1 hour movies, it usually finishes the synchronization in ~20 seconds, which compares favorably to the project linked in your comment (13 minutes unoptimized, 2 minutes optimized).

smacke··on Show HN: Automatically synchronize subtitles with video
By speech recognizer, are you referring to a method that would perform speech-to-text transcription? If so, you could probably do better when your video and subtitles are in the same language, but you would lose the ability to be language-agnostic (unless you manage to solve machine translation well enough :)

Interestingly enough, I haven't actually found any cases where the synchronization doesn't work (assuming the only problem with the subtitles file is a time offset), so it actually looks like simple voice detection might be good enough for the target use-case of longer TV episodes and movies, although further evaluation is necessary.

smacke··on Show HN: Automatically synchronize subtitles with video
Thanks for flagging -- this seems to be a common point of confusion; I'll work on updating the readme.

Let's say reference.srt is a set of English subtitles, and you have an out-of-alignment French subtitles file unsynchronized.srt. You could use the video directly as a reference in order to synchronize the French subtitles, but it will take a bit longer since we need to extract audio and perform voice detection over the whole video file. In this case, it will be faster to use your already-synchronized English subtitles file reference.srt.

smacke··on Show HN: Automatically synchronize subtitles with video
I think they were referring to the fact that "reference.srt" would already be synchronized.
smacke··on Show HN: Automatically synchronize subtitles with video
Great question. The main use case I can think of is when reference.srt and unsynchronized.srt are in different languages, and you want to eventually merge them into a single dual-language subtitle file.

EDIT: Oh, I should also mention that you don't need a reference.srt -- it can look at the video directly and use that as a reference.

smacke··on Show HN: Automatically synchronize subtitles with video
Cool! It looks to me like most of those approaches are a lot more sophisticated and work even when the text doesn't have any time annotations. At a first glance, it looks like the major downside is that these approaches are geared toward particular target languages. Subsync's advantage is its simplicity: by limiting the acoustic analysis part to simple voice detection, it can perform alignment between subtitles and audio/video in different languages, which is great for when you want to watch a TV episode in another language, but your video includes the previous episode's recap and your subtitles do not (or vice-versa). I'll definitely be taking a deeper look at these to see if there's any ideas / functionality that can be further incorporated.
smacke··on Show HN: Automatically synchronize subtitles with video
Yes! it should work for this too, as long as ffmpeg recognizes the audio format :)
← PreviousPage 3 of 3