I explored using an off the shelf syntax highlighting tokenizer (Pygments specifically as a it was a Python prototype) [1] to build character-based diffs. Got surprisingly good results from what I tested. Most off the shelf syntax highlighters support a huge variety of languages. One advantage to using a syntax highlighting tokenizer/lexer is that they are designed to be fast (many of them need to be real time in text editors after all), and handle "degenerate cases" very well (text editors spend a lot of time on work in progress code that doesn't fully describe a complete compilable unit; version control systems just as well need to be good at storing work in progress states).
The finished prototype I built created standard character-based unidiffs, and it sounds like could possibly be slotted directly into Pijul now. I'd be very interested to experiment further with it now that one of the patch algebra VCSes has finally adopted a "chunk" option.
[1] https://github.com/WorldMaker/tokdiff
(ETA: Suppose the first step is that it is time to get the band back together and update the prototype to Python 3.)