It’s sort-of optional for most playback stacks because they leave frame reordering to individual decoders as a codec-specific implementation detail, but Apple’s stack actually cares about frame-accurate random access so it relies on the codec-independent container timestamps.
The inaccurate seeking you get without container pts is okay for playback but it falls apart with editing or stuff like av1an.
I have a bit of a love-hate relationship with ffmpeg. Love because it does things nothing else does (e.g. has a main-profile software H.264 decoder) and is so comprehensive, hate because code quality varies (lots of long-unfixed bugs/quirks in trac) and documentation/API/CLI experience is poor. E.g., I wish the documentation for this new bsf explained what you just said.
I really wonder why it costs people so much to write the whys of everything technical they do in at least a source code comment, but it is indeed an everyday's uphill battle.