The reason here is the the "fast way" and the "slow way" work very differently behind the scenes.
The "fast way" looks only needs to look at the container, the bits of metadata that tell a player which bits of data need to be given to the decoder at which time. It can just take a blob of data and stick it in another blob of data without looking at the contents.
The "slow way" actually decodes the frames, that is, it takes the blobs of compressed data and turns them into actual pixels, which especially in the case of 4K video, is very slow.
The reason the "fast way" might be less accurate is that the frame you're asking for might not be possible to obtain without decoding the video. Modern video codecs have different kinds of frames and some frames depend on the frames before or after them. If you took such a frame and just jammed it into another video, things would break, because the other frames it refers to are missing.
> I'm still curious as to how a tool like ScreenFlow can achieve faster trimming/concatenation/subtitle overlaying, than FFmpeg.
It's likely that they have optimisations that FFmpeg doesn't or cannot have. FFmpeg has a bit of an emphasis of being able to play and handle pretty much anything you throw at it, no matter how broken. It could be that accurate input seeking is difficult while preserving that reliability.
One option that's probably not an option for you but you might consider is encoding your video in an all-intra format. This is fairly standard in the video editing world. All-intra means that all your frames are independent of each other and can be moved around by editing software without decoding anything. Doing this will result in larger files, however.