Locale handling and most of the str* function in libc indeed are quite bad. The only sane function for "C strings" is snprintf.
In a localized world searching for delimiters also starts to make less sense. Eg. I have been told it doesn't make sense to break on whitespace for Chinese text.
To be clear, I am not saying these functions are bad or evil or to blame for their limitations (a bunch of the problem space wasn't invented yet when they were introduced), just noting they have limits. A bunch of more recent languages and libraries have the same or similar issues, too.
UTF-8 continuation characters are limited to the range \200 through \300 so there's basically zero chance that if you choose something like comma as your delimiter that it's going to tokenize the middle of a multibyte sequence.
Also take into consideration that, under the hood, functions like strpbrk() are typically accelerated by CPU instructions such as PCMPISTRI which doesn't support UTF-8 natively but it does support UCS-2.
Not just "basically;" there is no possible collision between ASCII characters and any valid multibyte encoding. This can be seen somewhat visually in this table[1] and is an intentional aspect of the UTF-8 design.
On joiners / combining characters: I'd encourage using composed normalization (NFC) rather than decomposed normalization (NFD).
Just curiosity: are there any glyphs that lack a single codepoint representation, where one of the joined codepoints is an ASCII character? (That only helps after normalization, of course.)
The ISO C library string handling stuff is for systems programming, not for scanners and parsers for natural written language.
> I have been told it doesn't make sense to break on whitespace for Chinese text.
It could make sense to break on whitespace in some programming language or data format that allows Chinese (and other) identifiers.
A command interpreter that allows Chinese arguments (such as file names) wants to break on spaces, as usual.
> if a delimiter appears as part of a multi byte sequence, you may see strange results.
UTF-8 was designed by a dyed-in-the-wool C-and-Unix engineer, who ensured that such a thing can't happen. No character in the 0x00-0x7F range can occur in a multi-byte character.
The byte which starts a UTF-8 character cannot occur anywhere other than at the start, which is why we can use strstr to look for it.
I quit using it myself because I could never remember just what the exact protocol was for 0.