1. If a favicon is missing or in a format that's not compatible with the browser. The browser will look for a /favicon.ico file at the root of the site. It's worth having that as a backup or you end up with no favicon and 404 errors in the console.
2. Google now shows a favicon in some search results. I'm pretty sure it works fine with SVG. But, if you use an emoji you end up with a missing glyph character as your favicon. it looks bad.
3. If it's inline it won't be cached
I went a bit overboard optimizing the favicon of my site, and found that:
If you design your favicon as colored squares on a 16x16 grid, and save it as a 32x32 .ico file at https://example.com/favicon.ico
You end up with a smaller file size icon, that can be cached, doesn't have any markup on your site, and scales up perfectly on all the icon sizes.
here's an example: https://doodad.dev/favicon.ico
it's just 625 bytes (261 minified)