There are many approaches to this problem. E.g., Jupyter notebooks implement one that has matured in the wild over a decade. There's this very flexible markdown-it plugin that implements anther
https://github.com/goessner/markdown-it-texmath, and my version of it here
https://github.com/sagemathinc/cocalc/blob/master/src/packag... which I rewrote in typescript with a focus on the same semantics as Jupyter has, but for CoCalc. I've also been working on using unifiedjs to provide more general latex for Markdown (not just formulas) here
https://github.com/sagemathinc/cocalc/pull/5982Parsing math in markdown is easier if you use a plugin to an existing markdown parser, rather than trying to do some hack outside of that (which is what Github probably does, and also what Jupyter does).