Very cool to explore the code here. Have you looked at Visx? It’s a D3+React mashup library, used by Airbnb and Notion for graphs. https://airbnb.io/visx/gallery
The approach is kind of in-between what I suggest and what a classic JS library like Vega does.
It is definitely a very good abstraction imo!! But I'm always scared to be stuck with a customisation I cannot do because of a layer of abstraction.