Picking a set of domain specific targets and honing a tool to service that seems a good way to start building - rather than a general solver.

I also love graphviz but had a project to create an app for legible quality management, and it was a headache to shoehorn some concepts into "nodes in an edge connected graph" way of thinking (ended up programatically scripting LaTeX).

With so many visualisation concepts there's room for niche approaches (same in plotting; Matlab/Octave vs Gnuplot vs Pyplot vs R and so on)

IDK if it's quite the use case you're looking for, but at my company, we turn our CI YAML for our monorepo into Dot (Graphviz), and graph it, to visualize the dependencies between steps in our CI, & to highlight the critical path.

It's an ~100 node DAG. Graphviz struggles a bit with it, particularly with edge layout.

(Though really, the layout I want for it is a Gnatt chart style layout.)

I have made software architecture diagrams, but more "descriptive" than "prescriptive" ones, by generating graphviz from source code analysis tooling. (The objective here was getting oriented in a large codebase where there wasn't really a rigorous diagram of anything written down anywhere, and trying to find sort of a min-cut where putting a line through the graph of imports would cross the fewest edges)

In my experience graphviz handles up to thousands of nodes and tens of thousands of edges without really breaking a sweat.

edit: I've done the same thing with extracting AWS diagrams from aws describe* calls, with good results.

Laying out larger graphs is tricky, often because the size simply stands in the way of generating anything that's useful to the viewer. To add to that, most layout algorithms prioritize optimizing certain criteria, while other parts of the visualization emerge from that; and if the human viewer chooses a layout algorithm because of one of the latter properties, they are often surprised that the result doesn't look like they envisioned (because, hey, the algorithm optimized something entirely different). We see this disconnect fairly often in our own customer support, but I haven't really found a good way of putting an explanation in writing.

Then there's generally the problem of larger graphs, which tend to devolve into a tangled mess and hairballs, simply because they often tend to be well-connected. If there's no way of pruning them beforehand, or perhaps grouping, aggregating or clustering (in a way that makes sense to the viewer, not necessarily only structurally), then it can be hard to get good result.