Why are the nodes rotated like that? It seems very distracting, and I think that might also be partly causing your "layout is too spread out" issue? [1] It could also be exacerbating your second issue of crossing edges - you are getting crossing edges in even the simplest graphs (e.g. your "f1" function example, with 5 nodes).

GraphViz's "dot" algorithm (i.e., Sugiyama-style graph drawing algorithm) [2] should give a fairly compact representation that is organized into layers, and avoids crossing edges in at least simple cases, but rotating the nodes would again "spread out" the layout by forcing increasing height of each layer.

Under "Possible solutions" you mention "create a better graph layout algorithm" - that sounds quite ambitious, wouldn't this be a PhD-thesis-level research task in itself?

The only graph drawing library I'm aware of that might be competitive with GraphViz's algorithms is MSAGL [3] but that's a .NET library.

[1] https://github.com/rgleichman/glance/issues/1

[2] https://en.wikipedia.org/wiki/Layered_graph_drawing

[3] https://www.microsoft.com/en-us/research/project/microsoft-a...