I really want to learn the algorithms behind automatic layout engines. Are there any books on this topic?
The folks at ELK are doing fantastic work in public as an academic team out of Germany.
https://www.eclipse.org/elk/documentation/algorithmdeveloper... https://www.rtsys.informatik.uni-kiel.de/en/team/soren-domro...
Graphviz's papers are also must-reads (for hierarchical layouts): https://graphviz.org/theory/
In general they are surprisingly readable even without an academic background. Sometimes they go into proofs of correctness like method X really results in minimal edge crossings, and those are gnarly math, but skippable. The bad part is that a lot of these research papers have code that's just broken or wrong, or is NP-hard.