Not sure it's fair to say that graph theorists rarely use matrices to represent graphs: for many problems in graph theory some of the simplest solution involves representing the graph as a matrix (eg Hoffman-Singleton theorem).