A matrix is a collection of elements and nothing more. Indexing starts at 1 because that's the first element in your matrix, and numbering the first element 1 makes sense. Numbering from 0 makes sense when it represents an offset into something, but a matrix isn't that. 0-based indexing just always feels to me like letting the implementation details leak out. (I don't feel this way when an array actually represents a chunk of memory, rather than a math object.)
The proliferation of +1 and -1 depends on the application. Some work better with 1-indexed, some work better with 0-indexed. Personally I get annoyed at having to use `len-1` too often when working with 0-indexed arrays. This is why some languages like Julia (and FORTRAN apparently?) let you choose your index-base, which... has trade-offs.