It is not the case that treatment of spaces is completely wild and ad hoc. PDF can, and frequently does, contain the text itself along with the display of said text. This is why applications like pdftotext can often give you correct plaintext output.
OCR applications are able to make PDFs from scanned images accessible and searchable (and with the right spaces) precisely because they embed their output into that layer of the PDF meant for the text itself, not its presentation.