I'm not looking for a book on investing. I'm looking for something much lower level, that dives into the inner-workings of financial systems and how those systems have evolved over time.
To make an analogy with computing, I'm looking for something like SICP or "The Elements of Computing Systems" but for finance.
The book 'Wealth of Nation' by Adam Smith covers the allocation of human capital. Adam Smith argues the ultimate resource being allocated here, really, is time, and how individual's spend their time. When we measure output in dollars, what we're examining is the money value of time.
It's hard to get someone to agree that Finance can be a positive-sum game, if they refuse to accept that society is better off through the efficient allocation of human capital / time. Someone faulted me for being "greedy" and "selfish" to be a programmer in Silicon Valley. That by donating money to the Syrian crisis, I was merely "paying off the debt that you created". They insist I quit my job and go protest the United States government to allow Syrian refugees in. They suggest I quit my job, move to a 3rd world country, and help poor children. I sleep well at night because I read Wealth of Nations. I believe there is no place in the world that is comparable to Silicon Valley. Here, my skills / knowledge as a programmer is fully utilized. As such, I maximize my money, and, should I choose, donate money to causes I care about, to employ people who are more efficient at humanitarian efforts, but not necessarily programming.
Finance allocates money, which, we hope, leads to production of goods / services / jobs. The most efficient allocation of money is the one that produces the most. From first principles, I don't see anything inherently wrong with Finance. By putting money to work over time, rather than lay stagnant, Finance can be viewed as the time value of money. As a society, we'd want Finance and Financiers who help economies grow by efficient allocation of money.
This dichotomy of money-value-of-time versus time-value-of-money is not an original thought of mine. I was first exposed to this idea from a Finance PhD candidate's thought about their program. I couldn't find the article on Google. It'd be great if someone can link to it.
To be as pithy as I possibly can, finance is basically politics with numbers attached. Studying history has given me a great insight as to how and why modern financial systems work.
Be careful what books you read in your search, more than a few are rhetoric that's shaped like education.