As noted, it's not a criticism of a reading list tailored for you, I simply feel it's a really confusing reading list given the likely audience that will arrive there from Hacker News and the title. I also don't think it can be pieced together to form a good introductory reading list or that the contents as stated are "foundational" yet given the progression and holes.
Your (1) confuses me still - I admittedly don't know how you got started in mathematics or how that's relevant here?
Regarding (2), there certainly is a place for neuroscience to influencing thinking or introduce new ideas, and I'm certainly not disregarding the entirety of that potential intersection, but it's a very specific field that is still largely disconnected from the practical application of neural networks. This may change, slightly or substantially, in the future, but thinking again with the lens of the audience of Hacker News (where the incorrect adages of the style "deep learning learns just like a human brain" gets thrown around frequently), I rarely want to exaggerate the influence that neuroscience has on the field at this stage. I'd also lightly note that appealing to authority isn't an argument, though do note the author's pedigree.