Neurosurgery is the hardest residency, and the hardest physician career. However, this difficulty is in-part due to the bravado of the neurosurgeons themselves. During neurosurgery residency, you have to learn to be a brain and spine surgeon, undertake two years of research, learn how to do endovascular interventions, and (of course) manage all of the pre-/post- operative care including clinic. It is not uncommon for a single PGY-2 neurosurgery resident to follow 100+ inpatients. Suffice to say neurosurgeons are overburdened, yet they themselves have fought against reducing training hours.
Patients can generally expect to spend less than 30 minutes speaking to their surgeon before/after their surgery.
For those unaware, you can become a spine surgeon via either the neurosurgery or orthopedic surgery routes. Just my simple opinion, but I think spine surgery should be its own residency. Brains should be left to the neurosurgeons and extremities to the orthopedists.