Neutral grey makes sense in two cases:
- Relative color schemes in which your elements can be either lighter or darker than the background.
- Precise color grading, because white and black backgrounds shift color perception too much.
If you feel the background is too bright, either add more light in the room, or reduce the monitor brightness. It's all relative, it physically cannot have too much absolute brightness to hurt your eyes. The daylight is orders of magnitude stronger but you have no problem with it because your eyes adapt. What hurts them is excessive contrast: staring at the monitor in the dark room, pure black color schemes on OLED screens, etc. This looks jarring and breaks eye adaptation.