SDL does not need a windowing layer... There might be better docs somewhere but this should act as enough proof for you without linking directly to the source code.

https://wiki.libsdl.org/SDL2/SDL_HINT_KMSDRM_REQUIRE_DRM_MAS...

SDL can provide a context in the native windowing system, if you ask it to. Or not. You have to ask for it though, it's not the default.

I think sdl works directly with fb when using fb as a backend,no windows system is needed