You wouldn't use it in an offline-first application. But you can use it an offline-first application, and it isn't even very difficult.

I have built electron apps for fun that utilize React Server Components entirely offline.