Sure. But that difficulty all comes down to whose shoulders you stand on. If the peripherals' functionalities and their registers are fully described in a datasheet, and you have a working toolchain, and a viable method for loading binaries, then you are set. Additional libraries on top would make it even easier, but you at least have the device's specifications.

Companies like Qualcomm, Broadcom, and Marvell are notorious for locking down their documentation. If you're not shipping millions of units, they're not going to give you the time of day. And even if they are willing to work with you, they're not going to provide you with full register maps, or really anything that would allow you to be independent of them. Instead, they'll give you binary blobs that handle low level details. Every datasheet is watermarked, and NDAs are required to get them.

Most of the other big players, STMicro, Texas Instruments, Nordic, Microchip, etc, are far more open. Just about all of their parts are well documented, and those documents are freely available.