Yes, the process trade offs are not so good if you want embedded flash. You need a process that can withstand the high voltages for flash erase, which means thick oxides and slow transistors.

At 320Mhz, you should have icache and paged code. A process that can clock this high should have no problem with RAM densities to offer more than 16kB. Better to mirror what the entire industry does: low clock speed and embedded flash, or better process with faster clocks, more RAM and external flash.

Quad SPI NOR is still SPI, which requires serial timings. You need to serialize address and data with each transaction. Parallel NOR has parallel address and data, and an order of magnitude improved throughput and latency.

There is really no point to 320Mhz with such slow code memory.