As I understand it the transaction throughput is limited by the block size. The difficulty adjusts so the rate at which blocks can be mined remains roughly constant, but each block is size limited so it can only contain so many transactions.
The purpose of this restriction is to protect the decentralization of bitcoin nodes. It is this decentralization of nodes that separates bitcoin from every other crypto competitor, and by some margin.
I'm not really sure how a small block size prevents centralisation?
An increase in blocksize means an increase in propagation delay. Too much delay and miners who are closest to the most recent block get a head start mining the next block. Mining is suppose to be a lottery, not a race.
Research done on the matter shows that for each kb above the current 1mb block, 80ms of delay propagation is added.
Couldn't/Wouldn't the difficulty be adjusted down in that case so the blocks are mined faster?
Yes. Correct me if I'm wrong, but I think it's like a regular bus service to a desirable destination. 40 seats every few minutes. Highest fares at any given moment get a ride. But if there are hundreds of people wanting to go, those willing to pay peanuts get left waiting.
Purely because there are more people wanting to make transactions. The speed of the network is always ~10 minutes per block.