Smaller batteries are capable of delivering less current, but also only capable at charging at lower currents. So you can’t charge the individual cells in a battery any faster than the battery as a whole.
If you tried to take a single little cell, and charge it at power level of an entire EV battery, all you’ll get is a small explosion and a fire. If you take one of the individual cells out of a Tesla, it’s not much bigger than a AA battery. The charging cable of a super charger is much thicker than a AA battery, and made out of a substantial better conductor than a battery. Just imagine what would happen if you took a battery the size of a AA cell, and plugged it straight into that cable, running at 480V and pushing almost a thousand amps. They peak at 250kW, about 50 households of power usage. Your little battery won’t stand a chance.
Maybe if they switched to a different chemistry like Lithium Titanate they could chaemrge quicker. For now it is easier to use LiFePo batteries and just swap them out.
(1) Heat dissipation. You're charging the battery with a much higher current than the discharge current. Even if you can push a coolant through the battery at the charging time, it adds serious thermal loads, and associated mechanical loads, to it.
(2) Power at the charging point. You're pushing dozens or even hundreds of kilowatts, this takes both high voltage and high currents. Cables and connectors aside, the particular charging place just needs a pretty beefy cable from the electric mains. A typical cable going to a single-family house has a 20-30 kW limit. Now imagine a station with 10 charging points.