Its not the ignition (if I ran a diesel engine with CH4 I'd get no particulate emissions), or even the type of fuel (if I burn diesel in an appropriate furnace I can eliminate most soot).
It's burning diesel in a diesel engine that's the problem.
Diesel is thick and has a low vapor pressure. When it is injected, therefore, it is more akin to tiny droplets of fuel whose surface burns. Eventually there is -locally- no air around the droplet and the droplet pyrolizes and becomes a particle of soot.
Ironically, diesel engines run very lean (excess O2). But in the neighborhood around the droplet, without good mixing, you get low concentrations of O2.
Gasoline has a low vapor pressure, so it is essentially a gas, not a droplet in the combustion chamber. Therefore there is a flame front across a fairly homogeneous mixtures of gases and there isn't a local lack of O2 (assuming the air ratio is right).
For a gasoline engine this breaks down in two scenarios: Cold engines and direct fuel injection:
- In a cold engine the T might not be high enough to vaporize the fuel and the engine won't start. Therefore more fuel is used and it is in the form of droplets. This is what the choke is for in carburated engines, and now the computer does the same.
- In a direct injection engine there might not be enough time to vaporize the fuel. High P are used (and to shove the fuel into a high P chamber) to encourage vaporization. But its not as effective as port injection.
Direct fuel injection, however, allows for leaner fuel mixtures and more control of the burn (higher compression ratio) so it's basically necessary to achieve efficiency standards.
Another feature/problem of diesel is their high compression ratio. This is how the diesel cycle is more efficient than the Otto cycle (otherwise, the Otto cycle - all else being equal - is more efficient). However, high compression ratios imply high NOx emissions (NOx is an endothermic reaction that lowers P -> high T and high P favor its formation).
So you can see that environmental goals in ICE work against each other. Lowering CO2, or rising fuel efficiency, means rising NOx or soot emissions (or finding a way to shift the curve). You need a transformative technology to improve NOx, soot and CO2 (like fuel injection in the 80s).
In F1, there are no emission rules engines don't have to last 300 000 km but there is a very strict efficiency rule. Here Mercedes claims to have achieved 50% thermal efficiency, which is insane for a small ICE running at 10k rpm and producing 500hp/L. A typical gasoline car gets at best 25~30% in its most efficient state.