It is unlawful to place orders deliberately intended never to trade. Obviously, every market maker has to place and cancel orders, but every one of those orders is submitted with the knowledge that they might trade, an eventuality the market-maker has to be prepared for.

In a layering spoofing scenario, the trader is deliberately structuring their orders to make it virtually impossible for them to trade. The point of doing that is to create the illusion of volume; the orders the spoofer places with actual trading intent will probably be in the opposite direction of their giant spoofing orders.

Obviously, lots of legitimate trading strategies rely on forms of deception (most often: "These piecemeal trades are not part of a concerted effort to buy a zillion shares, I promise"). Where the line is seemingly drawn is, your orders can conceal information from the market, but they can't directly inject fake information.

Here's a detailed SEC complaint, which accompanied a settlement, that does a decent job laying out how regulators establish someone's submitting orders with an intent to deceive rather than trade:

http://www.sec.gov/litigation/complaints/2015/comp-pr2015-4....