If the Don't Fragment flag (DF) bit is set, then internet fragmentation of this datagram is NOT permitted, although it may be discarded. This can be used to prohibit fragmentation in cases where the receiving host does not have sufficient resources to reassemble internet fragments.
One example of use of the Don't Fragment feature is to down line load a small host. A small host could have a boot strap program that accepts a datagram stores it in memory and then executes it.
What it seems to mean is things like PXE[1] or BOOTP[2].
Basically, DF was built to allow the sender to optionally force zero fragmentation by intermediate hosts en-route that are connected to networks with a smaller MTU than the packet size originally emitted by the sender. This was originally intended to be of use because the sender was somehow made aware of limitations in the recipient's network stack.
Probably 20 or more years ago it was used for awhile as a latency-related hack for certain applications (VOIP, video, low-latency finance, certain scientific experiments generating vast amounts of data, etc.) mostly on UDP[3], though we have better methods for those now that operate through other IP headers (QoS).
Theoretically, intermediate nodes could also use it as part of path selection during routing, though I have no idea if this has been done or is encouraged - eg. a packet from node A reaches node B en-route to node C. Node B has two routes to node C. The lower-cost route is available with a smaller MTU than the packet size, and a higher-cost route is available with a large enough MTU to accommodate the packet size. The DF flag could be used by routing logic at node B to automatically shuffle the packet across that higher-cost route.
I believe the Path MTU discovery[4] feature of modern Linux kernels also probably uses this mechanism combined with short TTLs (ie. maximum hop counts, which also cause ICMP errors to be returned on their failure, and are the basis of traceroute[5], itself an unintended hack based on unrecognized capabilities of the IP protocol's specification) to optimize long-lived traffic flows.
There must be other edge-case or optimization-related wishy-washy reasons to set it, too. The main thing to remember is: none of these were really intended by the authors of IPv4. This whole historic pile of what-if edge-case hackyness has been thrown out in favor of a better system in IPv6[6].
[1] http://en.wikipedia.org/wiki/Preboot_Execution_Environment
[2] http://en.wikipedia.org/wiki/Bootstrap_Protocol
[3] http://en.wikipedia.org/wiki/User_Datagram_Protocol
[4] http://en.wikipedia.org/wiki/Path_MTU_Discovery
[5] http://en.wikipedia.org/wiki/Traceroute
[6] http://en.wikipedia.org/wiki/IPv6#Simplified_processing_by_r...