It's not that it's difficult, really, but the tools are incredibly underpowered. For example:
* How do you vertically center an arbitrary element inside another where you don't know the height of any of them? Answer: The only reliable way is via "display: table-cell". It has several annoying limitations.
* How do you ensure that several horizontally adjacent boxes fill the width of the container and are all of equal height, regardless of their contents? Answer: Table again, but it's not always possible without JS because of how height percentages are calculated. (Also, all browsers including Chrome implement "height: 100%" differently.)
* How, given N elements arranged horizontally next to each other, do you ensure that they (1) fill the width of the container, (2) if they become too narrow (ie. min-width), any element(s) that don't fit will wrap below? (Note: Rule #1 must apply; when one element doesn't fit, the preceding "line" must expand to fit the full width of the container.) Answer: Not possible in CSS alone.
* How do you ensure that an element (eg., an image) of arbitrary width is always has the same height as it's width? (For example, imagine a grid which should expand to fit the window, and all cells should be quadratic.) Answer: With images, possible with an unintuitive "height: 0; padding-bottom: 100%" hack, not for other elements without JS.
* How do you center an absolutely positioned element of arbitrary width and height (think tooltip or speech balloon)? Answer: JS.
* How do absolutely position an element without specifying a width, so that the width becomes the minimum needed to display all of its contents up to a max-width? For example, a box saying "Hello" should take up the space required by that word and no more. Answer: Not possible in CSS alone, although you would have thought that "display: inline-block" would work. Firefox actually does it, Chrome/WebKit don't.
* How do you have an absolute element be the same width as another element? For example, a drop down menu should be at least the width of its menu item, and an autocomplete box should at least be the width of its input box. Answer: JS.
* How do ensure a label is vertically centered on the same text baseline as an adjacent textarea? Answer: Textareas are considered blocks, not text, so you cannot without computing the baseline and using padding or negative margins, and that is imprecise.
These are just a few basic real-world layout challenges. They should not be challenges, they should be simple to implement, but CSS doesn't help; it gets in the way more than it aids. In some cases, inconsistent browser implementations result in wildly different rendering. The result is JavaScript to fill the holes, and sometimes just to say "screw this, I will do in code".