It's both effects really. Celestial bodies' orbits do undergo decay and also their rotation undergoes decay. The question is which happened to which degree, and I think the rotational slow down is the dominant effect in Earth's case.

Orbits decay due to various forms of drag. The long-term rate of decay of orbits in the Solar system is relatively well established.

Rotation also is slowed down due to drag, but in our case there's another major force: the tidal influence from the Moon. Earth's Moon is a relatively large companion (at 1.23%[1] by mass). Both bodies influence each other tidally, and that influence saps away rotational energy and also Moon's orbital energy; the Moon already got tidally locked to Earth. Aside of that there's a (smaller) tidal influence from the Sun, which again saps Earth's rotational energy.

--

[1] https://www.wolframalpha.com/input/?i=mass+of+the+moon+%2F+m...