I think the difficulty in 3D polygon modelling is partially a tooling problem, and also a learning curve problem. The usual workflows I've seen/used that didn't involve sculpting (which requires being good at that too) all involve blocking out the model in several steps and applying semi-irreversible transformations, making it really hard to go back and fix a dumb mistake. Some editors like Blender are also notorious for having completely impenetreble interfaces, and even 3ds max and similar probably aren't too straightforward either. Ontop of all that, you're required to know a ton of technical details (skinning, normal smoothing, etc) that aren't explained properly for modellers anywhere I know of (building an engine/renderer is a fairly direct way to pick up that knowledge though).

3D sculpting and CAD modelling are entirely different beasts however (I've never actually seen CAD tools involved in a game asset workflow, but it could be done and would be fairly useful for inorganic assets). Sculpting is dead simple in principle, in the same sense that drawing is; meaning you don't need to worry about the tools in the same way. CAD is a different beast, but still works on some pretty simple principles (constraint-driven CSG).

I think the only real "hard" part right now is getting low-poly versions of assets that can be used inside of your engine. Making the high-poly assets might require more actual work (and 3D scanning could help there), but it's nonetheless fairly straightforward.

edit: "low-poly" here meaning "reasonable number of polygons", not visibly low-poly.