Contrary to popular belief, going closer to the Sun is as much difficult as going further away from it. Orbits require energy to change them regardless which direction you want to do it. You can go to tables and see going to Mercury costs almost the same as going to Jupiter: https://en.wikipedia.org/wiki/Delta-v_budget
There are handy maps of Sol system if you are interested: https://forum.kerbalspaceprogram.com/index.php?/topic/131682...
You can of course minimize the energy even further using gravity assists, but this would require multi-year journey.
There is another trick when you want to go very close to the Sun, which is to go further away first, and then use much less energy to actually stop your orbital speed and fall however deep into Sun well you want. Unfortunately, Mercury is just far enough from the Sun to make it unprofitable and it also takes an awful amount of time to execute.
That of course is true, if you are stationary. Then you just start falling like apple from apple tree. But almost no objects in space are stationary, typically everything orbits something and frequently multiple things at the same time. Think about a rocket that is on an orbit around Earth. It is not free to fall to Earth, it must expend delta V to slow down enough so that its perigee is within Earth atmosphere (and typically well within atmosphere or it will just bounce off it).
At the same time, the rocket orbits the Sun, so even if it leaves Earth it still is not free to "fall" into Mercury. It must expend delta V to slow down so that its perihelion lowers at least to Mercury orbit.
So the graph on xkcd is correct, but with regards to bowling balls you might want to shoot up or let fall down from a position that is stationary in some celestial body's frame of reference.