Being better aligned with astronomical events makes it more correct. The purpose of the calendar is to map the year, the orbit of the Earth around the Sun. The drift in the Julian calendar was not intended. It is a flaw.
Even if people were tracking dates, adjusting for a couple of days every 200 years wouldn't be that hard: nobody would remember the good old times when we did the harvest on September 22nd in 1234, and now we do them on September 20th in 1434.
Finally, matching up with astronomical events would sometimes put sidereal year (and day) at the forefront: a day that's ~4 minutes shorter than the solar day making the tropical year Gregorian calendar is based on. Things get murky quite quickly once you start going down that path of what "correct" really is.
Note that in the Gregorian calendar, Spring equinox in 2021 and 2022 fell or falls on March 20th. It's only pretty good when averaged out over a 400-year cycle.
Basically, all of these calendar systems are attempts to "square the circle": find something resembling the least common multiple of non-integer values (solar day length and tropical year length), and then try to mix in a bunch of events observed in a different coordinate system (to overly simplify it, all the night stuff is "sidereal").
So we get back to what is really "useful"?
If you don't care about knowing how many days ago, or on what date in the proleptic calendar of your choice something happened in the past just from the date inscribed on it (eg. imagine a letter dated January 5th, 1605), you would certainly be fine with just dropping 10-13 days somewhere along the way. I can, however, understand when someone thinks it's easier to be off from astronomical events for a few weeks to avoid all that administrative trouble, for instance. However, the biggest practical problem today would be that everyone else has written those 10-13 days off, so it's probably easiest to switch too, especially in the global world we've got today.
But there is nothing intrinsically better in the Gregorian calendar that makes it win on all counts. It's just another agreed-upon approximation.
Earth's rotation is — arguably — better measured against the stars, so a sidereal day is more "correct", yet it would not map to our daily routine at all.
So I believe the question is not what is more "correct" (we've long established that we are dealing with approximations at best, Julian calendar included: it already had leap years), but what is more useful? And that's what has driven adoption of the Gregorian calendar in most of the world, but I am fine if somewhere it's more useful not to have to worry about did some days just disappear at some point in the past.
You also seem to be overstating the importance of exact dates when it comes to agriculture: my experience is that +-13 days does not make a practical difference, especially if it slowly accumulates (it's not like you would suddenly have to do the harvest 13 days later from one year to the next — you actuslly had to do it 10-13 days "early" once the calendar was switched, and it didn't make a difference even then).
If human civilization continued using the Julian calendar, we wouldn't have been any worse off: nothing points at it that we would have been. I am not saying that Gregorian calendar is "worse" at all (though those born on Feb 29 might beg to disagree when they go 8 years between birthdays 2096-2104 :)), just that where it's better does not matter much.
In short: How long it takes to cook an egg doesn't change because someone inserted a leap second.
Of course, when people try to do the obvious thing, and measure time durations using a wall clock, that's when the fun begins. In that world, "a day from now" is not a consistent number of seconds in the future, due to the aforementioned leap seconds, daylight savings time, and, potentially, time zone shifts if the person doing the measuring is traveling, or is (or, perhaps, was) in a particularly "interesting" jurisdiction.