Add to that that written Cantonese is Mandarin, but most speakers use traditional characters, and it's a hot stew of pain.
(the grammar is a lot of fun though)
Adding to the difficulty is that some native speakers will enthusiastically correct your tone by over emphasiszing and embellishing their words (with trailing tones, ah's and lah's, etc). There are some people that I simply can't learn words from because I can never pick the tone. Luckily after a while you can pick those people fairly quickly!
Spoken Cantonese + Yale vs. spoken Japanese + Romaji.
Unlike Japanese, Cantonese has a very simple grammar and no registers (correction: honorifics), so it would be easier to learn to speak.
Spoken Cantonese + Written Chinese vs. spoken Japanese + Kanji + Kana.
Japanese script is less effort to learn, as it uses "only" 2000 Kanji. There's no such hard limit on the number of Hanzi you need. (But there's the complication of On-yomi and kun-yomi pronunciations of the same kanji.)
The other issue is the quality and quantity of learning material. Japanese wins by a country mile.
Students are taught around 2136 kanji in school, and there are a couple hundred more that aren't taught in school yet pretty much every adult knows and uses regularly. e.g., 嘘, the character for "a lie".
The fact kanji average over 2 readings each vs Chinese's average 1 reading each also makes Japanese quite difficult. When factoring in usage in names and extremely common kanji like 生, you can have well over a dozen possible readings which you can only learn through long exposure and living in the country. There are so many times where I think, "Oh, I've never seen that word but I can guess the pronunciation since I know the kanji", only to be shut down and told it's some weird exception.
Registers being different levels of formality (informal, formal, slang, etc)? If so, Cantonese has that in spades.
Your answer also ignores the tonal nature of Cantonese/Mandarin. Most guides can't event agree to how many tones there are. I get by with 7, but it can range from 6 to 10 depending on who you speak to (in contrast to 4 in Mandarin).
On a lesser scale, the hardest thing about learning German, as, again, an English speaker, but one who had poor grammar instruction and no grounding in Latin, the idea of cases for verbs and pronouns was... weird.
I've had a bunch of German, and just passing contact with Latin, but a lot of German grammar started making a lot more sense when I started thinking of those languages as similar in a sense, because of declension.
True that. Also, it's just a different use of tone - English uses tone to differentiate questions from statements, and to otherwise add meaning to words/sentences. There are other mechanisms for that in Chinese dialects.
"Really?": Convince me; "Really!": I'm indignant on your behalf; "Really hungry": exceptionally [hungry]; "Really good": average.
I thought of mentioning tones, but they're not as much of a problem for learners as they're made out to be. Cantonese has seven tones. However, there are no word pairs where the only difference is that one has a high level tone where the other has a high falling tone.
Maybe the added difficulty comes from needing to understand the grammar (particles) and the different forms of speech for different levels of politeness (after a few years, it now weirds me out when a white dude calls himself 'watashi').
Edit: Actually, if the methodology is "average time spent to learn" another possibility is that the average is slower because the barrier to learning the language is just low enough to attract a bunch of slow learners that wouldn't be interested in Chinese or Korean. (Reminds me of a study where decreasing page weight in KBs increased average page load time because users on the slowest connections were now able to load the page and skewed the average upward.)
If English had half the available consonants and vowels it had now, it'd be far more difficult to learn rather than easy. Japanese has the same problem.