Hah. Okay, here's an anecdote.
First time I had to look under the hood (and discover it's not exactly easy to get through the wiring) is when I set up my first homelab cluster. I wanted to run it over an IPv6-only network I had, and it while the control plane worked data plane did not - turned out that some places were AF_INET only. That was quite a while ago, and it's all totally okay. But it was documented exactly nowhere except for some issue tracker, and it wasn't easy to get through CNI codebase to find enough information to find that issue.
So, it wasn't doing something wrong, but I had to look into it.
So I ran that cluster over IPv4. But it was a homelab, not some production-grade environment - so, commodity hardware, flaky networks and chaos monkeys. And I had it failing in various ways quite frequently. Problems of all kind, mostly network-related - some easy and well-documented like nf_conntrack capacity issues, some were just weird (spontaneous intermittent unidirectional connectivity losses that puzzled me, and the underlying network was totally okay, it was something in the CNI - I had to migrate off Weave to, IIRC, Calico, as Weave was nicer to configure but too buggy and harder to debug).
I also ran a production GKE cluster, and - indeed - it failed much less often, but I still had to scrape and rebuild it once, because something had happened to it, and by then I knew that I'm simply not in capacity of proper diagnostics (but creating new nodes and discarding old ones, so all failing payloads migrate is easy - except, of course, that it solves nothing in the long term).
In the end, for the homelab, I realized I don't need the flexibility of K8s. I briefly experimented with Nomad, and realized I don't even need Nomad for what I'm running. All I essentially wanted was failover, some replicated storage, and a private network underneath. And there are plenty of already robust age-proved tools for doing all of those. E.g. I needed a load balancer, but not specifically K8s load balancer. I know nginx and traefik a little bit, including some knowledge of of their codebase (I debugged both, successfully) so I just picked one. Same for the networking and storage, I just built what I needed from the well-known pieces with a bit of Nix glue, much more simpler and direct/hardcoded rather than managed by CNI stack. Most importantly, I didn't need fancy dynamic scaling capacities at all, as all my loads were and still are fairly static.
As for that GKE cluster - it worked, until the company failed (for entirely non-technical reasons with upper management).
If you know what I did wrong, in general - please tell. I'm not entirely dismissing K8s (it's nice to have its capabilities), but currently have a strong opinion that it's way too complex to be used when you don't have a team or person who can actually understand it. Just like you wouldn't run GNU/Linux or FreeBSD on server back in '00s without having an on-call sysadmin who knows how to deal with it (I was one, freezing my ass in the server room, debugging NIC driver crashing kernel at 2am - fun times (not really)!, and maybe I grew dumber but a lot of Linux source code is somewhat comprehensible to me, while K8s is much harder to get through.)
This is all just my personal experience and my personal attitude to how I do things (I just hate not knowing how things in my care/responsibility work, because I feel helpless). As always, YMMV.