This is the right call. If you ever actually move the needle on anything important your fun 20% project will assuredly be taken over by middle management and you will get pushed out. If you have free time at a bank either slack off or work on low-effort projects that look good on your performance reviews.
Not all workplaces are toxic. You generally have to take charge of your own career. But often the politics align such that it'd be strange to kick someone off their 20% project if it's successful.
I imagine no one at Google used their 20% time project to bunk off, simply because it was 20% of their working time, and not 20% of their contracted hours. If you're supposed to do 40 hours a week but you actually spend 80 hours working on work and 20 hours on your 20% project then you aren't bunking off.
Working smart and not hard and reaping the rewards is not "bad".
The Protestant work ethic and living to work aren't healthy for everyone and I don't really see why we need to assign value judgements to people who do or don't subscribe to it.
Which is good if you're tired, overworked, didn't sleep well or just want to spend some time with family. I usually use it to read books not closely related to my daily work, but related to what my coworkers do. Like about subscription models, devops, scrum master roles.
What have you worked on? How did you find your clients or customers or projects?
Is it a consultancy? Or did you develop a marketable product?
What technologies, programming languages, databases, and tech stack did you use?
Keep the information anonymous, of course, if you want to maintain your privacy.