This is why non-compulsory voting hasn't ever made sense to me; it effectively biases the system towards those who care the most and enables those same people to influence the system further by making it difficult enough for others that they won't bother.
When it's compulsory to vote, things like weekday voting stop happening because laws will be introduced making it illegal to prevent your employees from taking time off work to go vote, so naturally both employers and employees alike want weekend voting.
Additionally, it incentivises the government to make voting accessible, because it is unreasonable for them to enforce voting via fines etc if it's not really possible for people to vote.
When I moved to the US from Australia I heard stories from my new colleagues about people in poor neighbourhoods lining up literally all day just to maybe get to vote before the booth closed, while those in richer neighbourhoods had practically no line at their voting booths. This is not how democracy should work. Your "freedom to choose not to vote" only serves to hurt any other freedoms you may have had.