I dunno, my M1 with 8GB is not even really usong swap afer a month of uptime and heavy loads memory-wise (VM's mostly).

Based on other comments: try not using chrome. I don't and I have 0 issues iwht this.