The numbers don't tell the whole story - Japan also has large non-acceptance-related barriers to immigration, including the massive language barrier, low wages and a poor working environment for skilled workers compared to western companies, and a weak welfare system. Immigration system aside, Japan just isn't an attractive country to move to for economic considerations.
edit: as for anecdotes, I know people who have moved to the EU and US through marriage, and it was a multi-year and/or multi-thousand dollar ordeal. I know people who moved to the US though work and it was the same thing (working in a EU office for a year to satisfy some random internal-transfer criteria). Meanwhile getting a visa in Japan is a $30 fee for ID card issuance and everyone I know here has gotten visas approved in days (marriage visa, not even an interview required) to months (work visas).