Slowly! Invite your friends, family, people who read your tumblr. One of the most important points is for you personally to post every single day, live on the site, always be there. Always posting, always replying etc especially at the start.
Initially people will just drop in randomly if there's no posts, no-one there they won't come back so you keep it going all by yourself at the start if you have too! Once people are coming by themselves you'll reach a tipping point where they'll start bringing their own friends over without you even asking.
Another point which I'm sure you know, but don't worry about the 'quality' of your content, people mostly don't care unless it's outright spam or junk. Never try and micromanage your site/content, I've lost count of the number of sites I've seen killed because of that, just relax and let your users basically be at home and post what they like.
And last but not least make sure you have an effective blocking option. People may love anime but they won't always love each other, be prepared for that, don't take sides and point them towards your block!
Edited to add, also regular backups! Nothing kills a site faster than it crashing and having to start over because there was no backup in place.