I'm sure other people will be able to give better tips for doing the open sourcing and scaling up, so I'm not gonna touch that. I'll start with a different approach. I think the first important part is to forget the "scale an open source project" and just focus on "start".
Build a software that solves a problem you have. Preferably a single problem. Then make it good. Give it out to people you know in the same field or post it to relevant forums online.
After that, "open source" it. (You can put the code publicly on Github from the beginning but here I mean start promoting it to get users and collaborators.)
The motivation behind this sentiment is that all the extra stuff around this (maintaining the community of collaborators, replying to bug tickets, checking out pull requests) takes a lot of time that you should use in building the library first.