I started with a code sample of ten or so lines of Visual Basic .NET code that could allegedly turn on my Insteon light switch. Then I made it control my other Insteon devices. Then I started adding email notifications. Then I figured out a command line interface made it easy for me to add new commands/features quickly and that I could also then pipe things like input from an app or voice to that interface.

It kind of grew organically. I don't think when I started if I planned to build what I've put together so far, I would have ever started, because it sounds daunting, especially given my level of experience when I started.

Do one thing. Then do something else. :)

As a sidebar, if you want the general knowledge type queries, Wolfram Alpha's API is incredible for this. They have a "spoken" API which presents the result of your query as a single text-string, ready to be output to a text-to-speech client. I think adding the ability to use Wolfram Alpha was all of a couple lines of code tops to my existing project. And of course, it's free for way more queries a month than you'd ever run on a personal use project.