If you use Sphinx for speech recognition and use pyttsx for text to speech (Windows Speech API, OSX NSSS, or ESpeak on Linux) it all works offline - see the "Jarvis's Brain" section.