With a small corpus these would run into similar difficulties but would likely perform much better than RNN/CNNs trained on the same corpus. Regardless of whether you use NN or something else your approch is the right one: start with model trained on a larger but similar corpus and then use the smaller one to modify the transition probabilities.
Leaving out the "Deep" stuff will shed popular clicks on your post, probably a lot, but would get the job done with less cumulative effort. Important if for some reason you cannot use third party models. The stochastic grammars I mentioned are a lot less fiddly to train than * to vec
This approach to generating philosophy is the exact opposite to the philosophical tradition - there's no effort to determine if any of this stuff is "true" in any sense, there's no traditional formal reasoning model behind it; just if it sounds good. I expect to see it weaponised on political billboards and youtube videos this year.
In particular, “the limits of my language are always your limits” is a horrendously mangled Fascist version of Sapir-Whorf.