You are on to something there. For the syllables I'm actually using a rule-based model from Python's "pyhphen" library: https://pypi.org/project/PyHyphen/
I am not totally happy with the results but have not had a chance to train my own
1,086 karma · joined November 9, 2009
I am not totally happy with the results but have not had a chance to train my own
In general, I don't fix a random seed so in general you can get different definitions (sometimes: I cache data)
The early results were that it works, but noisier datasets are tough. The urban dictionary corpus also has a ton of racist definitions
I'm using a blacklist to reject "real" words but it's surprisingly hard to build for rare words. I'm up to ~600K items after parsing Wikipedia tokens and it still doesn't capture everything.