"doing all of the web" will cause pretty much any approach to AI/machine learning/NLP to break down. i'm a big believer in it being the responsibility of the engineer employing these techniques to take stock of the problem at hand and find out what constraints you can take advantage of to achieve better performance/accuracy/prettiness of code. there's not really a silver bullet that you can just release on the internet with the task of bringing back incredibly useful information without "knowing what you're trying to analyze before you analyze it."