Hacker News Reader

Comment by avinium | Hacker News Reader

Anyone familiar enough with this library to explain how they've achieved such impressive performance? Otherwise, may dig into it for a blog post when I have time.

View on HN

There are deterministic finite state machines underneath implemented in lean C++. These automata allow to implement operations with strings optimally or close to that. In the readme file there is a link to how to recompile linguistic resources, if you look inside the makefile that is used for resource compilation you will the steps.

What is lean C++?

I think he just means efficient C++ programming.

#define WIN32_LEAN_AND_MEAN

Maybe ?

Thanks - will dive in and start checking things out.

Do you use any kind of library or typical design pattern for your FSM implementation? When I've used them, I typically always go back to very C-like code (enum for the states, a state variable, and a big switch statement inside a loop).

We don't represent automata as a code, that leads to compiler errors etc., we represent them as graphs, just as data.

Inside the code we use: state --> int set of states --> sorted / unique'd array of int's input symbol --> int output symbol --> int transition function is abstracted out behind an interface and implemented differently based on whether automaton is read-only or changeable and based on the state etc.

I'm going to have to think about that a bit. Maybe I'm missing the point, but it sounds like an interpreter. Thanks for posting the code. This feels like something I need to understand.

One advantage is the core appears to be written in C/C++. NLTK is python and SpaCy is a lot of C/C++ with a significant amount of python.

I’ll have to dig in further, but I suspect there’s a lot of optimization for the task at hand here. SpaCy and NLTK tries to be much more general for NLP.

I was more so referring to spacy, where (IIRC) a lot of the lower-level tokenization & parsing is written in Cython, so (I assume) the simple fact that Bling is written in C++ wouldn't explain the huge performance gains.

There’s still a good amount of overhead in Cython portions of code unless there are no python objects or GIL involved, and then there’s still overhead communicating between C and Python worlds.

That’s part of why I stopped writing Cython and just use pybind11.

Reply on news.ycombinator.com