bd82

Or you could take the middle ground and use a library meant to make it easier to create hand built parsers. as opposed to a library that generates a parser for you.

* Repo: https://github.com/SAP/chevrotain

* Online Playground: http://sap.github.io/chevrotain/playground/

This handles the repetitive parts such as lookahead functions creation or Parse Tree building, while also providing advanced features such as error recovery or syntactic content assist (auto complete).

* ECMAScript family of languages supported (JavaScript/TypeScript/CoffeeScript).

> 2) ANTLR was not good for speculative parsing or probabilistic parsing which would be the basis of today's generation of tools such as "Intellisense" not giving up on parsing when there's an unclosed brace or missing variable declaration.

ANTLR does have some generic error recovery capabilities such as single token insertion/deletion or re-sync recovery.

There is a whole chapter describing the algorithms in the ANTLR book: http://media.pragprog.com/titles/tpantlr/errors.pdf (unfortunately this link does not contain the full content of the chapter).

These kinds of generic heuristics is not something that would be simple to implement in most hand build parsers as they require full knowledge of the entire grammar and that information is often "hard coded" as source code not as a separate data structure.

Thanks for the article.

What about Error Recovery capabilities? I did not find any reference in the docs regarding that.

Is it something you would want to add to FastParse? Do you think it is even possible without a separate scanning phase?

Is it possible to detect and warn about the left recursion? Assuming a data structure representing the grammar is built at runtime to be interpreted.

Won't it be possible to detect the left recursion instead of going into an infinite loop?

There are a few small things that still require conversion. But that is because Typescript also includes capabilities from ES7 (ES2016).

For example - currently in Typescript but only planned for ES7: https://github.com/jeffmo/es-class-fields-and-static-propert...

or Support ES7: exponentiation operator https://github.com/Microsoft/TypeScript/issues/4812

Typescript seems to be more like ECMAScript.next + types rather than ES6 + types.

Thanks for the compliment.

I Still need to produce more examples (contributions are welcome :) ) There is one bigger (but incomplete example) of ECMAScript 5.1. https://github.com/SAP/chevrotain/blob/master/examples/types... It is written in Typescript.

About LaTex: I'm not familiar with LaTex, However if I understand correctly it is based on Tex which is not a context free language.

Meaning it can't be parsed with standard parsing tools.

See details here: http://tex.stackexchange.com/questions/4201/is-there-a-bnf-g...

Answer to BTW: But what about debugging and stepping through the grammar execution? I assume that in the case you are describing EVAL is performed on text generated "in memory". If you want to debug your code snippets (grammar actions) or just the grammar flow itself you still can't escape the generated code, even if you can escape generating an actual file.

Answer to BTW2: Could you describe such a use case? what would be the motivation for it?

The use case of only handling valid input is a very common one, Particularly during compiler construction.

But what about the use case of building an Editor (IDE) ? In this context it would highly beneficial to be able to handle invalid inputs as the user would still expect outline/navigation/formatting/.../ to work even when there are syntax errors.

Error Recovery algorithms are heuristics, they can never solve all the possible kinds of errors. Automatic error recovery is also not mutually exclusive with expanding the grammar to support some "tolerable" errors, Both can be used to improve the "robustness" of the grammar when fault tolerance is needed.

Also note that expanding the grammar will work best for simple common errors (missing semicolon), but what about when you need to re-sync an unknown number of tokens and unwind the parser call stack?

Also note these "heuristics" are not "mine", they are based on heuristics used in Antlr 3. Which is itself based on Academic studies in this topic: Specifically:

Josef Grosch. Efficient and comfortable error recovery in recursive descent parsers. Structured Programming, 11(3):129–140, 1990.

Rodney W. Topor. A note on error recovery in recursive descent parsers. SIGPLAN Not., 17(2):37–40, 1982.

Niklaus Wirth. Algorithms + Data Structures = Programs. Prentice Hall PTR, Upper Saddle River, NJ, USA, 1978.

There is nothing special about the parsing algorithm. It is currently a naive LL(1) (single token lookahead) Parser. I may try to upgrade it to LL(*) in the future.

However that does not mean Chevrotain it limited to LL(1) grammars. You can explicitly implement the lookahead function in any point the grammar.

So if you are trying to parse a well designed grammar like Python who explicitly declare that being LL(1) is a blessing not a curse. https://www.python.org/dev/peps/pep-3099/#core-language You won't have any issues.

If you are trying to parse a language that is nearly always LL(1) You also won't have many issues as you can define your own custom lookahead. Example: https://github.com/SAP/chevrotain/blob/master/examples/custo...

If you are trying to parse a language that is nearly always NOT LL(1) You may want to use a different tool to parse it(at this point in time).

Thanks for the feedback.

The Lexer and Parser both expose a collection of Errors. http://sap.github.io/chevrotain/documentation/0_5_19/classes...

http://sap.github.io/chevrotain/documentation/0_5_19/interfa...

Both contain information regarding the position of the error. For a lexer Error it is an offset. For a Parsing Error the erroneous Token is referenced, It contains line/column/offset and more...

If you want to build an Editor using codeMirror you will have to extract that error position information and highlight it using codeMirror's APIs.

example: https://github.com/SAP/chevrotain/blob/gh-pages/playground/e...

Chevrotain is not aware of codeMirror/ACE or any other Editor frameworks. It is just a ±3,000 LOC library meant for building parsers.

If you are indeed interested in building an Online Editor for some language, you may want to have a look here: https://github.com/SAP/demos_json_editor_services_chevrotain

It is a work in progress on using Chevrotain to build Editor services (outline/formatting/Ast Building/...).

Thanks for the feedback. :)

There is no way to generate a "parser file". Because no source is ever generated (not even "in memory").

If you want to bundle with the rest of your application. Your grammar is just another javascript file. Which will be packaged normally as you would bundle other web applications. (webpack/uglify/...)

* Note that due to Implementation details (using Function.toString) you may have to play around with minification options regarding name mangling. Example: https://github.com/SAP/chevrotain/blob/master/gruntfile.js#L...

I will document this clearly in an FAQ...

If you are referring to the Chevrotain runtime itself. It is available As a single bundled file (minified) on bower/npm for ease of consumption in a web application.

Another major difference is that Chevrotain does not generate any code, it is a Parsing DSL not a Parser Generator.

It is actually implemented with something very much like Macros, The Grammar’s implementation itself is parsed at runtime in-order to “understand” the grammar’s structure, once the grammar’s structure is known at runtime, capabilities such as Lookahead calculation / Error Recovery / Dynamically generated syntax diagrams (self documenting) become possible.

This means you just write plain Javascript, not some new language that is later compiled to a runnable JS parser. At first glance this may seem to be of little consequences to the user in terms of functionality. However it does provide many benefits to the user’s development workflow:

1. Easy debugging. You don’t have to deal with both a grammar and a thousands of lines of generated code, trying to understand how the generated code maps to the grammar and vice versa.

2. Fast feedback loops. No compile step…

3. No need for a build/packaging step in your central build or worse yet deal with committing generated code to your source control system.

4. No special none Javascript syntax. Can use your favorite Javascript editor to build your grammar, instead of relying on custom tools or worse yet a simple text editor. No special syntax to insert code snippets.

Hello.

I am the author behind Chevrotain.

I would say the two major functional advantages (as already mentioned in this thread) are Error Recovery and Performance.

Error Recovery online playable demo can be found here: http://sap.github.io/chevrotain/playground/?example=tutorial...

Performance in the jsperf benchmark is affected by the javascript engine used. On latest chrome (V8) and thus node.js which is the most common runtime, Chevrotain is ±3 times faster in the provided jsperf benchmark. http://jsperf.com/json-parsers-comparison/21

Shahar.