Parsing EDIFACT in Python with syntax trees | Hacker News Reader

Parsing EDIFACT in Python with syntax trees

github.com

25 pointsalexprengere17 comments

Edison is a Python library for parsing EDIFACT messages. Instead of treating a message as a flat list of segments, it models messages as a syntax tree: segments can define groups of child segments, recursively.

You define the syntax tree once, then parse, validate, or pretty-print messages against that structure. It works by converting the syntax tree to a state machine of possible transitions between segments.

Once you know the tree, edi is trivial to parse. You can even do it with a loop and a state to store which level has which tag and what the current count is.

The real challenge was to discover what the tree was. Like wayfair order listings, cargo delivery messages, which are also similar to how Rhenus does them. Every vendor has their own format and standards.

For anyone who's encountered this, you likely have access to these but either they are being sent to random email alias at your company or you need to request these (clearinghouses typically have this, but you can also request from the vendor/partner). I'll never forget how _angry_ I was to find out after finishing a project that there was a spec with everything I needed and nobody had bothered to send it to me while I was implementing the project.

Once I started parsing example messages with the above method, the format became apparent. I even made a viewer for it which was useful because people we were integrating with were also making mistakes and I was the one correcting them thanks to the viewer.

The hardest part was europeans taking 15 day holidays every couple of months leaving work to someone else who had to start over.

Even with the "tree", it is not that trivial, as a single segment tag may be repeated in the "tree", so you cannot just jump to that segment and execute a callback. You need to be aware of the possible transitions from any given "node" in the tree. Edison solves this by computing a state machine graph whose nodes are all segments, and edges are all transitions, so parsing is just a walk in the graph.

This takes me back 20+ years to when I had to create/parse EDI messages for various retailers when I worked at an independent record distributor.

I wrote a C library to parse the messages with callbacks in the style of Expat. There's binding for Perl to make it a bit easier to use. EDIFACT, X12 and Tradacoms are supported.

Anyways, unlikely anyone might need it these days, but there are some Perl scripts which can interpret the EDIFACT "directories" (basically message schemas), so you don't have to build your own message syntax definitions. It should be possible to convert the XML output that it generates into the Python needed to define message syntax, element definitions, etc.

Best of luck with it - there's a whole world of crazy industry standards to explore.

Oh, and consulting is a possibility if anyone needs it ;-)

-- https://github.com/davidcoles/medici

In case other HNers have also not heard of this: https://en.wikipedia.org/wiki/EDIFACT

Many may think that EDIFACT is easy to parse, which may be the case. But it is NOT easy to interpret.

There are hundreds of "substandards" per industry. For example for clothes or medicine, most vendors have custom fields outside of the standard which MUST be interpreted/sent or everything falls apart

In my corner of the world, all official grammar descriptions finish with a sentence like "or another interpretation by bilateral agreement" (where bilateral means between any two vendors using the same "standard").

What is a "real" syntax tree? I wonder if the intent was "concrete" (the contrast to an abstract syntax tree), or if it's meant colloquially (as in "you really get a syntax tree" or "you get an actual syntax tree") rather than as a technical term.

I think OP means that it will parse EDIFACT with the tree structure it actually has in its specification, instead of a simple list of messages like many people do.

EDIFACT normally has (possibly repeating, possibly nesting) segment groups and a lot of code trying to handle EDIFACT skips over that.

Surprisingly this problem of treating EDI files as lists instead of the tree structure it actually is, is widespread even among clearinghouses. In retail energy, clearinghouses would often support their own custom format, which typically was either CSV or Excel. If you used that, you would routinely run into data integrity issues.

At this point, I'm pretty convinced that if you're running into EDI (or its sister formats like HL7) processing issues, its because you've got the wrong data structure in place or you aren't keeping your rules up to date. Once you've got the right architecture in place, you can easily manage 100s of provider/partner-specific rules with one person at 20% time. If you don't your whole company is going to be dealing with data integrity issues forever

Care to share what "right architecture" could look like? There's surprisingly little public engineering reading about EDI...

Mentioned elsewhere in the thread, but it's possible to parse the (mostly) machine readable specs for EDIFACT messages to create schemas for reading massages.

A library that I wrote decades ago has some Perl which can do that to produce XML definitions from the UN/EDIFACT published directories. I hope that it can be useful to others: https://github.com/davidcoles/medici

> Care to share what "right architecture" could look like?

I can speak about ANSI X12 because I've used it (North America), haven't used EDIFACT (non-North America):

Each document type, like a PO or Invoice has a defined structure (by the standards group) which is conceptually nested/hierarchical but physically stored as a combination of tags at the line level combined with delimited elements within a line.

Traditionally the raw data would be pass through a translator tool that converted it into it's hierarchal structure and a person could define a mapping from that structure to their internal systems data structure.

With the internet and XML and JSON, many people/companies weren't aware of the standards and didn't have an EDI translator, so they just winged it and extracted what they needed from the raw data.

There is still a large volume of EDI based data movement and it will probably live for a long time due to cost to change plus having standard document definitions across companies reduces the level of effort to connect many partners to many other partners (it basically reduces the transformations down to a conversation about which things each company does uniquely in the document as opposed to starting from scratch with a completely custom document for every company).

Yes it was meant colloquially, like "the syntax tree describes the actual underlying structure of the flat list of EDIFACT segments".

Really nice! I've been preparing to parse this for a possible customer in another language. Thanks for the insights.

Reply on news.ycombinator.com