What makes EDI so hard? | Hacker News Reader

What makes EDI so hard?

stedi.com

81 pointsmooreds55 comments

Took me a while searching around in that article to figure out what EDI is:

> Since each business has multiple trading partners, and each of its trading partner operates on different business systems, “point to point” integration of these transactions (that is, mapping Walmart’s internal database format directly to the QuickBooks JSON API) would require the recipient to have detailed knowledge of many different transaction formats – for example, a company selling to three customers running on NetSuite, SAP, and QuickBooks, respectively, would need to understand NetSuite XML, SAP IDoc, and QuickBooks JSON. Maintaining familiarity with so many systems isn’t practical; to avoid this explosion of complexity, businesses instead use commonly-accepted intermediary formats, which are broadly known as Electronic Data Interchange – EDI.

There was I, having just read the headline on the main page and not yet the article thinking it stood for "equality and diversity initiative"...

that's DEI :)

I think we're accepting of all the variants.

Even DIE, Diversity, Inclusion, and Equity?

It is also one letter off from either EDA or EMI. Which was enough for me to in some superposition of "Electronic design automation can be complicated" and "Electromagnetic interference is confusing."

I cut my teeth on this kind of thing 25 years ago...it's the stuff that makes the world go around, but gets shuffled off to the back corners of the tech world. Way more interesting and nuanced than it gets credit. So many problems that were solved decades ago that have been forgotten as the world perpetually reinvents the wheel.

When I worked for a huge electronics distributor in the EU I got assigned the EDI hot potato by accident and kinda grew to like it. I got to talk to other companies's IT people that handled it on the their side instead of business facing sales or management people. Started with loving the XML formats and hating EDIFACT but that got reversed real fast. One of the projects I'm still most proud of from that time is consolidating all of our EDI systems into one, coherent (and beautifly designed I might add since no one interfered because no one cared ;) whole that used EDIFACT as an internal format to represent documents. That enabled developing integrations quickly by just writing a translator/parser between our format and whatever God awful format the other party used.

One thing I do hate with a passion still is the AS2 transmission later that some vendors used. My god, what a shitshow that is. No stable clients, discrepancies between implementations and all for what? Essentially transferring files. Ugh.

AS2 is interesting because it includes a cryptographically signed Message Disposition Notification (MDN) - whenever there's a question if a document was delivered, just give the MDN information to the partner. This has led to some interesting troubleshooting sessions, where it turns out that the message failed somewhere in the partner's system, and the only way for them to dig it out and identify the issue was to check logs for the MDN message ID.

I've also heard that MDNs hold up in a court of law; no idea if that's been tested. But it's a great way to confirm delivery of the payload without having to worry about parsing errors.

This was my favourite part working for an e-commerce store that had an archaic inventory/order management system. While not EDI, I wrestled with a multitude of large CSVs that needed to be parsed in order to have a complete picture of the inventory and orders.

I would love finding the system was able to export a slightly more helpful CSV somewhere in a hidden menu that would enable me to gradually remove my spaghetti code or reduce the number of CSV imports.

Maybe we wouldn’t reinvent the wheel so often if the old wheels

  PO1**1*2P*0.0001*PE*GE*EDI Core~

didn’t require memorizing a huge compression dictionary just to work with.

You don't have to memorize if you have a computer translate for you.

Why would you do anything else when the world is running at 9600 baud.

EDI Notepad was a great tool for this - unfortunately it looks like it was shut down a few years ago :( Old versions should work fine however, unless you're using a very recent standard. (We were using the 1996 X12 standard until 2015 or thereabouts)

Did you ever do any work with Sterling Software? I worked at the Commerce division supporting parts of the X.25 and ISDN network. Good times.

Definitely - I think anyone who has ever touched the EDI space is familiar with Sterling. My time hands-on in that world was pretty short, but it was good background to have for the projects that came later.

I develop an internal system for a company which sells diapers to big chain supermarkets in europe. It is connected to them through EDI. These are my 2 reasons why EDI is hard and fucked up in my opinion:

1) syntax - we are using XML which has weird element names and values and it's mixed camel case with other styles. It's not nice or possible to read without a manual/reference

2) brokers - you connect to others throught "EDI brokers" (dunno the right name in english). Basically you don't connect with other directly but it always goes through one or more brokers which are part of some EDI union or whatnot. So for example order from chain to a producer goes like this Supermarket -> Broker -> Producer. What this means is that different brokers pass you the orders in different way. So i guess it can vary a lot. Our broker is maybe the largest in the country and yet the only way how to programatically get messages from him is invoking a Java program, which downloads the message from his servers and that is possible to do exactly once. Then it's up to you to distribute it in your company as you wish. God forbid if you corrupt the message or something. This is very bad because you have to do a lot more work than just do a few REST requests to deal with anything. It prolongs basic integration by a lot.

Twenty years ago I was working with EDI without XML. The format was a series of lines, each with a starting tag followed by a delimited list of values.

Each transaction type had a grammar, so each starting tag could mean different things depending on its position within the grammatical structure of the transaction. We were new to EDI and had to roll out a bunch of partners fast, so we bought $60K software to help us. Turned out, that software had no understanding of the grammar; you had to use a cumbersome GUI to essentially write code from scratch to pull out what you wanted, with the only control structure available being GOTOs. Just pulling out a dozen important fields looked like a significant engineering effort.

Then our partners said they required us to display to the user everything they sent. I ended up ditching the $60K software and writing my own tool with a proper parser, so I could input the grammar of a new transaction type in an hour and sort everything out right. After that it was easy.

We didn't have to deal with brokers at least, we connected to everyone directly. I think the company became one of the brokers after I left.

This both brings back my past and a future project I have waiting for me.

The discipline of dealing with other people's systems is a great learning point.

The last person I integrated with was totally bemused by my huge list of questions. They always seem to assume that the way their system exports orders is the norm!

A:"When do you export orders, as soon as they are placed or on some schedule?"

B:"As soon as they are placed".

Some time later in the conversation...

B:"You needn't run it that early, we don't export orders until after 7.30am".

You really can't trust anything in EDI until you have seen the orders!

I just remembered a proprietry ERP system I worked with that accepted EDI orders placed in a certain directory on their Solaris server. Every 30 seconds it would empty that folder and begin to process the orders.

The company must have implemented a design internally to spawn a new process to deal with each new order. The first I noticed was that the log file it spat out was not in order, it had messages about different parts of different orders all intermingled... then 1 day I put 700 orders in and it launched 700 processes and the server ran out of swap space and crashed.... Their support team felt I was unreasonable! I had to rate limit afterwards with my system drip feeding the orders! It is funny to think that it is still in use I understand, and busier than ever!

The biggest problem with EDI is semantics. When humans talk to each other they have ways of disambiguating the words and jargon used. Mistakes still happen.

Semantics are embedded at different layers of different systems and machines don't have the common sense to notice different semantic schemas.

From the medical side; we had a pathology system, an ADT (admission, discharge, transfer) system, a clinical system, and a billing system. The semantics of what constitutes a demographic update of a patient record vs. an update of an encounter (visit) is bad enough when the same data may be denormalized in messages in both streams. Even with an established interchange format (HL7) each system encodes patients, their visits, their billing codes, and their results in slightly different ways. Sure; the pathology system and clinical system both understand HL7 and ICD9, but do they agree on what it means when a modifier gets added to one of the codes during an update? Or does an interface engine MITMing the connection have to generate a delete/insert in one direction to synchronize the semantic meaning in the downstream systems? Or what exactly does "gross description" mean to a pathologist vs. a clinician (not to mention a biller or admission clerk or me), not to mention the systems that are supposed to store this information in a meaningful way (it turns out the best quantifiable semantic representation of the latter is plain ASCII, or maybe a rich text formatted document).

I spent a decade working with ANSI X12. Writing software to generate it, streaming parsers for mutli-gigabyte batches, tools to display it, consume it, validate it, etc. Really interesting world to deal with. Every trading partner had their own interpretation of certain things. In my experience (healthcare) things slowly improved over time. Supposedly, FHIR will will take over in that space, but I have my doubts.

FHIR will take over, but I wouldn't call that a good thing. the development of FHIR is extremely haphazard, most of it will turn into a generic KV store and miss a lot of the great systems thinking in EDI. It's as if they lowered the barrier to entry at the expense of robust document exchange.

Hopefully Stedi can make adopting EDI easier and increase its popularily over the next 50 years and there isn't such a demand to create alternative systems.

Healthcare wasn't using HL7? Another rather old, heavy data exchange format.

It is used for exchange of clinical data like admission notifications and health records. X12 is used for administrative data like enrollment and claim submission. I have worked with both.

HL7 is a standards development organization, not a data exchange format. They publish several major standards including V2 Messaging, CDA, and FHIR.

FHIR can now be used instead of X12 for prior authorizations but in the US it can't be used for claims yet. Claims will continue to rely on X12 for many years.

I did medical billing EDI adjacent stuff with a VMS system (that ran decades past point of obsolescence).

One thing - in the past the EDI integrations just had to be simple, the feature set was so small - there wasn't room to go totally crazy.

Once they mandated full ehealth records and exploded the complexity of the medical IT side, clinician productivity (face to face with patient) went way down. Integration pain went way up. The old tab separated value row turned into GIGANTIC XML monstrosities for the smallest things - now thousands of these blobs in a document. If you put even 10,000 XML blobs in an exchange vs tab separated values - very quickly appreciated the old way of doing things. One annoying thing, even the person on other end you are dealing with lost all ability to troubleshoot issues.

For some reason in some areas - no EDI in the traditional sense is being used (ie, govt contracts with idiots who build latest buzzword filled solutions - total annoyance to interact with).

Yeah, worked in medical billing for a while. It's insane. One particular example I remember was that there was one code for "blood pressure taken while patient lying down" and a completely different code for "blood pressure taken while patient sitting up" (rather than, say, one code for "blood pressure" and another for "patient position").

I'm going to have to implement an EDI medical billing system soon (moving from XLS/FTP based billing, say what you will about it but relatively easy to understand and automate) and after looking over the spec, I am not looking forward to it at all. Ended up purchasing a product called Cozyroc which integrates into SSIS(SQL Server Integration Services) which can do SQL to EDI translation with a map provided, should make things easier but still not looking forward to it.

This is all actually better nowadays. There was an in between place where CSV/TSV/FTP style systems were being taken over by vendor proprietary XML blobs for ehealth stuff when they were trying to (poorly) push ehealth. So the variety was crazy (and I'm still not sold on the huge XML style documents though I know it's "semantic"). I'm not as close to it anymore, but my big sense is that things have settled down A LOT and become more standardized.

In the bigger EDI space UBL was supposed to be the XML standard that was going to crush the "old" standards - but I haven't seen it take over yet - but not close enough to space, someone else probably knows better if UBL or similar is the clear dominant player these days.

I don't think EDI itself is too complex or anything--after working with several REST/SOAP servers, EDI is just another standard you have to account for. Parsing 850s or generating 810s is just the name of the game.

What makes it super complicated and frankly pisses me off are the "exclusive" deals retailers have. Want to work with Target? Too bad, you have to work with CommerceHub. Macy's? Nope, gotta use SPS Commerce. This is why I like Shopify or even Amazon Vendor Central, which uses EDI. Wayfair was fortunately open to the idea of anyone linking to their EDI servers so long as there's a proper onboarding process.

My company, which is an ERP for wholesalers, doesn't have a way to directly work with the retailer. It always has to go through one of these middleware companies.

EDI is not hard!!!! Period! Like some file formats, it’s unnecessarily complicated. The biggest problem in this industry is that there’s a monopoly and many companies are being FORCED to use these arcane and costly service providers.

I build and integrate EDI solutions for small/mid and retail juggernauts.

We're a small business looking into EDI with some of our suppliers, but are finding the pricing quite painful for our level of business when talking to some service providers. Which service providers do you see as arcane and costly?

SPS, Truecommerce etc. If you need advise or guidance drop me a message.

Funny, those are the 2 companies we were speaking to. Would appreciate some advice/guidance, but I'm at a loss as to how I drop you a message here — doesn't seem DM'ing is possible on HN?

Connect with me via devsaikan ~the google mail~

What makes you think that it is unnecessarily complicated?

Have you done programming in other languages or worked with other formats other than excel and CSV?

The x12 argument was fine 5-10 years ago. Certainly not now. There are much better structures available.

It's not hard, it's complex. I spent a few years going through 834 and later at a different company, 811. For the first I embedded a javascript interpreter to do translations. For the second I put most of the logic in the code.

I work in container shipping in Europe. Everything is coordinated by e-mailing EDI files (baplie, coprar, iftdgn etc.) Back and forth. Tens of millions containers per year.

Even though it should be a standard, you can't just e-mail a file to someone and except it to work. There's always integration work needed on both sides. Even then, errors frequently occur where you have to manually adjust something. I work on the operational side, but have become quite the edifact expert just to be able to do my work.

Overall solid introduction to the space. EDI was one of the things that boggled my mind early in my career.

> What we’re seeing here is that while a standard can be opinionated about the structure of a document and the naming of fields, it cannot be opinionated about the contents of a business transaction – the contents of a business transaction are dictated by the idiosyncrasies of the business itself.

Maybe it's the engineer in me speaking, but this seems like a pretty unfortunate stance to take. Sort of similar to saying, "we have the concept of HTTP status codes, but we cannot possibly opine on the codes themselves."

A bit hyperbolic for sure, but not having opinions on critical multi-party information has led to things like the reluctance of financial services to support non-ASCII values for customer names.

It's basically Conway's law in operation, just that it crosses business boundaries in this case.

I once had to parse EDI docs sent from SAP using C#. One of the worst things Ive ever had to do in my 15 years as a developer.

Trying to support EDI reminds me of the saying: "The great thing about standards is that everyone has their own."

I haven’t done any work on EDI for many years so it might be a bit out of date, but in case it’s of any use to anyone I wrote an EDI parsing library and processor for many of the EDIFACT message specs:

https://github.com/davidcoles/medici

I've had to do some EDI in the Automotive world and Financial world (ACH and Wires)

Maybe I've been lucky in my travels, but none of them were terribly difficult, so long as you had a spec doc in front of you. (As a C# Dev, I'd usually use some form of FileHelpers alongside of MultiRecordEngine, if you're having to deal with things like 'correction' records, treat the resulting records as an eventstream and use those to build the final dataset.)

What gets -nasty- is when companies violate a spec in a subtle way. In one case, the 'response' file for ACH transfers from one bank 'reused' the leftmost digit in a certain field for their own code. This however made it extraordinarily difficult to reconcile to the original check due to the relative shortness of this identifier.

Nothing scares me as hard as doing computer stuff in transport/logistics/warehouse/trade/invoices 'n' stuff world

complexity of those industries is giant

Yeah a lot of people think those industries are "ripe for disruption" and then they realize no you can't represent it in JSON and you can't manage it on a phone-sized UI.

I've written several APIs and middlewares for freight, inventory, WMS and OMS platforms. You absolutely can represent data with json over "archaic" xml, and edi.

I set up our company with EDI through sps commerce - we didn't want to build a tight integration so we exchange flat files on SFTP.

flat files over sftp is what ive done as well did you do a JSON to XML converter on your API calls b4 sending them to SPS Commerce ?

X12 is also used for freight rail messaging in the US. Anytime one railroad interchanges a train with another (in addition to many other types of events), messages are sent. I deal with it on most days.

i love how most of the comments on here are talking about EDI back when they first started and here I am consolidating 6 different systems into 1 EDI VAN and its considered new capabilities -_-

It's format.. I once had to convert several EDI specs to XML.. It was horrid..

Ah, this triggered an old memory! I used to do integration implementations in Microsoft BizTalk Server, which was all XML based: all of the guts were designed to handle XML documents, so it had out-of-the-box components to translate non-XML ("flat file") messages to/from XML as they were received or sent. These components were universal and designed to operate off of annotations baked into the schemas that represented the XML versions of the messages - things like field sizes, expected locations of line breaks, records that could repeat, etc. The BizTalk developer tools included a schema builder tool ("BizTalk Editor") that helped you autogenerate, construct and mark up schemas with the declarative flat-file translation logic.

I only had to interact with EDI once or twice, but I remember when I discovered that BizTalk shipped with a huge set of premade schemas for most of the X12 and EDIFACT message types, plus extra pluggable components that were "enhanced" versions of the regular translator components that handled the more complex aspects that most "plain" flat files didn't have, mostly stuff around batching/splitting/validating I think. Many of the schemas were gigantic and unbelievably complicated.

Reply on news.ycombinator.com