Open Sourcing super expensive proprietary software | Hacker News Reader

Open Sourcing super expensive proprietary software

usethetree.com

8 pointsmqsiuser13 comments

So I have been open sourcing the heart of a super expensive IBM software product, which IBM currently/today sells regularly for 100 tsd (Dollar or Euro, doesn't matter much) per CPU core (discounts of 40% to 95% usually apply). This product is IBM WebSphere Message Broker or now IBM Integration Bus (IIB) and was thought to be IBM's "SOA" offering. Let's say mildly that SOA failed and unfortunately the remaining parts are not "a lot" (also not worth that much). I was in fact able to understand them and it's been possible for me to open source the very most important parts in a POC. So the very most important part is the message processing and transformation language "ESQL". IBM implemented it in c and I implemented it in Java. I called the project "use-the-tree" and open sourced it on github. A deployed version is online at www.use-the-tree.com.

Actually it turned out that I added a couple of improvements which makes my implementation superior to the one IBM has.

There are open source ESB's and Broker's, but I did not find (and I think it is unlikely that there are) comparable proper implementations.

I managed to work in this SOA trouble and Integration world for about 8 years and I am the contributor of important components to IIB which I open sourced as "GenericESQLUtilities". My contributions are confirmed (by IBM Hursley) and by the community (on www.mqseries.net) and are actually used. Basically I proved that xslt is a performance disaster and on the other hand I showed how to properly do message transformation and processing. I also just implemented stuff that IBM didn't know how to (implement), since they bought the product from another company (NEON) and certainly had some brain drain during that process.

Main areas of future work are (also documented in the git hub wiki):

  - Add support for JSON
  - Add all quirks that XML has
  - Investigate if JavaScript can be used (instead of Java) (and implement it)
  - Sorting (super cleverly with Tree Map ofc)
  - On-Demand Parser (would be awesome, probably to hard for 3 months)
  - Investigate on open source solutions for parsing flat files standards like EDI/EDIFACT or HL7/ORM formats and it's usability together with use-the-tree

The project is very new (a couple of weeks old), it's a POC and there is nothing in production. Though it is confirmed to hit the point.

So... I have already successfully contributed to open source (in this area) and there is constant change in the IT world (I see a clear decline of SOA and a push towards open source), so that I think that this project can have a significant success in the future. Again ... "use-the-tree" is the only thing which is left of SOA if you remove all the bad parts or the ones that have just failed... it is "SOA - the good parts" :). It provides strong guidance on how to do message transformation (to end up with the most efficient transformation code) as opposed to IBM's approach, which is "we provide you with every possibility and you better choose wisely".

Xslt would look like it, if xslt would be done properly.

SOA as a buzz-word has fallen out of favor, but SOA as a means of segmenting a large problem to later solve it in "chunks" has been an accepted practice since the dawn of engineering. I'll agree that Microsoft, IBM and Oracle (Tibco, etc) would prefer that you were passing SOAP/XML over enterprise busses, but our layer's of RESTful services are still a "Service Oriented Architecture" ... the loss of statefullness was a big win.

> I'll agree that Microsoft, IBM and Oracle (Tibco, etc) would prefer that you were passing SOAP/XML over enterprise busses

They better say bye to this idea... it's hard to see why curly braces are good and angle brackets are evil... they get used differently.

> Enterprise (Service) Bus

Buzz word, snake oil

There is no problem, there never was a problem: Build on existing (de-facto) standards and just don't reinvent the world (with WS-*). And do not use the beginning of the payload (like SOAP does). The solution was there already and the solution is HTTP and the 2048 characters of the URL (to encode a function call) and yes a machine readable data format, but NOT XML and yes, JSON is likely/probably good enought.

You usually pay the bill for what you are doing

Open Source is free (as in freedom) but also free of charge :)

> yes a machine readable data format, but NOT XML and yes, JSON is likely/probably good enough

I thought so too, until I realized json is terrible for binary. on it's own it can't reproduce the full capabilities of REST ie: http multi-part uploads.

You need a performant streamable binary format too, and my vote for that is protobuf.

Thanks for doing this. I have not used IBM's products, but I understand the motivation behind doing something like this.

It will be a wonderful exercise in digging up super expensive programs like this and creating open source versions of it. They will rescue some hapless programmer and/or provide consulting opportunities for another.

Do you have some programs that can benefit by having an open source version? It will be a neat exercise to learn new languages/platforms.

> Do you have some programs that can benefit by having an open source version?

It's always and ever beneficial to (have the ability) to look into the source code

And the price ;)

> It will be a wonderful exercise in digging up super expensive programs like this and creating open source versions of it.

> They will rescue some hapless programmer and/or provide consulting opportunities for another.

It's because SOA failed that this software isn't worth what it was (and still currently is) priced at.

It's been unclear, an endeavour, we are going along and we are finding things out.

It turned out to the misfortune of big companies, which develop software in the laboratory and to the favour of internet companies (so actual (customer) projects)

Things do not work out as people thought. Companies want to make (big) business and in the end there may be almost nothing left (over).

I guess it can happen to others, but e.g. open sourcing SAP might be much more of an efford (they have a lot of business know-how in their software)

> I see a clear decline of SOA and a push towards open source

everything old is new again.

http://wayfinder.co/pathways/53536427f7040a11002ae407/a-fiel...

There is nothing, there is no problem

SOA invented a problem and then tried to solve it:

You hack together HTTP + JSON and that's it (Roy Fielding... thank you for "REST", but I am not sure what you are saying)

The backend does not give enough to be just working on it... we are all full stack today and the front end is very interesting too.

i don't think SOA invented a problem at all. I think the problem existed, and has existed, for as long as we have been building distributed systems.

SOA over-engineered though, and basically strangled any practical use it had by having layers upon layers of enterprise blah built on top of it.

microservices is the new wave of the 'well, we have this problem' that is springing up.

it says you need a lightweight communication layer (basically REST by default), and then you use whatever transport suits the problem.

Full-stack is basically meaningless by the way. It really depends on how big your "stack" is, and once you take a far enough step back, the web front-end is such a tiny part of the problem space.

http://wayfinder.co/pathways/535e46a318fbce1100cf7f2a/the-va...

Are you concerned or have you looked into any patents IBM might hold on this?

This project is about Computer Science. It is about finding THE SINGLE RIGHT WAY THERE IS (for fast message transformation)

I helped IBM on some (important) parts of Broker... this will get interesting to divide this up ;)

Are there many people still at Hursley labs working on WebSphere Message Broker? I did my internship there as a software tester many years ago.

I've never been to Hursley. Just www.mqseries.net :)

Reply on news.ycombinator.com