Hacker News Reader

Comment by joshribakoff | Hacker News Reader

I had an integrator request this so I stood up a nodeJS server that only implements upload, not download. This way if they leaked their own password, a malicious actor is limited to forging data, and no real data can be leaked. Because it didn't work in FileZilla, they didn't want to use it. Worked at another company that shuffled data between big name gyms & health insurance companies, it also used CSV files sent over FTP in all directions, to my dismay. CSV isn't even a well defined format, and you get all kinds of impedance mismatches with different delimiter & escaping mechanisms, character encodings, BOM, etc. Other companies will just give you a SQL user & let you go to town mining their database directly. I don't understand whats so difficult about making an API, but sometimes it seems like no one wants to do it. You can't push back too much or they will just see you as a problem & decide not to integrate with you.

View on HN

Funnily, you're complaining about csv formatting, guess you never worked with ACH formatting. It's cobol fixed length files. I implemented parsing/creation for this file format at two different companies. Hired on at one company, first day, my manager said, "hey, you have experience with ACH right? we have this project...". That's when you learn to start leaving stuff off your resume.

NACHA cha cha cha!!

You can't spell Atlach-Nacha without ACH.

You can do a lot worse than ACH. It's hard to read, but it's simple and pretty well-defined.

What _really_ sucks is one-off fixed-width formats that aren't well defined, or that change suddenly (oh, you thought that field would always be populated? lol no.)

Yeah, ACH is surprisingly not-unpleasant, at least relative to nightmares like X12 EDI with its implicit looping constructs and billions of companion guides that supercede random parts of the base spec.

X12 EDI, been there earned the badge, both on the generating and receiving sides. 837, 834, 835 and others in health care - fun! Positional format, with situational meaning... really it is a fascinating format.

i wrote the X12 EDI stuff for a major clearwater distributor... also wrote the ACH stuff that dumped someone a file and their job was to upload it via SFTP to the bank. so many badges.

Don't you mean nightmare format?

Nightmares can be fascinating

Years ago I worked with a product that parsed line printer data, it was a nightmare trying to get every piece of optional data into the reports.

What _really really_ sucks is well-defined, reasonable, meticulously documented formats where all in-the-wild implementations stray from the spec in different ways :-/

Previously I had no experience with ACH but my current project is essentially reverse engineering a very proprietary version of NACHA and another file that is 400 fields long. Honestly the current parser does work but I can only assume due to some black magic or other voodoo.

Every time someone complains about file formats I bring up a similar story. It's hard to believe fixed width files are still being used today. Parsing these kind of files with 45 million records, each 600+ bytes is a huge pain. I chuckle when I think about the first time I dealt with these files and thought I could open them in an editor...

> It's hard to believe fixed width files are still being used today. Parsing these kind of files with 45 million records, each 600+ bytes is a huge pain. I chuckle when I think about the first time I dealt with these files and thought I could open them in an editor...

Depending on the business-application, fixed-width formats are fantastic because they allow for genuine O(1) random-access to records without needing to precompute an index first. Writing parsers for fixed-width formats is also much simpler, for example, because it eliminates ambiguity around null-terminators vs length-prefix in text/string/array data - for resource-constrained environments it's great because you can guarantee you won't need to dynamically-allocate buffers. I feel the only real arguments against fixed-width records are concerned with wasted space - which are mitigated if your system lets you compress them somehow - because they'll typically compress very, very well.

> I feel the only real arguments against fixed-width records are concerned with wasted space

Sometimes, fields can turn out to be too short. I have received a few letters from business with my last name truncated, because somebody way back when decided eight or nine characters are definitely enough for a surname.

Ha! I wrote an editor for fixed length files, it lets you map to a cobol copybook, or create the layout manually. I never released it. Anyone interested?

If it works on Windows, I'd love to give it a try! Our ERP system is written in COBOL and I have to deal with fixed-width files all the time. Does it also handle REDEFINES and level 88?

Yes it handles redefines and 88s, my emails in my profile, I have a Windows build.

Ugh, reminds me of the time I had to integrate with a bunch of school software. They used csv over ftp (no auth). You just had to connect to the server and get the personal information of a district's students. Worse is you could google the URLs... I built a system that solved the problem but left shortly after it began to be integrated nationwide.

wait.. are you saying that a district had public ftp with personal student information? Pretty sure that breaks the law.

Email me at josh.ribakoff at Gmail I'd be interested in chatting. I think we worked at the same company. Or direct competitors

Sure, I'll do.

ACH needs to die. Its a very old system, and you have no clue if a transfer ever actually goes through. Only ACH rejects ever get sent back, and there isn't a definitive window in which you'll get that reject.

Wait. ACH doesn’t even tell the sender if it succeeded??!

Yeah there are no acks. It sucks.

The lack of acks is why most companies receiving money through ACH will "season" the transactions for a set amount of time. Most of the returns occur in the first 3 business days so most seasoning periods are just around that long. The reality is you can receive a reject up until (I believe) 60 days after the transaction date. I've seen a lot of companies be a little smarter here and reduce the seasoning times for repeat users using the same source bank account. It is a fascinating system to work with every day. Wire transfers are their own bundles of fun too.

Now I finally know why my credit-union (which serves the high-technology industry exclusively) told me that they couldn't put a hold or otherwise segregate any cheques I deposit until they've cleared... because they have no way of knowing when/if they've cleared!

It's 60 days, + some possibility of up to 2 extra days at the end for Fed Reserve holidays, and then in cases of outright fraud, occasionally banks will ignore the rules and return stuff later than that.

NAK-but-no-ACK is apparently a super common pattern in enterprise (both internal and between parties.)

I have no idea why it is and why people are so resistant to adopting ACKs.

Why it's there in the first place is fairly excusable. It (at least) halves the number of datagrams. Why people still resist it I'm not so sure about.

But on the other hand, it's a complicated architectural decision. US military wants absolutely reliable communications? They invent TCP which relies a lot on acks. Swedish telecom company wants fast and absolutely reliable communicating systems? Cue throw hands in air with a "there's no guarantees even with socks so better to simply never assume your message arrived and instead focus on detecting anomalous behaviour." They invent Erlang where there are no acks for messages.

They already have an API, it's called FTP or SFTP or whatever.

I don't know why everyone thinks they need an API to reimplement (poorly) basic unix functionality that's been around be for years.

Especially when FTP and HTTP performance is basically comparable and will depend more on the network and storage than the protocol.

Have we gotten to the point where anything not HTTP is considered old?

never mind that most "API" are JSON (or some such) delivered over (S)HTTP anyways.

But yes, anything but HTTP is "old". In large part because of corporate firewall rules that block anything but HTTP traffic (and more often than not filter even that traffic).

No. I'm referring to an API to query data in realtime instead of batching the entire freaking database and then having out dated snapshot of the data

>Because it didn't work in FileZilla, they didn't want to use it

Sounds like they didn't need much if any of what you did. They wanted X and you gave them Y. Y would require them to change the way they worked. Did they need query capability? Seems like they didn't and SFTP would have suited them better.

Honestly sounds to me like you went for an overly complex solution where something simple would have done just fine.

Our company and their company have a common customer. Neither of us need each other we want to make our customer happy. Customer would be happiest with realtime data but company B wants to send snapshots via sftp. So I setup an endpoint that speaks sftp. They even told me to "do something" to make it secure. Then they complained I used a port other than 22 and that it only worked on the command line. It sounds like the problem is the other company doesn't want to do any work / wants to send the file manually. I don't see how I went for an overly complex solution. I attempted to do exactly what they asked and they moved the target so of course I missed. In fact it's this company that is just trying to avoid doing any programming. But that comes with the territory. I respect your opinion but disagree with you

Fair enough, I didn't have the backstory.

But but rewrite all the things in node.js!! Don't worry about understanding the file format, which is well established, or what the users of the file are actually doing with it, also well established, the important thing is node.js!!!

Frankly once they hear it's node.js the users should be bowing down to worship!!!!

Actually I wanted him to make an https API for us to query real time data. I would have used PHP to consume his API because it's easier to find developers on our end and synchronous code is easier to maintain when performance is not a concern

why use a huge codebase like node for such a simple security critical task

When all you have is a hammer...

How else do you suggest you standup a sftp that cannot leak data even if the password is compromised? I'm all ears. Node is a runtime by the way and huge in what regard? The Linux codebase is also large.

First thing that comes to my mind is a PAM plugin that generates a new root directory for the user when they log in, named something sane under a fixed hierarchy. A recursive walk can later pick them all up and clean up.

Can't you just host a directory with -w- permissions? "Drop box" directories have been a staple of file sharing software for decades.

Seconding what kalleboo said, setting directory permissions has been a staple of file-sharing for decades. Setting up a node server to block read-access seems incredibly byzantine.

Good point but we also have to load balance this / have failover and provision users on the fly. I could have used messaging queues to provision PAM users on the whole cluster and this guy would have still complained because I used a non standard port. Not sure why that was a problem either but again we can't press back or question him. he would cancel the whole integration. Bottom line is our end user wanted real time data (presumably via an API)

what's wrong with nonstandard ports?

A lot of FTP servers have provisions to essentially "ignore" the user and proxy into the directory. As far as load balancing is concerned, most FTP servers can handle very high throughput because of the simplicity of the protocol. If you really did want to load balance it, haproxy can do that just fine.

> we can't press back or question him. he would cancel the whole integration

This attitude will not serve you well. There's saying no, and then there's getting more information (requirements!?!) to then guide people to better decisions.

You use chmod on the directory...writing any code for this kind of task would be crazy overkill.

It is a system administration problem, not write a new software problem.

You:

a) think about the scope of the problem and what you are trying to achive - you are blocking reads of the files by the uid/gid of the server. It is a known and solved problem. The tool is called chmod

b) think about the surface area of the attack - it should only be the SFTP server. We have ones that are very well known. I recommend the one that comes with OpenSSH but there are others.

c) think about business requirements - user's provisioning etc. That has been a solved problem for years - for the super-complex cases LDAP is used. For the simpler ones you can use PAM.

After thinking about that you just write the needed glue.

I did a similar thing, but decided to have LIST/MLSD/NLST, but stub out GET/RETR/MFCT/MFF/MFMT to return nothing or static data.

What did you use to implement?

Wrote it myself because it was Elixir so why not. I'd be more careful doing this in C.

Reminds me of a time when I had to implement a parser for a certain kind of file to transfer money using an "API" provided by a financial institution in Asia. Yes we are using CSV too, also sent over SFTP. The file format is, unsurprisingly, ill-defined.

Wouldn't FileZilla have been happy with a fake blank directory response?

Yeah I added that but the particular node sftp implementation I used from github just simply didn't work with FileZilla. Only the command line. Don't know why.

Whenever someone says “from github”, I mentally replace it with “from some random guy on the street” to give it its appropriate intuitive feeling.

Sure I don't care who wrote it if it works and passes our code review and the license matches. Plenty of crap software comes out of big name companies and plenty of random people on the street can write good code.

If you have no other alternative and can’t take the time to write it for yourself, and need something now, sure, go nuts and take something random off Github, PyPI, npm, or whatever.

But code that does not have upstream development and support is dead code. A piece of code in production is like an Internet connection or electrical or water hookup – it must be, so to speak, “connected” upstream to whoever is providing continuous upgrades and security/bug fixes. Otherwise, it is (to continue stretching the analogy) like a mystery battery or water barrel – it could go bad in many ways at any time and you wouldn’t know it until your equipment starts to fail because of low voltage or voltage spikes, or you start to die of legionnaires’ disease. And unlike a battery or water, there are only cursory, no good and thorough, ways to test their quality, and no way to test their age as measured by the way they interact with the ever-changing outside world.

I've always been overly eager to add dependencies to projects and while I know why it may not be good I was missing a good intuitive sense of why. This is such a good way to view it. Thanks!

Reply on news.ycombinator.com