Hacker News Reader

Comment by skrebbel | Hacker News Reader

IKVM has saved my ass a number of times. The ability to, at times, give up on the .NET ecosystem and peek into the Java ecosystem for suitable open source is fantastic.

We once had a project where some genius long ago had mandated that parts communicate via FTP. Eg one program uploads a file, another program monitors the directory. Total clusterfuck, so we wanted reliable automated tests. Turns out that doing an FTP server in .NET is pretty difficult. Full-fledged commercial libraries exist, but we just wanted a stub for testing.

IKVM, Apache Mina FtpServer solved it in two hours for us. I suspect many .NET shops use IKVM this way.

Sad to see it go, but it must've been an incomprehensibly huge one man hobby project for 15 years! If I were Jeroen I'd quit a long time ago :-)

Let's hope some shops get sufficient value out of it to maintain a fork. Jeroen if you read HN, thanks!!

View on HN

Do these parts need to communicate rarely, but with HUGE binary blobs? Because in this case it does sound like a good design, actually.

Eve then a HTTP POST/PUT is better than FTP, but the parent sated this was not their decision, but a constraint.

We have the same setup at our place. FTP communication isn't a "clusterfuck", it is the tried and tested communication method between any two large companies that aren't banks. To us, it is the only highly available and reliable one. Web services have downtimes and re-sends. Our partners practically never turn off their FTP machines.

The only difference at our place is that we assume FTP works and do our tests on the file system. We don't test whether FTP works. We know it does.

How did you deal with large files, dropped connections, and so on?

I mean, sure, FTP the protocol works fine, and so does virtually all FTP software, but for communication it lacks an important feature: the ability to tell when a file completed uploading.

When we started to upload reasonably large files on reasonably fast file systems this became problem. Our file watcher noticed that a new file arrived, but it's just started uploading. We tried polling the FTP server for connection status, but it turned out that the client supported and did at time use, resumed uploads on connection drops. We ended up having to parse each file just to tell whether it had finished uploading. The file fortunately was an enormous XML dump, so we could check completeness pretty cheaply.

All of this work would've been avoided with a transactional protocol such as HTTP.

I've almost exclusively seen this handled using an upload with a temporary filename (usually convention based) and then renamed on completion. Watch then for the existence of the renamed files. Clean up temp files after a defined amount to time (30 days for example).

Ahyes, smart. We didn't control the client, unfortunately, so we couldn't do that.

(we tried to communicate with the people who did but they turned out to be the supplier to the supplier to the customer of our customer, 5 countries away, and fixing anything at all was not high on their priority list :D)

Afterwards change the extension to .xmlok or .xmlfail (alt: zip it), no need for vendor adjustments

Did that to, even open sourced a client to remove folders depending extension and x days https://github.com/NicoJuicy/CLI-Cleanup-IO

Ftp was because of legacy though, I would have preferred http post

Not sure what you mean by huge, but our partners frequently upload 100+ MB files in various XML, IATA and EDI formats. We also use an FTP client locally to read the file back. We don't trust the file system state when it is being controlled by a server.

I'm guessing that the FTP server does the job of not showing half-finished files, since I've never heard of a half-finished file being read back from an FTP server in 15 years. I may have to validate this assumption, though.

The benefit of FTP is that it allows you to resume transfers which is especially important with huge files. HTTP is not particularly nice in that regard. It can also be unclear where exactly the HTTP server is storing the intermediate data. If you're not careful it'll try to store the file in-memory before writing it to disk.

Like tvjames mentions, you can rename the file once complete. Alternatively you can ask for a .md5 file to be uploaded as well and, if all else fails, you can simply poll for the "last changed" time of a file and assume that if it wasn't touched for 15 minutes it's probably done.

I think FTP the protocol would've been fine, and indeed is best suited for the job. The problems really stemmed from having to poll the filesystem because the FTP server was also not under our control. If you embed the FTP server in your app (say, with IKVM ), you can keep the whole intermediate file where you want (or even in RAM if it fits) and this entire problem shrinks 10x.

I beg to differ. HTTP is far better for any usecase than FTP.

I understand that sometimes one has to work because of constraint imposed from parties beyond the control of the project. This may force one to support FTP. But FTP is still a terrible protocol. Uploading files to a directory and processing those from a batch job is not a proper solution to an event driven solution.

> To us, it is the only highly available and reliable one.

As one who has had some experience with FTP I must say that you should consider reading after the meaning of HA and reliability.

> Web services have downtimes and re-sends

This is pure bullshit, pardon my French. How does FTP save you from a connection trouble? How does FTP have less downtime? FTP is a stateful protocol, cannot be simply load balanced and made HA. Also idempotency makes some HTTP operations safe to redo if connection troubles arouse.

For really huge amounts of data there is BitTorrent.

I don't have a fr-en translator available so I will try to keep it simple for you. FTP does not require restarts or reloads, and lets you continue broken uploads. The only scaling is you can do here is vertical scaling. Sorry if that doesn't fit your world view where every service must be containerized and horizontally scaled. No matter what you do, some ignorant losers will keep on using their single server ftp and call it reliable because that is their experience.

Reply on news.ycombinator.com