The only difference at our place is that we assume FTP works and do our tests on the file system. We don't test whether FTP works. We know it does.
I understand that sometimes one has to work because of constraint imposed from parties beyond the control of the project. This may force one to support FTP. But FTP is still a terrible protocol. Uploading files to a directory and processing those from a batch job is not a proper solution to an event driven solution.
> To us, it is the only highly available and reliable one.
As one who has had some experience with FTP I must say that you should consider reading after the meaning of HA and reliability.
> Web services have downtimes and re-sends
This is pure bullshit, pardon my French. How does FTP save you from a connection trouble? How does FTP have less downtime? FTP is a stateful protocol, cannot be simply load balanced and made HA. Also idempotency makes some HTTP operations safe to redo if connection troubles arouse.
For really huge amounts of data there is BitTorrent.
I mean, sure, FTP the protocol works fine, and so does virtually all FTP software, but for communication it lacks an important feature: the ability to tell when a file completed uploading.
When we started to upload reasonably large files on reasonably fast file systems this became problem. Our file watcher noticed that a new file arrived, but it's just started uploading. We tried polling the FTP server for connection status, but it turned out that the client supported and did at time use, resumed uploads on connection drops. We ended up having to parse each file just to tell whether it had finished uploading. The file fortunately was an enormous XML dump, so we could check completeness pretty cheaply.
All of this work would've been avoided with a transactional protocol such as HTTP.
(we tried to communicate with the people who did but they turned out to be the supplier to the supplier to the customer of our customer, 5 countries away, and fixing anything at all was not high on their priority list :D)
Ftp was because of legacy though, I would have preferred http post
I'm guessing that the FTP server does the job of not showing half-finished files, since I've never heard of a half-finished file being read back from an FTP server in 15 years. I may have to validate this assumption, though.
Like tvjames mentions, you can rename the file once complete. Alternatively you can ask for a .md5 file to be uploaded as well and, if all else fails, you can simply poll for the "last changed" time of a file and assume that if it wasn't touched for 15 minutes it's probably done.