F*EX – Frams' Fast File EXchange | Hacker News Reader

F*EX – Frams' Fast File EXchange

fex.rus.uni-stuttgart.de

72 pointsTomte23 comments

Classic and relevant quote from 1985-ish: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."

Today, the equivalent may be an autopilot Tesla Model X full of microSD cards. Terabyte-per-second calculations left as an exercise to the reader.

Some places straight up mail hard drives back and forth. It can be quicker per GB and/or cheaper depending on what links are available. Also, you generally wont have that bandwidth lost to other users' on the network. The shipping companies are happy to take their money, too. :)

My lab does this. The last step in each experiment is to unplug the hard drive and carry it back to the office for analysis. We have gigabit Ethernet but this is still way faster.

Assuming the internal storage volume of the Tesla (which I assume is actually the boots/trunks of the car) is 88.1 cubic feet [https://www.tesla.com/support/model-x-specifications] or 2.49 m^3.

A microSD card is 165 mm^3 = 1.65e-7 m^3.

Dividing one by the other: 15090909 microSD cards could be put in the car [This is simultaneously probably unrealistically high due to the assumed ideal packing and also low due to the large amount of space left in the car].

256GB is the largest microSD card I can actually buy: This is gives us 3.8 exabytes of data hurtling down the highway.

How to efficiently load/unload this data is beyond me. (Short of magic wireless microSD).

V unira'g purpxrq nal bs guvf, V'z n ovg qehax.

I wonder if there have actually been 15M 256GB microSD cards even manufactured at this stage.

A Google search for “88.1 cu ft / 165mm^3 * 256GB” largely confirms your calculation (just over 3.8 XB of data). Can’t say I’m surprised - MicroSD cards really are crazy dense these days.

If it takes an hour to get to your destination (ignoring load/unload time), that’s an aggregate data rate of around 1000 TB (1 PB) per second. The entire Internet averages something like ~25 TB/s, so our Tesla X has the bandwidth of forty Internets.

Of course, there’s a downside: that many MicroSD cards would actually weigh around 3.7 tons (at 250mg apiece), which is heavier than the Model X itself! Might be harder than expected to make the drive itself.

Relevant XKCD (because there always must be one): https://what-if.xkcd.com/31/

A similar calc with LTO-8 tapes comes out at 126PB per Model X. Not as dense, but the advantage is that tape robots and tape libraries go a little way towards automating the load/unload.

Today, the equivalent is an Amazon Snowmobile, a 45 foot shipping container weighing 34 tons transported on a semi-truck. Transfers about 100 petabytes at a time. They claim it can be filled "in less than 10 days". Of course, multiple Snowmobiles can be used in parallel if you have exabytes to transfer.

Assuming it takes about 3 weeks to turn around, my arithmetic says each truck has a bandwidth of about 460 Gbps.

"Q: How do I connect my data center to Snowmobile?

Each Snowmobile comes with a removable high-speed connector rack on wheels with two kilometers of ruggedized networking cable."

Source: https://aws.amazon.com/snowmobile/

Less than 1GB, encrypted, 24 hour expiration or one time use URL then it denies the URL exists or existed, and deletes the file. https://send.firefox.com/

Shameless plug: I wrote a CLI interface for it: https://github.com/nneonneo/ffsend

magic-wormhole is a pretty neat solution and lets you transfer where both ends are behind NAT. https://github.com/warner/magic-wormhole

Interesting. My questions (1) what is the timestamp of this page (created or last modified)? (2) Could you use a torrent client to solve this problem?

1. From http://fex.belwue.de/fex.html, it looks like the software can't be any older than 2006. Looking at the source, it uses things like rel="prefetch". The style is just an explicit aesthetic, then, like that of Craigslist.

2. Some of it could, yeah. Torrents make it easy to share files directly from your computer to a peer's. FEX solves a slightly different problem, though—the asynchronous "put the file somewhere, then let the peer get it while your computer is offline" use-case. It's somewhat cumbersome to use torrents for this (though possible: you and the server join the swarm, the server leeches from you, you leave the swarm, the peer joins the swarm, they leech from the server.) But, if you do use torrents that way, you won't [by default] get the "auto-deletion of the server's copy" property that FEX has.

(You could probably match 99% of FEX's functionality with a torrent client running on a server + a daemon running alongside it that can control the torrent client through RPC; but I don't know of any attempt to build such a thing.)

looks like the software itself is from 2006 - 2016: http://fex.belwue.de/fex.html

It references an xkcd from September 2011, which gives you a lower bound for last modified at least.

I use this software at work and can say that it is in active development (last version: 20180528 (Arch Linux)... that means just a couple of days ago). But my favorite part of F*EX is the sharing of versioned file archives (https://fex.rus.uni-stuttgart.de/sharing/ ).

A more modern approach is to stream the file from the sender's web browser directly to the recipient's web browser, using WebRTC and related web APIs.

An example implementation, on top of WebTorrent: https://instant.io/

Does it encrypt? Is there any privacy? They complain about the lack of privacy for other services so I assume they're doing better somehow?

You run it on your own server so it offers as much privacy as you can provide.

Can you put it behind nginx or does it only work as a stand alone service?

EDIT: probably, https://fex.rus.uni-stuttgart.de/usecases/reverseproxy.html

A time before Syncthing, dark indeed.

Not going to reliably without a resumable upload/download manager with fallback to traditional means. Murphy’s Law insists a transfer will get to 99.753% of a 3.5 PiB data set and time-out.

AWS Snowmobile/Edge, Snowball and Import/Export; SneakerNet™ and PostalNet™ for other needs.

"Main features of F*EX

...

RESEND and REGET for resuming after link failures at last sent byte"

Reply on news.ycombinator.com