dabiged

Not a book, but MIT OCW has a class on Aircraft Systems Engineering 16.885J (focusing on the Space Shuttle) that is an excellent example of this. The class starts by looking at the requirements for the system, then examines every single subsystem on the Shuttle, and explains, in detail, why it was built the way it was. Almost always the answer is "because of the constraints placed on the system by the rest of the vehicle" + it has to be as light as possible.

I initially watched the lectures because of an interest in aerospace and it is a fascinating historical series in its own right with some incredible speakers. The lessons for systems engineers are numerous too.

I agree. This would work, but you still need to get the data off the tape, and onto the SSD. The only way to do this is 60MB/sec.

The data is binary formatted floating point data, generally using the IBM floating point format, though IEEE floats are also allowed. Standard compression like gzip basically does nothing. All tape manufacturers claim their hardware based compression gives 2/2.5/3 times the tape capacity and it is all baloney. A 1TB tape holds 1TB of data.

There have been multiple attempts over the years to implement all sorts of algorithms but they have all failed to gain traction. For example Wavelet based compression works fantastically well for some data (i.e. real signals with band limited frequencies), and catastrophically badly for others (earth models defined with piecewise functions containing non-differentiable points). Some people are happy with lossy compression, others are not.

The only compression I have seen worth anything is in deep water, where there is literally no data in the water column (i.e. 500 consecutive zeroes in a 6000 byte block), gzip gives you the that section free.

Just use a cloud provider. If you are worried about privacy encrypt it.

For example, AWS Glacier deep archive would be years of storage for the cost of shipping a tape drive. If you have bandwidth issues use snowcones.

I am talking about bottlenecks on reading from the tape. Sending the data to /dev/null eliminates any write bottlenecks and we still max out at 60mb/sec on a 250mb/sec drive.

The main issue is that the standard was written back when file formats and tape formats were essentially the same thing. Data was read from tape, processed and written to a second tape. The memory of machine at the time were of the order of 1 block.

Decoupling tape formats from file format effectively resolves the issue, that is: read the tape in 6k blocks, write the file to a modern file system. Set the tape block size to 10mb. Write the file back to tape as a tar. You get the bandwidth limit of the tape drive.

The issue is almost all legacy industry software is still designed to read and write directly to tape objects so your tape isn't readable in all the software this data is designed for.

I work with a lot of tape in the oil and gas sector and one of the major issues we see is that industry wide file formats for data exchange are still based on old tapes. These formats have fixed block sizes that haven't changed since the 1950's. Modern tapes have exponentially larger capacities and bandwidth but due to the use of these file formats we rarely see 20% of the theoretical throughput on a brand new tape drive and cartridge. The industry has no desire to change the file format so we are stuck in this situation where tape read/write times grow exponentially with every new tape generation.

It is as if the author has completely forgotten the last 40 years of human habitation in low earth orbit, onboard tin cans in space.

I am on 35/7 ish, 20/1 when it rains, within 10km if the CBD of Perth, the fourth largest city in Australia. There are point-to-point wireless providers that tech savvy folk use that will get speeds in the range you are used to. I haven't taken it up because I rent and it is a substantial upfront charge for the hardware.

Who knows you might win the NBN lottery and get fibre at your house!