Hacker News Reader

Comment by an1sotropy | Hacker News Reader

I was hoping for some explanation of how this is envisioned to augment or coexist with HDF, a popular existing format for scientific data (but didn't see one in the docs). Did I miss it?

View on HN

From the introduction: "On the other end of the spectrum, formats such as HDF5 and BLZ address problems with large data sets and distributed computing, but don’t really address the metadata needs of an interchange format. ASDF aims to exist in the same middle ground that made FITS so successful, by being a hybrid text and binary format: containing human editable metadata for interchange, and raw binary data that is fast to load and use. Unlike FITS, the metadata is highly structured and is designed up-front for extensibility." [0]

Frankly, I feel like making the metadata of a scientific data structure human-editable is something of a mis-feature, or at best a non-feature. I use metadata in HDF5 files as a form of provenance tracking and I'd rather there be some friction to editing it.

[0] https://asdf-standard.readthedocs.io/en/1.0.3/intro.html

Metadata needs that, e.g., h5ad solves? I think there's quite a bit to improve on for h5 (it's very slow), but h5ad adds great ways of managing indices and meta data.

Good thing someone already invented NetCDF to address the metadata needs too...

In limitations it states: >While there is no hard limit on the size of the Tree, in most practical implementations it will need to be read entirely into main memory in order to interpret it, particularly to support forward references. This imposes a practical limit on its size relative to the system memory on the machine. It is not recommended to store large data sets in the tree directly, instead it should reference blocks.

I would guess that HDF5 would be the better choice for large datasets. However I quite do not understand the capital 'Tree' in this sentence and what that means for practical data sets.

It is envisioned that this will replace HDF5, at least for astronomy — based on this 2015 paper.

https://www.sciencedirect.com/science/article/pii/S221313371...

I’ve used HDF5 before but not ASDF so I can’t fully evaluate their points one way or the other.

Reply on news.ycombinator.com