796 karma · joined January 7, 2012
[ my public key: https://keybase.io/lgierth; my proof: https://keybase.io/lgierth/sigs/96UTewVcgE6mn2Hh1Tg3WCHUDRP_jWSFookoCcfruAg ]
Ideally your approach to pulling the data from its source into IPFS is somewhat deterministic, so that you get de-duplication and generally more efficient updates.
Could you be more specific? We're aware of a good bunch of reasons for IPNS's varying performance, but hashing isn't one of them.
On your point about the gateway -- I've co-maintained the ipfs.io gateway for the last 3 years and my perspective on its importance is the upgrade path from the existing systems. When developing and deploying new protocols, you need to onboard both users and applications/websites, and the most effective way to do so is seamless interoperability between old-web and new-web. I would imagine it tricky to get any critical mass of adoption if everyone has to choose either old-web or new-web.
Also: Being able to inter-link any content-addressed data structures. With IPLD [1], you can link from an IPFS directory to a Bitcoin transaction, to a Git repository, to a Torrent.
[1] https://ipld.io && https://github.com/ipld/ipld#implementations
An example of slightly different approach (a sharded index) is the wikipedia-on-ipfs experiment from last year: https://github.com/magik6k/distributed-wiki-search && https://github.com/ipfs/distributed-wikipedia-mirror
What Cloudflare adds to the mix is their infrastructure for automatic TLS certs.
Firefox has been supportive of the effort for some time already, working on libdweb: https://github.com/mozilla/libdweb
Which is ARIN/RIPE/APNIC/AFRINIC/LACNIC and the DNS root zones, and ICANN on top.
Not to say that's only bad, just trying to illustrate that in this case, D&G's point is actually pretty tangible.
I work on IPFS/libp2p and the "only" two known attacks on the official gateway ipfs.io are in Spain (DNS hijacking) and China (dito).
OONI has pretty good data on interference around the world: https://ooni.torproject.org/
(edit: clarified that the attack is only on the ipfs.io gateway, not the IPFS network)
HN discussion: https://news.ycombinator.com/item?id=17060437
Apart from that IPFS is built and layered in a way that makes the application layer independent of transport semantics - the network in terms of connections and hosts is abstracted away.
Other data formats that are ordered, and serialize somewhat more deterministically, are CBOR, Protobufs, or s-exp.
We've switched the default browser-to-browser transport in js-libp2p to a websockets relay recently, so you should give it another try.
- avoid data corruption when downloading/transferring/copying datasets
- notice changes/updates in the original dataset
- dataset versioning (think how e.g. git turns directories and files into hash trees -- also called content-addressing)
- most importantly: stable names without a naming authority
Do you have plans for adding file hashes to the datasets, e.g. sha256? This would make it a lot easier to integrate with other systems.