I can't say thank you enough for what you have done. None of this was a failure.
> This needs some sort of WRT54G kinda moment, synology or ???, you needed to have a last mile partner or a host (vpn in to services)
While I love the idea of everyone running their own hardware, I've been thinking recently that physical separateness might be the least important aspect of decentralization. I think alignment of incentives is more important. What I mean is that having your apps/data on google's servers isn't necessarily a problem, but being locked into their platform and unable to easily change is. When it's simple and convenient for customers to change providers, companies are incentivized to maintain high quality products and features such as privacy guarantees.
Was following Sandstorm closely, and despite the fact that im going with different choices compared with Sandstorm, your project was one that i instantly recognized to be tackling in the same domain.
I mean, i think your meditation and problem solving are in the right place, we just need a couple more iterations , research and efforts to get there.
I will try to give it a shot too, been working on this on a couple of years over the chromium codebase, with a design that was not quite right in the beggining, that needed couple more of iteration and tought, but that in the current incarnation, i think i might have something to show.
I hope i can count with your comments and criticism when i launch it, which i believe, its only a couple of months from now, as i have a great respect for your work and your vision.
You helped to move the needle forward in a domain we need desperately of a solution or else all of our data and freedoms are at stake (not to mention it can even be a better technology than the current cloud-corporate centric vision that are designed to maximize corporations profit and data centralization, where we dont actually own anything, anymore).
So thanks for all of your work, it was really inspiring.
Can you give an example of what you're talking about here?
Your applications and executables are also part of the data and move together with them.
But sandstorm is something else. It composes well. You can have an application that is a good document editor and another one that offers a spreadsheet that works for you, and have a unified layer that handles identity, authorization, document management (collections, sharing).
Short slogans as "liberate your data" or "self host your apps" focus on the end goal without highlighting what is really stopping us from doing it: the integrated, cohesive experience many of us expect/need, especially in the enterprise environments.
I recently went through an acquisition where we had to switch from gsuite to office 365. Oh my. What a mess.
In the ideal world we would have had our stuff in sandstorm grains, and after the merger our new colleagues would have access to them even if we wouldn't necessarily have picked the same spreadsheet app for example.
Now, for those of us that Word doesn't work well and were happy with Google Docs, well, we don't have a choice: we cannot possibly give a gsuite account to every employee in the larger company and thus we have to migrate in order to not preclude potential collaboration
SaaS providers basically have vertical monopolies here and I think that hurts consumers.
Can you talk about how this informs your work on Cloudflare Workers? Should I be able to take my workers and run them elsewhere, and on software that's not second-rate, as you put it? Should all the proprietary code behind Cloudflare Workers be something I can pick up and take to another host?
I'm not entirely sure yet how that would work but it's something I like to think about.
- For reasonable performance and reliability, code needs to run close to data. You can't be accessing all data remotely over the internet.
- Requirements to use "standard" data formats prevent apps from innovating new features not covered by the standards.
- Different applications call for different kinds of databases, and there are still lots of new ideas to explore in database technology. It would suck if all apps were forced to use the lowest-common-denominator database protocol supported by user storage.
- There are tons of classes of data that work rather well with streaming modes of computation, especially video, music, and images, which make up most of the average user's data. It can even work for bigger data. I recently had my whole genome sequenced. They're sending me the data on a 500GB hard drive, which I'll upload to the cloud and use tools like iobio[0] (which I work on) to analyze it remotely. Even in cases where you can't use sampling, often the actual computation is as much of a bottleneck as the network throughput is.
- I'm thinking more along the lines of a true hard drive analog, ie the apps can store whatever arbitrary data they want, including binary blogs and special config formats. The service provides a simple API for storing/retrieving files, notifying on updates, and handling auth/permissions.
- Again there are a huge class of problems that work just fine with flat filesystems. I think databases are heavily overused, especially in cases where only 1-100 people need to use the instance. You could run into issues with large collections of music or photos, but I don't know anyone who keeps more than a few hundred photos in any one directory.
[0] http://iobio.io/
However, the flat filesystem model would be awful for most productivity apps. Think GMail, Google Docs, Calendar, Slack, Trello, GitHub, Jira, etc. These are the use cases I care about.
Video, music, and images might be the bulk of most users data by raw volume but certainly not by frequency of access or importance.
I'm not aware of any providers that offer the ease of google drive and the access of S3 (ie can I host a website on it). As far as I know both GDrive and Dropbox removed the ability to host files on the web in the last couple years. Do you know of any comparable products that offer both of those? I'd be very interested in looking at them. I think S3 is the closest, because you can access the filesystem through their browser APIs, and publicly over HTTP. And it got popular enough for people to make open server implementations. The problem with S3 at the moment are that it wasn't designed for this use case, so it's missing features like Firebase-style update events.
> However, the flat filesystem model would be awful for most productivity apps. Think GMail, Google Docs, Calendar, Slack, Trello, GitHub, Jira, etc. These are the use cases I care about.
GMail, for sure. GDocs, I can't think of why you couldn't implement it for a reasonable number of users with nothing but texts files using atomic updates, or maybe come up with a standard protocol for invoking text-based CRDTs. Or just send the change updates as git diffs and if someone tries to change a line that would result in a conflict reject that change and send back the current state of the document. This would be sufficient for any app for the which the data can be expressed as text files. Worst case you could even allow something like CF Workers for cases where you absolutely need centralized logic, but I think there are a large number of tasks you can accomplish without that. Even something like slack I think you could implement with text files on a disk. Pre-SSDs I'd have to agree with you, but nowadays I think a generic filesystem is good enough for most things that don't involve scale.
> Video, music, and images might be the bulk of most users data by raw volume but certainly not by frequency of access or importance.
True
I think the most important piece of tech that Sandstorm was working on was the capability based security and the powerbox concepts (which I recognized as being similar to Android's capability apis but for the web). I don't see the decentralized data or local servers being an easy sell any time soon, but I can see the capabilities security working really well within the browser, and more importantly with existing SaaS ecosystems.
If we could get a new browser API for websites to offer to register available APIs for jsonschema based type defs through a manifest file, then users could allow those APIs to be included in their local browser registry. Other pages could request (or offer) data from those APIs, and with the user's permission, the browser could do the necessary oauth handshake and use the API, passing data to/from another service in a secure way behind the scenes.
As well since it would be a browser API, a webpage looking to call an API would be able to request provider info and show the appropriate in-app UI for selecting a service, which sounds like one of the problems Sandstorm had.
No one would have to worry about whether they implement a dropbox api or onedrive api, the page would just have to show the 'file service' selector and call the user selected api with a file. So I'd expect every SaaS out there would jump at the chance to provide an easy access way for users to use and pay for their service.
Furthermore, the browser could provide external connectors as well so I could have some native external app that registers a provider with the browser, allowing me to send my in browser data to an external native app (or vice versa). For example right click menu on a .drawio file and send to DrawIO or send to the currently opened Google Slide. Or send a contact from your fav web contact manager to the your native Skype app. Seriously I can think of so many uses for this! And it fits in perfectly with the existing file or clipboard apis as well.
I hope someone sees this excited rant and can make it happen! :)
I also really do hope the work that's been done in Sandstorm continues on and pushes things further for all of us. Thanks!
Ya this looks like it could've definitely fit the bill. Too bad it never took off. And worse yet, both major browsers tried their own version and neither took off.
This just seems so easy an idea to get on board with, like VSCode solving the n-to-1 problem with their language server api. I don't understand why it fizzled out.
It stalled for many many reasons, including:
- XML is kind of awful
- Over-general solution without clear advantages for a specific use case
- Mis-aligned incentives
- easy to abuse open query capabilities
- Hard to use for both data publishers and consumers
I think that JSON Schema with incremental enhancement via JSON-LD is a more promising tech stack for another try. That would let you take advantage of the massive investments in the current API ecosystem while carrying forward the best parts of the old Semantic Web Effort.
Of course the incentives are still hard to sort out. Commercial entities WANT lock-in. They will Embrace/Extend/Extinguish anything they can because they need a moat to make money. Honestly anti-trust regulations might be needed.