The Sandstorm Team Is Joining Cloudflare
sandstorm.io
sandstorm.io
But, here's the things I expect to be repeating a lot:
- Sandstorm is still an independent company, under control of Jade and myself.
- Sandstorm is "my baby" and I'm not going to stop working on it just because I have a day job. Yes, it will slow down a bit -- but on the bright side, we were previously spending the vast majority of our time trying to figure out enterprise sales (rather than actually building cool features), and we don't have to do that anymore. So, the difference may not be as big as you'd think. See: https://sandstorm.io/news/2017-02-06-sandstorm-returning-to-...
- Nevertheless I'm pretty excited about the work I'll be doing at Cloudflare as well -- which includes Cap'n Proto and other fascinating infrastructure tech.
Whenever I hear that I'm subvocalizing 'for now'. How do you see this in the longer term?
What kind of hostile commercial interest there might be, especially when the technology is protected by open-source licenses?
If there were a non-hostile commercial interest, akin to Citus in Postgres, won't it be a win for all?
So in some ways better than an acquihire and in some ways worse.
Even Sandstorm not gaining traction with their B2B strategy is likely a blessing in disguise: with enterprise money coming in they would have felt a hard pull to do more things that enterprises like. No matter how much conviction you have, making payroll often trumps that.
Now, they've proven that the enterprise route doesn't work. My hope is that the project can continue to improve and gain traction to the point where it sets up or joins a foundation, then potentially try to restart a complementary commercial venture on top of that.
Well, no... I don't think we proved it didn't work, only that we lacked the expertise to make it work. With more time and some good hiring I think we could have fixed the problem, but the investor money ran out, so...
Which, if that's the case, means that Sandstorm still is legally independent of Cloudflare, and the original Sandstorm devs can pretty much continue to do whatever they want with it. There would be some gray areas, like what will happen to future code or other IP they create for Sandstorm while employed by Cloudflare, or while using Cloudflare systems/resources; and of course their continued employment with Cloudflare may be contingent on their doing things with Sandstorm that Cloudflare likes, or at least doesn't disapprove of too strongly. But if the Sandstorm devs are just selling their future labor and not the company itself, they'll have more independence than they'd have gotten in an acqui-hire scenario.
Cap'n Proto will presumably end up with mixed copyright, but it's MIT license anyway so who cares?
CF being "cool" shouldn't put your project and its users (me!) at risk.
Even while working full-time on Sandstorm, on weekends I'd work on "things I want to code" rather than "things I need to code".
I usually take a week vacation when there's a new FF to play. :P
Sorry, I meant for that to be clear in the post but I guess I failed.
I'm currently in the process of leaving my full time job to work on my OSS stuff full time again. It's a tough move because you do eventually run out of money if you can't get enough grants, donations or a following on Patreon. That's how I ended up back in a full time gig myself.
This matters because Sandstorm Development Group, Inc. (and not Cloudflare!) has the ability to grant proprietary licenses to the Sandstorm codebase.
Personal thanks to you for both Cap'n Proto and Sandstorm.
Note that Oasis is extremely stable. We haven't had an incident requiring human response in weeks -- and the only kind of issue we regularly had before that is now automated.
With some of the rough corners filed smooth (e.g. Windows support), a bit more marketing and promotion, and eventually more big names advertising their use of capnp, my job getting buy in from my peers would be much easier.
http://dbeck.github.io/5-lessons-learnt-from-choosing-zeromq...
2) protobuf in the proto3 design doesn't cary default values. So if you have a bool field and want to explicitly send false, well you have to change it to some other type or use the default values all the time
3) protobuf generates incredibly large serialization/deserialization support coce for each template. For some languages like Python in can be in hundreds of kilobytes. Cap'n proto messages are significantly smaller
There is more for CnP but Protobuf has much better support and is by default used in projects like gRPC. Also new CnP is lacking speed in new development in comparison to Protobuf.
But I'm using in one of my side projects and I'm very happy with it
Does this mean that individual fields are encoded/decoded lazily?
(But every widely-used arch these days is LE.)
In our case we've now over 1T of rows. Storing default variables would be incredibly inefficient and expensive.
Is nullability of primitive types what you're missing? As in, having a boolean value that can be true, false, or null. If you need that, defining your field as google.protobuf.BoolValue instead of bool gives you that (same with the other primitive types: https://github.com/google/protobuf/blob/master/src/google/pr... ). But making primitive types never null aligns proto3 with most programming languages, making the generated code more idiomatic and performant.
Does capnproto provide similar RTTI information to walk through properties or recurse into other messages?
It's actually considerably easier to implement for Cap'n Proto since the in-memory objects are actually backed by byte buffers containing the wire format. No need to compute offsets of class members (which technically violates the C++ standard though it works on all compilers).
Congratulations, I'm really happy for you, Jade, and the rest of the team.
I am sad about the projected lower development activity in Sandstorm in the coming months. You had a tremendous community/brand built up, which is very hard to do for decentralized web technology initiatives -- several independent chat groups I'm in are lamenting the news today. If you're aware of any groups discussing further development in the decentralization space, please let me know (email's in profile, or I will check this comment every hour or so).
Seriously great work so far.
Really....seemed like they massively downplayed to me.
What leads you to believe that CloudFlare was impressively fast and transparent in this case? Especially since statements from Project Zero seem to imply that they were anything but.
(Note: All of the above is based on my external observations, as I was not yet an employee nor did I have any internal access at the time.)
https://www.washingtonpost.com/blogs/blogpost/post/google-en...
Even someone as popular as Steve Yegge that probably could have told Google to GFY basically switched to PR speak the next day:
> Yegge wrote a mea culpa the next day and praised Google PR for not coming down on him. He took the post offline but let others keep their copies. And then he stopped ranting and presumably went back to work. If only some politicians could learn from his example. When you make a mistake, the more you talk about it, the longer the story lives.
I don't think anything he says has been a PR disaster and I think his POV is genuinely correct (that could be just because he is persuasive :P).
I think the truth is people want the white lie of PR speak because when you stop and try to be honest, many people will try to use it against you later so its safer to use the PR filter that is a white lie than be 100% transparent.
The researchers massively overplayed the impact and the public massively misunderstood the effect.
When time allows, I'd very much appreciate a reflection on the economic viability for innovative infrastructure software in general (hosted vs enterprise sales, also taking RethinkDB's demise and that of other "new stack" startups into account, etc.), and I'm sure others here will as well.
Of late, offerings that provide a hybrid approach to software seem to be emerging. Gravitational and Relicated are examples of toolsets for these types of modelss. Take logging as a service and the more traditional self hosted log solution. What would work best users would be one piece of software, which could run both on-premise and in the cloud on someone else's servers, or even someone else's service. On-prem, it might be managed by those who wrote it. Hosted, it might represent a segment of a larger market play for intelligent time series analytics and sold as a monthly revenue rollup.
Deploying these types of models will be interesting, but it's totally doable using some type of lightweight federation across services. Sandstorm represented a portion of the infrastructure needed to do simple deployments that felt like a hosted service. A service proving cryptocurrency payments for their APIs might be another piece of the puzzle.
I continue thinking about these new markets as new technology enables them.
Every VC-funded startup is premised implicitly on "full monopoly" as the endgame for exits that demand extremely high returns from investors. This worked really well from 2002 to 2016, and is still a really good model for a lot of software innovation, but we need to start seriously thinking about other funding models -- grants, not-for-profits, institutions like Harvard/MIT/Stanford, etc., to be involved in funding the next generation of web software technology.
I know this isn't a super popular opinion here :). I'm pretty confident in the reasoning, though. The solution to the sociopolitical problems Facebook unearths isn't "another Facebook", it's rigorous rethinking of the relationship between users, companies, data, and new software applications. Sandstorm was exactly the right next step, but unfortunately kentonv & co had to spend most of their time on enterprise sales, because this is the tail end of what the VC-funding-only ideology expects you to do if you want to enact widespread software innovations.
Another approach I think will work well: https://solid.mit.edu/
With more investment I think we could have made it work, but it turns out that once you run into any kind of "trouble" as a startup, getting further investment becomes incredibly hard -- and we were pretty bad at the fundraising game to start with (it is, essentially, sales, after all).
I don't think we have any particular evidence that Sandstorm wasn't viable as a technology or a business -- we just personally did not have the right skillset for the business model we chose. :/
That's how I predicted it would fail, too. You were at least able to figure out the problem. Many people will blame everything or everyone else. Enterprise sales is it's own beast that's much different from how things operate in the Valley or smaller shops. That's only half the problem. The other half is you usually have to integrate with systems designed not to integrate with 3rd parties easily. Lock-in loving systems. The healthcare startups are particularly being hit hard by this.
I keep thinking there's potential to get a business going that specializes in the sales and/or enterprise features at reasonable price. A public benefit or nonprofit company that forces the charges to be limited to something reasonable. They can help all these new companies bootstrap the enterprise-specific aspects of their goods for a price. What you think?
EDIT: disregard the above, that this was a careless misunderstanding of the title. TFA says "Sandstorm, for its part, remains an independent entity, and I won’t be working on it during my day job at Cloudflare. However, I will be working on Cap’n Proto."
If you didn't follow that link in the first paragraph and didn't hear about it already from some other source, https://sandstorm.io/news/2017-02-06-sandstorm-returning-to-...
IMHO this is all really good news, because people who were on the fence about Sandstorm before can feel more free to commit resources to developing it, without worrying that Sandstorm the company is going to come along and make decisions against their interest, or try to make their work proprietary, or leverage it for the company to make more profit at their expense. (Not that we thought those things would happen...)
They would certainly have less time to work on Sandstorm if they were not able to go right out and get full-time jobs, once it was determined Sandstorm the company was out of money! IMHO there is nothing but good news in these announcements.
If anyone is looking for alternatives: http://alternativeto.net/software/sandstorm-io/ . I think Cozy, Yunohost, Cloudron, arkOS are the closest alternatives (Bitnami is about 1-click images and DO/linode are about cloud servers). Would be great to hear about other alternatives...
Given that, and given that it's FOSS, then as long as the community is still interested in the project, the project should keep seeing development, regardless of the amount of time the original developers can put in. Even if that means a LibreOffice- or MariaDB-like fork.
I think of those listed projects only few are backed by for-profit entities (like sandstorm itself). Cozy in particular seems very well funded (https://www.crunchbase.com/organization/cozy-cloud#/entity).
I also think the alternatives link is a bit misleading because those projects aren't really direct alternatives. I have investigated a bit in this space (for my personal hosting):
* Cozy is simply for a personal server for email/contacts. This is more like ownCloud. It's not meant for hosting internet facing websites.
* Sandstorm is sort of trying to build a Google Docs style portal. People share these docs with each other. This is why it has a "frame" around every doc/app. It's also not meant for hosting internet facing websites (by this i mean, it is not optimized for hosting your personal public blog).
* Bitnami is 1-click images for opensource apps. I am guessing their main user base is people who want to install a stack quickly. It's a glorified docker pull or curl.
* Cloudron is about running apps on a server. This is automation (ssl, backup, install/update etc) of what you would be doing today, if you have to setup a server and install apps like rocket.chat, gitlab etc. Cloudron uses docker for the apps. Yunohost does the exact same thing without docker.
It's an Intranet portal, essentially—like the one you get from setting up Apple's Server.app. But it could certainly be used to host an app that builds an Internet-facing website.
For example, you could launch an authenticated-users-only Wordpress instance on Sandstorm, to allow your team to collaboratively create a website. Then you'd install the Simply Static plugin in that Wordpress instance, and use it to shove a static-slug copy of the site out to a public web-enabled S3 bucket or wherever else you like.
In fact, there's probably a good nascent architecture involving Sandstorm + Minio + nginx where you use Sandstorm on the intranet-side, as a multi-product Content Management System for a public-facing site.
We don't currently use the RPC framework specifically though only the serialization.
So instead of running a whole Gitlab server, you'd run a Sandstorm server that hosts Gitlab grains, each consisting of a single repository. Gitlab itself is no longer responsible for sharing and authentication, the Gitlab grain lets Sandstorm handle that. Apps need to be rewired to fit into this paradigm, but once they do, it feels like one coherent platform rather than a bunch of separate individually hosted apps that maybe you glue together somehow.
For Etherpad, instead of running an Etherpad server that hosts many notepads, you create an Etherpad grain that has just one shared pad. For a drawing app, the grain would be just one drawing. For a photo gallery app, maybe it would be just one gallery. (These are all real examples, these grains all exist and work as I'm describing.)
To clarify: does a "grain" just run in one place, or is each "grain" a cluster of containers that can be spread-scheduled between disparate regions (i.e. "the Oasis service" and "your own machines" at the same time)? And, if the latter, do the cluster nodes have awareness of their locations, and can take advantage of that?
I think the grain runs in one place, I think to use the container analog it would be one container, but as a point of clarification I believe that it is not using one of the container runtimes like docker or rkt that you will be familiar with, but uses its own sandboxing instead.
I have not tried running the newly opened up Oasis backend so I can't really speak toward how it works. My experience with a single server running Sandstorm has been good, painless automatic upgrades, I would expect from what I know about this team, that since it's their dogfood they made it taste good.