You would need to contact us to buy an enterprise license. (Or wait four years for the license to convert to Apache 2.)
1,289 karma · joined December 12, 2012
github.com/benesch
nikhil.benesch@gmail.com
[ my public key: https://keybase.io/benesch; my proof: https://keybase.io/benesch/sigs/XTirUcJbA7dNwrQRe_-fze5YiXsPDPIs2sMq862fyDE ]
Tapping into the consistency-preserving features in Materialize is a bit complicated at the moment, but we're actively working on improving the integration of our consistency infrastructure for MySQL and PostgreSQL upstreams.
The biggest difference is one of intended use. Noria is, first and foremost, a research prototype, intended to explore new ideas in systems research. Materialize, by contrast, is intended to be a rock-solid piece of production infrastructure. (Much of the interesting research, in timely and differential dataflow, is already done.) We've invested a good bit in supporting the thornier bits of SQL, like full joins, nested subqueries, correlated subqueries, variable-precision decimals, and so on. Noria's support for SQL is less extensive; I think the decisions have been guided mostly by what's necessary to run its lobste.rs and HotCRP benchmarks. Make no mistake: Noria is an impressive piece of engineering, but, as an enterprise looking to deploy Noria, there's no one you can pay for support or to implement feature requests.
One area where Noria shines is in partial materialization. Details are in the Noria paper [1], but the tl;dr is that Noria has a lot of smarts around automatically materializing only the subset of the view that is actually accessed, while presently Materialize requires that you explicitly declare what subsets to materialize. We have some plans for how to bring these smarts to Materialize, but we haven't implemented them yet.
Also worth noting is that Materialize's underlying dataflow engine, differential dataflow, has the ability to support iterative computation, while Noria's engine requires an acyclic dataflow graph. We don't yet expose this power in Materialize, but will soon. Put another way: `WITH RECURSIVE` queries are a real and near possibility in Materialize, while (as I understand it) `WITH RECURSIVE` queries would require substantial retooling of Noria's underlying dataflow engine.
One of the creators of Noria, Jon Gjengset, did an interview on Noria [2] that covered some of differences between Noria and differential dataflow from his perspective, which I highly recommend you check out as well!
[0]: https://twitter.com/frankmcsherry/status/1056957760435376129...
[1]: https://jon.tsp.io/papers/osdi18-noria.pdf
[2]: https://notamonadtutorial.com/interview-with-norias-creator-...
Window functions are a particular favorite of mine, but we haven’t seen much customer demand for them yet, so they haven’t been officially scheduled on the roadmap. They require some finesse to support in a streaming system, as you have to reconstruct the potentially large window whenever you receive new data. Probably some interesting research to be done here, or at least some interesting blog posts from Frank.
Please feel free to file issues about any of these functions that you’d like to see support for! We especially love seeing sample queries from real pipelines.
We definitely have some performance engineering work to do in Materialize, but don’t let the lack of vectorization scare you off. It’s just not as important for a streaming engine.
SQL is lacking in elegance but abundant in popularity. Building the SQL translation layer has been a fun exercise in bridging the two worlds.
I think we're actually arguing the same points here. It's not that every use case needs single-digit millisecond latencies! There are plenty of use cases that are satisfied by batch jobs running every hour or every night.
But when you do need real-time processing, the current infrastructure is insufficient. When you do need single-digit latency, running your batch jobs every second, or every millisecond, is computationally infeasible. What you need is a reactive, streaming infrastructure that's as powerful as your existing batch infrastructure. Existing streaming infrastructure requires you to make tradeoffs on consistency, computational expressiveness, or both; we're rapidly evolving Materialize so that you don't need to compromise on either point.
And once you have streaming data warehouse in place for the use cases that really demand the single-digit latencies, you might as well plug your analysts and data scientists into that same warehouse, so you're not maintaining two separate data warehouses. That's what we mean by ideal: not only does it work for the systems with real-time requirements, but it works just as well for the humans with looser requirements.
To give you an example, let me respond to this point directly:
> Secondly, almost all data is useless in its raw form. The analysts had to perform ELT jobs on their data in the warehouse to clean, dedupe, aggregate, and project their business rules on that data. These functions often require the database to scan over historical data to produce the new materializations of that data. So even if we could get the data in the warehouse in sub-minute latency, the jobs to transform that data ran every 5 minutes.
The idea is that you would have your analysts write these ETL pipelines directly in Materialize. If you can express the cleaning/de-duplication/aggregation/projection in SQL, Materialize can incrementally maintain it for you. I'm familiar with a fair few ETL pipelines that are just SQL, though there are some transformations that are awkward to express in SQL. Down the road we might expose something closer to the raw differential dataflow API [0] for power users.
[0]: https://github.com/TimelyDataflow/differential-dataflow
Efficiently maintaining views over arbitrarily complex computations is one of the two hard problems in computer science for a reason [0].
I'm not sure I'd say it's "more OLAP." ksqlDB is about as OLAP as it gets, considering it doesn't support any sort of transactions or consistency. We think Materialize is quite a bit more powerful than what ksqlDB offers, thanks to the underlying technologies (timely/differential). For example, our joins are proper SQL joins, and don't require you to reason about the complicated and confusing difference between a stream and a table (https://docs.ksqldb.io/en/latest/developer-guide/joins/join-...). We also have preliminary support for maintaining the consistency properties of upstream OLTP data sources, and we'll be rolling out a more complete story here shortly.
> Seems that it's perhaps lacking the richness of how ksqldb uses Kafka Connectors (sinks and sources), but I don't see any reason you couldn't use Materialize in conjunction with ksqldb.
Is there something in particular about ksqlDB connectors that we don't seem to support? Our CREATE SOURCE command is quite powerful: https://materialize.io/docs/sql/create-source/.
> What connectors (sinks and sources) do you have or plan to develop? Seems like it's mostly Kafka in and out at the moment.
We already support file sources in a variety of formats, and support for Amazon Kinesis is on the short-term roadmap: https://github.com/MaterializeInc/materialize/issues/1239.
Are there other connector types you'd like to see?
> Can I snapshot and resume from the stream? Or do I need to rehydrate to re-establish state?
At the moment you can't snapshot and resume, but support for this is planned.
Removing a nullable column will also work fine along the same lines: that column will simply be filled in with NULL in Materialize for all new data. Removing a non-nullable column is backwards-incompatible, of course, and so Materialize will be unable to process any additional events for that topic, and so you'd have to recreate the source and restart from the beginning.
For example, consider a map combinator. The output diff is just the input diff with a function applied to the data. More formally, for every incoming (data, time, diff) triple, an output triple (f(data), time, diff) is produced.
For a filter combinator, the output is (data, time, diff) if f(data) is true, or (data, time, 0) if f(data) is false.
A computation that wants to filter and map some data then just requires piping together these operators.
Things get interesting when you want to aggregate and join data, but Frank’s blog posts and documentation explain how you build that in a dataflow system far better than I can here.
If you get into it, the library itself is well documented on docs.rs and there’s an active Gitter channel.
(Disclaimer: I work with Frank at his company, Materialize.)
[0]: https://timelydataflow.github.io/differential-dataflow/
[1]: http://sigops.org/s/conferences/sosp/2013/papers/p439-murray...
Unlike in C, there are no macros like __FILE__ and __LINE__, so there is no alternative to runtime.Callers (short of preprocessing your Go source code).
The reason is that Go's standard library provides functions that allow retrieving a backtrace and symbolicating that backtrace at runtime:
* https://golang.org/pkg/runtime/?m=all#Callers
* https://golang.org/pkg/runtime/?m=all#CallersFrames
Unlike in C or C++, where you can link in something like libbacktrace [0] to get a best-effort backtrace, those Go functions are guaranteed to be correct, even when functions have been inlined. This is no small feat, and indeed programs compiled with gccgo will often be incorrect because libbacktrace doesn't always get things right when functions have been inlined.
I can't think of another language that splits its official web presence across separate domains like this. Rust (rust-lang.org), Python (python.org), Ruby (ruby-lang.org) all have sections dedicated to marketing/eduction (usually "Community" or "Success Stories" tabs), as well as hosting the language reference, documentation, and downloads.
The result sure is confusing. Why aren't the blog and playground hosted on go.dev, for example? I can only imagine this is the result of internal politics at Google; it sure smells like the Go team engineers didn't want the growing Go marketing division to corrupt golang.org.
I don't mean to imply that there's a conspiracy here. What I mean is that I think the Plan 9 heritage is clouding strategic decisions around the Go toolchain. What may have been the right decision to get a new language off the ground is not necessarily the right decision once the language is widely popular and stable.
> Migrating Go today to LLVM would of course be a big, time-consuming zero-velocity project; you'd want to be really certain that the payoff would be worth the effort.
A big project, yes, but it's happening! [3] If a few engineers working on gollvm for a year or two could improve gollvm to the point that it made the average Go program run 20% faster, I'd think that would absolutely be worth it to Google.
> GCC is not an easy project to deal with, either. For decades, its internal intermediate representation was undocumented and intentionally obfuscated to ensure FSF/GNU control over backends.
Very true in general, but the Go team has been lucky in that GCC core maintainer Ian Taylor has been a member since the early days. Gccgo has been a spec-conformant Go compiler since 2012 [2], and so nearly all of the GCC-integration bits have been in place for seven years. As a result, it's far more a matter of staffing the project so that the Go frontend's inliner, garbage collector, and escape analysis can reach parity with gc, rather than dealing with GCC/FSF politics.
> You can browse the original commit here. That stuff has all been rewritten in Go.
Yeah, I'm familiar with the lineage. I don't know that I'd say that it's been rewritten in Go, though, as the toolchain and runtime were converted fairly automatically with a "c2go" tool that Russ Cox wrote [0] around the Go 1.5 release. Have you looked at the resulting code much? Some of it has been rewritten to be idiomatic Go, but a lot of it is still very clearly C code that has been automatically translated [1]. See also the fact that go/src/runtime/runtime.go, go/src/runtime/runtime1.go, and go/src/runtime/runtime2.go all exist—a consequence of the fact that runtime.go, runtime.c, and runtime.h all existed in Go 1.4 [2].
[0]: https://github.com/rsc/c2go
[1]: https://github.com/golang/go/blob/7b294cdd8df0a9523010f6ffc8...
I suspect the community is uninterested because it’s hard to be interested in compilers that a) compile slower than gc, and b) produce slower code than gc. That's a worse compiler on all fronts!
For gccgo or gollvm to be useful, they need to provide some benefit. I suspect we'll see (b) fixed within a year. GCC/LLVM have far more optimizations than gc, and so it's mostly a matter of plumbing enough information from the Go frontend into the LLVM optimizer to unleash its full power.
I don't expect we'll see (a) fixed, unless something changes at Google.
> Also, Google also already invests a massive amount of resource into LLVM. In fact, the principal author of LLVM and Clang works at Google. But even when he was at Apple they were already shoveling resource into the project.
Yes, but Chris Lattner is not actively working on speeding up LLVM. His big project lately has been supporting TensorFlow via MLIR.
In short, the reasons gccgo produces worse code than gc are the result of fairly basic optimizations that are/were missing. For example, open-coding string slicing [0] and string equality [1] is enough to close the gap in the TimeParse benchmark in the blog post above. In fact, with those changes, gccgo actually produces better code than gc, because the full power of GCC can be unleashed on the IR, and GCC has more aggressive optimizations than gc.
> This isn't to say you should take everything anyone with more experience says at blind faith, simply that dismissing without understanding is counter to a growth mentality.
Quite honestly, I think you're more guilty of dismissing without understanding than I am. It's true that I haven't been programming for decades, but I've spent two years dealing with the Go toolchain's shortcomings on a large Go project, and six weeks specifically on a passion project to improve gccgo. Of course it's possible that I'm wrong! But I've spent a while thinking about it, and often just as relevant as experience is a fresh, outside perspective. It's easy for a team to get stuck in tunnel vision or groupthink.
[0]: https://github.com/golang/gofrontend/commit/62e3a8cc0a862b0a... [1]: https://github.com/golang/gofrontend/commit/89b442a0100286ee...
Sorry, but I think you’re still missing my point. It’s well documented that the original creators of Go did not want to build their language on top of LLVM, and did not like C++. That’s totally fine! Of course you have to work on things you like.
But Go is so stable, and has been for more than half a decade. That means Google could spin up a brand new team, entirely separate if they must, of folks who like both LLVM and Go, to build out an LLVM frontend for Go. It’s not like Go is a fast-moving target. There is a very stable spec.
Gccgo is proof that it is possible to have a GCC frontend for Go without too much work. Ian Taylor has been maintaining Gccgo for about ten years, working what seems to be half time. Imagine what could be done if there were a few more folks actively working on improving gccgo, rather than just keeping parity with gc.
> I do not see how a generic framework could be made as fast as custom build stack.
I don’t see why not. It’s mostly a matter of introducing ways of disabling the expensive bits when compiling/linking a simpler language like Go. Yes, it’s probably somewhat slower to develop, but everyone benefits from the work, and you also offload much of the burden of maintaining a compiler/linker for a half dozen different platforms to the LLVM team collectively.
I did learn enough in those six weeks to feel comfortable asserting that there is incredible potential in gccgo/gollvm, and that I think Google is making a mistake by continuing to invest so heavily in the gc toolchain.
[0]: http://meltware.com/2019/01/16/gccgo-benchmarks-2019.html
Right, you need to specifically throw money and people at the problem of making LLVM faster, not just at LLVM in general. Neither Swift nor Objective-C have "fast compiles" as part of their pitch. Much of the work on LLVM goes into producing the highest quality object code possible, which is a goal often at odds with compiling quickly, and part of the reason the choose-your-optimization-level flag (-O) exists, though -O0 compiles are still not fast enough.
> In hindsight the Go team made the right call to use their own toolchain.
No, we don't have the benefit of hindsight yet. We don't know what could have been if the resources that had been spent on the Go toolchain had been spent on LLVM instead.
If five highly-qualified engineers spent five years trying to speed up gollvm compiles without success, we'd have strong evidence that something about LLVM prohibits the fast compiles that are possible with the gc toolchain. But that's not the situation.
I agree that enthusiasm is important! And indeed, for the Go creators, their particular leanings might have been such that they couldn't get excited about building an LLVM/GCC frontend, and adapting the Plan 9 toolchain is literally the only way those three could have Go gotten off the ground. As a member of the Go team, you'd certainly know better than I.
But Go is long past a personal passion project. Go is over ten years old. Go likely has over a million developers [0]. Go 1.0 has been stable for about seven years, and the first meaningful changes to the language are just now being talked about. In my opinion, it is several years past due for Google to start investing seriously in a Go toolchain based on a mature compiler stack.
I realize the audacity of this claim and I don't make it lightly. But if I had the money to spend on a team of developers, I would spend it making llvm-as and lld fast enough and stable enough to be Go's assembler and linker, and abandon the custom Plan 9 ones.
> It never would have happened. How do you motivate people whose principal frustration is the state of C++ to work on a large C++ codebase?
Well, for one, once the language gets off the ground, you can write the frontend in the new language. Rustc manages to be almost entirely Rust, for example.
> Heterogeneity is a huge benefit to any ecosystem. Improving existing things is great, but building new things is also very important.
I agree, and I think Go is an interesting contribution to the P/L landscape—essentially it proved that stripping away a good deal of complexity (generics, inheritance, etc.) results in a very useful, highly productive language. But I don't think Go's custom assembler and linker are meaningfully contributing to the ecosystem. They're useful presently in that they improve Go developers' productivity with ultra-fast builds, but they're not suitable for use by anything but Go. Improvements to Go's linker and assembler benefit only Go. Improvements to lld or gold can benefit practically everyone using a compiled language.
In many ways Go seems like an excuse for Google to fund the continued development of Plan 9. Three of the five most influential people on the Go team (Ken Thompson, Rob Pike, and Russ Cox) were heavily involved in Plan 9. And it shows. Go's toolchain is a direct descendant of the Plan 9 toolchain; in fact, the Go language is really just an evolution of the special C dialect that Plan 9 used [1]. Indeed, for a while, the Go compiler was written in this special dialect of C, and so building Go required building a C compiler (!) that could compile this custom dialect of C, and using that to compile the Go compiler [2].
By all rights, Plan 9 was an interesting research project, and seems well loved by those familiar with it. (I'm not personally familiar; it was well before my time.) But it never took off. What we ended up with is Linux, macOS, and Windows.
Go very much wants to be Plan 9. Sure, it's not a full-fledged operating system. But it's a linker, assembler, compiler, binutils, and scheduler. All it asks of the host system is memory management, networking, and filesystem support, and it will happily replace your system's DNS resolution with a pure Go version if you ask it to [3]. I wouldn't be surprised if Go ships its own TCP/IP stack someday [4].
This is, in my opinion, craziness. What other language ships its own assembler?! [5] To make matters worse, the assembly syntax is largely undocumented, and what is documented are the strange, unnecessary quirks, like
> Instructions, registers, and assembler directives are always in UPPER CASE to remind you that assembly programming is a fraught endeavor. (Exception: the g register renaming on ARM.)
> In the general case, the frame size is followed by an argument size, separated by a minus sign. (It's not a subtraction, just idiosyncratic syntax.)
> In Go object files and binaries, the full name of a symbol is the package path followed by a period and the symbol name: fmt.Printf or math/rand.Int. Because the assembler's parser treats period and slash as punctuation, those strings cannot be used directly as identifier names. Instead, the assembler allows the middle dot character U+00B7 and the division slash U+2215 in identifiers and rewrites them to plain period and slash.
The excuse for the custom toolchain has always been twofold, that a) LLVM is too slow, and fast compiles are one of Go's main features, and b) that the core team was too unfamiliar with GCC/LLVM, at least in the early days, and attempting to build Go on top of LLVM would have slowed the speed of innovation to a degree that Go might not exist [6].
I've always been skeptical of argument (b). After all, one of Go's creators literally won a Turing award, as this document not-so-subtly mentions. I'm quite sure they could have figured out how to build an LLVM frontend, given the desire. Rust, for example, is quite a bit more complicated than Go, and Mozilla's developers have had no trouble integrating with LLVM. I suspect the real reason was that hacking on the Plan 9 toolchain was more fun and more familiar—which is a very valid personal reason to work on something! But it doesn't mean it was the right strategic decision.
I will say that (a) is valid. I recently switched from writing Go to writing Rust, and I miss the compile times of Go desperately.
That said—and this is what I can't get past—the state of compilers would be much better off if the folks on the Go team had invested more in improving the compile and link times of LLVM or GCC. Every improvement to lld wouldn't just speed up compiles for Go; it would speed up compiles for C, C++, Swift, Rust, Fortran, Kotlin, and anything else with an LLVM frontend.
In the last year or so, the gollvm project [7] (which is exactly what you'd expect–a Go frontend for LLVM) has seen some very active development, and I'm following along excitedly. Unfortunately I still can't quite tell whether it's Than McIntosh's 20% time project or an actual staffed project of Google's, albeit a small time one. (There are really only two committers, Than and Cherry Zhang.) There are so many optimizations that will likely never be added to gc, like a register-based calling convention [8] and autovectorization, that you essentially get for "free" (i.e., with a bit of plumbing from the frontend) with a mature toolchain like LLVM.
There are not many folks who have the knowledge and expertise to work on compilers and linkers these days, and those that do can command high salaries. Google is in the luxurious position of being able to afford many dozens of these people. I just wish that someone with the power to effect change at Google would realize that the priorities are backwards. gccgo/gollvm are where the primary investment should be occurring, and the gc toolchain should be a side project that makes debug builds fast... not the production compiler, where the quality of the object code is the primary objective.
[0]: https://dave.cheney.net/2013/10/15/how-does-the-go-build-com...
[1]: http://doc.cat-v.org/plan_9/programming/c_programming_in_pla...
[2]: https://docs.google.com/document/d/1P3BLR31VA8cvLJLfMibSuTdw...
[3]: https://golang.org/pkg/net/
[4]: https://github.com/google/netstack
[5]: https://golang.org/doc/asm
[6]: https://golang.org/doc/faq#What_compiler_technology_is_used_...
(Atomic broadcast can be reduced to consensus, but frames the problem differently.)
Indeed. What you're describing is one of the main motivations for Raft. Paxos showed that distributed consensus was mathematically sound, but did little to guide implementors in actually building such a system. Raft is not a fundamentally new consensus algorithm; just an incremental improvement that formalizes many of the improvements that you needed to make to Paxos anyway, like membership changes, log compaction, and multi-decree support from the get go.
If you're interested, the Raft paper is quite readable and goes into this in detail. [0]
> The main problem I see is that in most consensus systems, any member can be nominated as leader. But if the member was inducted during a partition event - which one could do if the partition were long duration - then nominations for new leaders will go out. What happens if the new member gets elected? How do the partitioned machines find that leader when they return? How do the partitioned machines find that leader when they return?
This isn't actually the tricky bit, as it turns out. Communication flows from the leader to the other nodes, so the leader, even if it's a newly-inducted node, will initiate the connections to the partitioned nodes when the partition resolves. (The leader necessarily knows the addresses of the partitioned nodes, because the leader knows about all committed entries, and the identities of all the nodes in the cluster are committed into the Raft log.)
Again, the Raft paper does a great job explaining cluster mebership changes—much better than I can!