Here is the current speaker list - https://www.hytradboi.com/#speakers
Bunch more in the pipeline that I'm excited about.
3,462 karma · joined November 17, 2008
jamie@scattered-thoughts.net
Bunch more in the pipeline that I'm excited about.
This is very much counter to the point I was making. I didn't find that learning haskell or ml was anything like learning math, especially the kind of math that I've seen people get use out of in day-to-day engineering, and I don't understand why this idea is so persistent.
If you don't know any math, it might be worth learning some of whatever kind of math is applicable in your field. It's probably not haskell.
> It does help to be more familiar with regexes and regular languages.
Regexes are certainly useful, but I don't think that learning a lot about automata theory is a particularly efficient way to get better at using regexes in practice, compared to eg doing a bunch of drills on https://www.executeprogram.com/courses/regexes
I tried as much as possible to only write things where I could think of multiple concrete examples in my own experience.
https://www.airtable.com/product/views https://www.airtable.com/product/automations https://www.airtable.com/integrations
It's getting to the point where it can handle most of the internal software needs of a small-medium company and also has a ton of javascript hooks for covering that last 20%.
These days I'm writing at https://scattered-thoughts.net/ and working on https://github.com/jamii/dida and https://github.com/jamii/imp. Same ideals, but backed by many more years of ~~failure~~ experience.
Also still complaining about things https://scattered-thoughts.net/writing/against-sql and breaking things https://scattered-thoughts.net/writing/internal-consistency-...
I suspect it's also going to be fragile. The next time we hit a recession or, god forbid, another pandemic, sponsorships will probably be the first thing to go.
But it does help fill gaps. There are a lot of things I've wanted to code, study or write about that I think people will get value from but that I could not get any employer to pay for. So I really appreciate being able to work on things just because they're valuable, rather than because there is a way to capture the value and make a profit.
On the low temporal locality side, the only consistent system I've seen so far is differential dataflow (and materialize) which does internally track which results are consistent and gives you the option to wait for consistent results in the output.
It didn't make it into https://scattered-thoughts.net/writing/internal-consistency-... because it has severe limitations for low temporal locality operations:
* As of Spark 2.4, you can use joins only when the query is in Append output mode. Other output modes are not yet supported.
* As of Spark 2.4, you cannot use other non-map-like operations before joins. Here are a few examples of what cannot be used.
* Cannot use streaming aggregations before joins.
* Cannot use mapGroupsWithState and flatMapGroupsWithState in Update mode before joins.
* There are a few DataFrame/Dataset operations that are not supported with streaming DataFrames/Datasets. Some of them are as follows.
* Multiple streaming aggregations (i.e. a chain of aggregations on a streaming DF) are not yet supported on streaming Datasets.
* Limit and take the first N rows are not supported on streaming Datasets.
* Distinct operations on streaming Datasets are not supported.
* Sorting operations are supported on streaming Datasets only after an aggregation and in Complete Output Mode.
* Few types of outer joins on streaming Datasets are not supported.
The rest of spark doesn't really fit the post at all afaict - there is no streaming or incremental update, just batch stuff.
Yes, I think so. If you want to be able to handle out-of-order data, I don't think there is a way to garbage collect old time periods and still produce correct results unless you stop accepting new data for those time periods.
With the differential dataflow code I wrote for this post, that cutoff point for gc is also coupled to when you can first emit results, creating a sharp tradeoff between latency of downstream results and how out-of-order upstream data is allowed to be. With bitemporal timestamps or multiple watermarks those are decoupled, so you can emit provisional results and correct them later without giving up internal consistency, and also make a totally separate decision about when to garbage collect. That doesn't remove the problem entirely, but it means that you can make decisions about latency and decisions about gc separately instead of them being controlled by one variable.
With flink datastreams there are a lot of different levers to pull. The code that Vasia contributed recently waits for watermarks like differential dataflow. There is also a notion of triggers that emit early results from windowed operators, but these are not coordinated across different operators so they can result in internal inconsistency.
The flink table api, as far as I can tell, mostly just ignores event-time outside of windowed aggregates. So it doesn't have to confront these tradeoffs.
> the broader context is building a view over transaction decisions made by another system(s).
Most the time this is fine. If upstream is:
* a transactional database, we get ordered inputs * another system that has a notion of watermarks or liveness (eg spanner), we can use their guarantees to decide when to gc * a bunch of phones or a sensor network (eg mobile analytics), then we pick a cutoff that balance resource usage with data quality, and it's fine that we're not fully consistent with upstream because we're never compared against it
The hard case would be when your upstream system doesn't have any timing guarantees AND you need to be fully consistent with it AND you need to handle data in event-time order. I can't think of any examples like that of the top of head. I think the answer would probably have to be either buy a lot of ram or to soft-gc old state - write it out to cheap slow storage and hope that really old inputs don't come in often enough to hurt performance.
---
This has been an interesting conversation and I'll probably try to write this up to clarify my thinking. I'm not on hn often though, so if you want to continue feel free to email jamie@scattered-thoughts.net
> What is a valid subset to consider?
For these purposes, any subset.
> What is "correct output"?
I absolutely agree that deciding whether an output is correct requires actually defining the intended semantics of the streaming program. I don't believe that exists for flink or ksqldb. (It does for differential dataflow, to some degree - https://link.springer.com/content/pdf/10.1007/978-3-662-4667...). If the outcome of this testing was that streaming systems gave a clear definition of their intended semantics which allowed for this degree of error, I think that would be excellent. That's what I was getting at towards the end of the post: "Ideally the behavior would be sufficiently well defined that it's possible to write tests that examine intermediate outputs, because those are the outputs that real deployments will experience."
But in the meantime, for the deterministic sql queries here I think "run the same computation in a consistent batch system and compare the output" is a highly reasonable set of semantics by which we can judge correctness.
This framing is laid out more explicitly in intro to the previous article - https://scattered-thoughts.net/writing/an-opinionated-map-of...
> Why would the example be internally inconsistent (per the definition)? The out of date debits data is a subset of the input so far, is it not?
The debits data is using one subset and the credits data is using another. So if we ask "what subset of the inputs is balance computed from" the answer is sort of this one and sort of that one. The end result is that there is no single subset of the data that could produce those results in a consistent system.
> the goal is not to permit arbitrary subsets of data but to restrict the considered subsets of data to a consistent selection.
Yes. The goal is that the final output is computed from a well-defined subset of the data, not from a mishmash of different subsets. I think we're roughly in agreement on the intended behavior and just not on the language used to describe it?
> systems that implement full/immediate consistency.
I don't know what full consistency would mean for a streaming system if not internal consistency + eventual consistency. If you have both, you can send an input and then just block until you see the output for the corresponding timestamp.
Part of the trouble is that all our language for describing consistency comes from read-write transactions in databases and doesn't translate directly into streaming systems. The closest analogue is probably snapshot isolation - https://jepsen.io/consistency/models/snapshot-isolation - but you can't just port the definitions over to a system that doesn't have a concept of transactions.
> If the watchType is set to FileProcessingMode.PROCESS_CONTINUOUSLY, when a file is modified, its contents are re-processed entirely. This can break the “exactly-once” semantics, as appending data at the end of a file will lead to all its contents being re-processed.
You can't join on updating_tx because the credits and debits per account are disjoint sets of transactions - that join will never produce output.
I did try something similar with timestamps - https://github.com/jamii/streaming-consistency/blob/main/fli.... This is also wrong (because the timestamps don't have to match between credits and debits) but it at least produces output. It had a very similar error distribution to the original.
Plus the join is only one of the problems here - the sum in `total` also needs to at minimum process all the balance updates from a single transaction atomically.
> The only work I've seen on this is
Honestly, my brain in the morning...
Sort of. When you make a single read of a single key at the end of a streaming topology, that's equivalent to reading a all the inputs and doing the computation yourself. Internal consistency means that every path through the graph back to the the inputs should be reading the same set of inputs. Things go wrong when one path has processed more inputs than another, or when a single input produces multiple internal updates and those updates aren't processed atomically.
> isn't it possible for subsequent reads to lock into the maximum transaction id of the first read
Your example would still allow credits to have read more inputs than debits, but it's on the right track.
In differential dataflow aggregates wait until they see a watermark before emitting output, so the sum in `balance` will wait until the watermark for eg time 7 before emitting the sum for time 7. That allows the output from the join to settle.
Of course doing that efficiently is non-trivial...
That's true in that the table api itself is built on top of the datastream api. But to run the example you'd have to implement your own joins and retraction-aware aggregates and then we'd be testing the consistency properties of that implementation, not of the datastream api itself, and we might as well test a mature implementation like the table api instead.
It seems that this would still leave the problem where each transactions causes two deletes and two inserts to `balance` and then `total` sums those one by one? These errors are very transient but they still cause at least 3/4 outputs to be incorrect.
> tumbling event-time windows
Will this work for the join? For a given account, the last update to each of credits and debits may have arbitrarily disparate event times and transaction ids. It doesn't seem like there is window you could set that would be guaranteed to connect them?
That's true in that the table api itself is built on top of the datastream api. But to run the example you'd have to implement your own joins and retraction-aware aggregates and then we'd be testing the consistency properties of that implementation, not of the datastream api itself, and we might as well test a mature implementation like the table api instead.
> The file source can be used for testing streaming behavior, for example by using a directory and using a PROCESS_CONTINUOUSELY watch type.
Ah, I was only looking at the table api version (https://ci.apache.org/projects/flink/flink-docs-release-1.12...) which doesn't mention that option. I guess I could make stream and turn it into a table?
> This is expected, that's how Flink deals with late data...
I should clarify in the article that I was surprised not because it's wrong, but because it's very different from the behavior that I'm used to.
> Also, it seems very odd to me that a watermark of 5s is used while the expected out-of-orderness is 10s. Why did the author choose to do that?
To explore how late data is treated.
There are some plausible sources of inconsistency that from late data handling that I didn't get to demonstrate in the article. For example, it sounds like even without allowed_lateness if two different windowed operations had different window sizes they might drop different subsets of late data, which could causes inconsistency between their outputs.
> The standard library includes a set of allocators which don't reuse allocations, preventing use-after-free, and which catch double-free. I'm not clear yet on how high the runtime and memory overhead are though, which will dictate when it is practical to use these.
I didn't include it in the table because I'm not yet convinced that the overhead will be low enough that people will actually ship software using those allocators. (All the zig programs I've written so far use the libc allocator and are definitely susceptible to UAF)
Perhaps I'll spend some time measuring it this week and post an update.
This is checked at runtime in ReleaseSafe.
jamie@machine:~$ cat test.zig
pub fn main() void {
var x: usize = 3;
_ = @intToPtr(*u32, x - 3).*;
}
jamie@machine:~$ zig run test.zig -OReleaseSafe
cast causes pointer to be null
Aborted (core dumped)
If the GPA is practical to use in production, that will be a different story. But it doesn't sound like it's there yet.
Accessing a memory location with one type as if it's another. In C or C++ it's usually because you accessed a union without checking a tag somewhere else.
One classic way this happens is in an interpreter where you have a big enum for all the different possible data types:
https://github.com/MaterializeInc/materialize/blob/1b9b3cfab...
And then in various builtin functions you expect particular types:
https://github.com/MaterializeInc/materialize/blob/main/src/...
In rust and zig this code will produce a runtime error if you screw up, but in c it's easy to forget to check the tag and then you get UB. Similarly unwrap is checked in rust and zig but the equivalent in c - dereferencing a pointer that you are pretty sure is not null - is not.
Of course rust and zig both have support for c-style unions too, but they're not the first thing people reach for.