Yes, totally. In my definition the difference is that a stream processing system let's you define the frequency with which output is produced rather than forcing it to be "at the end of the data". This doesn't preclude blocking operations.
924 karma · joined June 15, 2009
I think in cases where you are running totally different computations in different systems the Lambda architecture may make a lot of sense.
However one assumption you may be making is that the stream processing system must be limited to non-blocking, in-memory computations like sketches. A common pattern for people using Samza is actually to accumulate a large window of data and then rank using a complex brute force algorithm that may take 5 mins or so to produce results.
One of the points I was hoping to make is that many of the limitations people think stream processing systems must have (e.g. can never block, can't process large windows of data, can't manage lots of state) have nothing to do with the stream processing model and are just weaknesses of the frameworks they have used.
So assuming the problem was equally prevalent at software startups in both countries would expect to hear about it a lot less from Germany.
:-)
Basically I think all this "it can't be measured" stuff is a bit dishonest. My wife is a high school teacher, and amongst her department everyone knows who is good and who isn't, same as any job. The only difference is that that knowledge can't possible have any effect on your career.
In any case, the point you are making about standardized testing is completely beside the point. No one who advocates using testing advocates using only testing for teacher evaluation. And no one advocates using the raw scores. Of course teachers with poor students in bad schools will have worse raw scores then teachers with rich students in good schools. The proposed measurement is virtually always some kind of "value added" score that attempts to control for these variables and measure the teachers impact.
You can argue this kind of evaluation is not fair (it is certainly less scientific then test scores), but the point is that on the whole it works. And teachers don't have it. They mostly cannot be fired, and receive compensation on a schedule based purely on years of tenure and education.
A good analogy would be if programmers were given tenure after two years at a company and then could not be fired regardless of productivity or any other offense other than gross misconduct. If the programmers then (unsurprisingly) resisted every attempt, no matter how small, to reward high performance and punish incompetence, we would have an equivalent situation.