potatoyogurt

Yes, that is definitely a downside for these tests. The worst is when the text of some exception is printed and it includes line numbers. It does still require some discipline to think about what you're printing and avoid output that will be very noisy. This problem is mitigated quite a bit by the ease of accepting changes when these tests fail for obviously nonsense reasons though (just hit a couple buttons in an emacs buffer).

A very small number of people trade by voice on trading floors and they are generally professionals. This is categorically different from phone apps that reach into every ordinary person's living room. No one is advocating for a return to pre-electronic markets.

Derivatives have underlyings. An option is the right to purchase or sell a stock (or something else); a future is the right/obligation to accept delivery of something at a future time. While the payout curve is more complicated than straightforward ownership, they derive their value from their relation to some underlying assets.

What did the US do to yogurt?

If you don't allocate, you won't garbage collect. GC pauses would be really bad, but you can avoid them, or at least avoid them happening at bad times, if you're careful. It does requires knowing in some detail how Ocaml manages memory.

I think maybe you're getting the wrong impression about the tone I intend (my fault, I am not putting a lot of effort into tone or careful writing). I realize that what I wrote can be read very passive aggressively. I meant it more just as acknowledging that I made a dumb mistake and chiding myself for it. In person, that would have been clearer.

This week has been busy and stressful and I wrote both comments quickly and off the cuff. It's not that I can't tell that acknowledged is a passive participle, it's just that I didn't even look at it because I was pattern matching against common mistakes I've seen people make. It was only afterwards that I noticed that the parent comment specifically pointed to the word "acknowledged."

> I find this attitude troubling in combination with the argument made all over this thread that you can tell the people who say not to use passives are full of it because they can't even identify a passive when they're looking at one. (And you're at least flirting with that argument in your comment above.)

This is not what I'm saying, and it's not the argument I would make if I were actually making an argument. I said very few words, and not very carefully chosen ones, and you're putting words into my mouth. I made no case for or against passive voice. I just thought I saw a mistake briefly and I was trying to point it out.

okay, you're right, i wasn't paying attention. shame on me. Still, I think it is a stretch to point to the use of this passive participle as the reason why this sentence works or doesn't work.

While I agree with the general point that basic and direct sentence structure is not always the right choice, there is no passive voice in the sentence you quoted.

"it is a truth universally acknowledged" -> impersonal construction

"a single man... must be in want of a wife" -> this is active voice, man is still the subject of the sentence and be is the the only verb. "Want" is a noun. See https://www.dictionary.com/browse/want, definition 14.

here is a french press for $15. It can arrive on thursday. https://www.amazon.com/dp/B07HC22KQH/ref=sbl_dpx_B00008XEWG_...

Is google not giving its employees monitors during this time? I find that surprising but also you could just go to Best Buy.

I wonder as well. I did a project with him when I was a teenager and we didn't entirely get along. He was a very talented guy, though, and I wasn't entirely fair to him. I hope he's doing all right.

I don't get it, I don't understand how the author of the article has a PhD if this is what he thinks. Like,

> Why do advanced-math classes bother with proofs almost no student can follow?

Because that literally _is_ mathematics, and the point of working through them is because students can actually understand them and learn to do the same thing with new problems. How does he expect anyone to create or discover anything new with just arithmetic 101?

There's a kernel of a good argument in that signaling probably is one of the main values that students get from a college degree, but it's overstated to such a degree that it crosses over into idiocy.

Thanks for clarifying! I was 90% just joking, but I wasn't actually aware that it had such a long history.

Did you read his link? It directly addresses your point. I will summarize and add some context, since I don't know if you read Russian. The word мир in Russian has two meanings, "peace" and something that is translated in different contexts more like "the world" or "society." These were once different words with different spellings, but during the Soviet Union, authorities decided that some letters had to be purged because they were naughty, so orthographic reforms were carried out that standardized some spellings and eliminated a few letters. So the previously distinguishable words "мир"(peace) and мір (the world/society) became orthographically indistinguishable. However, since Tolstoy published prior to the Soviet Union, you can tell what the likely intent was by how the title was spelled in pre-revolutionary publications. In almost all cases, it was spelled мир. There is just one instance when it was spelled мiр, and it seems likely that it was a misprint.

That is basically LiveRamp's business model. There are other companies who do this as well. See https://en.wikipedia.org/wiki/Data_onboarding

Unless you're also going to disclose all your cookie ids, mobile device ids, connected tv ids, transaction history, income/demographics/consumer preferences, etc. etc., there will still be a lot of room for adtech companies to gather information about you. And even if you do, there is still attribution of purchases to impressions or clicks, audience segmentation, ad retargeting, and many other lucrative areas for companies to trade your data.

Why do you think Java is so much harder to refactor? Java and python are not that much different, and Java has static types which aid refactoring as well as a really strong tooling ecosystem. I haven't worked with large python codebases, but it hasn't really been my experience that refactoring Java is unusually hard, except insofar as average code quality among projects written in Java is probably lower.

Yeah, inflation is not negative, so it doesn't seem reasonable to say that the value of a dollar is rising.

hah, you're right, I definitely am looking at it with distributed system goggles on. With just one machine, yes, it will probably make sense to fully process one job before moving onto another (although there are some cases where this may not be true, such as if you're limited by having to wait for external APIs. But this isn't a strong argument because calling external APIs probably isn't something you want to be doing in Hadoop anyway). If your workload scales, though, you will eventually have to spread out onto multiple machines anyway. At that point, even if you process each job on a single machine, you either need a system to coordinate job allocation across machines, or you have to accept some slack if you only schedule jobs on individual machines. In the former case, we're taking rapid steps towards YARN anyways. I'm sure non-Hadoop, pre-packaged systems exist that can handle this, although I'm not personally familiar with them, but my point is just that the more steps we take towards the complexity of a distributed computing model, the more it starts to seem reasonable to just say "okay, let's just use hadoop/spark/etc. and get the extra benefits they offer as well (such as smooth scaling without having to think about hardware), even if it's more expensive than a setup with individual nodes optimized for our purpose." It's now really easy to spin up a cluster, and with small scale, costs are not that big a deal. With large scale, your system is distributed in some way anyway.

I don't think it's an obvious decision, and you're absolutely right that people are usually too quick to jump to distributed systems when they really don't need them. But I think there are a lot of arguments for using something like Hadoop even before it's strictly necessary. I think part of the disconnect is that we have different backgrounds, so we both look at different things and think "oh, that's easy" vs "oh, thats like a pain."

Just replying to the top bit, since we're exchanging multiple comment chains. Hadoop, Spark, etc. are built on top of a job scheduler, so if you have multiple competing jobs, it will allocate tasks to resources in a fairly reasonable way. If there is too much contention at some point in time, your jobs will all still run, and portions that can't be scheduled immediately will be delayed while they wait for free nodes. The processing model also generally leads to independence between jobs. You can do job scheduling on a single machine too, of course, but you get it for free with Hadoop/YARN. You don't get it for free with chained bash scripts or a custom single-node solution. And without sophisticated job scheduling, you can't really come too close to fully allocating your machine (when you're not watching carefully), because you'll accidentally exhaust resources and break random jobs.

edit: if you're running on a cloud, you also may be able to autoscale to deal with spiky usage patterns.

I'm not saying that past a terabyte is a point where you definitely want to use distributed processing, just that at that point, you should really strongly consider it. There is usually a lot of fuzziness around estimates you get about what sort of data volume you'll need to deal with, and it's not uncommon for it to vary by integral factors between days. If you're pushing the limits of what your system can handle without needing a dramatic rearchitecting, then that's a big risk, and it's not necessarily premature to build in the flexibility to have the option of scaling in the future if you need to. If you hit that 4TB and you still need more, it will be a big headache.

I can't really comment on rates of machine failures, but I have seen it happen before, even just for stupid reasons like someone in a data center unplugging a machine.

I agree with your first two points. It's definitely possible to efficiently process quite a bit of data on a single machine, although I think that past a terabyte, there begin to be strong arguments to a distributed approach even if you can theoretically handle it on one machine (scalability if requirements change, resilience to machine failures, etc.).

I still disagree about the article itself (even outside of the conclusion), but perhaps I am reading it uncharitably and other people are not getting the same impression. I do feel that it would be easy for someone who is not very familiar with these technologies to get the wrong impression. Misusage does probably go mainly in the other direction (of people overusing Hadoop rather than underusing), though, so maybe that is not so important a concern.

You're probably right. I'm thinking of specific coworkers when I think of the level of knowledge that should be expected, and generally the standard I would hold coworkers to would include understanding this. But it is probably not as widely understood as I would hope.

You're absolutely right. Probably my estimate of "anything greater than 1TB" was not quite the right number. At that point, though, careful indexing is probably the more expert option. It's easy to throw data into EMR now if you're not super concerned about performance/cost, and it doesn't require you to think through how your data will need to be indexed to support your future needs.

That's a good point. It's really hard to give a clear cutoff because there is quite a bit that you can potentially do on a single machine now. But there are disadvantages to doing your processing on a single really big machine even when you can find a way to handle it. If you have inconsistent workloads, a single machine may be harder to schedule to be close to fully allocated, and if it goes down, then your workflow is dead, possibly until you take some sort of manual action, whereas a cluster of machines is already built to be resilient to individual node failures. It all depends on the use case, which makes it really tough to give hard and fast rules.

I agree in general. But I think you underestimate how large the set of use cases are where people are processing > 1TB of data. This includes quite a bit of the adtech industry, for instance, even many startups. It also includes data warehouses for other industries, such as in health tech. Of course, these people generally are experts, since it is their core business, so they know well what tools they need. I agree that for some analytics department in a random company whose core business isn't processing data, Hadoop is more likely to be a resume item than something that's really needed.

They're well understood by anyone who has used these technologies professionally. I probably should have been ore precise with my language, though. It's really a grey area as to where the boundary is and it depends a lot on your specific application. But as a general rule of thumb, my feeling is that for tens to hundreds of GB, you should consider it. And for TBs or more, you almost certainly want to be doing something distributed. Hadoop isn't necessarily the best option then, but it's a powerful tool. I don't know if there's any resource out there that really goes deep into the tradeoffs involved though. There probably is, given how popular the subject is, but I'm not aware of one.

The problem with the article is that if it's for a general audience that doesn't understand the tradeoffs of a system like Hadoop, it really paints a picture that it is just a bad, slow tool. It barely acknowledge just how rigged the comparison is at all, aside from mentioning that you might need something like Hadoop for really big data in the conclusion, while it is peppered with unnecessarily snide comments about Hadoop that will probably be more memorable. I think it is liable to leave readers more confused about the tradeoffs involved after reading than before.

> Although Tom was doing the project for fun, often people use Hadoop and other so-called Big Data™ tools for real-world processing and analysis jobs that can be done faster with simpler tools and different techniques.

"Big Data™" is unnecessary shade.

> One especially under-used approach for data processing is using standard shell tools and commands. The benefits of this approach can be massive, since creating a data pipeline out of shell commands means that all the processing steps can be done in parallel. This is basically like having your own Storm cluster on your local machine.

It is entirely unlike having a Storm cluster on your machine, and trying to do your data processing as chained shell commands will rapidly become cumbersome if you try to do actual complex processing.

Yes, I get that the author is trying to point out that simpler tools can work for many cases, but the tone of the article makes it seem like that author is just generally saying that EMR/Hadoop is bad. He does not acknowledge just how weighted the test he did is against Hadoop or give any indication of what the tipping point is where you actually want to start considering something distributed. This paints a really misleading picture for anyone who does not already know a fair amount about these technologies.

Best fit is something you can argue a lot about. There are a lot of data processing tools out there now, many of which have come out after Hadoop. But if the comparison is against some process running on a single machine, then the use case is not narrow. It includes basically anything where you're processing more than 1TB of data in non-trivial ways (i.e. not just a map operation) and are okay with batch processing.

in other words, "I used a chainsaw to cut an apple and it SUCKED at it."

If you're processing an amount of data that comfortably fits in memory on a single machine, then obviously Hadoop is going to perform poorly in comparison. The costs of scheduling a job onto N mappers/reducers, transferring code to each node, waiting for the slowest mapper/reducer to finish, transferring data from mappers -> reducers, replicating output on HDFS, etc. are well-understood. It's true that many people try to use Hadoop when they'd be better served with simpler solutions, but that does not justify the amount of shade that the author throws at it.

It's not really about ordinary users like you. Google is shifting their strategy for Google Drive more towards enterprise, and this is a useful feature for enterprise users. Have you ever seen one of those email signatures stating "this email is property of XXX corporation, etc. etc." or seen an email sent to a large group of people in a company with strong exhortations not to send it anywhere else? This is a strict improvement over that. Yes, people can actively work around the printing restriction, so it's not 100% secure. But people were already using email for sensitive information, and this decreases the ways the information can get leaked from negligence or active subversion to just subversion.