davidmr

Like any other thing the SEC would consider a security, it’s complicated and technical. The article refers to https://www.sec.gov/litigation/investreport/34-81207.pdf as the SEC’s basis for considering how “coins” relate to securities.

“However, the Dragonfly teams were instructed that they were not permitted to discuss the issue directly with Brin or other members of Google’s senior leadership team, including Pichai, co-founder Larry Page, and legal chief Kent Walker.”

If that’s accurate, I cannot possibly understand why Beaumont is still employed at Google unless the senior leadership team decided intentionally to limit their exposure to the project so they could claim plausible deniability in the case it blew up like it has.

Obviously I’m not on the inside of this story, but from the outside, it’s getting increasingly shameful by the week.

They’d have to pay me a lot of money to do it, that’s for sure. I’d love to see the disaster recovery plans. Every major Lustre site I’m aware of has had a data loss “incident” at some point in their history. It’s possible AWS has it all figured out with background backups and block device replication and whatnot, but I’m skeptical.

This was inevitable, and I’m sure the engineering from Amazon’s side is impressive because Lustre is an absolute beast to run well at scale, but I’m not sure how great an idea it is for most people.

Coming from an academia HPC background and then moving into the private sector, I’ve mostly come to believe that parallel filesystems (especially POSIX-compliant ones) are rarely the right solution outside of MPI simulations. Like NFS, it makes it extremely easy and attractive to implement anti-patterns like using the filesystem for IPC or generating a bazillion files and then needing to reduce them to move to the next stage of the pipeline. In my experience, it’s rare that people don’t regret doing that sort of stuff in the long run.

That said, I’m sure the AWS team knows their customers and what they’re doing better than I do!

Well that’s fair. If it’s all unknowable anyway, I’d be happy to know that. Anything that lessens the blogspam on the topic!

That’s true of any kind of success in any organization, yet there are still studies that show that certain types of traits indicate success at certain types of organizations, etc. Even in a general sense, I’d be happy if these types of conversations could have data-driven discussions.

Maybe I’m just a grouch in general, but I pray for a time when we have sufficient data available that this ceaseless and monotonous drone of posts about whether remote work is the best thing since sliced bread or if it’s the 7th sign of the apocalypse can just all be replaced by “here are the data and there are some pretty clear conclusions that answer questions X, Y, and Z”.

Anyone who knows more about this than I do, are we close to that day?

Other than for people with limited mobility, can you help me understand why people think this is so? To me, it seems like evolution solved this problem for us when we became bipedal. It’s entirely possible (even likely) that I could just be a crank, but I genuinely don’t see the point of scooter rentals for healthy, sufficiently mobile people.

Purely out of curiosity, what jobs is your employer trying to fill that an intuitive understanding of said theorem is required?

I took a lot of linear algebra and I can’t recall having covered it.

Here’s something I wrote in this thread:

https://news.ycombinator.com/item?id=15222470

At least on the real big systems, most of our work was with a small number of research groups basically doing the same workflow: start your job, read in some data, crunch, every 15 minutes or something slam the entire contents of system memory (100s of TB) out to spinning disk, crunch, slam, your job gets killed when your time slice expires, get scheduled again, load up the last checkpoint, rinse, repeat. Because of the sheer amount of data, it’s an interesting problem, but you could generally work with the researchers to impose good I/O behavior that gets around the POSIX constraints peculiarities of the particular filesystems. You want 100,000,000 cpu hours on a $200M computer? You can do the work to make the filesystem writes easier on the system.

Coming into private industry was a real eye-opener. You’re in-house staff and you don’t get to say who can use the computer. People use the filesystems for IPC, store 100M files of 200B each, read() and write() terabytes of data 1B at a time, you name it. If I had $100 for every job in which I saw 10,000 cores running a stat() in a while loop waiting for some data to get written to it by one process that had long since died, I’d be retired on a beach somewhere.

The problem with POSIX I/O is that it’s so, so easy and it almost always works when you expect it to. GPFS (what I’m most familiar with) is amazing at enforcing the consistency. I’ve seen parallel filesystems and disk break in every imaginable way and in a lot of ways that aren’t, but I’ve never seen GPFS present data inconsistently across time where some write call was finished and it’s data didn’t show up to a read() started after the write got its lock or a situation where some process opened a file after the unlink was acknowledged. For a developer who hasn’t ever worked with parallel computing and whose boss just wants them to make it work, the filesystem is an amazing tool. I honestly can’t blame a developer who makes it work for 1000 cores and then gets upset with me when it blows up at 1500. I get grouchy with them, but I don’t blame them. (There’s a difference!)

But as the filesystems get bigger, the amount of work the filesystems have to do to maintain that consistency isn’t scaling. The amount of lock traffic flying back and forth between all the nodes is a lot of complexity to keep up with, and if you have the tiniest issue with your network even on some edge somewhere, you’re going to have a really unpleasant day.

One of the things that GCE and AWS have done so well is to just abandon the concept of the shared POSIX filesystem, and produce good tooling to help people deal with the IPC and workflow data processing without it. It’s a hell of a lot of work to go from an on-site HPC environment to GCE though. There’s a ton of money to be made for someone who can make that transition easier and cheaper (if you’ve got it figured out, you know, call me. I want to on it!), but people have sunk so much money into their parallel filesystems and disk that it’s a tough ask for the C-suite. Hypothetically speaking, someone I know really well who’s a lot like me was recently leading a project to do exactly this that got shut down basically because they couldn’t prove it would be cheaper in 3 years.

I definitely agree with you. I’ve been paid more or less to run GPFS over the last 15 years or so (finally free of it for the last few months), and while I hate it with a searing passion, a) it’s far better than the alternatives as long as you can afford it, and b) the people who lead and do the real work on the project are very, very bright. If they keep these people working on hard problems, they’ll do well. If they try to replace them with inexpensive staff, it will fall apart in months.

Parallel POSIX-compliant filesystems at scale are an astonishingly hard problem, and adding the feature set they have while keeping it relatively stable and performant is really worthy of admiration. Every one of the dozens of conversations I’ve had with their developers and technical leads have left me more impressed. They literally can’t test it internally at the scale that their large customers run it, but they’re really good at finding problems and pushing out hotfixes that work pretty well.

That said, if I never have to touch it again for as long as I live, I’ll be a very happy man. From an HPC perspective, I think we’re long past the scale where parallel filesystems should be POSIX-based.

Even by the standard of what passes for legal advice on the internet, this is especially terrible advice. I also live in the Midwest and am currently sitting out a noncompete agreement (which, since it must be paid to be enforceable in my state, has been quite enjoyable).

Like you, I’m also not a lawyer, but I did engage the services of a firm when I left. My lawyer informed me in no uncertain terms that my noncompete agreement was enforceable and cited several cases where a) their firm took employers to court to have the agreements tossed and lost and b) unsuccessfully defended people who did what you did and were sued by their previous employer.

In general, “fuck off, sue me” usually isn’t a great legal strategy.

“To a unique gateway” with a static IP is hardly an actual answer. Under whose AWS account is this server provisioned? Who has root?

I mean, I don’t understand any of this, but I doubt it’s worth trying to understand. At any rate, this is a strange HN submission title for a paper whose abstract claims to have unified quantum mechanics and gravity. Also, if a 2016 paper really did do what it claimed in the abstract, my untrained intuition tells me that it’s bunk because it’s languished for 2 years in obscurity. Oh, and he solved dark matter too. That’s a nice side effect.

This seems relevant: https://www.scottaaronson.com/blog/?p=304

I've read this comment 10 times and I still can't tell if it's satire or not. Well done, either way.

I don’t deny at all that it happens, and I understand that management has a difficult path to tread, but he just threw out a claim that it had happened to him with absolutely no evidence or even plausible story of who did it or why.

Am I missing something? He says PC Magazine fired him because he was critical of his 5G article and that rankled the magazine's sponsors. Who specifically are the sponsors he's thinking of? Is there any evidence whatsoever that this is what happened?

So they linked his article about 5G to a different one. Why does that mean he was fired for writing the article? I feel like an application of Hanlon's Razor (not quite right, but I don't know of someone else's who fits the situation better) is called for. Is it more likely the editors thought his article sucked and used their editorial discretion to redirect his 5G article to a different one or that there was a conspiracy between his advertisers and the magazine's management to fire a guy who's been wrong on just about everything for the last 30 years because he hit the nail too close to the head?

I don't buy it.

While I get that this discussion is taking place in the context of contentious CoC debates in other projects, what I don't understand is why this is a big deal in the Linux case.

Yes, people have been complaining about Linus' occasionally unambiguously abusive behavior for a long time, this genuinely seems like Linus had a bad few days, was pulled aside by some subset of the maintainers, and came to the conclusion that he should make some changes to his behavior and to the community of the project he controls.

I didn't see any threats or endless contentious debate derailing the project, so I'm genuinely struggling to understand why people are upset about this. If his previous behavior was acceptable, surely his new behavior will be acceptable as well?

Similarly, I spent 8 years building research and production trading infrastructure for a trading firm. The quality of data, tools, and trading execution infrastructure produced for these purposes was so far ahead of anything I could access without building my own HFT firm that I resolved to find myself on the other end of their trades as infrequently as humanly possible.

I'm definitely the dumb money in the market, trading on stone age tools by comparison.

I'm not a lawyer, but this definition can't possibly be correct, can it? If you're a janitor and you come across a piece of paper or a conversation between bigwigs discussing a merger and sell it to a trader, the trader and the janitor can surely both be sent up the river for insider trading, right?

At least in my limited experience, matrix multiplication optimization is some combination of a) so specific to the specific matrices being used that the optimization amounts to finding "tricks" to reduce the problem (dgemm to sgemm or sgemm to syrk, etc.) and thus is not generally useful or b) so highly tied to IP that general optimizations (be they mathematical or advances in GPU or CPU low-level code) are kept secret.

We're now well out of my knowledge base, but your comment certainly makes things interesting. If they have similar NUMA latencies (big if!), I guess maybe the edge would go to whichever chip had higher NUMA bandwidth? Just sort of thinking out loud...

Or perhaps it has absolutely nothing to do with memory speeds, but I can't think of what else would have such a dramatic effect. I'd love to see the same test with a single Xeon.

I wonder if there were NUMA effects at play since the Intel server was dual-socket and the EPYC was not. It's not entirely surprising that a CPU-heavy workload reading from filesystem cache (i.e. memory) could cause a substantial amount of nonlocal reads that the single-socket server wouldn't have to deal with.

That said, that's just the first thing that popped into my head to explain the result. Either way, the results here are extremely encouraging!

What specific quantitative reasons do you have to suggest a questionnaire is not appropriate for this type of study? Did I miss some other study somewhere that suggests a questionnaire can never produce good science?

That to me read more as CNN getting someone uninvolved in the research to give a quote that makes for good copy. I agree that it's not a particularly useful thing to say, but it's important to note that Collins is not involved in the core study.

Given that Stanley just said precisely the opposite, I'm wondering why you made this comment. I'm not being a jerk, but I wonder why you spoke so authoritatively. Do you think that he's lying, or were you just speaking in generalities?

I wondered that as well. I guess my imagination for uses isn't particularly good, but all the uses i bought my 16 core workstation for (mostly research computing development) would really suffer with the memory performance of the 32 core chip.

I don't know enough about video games, but I would naively think that the memory latency would be a big deal there as well.

At any rate, I learned a lot from this article. Anandtech's reviews always seem well-written and well-researched.

I dunno, maybe. They're so completely different that it seems extremely unlikely to me that anyone will confuse the two in a conversation with any context whatsoever. The conversation you made up would almost certainly start with a discussion about scientific parallel programming. Any discussion I've ever had about charm++ (too many, I'm afraid) wouldn't have been confused for a discussion about an IDE.

Either way, it's done.

If it makes you feel any better, there was no particular shortage of idiots there either, just like any other place.

I worked at one of the above and can confirm that the interns, particularly the R&D interns were producing truly excellent work at the forefront of their respective fields, particularly with the mentorship of the senior staff. I'm serving a 1 year non-compete at the moment, and the only thing I find myself actually missing is the internal colloquia for some of the in-house projects and research.

It's very, very cool of Jane Street to publish some of these details. All the firms are doing approximately all of the same things tech-wise, so I think Jane Street might get a leg up on the next batch of recruits just for being willing to give them some public credit.