p4ul

With all the overdose deaths and the lives ruined by opioids, I guess I'm a little bit surprised this didn't result in criminal charges for someone at McKinsey. Surely there must be some emails or texts that are inculpating.

Indeed! I feel like we as a community have taken a wrong turn with our use of notebooks. I think they have benefits in some specific use cases (e.g., teaching, demos, etc.), but otherwise, I think they mostly encourage bad practices for software development.

This is correct, sadly.

I've done some work with the public data from the US Occupational Safety and Health Administration (OSHA). OSHA does a great job of making detailed data on inspections and accidents public. If memory serves me correctly, it turns out there are about two deaths per day from work-related causes in the US...

I think there is a fair bit of personal tastes in the answers to these kinds of questions. Personally, I've become a fan of Josh Comeau's "delightful React file/directory structure" [1].

I think it makes a ton of sense, but again, I suspect there's a fair bit of personal preference in these kinds of things.

I'm eager to see what other styles and architectures people suggest!

[1] https://www.joshwcomeau.com/react/file-structure/

Very cool! Thank for this, claytonjy!!

I usually assume these companies are using some of the popular schedulers (e.g., Slurm, MOAB, SGE) that have existed in the HPC community for many years.

I have anecdotally also heard that some are using k8s, but I've not seen that myself. Slurm [1] is basically built for this stuff; that's definitely what I would use!

[1] https://slurm.schedmd.com/documentation.html

They are going to have to pry Firefox out of my cold, dead hands!

I've only used Arc a bit, and generally had a good experience, but I do wish that they had chosen not to rely Chromium. I understand that would have massively increased the amount of work required initially, but I really worry about our collective reliance on Chromium. I don't know... maybe I'm a pessimist.

You are correct!

This is Cunningham's Law [1]! And it seems exceedingly plausible after observing behavior on many online forums!

[1] https://meta.wikimedia.org/wiki/Cunningham%27s_Law

Very cool! I hope the Rust crate gets to take advantage of this. It might just be me, but I feel like the Python package gets all the love from the Polars devs.

This is exactly what I would argue. PostgreSQL makes it straightforward to create extremely powerful extensions.

PostGIS is one such extension, and I would argue that if your use case involves geospatial data, then PostGIS alone is enough of a reason to use PostgreSQL!

At least in the US, an institution’s “facilities and administrative” F&A rate gets negotiated with the Federal government every four or five years.

I know of some institutions with F&A rates at or above 70%!! I presume that an institution trying to negotiate a higher F&A rate than this would have some significant pushback!

In the US, most employers roughly follow the guidelines that the Federal government uses [1]. Which is to say, that employment is "at will", and either the employer or employee can terminate the relationship at any time. The obvious caveat is that someone cannot be fired for a reason that would be illegal (e.g., firing someone because of age, gender, religion, etc.).

In practice, however, most organizations are extremely risk-averse when it comes to wrongful termination lawsuits. And as such, they'll have a lengthy "performance-improvement plan" (PIP) that can last three to six months and is meant to serve as evidence that an employee's poor performance was the reason for dismissal.

There are also, of course, extreme cases where someone can be fired "for cause" when they've committed some kind of egregious violation (e.g., embezzling funds, fraud, corporate espionage).

[1.] https://www.usa.gov/termination-for-employers

Yes! I couldn't possibly agree more; this is truly my fondest wish!

I'm inclined to agree with this. I'm super excited to see how Candle progresses in the coming years!

I might be wrong, but the wording of that question suggests it was only answered by people who consider themselves part of an underrepresented or marginalized group. So only a subset of the survey's responders answered.

This is also supported by the fact that the question was only completed by 1322 responders, which is about 10% of total responders.

Oh, I actually had not yet heard of the DataFusion crate! Thanks for sharing!

And to answer your question, we use the Rust API because all of our backend services are in Rust, and we like to stay in Rust whenever possible.

This echos my personal experience as well.

I would also add that there now seems to be more of an emphasis around being intentional in how languages foster community. The C and C++ communities (in my opinion) became how they are not by design, but mostly by accident or chance.

I absolutely love Polars, and I use the Rust crate all the time!

I think my only gripe is that the Rust API seems (to me at least) to be less well documented than the Python API. I guess Python is the de facto data science language, so maybe that explains it.

This looks super cool; thanks so much for sharing!

I'm probably an outlier, but when I saw this name I thought of the eating disorder [1].

[1] https://en.wikipedia.org/wiki/Pica_(disorder)

I would modify this slightly. We do experiment with humans; we just cannot do _certain kinds_ of experiments with humans because of ethical considerations. We constantly do clinic trials, which are essentially experiments with a high degree of controls and randomization.

I think probably the major challenge for the social sciences is the nature of the phenomena under study. Psychologists, sociologists, and economists aren't studying physical quantities that can easily be measured. Instead, they typically study abstract and intangible variables. Measuring these accurately is very difficult.

I am really curious about this, too! From time to time I'll switch to using a more full-featured IDE for a while, but I always come back to Vim!

Slurm is absolutely ubiquitous in the high-performance computing (HPC) community. I believe its only similar competitors in the HPC space are the SGE [1] and Torque/PBS [2] resource schedulers.

I'm not sure of the exact numbers, but I would guess that an overwhelming majority of the Top 500 Supercomputers [3] are running Slurm. And as others have noted, research computing centers in academia all mostly run Slurm. And Slurm also dominates in the DoE national labs in the US.

Oh, and as a [potentially apocryphal] fun fact, the name "Simple Linux Utility for Resource Management (SLURM)" is a backronym from the soda in Futurama! [4]

[1] https://en.wikipedia.org/wiki/Oracle_Grid_Engine

[2] https://github.com/adaptivecomputing/torque

[3] https://www.top500.org/

[4] https://futurama.fandom.com/wiki/Slurm

Thank you so much for sharing this! It's definitely an interesting and valuable perspective.

Oh, yes, I was aware that R (and its predecessor S) have a native dataframe object in the language.

It seemed that gmfawcett was indicating that there was a dataframe library in _Python_ that existed prior to Pandas. I was curious what that library was/is, as I'd not heard that before.

Same! And it's been an absolute delight!

Oh, and another important difference is memory layout. The dataframe implementations mostly (or all) use column-major format. Whereas most conventional SQL implementations use row-major format, I believe.

The way that I've heard the story, S was short for "statistics", and R was chosen because the authors were _R_obert [Gentleman] and _R_oss [Ihaka].

Statisticians are funny!

I'm worried it's going to get grizzly.