clusterhacks

Several comments mention the data immutability of Datomic as a plus and I just wanted to say you can totally make a plain-old-RDBMS table append-only and get those benefits. I'm sure this is commonly done.

I did it with a timestamp on the tables that was captured at insert time. All reads were against views of the tables that were defined such that the only tuples returned were the "most recent" tuples by appropriate data fields and max(timestamp). "Deleted" records were just indicated by a flag.

This preserved the ability see the full history for a tuple from creation, all mutations, all the way to deletion. This scaled reasonable well up to low millions of tuples on a normal, single database server. But it was for an internal project, so the number of clients hammering at it was quite low.

Multiple times in the last 3-5 years, I have coded up wrappers to new/evolving C++ libraries using Rcpp for internal use by bioinformaticians.

Especially anything using multi-threaded algorithms or from the recent algorithms research world, I really get a kick out of hacking into a usable R module. It is almost like doing computer science again (as opposed to the real day job, which is data munging for reporting tools).

That sounds interesting. Are there easy sources for seeing what companies drive US equity market volume? My google-fu failed me but maybe I am missing something specific in the fintech industry that tracks that kind of thing . . .

Doesn't this ignore the reality on the ground that Clojure is a niche (but wonderful) programming language that offers very few career opportunities? It simply isn't big in any specific domain.

I also don't buy that you see more Clojure the "further you get from silicon valley ..." without some actual (or even anecdotal) data to support that claim.

I'm not saying there are zero instances of Clojure in production systems. But it is hard for me to think that it represents a large number of professional employed software engineers. The 2023 StackOverflow survey show 1.38% out of 67,053 respondents in their "most commonly used programming language" section naming Clojure. That is less than 1,000 developers. The public university I work at graduated 500+ CS undergrads in 2023. I would bet $100 that zero of those got a first job that paid them to write Clojure professionally. I would bet $1000 for less than ten grads. I would bet $50,000 for less than 25.

I'm not trying to be cute but don't you think there are less than 10,000 software engineers in the US (maybe the freaking world) who primarily write Clojure code in their day job?

Because I'm curious, job searches at the company site of your list for 'clojure':

  Nubank - enhum resultado com a palavra "clojure" encontrado (no results)

  Guaranteed Rate - no results

  Kroo Bank careers page - no tech listings currently

  Griffin - open listing for software engineer (London or Remote within the UK, Germany, Sweden or Ireland) - description: Our backend stack is Clojure, FoundationDB, Kubernetes and AWS. Our frontend stack is CLJS, Reframe, Reagent, React, Stitches, Storybook, and Playroom.

  Pennymac - Sorry, no jobs were found that match your search criteria. 

  Two Sigma - No jobs found

  Dividend Finance - 0 JOBS FOUND

  Treasury Prime - There are no current openings. (for software engineers)

  Gravie - Lead software engineer position - description: Advanced programming experience in multiple programming languages ( Java, Kotlin, Groovy/Grails, JavaScript/TypeScript or Python) along with Clojure/ClojureScript or another functional programming language

Indeed.com returns 15 jobs posted in the last two weeks (US) for the Clojure keyword. LinkedIn returns 34 job posting in the last month (again, in the US).

I love Clojure - it has always "worked" in my head as a language to solve problems in. I was at the first Clojure/conj. I have deployed production code that I wrote in Clojure. Clojure made me a better developer in other languages.

But in the big picture for professional software engineers, Clojure might as well not exist.

Ok, so this is kind of funny but I also want to know - did/do you feel productive with J for stats work?

Did you ever switch to R? If so, how would you describe the two languages to a non-programmer?

One thing that has helped me relax is that my partner has a career that is very different than mine and we are a dual-income household. We have, for the most part, lived a life that can be supported on just one of our incomes. Our mortgage gets paid and we all eat on one of our household salaries. We are mostly frugal but not miserly. We have savings and investments that would help cushion income loss.

Academic credentials have been helpful to me - I have a MS in CS. Your profile indicates you are mechanical engineer by training. It may be worth considering if a part-time or online MS in CS is worthwhile. Especially consider this if your company offers an educational support benefit - a MS or MBA on the company's dime is attractive. I have a vague feeling the extra credentials matter much less for FAANG or software companies than working in healthcare or finance as a software engineer.

Meaningful efforts are worthwhile, even if it has to be personal projects, where you can say "I actually did this and the effect was <insert something cool>." My current organization always gets a large number of "aspirational" engineer applicants when we have openings. These types of folks can be summarized as having lite IT backgrounds. Maybe they have a little training in programming but are now mostly spreadsheet jockeys or have been doing desktop support. Or they are career-switchers who did a short javascript bootcamp and have held a number of entry-level jobs but don't seem to have actually programmed much. Don't be one of these, no matter what it takes.

To be clear, I am saying standardized tests (SAT and ACT) are useful for admission decision purposes today where the average high school graduate who applies to a college has a 3.5-4.0 GPA on a 4.0 scale. I am also being rather specific in saying that limited-seat institutions or heavily over-subscribed schools where the number of applicants greatly exceed the number of enrollment spots need some way of filtering applicants.

A thing that I feel is VERY true is that most "highly selective" colleges and universities probably get a relatively small percentage of "instant admit" applicants and a much larger number of "probably will be successful here" applicants. A higher standardized test score (ignoring some nuance here) is a useful tool for entering the admissions lottery in that second group. It isn't all there is - I see high school grads with good standardized test scores, the default high GPA, and not much or any interesting things otherwise struggle to gain admission at over-subscribed schools.

Switching to actual college coursework, when I have taught undergrad students, I use demanding individual projects (evaluated by oral presentation, demo, and questioning) as a significant part of the course. It mostly doesn't scale as an approach but my (biased) feeling is that for the "middle 50%" of students, it is a valuable and worthwhile effort.

All of this gets really murky and nuanced depending on what your real goals are. Those goals are often quite different depending on perspective.

Seriously? That just seems ridiculous. Is this an aging thing, where folks are worried about low testosterone or something?

Wish I could upvote more "to increase the likelihood that they can get a good job or get into grad school." I taught as an adjunct at a great public university. Undergrads seem to think there are two grades - A or failing.

But that attitude starts in high school. As a parent of kids there now, rampant grade inflation and an "infinite re-do" approach to assignments and tests really sets the tone for kids to feel quite bad about themselves for a B+.

I'm bullish on standardized tests becoming more, rather than less, important for admissions to competitive (or more honestly, limited-number-of-student-slots) universities at the undergrad and graduate level.

Share some numbers - I'm totally curious how many applicants you are getting for roles total/applicants that you easily identify as using LLMs. Even ballpark would be ok, "We got 40+ applicants for a senior software engineer, at least 50% had identical answers to our questions."

I have some theories about the duplicate answers that go beyond LLM usage . . .

Now that's interesting - I haven't paid much attention to the HPC job market in years. Are you a HPC system administrator, HPC dev, or both?

When I was at IBM, the whole patent/publication process was intensely gamed. Most employees in my division weren't particularly interested but made efforts to patent/publish because it was part of moving up salary bands.

A few true sociopaths went totally scorched earth with it - digging up years-old reports generated by other teams, getting them published after minimal "updates", and claiming lots of bonus dollars along the way.

It was a very weird experience working at IBM for me . . .

You may be interested in looking at the current Taulbee survey[1] - it surveys PhD granting CS and CE programs and collects MS and BS degree numbers as well.

I also found [2] that shows "computer and information science" graduates having a higher-than-average unemployment rate than other degrees. In 2018.

My anecdotal experience over the last 15 years as a person on the hiring side of the table is that my organization has received multiple qualified applicants for our open positions, several of who are almost always new CS grads looking for their first job. And my organization is in an area with multiple companies that hire software engineers and we typically pay somewhat below market. That experience makes me question if there is now (or even has been in that 15 years) an actual shortage of candidates for these types of jobs.

  [1-pdf] https://cra.org/crn/wp-content/uploads/sites/7/2023/05/2022-Taulbee-Survey-Final.pdf
  [2-pdf] https://nces.ed.gov/programs/coe/pdf/coe_sbc.pdf

Honest question: What changed? A few months ago you commented on HN and said you made $190K on a 4-day work week? Did your company do a lay-off? Freelance dry up?

Really? I looked up some data about working in the trades awhile back (so numbers maybe slightly higher in 2022 data than in my comment, but still close).

The U.S. Census Bureau lists the annual real median earnings at $41,535 in 2020 for all workers with earnings[4] and lists the annual median earnings at $56,287 in 2020 for people who worked full-time, year round. [1]

  Plumber - median $56,330, demand growth, slower than average [2]
  HVAC tech - median $50,590, demand growth, slower than average [3]
  Electrician - median $56,900, demand growth average (actually 9%) [4]
  Framer(Carpenter) - median $49,520, demand growth, slower than average [5]
  Bricklayer(Mason) - median $47,710, declining demand. [6]
  Software engineer - median $110,140, demand growth, higher than average [7]

  [1] https://en.wikipedia.org/wiki/Personal_income_in_the_United_States
  [2] https://www.bls.gov/ooh/construction-and-extraction/plumbers-pipefitters-and-steamfitters.htm
  [3] https://www.bls.gov/ooh/installation-maintenance-and-repair/heating-air-conditioning-and-refrigeration-mechanics-and-installers.htm
  [4] https://www.bls.gov/ooh/construction-and-extraction/electricians.htm
  [5] https://www.bls.gov/ooh/construction-and-extraction/carpenters.htm
  [6] https://www.bls.gov/ooh/construction-and-extraction/brickmasons-blockmasons-and-stonemasons.htm
  [7] https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm

At Harvard, 28% of students receive no aid at all. So they pay full sticker price.

Average net price by income in 2021-2022 [1]:

  Income               Cost
  0-$30,000         -- $5900
  $30,001-$48,000   -- $3,002
  $48,001-$75,000   -- $4,180
  $75,001-$110,000  -- $17,037
  $110,001+         -- $52,634

If your household makes $110,000 per year in California, the take home pay I found calculated online was ~$75,000. So that means paying around 22% of take home pay to go to Harvard? Can the average household actually put 22% of their take home pay into tuition without taking on debt?

[1] https://nces.ed.gov/collegenavigator/?q=harvard&s=all&id=166...

"which I believe is due to bloated administration."

In my state's public university system, the amount of funding provided by the state to that system per full-time equivalent student has dropped to its lowest level (in 2017 according to the source I checked) since 1995. So the state commitment to funding is lower on a per-student basis. Universities have responded by raising tuition and other costs. All of those facts mean that the real, out-of-pocket cost to students has gone up.

Looking at what information I can find on staff counts, the flagship state university data report in 1999/2000 shows a "grand total" of 10,000ish employees and a quick web search shows 13,000ish in 2023. I think we would have to slice and dice those numbers much better to see what the actual cost differences were for those two time periods.

The number of students seems to have grown to about 128% of 1999/2000 total enrollment numbers to today's total.

There is a deeper discussion to be had about the policy of funding state universities. Polls like the article show all segments of the public surveyed have declining confidence in higher education. That suggests to me that in general, people will be less inclined to have tax dollars invested into the public university system because they may see it as wasteful or not efficient.

There is some much to unpack around universities and cost . . . and, of course, private university tuition costs have exploded. Although at those types of institutions, I think many non-name-brand (or non-ivy-ish) schools have a very high "sticker" cost that maybe no students actually pay . . .

I taught an entry-level programming class in a university setting. There were weekly programming practice "labs" and a more involved independent project with several check-ins with me over the semester.

I gave an open-book final exam that asked students to write a for loop on one of the questions. I would say around 20% bombed the question despite the very book sitting beside them having a chapter on looping. With examples.

After the class ended, I ultimately concluded a non-trivial number of students probably didn't do any of their own coding during the course. I'm pretty sure I have worked with at least two or three people in coding positions who couldn't actually code and relied on outside (spouse? out-sourced?) support.

Apologies! Not invisible or forgotten!!

Do you remember any rough numbers on started vs finished on the two CTF tracks y'all put up?

FYI, the six stockfighter challenges were some of the most fun I ever had hacking around. I enjoy puzzle games like Exit and happily spend money on them. I would probably have paid to play the starfighter challenges even though the embedded one wasn't particularly my cup of tea. I wonder how many other nuts like me might have happily paid a monthly subscription for a few releases per year . . .

I also remember you saying in an early thread that clients' hiring processes were a mess and I think implied most treated Starfighter as just another candidate funnel. Did any of your clients take up a challenge/CTF process after working with Starfighter?

I know Thomas says hiring with challenges/CTFs is a competitive advantage but I'm not so sure. If it was true, wouldn't we see more of it out in the real world?

Thanks!! No surprises here from what I read, but definitely appreciate your thoughts.

Got a short list of companies in addition to Fly.io that are using a hiring process you like or love?

Probably depend on staging for this disease? If I was diagnosed with pancreatic stage 4 (metastatic) disease, the ship has maybe sailed on meaningful improvements in 5 year survival. I would probably opt-in to palliative surgery to improve quality of life for the next few months if the recovery didn't seem too gruesome.

Stage 1 or Stage 2? 100%. Otherwise, you are just waiting to be stage 4 . . .

Totally agree it was a blast. I learned a ton about trading just reading various recommended resources (Harris Trading and Exchanges was a solid intro) and then solving the levels.

I don't think very many players did all the Starfighter trading challenges. patio11 or tptacek might be able to confirm, but I vaguely recall being one of like 100 or so to do all six levels of out of 10,000 that started. But I also had free time for playful coding and was nerding out on online challenges back then.

When Starfighter came out, I was energized and hopeful that hiring would go in this direction for programmers or that it would open doors to me as a job seeker that I couldn't every push through. No such luck.

From the hiring side, I have been on two hiring committees this year. Most applicants lacked any meaningful programming projects they could use to demonstrate skills. Something like completing a series of online challenges would be a difference maker.

That said, one candidate mentioned doing HackerRank challenges to learn Python and SQL, but totally flamed out on being able to even talk through a for loop in Python or come up with a simple select against a table in the in-person interview. Maybe nerves, but it also felt like the candidate's experience was 100% as just a warm body in various cubicle farms. It was very awkward. Several non-developers on the hiring committee felt like the candidate had actively lied about previous experience . . .

It was fun! I did all the stock trading challenges but none of the embedded device challenges.

patio11 once said that the code wasn't in an open-source ready state.

There were some folks who went so far as re-implementing/duplicating the backend trade simulator in other languages. Some of those were on github . . .

We use similar questions with the exception of #6 - our data sets are small enough that just getting a correct query is fine. We add a small relational design exercise to see if they understand data modeling 1-to-N and M-N relationships.

Totally curious - what is your fail rate like for applicants at the first stage for the window function? We are happy with more junior engineers who usually haven't used a window function but can currently do simple queries and an easy candidate to move forward with will come up with a group by subquery . . .

I'm guessing that 2/3 to 3/4 of our in-person interview candidates look (on paper) like they could handle simple queries and grouping but struggle in the interview context.

Yes, the financial trade-offs for getting a PhD versus simply earning median software engineer salary is extreme. So much so that most people should only consider the PhD if that trade-off simply isn't a consideration.

Somewhat more problematically, "a pretty respectable level of knowledge and ability for CS research" is table stakes for just about every grad student in a CS program. This is especially true at better programs. You might be sadly surprised just how little your industry programming experience is valued in academic CS research - in my grad program, being highly skilled in multiple programming languages was assumed.

By better program, I mean programs like an R1 university CS research program - I am specifically excluding terminal, MS-only CS programs in that statement. Many of those terminal, MS-only programs are actually credential programs for career changers (people who want to upgrade to a SW engineer job and had unrelated undergrad majors).

You might want to look at how the papers and theses you have been enjoying were produced. I strongly suspect that the writers were mostly grad students working under a current professor who had grants to support grad students to do that work. There probably isn't much (any?) demand to hire programmers to do research programming work - that is what research assistantships pay students for.

Maybe a good analogy here would be think about someone who has a physics undergrad, has worked in applied engineering in the energy industry for 16 years, and is now interested in being hired into a university role that allows that person to do more of their day-to-day work directly in particle physics research. What physics department is going to hire that person? Why? Keep in mind there is a verifiable surplus of physics PhD postdocs looking for that gig . . .

All this isn't meant to be overly discouraging, just my thoughts. Honestly, just keep a list of universities you wouldn't mind working at and once a week or so, browse all their job listings to see if something fits your interests and apply if it does!

While it might be better if you better defined what you think the "academic career path" is, I'll throw out my experience . . .

I work as a programmer at an academic/research institution. But the job is just normal IT and light programming. I think finding a role similar to mine would be easy for you. Your MIT math undergrad will probably be more than enough credential signaling for that type of job to go along with your professional experience.

My path back to this environment for work goes roughly: earned MS in CS directly after BS in CS, two longer tenure jobs in industry (8-ish+ years total) after grad school, returned to academic/research institution because of personal network and also having skills needed for new role at institution. I have been here for 10+ years. Much of that time it seems like my earnings were at least 20% less than market rate for my region. But the pace of the job was easy and many people "settle in" to these jobs.

There is some autonomy here and a little room to stretch your day-to-day work to fit personal interests. But much less than you may expect. The work environment is not THAT different than industry and I personally have found that it is oddly more political here than expected with respect to who gets to do what work. I have been able to be involved more directly with researchers as a "side gig" when my day-to-day job responsibilities allowed. I probably would have left this role but personal+family responsibilities made the flexibility of the job hard to move on from. I am also in an unusual position wrt to funding - our group is heavily funded by hard money, so we are significantly less dependent on grants than even some research/teaching staff at my institution.

Your academic interests are probably going to be hard to fit in - those are "pure-ish" CS topics. I know some PhD applied math guys who do research computing support and their roles are often limited to helping with existing libraries with users of a research compute cluster. They aren't doing much applied math . . .

Are you willing to pull an extra stint of time in grad school for a MS or PhD? Joining a good research team and publishing with them could build up your CV for the academic job market. It isn't super critical unless you want to be research faculty somewhere . . . Some CS programs have a "professor of the practice" job role that could be attractive but it usually involves significant entry-level teaching requirements.

Not sure if any of this is helpful. Good luck figuring it out.

"Business users often ask for things without an ounce of rationale."

Deserves extra upvotes just for this statement.

This has always been painful to me working in the data analysis and reporting space. When I get many requests for dashboards or reports that lack an answer to the question of "how will this be used?", I seem to find the requesting groups are cost-centers in the larger organization and are somewhat obsessed with processes and procedures.

This is rarely a good group to build a career with . . .

Sure, but UIUC total sticker price is $32K-ish. And approximately 36% of students pay that full cost if only 64% receive any financial aid based on:

https://nces.ed.gov/collegenavigator/q=illinois&s=all&pg=2&i...

Something I have found interesting (as the first of my two kids rolls over into the college age group next year) is that most "brand name", well-known private schools clearly expect our family to pay the full $80K+ total sticker price.

While I share the same feelings of not being particularly wealthy as OP, it is clear that some colleges disagree with my personal wealth assessment . . .

I gave an honest floor salary to a company prior to interviewing. Their offer was 20% less than my floor. I think they thought it was my "best case" number despite very clear communication that it was my minimum to consider leaving my current role.

I walked away vaguely annoyed with the hiring company.

The article is like four paragraphs.

I think the contribution is better identification of risk factors than were previously known?

  The most important variables influencing KECAN included 4 known risk 
  factors(age, race, sex, BMI) and 9 novel (COPD, greater Hct,lower HDL, 
  greater LDL, lower serum CO2, lower Na, lower BUN, lower ALT, and 
  greater WBC).

The AI part seems like a buzzword-y add-on?

"We collected prescriptions, laboratory results, and International Classification of Diseases diagnoses 1 to 5 years prior to index. We randomly divided the cohort into training (50%), preliminary validation (25%), and testing (25%). In the preliminary validation set, simple random sampling imputation and extreme gradient boosting machine learning were most accurate. In the test set, we compared the final model, the Kettles Esophageal and Cardia Adenocarcinoma predictioN (K-ECAN) Tool, to HUNT, Kunzmann, and published guidelines."

I'm 100% not knowledgeable enough to parse that out, but I think maybe they ran XGBoost and did some hyperparameter tuning?