clusterhacks

"... our bet is that Deep Learning will overtake traditional BI workflows ..."

This is an interesting perspective. I have spent years in the traditional BI space and my gut feeling there is that analytics there are very much not fancy. Simple stuff seems to be where the real ROI is at.

Are you saying that data storage, data model, etc that Activeloop puts in place to better support deep learning workflows will replace the data storage, data model, etc as the store of information but visualization and querying will still be like BI work? Or alternatively, are you saying that deep learning is on a roaring path to replace traditional BI analytics?

The comparison feels just a bit off - like, Malmsteen's popularity rounds to zero, even in the world of 80's shredders aficionados. Gladwell has multiple best-sellers that are popular to general audiences. I think maybe a better comp would be someone like Reb Beach? Player with known technical chops but in a somewhat generally popular band that was not appreciated by guitar-store superheroes . . .

Course, maybe not using an obscure hair-band reference would be better.

I have seen organization-level layoffs and more targeted layoffs.

In the (only) organization level layoff I saw, someone stuck a pin in an org chart at a pretty high level. Everybody (and I mean everybody) in the tree underneath that point lost their job. Longtime employees at that company were in total shock that day. This was a layoff of a couple thousand employees.

I also saw a salary-based layoff happen. In that one, a number of software engineers in a division making over $X were let go in a cost-saving maneuver. Macroeconomic conditions continued to decline and a few months later, most early tenure (say 3-4 years or less) were basically let go.

I was new hire at the org-level layoff place I mentioned. New hires were exempted from the "resource action." Over the next 6 years that I was there, there were periodic layoffs but these were mostly targeted at folks judged as not doing particularly well or in positions that simply weren't needed anymore.

My experience is that software businesses and teams have an internal pressure to grow the number of software engineers on staff. I have seen three things swirl together to cause that.

1 - During periods of company growth (increased revenue and profits), there is a tendency of leadership to not have their hands on the wheel. You tend to see growth in company "cost-centers" instead of "profit centers." So you'll get new efforts that don't directly contribute much in increasing revenue. Early in my career I wasn't tuned into the need to really build your work and projects directly into profit center activities.

2 - Internally, I have seen people grow the number of staff underneath them as an almost weird way of justifying a promotion (eg, "executive VP mucky-mucks have N employees and I'm now managing N employees, promote me to executive VP mucky-muck").

3 - I have seen smaller companies really focus in on the revenue and/or profit per employee number to guide hiring plans. So during a good period of revenue/profit growth, a CEO/owner-type will say something like "It looks like every engineer I add at $1X0,000 is resulting in $[2|3|4]X0,000 additional revenue or profit - let's just keep adding engineers." These napkin calculations are devoid of deeper market analysis.

If you went one-on-one with every single software engineer at a big tech company or in a smaller org, you probably wouldn't find many who aren't busy with work. You just probably would find a bunch on "nice to have" work instead of real revenue generators.

Did Microsoft use the source code of Windows (in whole or in part) as training input to Copilot?

I have worked at a couple of organizations where bullying and internal competition with negative consequences were rampant. It is a very disheartening experience.

The worst was at my first employer. I was naive and several times was publicly eviscerated by folks clearly interested in dumping the blame for project failures and schedule slips onto some other person. I weathered those experiences because I was doing very well on other projects and had support from management and peers on those efforts. Otherwise, I'm sure I would have been one of the endless layoffs that seemed to occur every quarter for "low performers" in that stack-ranked hellscape.

My current employer (longest tenure of my professional career) has many of the same problems, particularly at the staff level between organizations. I am ruthless about strictly defining precisely my prereqs for any work assignment and quietly decline or avoid working with several groups at all. It is really unfortunate because the potential for much more impactful and meaningful work is lost.

Maybe, but I had friends making all-cash, over-asking-price offers last summer but getting outbid. Interest rates never factored into their thoughts at all . . .

They simply noped out on shopping because the market seemed so irrational to them personally. I wonder if other similarly wealthy buyers looked around and just did the same . . .

Ugh. I do NOT miss commuting into RTP. What counts as a 1 hour drive these days? Dating myself, but I remember co-workers at Nortel commuting into RTP from well east of Raleigh . . . that was awful even years ago when Nortel still existed.

Know much about folks buying that far out these days? I mean, I hear apocryphal stories about Apple/Google engineers buying up houses unseen in the immediate RTP area but I am not entirely sure I believe those.

Anecdotally speaking re: RTP, NC but I think the area may have peaked back in summer. Inventory is sitting much longer. I know sellers who sold under asking for a price much more similar to previous years recently.

New construction is always a bit behind here and we will definitely see higher prices in central NC with the various tech companies increasing their presence in the area.

But it was complete bonkers until August. Sellers were getting double-digit all-cash offers 10%-20% over asking price. Never seen anything like that here before.

I don't have a good answer to your question here are some results from a quick google search:

https://blog.prepscholar.com/average-sat-scores-over-time

I think a sometimes-easy-to-overlook factor is that many state flagship institutions like UT-Austin have a strong bias or preference for in-state applicants. Famously, I remember being told that out-of-state applicants to my state's big, somewhat well-known, public university needed better credentials than highly selective private institutions required/recommended . . .

I live and grew up in the mid-Atlantic US area. You would be surprised at the number of people who will express deep distrust and even disgust at government programs for poor people. I have heard, many times, about how these programs are "taken advantage of" by people who "made bad choices" and "why should we support people who won't support themselves."

A very specific example - I had a close relative who was adamantly opposed to subsidized housing. This was despite working in the food service/hotel industry directly with people who earned too little at work to afford housing in the area without help. This relative could easily express the idea that her employer paid very little and ruthlessly limited work hours to avoid having to pay benefits (like, you know, health insurance) but not connect how her co-workers had to live in subsidized housing because they simply didn't earn enough (ever in their lives) to afford any other choice.

I suspect some similar thinking around providing a better meal plan for kids. During the pandemic, my (relatively wealthy) school district did provide free breakfast and lunch for all students. However, that was based on COVID federal grants and has expired. Students can still apply for free and reduced price meals and even in our "wealthier" district around 17% of students qualify for those programs. This roughly tracks our state statistics which report around 20% of families with kids in schools have problems with food insecurity.

We have a deeply conservative state legislature. The state itself has very large disparities in income and poverty based on area. I am honestly not particularly well-informed on free and reduced meal plan numbers in different areas. It is unclear to me if it is "politically viable" to provide additional subsidies or financial support to those poorer districts.

Oh, I totally was not questioning the legitimacy of the statement - most of my work is not in service of producing software that is sold. My work is instead a mess of bad practice to just get something accomplished for others.

More generally speaking, I wish more people were like this creator - unafraid to just dive in and get stuff done. I have done a little bit of teaching introductory programming and one of the things I emphasized is that "works but is suboptimal along some professional measure" mostly rounds to "works" even in professional settings.

Part of my actual day job is working with non-professional software developers and helping take "non-professional" prototypes to "better" implementations. Having a working prototype is often a much better starting point that a poorly specified set of requirements . . .

Me too. I laughed at "Oh, I'm not a programmer. I had to write all the code to drive the OLED screen . . ."

Inspiring. Love when people do this. Makers gonna make . . .

I had two past co-workers who were almost certainly having someone else do their work, I think (especially) at non-software companies folks just don't get caught.

Frankly, the work environment was slow-paced and generally non-confrontational so both co-workers I suspect of this behavior just managed to tread water. When I joined the group there was a very tight-knit group of 3-4 developers who were very protective of each other. There was always a handy reason why some schedule was slipping and the other fact is that in hindsight, there just weren't very high expectations for them to accomplish anything.

Yes! I have a long-ish tenure at my current org (10+ years). I have had at least two co-workers in that time window who were clearly having someone else do their work assignments off-hours.

Honestly, it kind of worked for them. I was very naive about it and it took me a long time to realize that the reason both co-workers could not discuss technical issues at work (or "their" own code) was that they simply didn't know anything (and it wasn't "their" code). As far as I know, only a very small number of tech people suspected anything.

You are at UCSD - surely they have co-op and internship recruiting programs on campus? When I was an undergrad (longer ago that I want to admit), I did several co-ops and a summer internship. These programs are often hiring pipelines for employers. All my job offers, even after doing a MS in CS after my BS in CS, came from companies I had worked for already as an undergrad.

Also, (and I have suffered for NOT doing this better), reach out to colleagues or even recent UCSD grads from the CS program and ask for referrals. Just be low-key about it - you aren't asking for any "big" favors or nepotism-like stuff, just maybe an internal referral to their company? I think people should do this more often. Employers often offer current employees a bonus for every hired referral so those folks may get something out of it too.

Lastly, submitting resumes online seems to often be a blackhole of nothingness. Because I'm not Mr. Professional Network I have almost always applied using online submissions to jobs and haven't always had a good response rate with that method. But my longest (and current, maybe forever) gig came from a referral from someone in my social network.

[EDIT]: I noticed one of your other comments that implies you could maybe graduate sooner rather than later. This may be a controversial idea but think about taking LONGER. Double major or really think hard about getting into some big name company internship programs.

You aren't leaning on your story so much as you are demonstrating successful experience navigating a world that isn't laid out in a way that made it easy.

Retiring from the military? I expect you know deeply how to navigate a bureaucratic system with sometimes frustratingly weird rules. (translation - can work through grad school institutional requirements without hand holding)

Possibly at least partially funded by outside sources? (translation - wait, I don't have to pay the full cost of the student? Has healthcare already? FREE FREE FREE labor to advisor)

Earned undergrad degree while working full time and caring for a special-needs child? (translation - nothing here is going to knock this person off course)

Truthfully speaking, almost everyone has a good story to tell. People are generally interesting to me and I think to others - especially the type of other people you would probably prefer to work with.

Maybe I am being overly optimistic about it but maybe remember that grad school is not a magical place where only super-geniuses tread the halls (internal opinions on that may differ </slight sarcasm>).

I don't have any experience with universities that have such a competitive aspect to entry into specific fields of study for students.

My gut is that is may be very good for the university reputation if the university is already pretty selective on admissions. But pretty bad for even a better than average, but not top %, student who has their heart set on a specific major.

I have taught as an adjunct and was impressed by the average undergraduate at a large state school. As a person who very much doesn't like an entirely "winner takes all" mindset in life, I'm curious about in the extra pressure of getting into your preferred program.

Mind sharing your thoughts on how that extra internal selectivity works out for students? I mean, clearly students complain about the process if it doesn't work for them individually, but what do those students mostly do? "Settle" for another program? Transfer?

Your chances of admission are probably significantly better than you think.

I did my BS in computer science at a mid-lower-tier state school - it wasn't like I was all-world Stanford/MIT/CMU or anything. I did a bunch of GRE practice tests and have always been a very strong standardized test-taker so I had that going for me.

I also had cultivated strong references at my undergrad institution. My references were professors I had multiple classes under and that I had actually spent time with during office hours. You may have those sources available to you but I would guess that a decent fallback might be former commanding officers?

The school where I got my MS also had stories about active duty Air Force students coming in full-time for a PhD, mowing through the check-lists of to-do work, and finishing their PhD in the three years the AF gave them to do it.

Your military background may be a strong point in your favor. Also, I vaguely remember that military retirees often had some tuition assistance programs. Grad schools LOVE students with external funding sources. Look into this.

Lastly, it sounds like you have a good story to tell. Tell that story in your application letter!

I applied to and got into a bunch of good universities to study for my MS in computer science.

My biggest mistake, BY FAR, was not visiting all those schools, talking with prospective advisors and current students, and using my gut instincts on those conversations to guide my decision.

I picked the school that I felt was the best but wound up poorly matched with an advisor. I had been a very proactive and driven undergrad (returned to undergrad in my late 20s) but reverted to a passive approach in grad school - it was a terrible mistake. By the end of my first year, I was completely demotivated and very unhappy. I turned it around my second year by dropping my advisor and taking some control in the process back. But I also totally punted on a PhD and settled for a MS. But it left me very unsatisfied in the process and in a position of asking "what if" in many ways.

One big part of why I was dissatisfied is that in hindsight, many of the professors I interacted with were simply not very friendly or considerate of students. Maybe they were used to being able to be more dismissive of undergrads coming straight to grad school, but as an adult I expected adult interactions. That experience was partly my fault - there are always people who are prickly to deal with. I should have been a little more self-directed about making sure I was working with better people even if it meant maybe doing thesis or research work in an area that was slightly less interesting to me personally.

FTA "... all university rankings are essentially worthless. They’re based on data that have very little to do with the academic merit of an institution . . . "

While I strongly believe that a motivated university student can maximize their ROI at any institution, it would also be nice if there was an easier way for prospective students to identify schools that would offer the best opportunity for students. But campus visits don't tell you much about actual classroom teaching experiences from the learner perspective.

Any ideas out there?

When I talk to high-achieving parents of kids picking schools, the parents seem to say that choosing a school was mostly a matter of coming up with some arbitrary evaluation function (eg, "wanted a small school with a few hours of home", "interested in a big flagship state university", "wanted an Ivy", etc).

Sounds like you have several experiences as a Clojure job seeker. Has that been driven more by your personal Clojure network or kind of scatter-and-spray applying to jobs that list Clojure as a possible skill?

I ask because I occasionally look at Indeed or LinkedIn to see if I can get a feel for the current Clojure hiring market. The results of those searches seem to be trending downward for me over the last few years. Which doesn't mean there are fewer jobs than before but makes me question if more folks are Clojure job searching on various Clojure-focused resources instead.

There were only 658 responses for Clojure out of 35,695 respondents - see (hover over the data points - for example, Java had more than 10,000 respondents):

  https://survey.stackoverflow.co/2022/#section-salary-salary-and-experience-by-language

My guess is that many of these 658 Clojure respondents are highly experienced engineers. I bet that if we took the raw data and looked at say, the top 1,000 Java respondents by experience that most of the salary differential you see would probably disappear?

I think "niche" language developers may be not only experienced engineers but are also be the type of engineer who is actively seeking out "better" ways of building software. In other words, a programmer seeking to improve their craft and to think critically about the tools they use for software development. This type of self-analysis and self-criticism probably leads to being a better developer. Maybe similar to musicians? I think we all know musicians who can play a few songs from what amounts to muscle memory but we all also know a much smaller number of musicians who are deeply creative and skilled - in my experience, the primary difference between the two types of musicians boils down to time and some form of deliberate practice.

Stack Overflow makes the raw survey data available via CSV. I'm going to download it myself and dig a bit - others can do the same.

Oh, early in my career I was very hung up on the idea that specific programming language skills were the key to making that higher pay. I now believe that being highly paid has very little to do with specific programming language skills beyond some relatively low floor of competence with the language. There is some luck in making a high salary but generally you have to work for an org willing to pay high salaries, be visible to people making salary decisions, and then make actual valuable contributions. Of course, there are also people without those results who climb the salary ladder by simply understanding the organizational politics of how to move into higher paid positions.

This is a super-interesting question.

I don't think it is quite the answer to your question, but I have seen combined heatmaps/dendrograms that feel like a possibility for this? Maybe the heatmap deals with activity between nodes and the dendrogram handles the community cluster relationships? I see them often in bioinformatics

Ex: https://en.wikipedia.org/wiki/Dendrogram#/media/File:Heatmap...

Years ago, I used some personal projects and a database certification to make some desired job changes. I advise any person to leverage both training (traditional education, certification, etc) along with a demonstrable portfolio of work to do the same. Even now, I occasionally do a MOOC and earn the certificate just for personal interest.

I hope hiring orgs don't run away from candidates simply because they undertook a certification. But I understand the caution - my org has had bad experiences with coursework-only MS in computer science career changers who turned out to not have any real skills and maybe not even any desire to learn any real skills.

A fun essay that really tries to nail down the "Why" in a real-world way.

It mentions that functional programming removes the tripwire of state mutation as one of the "whys", but my experience slightly differs here.

I found, after years of struggling with the "why" of object-oriented programming and its early obsessions with design patterns, that functional programming simply resonated with me personally. To this day, I have a bit of inner cringe when reading (for example) Java code.

Early in my career, I thought my lack of "getting OOP" was due to some profound missing piece of understanding. But then Lisp (and Clojure) just made sense. Thinking functionally just works for me. It's not so much that I couldn't use OOP but just that each time I worked in deeper OOP projects, I felt some internal friction or vague uneasiness with the code.

I did find that functional programming made me significantly more comfortable with OOP languages. YMMV.

I did side consulting at $70 for someone getting a business off the ground.

After the company was more up and running, I switched to a flat $1,000 per day to avoid being asked to do too many "small" tasks.

A candidate who can't communicate effectively seems unlikely to be of interest to you, right? If their communications don't clearly tell you about their past experiences and skills relevant to your needs, I don't think you need to overthink this.

When I am on hiring committees at my org, I always come up with a simple rubric to screen applicants and narrow down who we want to include in the process going forward. It is a very simple rubric, usually just "Yes/No" answers to questions related to the needs of the position. We can usually use these rubrics to stack rank applicants and interview top candidates.

We do get some simply terrible cover letters or email communications from candidates (both native and non-native English speakers). These are usually so bad that the candidates in question don't get past the rubric anyway . . .

I don't get it - conceptually isn't a site/service provider selling the ad space to the highest bidder already? Isn't the goal to max out that revenue?

Maybe allowing your "best" consumers to opt out of ads makes overall CTR drop to some level below which advertisers will no longer want to use your site.

Any insight into numbers for this? It is way outside my knowledge.

I'm guessing, for example, if I said to you - hey, I like your site and will pay $1,000,000 a year to see it without ads that might be a deal you would take?

I don't think people will pay for content. And the ones who will pay for content may be overwhelmingly attractive targets to advertisers in general? So you will never get any support from platforms for your "subscription ad-space buying service."

But Youtube has something like 20 million premium subscribers - I'm one myself with a family plan. But that is against an estimated 2.6 billion active users. Like, 1%? Round up and call it 2%.

I wonder how many of those 2% of users are the most attractive and desired targets of advertisers?

OTOH, if you said to me you had a service that would bid like 1% more for the adspace on any site or service I consume and that $X00 per year would remove all ads from those sites or service I use . . . I'd buy that.

Putting on my blackhat for a minute. I'm a creator with some small percentage of consumers "opting out" via this type of tool - how can I buy ads on my site/service at an inflated price to optimize my income . . . if I know 2% of my users are willing to pay X+1% shouldn't I try to drive up X? Arbitrage?