I discussed some related soft skill tactics in this post - for the specific problem of managing technical respect for a specialist team (in this case a machine learning team).
- https://managingml.substack.com/p/the-myth-that-machine-lear...
4,165 karma · joined May 20, 2018
- https://managingml.substack.com/p/the-myth-that-machine-lear...
It’s mind-blowing that you see a 6 month investment in just doing basic leadership 101 as a massive bureaucratic hassle. I think it’s safe to say that the low performer you’re referring to is not actually the real problem.
It’s perfectly fine (good even) to work in a large company where your value add is not immediately clarified and is just part of a large agglomerated process. Most net benefit to consumers occurs this way.
It’s also fine if you want to work in the start-up manner, but it’s not “better” or “more correct” or anything.
The author is taking it to a deeply unreasonable extreme that shows more about the author’s arrogance than about any sincere or earnest desire to help customers.
The degree of behemoth incrementalism is so extreme that outside the immediate blast radius of your work, there is no serious, intellectually honest way to connect the dots between your effort and the user. It’s like shipping a rover to Mars. You do a shitload of work and hope several time units later when it actually lands, that your work has some barely perceptible positive contribution to the sum total outcome.
And all this is even worse in companies where it’s not clear to anyone whether the user is the product or the customer. Just imagine that. “How did you help the user today?” should make you freeze like a deer in headlights. Do you mean the ad company who can target Starbucks-drinking soccer moms in Texas better now because of our cool new image filter for posting kid sports pictures? Or do you mean the soccer moms themselves? Or do you mean the NSA we are allowing access to all this data on both the advertiser and the soccer moms? Or do you mean the VCs?
Leaders who say platitudes like “what did you do for the customer” are just extremely arrogant know-nothings. They can use the stick of “the customer always comes first” to indiscriminately beat, shame or fire anyone that they conveniently need to attack, no matter how unrealistic it is to demand this kind of direct customer impact accounting.
Employees aren’t braindead morons who subscribe to your company like a religion. But that is exactly what this type of thinking is meant to induce.
Ugh, this refuted myth again?
The “insecure, bad leader who blames subordinates for their own failures” meter is going off the charts with this one.
My forecast is that prior to the pandemic there were X “FAANG-paying jobs” across ~5 big cities in the US. After wider remote work normalizes in the next two years, there will be 10X that number of jobs and the average wage will likely only go up.
There will be room for all the headcount there currently is across the ~5 largest tech hubs, plus nine times that amount in additional, new headcount. People from all kinds of regions of the US and the world will have much greater chances at these jobs. And it will create spillover effects where lesser tier employers have to raise wages to compete for the remaining candidate pool and so on.
In mid-sized cities and large urban areas, the police are generally extremely corrupt and integrate with high level politicians and corporate interests for bribes, carrying out policing actions to serve cronyism, etc.
In small towns, police typically align with small town politics and specific locals with political power, ranging from judges to attorneys to business owners. If they decide you’re on the wrong side, they’ll find reasons to harass you, use speeding tickets, littering, frivolous fines, etc. Of course this often affects underrepresented groups, but can easily affect people in the majority if they just don’t like you.
Most communities see the police as a group to strictly stay away from. You hope that in a real emergency they will do their job, but more often they are just authority theater looking for a way to jam you up or exploit you. They absolutely are not good natured community members by default, and even if some specific cop is trying to do right by the community, nobody can assume that is trustworthy and everyone has to just stay away.
Policing, especially the procedural mechanics of fining or arresting people, is just fundamentally broken from first principles in the US, and most police systems exploit this for corrupt gains and abuses of power. Exceptions are so rare as to be ignored and assumed irrelevant.
I hope people will read it, because a direct reading of it does support my position and does not support your position.
If the competitive wage rises but a company can’t afford it, that’s not the worker’s problem to accept a bad wage to help the company. That’s the company’s problem to hire better leadership or cut executive pay in favor of employee pay or get acquired / raise funding or go out of business. None of that changes the reality of competitive wages.
At the root it sounds like your question is more about aggregate market allocation of labor. But why should workers accept inefficient allocation at worse wages? That just seems like an aggregate market expression of “these businesses are unwanted by society” - the same happens to 90% of startups all the time.
Ultimately your business’s total revenue is a function of external demand for what you sell. Within that limit, some employees have a big impact on realizing that revenue, some have a small impact. What they are really worth to you is their share of contribution of revenue by that impact.
What I have said is that virtually all companies employing software engineers experience this impact from software engineering in similar ways (meaning the engineers’ value is much closer to their direct impact on revenue than some lower “market rate” wage floor).
It’s frustrating that you are misinterpreting this as me saying other companies earn the same profits, in absolute dollar terms, as tech companies. Because I never said that at all - and more importantly, that does not need to be true for my earlier statements to be correct.
A company doesn’t have to make the same profit as FAANG to justify that the fair value of one of their software engineers according to their revenue contributions within the company’s (much lower than FAANG) profits is already at a FAANG salary level (which they already have the ability to pay).
Remote work just takes away the greedy rent seeking mechanism (local “cost of living wage” excuses) that had been used to artificially suppress what should have been a FAANG-level wage (paid out of the current revenue) all along, based on the way the company leverages software engineers for their contribution to existing, current profits - not any hypothetical ability to generate future FAANG level revenue per employee.
May I ask where that is? Because it could not be any location in the US.
Governments are deeply incapable of stopping even just basic crimes and fraud. If they can’t stop overtly criminal acts often committed with no technology to assist in obfuscation, why would anyone believe they can (a) detect network traffic somehow related to Bitcoin and (b) trace it back to an individual and (c) prove that individual has broken some law? Let alone to do this to perfectly law-abiding citizens conducting perfectly legal purchase of Bitcoin.
Your point is very bizarre to me. Are you suggesting a government so incapable of managing fiscal activities that it descends into hyperinflation crisis is somehow magically super competent at extremely bespoke and difficult internet traffic surveillance of legal cryptocurrency transactions they have no reason to need to invest in expensive capabilities to monitor in the first place?
People seem to think that opening up India as another source will drastically lower wages, but I don’t see any reason at all that this will be true - and if the reasoning is just “more supply generally means lower prices” then you’re totally wrong and supply & demand 101ism should just get straight ignored in this conversation.
Hiring a competent and trustworthy software engineer, even at entry level, is extremely hard and expensive. For every 100 computer science applicants, maybe 15 are hirable, 4 are good, and 1 is sincerely worth it.
You can increase the candidate pool but that rate doesn’t change. And if the sources of education and training are substandard in other markets (only a fraction of Indian students can get education or training on par with mid-grade or higher Western universities), the rate of acceptable candidates is even lower. Factoring in the overhead of visas, it is just super unlikely to result in widespread wage reduction.
Besides that, there is still employer competition. Employees don’t typically bid on jobs by quoting lower prices to undercut peers, especially not in a collaboration-based industry like software. Rather, employers compete.
A lot of companies would gladly double their engineering headcount even at San Francisco wages, if only they could actually find that many acceptable candidates.
I admit these dynamics could change. For example if the fraction of computer science job seekers who are actually capable of doing the job at a minimally acceptable level grows much higher than ~20% then perhaps replacements will be cheap and common.
But right now the industry is not at all like that. I welcome remote workers from all over the world. If they are good enough to get hired, then they deserve a wage close to their impact on revenue just like their SF or NY counterparts. I don’t fear this will cause wage depression for me one bit. It will give them access to better quality of life they have earned, and will make our teams more culturally diverse and globally minded. I think all around it’s only a good thing.
The “least they can pay” floor is just being moved up closer to the fair level indicated by the actual contribution to revenue.
Remote work just strips exploitative regional employers from their many decades of greedy free lunch setting the “least they can pay” floor much lower.
The other major problematic section is on energy use and unqualified comparison of units of large model training with carbon emissions of other activities (with no attempt to account for societal value of ML model research or training, nor economies of scale as training technology democratizes it and makes it cheaper). It is similar to the incredibly specious diatribes from Stephen Diehl lately balking at the raw energy numbers of Bitcoin mining, as if those numbers mean anything or relate to any conclusions on their own.
I wrote up a lot more of my thoughts on Gebru’s bad article when it was first leaked. Here’s that comment:
- https://news.ycombinator.com/item?id=25314824
I also feel you are very commonly involved in lengthy downvote / flagging threads with disingenuous takes on others’ comments and deliberate unwillingness to apply charitable takes or avoid taking things out of context. I am sure from your POV you don’t feel that way and I’m not asking anyone to agree with my assessment; I have zero goal of citing things to back it up. I am just going to preemptively drop off from replying to you any further.
They are just using rent seeking opportunities to disingenuously pay uncompetitive salaries and reap surplus for executive pay.
Either those companies will get better management and leaders who successfully rebuild the business model to account for this, or else executive and manager pay will go way down to reflect the reality that competition for high leverage software talent takes away the cushy executive surplus, or else the market in general will deem the company to be unwanted and not useful and it will decay out of business or get acquired.
The company has no right or expectation for things to stay in the current unfair state. The “cost of living wage” free lunch is ending, so adapt or fold.
The premise is: why do we care about some forms of inequality (financial inequality among households in one nation, or protected class inequality within employment law in one nation), but we don’t care about other things (inequality between socially lonely people and social butterflies, inequality across country boundaries, inequality of physical attractiveness).
Hanson suggests it just so happens that we develop “morality” around some forms of inequality because it offers easy ways to politically coordinate to grab money or power and punish your rivals.
Other forms of inequality that involve just as much harm, but where it’s not politically easy to drum up support for physically taking money or power, just happen to conveniently not turn out to be big moral concerns, or may even have other competing morals invented over time that make it “wrong” to even treat it as inequality (like the hallowed status of a “right” to control what happens to your body counting for more than inequality based harm of variation in genetic sexual attractiveness outcomes - which is a massive double standard compared with no such “right” to control what happens to the fruits of your labor or intelligence, which undergo forced redistribution via taxes because we just so happen to believe inequality of financial outcomes is not acceptable).
For example, most people don’t give a shit about animal suffering on the scale of agribusiness. Most people don’t give a shit if ugly people suffer loneliness or unhealthy lack of sex or job discrimination from their appearance. Most people don’t care much about structural inequality in countries besides their own.
These are just convenient moral omissions because in each situation it is not easy to see how you could “redistribute” attractiveness, social companionship, or power structures of some other country. Plus animals don’t have any money or status to give you in exchange for allyship, while big agribusiness has plenty of lobbying power to crush you with.
I wish more people appreciated this. We don’t, as a society, advocate for race, gender, or pay equality because it’s “right.” We just know that aging white patriarchy is full of convenient targets we can take money and status from. Conveniently for other domains of inequality, like animal welfare, where no such easy targets exist to take from, it just somehow doesn’t qualify as a moral issue worthy of advocacy.
[0]: https://www.overcomingbias.com/2013/08/inequality-is-about-g...
I think this is wrong. This is what ML researchers tend to think when they are ignorant of why DevOps best practices exist and what other teams need in order to provide underlying infrastructure support. You’re only thinking of “what you need” from the point of view of the developer experience of the ML engineer, which frankly is usually the least important part by a wide margin.
It’s just a hype vehicle to cram in whatever social justice outrage du jour that some liberal source wants to push as an agenda. I say this as a liberal who is sympathetic to most of those issues, but finds their representation in AI ethics or fairness debates to be ignorant and juvenile power grabs.
The whole Timnit Gebru affair is a direct example. The paper Google didn’t want to approve for publication was really poor. Lots of specious arguments. Lots of declarations of opinions as if they were fact (eg. nobody is required to agree with or accept woke vocabulary, let alone ensure NLP models are kept updated with it), and lots of sanctimonious assertions about various agendas that nobody has to agree “are right” and have zero appropriateness within a science publication.
Software is fundamentally different and the revenue impact can be multiples or even orders of magnitude higher, in really any industry or company type.
Software engineers, even in Silicon valley, are underpaid relative to revenue or profit contributions.
Paternalistic and entitled employers in other areas have milked the opportunity for rent seeking by suppressing wages due to disingenuous cost of living policies for a long time. Finally workers have options that let them say, “no, I just won’t take your poor pay job. I don’t care that you think someone in Kansas City or Wyoming should earn X% less for doing the exact same job, and I won’t accept that anti-worker paternalism. Improve your offer or get lost.”
I’ll give a concrete example. In my org we use Dataproc on GCP as a model training task execution paradigm. You define your base environment via some Docker container, put it in GCR, and then define Dataproc jobs in terms of the base environment, the backing compute resources, any GCS bucket connections, and any ML-specific config like hyperparameters.
A human being never under any circumstances triggers these jobs. Instead a human user deploys the config as a cronjob or regular job in Kubernetes, and then a scheduler picks them up and runs them. For experimental workloads only, developers can manually trigger a Kubernetes job.
Each job consults config, spins up the appropriate Dataproc cluster, runs the job (with visualization tools exposed on ports at the cluster node IPs), and saves artifacts to GCS when done.
All of this is controlled via clean and easy internal CLI tools and wrappers to make it simple for any developer.
The number one thing this ensures is that no work ever exists in notebook format, beyond tiny scratch work a developer might do strictly to debug code or try a small data proof of concept.
The number two thing this ensures is complete reproducibility. Since every possible training task must go through code review, commit all config to version control, get impounded into a container, and execute via a deployed Kubernetes job, it is by definition impossible for someone to execute an ad hoc task that other engineers can’t rerun or have to follow weird setup steps to recreate (it’s all impounded in the container).
The third thing it ensures is that all accuracy, monitoring and results artifacts are explicitly tied to the Kubernetes job that controlled the process. It is not possible for some accuracy result to float around untethered from a job ID that uniquely and conclusively ties it to all relevant code for the job. This can be facilitated through MLflow or whatever else.
Getting data scientists to “wear corrective shoes” and learn to reorient their way of working to align it with this process has universally paid dividends, both for letting the data scientists experiment faster and more reliably, and for ensuring model training adheres to SRE-related compliance and best practices, so it is pluggable into various tools and constraints those non-ML support teams need in order to do their jobs and offer support to ML teams without getting hit with unstructured notebook spaghetti and bespoke execution paradigms.
I’d also like better integrations with alternate cloud storage providers, but this is true for Apple products too.
Combine with the fact you can literally ssh to your Remarkable and write your own apps, it’s a no-brainer.
You have it backwards. Silicon Valley salaries are not “inflated” nor are they unreasonably high. In fact even at large FAANG companies, engineers are underpaid relative to actual revenue contribution, it’s just that competition drives those companies to pay closer to a fair value wage.
Across the US, engineers are massively underpaid due to specious rent-seeking policies by employers who tether pay to cost of living (treating it like an allowance for employees instead of an earned wage).
This change where remote employees can earn the same high salary based on their value instead of being a paternalistic allowance based on their employer’s opinion of where they live and what they should be allowed to afford, it is a good thing, because all these rent-seeking companies acting like vampires on low-wage suppression due to location are going to have to radically change, get rid of their paternalistic attitudes, and start being fair with employees.
A manager should not be making tactical implementation decisions - they should trust their team to make those decisions, and offer higher level architecture & product guidance about the problems worth solving, the context and the constraints. Managers should be somewhat like 1/3 principle engineer, 1/3 product manager and 1/3 personnel / recruitment / rewards manager.
The absolute worst is when a company very openly embraces “team lead” or “player coach” roles, which are essentially an unspecified dumping ground of any arbitrary, flailing responsibility from any domain, with no clarity and with unrealistic expectations of “wearing many hats” and getting no staffing or team growth yet still being expected to deliver high project throughput and still somehow manage personnel issues, compensation, team building and so on.
“Team lead” is a bad cop out by senior leaders who are too weak to defend org structures they designed. They want to have their cake (take credit for constant reorgs) and eat it too (drop in to any part of a reorg that’s not succeeding and arbitrarily redefine a team lead role to mean whatever they want it to mean at that moment, for political convenience).
Even though it might be fun to still do coding, your life as a manager is so much better if you refuse anything like “team lead.” Don’t do it. Only accept roles with super explicit clarity that your job is to be a manager, not an implementer, and require a company culture that really backs this up and affirms it.
- https://managingml.substack.com/p/ml-model-training-is-an-et...
As an ML manager one of the things that I dislike about the Databricks model is the reliance on notebooks and bespoke interactive cluster workflows.
ML workloads are stuck in the stone age because there’s no common pattern to plug them into existing ETL frameworks, but really that is what they are.
Model training is an ETL that just happens to need unique domain tools to visualize how it’s succeeding / track observability metrics. But apart from that, you really need to automate model training and experimentation.
It is a total antipattern to ever use notebooks, or even non-notebook code developed interactively. Rather you should be submitting tasks to a task queue that maps your ML exploratory workload or training workload to a scheduled compute resource, runs it with observability baked in, and treats outputs like schema’d outputs of more traditional ETLs.
Instead of that, Databricks is much more of an early 2000s MATLAB model. Make it addictive and easy for unpragmatic researchers detached from real production use cases, then figure employers will have to adopt it since all the expensive-to-hire researchers can only be productive with it.
Long term I think it’s a very bad gamble. Just consider how much open source Python tools have eaten MATLAB’s lunch in the past 10 years.
Anyone actually talented just leaves after 2-4 years, so you get zero cohesion or snowballing effect of talented tech leaders who other good people actually want to work with.
These places become total career death. Eventually it becomes a place where only people with kids or other significant life obligation go, specifically because they value no-ambition bureaucracy that trades off compensation and risk taking in exchange for work life balance.
Those places are so soul crushing. Avoid at all costs.