mlthoughts2018

> “ If your heavyweight runtime is being launched 1000s of times to get a job done every day or multiple times a day, consider optimizing. Which may include changing the language. That's hardly controversial”

No, that is controversial because the time saved per run (even 1000s of times per run with multiple daily runs) is never going to come close to amortizing the upfront sunk cost of that migration and future maintenance.

I’m specifically saying in the exact case you highlighted, people will short-sightedly think it’s a clear case to migrate out of the easy-but-slow interpreted language or never start with it to begin with, and they would be quantitatively wrong, missing the forest for the trees.

> “ Programmers need to think long and hard about your process invocation model. Consider the use of fewer processes and/or consider alternative programming languages that don't have significant startup overhead if this could become a problem (anything that compiles down to assembly is usually fine).”

This is backwards. It costs extra developer overhead and code overhead to write those invocations in an AOT compiled language. The trade off is usually that occasional minor slowness from the interpreted language pales in comparison to the develop-time slowness, fights with the compiler, and long term maintenance of more total code, so even though every run is a few milliseconds slower, adding up to hours of slowness over hundreds of thousands of runs, that speed savings would never realistically amortize the 20-40 hours of extra lost developer labor time up front, plus additional larger lost time to maintenance.

People who say otherwise usually have a personal, parochial attachment to some specific “systems” language and always feel they personally could code it up just as fast (or, more laughably, even faster thanks to the compiler’s help) and they naively see it as frustration that other programmers don’t have the same level of command to render the develop-time trade off moot. Except that’s just hubris and ignores tons of factors that take “skill with particular systems language” out of the equation, ranging from “well good luck hiring only people who want to work like that” to “yeah, zero of the required domain specific libraries for this use case exist in anything besides Python.”

This is a case where this speed optimization actually wastes time overall.

Docker / containers are necessary but not sufficient. For example, in a machine learning CI / CD system, there could be a fundamental difference between executing the same step, with the same code, on CPU hardware vs GPU hardware.

Many CI systems try to strictly enforce hermetic build semantics and disallow non-idempotent steps from being possible. For example, by associating build steps with an exact source code commit and categorically disallow a repeat of a successful step for that commit.

Isn’t Drone also an example of the author’s ideal solution?

[flagged]

I do agree with the consistency thing. One bad culture failure mode is a case where one or two senior executives run amok enforcing their view of culture, while the CEO fails to enforce consistency and a bunch of other executives, directors, etc., just try to “stay out of it” and remain culturally neutral (which isn’t really possible).

I saw this in one company where it was the CTO running amok with an aggressive culture of yelling in meetings, slamming doors, and emphasizing arbitrary deadlines.

I saw it in another org where it was the CPO instead, trying to install Dilberty consultant snake oil with constant reorgs and zero accountability for product managers.

It’s really sad that you are blind to your own severe discrimination. If you think in sweeping generalizations like, “leftists make the mistake...” you really need to step back and spend time not commenting at all and work harder to gain more self-awareness about your own prejudices.

I know you will not like this comment and you will want to knee-jerk reply to “refute” it, but you need to resist that urge and admit you have really significant and really troubling prejudices that come through like a megaphone to others observing you, and spend time just dealing with that.

That you feel you are identifying solutions, or even a definition, not understood or pursued by people creating things like microinequity training, just reveals your ignorance.

I think my earlier comment which you described as a “no u” comeback was actually really apt and crystal clearly accurate based on your follow-ups.

sounds suspiciously actually like you don’t know what systemic racism is...

Posts that masquerade as “legitimate dissenting opinions just not getting a fair shake from the left by golly” but which really seek to undermine structures by which systemic racism can possibly be called out and held accountable for harm (like the framework of microaggressions) deserve to be flagged and shut down. They don’t serve any free speech or intellectual honesty purpose to preserve discourse and the psychological safety to disagree, and they are roundly disingenuous tools for filibustering actual legit discourse and progress. It’s just taking crass tools of hate groups and dressing them up with five dollar words and armchair discussion of academic freedom of expression, when really it’s just hostile noise jamming against progress.

Thankfully this getting flagged is still routine. Can you imagine how sickening a place this would be if this kind of post didn’t get flagged? It’s horrific, yet parading around as if it was merely legit discourse that is snubbed unfairly by the left.

Robust potential functions are a huge part of ML. Many people have researched robust potential functions for use as loss functions. Its use in ML algorithms predates use for SLAM, tracking algorithms, etc., which only used it after classical ML.

Neural nets typically don’t benefit much from it because you can use batch normalization, dropout and clever activation functions to achieve the same results, by having the network learn diminished sensitivity to outliers that produce neurons which saturate the low end of an activation function.

This is preferable because many of the robust potential functions involve absolute values, order statistics and other non-differentiable quantities that are hard to put into backpropagation-based optimizers. You almost always would need to relax the loss function to something that trades off smoothness against outlier robustness, where convergence will be slower and slower as you crank the trade off closer to outlier robustness.

I don’t see any reason your comment would be accurate. It all depends on trade-offs. If the goal is to reduce cost, then you could equally argue to get rid of medicare entirely.

Clearly the goal (no matter what anyone’s separate normative opinion is) is to balance some complex tradeoff between costs borne by tax payers, costs borne by corporations (through taxes and through employer based healthcare for working age adults and their dependents), and a high level of access for all people.

Nothing about this censored data effect can say anything about the morality of different regions of that trade off space.

As to your second comment, this article is not massive news and it seems laughable to say it is. It’s just a blip in the news cycle, using some data artifact to drum up attention to something that is already well-known for any econometrician or health policy analyst.

Full disclosure: I personally favor nationalized medicine and welcome higher taxes across the board. Nonetheless I don’t find your comment to be accurate or valuable.

This has been discussed widely in computer vision for many decades

- https://www.routledge.com/The-Ecological-Approach-to-Visual-...

- http://fmdb.cs.ucla.edu/Treports/soatto_extended_v18.pdf

There would almost surely be a censoring effect no matter what age cutoff you pick, and even allowing for typical government incompetence, surely at some level health econometricians are involved in this type of policy and are aware and facilitate whatever trade-offs are being sought.

I love roasting poor government policy decision making as much as the next person, but that would be a bridge too far.

Companies don’t pay financial rewards for doing the data plumbing work, yet it’s often much harder, prone to unusual failure domains mixed with high scaling, and carries more stringent on-call and incident triage responsibilities.

Given that it’s (a) more difficult, (b) more business critical, (c) more stressful and requiring an incident alerting on-call rotation, then “data work” should be much better paid and offer job security and career growth.

Yet no company I know of pays expert modelers & researchers less than expert data platform engineers.

So either the companies know something you don’t (e.g. that data platform work is more commodity and easier to replace than rarer modeling talent) or there’s a free lunch you can get by exploiting the arbitrage opportunity to pay data platform experts more and consume correspondingly higher business value that other orgs are missing out on by putting modeler / researcher higher on the status hierarchy than data platform engineer.

My perspective after many years of experience managing machine learning teams (both platform/infra and research/modeling) is that data platforming is just a worse job. It’s unpleasant and stressful and business stakeholders who are removed from backend engineering complexity and just want the report or just want the model couldn’t care less about organizational structures and workflows that support healthier lives for intermediate data platform teams. Because of this, the pay and bonuses for data platform roles should be much higher, but politically speaking it’s impossible to advocate for that, so it becomes a turnover mill where everyone burns out to keep the existing shitty system running, with comparatively low pay and low autonomy, and so nobody ends up wanting to join that team or do that work.

There are so many issues besides just saving for a downpayment.

How will you keep a job that supports the mortgage? Companies are exceedingly disloyal and looking to cut labor costs whenever possible, whole industries go through huge labor contraction, not to mention accumulating effects of automation.

Even if I saved a downpayment, I would not feel comfortable entering into a mortgage like this at all. It just adds stress to your life and you get none of the independence or peace of mind you expect to get from private property ownership.

I’m only getting 503 varnish cache errors when I try to actually visit any artworks through this.

Your comment is exceedingly political. It’s unfortunate you mistakenly feel you are politically neutral and that you represent “actual” free speech, whereas companies who don’t choose to render services to insurrection-plotting fascists and racists apparently only practice “convenient” free speech.

I hope you can step back and evaluate the mistake in your perspective. Number one, you are demonstrating that you care more for politics than for free speech (let alone the fact that your viewpoint literally leads to people being harmed in preventable ways where the prevention methods don’t threaten free speech). Number two, you are trying to define the conversation in a way where you are neutral politically and merely hold up an ideal principle, but that’s very clearly false to huge degree and you’re not admitting the political bias and blinders you are yourself bringing into it.

Generalization to previously unseen examples is one of the core components of ML models.

I think your perspective is actually way off the mark here. AI excels at long-tail problems where the cost of failure is high, precisely because human failure is such an expensive problem in those cases and the nature of long tail problems prevents it from being possible to apply QA to every use case. In other words, you know you are forced to deal with getting it wrong a lot and paying the high failure cost, so using a system capable of optimizing that trade-off explicitly is often much better than pretending as if a human in the loop is somehow sparing you the failure costs when they aren’t (and in fact they are simply less efficient than algorithmic solutions).

What constitutes a useful sequence of facts in root cause analysis is not just some platonic existing thing. It’s a complex problem involving mind-melting log sleuthing, correlating all kinds of disparate metrics, comparing against timestamps of merges and eventually synthesizing the results.

Even seasoned veterans who know systems inside and out struggle with the sheer volume of logs, metrics and facts to compile. And most of the time their approach is based purely on inductive experience with similar incidents combined with heuristics.

This is precisely the kind of problem that ML solutions excel at. It has many hallmarks of a good fit and almost none of the hallmarks of “solution in search of a problem” ML over-engineering.

“Root cause” is relative. To the CEO the root cause is “some engineering thing broke.” To the data engineer the root cause may be that a human config error led to a rogue process that caused a VM disk to fill up. To a quantum super-intelligence the root cause may be that in Everett Branch 2765425 atom 67896533 collided with atom 78532578.

(Just kidding. The space of Everett branches is not countable.)

You seem fundamentally confused, for example with tools like Cython.

Many extension module implementations in Python are literally as fast as pure C (not just nearly as fast with minor extra CPython overhead, but literally as fast as pure C by deliberately bypassing CPython VM loop and data models).

You are incorrect. This is only true if you can precompute all sort orders (many types of sort orders cannot be precomputed and depend on additional context data only available at query time, especially for personalization or trending solutions). Additionally you must deal with precommitment to very poor sharding properties with Solr. With our Python approach, we could hold all results (billions of content items with trimmed down data structures representing only the thinnest container needed for each sort order representation) in memory easily, and dynamically resort on the fly or double sort by multiple sort orders that each required contextual data only available at query time.

I worked previously for a very high traffic ecommerce company (Alexa top 300 site).

As part of the search team, I worked on a project where we deliberately rewrote the whole product search engine in Python and Cython, including our own algorithms manipulating documents for deletion, low latency reindexing after edits, and more.

We did this because SOLR was too slow and the process of defining custom sort orders (for example, new sort orders defined by machine learning ranking algorithms, and needing to be A/B tested regularly) was awful and performance in SOLR was poor.

It was a really fun project. One of the slogans for our group at the time was “rewriting search in Python for speed.”

The ultimate system we deployed was insanely fast. I doubt you could have made it faster even writing the entire thing directly in C or C++. It became a reference project in the company to help avoid various flavors of arrogant dismissal of Python as a backend service language for performance critical systems.

Maximum memory per instance (8GB) is an extreme limit. Disk and CPU limits per container instance are also quite bad.

And, laughably, for any workload just a bit out of reach for Cloud Run, GCP docs immediately recommend switching to GKE (and even Anthos).

Imagine having a high RAM workload that is just a simple RPC service. Many (probably most) machine learning services fit this model. Many routine ML models require more than 8GB RAM just to load the model, bit it’s a good use case for serverless non-lambda infra because it runs out of a very unique Docker image and does nothing but serve stateless model predictions.

Needing to bring in all the machinery of GKE or pay out the nose for Anthos just because you need exactly the same operational model as Cloud Run just with high RAM or CPU is a really poor customer experience that feels deliberately set up to push you towards more expensive Kubernetes products.

GCP has vastly worse documentation than AWS unfortunately.

The resource limits are so severely restricted on Cloud Run that I don’t think it’s fair to compare it to Fargate. The space of use cases solved well with Fargate is far, far larger.

Wow, looking at this thread, you appear to either be just a troll stirring up racism arguments, or else you are yourself a sickeningly extreme racist. It is just bonkers crazy reading some of your comments on a usually civil platform like Hacker News.