mlthoughts2018

> “ clients do not know what they are getting in advance, because they are looking for knowledge that they themselves lack. They cannot measure the results, either, because outside factors, such as the quality of execution, influence the outcome of the consultant’s recommendations.”

There is a ton of this in cloud computing as well, especially in the AI / ML space. It’s marketed as a production-ready, plug-in solution, on the hopes that the customer isn’t smart enough to evaluate what they are getting. It’s sold as a replacement for an earnest need to understand problems and solutions.

Just consider how many cloud vendors charge pay-per-usage for things like pre-built ML models. It makes no sense to pay by usage unless you have a very clear understanding of what accuracy and business impact of failures you’ll get at that usage rate (which depends on your product / customers / data, not primarily on the vendor’s managed model).

So either you just trust the vendor and you are flying totally blind in terms of the real accuracy or business value you get per dollar, or you have to go ahead and hire expensive ML experts to maintain that vendor integration and understand its business impact. But in no world can you just buy business value direct from the vendor without someone in the loop actually producing clarified understanding.

I think it’s the same with big box consulting. The customer wishes to offload political battles and status turf wars to an outside authority that can just tell them the business value answer without anyone in the loop actually achieving understanding or clarity.

It’s pure fool’s gold. The corporate equivalent of obsessively visiting psychics or tarot card readers.

I agree. I once worked for a company that wrote a blog post on reinventing and upgrading A/B tests, but the ugly truth was that the company couldn’t run an A/B test to save its life. Every A/B test was a disaster, there were fragmented different frameworks for A/B tests in different parts of the product, with inconsistent and unreliable data, and the core clickstream ingestion system that was the foundation for any possible way of testing would crash and go down for hours every few weeks, and product management would just silently ignore any effects of missing data or correlation between failures in different product features. Even though we had a team of statistical researchers, they were treated as if legit stats 101 concerns were just ivory tower academic hair splitting and often were silently omitted from being in A/B test design, recap or decision meetings, which were instead run by product managers with no stats training.

I remember interviewing a candidate once who said he was excited about the role because of that A/B testing blog post, and I just thought - geez what a completely soulless bait and switch ploy.

There is just absolutely no good-faith way anyone can disagree with requiring user consent to be tracked. It is as close to “pure evil” as you can get for Facebook to campaign against it.

Gotta mainline those photons.

I think one one interesting thing is that megacorps that seem to integrate big acquisitions (Amazon -> Whole Foods, Microsoft -> Github & LinkedIn, Facebook -> WhatsApp), all seem to be megacorps that don’t mind doing shit in a messy, scrappy way.

Whereas companies with a strong fixation on design & aesthetic seem to have a harder time. To fit into Apple, everything must look and feel like Apple. There could never be a new “Whole Foods” tab where things look and feel like Whole Foods. So Apple melts it down and recasts what it can into Apple - a lot like the Borg.

Certainly there’s a lot of credit due to “lessons learned” - but they key part is that the main lesson learned was to prioritize spot change flexibility over a “pluggable” model of extensibility enforced with a type system design and rigidity. Any smaller scale tactical improvements in code structure paled in comparison to that core property.

> “You say this, but then never address this claim in the text below. Could you expand on it?”

I believe I did answer this in my original comment, so I will just refer you back to that.

I disagree that there was any conflating going on. Type system designs enforced with static typing are a hallmark aspect of most of these design patterns around things like type classes, interfaces and inheritance. Of course similar things can exist in dynamically typed languages but they are not the same. For example “interfaces” in Python are just duck typing conventions (apart from built-in CPython data model properties). That duck typing interface is not at all similar to interfaces as a type system design pattern in a statically types language. Any similarity is purely semantic.

As for examples, one example that I worked on heavily involved a Scala system for managing DAG dependencies in task execution. The system was set up using phantom typing and a bunch of sealed case classes such that for any logical type of Task that could exist in a DAG, the task had an “Active” and “Passive” variant, where the “Active” variant could only be obtained through a monadic validator processing a “Passive” variant.

The goal was to use the type system itself to encode the concept of “this task has passed through validation and it’s allowed to be processed.”

Because this was designed at the type system level, it created huge problems and never added any real value in the sense of making it “logically impossible” to create invalid Tasks. Number one, it led to huge, painful boiler plate to create the case classes for every type of Task and specialize the type class with a “validator” function. Number two, we eventually realized there were many different aspects to “validation” that did not map well to the concept of “passing through a validator.” For example, some tasks depended on data that didn’t exist in the required location at a certain time, and hence needed retry logic to validate. Some situations involved re-running an already complete task (usually for resource usage observability reasons, or because an external data dependency changed). At any rate, baking validation status into a static type via the phantom type design was nothing but a headache. For all the beautiful code supposedly protecting us from processing invalid jobs, all that we got was difficult constant refactors.

Eventually we abandoned it and just used Luigi instead, and wrote all DAG management code in Python. It was the best decision we made. We lost zero safety and our defect rate did not get worse. Testing caught all the same bugs that compilation would have caught in Scala, and more, with less total code. And because the nature of the tasks in Luigi was just “whatever arbitrary Python you want” it was super easy to write effective validators, accommodate new use cases on the fly, and keep the code clean without dogmatic adherence to a precommitted type system design. Luigi happening to use some lightweight inheritance patterns was forgivable, given the dynamic typing flexibility.

Great to see the excitement and success story for this hardworking person.

Depressing to see leetcode and data structures interview prep heralded as a positive thing and treated as a neutral, accepted, “nothing wrong with this” aspect of the path to getting a job.

The “types do not impede, they help [clarity of thought / program structure / catching many important classes of bugs]” claim is hogwash.

I say that as someone who spent most of a decade doing Haskell and Scala professionally in large companies that built lots of developer tooling and workflows for them.

The most critical aspect of business software is to be able to drop into any particular local section of the code and make significant changes according to shifting business constraints. Anything that enforces a program structure that makes this harder to do, or requires a sequence of significant refactors around things like type class design / OOP interfaces and so forth, is strictly a loss for the business, not a win, even considering correctness, safety, a developer efficiency as critical success measures.

It’s often much worse than “a loss for the business” too, given that 99% of the time, those type class designs and OOP interfaces or nested inheritance models were premature abstractions and all the extensibility or well factored SOLID code (or equivalent ideas in FP) winds up being sheer debt that fails to be extensible in the ways that reality turned out to require but which nobody foresaw.

In a world with excellent foresight and ability to hit pause to refactor architectures, then baking in domain modeling constraints through type system designs would be great. Unfortunately that doesn’t map to the real world at all.

I think this extends outside of Python. Performance and safety are trade offs, not absolutes, and the balance of needing safety or performance vs extensibility vs ease of development may result in dozens or hundreds of different trade off needs in different parts of a single application.

One consequence is that it never makes sense to use static typing or compilation as application-wide absolutes for any language or paradigm.

You should virtually never be writing whole applications in Rust, C, C++, Java, Haskell, etc. It is a huge sign of bad premature optimization and dogmatism. Compiling subsets in these languages and then exposing them through extension modules in other languages that don’t force those constant trade offs is almost always superior, and it’s very telling about poor engineering culture when this results in debates or vitriolic dismissiveness from people with prior dogmatic commitments to static typing or AOT compilation.

No, it is not too much to ask in the specific case of Gebru’s bad paper. Several of the arguments are specious, like comparing the total energy consumption for training GPT with car trips, or demanding that NLP researchers have to keep up with rapidly changing activist “woke” vocabulary and ensure their models are respecting it.

These are ridiculous claims, and it’s fair to respond to them by saying, “well, what exactly do you imagine a solution or mitigation looks like?”

Essentially, by the nature of how specious Gebru’s stated problems are, they demand clarity over what an “ethical solution” even is, conceptually, and why everyone would have to agree.

For example, you could discuss economies of scale or train-once-finetune-everywhere approaches with GPT that reduce total energy needs. Or you could discuss how researchers can register the corpus they use and the snapshot of time it was grabbed, with an open understanding that as long as the methods and data are reproducible, there is no research ethical issue with studying that corpus, no matter how much bias or lack of woke vocab a given person believes it has. (And also, nobody is required to just accept activist language as important or valid.)

Gebru did none of this. The article could literally be summed up by Gebru saying, “I think <supposedly shocking evidence> is bad, therefore its connection to something in ML is bad.”

E.g. “I think, subjectively, that the raw energy use to train GPT is bad. Here are some shocking comparisons. Therefore GPT is bad.”

It’s incredibly unrigorous and juvenile. Dean’s comments that it needs to clearly state mitigations is actually a super generous, polite way of saying the paper is just subjective amateur hour.

I feel like if I am worried about those issues, when would I ever use sqlite instead of Postgres?

Sqlite seems like it is only appropriate for the knife’s edge boundary between small data, low reliability, in-memory situations (better served by server applications using tools like pandas or R) and bigger data, transactional structure, reliability constraints (better served by Postgres).

I just can’t understand what use cases live in between them and are better served by sqlite.

We did deploy it. This whole architecture was chosen in an adtech business I worked in before, supported by data platform teams, analytics teams and more. The RFC process was intense and the decision was vetted with seriously heavy effort.

Once in memory, the flatfile’s data can be mutated in place, and creating the equivalent of a WAL-like system or oplog to protect against sudden failure & loss of in-memory data is a pretty easy part to solve.

No, I meant using eg pandas even for that. We built an data annotation processing system at my last job like this. Literally customer specific csv files that got booted up and then an online flask service received restful updates of data annotations transactionally written back to the backing csv.

Managing it as separate csvs per customer allowed some incredible optimizations for fanning out processing and performing reporting and dashboarding. The process running pandas allowed us to do much nicer aggregations, pivots, filters, etc., and by not writing it in SQL, we had so much more flexibility in application code and especially in unit & integration testing code.

If data size per customer was going to grow substantially larger, we would have needed to migrate the workload to a backing SQL database, likely Postgres, but the nature of the problem meant this axis of data size was not a problem (every separate csv represented a completely isolated advertising campaign from a customer, with only up to a few million records per campaign).

Using a flask server program to do this in pandas was an aspect that really, really paid off for us.

Can anyone share some cool applications or bigger production systems built with sqlite? I love virtually everything I read about it, but I have never encountered a single use case where I am not better served by storing the data in some flat file (parquet, csv, etc.) and running a pandas program or simple Spark program to analyze it (locally on my laptop, or through a managed service like Dataproc or Databricks - it’s exceedingly cheap for any data size conceivably processable with sqlite).

I am asking sincerely, what is the comparative advantage of sqlite or some example scenarios or trade off circumstances where sqlite is a comparatively more effective solution?

It’s framed as a solution to energy reliance on fossil fuel because it is a solution.

This is very very stupid. I mean it: stupid.

The act of planning is not about the future, it’s about the present. I haven’t experienced the future yet, so it has no value to me. But a chained sequence of events that have a certain likelihood of occurring does have value to me now in the form of the expectation (which I experience today) of the outcome.

The author also presumes no one can value states of the world except insofar as they are directly, materially experiencing that state of the world, but this is wrong.

For example, I’ve never visited Australia, but I would value a state of the world where Australian cryptography laws and email surveillance laws are reformed. I get value just from the concept of that state of the world. I suppose it’s similar to getting value from consuming a fictional work of art - I experience the concept of it, but it doesn’t materially exist in my presence. I don’t benefit from it (say, like an Australian citizen would), but I benefit from the concept of it and associated probabilities of the world changing to a certain state.

So if I am on my death bed with a special button laid out in front of me, which if pressed will automatically cause all Australian politicians’ pants to fall down until they vote to reform cryptography laws, and I believe there’s high probability the button would work and the embarrassment would work, then my impending death should not make me indifferent to pressing the button. With those precious few moments of life, I am experiencing something materially of value (a chained sequence of cause & effect with high probability to lead to a state of the world I value). The world I am in contains aspects of value I can experience that are functions of sequences of events leading to worlds I am not in. There’s nothing irrational, non-utilitarian or inconsistent about this.

No, it is not easier. Working on them across multiple repos is just as easy. In either case you write dev tools to apply changes across units. If the units are repos, it adds no additional complexity.

Just because repo structure can relate doesn’t mean that such a relationship involves any properties of monorepo structure vs polyrepo structure.

Any messy code organization or dependency issue you can have in a monorepo, you can also have in polyrepos. Any automation or central controllership you can have in a monorepo, you can also have in polyrepos. Any scale-out chores or overhead work you can have with polyrepos, you can also have with monorepo.

The issues of monolith / poorly coupled dependency applications vs microservices has absolutely nothing to do with monorepo vs polyrepos. Neither option makes any class of tooling easier / harder to build or maintain. Neither makes conceptual organization easier / harder to get right, neither makes discoverability or cross-referencing easier or harder. That whole axis of concern is just fundamentally unrelated to the application monolith vs microservices question.

That wouldn’t make sense, given that repo structure truly is unrelated in any way to the distinctions between monolith applications vs microservices.

The article doesn’t have any connection to monorepo / polyrepo axis. You can use either repo structure for a monolith application, or use either repo structure for separated microservices. The article is just not related to repo structure in any way.

No, I can’t speak on their behalf on Hacker News, so it is important to me to stay disconnected from my employer.

“But first, an anecdote... when I was a kid...”

Nope. See ya. Whatever follows next is not going to be worthwhile.

Yep, I was going to suggest this too. It feels copy/pasted from Talk to Transformer.

I get it from ~15 years experience as a manager across many engineering teams in large ecommerce companies. I’ve seen it literally hundreds of times. Someone absolutely aces leetcode interviews and appears to walk on water with instant solutions to non-cookie-cutter dynamic programming problems, exotic tree / heap / graph data structure problems, word problems that are hard to map to data structure problems.

They join with a big offer and it goes nowhere. They don’t know how to choose simpler methods, do things a bad fast way, be incremental, follow evidence from quick user feedback and only optimize when you really have to. Their output is poor, but because they see themselves as high status due to leetcode superiority, it just becomes a problem for everyone. Eventually they think the issue isn’t their poor output but rather some ceiling they are limited by and they quit.

Meanwhile plenty of other people do terribly badly at the dynamic programming / data structure trivia, but they are great at the actual job. They work incrementally, they are less prone to over-engineering or premature optimizations, they focus on the business impact, all while still writing clean, simple code and carefully picking their battles in terms of significant data structure work or optimizations.

The sheer randomness of outcomes leads to leetcode performance just being totally uncorrelated with success in the role as defined by any measure of business impact or effectiveness.

I think it’s been bog standard practice to run flask via uwsgi or gunicorn with async workers and use multiple process based workers per deployed server unit (eg per pod in Kubernetes).

What matters is that the cumulative latency & throughput solve your problem, not how fast you can make one singular async worker thread.

I figure most people running complex web services in production would just do an eye roll at this post. Nobody's going to switch to PyPy for any of this.

My team at work runs several complex ML workloads, and we use the exact same container pattern for every service running gunicorn to spawn X async workers per pod and then scale pods per service to meet throughput requirements. Sometimes we also just post complex image processing workloads to a queue and batch them to GPU processor workers. In all these use cases, super low effort “just toss it in gunicorn running flask” has worked without issue for services supporting up to peak load of thousands to hundreds of thousands of requests per second.

Not true. Success in leetcode interviews does not correlate with success or impact once in the job. The current methods have poor accuracy and poor recall. The only reason they are widely used is political gatekeeping. You need the ability to imagine failure or argue that you, the existing employee, are superior, in any interview you want, for political purposes, so the interview has to be complex enough to facilitate this.

Tricky / complex interviews are a sign of internal political arms race and peacocking, nothing else. Absolutely great candidates fail all the time. Absolutely terrible candidates pass all the time.

I am long on Bitcoin and also believe Tether is a fraud. I expect Bitcoin will significantly drop, but I realize I could be wrong. If it does drop, that is nothing but a large buying opportunity to me. If it doesn’t drop, I’ll be happy to just buy smaller amounts on a steady basis.

I don’t understand why a crossword puzzle company spends so much effort on a journalism side project.