jvans

Yes. Use GBTs and off the shelf language models you don't train yourself.

This holds back people of every level. Paradoxically I think it holds senior people back more because once someone gets a sufficiently senior label, they think they should have all the answers and stop asking questions

In my experience the more familiar that someone is with the code, the more they think pushing code into smaller functions is the correct path. They have already built up a mental model of the code at hand, so the cleanest implementation to them is one with very few lines.

But the next person to come along has to bounce back and forth, performing mental stack push/pop operations to create the same mental model which is much harder to do when you don't have any of the original context

Building your own bayesian model with something like pymc3 is also a very reasonable approach to take with small data or data with too much variance to detect effects in a timely manner. This also forces you to think about the underlying distributions that generate your data which is an exercise in itself that can yield interesting insights.

I would say the same logic applies to products/features and influences the design component. You create a product that reaches some local maximum of utility but there's a constant pressure to churn out new features with increasingly small gains in customer utility. Those features need UI elements, items in menu bars, etc etc which results in a bloated UIs full of esoteric features

I would make a distinction between adding an index to sqlite/postgres and adding one to elastic. The ANN algorithms usually require the index fits into RAM which can lead to high memory requirements on large datasets. Something like elastic will do a lot better because of its ability to horizontally scale

I assume that is the rationale behind a dedicated database but I generally feel the same way as you

Yes, definitely. I am a generalist SWE, learned ML through Jeremy Howard's course and transitioned to the ML team at my company. 98% of ML in the real world is building a robust software system to support the models. The last 2% is actually modeling and simple models go a long way.

Besides being false, the problem with making the scientific certainty the bar for whether to ban these chemicals is that the default action is to keep them, harming people for decades before science catches up.

The bar should be for proving safety, not proving harm

I have often thought along similar lines, that the effort involved in building software seems to indicate a level of abstraction that is missing. The general theme of the comments seems roughly what you should expect from a very bold, paradigm shifting proposal. Good luck with your efforts and don't let this discourage you!

I will make one minor suggestion that I hope is constructive. I found the post difficult to read, largely because you rapid fire introduce a bunch of completely new concepts and propose a solution to many problems at once. You make a passing comparison to "just event sourcing and materialized views", although this was the easiest way for me to understand what you are doing. Starting from event sourcing and materialized views puts the reader on a ground they already understand, and moving on from there to why rama is better/what it adds on top, would be an easier transition.

I found that book impenetrable. I'm sure it's the most rigorous textbook on the subject but it is not explained in an intuitive or friendly way.

I will check out the stan guide though, thanks!

> It's difficult. Applying these techniques efficiently and correctly is way more difficult than standard frequentist methods

Do you have any good resources/examples for applying these methods effectively? I've read Statistical Rethinking which is a good introduction to these methods at a high level but I find when I dig into an actual problem I have a lot of gaps and wish there were more real world code examples I could learn from.

It doesn't have to be outright fraud. I imagine the vast majority of manipulations are extremely subtle. Intentional selection bias, multiple hypothesis testing without correction or reporting the failed hypotheses(p hacking). Most of these would improve your chances of finding important results significantly and be almost impossible to detect

I imagine the number of people who fit the venn of data engineer with signal analysis knowledge/skills is pretty small. Is hiring for that kind of role difficult? How would you recommend people with skills in SWE/DE become hireable in the signal processing world?

The incentive structure of the PM role is to pump out new features when a lot of the time making something more reliable or easier to use has more user value. Design and Engineering should jointly take on this role

Next step: tweak enough chemical bonds to give it a different acronym and dodge litigation for another 20 years

If you aren't writing the code you don't understand the problem and can't possibly write the documentation

it's a classic bike shedding exercise: https://en.wikipedia.org/wiki/Law_of_triviality

Does that mean that 5+ years go by before code starts running in production? How do you cope with the existing system evolving as the rewrite is taking place?

I am sure there are good reasons for it, the postgres proposal for switching concurrency models[1] comes to mind. I actually just posted this question to see what people come up with: https://news.ycombinator.com/item?id=36646299

My knee jerk reaction to rewrites is so negative I actually would like some counter evidence on when it worked out.

[1]https://www.postgresql.org/message-id/4658520e-5cd0-6242-e54...

Some non fiction books that thoroughly convinced me of their thesis with their rigor:

1. Bottle of Lies - Eban

2. Capital in the Twenty-First Century - Piketty. I will admit this was difficult to push through but by the end of it you really appreciate how he takes so many different angles to prove his point. Type 2 fun for sure.

Fiction books with great style:

1. A Deepness in the Sky - Vinge. -- This book is so mind blowingly exceptional I am shocked I don't see it recommended more on these types of questions. Vinge absolutely nails the micro and macro structure of the book. The build up, introduction to and revealing of mysteries, are all perfectly timed to keep you hanging onto every word.

2. All the Light We Cannot See - Doerr

3. Cloud Atlas - Mitchell

I don't think this is hyperbole, i completely agree with it as written

The incrementally part is extremely important and often overlooked. I have seen project after project attempt some major big bang rewrite that always takes way longer than expected, delivers less value than promised, and causes problems in areas that used to work fine. We often understand some big picture things better as time goes on but there's a thousand small decisions baked into the existing code that are very easy to overlook

On incremental improvements you should be able to stop what you are doing within a week or two and be ok with leaving the code like that for a long time.

The devil is always in the details and it's too easy to present a high level intelligent sounding argument that ignores all the important complexity.

In any highly nonlinear system it's extremely difficult to predict side effects of changes like this. It's plausible there are benefits to things that seem mild or harmless and eliminating them can have dramatically negative consequences.

The suits know the profit math and find the risks acceptable. The engineers know the mortality math and do not.

The types of effects you want to measure would take months or years to show and are the combination of many different small decisions. Teams need to apply common sense thinking, empirical data, and the willingness to wrestle with uncertainty. All of that is hard, blindly following a/b test results relieves people of that cognitive burden

I agree but a truly nuanced approach to interpretation of A/B tests is rare especially when mixed incentives are involved. Ignoring empirical evidence is bad and taking it as gospel is also bad.

This is why I sometimes hate a/b testing. I'm sure someone at booking a/b tested these things and saw an increase in revenue. The thing that these tests don't measure are very long term effects where people either start to hate your product and look for alternatives, or become so numb to the changes that the initial novelty effect wears off. The person who ran the test gets a promotion for increasing revenue during the quarter but the net result is a massive negative for the longevity of the product.

I am not exactly sure which surgeries you are talking about but for people curious there are basically 3 surgeries to consider:

1. MMA - Break both jaws and advance them forward to expand the airway. Most invasive, most effective. It is a major surgery and should not be taken lightly.

2. Maxillary expansion - Less invasive depending how it's done. Be skeptical of maxillofacial surgeons who do Le Forte osteotomies as part of the procedure. There are many variations of this surgery. EASE is probably the best

3. Nasal surgery.

If you can afford it consider Kasey Li https://drkaseyli.org/ease/

my point is the literal interpretation doesn't come close to capturing the business value