rwilson4

The data generating process is the random assignment of people to experiment groups.

The potential outcomes are fixed: if a person is assigned to one group the outcome is x1; if another, x2. No assumption is made about these potential outcomes. They are not considered random, unless the Population Average Treatment Effect is being estimated. And even in that case, no distribution is assumed. It certainly is not Gaussian for example.

Under random assignment, the observed treatment effect is unbiased for the Sample Average Treatment Effect. So again, the data generating process of interest to the analyst is random assignment.

I agree, but that's not the whole story. A/B testing isn't just for identifying the best option, it's for figuring out how much opportunity there is in a particular facet of your business. If you try a few reasonable designs, and some designs have much better performance than others, then it makes sense to continue investing time to further improve. You can take what you've learned and try to come up with even better designs.

On the other hand, if the variants mostly perform the same, why spend more time on it? Go focus elsewhere.

It certainly is a logical possibility that the next design you try will be much more impactful, but after trying several variants unsuccessfully your time is probably better spent elsewhere.

The author discusses "pull forward", where the real impact of a change is to make people purchase earlier, but we don't necessarily observe incremental purchases. This isn't necessarily bad; I'd rather have a dollar today than a dollar next week.

This can be quantified by plotting the incremental conversions observed by day x. We migh see a big initial lift that degrades over time. If it eventually degrades to zero, there are no truly incremental conversions, just pull-forward. But if we end up pulling forward a meaningful number of purchases by a month or more, that can be valuable to the business!

I wouldn't immediately jump to a complicated mathematical model to handle this situation, I would consider the business implications first and foremost.

I also urge anyone considering Bayesian methods for A/B testing to read up on the likelihood principle vs the strong repeated sampling principle (I documented my thoughts here [0]). Bayesian methods always satisfy the likelihood principle; frequentist methods always satisfy repeated sampling. In many situations both methods satisfy both principles, and then the two approaches will give similar answers. But based on many years doing A/B testing, I wouldn't give up repeated sampling lightly. Bayesian and frequentist methods are not blindly interchangeable.

On the other hand, if repeated sampling is not important in your use case, then by all means prefer the Bayesian approach! I just want people to consider the trade offs.

[0]: https://adventuresinwhy.com/post/bayesian_ab_testing/

Correlation typically means y is a linear function of x, but people usually interpet it (incorrectly) as: knowing x tells you something about y. If y = x^2, then y is determined completely by x, but since it's nonlinear the correlation may actually be zero depending on the distribution of x. This paper proposes a statistic that will indicate if y is related to any function of x, linear or nonlinear.

If the relationship is linear, I'm guessing a test based on Pearson has greater power.

Sure, or Nesterov's accelerated gradient descent or something. I'm just not sure what the authors mean by "cheap iterations".

Update: your other comment clarified this for me. It's cheap relative to other methods that work with non-self-concordant objectives.

Theoretically yes, but: typically with neural networks, and esp. deep networks with millions or billions of parameters, 2nd order methods which require calculating and factoring the matrix of 2nd derivatives are prohibitively expensive. 1st order methods such as gradient descent skip this, albeit with a tradeoff in convergence. The method proposed in this paper is impractical for large systems such as computer vision or language models.

This is an approach for solving convex optimization problems. An optimization problem is when you want to find the best option, or the fastest, or the most profitable. If you're asking questions with superlatives (most, best, fastest), you may be working on an optimization problem.

Many optimization problems are NP Hard and cannot be solved in a reasonable amount of time. Convex optimization problems are a special subclass which can typically be solved efficiently (meaning in polynomial time).

This paper provides a new approach to solving Convex Optimization problems for systems where other methods struggle.

"Cheap iterations" led me to expect a 1st order method, not requiring computing the matrix of 2nd derivatives, which is the bottleneck in large systems. But this does indeed rely on the Hessian, so I'm not sure what makes the iterations "cheap".

Haven't read the paper yet, but Boyd and Vandenberghe's book, Convex Optimization, is available for free on the authors' website and covers all the foundational material.

Update: I've skimmed the paper and here's the gist as I understand it. The paper alleges that Newton's method can have convergence issues, even when using line search. Their approach resolves these convergence issues. It's 6am where I am which isn't great for processing this stuff, but I write a lot of custom Convex Optimization solvers, and I've never had convergence issues with Newton's method with the default line search parameters suggested by the book mentioned above, even with some bad condition numbers that throttle off-the-shelf solvers (which is why I write my own).

The paper talks about speed of convergence, which initially made me think this was a first order method, not requiring calculating the Hessian (matrix of 2nd order derivatives). For large systems this is really slow and 1st order methods speed this up by working only with the first derivatives. But that's not what this paper is: they still use the Hessian but add regularization, which presumably just improves the condition number. Reminds me of proximal algorithms but haven't explored that connection.

Update on the update: the paper addresses situations where the self-concordance assumption does not hold. TBH I didn't pay much attention in class when we went over self-concordancy and convergence, but the contribution of this paper seems to be extending convergence guarantees even to non-self-concordant systems.

Have you considered a .dir-locals.el file?

I use elpy regularly for python development. Handles virtualenvs really well, makes it super easy to run one or all test cases. Good support for linting and reformatting (black).

I should note that I don't work on large codebases. My codebases have ~20 files and <20,000 lines of code total. I know some folks prefer LSP and that might be the reason.

Working in marketing has made me (sadly) realize these articles are meaningless. Articles like this don't mean the recommended brands are the best. It just means those brands paid to be recommended.

For anyone else who had no idea what OP was doing with storm petrels: https://en.m.wikipedia.org/wiki/Bird_ringing

I recently bought a Kinesis Advantage 2. Could you be more specific about how you've mapped Ctl and Option? I mapped Delete and End in the left thumb cluster to Ctl and Option resp. What did you do?

People with kids. My daughter's preschool is more expensive than my college was!

The intellectual honesty one hits hard. A luxury indeed!

I'm really curious about this possibility. How would we prove or disprove this?

“The Book” is the hypothetical collection of “best” proofs for each theorem. “God’s favorite proofs” if you will.

Any recommendations on QCD textbooks for Physics grad students? I left that field a while back but I had always hoped to understand it, beyond just hand-wavy explanations.

“Many learning tasks” is a wiggle term. Sure, edge cases exist, but the methods do work impressively well in many cases.

Also, if you don’t mind me asking, what kind of support from your manager would be helpful in this situation?

I like that! I’ll think about how I could incorporate it. Thanks!

That may be their perspective, true! But if so I would consider that a red flag indicating burn out. Why are they so worried about deliverables? Do they feel the timelines are too aggressive? Are they really so overloaded they can’t take 20 minutes to think about their work from a different perspective? What other problems do they care about? As a manager I need to tease out these things. But each team is unique; what works for my team might not work for yours!

In 4 years of doing 1:1s this way, at different companies and with dozens of teammates, I have received mostly positive feedback. And I’ve definitely adapted the process to address concerns. But again, every person has their own individual needs and preferences.

Early on as a manager I made the mistake of assuming I knew what was going on in someone’s head just by working closely with them. I was wrong. You can’t even just directly ask because of the complicated dynamics. The best way I have found is frequent conversations that challenge people to dig down into their values and perspectives. Lists like this one are a good starting point, but as others have said genuine empathy is required.

I treat them as prompts. I’ll give my team as much fore notice as possible what the topic will be and ask them to really think about the question in advance. Otherwise the conversations are often shallow. My team often writes out their thoughts and we go through it, it’s been pretty successful I think!

My 1:1s have the following format:

- Current tasks

- Career development/Goals (we have 6 week and yearly goals that the employees select for themselves with a little guidance from me)

- Special topic, different each time and very much aligned with the posted list. My question for next time is about values.

- Feedback, both delivering and soliciting (this has proven to be the hardest for me, as it takes time to think about and articulate feedback well)

I do the 1:1 at a nearby coffee shop since I think people feel more open outside the office.

Skip the first 50 pages or so. The rest is golden.

Credit where it’s due though. The low barrier to entry and habit-forming nature got me learning Italian. I’m definitely not fluent but I can have conversations with people in cafes and restaurants and it’s pretty fun! Recently I’ve moved on to Babel and I like that too.

It also seems to ignore strike price and the scenario of an acquisition where only the VC and founders get paid.