e10v_me

> I think they'll only serve to create a barrier for junior developers.

I proposed some ideas how to lower entry barriers: https://e10v.me/matching-markets-congestion/#entry-barriers-...

> I'm a supporter of stricter documentation requirements—ADRs, tests, and so on.

But what would you do if there were a lot of incoming PRs and your capacity was not enough to review if they conform to requirements?

I see more comments from projects' members that they just start banning people or closing incoming PRs. I'm not sure that helps junior developers.

In one of my previous posts, I discussed congestion in the job market caused by the surge of AI tools that scrape job descriptions and auto-apply to jobs. Since that post, another problem has emerged: progress in the capabilities of coding agents has caused a sharp rise in vibe-coded pull requests in open source repositories on GitHub. This problem can also be framed as a matching market congestion problem.

I became familiar with the problem while working in a services marketplace and solving matching-market-related problems there. That gave me direct practical experience with the typical issues. In this post, I want to share that experience and knowledge.

I explore the services marketplace, a dating platform, job search, and open source contribution through the lens of matching market design and identify a common pattern: lowering search and application costs leads to more applications, resulting in less effective matching due to reviewer overload.

I argue that just automating application screening and review with AI doesn't fully resolve the problem. In some cases, it makes it even worse by creating a self-reinforcing feedback loop: more applications → more automated filtering → even more applications. AI automation tools lack private information about applicant fit and intent.

As an alternative, I propose to redesign incentives so applicants bear more of the cost of low-value submissions and use their private knowledge to apply more carefully. The proposed solution is a reputation-credit-based system for GitHub-like platforms: non-transferable reputation credits are earned through valuable contributions and debited through low-quality pull requests and issues.

I'm surprised by how many people has fallen for that. I also wonder how many of them are the author's friends or bots.

I published a practical comparison of Python packages for A/B test analysis: tea-tasting, Pingouin, statsmodels, and SciPy.

Instead of choosing a single "best" tool, I break down where each package fits and how much manual work is needed for production-style experiment reporting.

Includes code examples and a feature matrix across power analysis, ratio metrics, relative effect CIs, CUPED, multiple testing correction, and working aggregated statistics for efficiency.

Disclosure: I am also the author of tea-tasting.

I was thinking about the labor market congestion problem and came up with a solution that is often used in service marketplaces: pay to apply. Then I asked myself what this solution has that AI doesn't. That’s how I arrived at the analogy that prices act as model weights: they encode market information. An important difference: prices incorporate signals from dispersed, hard-to-observe data that an AI/ML model may not access.

P.S. Paying to apply may sound provocative and require thoughtful consideration and careful testing. Payments can be made with platform-issued virtual points, available in a limited supply. But here, I focus on why price signals may address this problem better than AI-based screening.

Funny, the same) When I was switching from R (data.table) to Python, it was painful. Not only because it was slow, but because of the API. At that time, I thought that maybe it's because of switching to something new. Several years later, switching from Pandas to Polars API was a real joy (Ibis is also good is that sense). So, I learned that it had been Pandas fault all along))

Congrats! What’s impressive is not just the speed of the tools Astral develops but also the speed of delivery.

I wonder, if you plan to extend the functionality of building and publishing packages. For example, support for dynamic version (from github) and trusted publishers.

Here’s another take on the Dunning–Kruger effect. I made two main points:

1. Consider N independent observations of two variables, X1 and X2, with imperfect correlation. Next, we assign percentiles to them: P1 and P2, respectively. We then take a subsample of size N/4 with the lowest values of P1. The statistical fact, not psychological effect, is that, in this subsample, the average of P1 will always be less than the average of P2. With a large enough sample size, the difference will be statistically significant.

2. Percentile is a measure with bounded support (between 0 and 100). It's not correct to use it as a measure of abilities in this experiment. Participants from the top test score quartile can overestimate their abilities by a maximum of 24 percentiles and an average of 12 percentiles. Participants from the bottom test score quartile can overestimate their abilities by a maximum of 99 percentiles and an average of 87 percentiles. There is certainly a bias here.

The first point is not particularly new; there are published papers on this topic. However, not everyone is aware of this, so it's worth mentioning. I also provided a Streamlit app for simulations and a source code.

The second point is somewhat novel. At least, I haven't encountered any references to it in the context of the Dunning–Kruger experiment.

I also encourage you to think about the question: if a person scores the maximum number of points in a test, does this mean that they cannot overestimate their abilities?

Thank you for explanation and for drawing a vivid picture) I will add FWER and FDR to the roadmap. Which specific controlling procedures do you find the most useful on practice?

I'm considering the following: - FWER: Holm–Bonferroni, Hochberg's step-up. - FDR: Benjamini–Hochberg, Benjamini–Yekutieli.

Thank you for explanation. If I understand correctly, you use this approach to increase sensitivity (compared to NHST) using the same data.

Most probably, in your case, higher sensitivity (or power) comes at the cost of higher type I error rate. And this might be fine. Sometimes making more changes and faster is more important than false positives. In this case, you can just use a higher p-value threshold in the NHST framework.

You might argue that the discrete type I error does not concern you. And that the potential loss in metric value is what matters. This might be true in your setting. But in real life scenarios, in most cases, there are additional costs that are not taken into account in the proposed solution: increased complexity, more time spent on development, implementation, and maintenance.

I suggest reading this old post by David Robinson: https://varianceexplained.org/r/bayesian-ab-testing/

While the approach might fit in your setting, I don't believe most of other users of tea-tasting would benefit from it. For the moment, I must decline your kind contribution.

But you still can use tea-tasting and perform the calculations described in the whitepaper. See the guide on how to define a custom metric with a statistical test of your choice: https://tea-tasting.e10v.me/custom-metrics/

Thank you, I'm aware of this. But I don't understand how your link answers my previous message. I was asking for example of how to fit it using only aggregated statistics (focus on "aggregated"). I'm afraid the MCMC or other Bayesian sampling algorithms are not the right examples.

You would need probability distributions anyway. In Python, SciPy is the most mature and popular package with probability distributions. And it depends on NumPy. But I'll gladly consider better alternatives if you propose them.

I thought about multiple comparison corrections. Here what my thoughts were:

1. Experiments with 3 or more variants are quite rare in my practice. I usually try to avoid them.

2. In my opinion, the Bonferroni correction is just wrong. It's too pessimistic. There are better methods though.

3. The choice of alpha is subjective. Why use a precise smart method to adjust a subjective parameter? Just choose another subjective alpha, a smaller one :)

But I can change my opinion if I see a good argument.

Thank you! I hope it will be useful for you.

Regarding your question, first, I'd like to understand what problem you want to solve, and whether this approach will be useful for other users of tea-tasting.

Thank you. I will think about it. There are many different things I can add. The idea is to focus on the most needed features first. And the word "exotic" tells for itself ;)

Not at the moment. If you have a specific method in mind, I will gladly look into it.

> Most common statistical software (e.g. statsmodels) will support this grouped format.

Interesting, I didn't know this about statsmodels. But maybe documentation a bit misleading: "A nobs x k array where nobs is the number of observations and k is the number of regressors". Source: https://www.statsmodels.org/stable/generated/statsmodels.gen...

I would be grateful for the references on how to apply statsmodels for solving logistic model using only aggregated statistics. Or not statsmodels. Any references will do.

CUPED is easier than stratifying users. Or, probably, you mean post-stratification. Still, CUPED is easier, on my personal opinion :)

I will consider it. Thank you for the suggestion.

I cannot agree more. It's one of the reasons I've developed the package. With tea-tasting, I can save time and focus on the more important tasks, like experiment design.

Thank you for the interest and for the suggestion.

Yes, one can analyze A/B tests in a regression framework. In fact, CUPED is an equivalent to the linear regression with a single covariate.

Would it be better? It depends on the definition of "better". There are several factors to consider. Scientific rigor is one of them. So is the computational efficiency.

A/B tests are usually conducted at scale of thousands of randomization units (actually it's more like tens or hundreds of thousands). There are two consequences:

1. Computational efficiency is very important, especially if we take into account the number of experiments and the number of metrics. And pulling granular data into a Python environment and fitting a regression is much less efficient than calculating aggregated statistics like mean and variance.

2. I didn't check, but I'm pretty sure that, at such scale, logistic and linear regressions' results will be very close, if not equal.

And even if, for some reason, there is a real need to analyze a test using logistic model, multi-level model, or a clustered error, in tea-tasting, it's possible via custom metrics: https://tea-tasting.e10v.me/custom-metrics/

Agree. I also suggest looking at Alex Deng's unfinished book on causal inference and, particularly, A/B testing: https://alexdeng.github.io/causal/

Alex Deng worked with Ron Kohavi at Microsoft Analysis and Experimentation Team and co-autored many important papers on the topic, including paper about CUPED.

Yeah, I know :) But it's in the roadmap. Btw, aren't Barnard's test or Boschloo's test better alternatives?