Comment by colincooke | Hacker News Reader

Comment by colincooke | Hacker News Reader

A simpler, if perhaps less effective, idea would be to just take advantage of ensembling. Get many estimates from everyone qualified to give them, then just fit a normal distribution to your final results. The average of many estimates tends to actually be quite accurate sometimes, with the famous example of visitors to a county fair guessing the weight of an ox[0]:

"At a county fair held in Plymouth, 800 visitors had participated in a competition to guess the weight of an ox. While most people's estimates were too high or too low — falling an average of 37 lbs. away from the true weight of 1,198 lbs. — the median of everyone's guess was off by only 9 lbs., or less than 1 percent of the true weight of the ox."

Curiously there is also some "within individual" averaging that helps too. If you would ask the same person to estimate the project timeline a few times, spaced temporally, then take the average, it'll probably be more accurate than a single estimate.

[0] https://www.npr.org/sections/13.7/2018/03/12/592868569/no-ma...

You're right! Ensemble forecasting is great! Vistimo proper (https://www.vistimo.com) does both -- it gets individual high/low estimates from multiple people, and creates a combined distribution from all of them (though each estimate is a log normal, not normal, since it's a time estimate).

But Vistimo is a full project management tool; that space is full and switching project management tools is expensive, so it's hard to get people to adopt.

Vistimo Quotes is a much lighter-weight way to get many of the same benefits without having to talk your whole team into signing up.

I'm curious if you allow the project to enter actuals after the fact, then you can compare them to individuals/aggregate estimates so you know how biased your model/individuals/the organization is on these types of estimation efforts. You could give that as feedback as an additional service: "You're constantly overconfident on X type tasks..."

Not at the moment. This is the dream, but it's statistically much harder than it seems!

For each task you have one distribution, and one sample. What's the likelihood that sample was actually from a different distribution?

Instinctively it feels easy to judge "your usually closer to the worst-case than the best-case" but mathematically it's not so straightforward, especially when the estimate doesn't include sick days, splitting time with other projects, etc, but both the projection and the reality do.

We tried this at $OLD_JOB and what I've found is that when there is a larger disparity between estimates, it hints at information asymmetry, but that asymmetry can still exist when everyone has similar ideas about timelines, and so you can still get drastically wrong estimates when everyone thinks they're in agreement.

Especially when people have different experience or lack experience around a specific aspect.

Getting a data scientist to estimate how long the front end will take to build will not be as helpful.

https://en.wikipedia.org/wiki/Prediction_market

https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

Exactly, I was going to suggest a prediction market, that would be cool. Especially if employees have to use real money!

That would be horrible! You’d get shortcuts taken and perverse incentives

How was 1 std dev 37 lbs for that??? I feel like if I guessed I might be off by 500 or more lbs. I would have guessed "slightly less than a small car".

37 lbs seems astronomically accurate unless this is a county fair of farmers...

Reply on news.ycombinator.com