Hacker News Reader

Comment by rachitnigam | Hacker News Reader

Author here. Acceptance criteria are hard to lock down, especially in new projects. The whole point of research is to explore domains and to expect a lot of dead ends.

I’m unsure what a concrete acceptance criteria will look like given that we want to encourage researchers to explore.

View on HN

Acceptance criteria in the CS department of my alma mater is on a scale of 3-4 peer peer-reviewed papers, accepted in some journals. I would also be careful about what the point of research might be. The most important purpose of a PhD student is to produce a thesis in a foreseeable future, to qualify himself for scientific work. The success of the thesis depends on the advisor(s), and later the department and the reviewers. Thus, it's important to choose a topic, which is NOT an ultimate dead end. Even if your topic leads to a successful thesis, it might be a dead end later on. Because your advisor has no network, or because nobody is interested in your results etc. And even if you can continue with research, you are rather occupied with fund raising, networking and marketing. If I had to do a PhD again, I would try to figure out as fast as possible what is expected to do in the specific configuration I'm in, and then try to minimize all obstacles and distractions which might be in the way (teaching, assistance for the prof, fund raising, conferences etc). After that, if your're are lucky, you have 1-2 years left to write down your stuff.

(off the cuff answer) Well the basic acceptance criteria should relate to the research goals, so that the exploration is focussed in some sense.

There is clearly scope for iteration here. If I was assessing an open ended exploratory project, I'd expect to see that a problem space has been defined and that some sort of question has been posed within it. Yes, it might take some time to find out what the question actually is, and the question might be to some extent retrofitted.

Once the question has been defined then you can imagine research subgoals that involve creating the required tooling / infrastructure, collecting test data, running a trial, analysing the outputs, etc. Actual software dev activities should be traced back to one of these research subgoals, or you'd have to question why they are being conducted at all.

> Acceptance criteria are hard to lock down, especially in new projects

In industry research teams it is essential to lock them down at some point and you may not get paid if you don't complete against them (accepting that a negative result might be a valid research outcome). This ability to focus was the hardest thing I had to learn when I transitioned from academic to industry research. Even in an open-ended exploratory academic context, you still do need to prove, as the GP said, that you have in some sense answered a question.

Better to have an answer that is wrong, than something that isn't an answer at all.

It sounds like you're saying that acceptance criteria are hard to define when there's high uncertainty (a lot of activities are high risk of not yielding anything interesting), which makes sense.

In that case, it seems like...

* acceptance criteria over "explore" activities (e.g. achieving failure, we are running X high risk/ high reward activities per Y period of time; someone signed off on its value before running it)

* acceptance criteria over "exploit" activities (e.g. I need to produce these very clear things of value)

In retrospect, though, I think my research community (in psychology) was very open / interested in learning about dead ends. The weirdest outcome is when an advisor learns something from research, but then feels like its obvious, so the research gave them something, but is perceived as low value.

Reply on news.ycombinator.com