So I wrote a very simple python script that would randomly generate layouts of who would sit in each room and next to whom. Every time it gave me a result I scanned it for conditions that would make it not work and add a rule to skip such configurations. After about six such edits I got a layout I thought was acceptable. The team as far as I know was happy and nobody questioned it for the entire time we were there. This saved me time because I didn’t have to pre-program all the conditions, only add ones I had already seen not work. Saved both CPU and brain cycles, so to speak.
This is quite similar to education research on Teachable Agents [1], which is an embodiment of the idea that to know something you must be able to teach it to someone else. In Teachable Agents you teach the computer rules (e.g. how ecosystems work), and then it gets a quiz where it compares the output to the right answer. When its wrong, you as the teacher must figure out whether the rules you taught it were correct and/or if it needs more rules.
Teachable Agents works for things where there's a right or wrong answer, because the computer is doing the test proctoring. But in your method the human is doing that, and I think it works quite well for things of a more qualitative nature like the arts, with the human playing the role of the critic or curator.
[1]: https://slate.com/technology/2015/04/teachable-agents-making...
- generate a couple examples
- see something wrong
- add a rule
- repeat
(For example, in the NFL, the teams in each division play each other twice, and then get the rest of their games by rotating through teams from the rest of the league)
Many times people have specifically researched why a certain method is better than status quo in practice, on real world data with real world operating constraints.
A good example is the paper “Let’s Put the Garbage-Can Regressions and Garbage-Can Probits Where They Belong” - explain an extremely common and hugely severe problem with “garbage can” regression models.
Try writing a simple calculator app with ML and you won't even have marked up enough training data by the time I have written it algorithmically. When it's finished, the ML one won't be as reliable.
I've been involved in a few projects where ML was a candidate approach. Mostly the answer was just to write down what we already know about the problem domain instead.