Research in machine vision suggests this is possible, and even has some positive effects, but it significantly degrades capabilities.
Research in machine vision suggests this is possible, and even has some positive effects, but it significantly degrades capabilities.
But this would make APIs increasingly annoying and unreliable with false positives and such. It seems like another advantage for local LLMs.
On a train/test/eval split. But the degradation is lower on OOD data. Which suggests perhaps the degradation is merely "less overfitting".
Do you know or have any references on this? If one disregards the emphasis on alignment and "merely" considers the "less overfitting" aspect, that would seem very profound in and of itself, a capability to avoid overfitting.
If you look at historical debates in the sciences, where universal truths are sought, candidate truths are intentionally and adversarially stretched to apparent or real inconsistencies in order to test the universality of a claim.
Think of say the back and forths between Einstein and Bohr concerning entanglement. They are assuming adversarial roles, tricking each others belief system into expressing the absurd. Together they mapped out predictions for non-obvious or outright bizarre aspects of reality. If no-one dares to take a potentially vulnerable position, there will be nothing to attack, but also nothing to disturb the scientific mind and prod the community into settling the matter by measurements.