Cost as a perf metric is meaningless and the history of computer benchmarks has repeatedly proven this point.
There is a reason why we report time (speedup) in spec instead of $$
The price you pay depends on who you are and who is giving it to you.
There is a reason why we report time (speedup) in spec instead of $$
The price you pay depends on who you are and who is giving it to you.
With the other threshold, it creates a disincentive for models like llama-405B+, in effect enshrining an even wider gap between open and closed.
And even if it were, if said guy has such amount of compute, then it's time to use some of it to describe the model's safety profile.
If it makes sense for Meta to release models, it would have made sense even with the requirement. (After all the whole point of the proposed regulation is to get some better sense of those closed models.)