how odd! What exactly is lmsys using? Some hidden API that google give them so they can have a better ranking there?
From the vertex ai:
API_ENDPOINT="us-central1-aiplatform.googleapis.com"
PROJECT_ID="test00"
MODEL_ID="gemini-pro"
LOCATION_ID="us-central1"
curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
"https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION_ID}/publishers/google/models/${MODEL_ID}:streamGenerateContent" -d '@request.json'
curl \
-X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=${API_KEY} \
-H 'Content-Type: application/json' \
-d '@request.json'I don't know about that second part - but it would make sense that google (and others) may want to use lmsys's arena to benchmark their models.
After all, Human A/B tests are far better then the current automated benchmarks.
I would like more info from lmsys as to how they're accessing these though.