Google is reportedly comparing Gemini’s responses against those generated by Anthropic’s Claude artificial intelligence (AI) models. As per the report, the Mountain View-based tech giant asked third-party contractors who work on evaluating the quality of Gemini’s responses to also rate whether Claude’s responses are better. While it is not an uncommon practice, using third-party AI models for this purpose requires obtaining permission from the company. The report could not confirm whether the tech giant had received any permission from Anthropic.
Gemini Reportedly Being Improved Using Claude
According to a TechCrunch report, contractors working on Gemini are comparing its outputs against those created by Claude, a direct competitor of the former. The publication claimed to have seen internal correspondence between Google and the contractors, where they were told to follow this practice.
The report also claimed that the contractors are given up to 30 minutes per prompt where they are tasked to determine if they prefer Gemini or Claude’s response. Notably, these contractors are typically subject matter experts and they evaluate the chatbot’s responses on niche and technical topics across several parameters such as truthfulness, precision, and verbosity.
After the practice began, some contractors reportedly began noticing references to Anthropic and Claude in Gemini’s responses. The publication claimed that one Gemini output directly stated “I am Claude, created by Anthropic.” Typically, this should not happen if Gemini’s responses are simply being compared against Claude’s. This raises the concern of whether developers are feeding Claude’s output to Gemini in instances where the former’s responses are better.
In its commercial terms of conditions, Anthropic states that those accessing Claude are forbidden from building “competing product or service” or training “competing AI models” without obtaining approval from Anthropic.
Speaking with TechCrunch, Google DeepMind’s spokesperson Shira McNamara said that while the division does compare model outputs for evaluation purposes, Gemini is not trained on Anthropic’s AI models.
Last week, a report claimed that the contractors assigned to evaluate Gemini’s responses were being told to rate the chatbot’s outputs even when they were outside of their expertise. While earlier, contractors could skip such questions, the option to do so was reportedly being removed.