2 reactions
Is Google’s Gemini Really Smarter Than OpenAI's GPT-4? Community Sleuths Find Out
Google launched its latest artificial intelligence (AI) model Gemini on Dec. 6, announcing it as the most advanced AI model currently available on the market, surpassing OpenAI’s GPT-4.
Gemini is multimodal, which means it was built to understand and combine different types of information. It comes in three versions (Ultra, Pro, Nano) to serve different use cases, and one area in which it appears to beat GPT-4 is its ability to perform advanced math and specialized coding.
On its debut, Google released multiple benchmark tests that compared Gemini with GPT-4. The Gemini Ultra version achieved “state-of-the-art performance” in 30 out of 32 academic benchmarks that were used in large language model (LLM) development.
However, this is where critics across the internet have been poking at Gemini and questioning the methods used in the benchmark test that suggest Gemini’s superiority, along with Google’s marketing of the product.
“Misleading” Gemini promotion
One user on the social media platform X who works in the field of machine learning development, questioned whether Gemini’s claim of superiority over GPT-4 was true or not.
He pointed out that Google may be hyping up Gemini or “cherry-picking” examples of its superiority. Still, he concluded, “my bet is that Gemini is very competitive and will give GPT-4 a run for its money" and that competition in the space is good.
However, shortly afterward, he made a second post saying Google should be “embarrassed” for its “misleading” promotion of the product in a promotional video it created for the release of Gemini.
Google, this is embarrassing.You published an impressive video showing Gemini answering your questions. It looked awesome. It looked real-time.But it was a lie. None of that happened as recorded and presented to the public.Instead, you cherry-picked frames and edited a… pic.twitter.com/GjyqWPyaIu
— Santiago (@svpino) December 6, 2023
In response to his tweet, other X users spoke out about feeling deceived by Google’s portrayal of Gemini. One user said claims that Gemini would end the era of GPT-4 are “canceled.”
Another user, a computer scientist, agreed, and called Google's portrayal of Gemini’s superiority “disingenuous.”
Users pointed out that Google had included benchmarks that used an outdated version of GPT-4, rather than its current capacity, and therefore the comparisons were redundant.
Another area of concern to social media sleuths was in the parameters that Google used to compare its Gemini model with GPT-4. Moreover, the prompts given to both models were not identical, which could have major implications for the outcomes.
this is pretty weirdusually when you benchmark… you compare the results of the same exact test…Took someone else mentioning this for me to notice
— bryankyritz.eth (@kyritzb) December 6, 2023
The user also pointed out that the results were achieved using tests carried out on a model that “isn’t publicly available” at the moment. Another user pointed out that scores could be different if the advanced model of Gemini was tested against the advanced version of GPT-4 known as “turbo.”
Google launched its latest artificial intelligence (AI) model Gemini on Dec. 6, announcing it as the most advanced AI model currently available on the market, surpassing OpenAI’s GPT-4.
Gemini is multimodal, which means it was built to understand and combine different types of information. It comes in three versions (Ultra, Pro, Nano) to serve different use cases, and one area in which it appears to beat GPT-4 is its ability to perform advanced math and specialized coding.
On its debut, Google released multiple benchmark tests that compared Gemini with GPT-4. The Gemini Ultra version achieved “state-of-the-art performance” in 30 out of 32 academic benchmarks that were used in large language model (LLM) development.
However, this is where critics across the internet have been poking at Gemini and questioning the methods used in the benchmark test that suggest Gemini’s superiority, along with Google’s marketing of the product.
“Misleading” Gemini promotion
One user on the social media platform X who works in the field of machine learning development, questioned whether Gemini’s claim of superiority over GPT-4 was true or not.
He pointed out that Google may be hyping up Gemini or “cherry-picking” examples of its superiority. Still, he concluded, “my bet is that Gemini is very competitive and will give GPT-4 a run for its money" and that competition in the space is good.
However, shortly afterward, he made a second post saying Google should be “embarrassed” for its “misleading” promotion of the product in a promotional video it created for the release of Gemini.
Google, this is embarrassing.You published an impressive video showing Gemini answering your questions. It looked awesome. It looked real-time.But it was a lie. None of that happened as recorded and presented to the public.Instead, you cherry-picked frames and edited a… pic.twitter.com/GjyqWPyaIu
— Santiago (@svpino) December 6, 2023
In response to his tweet, other X users spoke out about feeling deceived by Google’s portrayal of Gemini. One user said claims that Gemini would end the era of GPT-4 are “canceled.”
Another user, a computer scientist, agreed, and called Google's portrayal of Gemini’s superiority “disingenuous.”
Users pointed out that Google had included benchmarks that used an outdated version of GPT-4, rather than its current capacity, and therefore the comparisons were redundant.
Another area of concern to social media sleuths was in the parameters that Google used to compare its Gemini model with GPT-4. Moreover, the prompts given to both models were not identical, which could have major implications for the outcomes.
this is pretty weirdusually when you benchmark… you compare the results of the same exact test…Took someone else mentioning this for me to notice
— bryankyritz.eth (@kyritzb) December 6, 2023
The user also pointed out that the results were achieved using tests carried out on a model that “isn’t publicly available” at the moment. Another user pointed out that scores could be different if the advanced model of Gemini was tested against the advanced version of GPT-4 known as “turbo.”
| 14,000 mlx | |
| 17 | |
| 2 |