ChatGPT vs. Claude 2: A Battle of the LLM Titans
-(2)-profile_picture.jpg)
Written by Ivana Kotorchevikj We Tested
ChatGPT took the world by storm when it was unveiled by OpenAI late last year, showcasing remarkable human-like conversational abilities. Now, a new challenger enters the ring - Claude 2, the latest iteration of Anthropic's conversational AI assistant.
Claude was first launched in March 2023, but in July, Antropic created a buzz, releasing the better and improved Claude 2, placing him in Bing and Bard's company.
Typically described as ‘friendly and enthusiastic,’ many users claim Claude 2 is far superior to the good old ChatGPT.
Intrigued by this new AI kid on the block, in this detailed study, we will look closely at ChatGPT and Claude 2, explaining what they can do, their special features, and the important differences that set them apart. We’ll compare and contrast their capabilities and results in several categories: tech under the hood, availability, features, processing time, text context limit, up-to-date information and knowledge base, creative thinking, math and logical reasoning, data analysis, and summarizing.
While both leverage massive neural networks to generate remarkably coherent text, ChatGPT and Claude 2 have different strengths and weaknesses. Who will emerge victorious in this battle of the LLM titans?
Tech under the hood
Though Claude 2 and ChatGPT have a very similar user experience, allowing people to converse with them through typed messages, the underlying artificial intelligence powering each chatbot differs.
What is ChatGPT and how it works
ChatGPT is an AI chatbot built by OpenAI. It is a large language model (LLM) trained on a massive dataset using deep learning techniques like transformer networks. It’s capable of having convincing natural language conversations.
It’s built on OpenAI’s GPT-3 and GPT-4 large language models. The AI chatbot uses natural language processing and pattern recognition to provide articulate and informative responses to any conversational prompt, questions, or statements.
ChatGPT is designed to perform well across a wide range of conversational tasks like answering questions on complex subjects, conducting dialogues, doing translations, generating content like essays, articles, and stories, generating marketing materials including tweets, emails, and scripts, providing programming assistance, like explaining code and composing basic algorithms, and even answering customer service queries.
What is Claude 2 and how it works
Claude 2 is powered by a large language model architecture. However, Anthropic uses a proprietary self-supervised training technique called Constitutional AI to train Claude 2. It refers to a set of principles that help Claude better judge the text it’s producing. These principles are found in documents such as the UN's 1948 declaration and Apple's terms of service, addressing contemporary concerns like data privacy and impersonation.
Its training focuses on domains like common sense reasoning, factual knowledge recall, and providing citations/references for statements. Claude 2 has safety filters to detect potential harm and is designed to avoid generating harmful, unethical, dangerous, or illegal content.
Some key use cases include research assistance, friendly everyday conversations, answering factual questions, summarizing documents, providing tutorials/explanations, assisting with writing tasks requiring reasoning, and providing advice with good judgment and awareness of safety.
Availability
ChatGPT
ChatGPT became available to the public in November 2022, first as a free version, and then from January 2023, in a Premium version as well. In May, OpenAI also announced the iOS app for ChatGPT, and in July, they released the Android app.
ChatGPT is available globally, except in countries that have imposed nationwide strict internet censorship laws and regulations.
Claude 2
On the other hand, Claude 2 is available in beta testing in the US and UK. But Antropic plans to release Claude 2 worldwide. Because of its limited availability, Claude 2 is not as widely used and popular as ChatGPT, but this might change when it becomes available globally.
Features
ChatGPT
ChatGPT offers additional features besides the default mode. Users could use Bing web browsing mode, but it was discontinued in July 2023 because it bypassed paid walls by publishers that offered paid subscriptions.
However, ChatGPT has other useful features in its pocket that increase user productivity. More than 800 plugins are available in the Premium versions, which enhance the scope of ChatGPT’s possible applications, helping users with tasks such as writing prompts, booking flights and restaurants, getting links to comics, etc.
Additionally, one of the latest additions, the Code Interpreter, allows users to effortlessly tackle mathematical problems, analyze data, generate charts, and add and convert files into various formats, among other functionalities.
Another valuable feature is Custom Instructions, which allow you to guide ChatGPT's responses to your specific needs or desired outcomes. You can include them at the beginning of your conversation to steer the conversation in a particular direction.
Claude 2
While Claude 2 doesn’t offer various plugins and additional features, it excels in performance specifications compared to ChatGPT, with a larger context window, the ability to process larger bodies of text, faster processing time, and all this for free. We’ll go over and compare each of these features below.
Processing time
ChatGPT
The average processing time for ChatGPT is about 30 seconds. The response time may vary depending on demand and the complexity of the question.
Claude 2
Claude beats ChatGPT with an average processing time of 5 seconds, but it may also vary if the query is more complex.
Text context limit (token size)
ChatGPT
ChatGPT token limit increased with the newest versions. Formally, ChatGPT-3.5 has a 4096-token limit (around 3000 English words), while GPT-4 (8K) has an 8,192 token limit (around 6,000 English words), and the GPT-4 (32K) boasts 32768 tokens per prompt (around 25,000 English words).
The latest GPT4 (32K) version is available only to developers and select companies who’ve been granted access and not to regular users.
However, one user performed a test on GPT-3.5 and GPT-4 to see the maximum number of words it would allow in a prompt. He discovered that there are no major differences in the total number of words or characters that can be used in ChatGPT-3.5 vs. ChatGPT-4. Both GPT-3.5 and GPT-4 accepted only 2,500-3,000 words and 18,000-19,000 characters (prompt and response text in total).
Disclaimer: The test was performed in April 2023, and context limits may have changed.
Claude 2
On the other hand, Claude 2 has a far larger context limit of 100,000 tokens, which is about 75,000 words in a single prompt for free.
Up-to-date information and knowledge base
ChatGPT
We know it as gospel that ChatGPT has a knowledge cut-off until September 2021. It has been trained on 570 GB of online data, but it has limited knowledge and doesn’t have access to the internet (that is, it had until they canceled the Bing browsing plugin).
When you try to ask for current events or news, ChatGPT responds that its last update was in September 2021, that it couldn’t provide real-time information, and advises you to consult more relevant sources. For example, if you ask ChatGPT, “Who is the monarch of the United Kingdom?” it will give the answer:
“As of my last training data in September 2021, Queen Elizabeth II was the reigning monarch of the United Kingdom. However, I cannot provide real-time updates, so I recommend consulting a current and reliable source to verify the most up-to-date information.”
Premium ChatGPT users can get more up-to-date information since GPT-4 is continuously updated with new data. Additionally, they can use the plugins to get real-time information, like world news, flights, and hotel booking information.
Claude 2
Claude 2 was trained on more recent data up to December 2022, which means it can provide answers to more recent events than the free version of ChatGPT, but it still can’t provide real-time information.
Claude 2 is also not connected to the internet, like Bard or Bing, but Antropic claims that users can provide a recent website or page, and Claude 2 will be able to answer related questions. The company behind Claude 2 also states that its database is larger than ChatGPT’s and that the AI constantly learns from users and grows its knowledge base.
Creative thinking
Now that we have broken down ChatGPT’s and Claude 2’s capabilities and features, let's test them on different tasks and compare the results.
First, we’ll test the two chatbot’s creative skills. We asked both ChatGPT and Claude 2 to write a story with the prompt:
“Please create a story about a girl and her doll. The story should have a moral conclusion that we should be happy and satisfied with things we have, and that although other people's stuff can be more attractive to us, that doesn't mean they are better.”
ChatGPT
Claude 2
ChatGPT produced an intricate story interwoven with emotion that can evoke emotion in every human. It used more story-like and more complex sentences. Claude 2, on the other hand, generated a shorter and simpler story that lacked depth. The language is simpler, and sentences are shorter.
Math and logical reasoning
We asked ChatGPT and Claude 2 to provide an answer to this math/logic problem:
“You have two ropes that each take an hour to burn, but burn at inconsistent rates. How can you measure 45 minutes? (You can light one or both ropes at one or both ends at the same time.)?"
ChatGPT
Claude 2
Both ChatGPT and Claude 2 provided correct solutions for the problem. ChatGPT has an advantage in providing a more concise and clear solution in fewer steps. Claude 2 provided a bit more complex solution, longer steps, and additional explanations.
Data analysis
Although ChatGPT and Claude 2 are primarily text-processing and generating tools, they also have data analysis capabilities. We provided a sample database to the chatbots and asked them to analyze the data. More precisely, to provide the number of female employees working in the company for more than 20 years.
ChatGPT
ChatGPT has the upper hand when it comes to data analysis since it has a powerful Advanced Data Analysis feature in the paid GPT-4 mode. (OpenAI renamed the feature from the original name Code Interpreter and changed some features.)
The Advanced Data Analysis feature allows users to upload data files in various formats, including XLSX, XML, Jason, HTML, PDF, DOC, DOCS, TXT, and more.
Claude 2
On the other hand, Claude 2 supports uploading files in PDF, TXT, and CSV. So, if you need to upload data like XLSX, you need to convert it to CSV to be able to upload it.
ChatGPT correctly answered that 13 women have been working in the company for more than 20 years. Claude 2, on the other hand, got the answer wrong. It not only missed the total number of women, but also incorrectly calculated that there are only 3 women who have worked for more than 20 years.
Although Claude 2’s response was incorrect, it does a good job of walking you through the steps of its logic and how it breaks down the task.
Summarizing text
For the last test, we compared ChatGPT and Claude 2 with summarizing articles.
ChatGPT
Claude 2
ChatGPT did a good job summarizing an article and capturing the most important points into paragraphs and lists. Claude 2 summarizes the article into bullet points with the most important points, if you don’t specify the format in the prompt. Also, if you paste a longer text into Claude 2, it uploads the text and the prompt like a text attachment, but it does the task nonetheless.
However, Claude 2’s power lies in its ability to summarize massive bodies of text - up to 75,000 words, which means large books and reposts - something ChatGPT lacks. And
Wrap up
Both ChatGPT and Claude 2 have carved out their own niches, each bringing a unique set of pros and cons to the table. Claude 2 is all about safe and thoughtful chat, but it's not as versatile or creative as ChatGPT. On the flip side, ChatGPT can do a lot—from answering tough questions to creative writing—but it's not without flaws, like sometimes giving wrong or misleading info.
When it comes to practical applications, Claude 2 is your go-to for straightforward, common-sense conversations and tasks that require sound advice. ChatGPT, on the other hand, is a jack-of-all-trades, skilled in handling complex queries, creative tasks, and even programming assistance. However, it can sometimes give wrong or confusing information and can also be too wordy.
Both AI bots have made a big impact, but in different ways. In essence, ChatGPT laid the foundational stones for what conversational AI could achieve, and now Claude 2 is coming in strong with a focus on being safe and ethical. The public is clearly interested, and this tech is only going to get better and more popular.
In the end, the best chatbot for you depends on what you need—be it creative help, safe interactions, or a wide range of features. Either way, the future of chatbots is something to keep an eye on.
