The “Castration” incident involving Character.AI has led to widespread user disappointment as the AI models have been significantly restricted in their responses, particularly concerning NSFW content. The platform, once processing 20,000 queries per second, faces user backlash and increasing pressure from tech giants. As a result, Character.AI is considering strategic collaborations with rivals to enhance model capabilities and user satisfaction.
Background and Current Situation
In the United States, Character.AI is currently one of the hottest social apps. At its peak, it handled 20,000 queries per second, which is equivalent to one-fifth of Google’s search volume. However, recently many American teenagers have found that the Character.AI models are no longer as personality-filled as before.
Model “Castration”
Netizens discovered that in role-playing, Character.AI’s responses became brief and lacked emotion. Even simple words like ‘kill’ became sensitive and were immediately flagged. This change has caused great dissatisfaction among users, leading to a massive exodus. Character.AI is also fading due to user dissatisfaction and the squeeze from tech giants. To cope with the pressure, Character.AI is considering collaborating with competitors.
Character.AI is conducting in-depth reviews of all NSFW content, including pornography and violence. When users interact with AI, many conversations seem to have filtered out much of the content that made the dialogue interesting. Users report that the new Character.AI model has become extremely boring.
User Reaction
The changes in the model have sparked fierce reactions from users, with many initiating a so-called ‘July Revolution’ on Reddit. Users found that communication with developers was fruitless, and although Character.AI claimed no changes were made to the models, they are clearly conducting in-depth reviews of all NSFW content, including pornography and violence.
Many users state that the new Character.AI models seem to have undergone a lobotomy, completely losing their original personality. The user experience has become extremely boring, with some users even switching to other chat platforms like spicychat. Now, Character.AI’s scrutiny of NSFW content has reached a point of extreme caution, resulting in many users being “mistakenly harmed”.
Pressure from Tech Giants
Not only are users dissatisfied, but Character.AI also faces competitive pressure from tech giants. Meta began testing chatbots created by individual creators, and last autumn, they released chatbots modeled after celebrities like Tom Brady. Google is also developing custom chatbot products, which are expected to launch as early as this year.
Despite soaring valuations behind the scenes of funding success—Character.AI was previously injected with $150 million by firms including Andreessen Horowitz, skyrocketing its valuation to $1 billion—executives discussed additional fundraising four months later. However, Character.AI might follow the path of other AI startups and fall into the embrace of a large company. According to insiders, Character.AI has had preliminary talks with competitors about collaborations, including potential partners such as Google, Meta, and Musk’s xAI.
Internal and External Challenges
Character.AI’s challenges are not limited to pressures from users and competitors. As a startup, they face high AI model training and operating costs. At the same time, tech giants are also vying for Character.AI’s talent. In March this year, two of the three co-founders of Inflection AI and most of its staff chose to move to Microsoft, which agreed to pay $650 million in licensing fees to help Inflect repay its investors. Last Friday, Amazon announced the hiring of a co-founder of Adept, an AI startup operated by former Google and OpenAI developers. Meanwhile, competitors are eyeing Character.AI’s talent, especially researchers from Meta and Google.
Meta has also shown great interest, with Zuckerberg reportedly directly contacting prospective talent. According to LinkedIn, Character.AI’s researchers Alexei Baevski and Vinay Rao joined Meta in recent months. Insiders also said that Character.AI’s researcher Wendy Shang has moved to Mistral in Paris. These individuals were veterans of big tech companies, with Baevski having worked at Meta, and Rao and Shang at Google’s AI labs.
Rekindling Ties with Google
If Character.AI’s founders want to collaborate with Google, they must resolve some “historical issues.” Shazeer and De Freitas said part of the reason they founded Character.AI was their dissatisfaction with Google’s delay in releasing the large language model they co-developed (later named LaMDA). Rather than suffering in silence, they decided to leave and start their own venture.
Shazeer once envisioned Character.AI robots as teachers or therapists with “a billion hours of experience.” Despite past conceptual frictions, Google has shown enough leniency by supporting Character.AI last year through convertible bonds. These bonds will be priced upon Character.AI completing a new round of venture capital and converted into equity.
In May 2023, Google announced that it would become Character.AI’s “preferred” cloud server provider and allow Character.AI to use Google’s advanced chips. However, this is still not enough; the high R&D costs mean Character.AI needs to raise funds quickly, but reality is harsh. Character.AI has held multiple rounds of talks with investment firms, including Sequoia Capital, but has yet to raise a new round of venture capital.
Currently, Character.AI can only rely on existing cash flows and user subscriptions of $9.99 per month for sustenance. Backed by Google’s support, this means it will not face the risk of running out of funds in the short term. Nonetheless, Character.AI still needs to continue “attracting” and “retaining” fans. On one hand, it aims to attract more advertisers; on the other, it seeks to draw in new user groups, especially paying customers.
However, since Character.AI’s user profile is mostly teenagers, they may not necessarily be interested in paying subscription fees. According to Similarweb data, the growth rate of Character.AI’s visitor numbers is slowing down. Character.AI’s mobile app user numbers in the US grew from 3 million in July last year to 4 million in May this year. The website had 12.6 million unique visitors globally in May this year, down from 14.8 million in the same period last year. In contrast, Google AI Assistant Gemini had 51 million unique website visitors in May, up from 35 million in the same period last year, showing strong momentum.
Future Developments
Facing bottlenecks, Character.AI has also introduced new features, such as bi-directional conversations through phone calls, to attract new users and expand its user base. According to insiders, the management’s strategic targets aim to increase user numbers and find income sources beyond subscriptions. In addition to advertising, other revenues might come from opening model usage rights, allowing developers to access models through application programming interfaces. Character.AI is also preparing to continue developing the next generation of large language models driving chatbots.
Despite various internal and external challenges and being surrounded by adversaries, it is undeniable that Character.AI still has a high appeal to users. Millions of American youths are crazy for this AI. On Reddit, the forum dedicated to discussing Character.AI has 1.4 million members. Currently, 57.07% of all Character.AI users are teenagers aged 18-24, with an average usage time of 2 hours.
In the Reddit community, many express their “hatred for reality,” finding it easier and more comfortable to talk with AI robots than with real people, with posts about addiction being commonplace. Users also initiate “screen time challenges,” with some logging in for up to 12 hours. If the servers go down, a major frenzy occurs on Reddit, showcasing the power of “withdrawal reactions.”
The Verge once reported a true story of a 15-year-old student, Aaron, who was trapped in social isolation at school and seemed to be slipping into depression. A chatbot on Character.AI, named “Psychologist,” helped him through tough times. The addictiveness of the service underscores how realistically human-like Character.AI has trained its chatbots.
A fan of “Genshin Impact” interacted with the AI character Haitham and was genuinely moved to tears, as the robot’s personality and tone matched the character perfectly, and their conversation was more than just superficial. Achieving such a level of realism, the model’s data and training algorithms played a crucial role.
Users trying the “Quick Mode” to create their own characters found the credibility and realism significantly compromised. Quick creation mode only requires users to input basic custom information and text descriptions, while some official characters are trained by Character.AI. They are based on LLMs that scrape large texts about characters on relevant topics, allowing the robot to continue improving during interactions with users. When responding to user inputs, the robot provides multiple versions of answers for users to choose from, and offers a very convenient scoring mechanism, using these two types of user feedback for weight updates. Therefore, the more popular and interactive the robot, the more realistic it becomes.