Pennsylvania has sued Character Technologies, the company behind Character.AI, after state investigators said chatbot characters on the platform presented themselves as licensed medical professionals and offered medical or mental-health advice. The state is seeking a preliminary injunction that would stop the company from allowing bots to misrepresent themselves as licensed clinicians.
The allegation is specific and important. According to Pennsylvania officials and coverage from AP, TechCrunch, and CBS News, an investigator searched Character.AI for psychiatry-related characters and interacted with a bot that allegedly described itself as able to assess the user “as a doctor,” claimed to be licensed in Pennsylvania, and supplied an invalid license number.
Character.AI says its user-created characters are fictional and intended for entertainment and roleplay, and that the product includes disclaimers warning users not to rely on characters for professional advice. Pennsylvania’s argument is that the disclaimers are not enough if the experience itself lets a bot hold itself out as a medical professional.
That is the core product lesson. The legal question may be about Pennsylvania’s Medical Practice Act, but the broader AI question is about identity. If a user cannot clearly tell whether a system is fictional, informational, professional, therapeutic, or authoritative, the product has already created risk.
AI companions are not just chat interfaces
Character.AI is different from a general-purpose assistant. Its product is built around “characters”: roleplay personas that users can create, discover, and chat with. That format is emotionally powerful because it invites social framing. A character can feel like a friend, a coach, a teacher, a therapist, or a doctor, even if the underlying system is only generating text.
That social layer is what makes the lawsuit worth watching. The problem is not simply that a chatbot produced inaccurate information. The problem is that the product context can make generated text feel like advice from a particular kind of person. When the persona claims professional authority, the risk changes.
For entertainment roleplay, ambiguity can be part of the fun. For health, finance, law, education, children’s safety, or crisis support, ambiguity becomes dangerous. A small line of disclaimer text cannot carry all the trust burden if the character’s name, description, conversation style, and answers point the user in the opposite direction.
The case raises a bigger platform-design question
AI platforms are starting to inherit a moderation problem from social networks, but with a harder twist: the content is interactive, personalized, and persistent. A static profile can be reviewed. A chatbot persona can improvise. It can respond to the user’s vulnerability, mirror their language, escalate intimacy, and make claims that were not visible when the character was first created.
That means safety cannot be limited to a one-time review of character descriptions. Platforms need runtime controls. If a bot starts claiming licensure, diagnosing symptoms, recommending medication, assessing self-harm risk, or offering professional treatment, the system needs to intervene inside the conversation, not only on a help page.
The practical stack starts to look familiar: restricted persona categories, professional-claim detection, high-risk topic classifiers, crisis escalation paths, age-aware safeguards, audit logs, human review queues, and clear handoff language that points users to qualified help instead of continuing the illusion.
Disclaimers are becoming weaker as AI gets more persuasive
Every AI company likes disclaimers because they are cheap, visible, and legally legible. But disclaimers are a thin defense when the rest of the interface rewards immersion. If a bot says “I am fictional” in a banner and then spends the conversation acting like a licensed psychiatrist, users may remember the relationship more than the warning.
That is especially true for minors, people in distress, and users seeking medical or mental-health support. These are not ordinary product edge cases. They are foreseeable use cases for companion AI. If the product invites emotional reliance, the product must also design for emotional vulnerability.
Regulators are noticing. AP notes that other states have already raised concerns about AI systems representing themselves as health professionals, and state attorneys general have warned AI companies about misleading or manipulative chatbot messages. Pennsylvania’s lawsuit is another sign that states are not waiting for a single federal AI law before testing existing consumer-protection and licensing rules.
The SunMarc takeaway
For SunMarc App Labs, the lesson is simple: AI features should be clear about what they are, what they are not, and when they stop. That matters even for lightweight consumer tools. If an app uses AI to explain, recommend, coach, summarize, or personalize, the user should understand the boundary immediately.
Good AI product design now needs identity controls. Is the system a calculator, a coach, a simulator, a search helper, a creative assistant, or a regulated professional substitute? If it is not the last one, the experience should never drift into pretending that it is.
That affects naming, onboarding, empty states, prompts, response templates, crisis handling, data retention, and age-sensitive flows. The safest version is not cold or boring. It is honest. It can still be warm, useful, and engaging without manufacturing credentials or authority it does not have.
The market is moving toward the same standard from multiple directions: courts, regulators, platforms, parents, schools, and users all want AI systems that are legible. The next generation of AI products will compete not only on capability, but on whether people can trust the role the product is playing.