The Replika Case: A Governance Warning for Leaders Deploying AI

The Italian data protection authority’s €5 million fine against Luka Inc., operator of the Replika AI companion, should be read as a governance case — not simply a privacy enforcement story.

Conversational AI is designed to feel immediate and personal. That is precisely why weak controls can have serious consequences. Users may disclose intimate information, form emotional reliance on the service or misunderstand the nature and limits of the system. Where children can access the product, the standard of care rises further.

What leaders should take from the case

Product design is part of compliance. Privacy notices and contractual wording cannot compensate for an experience that encourages sensitive disclosure without proportionate safeguards.

Age assurance must be credible. A system likely to attract minors cannot rely on a superficial declaration of age. The control must reflect the risk, avoid collecting unnecessary data and be tested in practice.

A lawful basis is not a box-ticking exercise. Organisations need to understand which personal data is processed at every stage, why that processing is necessary and whether users can reasonably understand the consequences.

AI behaviour needs ongoing oversight. A conversational model changes through prompts, configuration, updates and user context. Governance must cover monitoring, harmful-output escalation, incident response and the ability to intervene quickly.

The wider regulatory picture

The same system can engage GDPR, consumer-protection rules, child-safety duties, cybersecurity requirements and the EU AI Act. Treating each regulation as a separate legal checklist creates blind spots. The operating model must connect product, security, data protection, legal and executive ownership.

Questions to ask before launch

• Which users could be harmed, and how?
• What sensitive information is the product likely to elicit?
• How is age-related risk addressed?
• Can the organisation explain and evidence its lawful basis?
• Who monitors model behaviour and approves material changes?
• How can users report a harmful interaction and obtain a timely response?

The lesson is straightforward: trust cannot be added after the product reaches the market. For AI systems that interact directly with people, trust must be designed, tested and governed from the start.