INTRODUCTION
Artificial intelligence is widely used for legal and medical advice because it is fast and accessible, but it creates serious risks when it gives incorrect information. Users may rely on AI for legal procedures or health decisions, and wrong answers can lead to loss of rights, delayed treatment, or unsafe actions. The key issue is not just a technical error but legal responsibility for harm. Since AI has no legal personality in India, liability may fall on developers, platforms, professionals, institutions, or even users, with existing laws like consumer protection, torts, and data protection helping determine accountability.[1][2][3][4]
MEANING OF AI HALLUCINATIONS
An AI mistake occurs when an AI system produces false or misleading information but presents it as accurate. While this may seem harmless in general use, it becomes dangerous in legal and medical contexts. AI may invent legal citations, misinterpret laws, misdiagnose symptoms, or suggest unsafe treatments, leading to serious consequences. Unlike ordinary software mistakes, these issues are risky because they appear confident and authoritative, making users trust them without verification. This turns the issue into one of reliability and trust. Legally, the key question is whether those who develop or use such AI systems took reasonable steps to prevent foreseeable harm.
CONSUMER PROTECTION ACT 2019 AND AI LIABILITY
Under the Consumer Protection Act 2019, AI service providers can be held liable for deficient services, misleading claims, or negligence[5]. If an AI tool is marketed as reliable for legal or medical advice and causes harm due to inaccurate output, users may claim deficiency in service or misrepresentation, especially when terms like “accurate” or “expert” are used.
Disclaimers such as “for informational purposes only” do not automatically remove liability. Courts will examine whether the provider encouraged reliance, gave clear warnings, and took reasonable steps to prevent harm. In high-risk areas like law and healthcare, such disclaimers cannot override the duty of care.
TORT LAW AND THE PRINCIPLE OF NEGLIGENCE
Indian tort law, though uncodified, provides a strong basis for liability through negligence, which requires duty of care, breach, causation, and damage. AI providers offering legal or medical advice can owe a duty of care to users, including proper design, testing, warnings, and safeguards against risks.
A breach occurs if systems are released despite known inaccuracies or lack safety measures, for example, misdiagnosing medical emergencies or inventing legal information while still being marketed as reliable.
Causation is often disputed, as providers may argue that users should verify advice. However, when AI systems are designed and marketed to inspire trust, user reliance becomes foreseeable. In such cases, liability may still arise despite user involvement.
LIABILITY IN THE MEDICAL CONTEXT
AI-generated medical advice raises serious concerns because errors can lead to injury or loss of life. In India, the Telemedicine Practice Guidelines require that remote healthcare be provided by registered doctors following professional standards, including proper evaluation, consent, and safe prescriptions. When AI is used, liability does not rest only on the developer. Doctors must use their own judgment and cannot rely blindly on AI, and hospitals cannot avoid responsibility by blaming the technology. Medical care in India remains a human professional duty, with AI only acting as a support tool. If an AI system wrongly assesses a patient’s condition, liability may arise at multiple levels against the developer for faulty design, the hospital for improper use, and the doctor for negligent reliance.
LIABILITY IN THE LEGAL SERVICE CONTEXT
The same principle applies to legal advice. Legal practice is a professional service grounded in competence, diligence, and ethical responsibility. A lawyer who relies on AI-generated research without verification cannot simply blame the software if a filing contains fabricated citations or incorrect legal propositions. The professional duty remains with the advocate or legal practitioner using the tool.
At the same time, standalone legal-tech platforms that directly advise consumers create a separate category of risk. If such platforms represent themselves as substitutes for lawyers, simplify complex legal issues into misleading automated outputs, or fail to disclose serious limitations, they may face claims under consumer law and negligence principles[6]. This is especially important in India, where many users turn to low-cost digital tools because formal legal assistance is expensive or inaccessible. The law must therefore protect users from being misled into relying on tools that imitate expertise without corresponding accountability.
IT RULES AND PLATFORM DUE DILIGENCE
The Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Rules, 2021, as updated by the government, do not create a complete AI liability regime, but they show the broader regulatory direction. They impose due diligence obligations on intermediaries and reflect increasing concern with harmful digital content and platform responsibility[7]. MeitY’s 2026 materials further indicate that the due diligence framework has been strengthened in relation to synthetically generated information.
Although not every AI advisory platform fits neatly within the category of intermediary, the regulatory message is clear: digital actors cannot claim total neutrality when their systems produce or spread harmful information. This principle becomes significant when a platform knowingly deploys AI systems capable of producing dangerous legal or medical outputs without adequate controls. Indian digital governance is gradually moving toward the idea that platforms must anticipate and manage technologically generated harms rather than deny responsibility after damage occurs.
DIGITAL PERSONAL DATA PROTECTION ACT,2023 AND AI SYSTEM
The Digital Personal Data Protection Act 2023 adds an important dimension to AI liability where personal data is involved[8]. Many legal and medical AI systems depend on sensitive user information, such as health records, financial details, and identity data. The Act establishes a framework for lawful processing, transparency, and responsible handling of such data, while recognising individuals’ rights to data protection.
Although it does not directly address every harm caused by AI errors, it makes clear that inaccurate outputs may arise not only from flawed algorithms but also from poor data governance, such as incomplete or improperly processed data. This creates an overlap between data protection obligations and substantive liability. The framework has further evolved through the Digital Personal Data Protection Rules, 2025, reflecting continued regulatory development in managing data-driven AI systems.
WHO SHOULD ULTIMATELY BE LIABLE
The most difficult question is how liability should be distributed. There is rarely a single wrongdoer in AI harm cases. The model developer may be responsible for inadequate training, design, or testing. The platform operator may be responsible for poor safeguards and misleading marketing. The hospital, doctor, law firm, or legal consultant may be responsible for negligent reliance. Even the user’s own conduct may be considered in determining contributory negligence.
Therefore, Indian law should move toward a chain-of-responsibility model. The law should ask:
- who designed the system,
- who deployed it in a high-risk environment,
- who marketed it as trustworthy,
- who exercised final professional judgment,
- and whether the user’s reliance was foreseeable.
Such an approach is more realistic than attempting to assign blame to the AI itself. AI does not possess legal personhood. Responsibility must remain with the human and institutional actors who created the conditions in which harm became possible.
NEED FOR A RISK – BASED AI LIABILITY FRAMEWORK IN INDIA
India would benefit from a specific risk-based AI framework, particularly for sectors such as law, medicine, education, finance, and public administration. At a minimum, such a framework should require:
- classification of high-risk AI systems,
- mandatory human oversight in legal and medical applications,
- strong warning and disclosure standards,
- output logging and audit trails,
- incident reporting for harmful errors,
- and accessible remedies for affected users.
The current legal position relies on adapting existing doctrines, which is useful but incomplete. A dedicated framework would improve certainty for consumers, professionals, and businesses. It would also promote innovation more responsibly by making clear that high-risk AI deployment demands stronger safeguards.
CONCLUSION
AI-generated legal and medical advice presents a complex challenge for liability in India, where harm cannot be attributed to the machine itself but must be traced to human and institutional actors. Existing legal frameworks—including consumer protection law[9], tort principles, professional standards, and data protection regulations[10]—collectively provide a foundation to address such harms, even though they were not designed specifically for AI. The key principle that emerges is accountability across the entire chain: from developers and platform operators to professionals who rely on these tools and deploy them in high-risk contexts.
As AI systems increasingly position themselves as reliable sources of expertise, the corresponding duty of care must rise proportionately. Disclaimers alone are insufficient where reliance is foreseeable, and consequences are serious. While current laws offer partial remedies, they remain reactive and fragmented. Therefore, India must move toward a clear, risk-based AI liability framework that ensures transparency, human oversight, and effective remedies. Only then can the balance between innovation and accountability be properly maintained, ensuring that technological advancement does not compromise legal rights or human safety.
Author: Ankit Raj (Shobhit University, Meerut)
References:
[1] Consumer Protection Act 2019
[2] Telemedicine Practice Guidelines (Ministry of Health and Family Welfare, 2020)
[3] Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Rules 2021
[4] Digital Personal Data Protection Act 2023
[5] Consumer Protection Act 2019
[6] Consumer Protection Act 2019
[7] Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Rules 2021
[8] Digital Personal Data Protection Act 2023
[9] Consumer Protection Act 2019
[10] Digital Personal Data Protection Act 2023
