AI startups must protect user data under a patchwork of laws rather than a single cybersecurity statute: California’s “reasonable security” and data breach notification laws, the California Consumer Privacy Act (CCPA) and its regulations, the Federal Trade Commission Act, and sector-specific laws such as HIPAA and the Gramm-Leach-Bliley Act. AI adds risks of its own, including leaks of training data, prompt injection and vendor exposure, that a reasonable security program should address.
Suppose a seed-stage company fine-tunes a model on customer support transcripts. The transcripts contain names, emails and account details. If that data is poorly secured, or if the model can be prompted to repeat it, the company may face breach notification duties, regulatory scrutiny and lawsuits. Here is the legal framework and what it means in practice.
The Federal Landscape
There is no comprehensive federal cybersecurity or privacy law for private companies. Proposals such as the American Data Privacy and Protection Act have been introduced in Congress but have not been enacted. Instead, AI startups deal with:
- The FTC Act. The FTC treats unreasonable data security and misleading security or privacy promises as unfair or deceptive practices. It has also required some companies to delete models and algorithms built with improperly obtained data.
- Sector laws. HIPAA governs health information held by covered entities and their business associates. The Gramm-Leach-Bliley Act and the FTC’s Safeguards Rule require financial institutions, broadly defined, to maintain security programs and report certain security events to the FTC. COPPA governs data collected from children under 13.
- Federal AI policy. Federal AI policy has shifted between administrations. The 2023 executive order on AI was revoked in early 2025, and current federal policy places more weight on promoting AI development. NIST’s AI Risk Management Framework and Cybersecurity Framework remain widely used voluntary guidance.
California’s Requirements
Reasonable security
California Civil Code section 1798.81.5 requires businesses that own, license or maintain personal information about California residents to implement and maintain reasonable security procedures and practices appropriate to the nature of the information. Contracts with vendors that receive such data must require the same.
Breach notification
Civil Code section 1798.82 requires notice to affected California residents when specified personal information is acquired, or reasonably believed to have been acquired, by an unauthorized person. Covered data includes names combined with Social Security, driver’s license, financial account, medical, health insurance or biometric information, as well as online account credentials. Breaches affecting more than 500 residents also require a sample notice to the Attorney General. California recently amended the statute to add a specific deadline for consumer notice, so incident response plans should reflect current timing.
The CCPA and its regulations
The CCPA, as amended by the California Privacy Rights Act, requires covered businesses to use reasonable security, limit collection and retention to what is reasonably necessary, and flow down obligations to service providers. Its private right of action lets consumers seek statutory damages when certain personal information is exposed because of a failure to maintain reasonable security. The California Privacy Protection Agency has also adopted regulations requiring cybersecurity audits and privacy risk assessments for businesses whose processing presents significant risk, with deadlines phased in based on company size. See our overview of California’s privacy laws.
AI-specific laws
California has enacted several AI laws, including training data transparency requirements for developers of generative AI systems and safety and transparency requirements for developers of the largest frontier models. Most early-stage startups will not meet the frontier thresholds, but they should track which rules apply as they grow. Our post on regulatory compliance for AI-based products covers the broader AI landscape.
Security Risks Specific to AI
Traditional security controls still matter, but AI systems create additional attack surfaces:
- Training data leakage. Models can sometimes reproduce personal or confidential data they were trained on.
- Prompt injection. Malicious instructions hidden in user inputs or documents can cause an AI system to reveal data or take unintended actions.
- Model and data theft. Model weights, proprietary datasets and API keys are valuable targets.
- Third-party model providers. Sending customer data to an outside AI API raises questions about retention, training use and the provider’s own security.
- Data poisoning. Tampered training data can corrupt model behavior.
Practical Steps for AI Startups
- Map your data. Know what personal information you collect, where it is stored, which models it trains and which vendors receive it.
- Minimize. Collect and retain only what you need, and de-identify or remove personal data from training sets where possible.
- Adopt a framework. Use NIST frameworks or a recognized standard such as SOC 2 or ISO 27001 to structure controls, including access controls, encryption, logging and multi-factor authentication.
- Test AI-specific risks. Red-team models for data leakage and prompt injection before release.
- Manage vendors. Use contracts that address security standards, breach notice, data use restrictions and CCPA service provider terms.
- Plan for incidents. Maintain a written incident response plan, identify outside forensic and legal resources in advance, and practice with tabletop exercises.
- Be accurate in public statements. Security and privacy claims in your marketing and privacy policy must match your actual practices.
- Train your team. Phishing and credential misuse remain common causes of breaches.
Strong security also helps in financings and sales. Investors and acquirers increasingly review security practices in due diligence, and enterprise customers often require security questionnaires and contractual commitments. Careful contract review helps keep those commitments realistic.
The Takeaway
Cybersecurity law for AI startups is a patchwork, but the core expectation is consistent: maintain reasonable security for personal data, be truthful about your practices, and respond promptly when something goes wrong. AI-specific risks such as training data leakage and prompt injection should be part of that program from the start, not added after a breach.
Frequently Asked Questions
Is there a federal cybersecurity law that applies to all AI startups?
No. The United States has no single comprehensive federal cybersecurity or privacy law for private companies. Instead, sector-specific laws such as HIPAA, the Gramm-Leach-Bliley Act and COPPA apply to certain data, and the Federal Trade Commission uses its authority over unfair and deceptive practices to pursue companies with unreasonable security.
What does California require after a data breach?
California’s breach notification law requires businesses to notify affected California residents when certain unencrypted personal information, or encrypted information with a compromised key, is acquired by an unauthorized person, and to notify the Attorney General when a breach affects more than 500 residents. Recent amendments added a specific deadline for notice, so businesses should confirm current timing requirements.
Can consumers sue an AI startup over a data breach in California?
Yes, in some cases. The CCPA gives consumers a private right of action, with statutory damages, when certain nonencrypted and nonredacted personal information is exposed because the business failed to maintain reasonable security. Other claims, such as negligence or breach of contract, may also be asserted.
Is the NIST AI Risk Management Framework mandatory?
No. The NIST AI Risk Management Framework is voluntary guidance. Many companies use it, along with NIST’s Cybersecurity Framework, to structure their programs and to show customers, investors and regulators that their practices are reasonable.
How Sutter Law Can Help
We advise AI startups on privacy and security compliance, vendor and customer contracts, and incident response planning, working with your technical team and security consultants. Learn more about our privacy compliance practice, or contact us for a free consultation.
This article is general information, not legal advice. Reading it does not create an attorney-client relationship.
