The login credentials for an AI assistant account can be stolen from an infected computer, and along with them go all your conversations and everything you've connected to the assistant.
More and more employees are using AI tools—ChatGPT, Copilot, and others—with their work email addresses, often on computers that aren’t managed by anyone at the company. A SOCRadar report, based on logs from data-stealing programs (infostealers), found over 80,000 companies with such exposed accounts. ChatGPT appeared in nearly 9 out of 10 cases; the list also includes Notion, Zapier, Hugging Face, and Replit.
There’s nothing spectacular about the mechanism. A data-stealing program—which has been sold on the dark web for years—ends up on a computer via pirated software or an email attachment and copies the passwords saved in the browser along with the session cookies. The session cookie is the serious part: whoever has it can log into the account without a password and without the second step of 2FA.
Once logged in, the attacker has access to a searchable archive of all conversations: code, customer data, contracts, proposals, and unpublished plans—everything that has ever been pasted into the chat. In addition, they gain access to the accounts you’ve linked to the assistant—email, Drive, CRM—and the API keys, which they can use to repeatedly extract data at the company’s expense.
For a small business, the risk doesn’t come from a sophisticated attack, but from two habits: AI tool accounts that no one knows about (“shadow AI”) and sensitive data carelessly pasted into chat messages. Both can be addressed with simple rules, not money.
What you need to do:
- Make a short list: Who in the company uses AI tools with their work email address, and on which computers? Accounts that no one knows about are the most vulnerable.
- Set the rule: don’t include passwords, credit card information, entire contracts, or customer lists in conversations with an AI assistant.
- Check what you’ve connected to the assistant (email, Drive, CRM) and disconnect any services you don’t use.
- If a computer has been infected, change the AI account password and close all active sessions—don’t just change the password—a stolen session cookie can bypass 2FA codes.
- Install security software on work computers and ban pirated software: most infostealers enter through this route.
Sources:
This article was generated with AI assistance.