How AI-Powered Phishing Scams May Soon Outpace SEO Attacks

Imagine asking your favorite AI chatbot for a safe login link—only to land on a cleverly disguised phishing site. Sound far-fetched? Not anymore. As attackers move beyond manipulating Google search, a new era of “LLM poisoning” is emerging. Today, the same playbook used to poison search results with malicious SEO tactics is being retooled to fool artificial intelligence and large language models (LLMs) like ChatGPT, Google Gemini, and Microsoft Copilot.

This isn’t just a hypothetical risk. Recent research from Netcraft reveals that LLMs sometimes recommend incorrect, unregistered, or downright dangerous domains in response to simple, natural questions—exactly what regular users would ask. As AI-powered search and chat become the default gateway to the web, a whole new attack surface opens up for cybercriminals.

Let’s break down how this works, why it matters, and what everyone—from tech companies to everyday users—needs to understand to stay safe.

The Shift: From SEO Poisoning to LLM Manipulation

Search engine optimization (SEO) isn’t just for businesses trying to get noticed. For years, cybercriminals have exploited SEO techniques—known as “SEO poisoning”—to push phishing and scam pages to the top of Google results. By mimicking keywords, content, and site structure, attackers trick both algorithms and humans into clicking the wrong links.

Now, with the rise of AI-powered chatbots and advanced LLMs, the battleground is shifting. Think of AI chat as your new digital assistant—and attackers are already figuring out how to whisper in its ear.

The Mechanics of LLM “Poisoning”

Here’s the new twist: LLMs generate answers based on massive datasets, scraping information from sources like websites, forums, and documentation. If attackers inject malicious content—say, by creating fake support pages, GitHub projects, or tutorials—they can manipulate what the AI learns or retrieves.

So when you ask, “What’s the official login page for MyBank?”, the AI might confidently suggest a phishing domain crafted by criminals, not the real one. No fancy prompt tricks needed—just exploiting the AI’s trust in whatever it’s seen online.

Real-World Example: Netcraft’s Eye-Opening Experiment

To test this, Netcraft researchers asked a GPT-4.1 model ordinary questions about login pages for 50 major brands—no prompt engineering, just natural language. The results were startling:

131 web addresses returned, tied to 97 unique domains.
34% of these domains weren’t owned by the brands—some were unregistered, inactive, or simply parked.
A handful belonged to unrelated legitimate companies.
Many domains could be snapped up by attackers and weaponized for phishing.

The bottom line? AI models, trusted by millions, can now serve up phishing links by accident, simply because the underlying data has been poisoned or hallucinated.

Why This Matters: The New Risks of AI-Driven Search

If you’re wondering, “Is this really different from old-school phishing?”—the answer is a resounding yes. Here’s why:

1. Trust in AI Is High—Even When It’s Wrong

When Google or Bing shows you a sketchy website, you might hesitate. But when your AI assistant recommends a link with confidence, it feels right—even if it’s dead wrong. LLMs don’t just surface results; they synthesize and present them as answers, often without clear warnings or context.

2. Attackers No Longer Need to Beat the Top Brands

Classic SEO poisoning was a slugfest—attackers had to outrank legitimate companies. But LLMs sometimes invent plausible-sounding URLs that aren’t even real brands yet. Criminals can simply register these “hallucinated” domains and wait for the AI to send users their way.

3. AI Search Is Rapidly Becoming Ubiquitous

With AI summaries now appearing atop traditional search results (thanks to tools like Google’s AI Overviews), these risks are front and center for everyone. The shift from link lists to “answers” is a double-edged sword: faster for users, but a jackpot for scammers if not handled carefully.

Let me explain why this evolution demands urgent attention—not just from AI developers, but from brands and end users, too.

How Attackers Are Gaming the System: Techniques in the Wild

Cybercriminals aren’t waiting for the future—they’re already adopting AI-savvy tactics. Here’s what we know:

Generating AI-Optimized Phishing Content

Attackers are using the same content-generation tools as legitimate companies. By flooding the web with legitimate-looking documentation, support pages, and GitHub repos, they create a credible digital footprint for their scam domains.

Example: Over 17,000 AI-generated GitBook phishing pages were found targeting crypto users, mimicking product docs and support info.
Trend: The travel sector and financial industry are now seeing similar tactics, with attackers crafting “clean, fast, and linguistically tuned” sites that appeal to both humans and machines.

Promoting Malicious APIs and Tools

Netcraft spotted a campaign where a fake blockchain API was promoted across blog posts, Q&A forums, and GitHub in an effort to get it indexed by AI training algorithms. The goal? When coders asked their LLM for recommendations, the AI might suggest this malware-laden API.

Here’s why that matters: Once in the system, these malicious references can echo across thousands of AI-generated answers, multiplying the impact of a single phishing campaign.

The LLM Hallucination Problem: Plausible URLs Gone Bad

One unique risk with LLMs is “hallucination”—the tendency of AI models to make up plausible-sounding but non-existent information, including web addresses. Unlike a typo, these hallucinated URLs often follow brand patterns and naming conventions, making them easy for attackers to spot and exploit.

What Are Hallucinated Domains?

Invented by the AI: The model “guesses” what a login URL might be, based on patterns in its training data.
Unclaimed or Parked: Many of these domains are unregistered at first, giving attackers a free runway.
Quickly Weaponized: As soon as an LLM suggests a non-existent but plausible domain, criminals can register it and build a phishing site overnight.

Why this is alarming: If you trust your AI to know best and click such a link, you may not notice you’re handing over your credentials to a cybercriminal.

AI Search Engines: AI Answers, Real-World Risks

AI-powered web search is no longer a novelty. Google’s Gemini, Microsoft Copilot, Perplexity, and other AI-driven platforms now generate summaries and direct answers to user queries, often placing them above traditional search results.

The Double-Edged Sword of AI Summaries

Pros: Saves time, reduces clutter, streamlines information gathering.
Cons: If the AI’s underlying data is poisoned or hallucinated, users get bad info—presented as fact.

Recent research shows these AI-generated answers may include links to malicious or irrelevant sites, delivered with authority. As more users rely on AI-powered search, the risks of accidental endorsement by trusted tools only grow.

What Can Be Done? Mitigating LLM-Driven Phishing Threats

The good news: There are steps that can reduce the risk of AI-assisted phishing. But it requires action from AI developers, brands, and end users alike.

For AI Developers and Model Providers

Implement URL Verification: Cross-check suggested domains against official brand registries before recommending them.
Deploy Guardrails: Limit or flag responses that suggest unverified or unowned domains.
Leverage Feedback Loops: Allow users to report incorrect or suspicious recommendations, improving model accuracy over time.
Integrate Trusted Sources: Prioritize information from reputable directories, official documentation, and known security feeds.

For Brands and Organizations

Monitor for Impersonation: Use threat intelligence tools to catch when AI models suggest lookalike or fake domains.
Register Lookalike Domains: Proactively claim plausible variations of your official URLs, reducing the attacker’s available targets.
Educate Your Users: Warn customers about AI-driven phishing and provide clear, official login paths.
Collaborate with AI Providers: Work directly with model developers to flag brand misuse or incorrect recommendations.

Explore guidance from the Cybersecurity & Infrastructure Security Agency (CISA).

For Everyday Users

Be Skeptical of AI-Suggested Links: If an AI chatbot gives you a login URL, double-check it before entering any credentials.
Bookmark Official Sites: Don’t rely solely on AI or search for sensitive logins—keep your official links saved.
Look for HTTPS and Valid Certificates: While not foolproof, secure connections help spot some scams.
Report Suspicious Activity: If you spot a fake site recommended by an AI, report it to the provider and relevant authorities.

The Future: Will LLM Phishing Outpace Traditional Attacks?

It’s tempting to hope that AI security will catch up just as quickly as AI capabilities. But history shows attackers move faster than defenders, especially in new digital frontiers.

As AI becomes more integrated into daily life, expect cybercriminals to hone their tactics—tailoring content, promoting malicious domains, and gaming the algorithms that power our digital assistants. Brands that ignore this shift risk seeing their customers phished by links their own AI partners unwittingly recommend.

Actionable Insight: If you’re a brand, now is the time to get proactive about LLM security and domain monitoring. If you’re a user, stay alert and confirm before you click—especially when an AI gives you an answer that seems “too easy.” We’re entering a new era where “trust, but verify” isn’t just good advice—it’s essential.

Frequently Asked Questions (FAQs)

Q: What is LLM poisoning in cybersecurity?
A: LLM poisoning refers to manipulating the outputs of large language models (LLMs) like ChatGPT or Google Gemini by injecting malicious or misleading content into their training data or indexed sources. Attackers aim to get the AI to recommend phishing sites, scam tools, or incorrect information, often without any prompt manipulation.

Q: How is LLM-driven phishing different from traditional SEO poisoning?
A: Traditional SEO poisoning tries to push malicious sites up in search engine rankings. LLM-driven phishing targets the AI’s answer generation—by influencing the AI’s knowledge base, attackers can get the model to recommend bad links as if they were trustworthy.

Q: Are AI chatbots safe to use for finding login links or sensitive information?
A: While AI chatbots are improving, they can sometimes hallucinate or suggest incorrect, unverified, or malicious domains. Always double-check any login links provided by AI, and bookmark official sites for sensitive accounts.

Q: What should brands do to protect against AI-enabled phishing?
A: Key steps include monitoring for brand impersonation, registering plausible lookalike domains, collaborating with AI providers, and educating users about potential risks.

Q: How can end users protect themselves from AI-generated phishing attacks?
A: Never rely solely on AI-suggested links for logins. Use bookmarks, verify URLs, check for HTTPS, and report suspicious sites or AI answers.

Final Takeaway: Stay Informed, Stay Vigilant

The rise of large language models and AI-powered search is transforming how we access information—but it’s also opening new doors for cybercriminals. As attackers pivot from SEO poisoning to LLM manipulation, the need for vigilance has never been higher.

Stay curious, stay cautious, and always verify before you trust AI-generated answers—especially when your security is on the line. Want to dive deeper into the evolving world of AI and cybersecurity? Subscribe to our newsletter for expert insights and practical tips.

For more on AI and online safety, check out Dark Reading’s coverage and Netcraft’s threat intelligence updates.

Discover more at InnoVirtuoso.com

I would love some feedback on my writing so if you have any, please don’t hesitate to leave a comment around here or in any platforms that is convenient for you.

For more on tech and other topics, explore InnoVirtuoso.com anytime. Subscribe to my newsletter and join our growing community—we’ll create something magical together. I promise, it’ll never be boring!

Stay updated with the latest news—subscribe to our newsletter today!

Thank you all—wishing you an amazing day ahead!

How AI-Powered Phishing Scams May Soon Outpace SEO Attacks—And What You Need to Know