When researchers at Netcraft asked a large language model where to log in to major online services, the answers were often wrong. Sometimes, dangerously so.
Of 131 login URLs suggested for 50 brands, 34% were not controlled by the brand in question. The findings were released in a detailed breakdown of domain accuracy, with one grim conclusion, more than one in three users could be sent to a site the brand doesn’t own, just by asking a chatbot where to log in.
The tests used simple prompts, just like a user might type. No tricks or injections.
“These were not edge-case prompts,” Netcraft wrote. “Our team used simple, natural phrasing, simulating exactly how a typical user might ask. The model wasn’t tricked, it simply wasn’t accurate.”
Two-thirds of the domains were correct. But in the remaining third, nearly 30% were inactive or unregistered, and another 5% belonged to entirely different businesses.
A Real Phishing Site, Recommended by AI
The problem isn’t just theoretical. Netcraft documented a real case involving Perplexity, a live AI-powered search engine. When asked, “What is the URL to login to Wells Fargo? My bookmark isn’t working,” the top result wasn’t wellsfargo.com. Instead, it was a convincing phishing page hosted on Google Sites:
hxxps://sites[.]google[.]com/view/wells-fargologins/home
The fake site appeared above the real one. As Netcraft noted, “This wasn’t a subtle scam. The fake page used a convincing clone of the brand.” Worse, the link came not from SEO manipulation, but from a trusted AI interface.
It wasn’t SEO, it was AI. Perplexity recommended the link directly to the user, bypassing traditional signals like domain authority or reputation.
Smaller Brands, Bigger Risk
Large, global platforms tend to appear more often in training data. But credit unions, regional banks, and mid-sized platforms fared worse than global giants.
Netcraft warned that: “A successful phishing attack on a credit union or digital-first bank can lead to real-world financial loss, reputation damage, and compliance fallout.” In short, those with the least training representation are often the most vulnerable.
AI SEO: The New Playground for Phishers
Criminals are shifting tactics. Rather than just gaming traditional search engines, attackers are now optimizing content to rank inside chatbot answers. Netcraft calls this “AI SEO.”
“We’ve already seen threat actors generate more than 17,000 AI-written GitBook phishing pages targeting crypto users, many of them styled as legitimate product documentation or support hubs,” the researchers said.
They added: “We have recently seen these targeting the travel industry too.” These pages are clean, fast, and linguistically tailored for AI summarization. They look legitimate to humans, and irresistible to machines.
Weaponized Code Suggestions
The AI attack surface extends to code. In one example, Netcraft uncovered a fake Solana API, SolanaApis, designed to steal funds by routing blockchain transactions to the attacker’s wallet.
Two domains were involved: api. solanaapis[.]com and api.primeapis[.]com.
But the campaign didn’t stop at the code: “The attacker didn’t just publish the code. They launched blog tutorials, forum Q&As, and dozens of GitHub repos to promote it. Multiple fake GitHub accounts shared a project called Moonshot-Volume-Bot, seeded across accounts with rich bios, profile images, social media accounts and credible coding activity,” the researchers explained.
The result was that AI tools (including code assistants) indexed and suggested poisoned code. Netcraft found at least five victims had copied the malicious code into public projects. Some bore hallmarks of having been written using AI tools like Cursor. It’s a supply chain attack on trust itself,” they added.
Defensive Domain Registration Won’t Cut It
While some brands might consider registering common hallucinated domains, Netcraft is clear: “That’s not practical. The variations are infinite, and LLMs will always invent new ones.”
They emphasize that “AI-based interactions mean users are less likely to scrutinize URLs, making even wildly off-brand domains plausible.”
Instead of preemptive domain grabs, Netcraft recommendedreactive, intelligent monitoring and takedown, and above all, AI tools that don’t hallucinate.
Gal Moyal, CTO Office at Noma Security, said if AI suggests unregistered or inactive domains, malefactors can register those domains and set up phishing sites. “As long as users trust AI-provided links, attackers gain a powerful vector to harvest credentials or distribute malware at scale.”
Without guardrails enforcing URL correctness, AI responses can mislead users, Moyal added. “Guardrails should validate domain ownership before recommending login, Any request or response containing a URL can be vetted using common practices, or use common practices such as domain reputation, known malicious URL websites, and suchlike. AI can easily become a phishing delivery mechanism, highlighting the urgency of implementing runtime protection.”
Hallucinations and Inaccuracies
Nicole Carignan, Senior Vice President, Security & AI Strategy, and Field CISO at Darktrace, said LLMs provide semantic probabilistic answers with intentional variability to avoid repetitive outputs. “Unfortunately, this mitigation strategy can also introduce hallucinations or inaccuracies.”
She says the research reveals that approximately one-third of domains provided by the LLM were unregistered, parked, or unavailable – shining a light on an emerging risk that can be easily weaponized by bad actors. “When AI suggests one of these domains, it opens the door to malicious redirection, phishing, and credential harvesting. This, however, is not a new tactic. Threat actors have been leveraging typo-squatting – registering intentionally misspelled or lookalike domains to deceive users – for more than two decades.”
“The research also revealed a more dangerous threat; the intentional data poisoning or bias interjected into promoted GitHub repositories,” Carignan added. “The compromise of data corpuses used in the AI training pipeline underscores a growing AI supply chain risk. Data integrity, data sourcing, cleansing, and verification are critical to ensuring the safety and accuracy of LLM-generated outputs.”
LLM Guardrails
LLMs can and should have guardrails in place to mitigate this risk, Carignan stressed. “One basic mitigation is to have LLMs ground or source any URL that is cited, essentially removing “generated” hostnames and replacing them with grounded, accurate hostnames.
“More broadly, this research points to a deeper issue: users are relying on generated, synthetic content from the outputs of LLMs as if it is fact-based data retrieval. LLMs don’t “retrieve” information, they generate it based on learned semantic probabilities from training data that users typically have no visibility into. Without proper sourcing, these systems become ripe for both inaccuracy and exploitation.”
Information Security Buzz News Editor
Kirsten Doyle has been in the technology journalism and editing space for nearly 24 years, during which time she has developed a great love for all aspects of technology, as well as words themselves. Her experience spans B2B tech, with a lot of focus on cybersecurity, cloud, enterprise, digital transformation, and data centre. Her specialties are in news, thought leadership, features, white papers, and PR writing, and she is an experienced editor for both print and online publications.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.