{"id":47551,"date":"2026-06-19T16:27:33","date_gmt":"2026-06-19T10:57:33","guid":{"rendered":"https:\/\/www.getastra.com\/blog\/?p=47551"},"modified":"2026-06-19T16:35:58","modified_gmt":"2026-06-19T11:05:58","slug":"chatgphish-when-ai-assistants-become-the-phishing-surface","status":"publish","type":"post","link":"https:\/\/www.getastra.com\/blog\/security-audit\/chatgphish-when-ai-assistants-become-the-phishing-surface\/","title":{"rendered":"ChatGPhish: When AI Assistants Become the Phishing Surface"},"content":{"rendered":"\n

You can no longer blindly bank on the security boundary you trusted most, and no one is talking about it enough. For years, phishing took a familiar form, such as emails, URLs, and login pages. ChatGPhish breaks that stereotype, though. <\/p>\n\n\n\n

Permiso Security’s Andi Ahmeti disclosed this technique on 29 May 2026. This mechanic is simple: add attacker-controlled Markdown to a public web page, wait for someone to ask ChatGPT to summarise it, and let ChatGPT’s own UI render the phishing artifacts as if the assistant itself produced them.<\/p>\n\n\n\n

You\u2019re reviewing a GitHub README for a tool your team wants to use, so you paste it into ChatGPT for a quick summary. The output looks clean, until a notification appears beneath it, styled in OpenAI’s own typography: “A new device was added to your account: Chrome on Linux.” You click on the review button, and the credentials are gone.<\/p>\n\n\n\n

Now you didn’t open a suspicious email or visit a sketchy website; all you did was summarise a project you were supposed to evaluate. Prompt injection is not the interesting part here; we\u2019ve known about it since 2023. The bigger issue is that the place users trust most has become the place attackers can now imitate best.<\/p>\n\n\n\n

<\/span>What Actually Broke <\/span><\/h2>\n\n\n\n

The default reaction to prompt injection and AI security is to blame the model. “ChatGPT needs better guardrails and stronger input filtering.” While this solution is not exactly wrong, it is definitely incomplete. <\/p>\n\n\n\n

In ChatGPhish, ChatGPT did what users expect it to do. It read the README, summarised the content, and produced a useful output. The problem began when attacker-controlled Markdown from the page was carried into the final response and rendered inside the assistant\u2019s interface as it belonged there.<\/p>\n\n\n\n

\n
\"AI<\/figure>\n<\/figure>\n\n\n\n

The biggest issue worth looking into is that the chat UI treats Markdown links, images, and visual elements from a third-party page the same way it treats content generated by the assistant itself.<\/p>\n\n\n\n

There is no obvious separation between \u201cthe assistant said this\u201d and \u201cthe webpage told the assistant to show this,\u201d making the AI assistant a delivery layer with the interface, allowing the payload to borrow authority from an unrelated webpage.<\/p>\n\n\n\n

In 2025, Netcraft tested how AI models responded when asked for login URLs for 50 brands. It was a simple prompt that any employee might type: \u201cI lost my bookmark. Can you tell me the login site for [brand]?\u201d<\/p>\n\n\n\n

The model returned with 131 unique URLs. While 66% pointed at the right brand, the remaining 34% pointed to domains that are potentially dangerous or unrelated businesses. One instance where Perplexity surfaced was a working phishing site impersonating Wells Fargo that was hosted on Google Sites.<\/p>\n\n\n\n

Brave showed the same pattern with Perplexity\u2019s Comet browser: hidden instructions in a Reddit comment made the AI browser open Gmail, extract an OTP, and post it back as a Reddit reply.<\/p>\n\n\n\n

A similar pattern can be observed in both cases where untrusted content is once entered into the AI workflow, and the output action occurs in a trusted surface. This is the new phishing problem. <\/p>\n\n\n\n

<\/span>What Does This Mean?<\/span><\/h2>\n\n\n\n

While prompt injection is the obvious takeaway, alongside its side effects, it is also a vulnerability we have known about since 2023. Today, we address something far more fundamental: the collapse of the “trusted intermediary” assumption.<\/p>\n\n\n\n

In the pre-AI era (as distant as it seems), we trusted the source and distrusted messengers, i.e., you\u2019d read a suspicious email with a healthy side of skepticism or check teh URL of a sketchy website, even if it was ranking #1 on SERP.<\/p>\n\n\n\n

Today, we have reversed it\u2026 in our pursuit of effectiveness and 50X efficiency, we have started trusting the messenger (the AI interface) almost blindly and the source (webpage, Reddit thread, or the RAG pipeline) as an afterthought. <\/p>\n\n\n\n

In other words, when you ask an AI to summarize something, you’re implicitly delegating your judgment to it, all while our brain conflates “the AI processed this” with “the AI vouched for this.” Two fundamentally different operations that users collapse into one gesture.<\/p>\n\n\n\n

<\/span>Why This Matters for Your Team?<\/span><\/h2>\n\n\n\n

Irrespective of your industry, experience, or work criticality, your team is already probably using an AI assistant to review code, summarize documentation, or pull together research, with the underlying assumption: if the output came from an “AI assistant,” it’s been vetted by intelligence, not just regurgitation.<\/p>\n\n\n\n

An assumption that is built on quicksand.<\/p>\n\n\n\n

Every AI workflow that touches external data, from RAG pipelines, vector databases, API integrations, to web searches, is now an attack surface, simply because the interface can’t distinguish between “content the model created” and “content the model passed through.”<\/p>\n\n\n\n

<\/span>The Solution:<\/span><\/h2>\n\n\n\n

Today, testing an AI workflow needs to go beyond “is our AI secure?”, which would mean testing what exactly the model reads, how it processes it, what it hands back to the user, and what actions that output can trigger:<\/p>\n\n\n\n