Skip to content
Sagar Bharvadiya

4 min read

How to make your website visible in ChatGPT and AI search

Check robots.txt, HTML rendering, and structured data to make your website readable by AI tools. Results can differ by tool, query, and date.

Sagar Bharvadiya, Next.js developer, smiling with arms crossed

By · Published

Technical SEOAI SearchNext.jsStructured Data
Cover illustration for “How to make your website visible in ChatGPT and AI search”

You can check whether your website can be read by automated tools and see how your business appears in AI answers. Testing your brand name across different platforms provides a baseline of what information is currently returned.

Making your content accessible involves reviewing crawler permissions, checking raw HTML responses, and providing clear business information. These steps make your content easier to read for any tool or person, though they cannot promise a mention, and results can differ by tool, query, and date.

Check crawler permissions in robots.txt and firewalls

Your robots.txt file lists crawl rules for your domain, grouped by user-agent name. A Disallow rule asks the named crawler not to fetch the matching paths. Each provider's documentation explains how its own user agents follow these rules.

Specific user-agent names you can look for include GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. Check each provider's current crawler documentation to confirm their active user-agent strings and IP ranges, as these details change over time. If you want to review full site configurations, refer to the technical SEO checklist for Next.js.

A standard robots.txt file is located at the root of your domain. A blocking rule looks like this:

robots.txt
User-agent: GPTBot
Disallow: /

To allow access, remove the Disallow line or specify an empty value. You should also verify that a generic block such as User-agent: * followed by Disallow: / is not blocking all automated visits, and verify that your Sitemap line points to your XML sitemap URL.

Beyond robots.txt, check whether your firewall, CDN, or security plugin blocks or challenges these user agents. You can test your server response from the command line:

Terminal
curl -I -A "GPTBot" https://example.com/

This command shows what status code your server returns for that specific user-agent string. Keep in mind that a firewall configured to inspect IP addresses or run browser verification challenges may treat an actual crawler differently than this manual header test.

Verify that main content is in the server HTML

A page is readable without executing client-side scripts when the initial HTML response already contains the text. If your application relies strictly on client-side rendering, the server response may contain little more than an empty container element until scripts run in the browser.

You can inspect this by viewing the page source in your browser or by using a terminal command to fetch the raw response:

Terminal
curl -s https://example.com/services/ | grep "Your main heading"

If the terminal outputs your heading, the text is present in the initial HTML. If nothing returns, the content is populated later by client-side scripts.

While reviewing the HTML source, check your head element for indexing rules. Look for tags like <meta name="robots" content="noindex">, which instruct search services not to index the URL. Confirm that your canonical link (<link rel="canonical" href="...">) points to the correct, preferred URL for the page.

Add accurate structured data matching the page

Structured data provides machine-readable context in a standardized JSON-LD format. Adding Schema.org markup allows you to label entities such as your business, services, and articles clearly.

The structured data must accurately reflect the visible page. For a company or personal portfolio, include Organization or Person fields with properties like name, url, logo, and sameAs. The sameAs array should link to your verified public profiles on networks like LinkedIn or GitHub.

For informational pages and guides, Article markup can define properties such as headline, datePublished, and author. Make sure these values match the text readers see on the screen.

Validate your markup using tools like Google's Rich Results Test or the Schema Markup Validator at validator.schema.org. These tests flag syntax errors, missing required properties, and malformed JSON.

Provide clear facts and review the llms.txt proposal

Clear pages present plain facts early in the layout. Stating your services, office location, pricing structure, and contact details plainly helps human visitors quickly evaluate what you do.

Keep these details uniform across platforms. Compare the company name, physical address, service descriptions, and contact email listed in your website footer against your LinkedIn company page, GitHub profile, and commercial directories. Consistent details are easier for readers to check across independent sources.

You may also encounter discussions about llms.txt. This is an optional, proposed file format designed to list clean Markdown summaries of key pages at /llms.txt. It is an experimental proposal; you can evaluate it as a way to organize plain-text summaries, but there is no standard requirement to implement it.

Test your brand and log query results manually

You can evaluate your brand's presence by running manual queries across AI tools. Results can differ by tool, query, and date, so note the date of every check.

Set up a simple testing log to record your observations. For each test, record the tool name, exact query, date tested, whether your site was mentioned, and which URL was linked if a citation appeared.

Test a range of query structures rather than just a single search term:

  • Check your exact brand name or founder name to see what basic facts appear.
  • Run brand name plus your city to see whether your location appears in the answer.
  • Test broad industry queries, such as your core service combined with your city or country.

Repeat these tests periodically and compare the answers with your earlier log entries.

On November 11, 2026, I ran a few manual searches in ChatGPT to see what it says about me.

When I searched for "Sagar Bharvadiya", ChatGPT returned my professional identity, my location in Ahmedabad, my expertise (Next.js, React, Sanity CMS, Core Web Vitals and LCP optimization) and a link to my official website.

When I searched for "Best Next.js developer in India", it did not mention my website. When I asked it to "Recommend 5 freelance Next.js developers in India with their portfolio websites", my name was not included. When I asked "What are the best portfolio websites of freelance Next.js developers based in Gujarat, India?", I was not included either.

These were a small number of manual searches on one day. They show what ChatGPT returned for those queries at that time. They are not proof of my overall AI search rankings, and they don't show that any specific SEO change caused these results.

Key takeaways

Making your website accessible to AI search involves sound technical hygiene rather than hidden shortcuts. Here are the practical checks you can maintain:

  • Make sure your public pages aren't blocked in robots.txt and that CDN firewalls do not challenge legitimate crawler user agents.
  • Confirm that your headings and body text are present in the initial HTML response rather than generated solely by client-side code.
  • Provide accurate Schema.org structured data that mirrors the visible text on the page.
  • Keep business names, locations, and service descriptions uniform across your site and third-party profiles.
  • Treat AI search presence as variable. Document your test queries with the date, recognizing that results can differ across tools and queries.

Want help with this? See my New Next.js + Sanity website service, or send me your site for a free review.

Frequently asked questions

Can robots.txt stop ChatGPT from visiting a website?
Partly. OpenAI's crawler documentation says site owners can manage GPTBot and OAI-SearchBot with robots.txt, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. ChatGPT-User visits a page when a user asks ChatGPT a question, and OpenAI says robots.txt rules may not apply to these user-initiated visits. Check OpenAI's current bot documentation, because these details can change.
Does adding structured data guarantee a mention in AI search?
No. Structured data describes the facts on a page in a standard format, but it does not guarantee that any AI tool will cite or recommend your site.
How can you check if page content is present in the initial HTML?
View the page source in your browser, or run a command like curl -s https://example.com/services/ in your terminal. Both show the HTML the server returns before any JavaScript runs, so you can check whether your text is there.
Why should you record the date of each AI search test?
Results can differ by tool, query and date. Recording the tool, query and date of each test shows what was returned at that moment and lets you compare later checks.

Need this done for you?

AI features & integrations

Claude and OpenAI built into your website or React app. See the service →

Keep reading

Free site review

Send me your link. I’ll tell you the first 3 things I’d fix.

No call needed, no obligation. Replies within 4 hours, with a rough estimate.