A couple of days ago I wrote a post: if AI doesn’t recommend your company, the reason may lie at three levels — it doesn’t know you, it hasn’t retrieved you, or it retrieved you but didn’t choose you.
Today I’ll cover the concrete way to check the second level — and it is the only level where you can obtain hard evidence: open your server logs and see whether AI crawlers have actually visited.
The value of this is that every earlier judgment about “how AI sees you” is an indirect inference, while server logs alone are direct evidence — who came, how many times, which pages they fetched, and whether you yourself shut them out at the door; all of it is recorded in black and white.
Yesterday we ran this check on our own site and found two problems. One of them was a configuration error affecting every search engine and AI crawler, and it had been there for some time with nobody noticing. Below I’ll explain the checking method together with these two pitfalls.
Step 1: Count how much each crawler has fetched
Ask your IT operations team or website vendor for a copy of the site access log (usually on the server, with a filename like yourdomain.log), then simply count the entries by crawler identifier.
Common crawler identifiers:
- Search engines:
Baiduspider(Baidu),Googlebot,bingbot,Sogou(Sogou),YisouSpider(Shenma),360Spider - AI / large-model related:
Bytespider(ByteDance),GPTBotandOAI-SearchBot(OpenAI),ClaudeBot,PetalBot(Huawei)
Focus on two things: which ones are completely at zero, and the order-of-magnitude gap between them. If one of them stays at zero over the long term, it means it isn’t indexing you at all — so on that platform, whatever content optimization you do is wasted effort.
One easy mistake to watch out for: before counting, first confirm the time span the log covers. At first we saw only a few dozen entries from one crawler and assumed the volume was very low; later we found that the log covered just three hours — scaled up, that is actually several hundred visits a day, entirely normal. Before drawing conclusions from absolute numbers, divide by time.
Step 2: See what status codes the crawlers received
This step checks “whether you are blocking yourself”. Group each crawler’s requests by status code:
| Status code | Meaning | Does it matter? |
|---|---|---|
| 200 / 304 | Fetched normally | Normal; should account for the vast majority |
| 301 / 302 | Redirect | A small number is normal; if there are many, check whether your link structure is faulty |
| 404 | Page does not exist | A small number is normal; a large number means it is looking for you at the wrong addresses |
| 403 / 444 / 429 | Rejected, rate-limited | Red flag: you are blocking the crawler; check your protection rules immediately |
| 5xx | Server error | Your site was broken when the crawler arrived, which affects how it evaluates you |
If, like many companies, you have a firewall, CDN, or security plugin installed, this step is essential — because such protection can easily mistake high-frequency crawlers for scanners. We once nearly blocked a search engine crawler; we only caught it because we ran a round of “log only, don’t block” before enabling the blocking rules.
Step 3: Check whether it is fetching “the pages it should fetch”
This step is the easiest to skip, but it is very useful: list the URLs a given crawler has fetched and see where it spends its effort.
When we checked our own site, we found that one AI crawler mostly fetched images, old articles, and a few marginal pages, while the “Company Profile” page we had recently optimized specifically was not touched even once.
This tells you one thing: a crawler being able to get in does not mean it has looked at what you want it to see. If you have just finished optimizing an important page but find that the crawler has never fetched it, then that optimization has not yet taken effect on the AI side — you need to guide it there through your sitemap and proactive submission.
The first pitfall we found: every sitemap in the robots file was broken
Following the previous step to investigate “why the key pages weren’t fetched”, we looked at robots.txt — the first file every crawler reads when it enters a site, which tells it where the sitemap is.
It turned out to list 6 sitemap URLs, all of them invalid: every one of them redirects, and what comes back after the redirect is the site’s homepage HTML, not a sitemap at all.
Even worse: the one sitemap that actually works on the site was not listed at all.
This means: every visiting crawler — search engine or AI — reads the robots file, gets 6 invalid addresses, and ends up with nothing useful. It can only crawl links level by level to discover pages, which is far less efficient, and it easily misses new content and pages that no internal links point to.
This problem had existed for some time. No alerts, website access completely normal, backend all normal — you only find it if you go and check.
The self-check is very simple and takes one minute: open yourdomain/robots.txt in a browser and click open each address after every Sitemap: entry, one by one. The correct result should be a page of XML code; if a web page opens instead, that entry is broken.
The second pitfall: crawlers are looking for you at addresses that no longer exist
We also found that a search engine crawler was generating a large number of 404s. After categorizing these addresses, a pattern emerged — they were a URL structure completely unrelated to the current website (some old category and product page format).
This usually comes from a few sources: the website was redesigned or changed platforms in its early years, the domain was previously used by someone else, or a link pointing to an old address still remains somewhere.
Impact: A crawler’s daily crawl quota is limited. If it spends that quota on a pile of nonexistent URLs, less is left for your real content. What to do: If the volume is large, submit the dead links in the webmaster platform, or handle these URLs in a standard way, so they don’t keep spinning idle.
The last step: actively go “invite” the crawlers instead of just waiting
After the audit, if you find that a particular AI or search engine crawls your site infrequently, then in addition to fixing the issues above, you can also submit proactively.
- All major search engines have webmaster platforms, where you can submit sitemaps, push new URLs, and view crawl frequency and index volume.
- If you’re pursuing AI visibility, don’t overlook the search platforms run by the major model vendors themselves. Many AI assistants use their own ecosystem’s search capabilities when they go online — submitting and getting indexed within that ecosystem is more direct than pushing hard elsewhere.
- Most of these platforms support QR-code login, and site verification usually means placing a file or adding a snippet of code. Ask your technical team to help once, and you can then use it long term.
On sequencing, we recommend: fix robots and the sitemap first, then submit. Otherwise, even after you’ve invited the crawlers in, they still won’t be able to get that “directory”.
A ten-minute self-check list
| What to check | How to check | What normal looks like |
|---|---|---|
| Does the sitemap list the right URLs? | Open domain/robots.txt and click each Sitemap URL one by one |
It opens as XML code, not a web page |
| Which crawlers have visited? | Have your IT operations team count log entries by crawler identifier | Major search engine and AI crawlers all have records; none should stay at zero for long |
| Are you blocking yourself? | Count status codes grouped by crawler | Mostly 200; if 403/429 appears, check your protection rules immediately |
| Are the crawled pages the key ones? | List the URLs crawled by a given crawler | Key pages such as the company profile and core services should be among them |
| Are there many 404s? | Measure the share of 404s among crawler requests and the URL pattern | A low share; large volumes of 404s that form a pattern need to be addressed |
A few common questions
Q: I don’t have server access. How can I check?
A: Ask the website builder or your IT provider for a recent access log — this is a very routine request. If they can’t provide one, you can fall back on each webmaster platform’s crawl statistics for a rough picture; it just isn’t as complete as the raw logs.
Q: Is heavy crawling a good thing? Will it slow down my site?
A: Within normal ranges it’s a good thing — it means the crawler is actively indexing you. If it does create pressure, the right approach is to adjust the crawl rate in the webmaster platform, rather than blocking it with a firewall — blocking it cuts off your own indexing.
Q: The URLs in the robots file are wrong. Can I fix that myself?
A: It’s just a text file in your website’s root directory. Have your technical team update it; it takes a few minutes. After the change, remember to open it again to confirm — many problems come from “changed it but never verified it”.
Q: How often should I check?
A: Once a month is recommended. In addition, you must check after changing themes/plugins, migrating servers, or modifying protection rules. These are the actions most likely to change crawler access results without your knowledge.
Want to know whether AI and search engine crawlers are actually reaching your website properly?
Siwenlide provides corporate website and SEO/GEO content-led lead generation operations for SMEs in Beijing and surrounding areas: we reconcile each crawler’s crawl volume, status codes and crawl paths from server logs, fix issues such as robots, sitemaps and false blocks by protection rules — problems that “never raise an alarm but keep leaking” — then submit to and monitor the major webmaster platforms, and issue a comparison report every month. The two pitfalls in this article are exactly what we found on our own site following this process. Visit siwenlide.com (or take a look at our Company Profile) or call 400-686-2011 to talk.
