# SEO

XML Sitemaps and robots.txt: Build, Audit and Troubleshoot Them

XML Sitemaps and robots.txt: Build, Audit and Troubleshoot Them

An XML Sitemap is a discovery and inventory hint: it tells search engines which preferred URLs and files you consider important. robots.txt is an access protocol: it tells compliant crawlers which URL paths they may request. Neither guarantees indexing or ranking, and neither protects private information.

Choose the control by the outcome

Discovery: Sitemap plus crawlable links. Crawl access: robots.txt. Index exclusion: crawlable noindex or a real removal response. Duplicate consolidation: Canonical or redirect. Confidentiality: authentication and authorization.

The five controls are not interchangeable

Required outcomePrimary controlEvidence of successDo not substitute
Help search engines discover a preferred URLCrawlable internal links and a clean SitemapURL is linked, listed and fetchableRepeated URL Inspection requests
Prevent a compliant bot from requesting a pathrobots.txt rule for that crawler and hostRule matches the tested URLnoindex, which still permits crawling
Keep an accessible page out of Google Searchrobots meta or X-Robots-Tag noindexGoogle can fetch and read the directiverobots.txt Disallow
Permanently remove a retired public URL404/410, or 301/308 to a true replacementFinal HTTP response matches the decisionDeleting it only from the Sitemap
Consolidate equivalent duplicate URLsConsistent Canonical signals or permanent redirectsPreferred URL is used across links and Sitemaprobots.txt blocking duplicates
Protect private or confidential contentAuthentication and server-side authorizationUnauthorized requests cannot retrieve itrobots.txt, noindex or an unlinked URL

Does the website need a Sitemap?

Google says a small site of roughly 500 pages or fewer may not need a Sitemap when every important page is reachable through normal links. That does not make Sitemaps useless for smaller businesses: they remain valuable during launches, migrations, multilingual expansion and inventory monitoring.

A Sitemap complements site architecture; it does not repair orphan pages. Use the site architecture guide and internal-linking guide to create real discovery paths.

SituationSitemap valueMore important companion
New domain with few external linksHigh: exposes the preferred launch inventoryLinks from the homepage, hubs and relevant external profiles
Small, comprehensively linked service siteUseful but not essential for discoveryClear navigation and direct internal links
Large ecommerce, publishing or marketplace siteHigh: partitions large and changing inventoriesFaceted-navigation control, logs and stable URL rules
Multilingual websiteHigh when Sitemap hreflang is the chosen implementationSelf-Canonical locale URLs and reciprocal alternates
Image, video or news-led siteHigh when specialized files are hard to discoverAccessible media, valid landing pages and feature-specific requirements
Website migrationHigh: presents the final preferred URL setOne-to-one redirects, preserved content and production crawl QA

Choose a Sitemap format that can be maintained

FormatGood fitCapabilities and limits
XML URL setMost sites and CMS platformsSupports URLs, lastmod and Google extensions for localized, image, video and news data
Sitemap indexMultiple XML files or large inventoriesReferences child Sitemaps; it contains Sitemap locations, not page URLs
RSS or AtomPublishers with a reliable feedUseful for recent URLs; does not replace a complete historical inventory
Text SitemapSimple page-only inventoryOne absolute URL per line; no lastmod or extensions
HTML SitemapHuman navigation and accessibilityA normal webpage, not a substitute for protocol-compliant Sitemap submission

Build the Sitemap from the canonical inventory

The safest generator starts with the same source of truth that publishes pages. A URL enters the Sitemap only when its intended state is public, indexable and canonical. Automatically crawl-generated lists can preserve mistakes such as parameter duplicates, redirects or old staging routes.

Google attempts to crawl the exact absolute URLs provided. Keep protocol, hostname, path casing, trailing-slash policy and locale path consistent with the final Canonical.

URL inclusion gate
  • Absolute, fully qualified production HTTPS URL.
  • Preferred Canonical version with the correct hostname, path and locale.
  • Expected to return HTTP 200 and meaningful visible content.
  • Allowed by robots.txt and not protected by login.
  • No robots meta or X-Robots-Tag noindex.
  • Not a duplicate parameter, sort, filter, session or tracking variation.
  • Reachable through at least one useful crawlable internal link.
  • Appropriate for users to land on from search results.
  • Included once in the correct child Sitemap.
  • Removed automatically when its publishing or index decision changes.

Exclude URLs that tell a different index story

URL stateSitemap treatmentCorrect technical treatment
301/308 permanent redirectExclude source; list only the final preferred URLOld URL redirects directly to the closest equivalent
302/307 temporary redirectUsually exclude temporary source from a preferred inventoryConfirm which URL should remain indexed
404/410 retired URLExcludeKeep the real removal response and repair internal links
5xx or persistent timeoutDo not present as healthy inventoryFix the server or application cause
noindex pageExcludeKeep crawlable until Google can process the directive
Canonical duplicateExclude duplicate; include preferred representativeAlign internal links, hreflang and Canonical
Internal search or weak filter pageExcludeUse deliberate index/crawl rules based on value and scale
Staging, preview, admin or account URLExcludeProtect private environments and sessions with access control

Respect protocol limits and file scope

A single Sitemap is limited to 50,000 URLs or 50 MB uncompressed. A Sitemap index can reference up to 50,000 Sitemap files and is subject to the same uncompressed size limit. Compression reduces transfer size, not the uncompressed protocol limit.

Use UTF-8, entity-escape XML values and keep Sitemap files at a stable public URL. Unless submitted through Search Console, a Sitemap normally affects descendants of its parent directory; hosting it at the site root avoids accidental scope restrictions.

Protocol itemRequirementAudit check
URL setMaximum 50,000 URLs and 50 MB uncompressedCount entries and uncompressed bytes before release
Sitemap indexMaximum 50,000 child Sitemaps and 50 MB uncompressedChild locations resolve and contain the intended host inventory
EncodingUTF-8 with valid XML and escaped entitiesParser accepts ampersands, non-Latin paths and namespaces
LocationStable accessible URL within valid scopeRoot placement or verified cross-site submission is intentional
Page URLFully qualified absolute URLNo relative paths, development hosts or mixed protocol
OrderingNo ranking meaningDo not waste effort sorting by supposed importance

Partition child Sitemaps for diagnosis, not decoration

Splitting a small inventory into hundreds of files creates maintenance noise. Partition when a group has a distinct template, owner, update pattern or risk so Search Console filtering reveals an actionable cohort.

SEOWithJack currently uses one root Sitemap index that references post, page and category child Sitemaps. The post Sitemap includes English, Malay and Simplified Chinese article URLs, while the robots.txt file points to the one root index. That pattern is simple enough to audit and specific enough to isolate content types.

Possible child SitemapUseful whenPoor reason to create it
Posts or articlesEditorial URLs share publishing and refresh rulesEvery month needs a permanent file regardless of workflow
Pages or servicesCommercial/static templates need separate monitoringThey merely have lower URL counts
Products or categoriesDifferent templates and Canonical risks need cohortsTo claim greater priority
LocalesTeams or platforms maintain languages independentlyTo avoid implementing correct hreflang relationships
Images, videos or newsSpecialized Google extensions and validation applyThe website happens to contain one ordinary image
Legacy migration cohortTemporary monitoring of moved URLs is documentedRedirecting old URLs are included as preferred pages

Use lastmod as a factual change record

Google says it may use <lastmod> for crawl scheduling when the value is consistently and verifiably accurate. It should reflect the last significant change to the page, not the Sitemap generation time, deployment timestamp or copyright year.

Google ignores <priority> and <changefreq>. The unauthenticated Sitemap Ping endpoint is also deprecated and returns 404; keep a stable Sitemap URL and use Search Console, robots.txt or the Search Console API instead.

ChangeUpdate lastmod?Reason
Rewrite main guidance with new evidenceYesThe page meaningfully changed
Add or correct important structured dataYesGoogle identifies structured data as potentially significant
Change important internal links or navigation contextUsually yes when material to the pageDiscovery and page relationships changed
Fix one typo or compress the same imageUsually noThe page purpose and information did not materially change
Update only footer, copyright year or global CSSNo for every content URLA shared cosmetic build is not a content refresh
Republish unchanged text with today’s dateNoIt creates an inaccurate signal and misleading freshness
Aggregator page changes automaticallyOnly when the system can calculate a truthful meaningful updateOmit lastmod if confidence is low

Align Sitemap, status, Canonical and index rules

SignalPreferred indexable URLRetired URLDuplicate URLPrivate URL
HTTP response200404/410 or relevant 301/308200 or redirect based on product need401/403 or authenticated access
robots.txtAllow crawlingUsually no special rule neededAllow when Google must read Canonical/noindexNot a security control
robots meta/headerIndex allowedNot required for true removalUse noindex only when exclusion is intendedNot a security control
CanonicalSelf-referential preferred URLNone on removed responsePoints to equivalent representative when appropriateNot relied upon
Internal linksUse preferred URL directlyRemove or updatePrefer representative URLOnly inside authorized experience
SitemapIncludeExcludeExclude nonpreferred duplicateExclude

Multilingual Sitemap and hreflang rules

XML is one of three equivalent Google-supported hreflang implementation methods. If you choose it, every localized URL receives its own <url> entry, and every entry lists itself plus every other alternate through identical xhtml:link annotations.

Do not use hreflang to repair translated navigation wrapped around untranslated main content. Each locale URL should be a real, indexable localized page with a self-Canonical and supported language or language-region code. Review the multilingual SEO guide for the complete decision model.

Locale-cluster validation
  • Every indexable language URL has its own Sitemap URL entry.
  • Each entry lists itself and all other alternates.
  • Alternate sets are reciprocal and identical across versions.
  • All href values are absolute production URLs.
  • Each alternate resolves to HTTP 200 and a self-Canonical in that locale.
  • Language codes use supported ISO language values and optional region values.
  • x-default is used only for the deliberate fallback destination.
  • Redirected, noindex, blocked or untranslated variants are not declared as valid alternates.
  • The same mapping is not contradicted by HTML or HTTP-header hreflang.

Use specialized Sitemap extensions only when they solve discovery

ExtensionUseful forRelease gate
ImageImportant images Google may not otherwise discover, including some JavaScript-reached assetsImage URL is crawlable; use current supported image tags
VideoPages where video is the main content and media details help discoveryLanding page, thumbnail, player/content URLs and required fields are accessible
NewsEligible news publishers with current article inventoryFollow the current Google News Sitemap requirements and freshness window
xhtml hreflangLocalized variants when Sitemap is the selected annotation methodEvery version lists the complete reciprocal locale set
Combined extensionsA page legitimately qualifies for more than one typeDeclare every namespace once and validate the combined XML
NoneOrdinary pages already discoverable in HTMLDo not add unsupported metadata for appearance alone

Submit the location, not the file contents

Search Console submission tells Google where the hosted Sitemap lives; it does not upload the XML to Google. The file must remain public and fetchable. A successful submission means Google could process the file—not that every listed URL was crawled, indexed or ranked.

Discovery or submission methodUse caseImportant limitation
Search Console Sitemaps reportManual submission plus fetch and parsing feedbackRequires owner permission and shows submitted files for the property
robots.txt Sitemap linePersistent public declaration discoverable by crawlersUse a fully qualified absolute URL; it does not guarantee processing
Search Console APIProgrammatic management for many verified propertiesAutomation still needs validation and ownership
RSS/Atom with WebSubBroadcast recent publishing changesFeed is recent inventory, not necessarily the full Sitemap
Deprecated Ping endpointDo not useGoogle returns 404 and gains no useful signal

Read Search Console reports at the correct level

Use the Sitemaps report to confirm fetch history and parsing errors. Use the Page indexing report filtered by “All submitted pages” or a specific Sitemap to understand the index state of that cohort. Use URL Inspection for a representative URL—not to extrapolate a whole site from one page.

A gap between submitted and indexed URLs is not automatically an error. Redirects, duplicates, noindex pages and removals should not have entered a clean Sitemap; valid pages may still require investigation of quality, Canonical selection or processing.

QuestionBest evidenceDo not conclude
Could Google fetch and parse the file?Sitemaps report status, last read and errorsSuccess means every URL is indexed
Which submitted cohort is not indexed?Page indexing report filtered by SitemapEvery non-indexed URL is a defect
What does Google know about one URL?URL Inspection indexed data and live testA live test means the URL entered the index
Is generation output technically clean?Independent XML parse plus URL inventory crawlValid XML means URL decisions are correct
Are changes improving discovery?Comparable crawl/index cohorts and server logsA submission timestamp caused ranking gains

robots.txt scope starts at the exact host root

The file must be available as lowercase /robots.txt at the top level. Its rules apply only to the same protocol, hostname and port. A file on https://example.com does not control http://example.com, https://www.example.com, another subdomain or a nonstandard port.

Keep each production hostname intentional. A translated subfolder shares its host’s robots file; a translated subdomain needs its own. A CDN or application should return plain UTF-8 text—not a login page, HTML error template or redirect chain.

robots.txt locationControlsDoes not control
https://example.com/robots.txtHTTPS URLs on example.com at the standard portHTTP, www, subdomains or custom ports
https://www.example.com/robots.txtOnly the www HTTPS hostApex example.com or shop.example.com
https://shop.example.com/robots.txtOnly the shop HTTPS subdomainMain website or other subdomains
https://example.com/folder/robots.txtNothing as a robots root policyURLs under /folder/
https://example.com:8443/robots.txtOnly that host, protocol and portStandard HTTPS port

Understand groups, matching and supported fields

ElementGoogle behaviorAudit risk
User-agentBegins a group; Google selects the most specific matching token and combines duplicate matching groupsA generic group is assumed to override a more specific bot group
DisallowBlocks crawl requests whose path matches the ruleA broad prefix unintentionally covers useful URLs
AllowPermits an exception inside a broader DisallowThe exception is shorter than the competing blocking rule
Longest matchThe most specific matching path wins; Allow wins a same-length tieReviewers read top-to-bottom as if first rule always wins
Path casingRules are case-sensitiveThe live URL uses different capitalization
WildcardsGoogle supports * and end-anchor $ in path rulesRegex assumptions produce wider or narrower matches
SitemapAbsolute location; not tied to a user-agent groupRelative URL or wrong protocol/hostname
Unsupported linesGoogle ignores themcrawl-delay or plugin-specific directives are assumed universal

Start with the smallest robots.txt that expresses policy

If everything public may be crawled, an absent file or an empty applicable Disallow already means crawling is allowed. Allow: / is usually unnecessary. Add rules only for URL patterns whose crawl access is deliberately restricted, then test representative allowed and blocked URLs.

A simple service site might use one User-agent: * group, block internal search-result paths and list one absolute Sitemap index. It should not copy old WordPress admin rules after those routes no longer exist, nor block CSS, JavaScript or images required to render public pages.

Robots rule review
  • Every Disallow has a documented crawl-management purpose.
  • Rules match the exact production case, slash and parameter pattern.
  • A more specific Allow exception works under the longest-match rule.
  • No public page, CSS, JavaScript, image or API required for rendering is blocked accidentally.
  • No rule is being used as a security, Canonical or noindex substitute.
  • Specific crawler groups are based on that crawler’s published documentation.
  • The Sitemap line uses the final absolute production URL.
  • Staging rules cannot be copied silently into production.

Know how robots.txt response failures change crawling

robots.txt is operational infrastructure. Google generally caches it for up to 24 hours, sometimes longer when refresh fails. A change may therefore take time to propagate, and an outage can affect much more than one page.

robots.txt responseGoogle-documented treatmentOperational response
2xxProcesses the valid rules receivedConfirm content type, encoding and intended rules
3xxFollows at least five redirects; beyond that treats it like 404Serve directly at the canonical host root when possible
4xx except 429Acts as if no robots.txt exists, so no crawl restrictions applyDo not use 401/403 to throttle crawling
429Handled separately from ordinary 4xxInvestigate capacity and rate-limiting configuration
5xxInitially stops crawling for 12 hours; may use the last good version while retryingTreat as an urgent host reliability issue
DNS/network failureTreated as a server errorFix availability, TLS, DNS or edge behavior
File larger than 500 KiBContent after the limit is ignoredConsolidate rules and restructure URL patterns
Cached old versionMay persist for about 24 hours or longer during failuresAllow propagation time and verify server delivery

robots.txt cannot remove or secure content

NeedCorrect methodWhy robots.txt fails
Remove an accessible HTML page from Googlecrawlable meta robots noindexA blocked crawler cannot read the page-level rule
Remove an accessible PDF or other non-HTML filecrawlable X-Robots-Tag noindex response headerHTML meta is unavailable and a block hides the header
Urgently hide a result while permanent handling is preparedSearch Console Removals plus durable noindex/removal/authenticationA robots change alone is neither immediate nor durable index removal
Protect customer, staging or confidential dataAuthentication, authorization and network controlsrobots.txt is public, advisory and exposes named paths
Retire content with no replacement404 or 410 and removal from links/SitemapDisallow leaves the URL and its external references unresolved
Move content permanently301/308 to the closest equivalentDisallow prevents normal processing of the move
Consolidate duplicate URLsCanonical signals or permanent redirectsBlocking prevents Google from reading Canonical content signals

Generate differently for WordPress, static sites and applications

PlatformRecommended source of truthRelease risk
WordPress corePublished public content and the native Sitemap system when suitableSEO plugin, cache or multilingual plugin creates a second conflicting index
WordPress with SEO pluginOne deliberately selected Sitemap provider using final Canonical settingsSubmitting both native and plugin inventories without comparing them
Static site generatorBuild manifest of public indexable routesEvery build changes lastmod or includes generated redirects and utility pages
Headless CMSPublished records joined with route, locale and Canonical statusUnpublished or orphaned records leak into generated XML
Ecommerce/applicationProduct/category availability and index-decision serviceFacets, sessions, sort parameters and soft-deleted records expand inventory
Multiple hostsPer-host inventory with verified cross-submission only when intentionalOne robots file is assumed to control every subdomain

Protect a WordPress-to-static migration

Migration sequence
  1. Export the old Sitemap inventory, public crawl, Canonicals, hreflang and index directives.
  2. Map every old URL to an unchanged route, a true equivalent redirect, or a justified 404/410.
  3. Keep staging behind authentication and exclude its hostname from production Sitemaps.
  4. Generate new Sitemaps from the final published route manifest, not from a blind crawl.
  5. Compare old and new preferred inventories by content type and locale.
  6. Remove redirects, noindex pages, errors, search results and non-Canonical variants.
  7. Validate XML, child indexes, URL limits, response codes and production hostname.
  8. Publish redirects, pages, Sitemap and robots.txt in the same release.
  9. Confirm production robots.txt did not inherit a site-wide staging Disallow.
  10. Crawl every old URL and every new Sitemap URL after launch.
  11. Submit the stable Sitemap index in the production Search Console property.
  12. Monitor submitted cohorts, selected Canonicals, server errors and organic landing pages.

Diagnose symptoms in the right order

SymptomLikely layerFirst evidence to collect
Sitemap “Could not fetch”Access, DNS, redirects, content type or property scopeDirect HTTP response and Sitemaps report detail
XML parsing errorEncoding, escaping, namespace or malformed markupRaw response and XML validator location
Submitted URL blocked by robots.txtInventory and crawl policy conflictExact Sitemap entry plus matching robots rule
Submitted URL marked noindexPublishing/index-decision conflictRendered/head response and generator eligibility logic
Submitted URL is a redirectOld or wrong Canonical inventoryRedirect destination and route manifest
Google selected a different CanonicalDuplicate or inconsistent signalsPage pair, internal links, Canonicals, hreflang and Sitemap
Important URL absent from SitemapGenerator, publishing state or partition errorSource record and child-Sitemap assignment
Large unexplained unsubmitted URL setFacets, parameters, old routes or crawler trapsPage indexing filter, crawl export and server logs
robots.txt changed but testing looks oldCaching or edge deliveryResponse headers, host variants and elapsed cache time
Traffic changed after submissionNot proof of Sitemap causationQuery, page, release, demand and SERP cohorts

Run a release gate, not a visual spot-check

Production QA
  • robots.txt and every Sitemap return HTTP 200 on the production host.
  • Files are UTF-8 and XML parses with every declared namespace.
  • Sitemap index child locations are absolute, unique and fetchable.
  • Every listed page is 200, indexable, Canonical and linked internally.
  • No redirect, 4xx, 5xx, login, staging or non-Canonical URL is listed.
  • Locale clusters are complete and reciprocal when Sitemap hreflang is used.
  • lastmod is omitted or tied to a meaningful source-controlled change.
  • robots rules pass representative allowed, blocked and exception cases.
  • Required rendering assets remain crawlable.
  • The Sitemap index is declared once with its exact absolute URL.
  • Old WordPress URLs resolve according to the approved redirect map.
  • Search Console can fetch the Sitemap and cohort monitoring has an owner.

Common Sitemap and robots.txt myths

  • “Every URL in a Sitemap will be indexed.” It is a hint, not a guarantee.
  • “A Sitemap replaces internal links.” Orphan pages remain a navigation and architecture problem.
  • “priority 1.0 makes a page rank higher.” Google ignores priority.
  • “changefreq=daily forces daily crawling.” Google ignores changefreq.
  • “A fresh lastmod guarantees recrawling today.” It is useful only when accurate and does not create a deadline.
  • “The old Ping URL speeds up submission.” Google deprecated it and returns 404.
  • “robots.txt removes pages from Search.” A blocked URL may still be indexed without fetched content.
  • “Disallow protects confidential paths.” The file is public and provides no authorization.
  • “Block a noindex page for extra certainty.” Blocking may prevent Google from reading noindex.
  • “One robots.txt controls www, subdomains and HTTP.” Rules are scoped to protocol, hostname and port.
  • “More Disallow rules save crawl budget on every small site.” Unnecessary complexity can create larger failures.
  • “A successful Search Console submission proves SEO is complete.” It proves file processing, not page quality or ranking.

Frequently asked questions

Does every website need an XML Sitemap?

No. Google says a small, comprehensively linked site may be discovered without one. A clean Sitemap is still useful for launches, migrations, multilingual inventories, specialized media and monitoring.

Does a Sitemap guarantee crawling or indexing?

No. Submission is a hint. The URL must still be accessible, indexable, canonical, useful and selected by Google’s systems.

Should redirected or noindex URLs be in the Sitemap?

No. A preferred indexable inventory should exclude redirects, removals, noindex pages and non-Canonical duplicates.

How many URLs can one Sitemap contain?

Up to 50,000 URLs or 50 MB uncompressed, whichever limit is reached first. Larger inventories should use child Sitemaps and a Sitemap index.

Should I use lastmod on every URL?

Only when the system can provide a consistently accurate date of the last significant page change. It is acceptable to omit it when confidence is low.

Do priority and changefreq help Google?

No. Google states that it ignores both fields.

Should noindex URLs also be blocked in robots.txt?

No. Google must be able to crawl a URL to read its meta robots or X-Robots-Tag noindex directive.

Can robots.txt protect a staging website?

No. Use authentication or network access controls. A public Disallow file is advisory and can expose the paths it names.

Where should robots.txt and the Sitemap be hosted?

robots.txt must be at the exact host root. A root-level Sitemap gives simple site-wide scope; its absolute location can then be declared in robots.txt and Search Console.

How should I monitor Sitemap performance?

Check fetch and parsing in the Sitemaps report, filter the Page indexing report by submitted Sitemap, inspect representative URLs and compare index outcomes by meaningful cohorts.

Official references

Need a practical next step?Make discovery, access and index signals agree.

Share the live Sitemap index, robots.txt, Search Console states and affected URL patterns. Jack can compare the generated inventory with Canonicals, responses, locale routes and migration redirects.

Discuss technical SEO on WhatsApp

Jack Lee

Jack Lee

Building Search Visibility with SEO, GEO & AI-Assisted Websites through practical projects and experiments.