Conrad DevOps

Train

SEO and crawlers

Sitemaps for a catalogue of hundreds of thousands of pages, robots.txt versus noindex, canonicals and crawl budget on a server-rendered shop, and why crawlers must get the same page as users.
Google — build and submit a sitemapGoogle — robots.txt introductionGoogle — cloaking

From the "SEO for a server-rendered shop" part of your Preparation tasks note.

Know

  • Sitemaps: at most 50,000 URLs or 50 MB per file, so a big catalogue gets many sitemaps plus a sitemap index; generated from the catalogue by a server route, cached for an hour, listed in robots.txt.
  • robots.txt: stops crawling of /api, /cart, /checkout, search and filter combinations; the list can live in runtime config. It does not remove anything from the index.
  • noindex: meta tag or X-Robots-Tag header; the page must stay crawlable so Google can read it. nofollow on links you do not want followed. Canonical URLs on every variant, hreflang for languages, structured data for products.
  • Crawl budget: follows response speed and the number of useless URLs; fast cached pages and a tidy robots.txt matter more than tricks.
  • Bots and rendering: detecting a crawler to skip personalisation, consent banners or tracking scripts is fine; giving it different content or a page without links is cloaking.

Fact check of your notes

Gotcha· A page without links for bots is cloaking

The note's plan is to show crawlers "a page without links so it cannot follow to other pages". Google's spam policy calls showing crawlers different content than users cloaking, and it is penalised. It would also remove the internal links Google uses to discover and rank deep product pages. To control what is indexed, use robots.txt, noindex and nofollow, and keep the HTML the same for everyone. Skipping tracking scripts and personalisation for bots is fine.

Gotcha· robots.txt does not de-index

A disallowed URL can stay in the index when other pages link to it, and Google cannot see a noindex on a page it is not allowed to crawl. First noindex, then disallow once it is gone.

Exercise

Exercise
  • Write a Nitro route that serves a sitemap index and a sitemap per category from a fake catalogue of 200,000 products, cached for an hour; check the file limits.
  • Decide, for five URL types of a shop (product, category, filtered listing, cart, search), whether each should be crawled, indexed, canonicalised or blocked, and say how.

Be able to say

Be able to say· How do you keep a huge server-rendered catalogue well indexed?

"Server rendering so the content is in the HTML, a sitemap index with one sitemap per category under the 50,000 URL limit, generated from the catalogue and cached for an hour, and a robots.txt that keeps crawlers away from the cart, checkout, APIs and filter combinations. Canonical URLs on every variant and noindex on thin pages, and fast cached responses, because crawl budget follows speed. What I would not do is show crawlers a different page; that is cloaking, and it also removes the links Google needs to find the deep pages."

Be able to say· What is the difference between blocking a page in robots.txt and noindex?

"robots.txt stops the crawl, noindex stops the indexing. A page blocked in robots.txt can still show up in results if other pages link to it, and Google can never see a noindex on it. So to remove a page I put noindex on it and keep it crawlable, and only block it once it is gone."

Your notes

Open in Notes

Nothing yet. Notes filed here show up on this page and the section overview.

Your Develit architecture

How to present the platform you worked on at Develit — isolated Cloudflare Workers services behind an orchestrating gateway, service bindings and Workers RPC, KV for exchange rates, Durable Objects for idempotency, Workflows and Queues, micro-frontends on shared Nuxt layers — and how it contrasts with Conrad's stack.

Conrad monitoring

Preparation for the monitoring part of the Conrad interview, written from your Monitoring note — metrics, logs and traces with the Grafana stack, front-end monitoring and Core Web Vitals, and load testing with k6.