Sitemaps for a catalogue of hundreds of thousands of pages, robots.txt versus noindex, canonicals and crawl budget on a server-rendered shop, and why crawlers must get the same page as users.