Technical SEO Basics: Crawling, Indexing and Speed Explained
· 3 min read · By PK Creative
Technical SEO is the plumbing. If Google cannot reach or understand a page, no amount of good writing will help it rank. The good news is that most issues are simple to find and fix.
Crawling and indexing in plain words
Google first crawls pages, meaning a robot follows links to fetch them. Then it decides whether to index them, meaning store them so they can appear in results. A page can be crawled but not indexed if Google judges it duplicate, thin or low value.
Sitemap and robots.txt
An XML sitemap lists the pages you want indexed, helping Google find new content. The robots.txt file tells crawlers which areas to avoid, such as admin screens. Keep the sitemap clean: only live pages that return a normal response, using the same address style (https and one consistent host) as your canonical tags.
Redirects and canonical tags
A redirect sends visitors from an old address to a new one. If your sitemap lists addresses that redirect, Search Console reports "page with redirect" and those URLs are not indexed. A canonical tag tells Google which version of a page is the main one when several addresses show the same content.
- Pick one host (with or without www) and stick to it
- Always use https
- Avoid redirect chains
- Point canonicals at the final address
Mobile and speed
Google uses the mobile version of your site for ranking. Make sure content is identical on mobile and desktop and that pages load quickly. See our guide to Core Web Vitals.
A monthly technical check
Once a month open Search Console and review four things: indexing status, crawl errors, mobile usability and Core Web Vitals. Then crawl your own site with a free crawler to catch broken links, missing titles and redirect chains.
Keep a log of changes you make to the site. When something shifts in the data, you will be able to connect it to an update.
Crawl budget and why small sites should care a little
Google decides how many pages to fetch from your site and how often. For a new or small site, this allowance is modest. Wasting it on duplicate pages, redirect loops, parameters and low-value pages slows the discovery of the pages that matter.
Keep your structure tidy: link to important pages prominently, avoid creating many thin pages and make sure your sitemap lists only canonical, working addresses. A clean site is easier for Google to understand and quicker to index.
Use Search Console as your dashboard
Google Search Console is free and shows indexing status, errors, queries and speed data. Check the Pages report monthly. "Discovered - currently not indexed" usually means Google knows the page exists but has not prioritised it; strong internal links and useful content help. Need help? Our SEO team can audit it.
Frequently asked questions
- Does my site need an XML sitemap?
- Yes, it helps search engines discover pages, especially on newer sites.
- What is a canonical tag?
- It tells search engines which address is the preferred version when several URLs show similar content.
- Should the admin area be indexed?
- No. Block it in robots.txt and add noindex so it never appears in search.
- technical SEO
- crawling and indexing
- robots.txt sitemap
- canonical tag explained