How Google Search Works: Crawling – Indexing – Serving

Google is your ultimate guru whom you run to when you have a question. It is basically your answer machine.

As explained by Google itself, in order to answer your question, it has to discover, understand & organize the web content. According to Moz, discovery is most probably the most important piece of SEO. If your site cannot be found by Google you will not exist in the Search Engine Results Page (SERP).

How does Google work?

Note: Matt Cutts answered this question in an 8mins long video. For the lazy ones out there who dont like reading, you can see his video-answer at the end of this article.

The short version:

Google follows 3 basic steps:

1. Crawling: This is the discovery phase where Google will search the whole internet for content.
2. Indexing: This is the understanding phase. In this phase, Google will try to understand what this page is about – then it will store it & organize it.
3. Serving: (also called Ranking) This happens when a user searches for a specific query. Google will rank all of the relevant answers it has in its index and serve him the best matching ones to his query.

Before 2003, the above 3 steps used to take around 1 month to finish. This was called the Google Dance. This meant that search results were always out of date. Can you imagine searching for « COVID-19 updates » and finding results more 1 month old? This sucks big time (& not in a good way).

Update Fritz to the rescue. Back in July 2003 (yes its Update Fritz birthday this month), Google released Update Fritz and moved to continuously processing its index of content rather than batch processing it. Phew.

Well, thats it. Thats the short version at least of it. If you want to know more, continue reading the long version.

What is crawling?

As mentioned above, crawling is the discovery process. Google releases its team of robots (known as crawlers or spiders) and they start foraging the web for content.

Some pages already exist in Google database from previous crawls, Googlebot keeps crawling those pages periodically to register new updates and discovers new pages by following links added to those known pages.
Google can also discover new pages when a website owner submits his sitemap to Search Console.

So, how does Google find a page?

1. Following links from existing sites or pages.
2. Reading sitemaps submitted in Search Console.
3. Responding to a manual request from Search Console.

How to improve your website crawling?

1. Submit your website sitemap to Search Console
2. Submit crawl requests for individual pages
3. Use a simple, readable and meaningful URL for your pages
4. Provide internal links in your website to your different pages
5. Check your pages coverage in Search Console
6. Make sure your website pages are being linked from other known websites
7. Make sure you are not denying Googlebots from crawling your pages in your robots.txt file

Note: There is no paid plan to get higher ranking or better crawling for your website. Google does not accept any kind of payment for this purpose. The only way to improve your SEO is by contacting us 🙂

What is indexing?

After crawling a page, Google tries to understand what it is about. This is called the understanding phase.

Google analyzes the content, catalogs images & videos and stores all of these data in its index called Caffeine.
Google pays special attention to the words in special tags or attributes, such as the <h1> & <h2> tags. This process is called indexing.

How to improve your website indexing?

1. Create short, meaningful page titles & descriptions
2. Use page headings that convey the subject of the page
3. Write useful content for your users, use text and not images whenever you can.
4. Deny Googlebot from finding pages that duplicate to another page on the web.
5. Use structured data or schema markup to help Google organize & display your data
6. Follow the Google Webmaster Guidelines.

Tip: Is your website indexed by Google?

The easiest way to find out if your website is indexed by Google is to search for the following query « site:yourwebsite.com » on google.com.

Example: site:wearemaze.com

What is serving? (or ranking)

Imagine a user searches for « buy baby products in Saudi Arabia » – What would Google do?

The giant search engine will search its huge index & tries to find the most relevant answer for this user from its index database. Results are then ranked by most relevant to least relevant.

In order to determine which website/page appears first in its SERP (Search Engine Result Page), Google will take into consideration more than 200 factors including things like user experience, highest quality & most appropriate answer, user’s location, user’s langage & device.
If you are the only baby products online shop in your hometown & your website meets the above conditions, chances are you will rank on top of the first page.

How to improve your website ranking?

1. Your website or page Speed Speed & Speed.
2. Make sure your page is mobile-friendly & loads fast.
3. Make sure your content is useful to your users & up to date.
4. Make sure your site has a good user experience.
5. Make sure to tell Google whom your website content serves (Location & Language).
6. Use structured data or schema markup to help Google organize & display your data.
7. Implement AMP (when applicable) for faster loading pages on mobile devices.
8. Work on creating good, fresh content that users want.

What should you do if your website is not on Google?

In most cases, Google will take between 2 days and 4 weeks to crawl & index your website pages. However, this really depends on many factors, how fast your site is, how useful Google find your pages content, how popular your site is and many other variables explained above.

While Googlebot is usually fully automated and most probably it will find your site without any manual work from your part, it is generally a very good idea to add your site on Search Console to make the process faster & easier.

Bonus Section: What is RankBrain?

Google’s core algorithm is known to be one of the best in the world. RankBrain is the machine learning part of this algorithm.

RankBrain keeps on learning & improving its predictions & because of that search results will constantly improve.
For example, if RankBrain sees a lower ranking website that provides a better result/experience for the user, it will adjust the SERP to reflect this. Basically RankBrain will raise that website position and lower others.

Conclusion

No one knows the ins and outs of Google’s algorithm and I surely don’t pretend to. However, we do know how Google find, organize, saves & ranks content.

If you want me to help you rank better go ahead and ask for a free consultancy.

Matt Cutts video answer