June 27, 2018
What Is Crawling & How The Search Engine Bots Work Whole Together?
June 27, 2018
What Is Crawling & How The Search Engine Bots Work Whole Together?
Manchun Pandit loves pursuing excellence through writing and have a passion for technology. he has successfully managed and run personal technology magazines and websites. he currently writes for JanBaskTraining.com, a global training company that provides e-learning and professional Software QA Testing certification training.
Newbie 6 posts
Followers: 4 people

We all know what Search engines are, but very few of us know about how the search engines Bots work. It is important to know about search engine functionalities. So, this blog intends to give you all the details about search engine bots and for this, we are going to discuss following points;

  1. What is crawling? Why is it a prime function of the search engine?
  2. How does the Search Engine Bots work efficiently?

So, let’s address the very first point, i.e.

What is crawling? Why is it the prime function of a search engine?

Crawling is where the acquisitions of any type of data about a website begin. The advertisements are present, it includes the scanning of all websites, gathering all the details about each and every page like relevant images, wherever the links are being crammed, page layouts, etc.

Let’s know how a website is crawled;

When an automated bot, that is also known as ‘spider’ visits pages back to back as it keeps on responding, with help of links that are used to identify about where we should go next. The search engine giant Google’s spiders have the capability to read n-number of thousands of pages every second.

Therefore, whenever the web crawler visits any of the pages, it gathers all the links present on that page and then keeps on adding all of them to the list that it has to visit immediately after this visit. Then move on to next pages added to the list, gathers all the links again and repeat the process. Web crawlers tend to revisit last pages that it already visited to see if everything is fine or anything is changed over there.

What we mean to say is that any of the sites that are already linked from an indexed site will always be crawled. You can also see crawling variations, like some sites, are crawled frequently and few are crawled less, some are crawled to the deeper depths.

Example- Consider the world wide web as a chain of showrooms in a mall.

Well, you can understand crawling in a better way through this example. Now each showroom is a unique document like a web page, JPG, PDF, etc. So, what search engines have to do is that it needs a way out to “Crawl” entire mall in order to find out all the shops on the way, so they choose the different ‘links’ as a best-possible path.

  • Crawling and Indexing- Both of them provides millions of pages, documents, files, videos, news, media, etc. on the World Wide Web.
  • Providing answers- Answering users queries mostly through the long list of pages (mostly relevant pages) that is being searched and ranked consequently for their relevancy.

How does the Search Engine Bots work efficiently?

The structure of web links aims to keep all pages bound together.

  • Millions of links present to enable the automated robots of search engines (usually named as ‘crawlers’ or ‘spiders’, in reaching a vast number of interconnected documents available over the web.
  • So, once the search engines find out all of these pages, they take the code from the same and store few chosen pieces them appropriately in the wide databases, so that it can be recalled any time in the later stages in case of search queries.
  • This can be done in a fraction of seconds because the search engines have created huge data-centers worldwide.
  • This complex storage facility holds a wide number of machines that process an unlimited amount of information within a fraction of seconds.

When an individual does any kind of search in Google or any other search engines, it can show the result instantly. The user can feel dissatisfied even it there is the delay of 1-2 seconds, hence, it has to be fastest possible for responding to users’ queries.

Finding answers in search engines-

To respond instantly, the search engines must be unbelievably fast and to find any information instantly, it has to go through millions of web pages within a fraction of seconds. For this, the search engine uses specific software robots and these robots are named as “spiders”. The spiders are used to build a long list of web results. So, when the spider builds this list of words present in websites, the whole process is known as ‘Web crawling’. When the spider initiates the work with the most popular website, it indexes the words, follows every link on the website, it travels as much as possible to find answers.

The spiders start its search from the most used servers and most popular pages. i.e it starts with popular and relevant pages.

The initiation of web search by spiders

Initially, Google has a specific server for availing URLs to the spiders. So instead of relying on internet service provider for DNS, that deals with the translation of server’s name into a specific address Google has a dedicated DNS for them to minimize any delay.

The Google spider focused on 2 things when it referred the HTML page, i.e.

  • The words present in a particular page
  • And where these words were located.

The Google spider was designed in such a way that it was able to index all the words on a page, leaving just ‘a’, ‘an’ and ‘the’. Initially, each spider was capable of keeping around 300 connections to Web pages open at the same time. Later on, it increases and multiple spiders are being used during a single search. Other search engine’s spiders may take a differing approach.  We also have other systems like ‘Alta Vista’ that goes in a different direction whole together and indexes all words including a, an, the, etc.

Over the years, our smart engineers have implemented many better ways to find précised results for any kind of queries. The engines use a vast number of algorithms (mathematical equations) to sort the relevant results and then to rank those results in according to their popularity.


Search Engines are a blessing because they answer every query of users, that too within a fraction of seconds. It is essential and interesting to know how these search engine bots work. It is a complex system that is hard to understand by everyone and definitely not as easy as it works for us in finding the search results quickly.

1 Comment
2018-06-29 00:38:20
2018-06-29 00:38:20

Interesting read Manchun. Thank you for providing this to us.

Add Comment