Most of our Search index is built through the work of software known as crawlers. These automatically visit publicly accessible web pages and follow links on those pages, much like you would if you were browsing content on the web. They go from page to page and store information about what they find on these pages and other publicly accessible content in Google's Search index.

When crawlers find a web page, our systems render the content of the page, just as a browser does. We take note of key signals – from keywords to website freshness – and we keep track of it all in the Search index.

The Google Search index contains hundreds of billions of web pages and is well over 100,000,000 gigabytes in size. It’s like the index in the back of a book – with an entry for every word seen on every web page that we index. When we index a web page, we add it to the entries for all of the words that it contains.

Because the web and other content is constantly changing, our crawling processes are always running to keep up. They learn how often content that they've seen before seems to change and revisit as needed. They also discover new content as new links to those pages or information appear.

Google also provides a free toolset called Search Console that creators can use to help us better crawl their content. They can also make use of established standards like sitemaps or robots.txt to indicate how often content should be visited or if it shouldn't be included in our Search index at all.

Google never accepts payment to crawl a site more frequently – we provide the same tools to all websites to ensure the best possible results for our users.

Our Search index contains more than just what's on the web, because helpful information can be located in other sources.

In fact, we have multiple indexes of different types of information, which is gathered through crawling, through partnerships, through data feeds being sent to us and through our own encyclopedia of facts, the Knowledge Graph.

These many indexes mean that you can search within millions of books from major libraries, find travel times from your local public transport agency, or find data from public sources like the World Bank.