How Google Works?

I. What is Google?

II. Google File System.

     Google File System (GFS) is a file system developed by Sanjay Ghemawat, Shun-Tak Leung and Urs Holzle for Google to handle rapid growth of the company’s infrastructure and to suit their needs. The company was looking for a way to improve the stability and reliability of their services and the only reasonable way to do that was to build their own file management system as no other system was able to handle massive number of requests over huge amount of servers.
     The GFS supports gigantic user population combined with very cheap computer equipment that tends to break down quite often. From the beginning of Google, the company used cheap computers running Linux to maintain the need for storage space for information gathered by web crawlers (and other services, like GMail, etc.). It caused problems with reliability of service and low efficiency. They had to create reliable software to run over unreliable hardware. It was a total change to the economy of the IT companies.
     The system was designed to be fully scaled, flexible and be easy to expand. It also doesn’t remove or override data, but rather append it. It is easier and faster way of storing data to add new files instead of updating and deleting old ones. Google system is also planned to resist numerous hardware failures and human error factors. That’s why each file in the system has three copies (default, but for high demand files even more) on separate servers to ensure that if one server goes down, there are two back up servers ready to take the request and fulfill the search process. And all of that is performed in milliseconds.
     Files managed by GFS range from hundred of megabytes to several gigabytes as it is better to operate on couple of huge files rather than handle with millions of small sized files. So, to manage efficiently GFS stores data in chunks of about 64 megabytes each – chunks are similar to clusters or sectors in regular file system – the smallest part of data that the system supports.
For example, to store 128 megabytes of data GFS will use 2 chunks. But there are files smaller than 64 megabytes, like 20 megabytes, what to do with them? Fortunately there are so few files small as that, that Google doesn’t bother with those files. Common files consume multiply chunks.
     GFS consists of master server and chunk servers. Master server contains metadata, like file names, file location on server and their sizes. When there is a request to certain file, the master server gives directions to the file in proper chunk server and the file is accessed.

III. Google PageRank™ system.

     The source of Google success is page ranking system called after Larry Page, Google PageRank™. It is a mathematical algorithm calculating the importance of a website according to many onsite and offsite factors.
     Basically, a website that has more inbound links will be higher in search engine results page (SERP) than the same website with less sites linking to it. It is an idea taken from the science world, where the importance of publication is measured through the number of quoting in periodics. Placing a link to some website Google treats as some kind of vote for this site and gives it some significance. But weight and quality of a website also counts. It is known that a government website is more important than a personal home webpage. It is like that because government have power and right to publicize official announcements, changes to law, new tax systems and so on. Our home page can only refer and link to the above information’s.
     That’s why a link from a site with bigger credibility and higher meaning weights more than couple or even some tens of links from an unknown pages with little or no importance. Logically it appears to use this system to prioritize sites on internet.
     Google sorts websites according to 10 scale system, where 0 is site with no importance and 10 is a well known, credible website with many visitors and high traffic (very few websites have PageRank™ of 10).

     Google, by adding some new features to its algorithms gives higher importance to websites that are loner on the internet. In natural way these sites should possess some number of inbound links from other websites and some constant number of visitors. That’s why new websites get higher link weights than old sites to close the gap between them. New sites have no or very few inbound links and Google tries to make it equal to everyone by doing that. New sites can much easier get high listings in search engine results and high PageRank™ in the early phases of their existence. With time they usually lose their high positions and only then it is possible to determine what weight does the website really have.
     It is not a good idea to buy a lot of links from a high PageRank™ websites to newly created web pages. Google assumes that such pages don’t have reach and developed content like sites that exist longer. It is common mistake while web optimization process. If link from a newly published website from a new web domain is added to numerous web directories and websites with high PageRank™ it is possible that such site will land in sandbox. Do not mistake sandbox with ban – a removal from indexes. Sandbox means lowered website PageRank™ and lower position in SERP, but is not equal to removal from indexing. Such site might still appear in SERP, but will lose its high positions for some or all keywords. The site will continuously be visited by Google web crawlers so don’t worry. After a while (from several weeks to several months) it will leave sandbox and be restored on full rights to search engine indexes.
     However the trademark rights to PageRank™ system has Google Inc., the patent rights belong to Stanford University, where Larry Page and Sergey Brin, the founders of Google studied.

     Google itself describes PageRank™ as:
“PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page's value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. But, Google looks at considerably more than the sheer volume of votes, or links a page receives; for example, it also analyzes the page that casts the vote. Votes cast by pages that are themselves "important" weigh more heavily and help to make other pages "important." Using these and other factors, Google provides its views on pages' relative importance.”

IV. Google web crawlers.

     There are different types of search engines that use different methods of searching the Internet resources.
     Robots read only links located on already found websites and based on that they create tree-like link hierarchy. Spiders read the whole content of a website, the title, links, the document text and information inside meta tags. There are also engines like Metacrawlers, which include Meta Search and Smarter Meta Search engines. The first one sends the search term entered by the user to indexes of different search engines simultaneously and returns requested information’s based on found search results. So they don’t have their own index servers and they do not search the internet in an active way. They use already existing indexes of other search engines. Smarter Meta Search engines use the same method with a difference. They use linguistic and collective analysis to determine even more accurate search results than Meta Search engines.
     Search engines do not index everything that is available on the internet though. They do not index:
· Multimedia files - mp3, mpeg, avi, jpg, gif, png
· Documents to which a password is required – e.g. mailboxes or intranet, an inside Internet of companies
· Sites that have been excluded from indexing by its author using robots.txt file or metatag robots
     All of the websites that exist but are excluded from search engine indexes for different reasons create so called “invisible network” or “invisible internet”. This invisible internet is three times bigger than all sites creating the visible internet. It is like that because companies or government institutions in an obvious way do not want to share with their data. That’s why they exclude their content from indexes or hide it inside intranet.

     Google calls its spider “Googlebot”. It divides into Freshbot and Deepbot. The task of Freshbot is to find new, fresh content on a website and that’s why it visits the same sites even couple times a day. Deepbot on the other hand is responsible for deep website crawling. Its main purpose is to create a full view, a picture of your website content, navigation system and all the inbound and outbound links. If you see a search engine results change it means the Deepbot was active.
     In Google the search process is divided into three parts. First the Googlebot crawls the internet in search for changes to existing websites and for new websites. It works like standard web browser. It sends request to a server for specific website and then saves it at sends it to index servers. It can request thousands of different websites simultaneously, but it deliberately makes it slower to avoid crashing web servers or overcrowding the real human requests to the same server. Googlebot finds sites in two ways: by adding a website through an add URL form (www.google.com/addurl.html) or by finding links by crawling the internet.
     When Google fetches a website it collects all the links on this site and adds it to “visit soon” URL list. That way in short time it can visit wide area of internet and make the search process faster. But it also causes problems. Google have to examine the “visit soon” URL list to check if there are duplicates of URL addresses and if so to delete the duplicate to prevent from visiting the same site too often.
     To keep the indexes up to date Google re-crawls sites on regular basis. For a newspaper site or a highly visited portal it can be daily, for a stock quotes much more frequently and for other pages once a month or several times a month.

     Googlebot sends then the whole text it finds on sites to Google indexer, an indexing database servers. They store the text sorted alphabetically by term - a keyword or phrase. To each term a list of documents in which it appears and where on site is attached so it’s easy to find the location of correct document for certain user query. To eliminate unimportant words, to improve the search process and to make it faster Google indexer doesn’t take into consideration most frequent words in each language. These words, called “stop words” (such as is, on, or, the, at, in, how, why) don’t make the search process any more precise, so they can be ignored.
     When a user enter a search term to Google, it sends the request to indexing servers to find out if the term exists in the database. At this point it is important to realize that because of the amount of data Google holds it would be to difficult to store all information’s in one indexing server. That’s why Google uses many separate servers where each holds some part of all data. The query is send therefore to different servers simultaneously and if the term exist in the database Google generates 1000 most relevant results based on more than 100 factors (such as PageRank™, metatags, age of a website, traffic on website and many more).
     At the same time to each document a special number is attached. The Document-ID is then send to file servers where a title and description of a website is added based on its metatags. If there are no metatags the title and description are generated automatically based on sites content. Also in this case there are many file servers working simultaneously.
     The last stage is adding advertisements to the search results taken from the ad-servers adequate to the search term. The ad-servers keep information’s about advertisers, campaigns and they determine which advert should be published on search results page. Those ad-servers are the main income of Google company. They bring 98% of overall revenue of this search engine.
     All the information’s are put together and displayed in a user web browser as dynamically generated website. And all of that is done is seconds.

References

I think now you get the idea how big Google really is. If you still feel doubts visit this page:
http://en.wikipedia.org/wiki/List_of_Google_products
It’s a comprehensive list of all Google projects with short descriptions. I’m sure you will find many interesting things there.

Very good and more technical articles about Google File System can be found here:
http://storagemojo.com/?page_id=152
http://www.baselinemag.com/article2/0,1397,1985047,00.asp
http://labs.google.com/papers/gfs-sosp2003.pdf (PDF file)

If you want to read more about how Google PageRank™ algorithm works visit http://en.wikipedia.org/wiki/PageRank where waits for you great article about technology that PageRank™ relies and was built on.
Full Google explanation of its ranking system can be viewed under this address:
http://www.google.com/technology/

For more information’s about how Google works take a look on these websites:
http://www.googleguide.com/google_works.html
http://www.portfolio.com/interactive-features/2007/08/google
http://www.portfolio.com/culture-lifestyle/goods/gadgets/2007/08/13/How-Google-Works
http://en.wikipedia.org/wiki/Index_%28search_engine%29

And for last, but not least: What Google thinks about SEO and what they suggest?
http://www.google.com/support/webmasters/bin/answer.py?answer=35291