28 Jul
Search engines are the gateway of easy-access information, but site crawler, their little-known sidekicks, play a crucial role in rounding up online content.
Google’s site crawlers are a vital component of the SEO ranking process. If you want your website to rank, your site needs to be indexed. To be indexed, website crawlers need to be able to find and rank your site.
What is a site crawler?
Site crawlers are the librarians of the internet, crawling web pages and indexing useful content. Search engines have their own site crawlers; for example, Google has its “Google bots.” These bots (known also as “crawlers” or “spiders”) visit new or updated websites, analyze the content and metadata, and index the content it finds.
There are also 3rd party site crawlers that you can use as part of your SEO efforts. These site crawlers can analyze the health of your website or the backlink profile of your competitors.
How does a site crawler work?
When you search for something and see a list of possible matches, it’s because of site crawlers. Site crawlers are complex algorithms created with massive computer programs. They’re meant to scan and understand a large volume of information, then connect what it’s discovered with your search term. But how do they get this info?
Let’s break it down into 3 steps every site crawler takes:
- Crawling your website
- Scanning content on your site
- Visit the links (URLs) on your site
All of this information is stored on a massive database and indexed according to keywords and relevance.
Google then hands out the top spots to the best, most reliable, most accurate, and most interesting content while everyone else is shuffled down the list. Unfortunately, not all websites will be crawled if they’re not “crawler friendly.”
That’s where 3rd party site crawler tools like the Site Audit tool can help. The Site Audit tool crawls your website, highlighting any errors and any suggestions you can use to improve the crawlability of your site.
How site audit tools can help?
If your site doesn’t get crawled, you’ve got zero chance of driving organic traffic to it. Sure, you could pay for ads to gain top spots, but as any SEO pro will tell you organic traffic is a pretty accurate indicator of a quality website.
To ensure that search engine crawlers can get through, you’ll need to crawl your own website regularly. Adding new content and optimizing pages and content is one sure-fire way to do this. The more people who link to your content, the more trustworthy you seem to Google. The Site Audit tool can help by:
- Using our specialized site crawlers to check the health of your website
- Checking over 120 issues that may be affecting your website
- Showing you exactly what to fix on your website (and why it’s important)
Fixing errors with the site audit tool
If you’re new to SEO, don’t panic when you see your report. No one likes seeing site errors and warnings, but it’s important to fix them as soon as you can.
Once completed, the Site Audit tool will return a list of errors it has spotted on your site. These issues are usually categorized as:
- Errors: These are high-impact issues, so treat them like a priority. These are any major issues that are preventing your site from being crawled or index.
- Warnings: These issues are still pretty important, but not as much as errors. Plan to tackle these next.
- Notices: These aren’t serious issues, but they could impact your user’s experience. Take care of these when all other issues are addressed.
The tool explains each issue and offers suggested fixes. You can filter or sort for specific issues in the “Issues” tab:
On the overview page, you will see your crawlability score. This thematic report offers an overview of the indexed pages and any issues preventing the bots from crawling the pages.
Work your way through these until you’ve completed each one on the list. If you’re a Trello or Zapier user, you can assign any of the tasks to a board or a task manager.
Once you’re done updating your site, run another audit. Upon completion, you can select “compare crawls” to see if and how your efforts are making an impact on your website’s health.
What web crawler bots are active on the Internet?
The bots from the major search engines are called:
- Google: Googlebot (actually two crawlers, Googlebot Desktop and Googlebot Mobile, for desktop and mobile searches)
- Bing: Bingbot
- Yandex (Russian search engine): Yandex Bot
- Baidu (Chinese search engine): Baidu Spider
There are also many less common web crawler bots, some of which aren’t associated with any search engine.
Why web crawlers matter for SEO?
SEO improving your site for better rankings requires pages to be reachable and readable for web crawlers. Crawling is the first way search engines lock onto your pages, but regular crawling helps them display changes you make and stay updated on your content freshness.
Since crawling goes beyond the beginning of your SEO campaign, you can consider web crawler behavior as a proactive measure for helping you appear in search results and enhance the user experience.
Crawl budget management
Ongoing web crawling gives your newly published pages a chance to appear in the search engine results pages (SERPs). However, you aren’t given unlimited crawling from Google and most other search engines. Google has a crawl budget that guides its bots in:
- How often to crawl
- Which pages to scan
- How much server pressure is acceptable
It’s a good thing there’s a crawl budget in place. Otherwise, the activity of crawlers and visitors could overload your site. If you want to keep your site running smoothly, you can adjust web crawling through the crawl rate limit and the crawl demand.
The crawl rate limit monitors fetching on sites so that the load speed doesn’t suffer or results in a surge of errors. You can alter it in Google Search Console if you experience issues from Googlebot.
The crawl demand is the level of interest Google and its users have on your website. So, if you don’t have a wide following yet, then Googlebot isn’t going to crawl your site as often as highly popular ones.
Roadblocks for web crawlers
There are a few ways to block web crawlers from accessing your pages purposefully. Not every page on your site should rank in the SERPs, and these crawler roadblocks can protect sensitive, redundant, or irrelevant pages from appearing for keywords.
The first roadblock is the noindex meta tag, which stops search engines from indexing and ranking a particular page. It’s usually wise to apply noindex to admin pages, thank you pages, and internal search results.
Another crawler roadblock is the robots.txt file. This directive isn’t as definitive because crawlers can opt-out of obeying your robots.txt files, but it’s handy for controlling your crawl budget.
Optimize search engine crawling
Search engine crawlers are incredible powerhouses for finding and recording website pages. This is a foundational building block for your SEO strategy, and an SEO company can fill in the gaps and provide your business with a robust campaign to boost traffic, revenue, and rankings in SERPs.
What are the main web crawler types?
Web crawlers are not limited to search engine spiders. There are other types of web crawling out there.
Email crawling – Email crawling is especially useful in outbound lead generation as this type of crawling helps extract email addresses. It is worth mentioning that this kind of crawling is illegal as it violates personal privacy and can’t be used without user permission.
News crawling – With the advent of the Internet, news from all over the world can be spread rapidly around the Web, and to extract data from various websites can be quite unmanageable.
There are many web crawlers that can cope with this task. Such crawlers are able to retrieve data from new, old, and archived news content and read RSS feeds. They extract the following information: date of publishing, the author’s name, headlines, lead paragraphs, main text, and publishing language.
Image crawling – As the name implies, this type of crawling is applied to images. The Internet is full of visual representations. Thus, such bots help people find relevant pictures in a plethora of images across the Web.
Social media crawling – Social media crawling is quite an interesting matter as not all social media platforms allow to be crawled. You should also bear in mind that such type of crawling can be illegal if it violates data privacy compliance. Still, there are many social media platform providers which are fine with crawling. For instance, Pinterest and Twitter allow spider bots to scan their pages if they are not user-sensitive and do not disclose any personal information. Facebook, LinkedIn are strict regarding this matter.
Video crawling – Sometimes it is much easier to watch a video than read a lot of content. If you decide to embed Youtube, Soundcloud, Vimeo, or any other video content into your website, it can be indexed by some web crawlers.
Why are web crawlers called ‘spiders’?
The Internet, or at least the part that most users access, is also known as the World Wide Web – in fact that’s where the “www” part of most website URLs comes from. It was only natural to call search engine bots “spiders,” because they crawl all over the Web, just as real spiders crawl on spiderwebs.
Making pages accessible to crawlers
Crawlers are very simple programs. They begin with a list of links to scan, and then follow the links they find. Sounds simple, right? Well, yes, it is, until you get to complex pages with dynamic content.
Think about on-site search results, Flash content, forms, animation and other dynamic resources. There are many reasons why a crawler would not see your website in the same way that your human visitors do.
In fact, many businesses take steps to ensure that web crawlers ‘see’ all of the content available. This is particularly an issue for websites with lots of dynamic content which may only be visible after making a search.
Here you can see how Google Search Console can be used to understand how many of your pages are indexed, which pages were excluded and why, along with any errors or warnings that were encountered when crawling your website.
What is the difference between web crawling and web scraping?
Data scraping, content scraping web scraping is when a bot downloads the content on a website without permission, often with the intention of using that content for a malicious purpose.
Moreover, it is usually much more targeted than web crawling. Web scrapers may be after specific pages or specific websites only, while web crawlers will keep following links and crawling pages continuously.
Also, web scraper bots may disregard the strain they put on web servers, while web crawlers, especially those from major search engines, will obey the robots.txt file and limit their requests so as not to overtax the web server.
Should web crawler bots always be allowed to access web properties?
That’s up to the web property, and it depends on a number of factors. Web crawlers require server resources in order to index content – they make requests that the server needs to respond to, just like a user visiting a website or other bots accessing a website. Depending on the amount of content on each page or the number of pages on the site, it could be in the website operator’s best interests not to allow search indexing too often, since too much indexing could overtax the server, drive up bandwidth costs, or both.
Also, developers or companies may not want some webpages to be discoverable unless a user already has been given a link to the page (without putting the page behind a paywall or a login). One example of such a case for enterprises is when they create a dedicated landing page for a marketing campaign, but they don’t want anyone not targeted by the campaign to access the page. In this way they can tailor the messaging or precisely measure the page’s performance. In such cases the enterprise can add a “no index” tag to the landing page, and it won’t show up in search engine results. They can also add a “disallow” tag in the page or in the robots.txt file, and search engine spiders won’t crawl it at all.
Website owners may not want web crawler bots to crawl part or all of their sites for a variety of other reasons as well. For instance, a website that offers users the ability to search within the site may want to block the search results pages, as these are not useful for most users. Other auto-generated pages that are only helpful for one user or a few specific users should also be blocked.
Conclusion
To make sure your site is indexed by search engines, make your website as crawlable as possible. You need to make sure it’s set up effectively to allow bots to explore every page they can. Google may change ranking factors in the future, but we know that user experience and crawlability are here to stay.
Running site audits regularly helps you stay on top of potential errors that can impact your site’s crawlability. Remember: website maintenance is a dedicated process, so don’t be afraid to take your time!