Free SEO Audit

Crawl the website for technical issues and get a prioritized to-do list with detailed guides on how to fix.

Something went wrong. Please, try again later.
Sitechecker crozdesk rating Sitechecker crowd rating Sitechecker capterra rating
Sitechecker trusted company

What is the Crawlability Checker?

The Crawlability Checker by Sitechecker enables you to check crawlability (verify a site’s accessibility for crawling), detailing its contents and appearance. It assesses the number of pages crawled and indexed, allowing for a comparison of these metrics. The tool allows customizing the scope of crawling, specifying both the number of pages and their types, including those marked as ‘no index’, ‘no follow’, and accessible to crawling robots.txt and XML sitemap.

The Google Crawl Checker can help you in many ways

Assess your website’s crawlability: analyzes links with ‘nofollow’ attributes, pages tagged with ‘noindex’, and includes URLs containing query strings for an in-depth crawl. It evaluates all images for SEO attributes, executes and reviews JavaScript to ensure dynamic content is properly handled, inspects CSS files for rendering issues, and follows outbound links to examine the site’s external link profile.

Finding mismatches in the search: identify and compare the number of pages crawled with those indexed.

Customizing crawl settings: provides flexible options to adjust crawl limits, with the ability to either adhere to or bypass the directives of robots.txt. It enables configuration of the number of pages to scan, and paralleled requests offers the choice to include or exclude pages based on options like ‘nofollow’ and ‘noindex’. Additionally, it features sitemap detection. This adaptability allows for precise customization of the crawl, focusing on specific types of links and content.

Key Features of the tool

Unified Dashboard: offers a centralized platform where users can easily monitor and manage all aspects of their website’s SEO performance. It displays various metrics and insights in a cohesive, easy-to-understand format, allowing for efficient tracking and decision-making.

User-friendly Interface: designed with simplicity and ease of use in mind. Its intuitive layout ensures that users, regardless of their technical expertise, can navigate and utilize the full range of functions without facing a steep learning curve. This accessibility makes it an ideal choice for both beginners and seasoned SEO professionals.

Complete SEO Toolset: provides a comprehensive suite of SEO tools that cover all necessary aspects of search engine optimization. This includes keyword research, backlink analysis, on-page SEO audits, rank tracking, and more. This extensive toolset enables users to conduct thorough analyses and implement SEO strategies effectively.

How to Use the Google Crawl Checker Test

Once the scan is initiated, the tool assesses your domain for crawlability. It conducts an in-depth analysis of the pages, identifying a range of errors like server response codes and pages that are not accessible. The complete site audit covers more than 300 potential issues, encompassing link tests, indexing, content assessment, code validation, and beyond.

Step 1: Enter a domain name of your website

Crawlability Checker
To receive results and access to Sitechecker's features for 14 days, start your FREE trial. Sign-up is easy with your email, Google, or Facebook account. No credit card is needed.

Step 2: Get results

The tool illustrates the number of ‘Crawled pages’ versus ‘Indexable pages’, providing a quick visual assessment of the site’s accessibility to search engines. Adjacent to this, a sidebar categorizes the issues affecting the site, such as ‘Links’, ‘Content relevance’, and ‘Mobile friendly’, showing the impact level on the site’s SEO.

In the main pane, there’s a categorized list detailing various SEO concerns like indexability problems and content relevance, each with the number of relevant issues found. For instance, it shows counts for ‘Meta noindex pages’ and ‘Meta nofollow pages’, which are crucial for search engine visibility.

The tool summarizes the site’s health with metrics such as ‘Criticals’ and ‘Warnings’, indicating the severity and quantity of issues detected.

Site Audit Dashboard

The tool features a crawlability test tool that audits a website’s HTML pages. The interface shows various SEO metrics for each page, such as indexability, robots.txt status, and HTTP status codes, with page errors like broken links or redirects.

A sidebar categorizes and counts the issues found, such as ‘Links’ and ‘Content relevance’, providing a clear overview of areas for SEO improvement. This tool aids in identifying and prioritizing fixes to enhance the website’s visibility to search engines.

All HTML Pages

Additional Features

The tool provides customizable crawling settings, enabling you to verify the accessibility of specific pages or site files, including sitemaps and .txt files. To accomplish this, navigate to the ‘Project Settings’, then ‘Site Audit’, and select the desired settings.

The Crawlability Test Tool empowers users with comprehensive features for crawling. You can include detects sitemaps for effective site navigation and allow setting crawl limits.

A suite of crawl settings offers further refinement, enabling the inclusion of nofollow links, links with parameters, and the crawling of various site assets like images, JavaScript, and CSS. This ensures that all critical elements that could influence the search engine visibility are covered. The ability to crawl external links extends the audit’s reach, providing a holistic view of the site’s SEO ecosystem.

The tool allows managing server load through adjustable parallel request settings. You can customize their crawling scope by choosing specific site elements to be crawled, and can opt to follow or ignore robots.txt directives. You benefit from API access for automated data retrieval, integrating the tool into larger SEO strategies seamlessly.

Together, these features make the Crawlability Test Tool a flexible and powerful solution for website owners and SEO practitioners aiming to optimize their site’s search engine presence.

Site Audit

Final Idea

The Crawlability Tester checks website crawlability, pinpointing errors and comparing crawled versus indexed pages. It features a user-friendly dashboard for SEO monitoring, a suite of SEO tools for analysis and strategy, and customizable settings for targeted audits. The tool simplifies SEO optimization by allowing users to adjust settings.

The "Noindex" status is used to identify pages that are set to be excluded from search engine results. This helps in ensuring that only desired pages are visible to search engines.
Yes, it can detect temporary redirects by displaying the HTTP status codes, such as 302, which indicates a temporary redirect.
You can customize the crawl to include or exclude content such as images, CSS, JavaScript, and external links, tailoring the audit to your specific needs.

What users are saying

Check other of our awesome SEO tools!
Content Optimization
Keyword Research
Link Building
Off-Site SEO
On-Page SEO
Rank Tracking
View more tools