Before keywords, backlinks, and content marketing will have any affect on your search engine optimization, Google must be able to crawl the navigational links on your site. Without the crawl, very few — if any — pages beyond the homepage will rank in organic search.
Search engines crawl every unique page they can find, and a portion of the pages they crawl get indexed. A very small percentage of the pages that make it into the index actually rank in search results, as shown in the image above (not shown to scale).
That makes the crawl critically important to your organic search performance.
If a search engine can’t crawl through a link to a second page, that second page won’t be crawled, indexed, or ranked. For example, let’s say your site sells shoes. On your homepage there are links to category pages for boots, men’s shoes, and sandals. If those three links aren’t crawlable, the pages for boots, men’s shoes, and sandals are probably not going to rank, at least initially.
What Makes a Link Crawlable?
In order to crawl the links on your site, search engines have to be able to find them. It seems simple enough, but some of the ways developers code links can’t be crawled by search engines. Even if you can click on a link and land on another page, that doesn’t mean that search engine bots can — or will — crawl it.
In order to be crawled, Google specifies that the link must be coded with:
- An anchor tag;
- An href attribute;
- A URL;
- A closing tag.
For search engines to associate relevance with the link, the tag needs to surround relevant anchor text. The resulting piece of code looks like this:
- <a href=“https://www.jumpfly.com/blog/category/seo/”>SEO blog posts</a>
Every search engine can crawl that link. However, links can also be coded in ways that behave perfectly link-like to your browser but that search engines do not follow as links. For example, the following links won’t get crawled:
- <a onclick=“changePage(‘https://www.jumpfly.com/blog/category/seo/’)”>Bad Link</a>
- <span onclick=“changePage(‘https://www.jumpfly.com/blog/category/seo/’)”>Bad Link</span>
The first one starts with an anchor tag, but then uses an onclick attribute instead of an href. The second one uses a span tag instead of an anchor tag.
Platform and site developers make decisions on how to code individual links every day as part of managing a website. These decisions usually don’t cause any problems for the customers on your site. However, if you want to drive more customers to your site via organic search, these decisions become incredibly important.
How to Check Your Links
If you’re somewhat familiar with HTML, you can check the links on your site to make sure they’re using a crawlable format.
CAPTION: Watch this short clip showing how to use the Chrome browser’s inspection tool to see how your links are coded.
In your Chrome browser, right click on any link and select “Inspect” from the menu, as shown above. A panel will open to the right or at the bottom of your screen that shows the code for the page with the element you right-clicked on highlighted.
Look for the anchor tag, the href, and the URL. If those three things are present, your link is crawlable. If there’s anchor text as well, you’re all set.
If your link is missing any of those elements, it’s probably not immediately crawlable. The search engine crawler will grab a snapshot of the rendered page as it displays in the browser and send it back to the datacenter along with the source code for extra processing.
After that additional process – which can take weeks – the search engine may identify the noncompliant links and set them to be crawled…. Or it may not.
Do you really want to leave that to chance?
Do yourself – and your organic search performance – a favor: Talk to your SEO manager and developer today about the type of links your site uses. If the crawling issue is widespread — in the header navigation, for instance — the traffic that you are missing out on could be substantial. Fixing the issue may not be easy, but it can be a game-changer for your organic traffic and conversions.
