6 Reasons why Google Bots are not crawling your blog post

How to make sure Google knows about your content

Even if you’re arachnophobic, there’s one kind of spider you want on your website. Search bots. These ‘spiders’ crawl through your pages looking for what to show search engine users and save a cached image of what they find. When someone enters a search query, results are pulled from content the bot has cached. Without search spiders, your site is invisible to Google, Bing, Yahoo, and other search engines.

Wrap your head around this: Google has more than 39 billion webpages cached. That number took a sharp dip starting in early June, 2016 (down from a peak of nearly 52 billion indexed pages in May, 2015), but it’s still huge. That’s one of the reasons Google doesn’t cache EVERYTHING. Another is quality – the search engine policies what it shows. Link farms, spam, and black hat SEO often land sites on blacklists. Google wants to see relevant, high-quality content.

Hopefully, you’re already offering great content and you aren’t out to spam anyone. So how do you attract the attention of search spiders?

Marketer, blogger, author, and entrepreneur Neil Patel thinks a great place to start is with your sitemap.

“In today’s hummingbird-driven world of search, there are a lot of SEO myths you need to be wary of. But, one thing remains the same: all things being equal, great content will rise to the top, just like cream. Sitemaps help your great content get crawled and indexed, so it can rise to the top of SERPs more quickly, according to the Google webmaster blog. In Google’s own words, ‘Submitting a Sitemap helps you make sure Google knows about the URLs on your site.’ Is it a guarantee your site will be indexed immediately? No, but it is definitely an effective tool that helps in that process.”

Sitemaps aren’t all that matter, as Neil will tell you in the article linked above. And if you’re having trouble getting indexed, chances are, you’re making one of the following mistakes.

1.   No XML sitemap

Bots need to know where to look. An XML sitemap shows them. Like GPS for your website, an XML sitemap shows search bots how to navigate through your content. It cuts the amount of time bots need to spend on your site down by up to 1,000% and makes it easier for your site to be indexed quickly.

If you don’t have an XML sitemap right now, add one.

Pro Tip: Don’t forget no-follow URLS. If you’ve got duplicate content on your site, use no-follows to keep search bots from crawling it and penalizing you!

2.   Messy Code

If coding isn’t your forte, talk to a developer. Bloated and non-W3C compliant code will cause you major problems. You also need to make sure you’ve got a good HTML to text ratio. Anything that slows bots down won’t do you any favors and if your site is too thin on content, Google might think it’s spam. The Panda update in 2015 was a good indicator that doorway and other content-thin pages (category pages, affiliate sales pages, and local search landing pages) are at risk of being removed from the big G’s index.

3.   You Don’t Share in the Right Places

The coders over at Google must have a thing for Reddit and Digg. Social news link submissions are indexed almost immediately and fade out of Google’s index within a few days – just like the news. But if a particular social news link tops its category or goes viral, it may stay indexed permanently.

If you aren’t using social news to increase indexing speed for new content yet, you’re missing out. It’s a great way to get your site indexed quickly.

4.   Bad Links

The number and type of your links can have a drastic impact on your search rankings and visibility in general. Before we get into what NOT to do, let’s take a look at what you CAN do, like submitting your site to reputable directories in your niche. It’s a great way to improve your visibility and get a few sites with high domain authority (DA) to link to you, something search engines like Google want to see.

You can increase inbound links from sites with high DA by actively seeking guest post opportunities on them, establishing social media profiles (which we’ll tackle in a bit), writing and submitting press releases, posting video on YouTube and Vimeo, and submitting articles to high-profile, high-quality article directories (be VERY selective).

Create stunning and shareable visual content, case studies, and slideshares to increase your brand’s reach and visibility. Also try following the Twitter hashtag #HARO and watching for opportunities to provide relevant interviews to reporters.

Each of the steps above will come with a backlink or ten – helping you quickly and easily create numerous inbound links from sites with high DA.

Do NOT submit your site to poor quality directories, unless you’re keen on increasing your spam factor. If you have useful content on your site (and plenty of it), check out blog directories like TopRank, Alltop, and Technorati, but don’t submit to crappy blog directories – they do more harm than good.

You also need to make sure that all your inbound and outbound links are pointing to live content. 301 redirects are no fun for anyone. Keep an eye on content that links to you and content you link to, as well. Good link hygiene is a great way to keep Google happy with you.

5.   You Didn’t Submit Your Sitemap

…and we’re back at sitemaps. We already discussed how important they can be in helping Google, Bing, Yahoo, and other search engines crawl your site, but there is a chance that those bots won’t know your site exists unless you tell them. If you’ve done everything you’re ‘supposed’ to do, check the most basic task. Did you submit your sitemap?

Google is very adept at discovering new sites, pages, and content quickly, but it isn’t flawless. Set up your Google Webmaster Tools or Google Search Console account so you can see what’s going on when Google’s bots search your page and minimize the impact of crawl errors quickly.

Keep in mind that Google isn’t the only search engine out there. To get the most impact from organic traffic, make sure you directly submit your site to all of the top search engines, including:

• Google
• Yahoo
• Bing
• DuckDuckGo
• Yandex
• Ixquick
• com

6.   Are You Anti-social?

If you groan every time you hear the phrase ‘social media’, you’re probably not using it right. Search engines are watching social signals, and having a presence on Twitter, Facebook, Google +, YouTube, Instagram, and more can be very good for your SEO. It’s also a great way to draw the attention of search engine bots.

There’s another huge reason to pay attention to your social signals. Like Neil Patel said,

“Social is a search engine too.”

YouTube is the second-largest search engine in the world, Facebook boasts roughly 1/7^th of the global population as active users, and Twitter’s search engine handles 2.1 billion queries DAILY. Google + is a Google product, and as a result, using it is likely to keep you on Google’s radar at least.

Social media as such isn’t necessarily a driver of sales. There are plenty of critics who argue that Google has openly stated social signals aren’t counted as a ranking factor – that’s only half true. Google has repeatedly said that they don’t watch all social signals, but they do admit to crawling social pages when they aren’t blocked.

Google doesn’t care how many followers you have, but if your content is showing up on social media, related posts are likely to show up in that form on search results pages, too. When you search for prominent brands, you’ll find their social profiles among the top results. That is a strong signal that, even though Google doesn’t use these pages in their ranking factors, you can improve your traffic if you maintain active social profiles.

Social media also gives you a chance to create relationships with your customers and potential clients, share your content to a wider audience, and catch the attention of key influencers. Obviously, this offers significant benefits for your overall marketing strategy. It also helps you gather more backlinks by increasing your content’s reach.

How to Make Your Site Search Spider Friendly

The steps above keep you on Google’s search bots’ good side and attract search bots’ attention. Keeping your site optimized for SEO is also key. Done correctly and kept up to date to reflect changes in Google’s search algorithm, SEO addresses many of the issues you’re probably facing if your site isn’t currently indexed.

Make sure your site has an XML sitemap and that you resubmit it after adding new content. Keep the code clean and don’t bloat it. You also need to make sure you maintain a good HTML to text ratio and check your link health regularly for 404 errors. Sharing your content on social news networks like Reddit and Digg, as well as on your social media channels, can also help Google and other search engines ‘see’ your pages.

Do you have any questions, comments, or tips? Share them in the comments below.

Thanks to Rotem Gal for sharing their advice and opinions in this post. Rotem is Vice President of Marketing at InBound Junction. You can follow him on Twitter or connect on LinkedIn.