55. Why Aren’t My Posts Getting Indexed?

[00:00:00] Stephanie: It is happened to many of us. You work hard on a blog post and wait for traffic to roll in. When that doesn't happen, you start troubleshooting and discover that your post hasn't been indexed by Google. Let's chat about why this happens and what you can do about it. Welcome back to the podcast. Your Independence Day parties might be over, but we're just getting started with another celebration.

[00:00:36] The first birthday of Pay Less for traffic. I can't believe it, but we've been having our chats for a full year now. So to celebrate this milestone, I decided that this month we're going to be tackling questions that I've received over the past year. I love when listeners tune into an episode and then send me dms with follow up questions.

[00:00:55] Sometimes I'm able to answer them right then. Other times I know the best answer will require a bit more context and discussion. Plus, I have a feeling that if one listener has this question, more of you do as well. So this week's episode is one of those topics that I thought could use a little bit more discussion.

[00:01:15] I had a listener reach out to me after she had checked her indexing report on Google Search Console. She discovered that she had a lot of URLs that hadn't been indexed, and she wanted to know more about why this happens and if there's anything that can be done about it. I gave her a few things to think about, but then I made a mental note to go ahead and cover this topic on the podcast, So in this episode, I'll give you a quick refresher on what Google indexing is, so we're all on the same page. Then I'll share some reasons why posts aren't indexed, and finally we'll chat about things you can do to fix it. I think I've shared on the podcast before that Google runs like one big filing cabinet.

[00:01:55] All of the content on the internet is organized and categorized so that when people come looking for something, Google can efficiently pull the best content from the filing cabinet to answer that question or solve that problem for that person. The process of putting this content where it belongs in the filing cabinet is known as indexing.

[00:02:17] Bots will periodically crawl your website to check for new content, and then Google will make a decision about where that content belongs in the index. Now, the important thing to remember is that you as a content creator have no say in where Google puts your content in that filing cabinet or whether they put it in the filing cabinet at all.

[00:02:39] That is 100% up to Google. All you can do is give signals to let Google know what your blog posts are about and to show that your content is worth getting indexed. And this is what we're doing when we focus on SEO. We're making it easier for our content to get put in the filing cabinet, so to speak. But even after all of your efforts, there are no guarantees that your blog posts will even make it into the filing cabinet.

[00:03:07] In fact, it's possible that your content will end up in a stack of posts from your website that Google is aware of, but chooses not to index. Because if you think about it, crawling all of the content on the internet is a resource heavy venture, especially now in the days of mass produced AI content. So if something is in the index, it becomes part of the crawling process for your website moving forward, because bots will need to track changes, new links, and all of that.

[00:03:39] So it makes sense that Google can be picky about what gets indexed. So even though Google can be picky and a little bit mysterious, one thing they're actually pretty transparent about is why they are not indexing a post on your website, for the most part, and I'll get to that in just a second. But if you log into Google Search Console, you can see the total number of posts on your website that have been indexed, and then how many have not.

[00:04:05] And when you click on the ones that have not, you get a breakdown of the different reasons and how many posts have been not indexed for that particular reason, and so I thought we would just run through some of the most common ones. I'm not gonna go through all of the reasons why posts don't get indexed, but just using my own Google search console as inspiration, I'll go through some of the most common reasons. So one of the first reasons why a post is left out of the index is that there has been a redirect or it has been deleted. When bots know that a post used to be there but no longer exists or has been redirected, it will show up as a non indexed URL for a while. Eventually, those will kind of drop off in your report.

[00:04:50] Another reason why a post is left out of the index is that the post is considered duplicate content of something that already exists on your website. Google might also recognize a URL as an alternate address for the same piece of content, so those alternate URLs aren't indexed. When looking at my Google search console, this is the reason for most of my own non indexed posts.

[00:05:18] My indexing report is full of URLs that have tailwind, Pinterest, or even Facebook tracking attached to the URL. So Google doesn't need to index these versions of the post URL. They're just going to index the original post only. Up to this point, all of the reasons for no indexing make a lot of sense. Why should Google keep indexing a URL that no longer exists, or a post that's clearly a duplicate of another piece of content on your website?

[00:05:49] Where indexing gets tricky is when you have posts with the label of crawled but not indexed. This is content that bots have discovered and crawled so they know it exists on your website, but Google hasn't decided to index it, and we don't know the reason. And I feel like this category can feel a bit subjective, but there are ultimately reasons why Google decides to leave the post out of the index.

[00:06:19] Sometimes these posts are considered thin content. They don't provide enough context to Google about the contents of the post. Another reason is they might be orphaned posts, meaning that they don't have inbound or outbound internal links. And so that sends a signal to the bots that this content is really not important in the grand scheme of your website, otherwise you would be linking to and from it.

[00:06:47] So as you can tell, that final category that crawled but not indexed is the one that should give you the most pause and think, oh was that not a good enough post? What can I change? And so let's talk about what you can do when you notice your posts are getting left out of the index. So you've noticed that your posts aren't getting indexed.

[00:07:10] What can you do about it? Well, first, if you're just getting started with blogging it in SEO, you can submit your site map to Google. So this process basically throws your hat in the ring for the crawling process, you're saying, I have a website, it has content, here's where you can find the list of the content on my website.

[00:07:32] I'd love for it to be added to the Google Index. And so a site map is actually just a list of your current URLs on your website, and this is public. Anyone can access it. And so what you do is you grab that location of your site map, it's usually the name of your website, so www.yourwebsite.com/sitemap, all one word, dot xml.

[00:08:02] And so if you type that in, you will see your site map, or you should. And so then what you do is you copy that URL and in Google Search Console, you can submit that site map to Google. So then once Google knows where to find your site map, you've kind of thrown your hat in the ring. Then Google will eventually start to index your content.

[00:08:26] But let's say you've already done all of that. You have an established website, but you notice some of your content is not getting indexed. So the next thing you can do is just take a really hard look at your content. And make sure that it's high quality, trustworthy, and authoritative. It's also important that you make use of internal links, so for every new post you publish, you should be linking out to existing blog posts on your website.

[00:08:52] I mean, we all know this is important, but the part that often gets overlooked is that you also need to go back into an existing blog post and add a link to this new piece of content that you've just written. This sends the signal to Google that this is an important piece of content since you have other blog posts linking to it.

[00:09:14] So if you notice that you have a piece of thin or orphaned content that has been left off the index. You can go ahead and update it and then ask Google to try indexing it again. In Google Search Console, the way you do that is you just add the URL to the site inspection toolbar when it pops up as not being indexed

[00:09:35] there's a link there where you can request indexing, and this is especially helpful if you start to update or publish content after a long stretch of not blogging because your publishing patterns can actually contribute to your website's indexing schedule with the Google bots. If they crawl your website at a particular interval and notice that your website hasn't changed at all, the bots are going to start crawling it less and less often, which makes sense.

[00:10:06] Why would they continue to use resources to crawl your website when nothing has changed for months or even years? So you can use manual indexing requests to try and speed up the indexing process when you get back into blogging. There are no guarantees that indexing will happen right away, but it can be worth a try.

[00:10:26] I know that this has been a quick introduction to indexing, but I hope it's made you feel better about that big number of non indexed posts that you see in Google Search Console. Ultimately, indexing is up to Google, but you can give your content its best chance possible by focusing on the SEO and blogging best practices that I share here on the podcast and in bring your own traffic.

[00:10:50] Before we go, I wanted to let you know that year two of the podcast is going to include a regular segment with listener questions like this one. If you'd like to submit your own question to be answered on the show, there is now a form on my website. Just go to stephanie royer solutions.com/podcast to fill out the form with your question.