Once upon a time, Greenlane had a client launch a new site in WordPress. It was in a staging area before launch, to which the environment was properly blocked from Googlebot. Instinctively upon hearing the news of the website going live, we decided to look for a robots <meta> tag.
Sure enough, every page was marked as “noindex, nofollow”. The developer forgot to remove the noindex tag. This tiny oversight could have easily cost the client hundreds of thousands of dollars if not caught. A major SEO issue averted!
Above I said “instinctively” because, well, this isn’t the first time I’ve seen a site launch set to block search engines. It’s probably not even the 50th. I worked on an eCommerce platform where many sites launched with this SEO issue. WordPress, as fine a platform as it is, makes it super easy to launch set to noindex. Since developers often build sites in staging areas, they’re wise to block bots from discovering their playground. But, in the hustle to push live an update or new design, they can forget a tiny (yet crucial) check box.
It’s the role of an SEO to monitor websites to ensure proper crawlability at all times. I recommend monitoring daily; with these tools, the time commitment is very low. Make it part of your early morning coffee and donuts routine.
I’ve gathered up four different ways you can monitor your sites for SEO problems without the use of server logs or an education in server administration. There are different kinds of advanced website monitoring (e.g., active, passive), but I’m keeping it simple and applicable for anyone. I wanted to pick a few that were diverse, free or affordable.
1. SEOradar
SEOradar is a great tool from Mark Munroe. The purpose of the tool is to alert you when technical SEO changes occur. You can set monitoring to be daily or longer, and cover changes in robots.txt, meta robots, internal links, schema, etc. Plus, you can get these alerts emailed to you.
From the website’s homepage: “SEORadar examines changes to pages and alerts users to issues with potential dire SEO consequences; including title changes, noindex tags, broken canonicals, 302s and much more. SEORadar checks for over 100 distinct site changes and warns users when they occur. Users can configure their own tests if we missed ones they need!”
Here’s an example of the dashboard for one of our enterprise clients. All is (mostly) calm today with only 2 minor changes to pay attention to. When they’re critical, we jump:
The interface is smart, allowing you to drill deeper into the alerts for identifying the SEO issue. I like being able to view two different source code documents together with the changes highlighted:
In this case, the tool caught a Javascript code change I made. I meant to do it, but how many times has a new site launched without proper tagging? Some changes happen daily as part of the website’s purpose; thus, the tool can be taught what to avoid and what to highlight.
2. Little Warden
Little Warden’s tagline is “Vital Alerts for Hidden Issues.” Little Warden is in the same class as SEORadar. I’m fond of the dashboard that gives you a quick glance of discovered issues:
The notification controls are pretty smart. Little Warden even integrates into Slack, allowing for instant notifications when an issue is found.
I also really the “change history” interface:
3. Uptime Robot
While SEOradar and Little Warden can track a lot of technical SEO related details, the biggest problem is website downtime. I’m sure we’ve all been there – it’s 2pm in the afternoon, you’re relaxed after lunch when somebody finally notices the site is down. You do some digging, and it turns out the site has been down since 4am.
Most companies don’t think to look at their own website routinely. We all just assume it’s alive and well. All webhosts suffer from periodic downtime, but poor hosts – even in this day and age – can really screw you on this one. We’ve seen Google essentially bail on websites that are full of server errors.
Website monitoring services exist to keep you in the loop, from SaaS services like Pingdom and SiteUptime, to downloadable applications. I’m particularly fond of Uptime Robot because it’s easily configurable and free for 50 monitors. If your site goes down, not only do you get an email alert, but you’ve got a great interface to follow along:
Plus you can change the intervals, starting at 5 minutes. Keeps the control in your hands:
(Bonus) Daily Rank Tracking and Analytics
The truth is, you don’t necessarily need the above tools if you are not interested in reacting quickly. If you’re old-school and don’t want more tools in your arsenal, this tip is for you.
I was once pitched by an enterprise-level all-in-one SEO solution. When they discussed their rank tracking features, I asked, “can you track rankings on a daily basis?” There was a smug response: “no, we do weekly rank updates – but I can’t see a reason why you would ever want daily rank tracking.”
Well, on very large sites where popular pages are crawled every couple days, and many updates are being made, daily rank tracking can help you quickly discover where an issue lies. It can lead you directly to the page that has the unexpected problem. Not to mention, if you’ve been stung by a penalty, daily rank tracking (and analytics) will often be your first indicator. I like to know that day – not in a week.
Analytics isn’t any different. The data pours in fast on modern analytics platforms. In less than 24 hours, most sites will be able to provide enough clues that a problem is occurring. You just have to monitor routinely.
Daily Rank Tracking
Honestly, this isn’t my favorite method of monitoring for SEO problems, but it’s an easy one. If you have a page Google favors, it will get crawled often – as often as daily. If it’s a deeper, less important page (in Google’s perspective), I’ve seen it take months to be revisited. So, if you’re going to rely on rank tracking, choose daily tracking, and monitor your oft-visited pages. Clearly, this leaves a lot of monitoring on the floor – be aware, there’s plenty of room for error and latency.
Analytics
Most analytics packages allow you to create a custom dashboard. Most also allow for daily emails. To make this work, your new job is to simply check your analytics a few times a day. For over a decade, this was the first thing I’d do when I got into the office. I wasn’t just looking at revenue, but hiccups. I’m looking for an atypical pattern.
It’s almost like a heartbeat monitor in a hospital. Clearly, this is not as simple as allowing SEOradar, Little Warden or Uptime Robot to send you an alert, but useful nonetheless.
What do you use to monitor your website? Let us know in the comments.