Sunday, 27 November 2016
New Funny Football Vines 2016 Goals Skills, Fails messi, messi wonder
Watch on YouTube here: New Funny Football Vines 2016 Goals Skills, Fails messi, messi wonder
Via https://www.youtube.com/channel/UCcZImr64sN4bR4BB_JA82vg/videos
Saturday, 26 November 2016
Friday, 25 November 2016
Backlinks from Client Sites, Sites You Own, Widgets, & Embedded Content: How to Maximize Benefits & Avoid Problems - Whiteboard Friday
Posted by randfish
When it comes to certain kinds of backlinks, avoiding penalties can be a real gray area. How can you earn the benefits without gaining the scrutiny of Google? In this Whiteboard Friday, Rand will teach you which rules to follow to keep you safe and on the up-and-up, all while improving your link profile.
Click on the whiteboard image above to open a high-resolution version in a new tab!
Video Transcription
Howdy, Moz fans, and welcome to another edition of Whiteboard Friday. This week, we're going to chat about a question we see a lot here at Moz, around what you should do with websites that you maybe design or build or do work for, your clients' websites if you're an agency or consultant, or a web designer or builder, sites that you own but are not your primary website, and widgets and embeds, blogrolls, all these kinds of things where you control the link infrastructure, or could control it, and should you.
I think one of the challenges here is to understand that many folks have recognized that, over the years, widgets, embeds, links from client websites have gotten other sites penalized, potentially even your sites penalized over the years, because you had all these links that you control pointing back to places, and to Google that can look really sketchy. So I want to talk through some best practices about how you can get link benefit and value from these places without getting yourself into trouble.
The challenge
All right. The challenge here is let's say that I own sneakerobsessed.com, but it is not my primary website, or maybe it's a client's website. But I do own sneakysneakers.com, and I'm thinking to myself, "Gosh, you know the fact that I control, I have the login for the admin here, the site owner, or me, would be fine with linking from these pages to these pages. What should I do there? I don't want to get into trouble. But I would love to get some benefit, and I think that these links could help me. Should I:
A. Add a link from every page here to a bunch of pages here or to my homepage?
B. Should I link to a variety of my pages, like take a few of these and link them to my homepage, take a few others and link to some internal pages?
C. Should I use a single page on this website to link back to maybe my homepage?
The answer is kind of, it depends. It depends.
My recommendations
Client websites
Site(s) you own
Anytime you're referencing other content you've created or things that you've done, or recognition that you have, or someone else from your organization, you would naturally link over here. That's the way you should play it, not with some specific process and checklist. Anything that matches a very standard pattern is going to be easily recognized by Google, and that can get you into trouble.
Blogrolls, syndicators, etc.
The only time you get into real trouble is if that blogroll is essentially just a paid manipulation. It's technically a blog network. It's not that you're being editorially endorsed by someone else. They're only linking to you because you're linking to them. You get into that reciprocity challenge. That's not to say you should never link to anyone who has you in their blogroll either. It's just that this has to look natural and editorial to Google, or you can get in trouble.
Syndicators, by the way, it's okay to link from every syndicated piece of content back to the original piece of content. In fact, that's the way it should be. If you do your own syndication, like I do sometimes on Medium, where I put up my blog posts that I've already put on moz.com/rand on medium.com/randfish, then you should have each of those link back to their original pieces, and that's just fine.
Widgets & embeds
So the idea here is that you have this piece of content here that's being embedded from your site. So Sneaker Obsessed, maybe the guys there went to Sneaky Sneakers. They saw a data graph of Nike shoes versus Adidas shoes sales over the last 12 months, and they were like, "Oh, man. I really want to show that. That's awesome." In fact, there's a little "embed this graph onto your own website." So they took that, and they put it on there.
More dangerous
You get into more dangerous territory with this type of thing when in the link between here there's:
- Keyword-matching anchor text
- No opt-out option, meaning there's no way to say, "I don't want to include the link to the original"
- When visitors are very unlikely to click that link; when there's no sort of, "Oh, why would I ever click on the attributed link from the embed?"
- Remotely controlled via JavaScript, meaning you can remotely update this link and anchor text, that gets real sketchy.
- Widget's purpose feels like it exists only for links, like it's not particularly useful, there's not a clear reason why this is a widget instead of just a graphic that other people can use or content they can syndicate, why make it a widget as opposed to something like a graph whose data can change, or an interactive content element, or a video player, or something like that?
- Any sort of payment or discounts that you offer or coercion to get people to embed it gets you into more dangerous territory.
Less dangerous
You're much less likely to have problems if you:
- Keep that anchor text branded or omitted entirely. It's non-branded anchor text. It's just your brand name, or it's very limited. It just says "Data Via," and via is the link itself.
- Opt-out of the link is available, meaning that someone could say, "Yeah, I want to embed that. Include a link back to sneakysneakers.com? No. No, thank you."
- There should be a compelling reason to click.
- That embed is static.
- It's not controlled by JavaScript.
- The widget feels like it's reference-focused, so there's actually some value there.
- Only embedded intentionally by those who are naturally and editorially choosing to include it.
That will keep you safe.
Hopefully, you will not encounter these problems. I think if you follow these rules, you'll be in the safe zone, and you'll also be benefiting from the link value that these can provide. I look forward to your comments. We'll see you again next week for another edition of Whiteboard Friday. Take care.
Video transcription by Speechpad.com
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!
Wednesday, 23 November 2016
Featured Snippets: A Dead-Simple Tactic for Making Them Stick
Posted by ronell-smith
Dr. Pete throwing down at MozCon 2015, flexin' in his retro Gatorade shirt
At MozCon 2015, Dr. Pete delivered a gem that perked up my ears when he discussed Google's featured snippets during his talk, "Surviving Google: SEO in 2020":
"Let's say you're No. 5 in a competitive query, and you're trying to get from No. 5 to No.1. That is incredibly difficult; that takes a lot of money, a lot of links, a lot of authority. You might be able to jump past No. 1 to No. 0 with this just by matching the question better. So it may actually be easier to get from No. 5 to No. 0 than it is to get from No. 5 to No. 1 ... Be a better match. Be a better answer to the question. It's good for users."
Something about those 98 words perked my ears up, especially the last two sentences.
"Be a better answer to the question. It's good for users."
Those words rolled around in my head for months, though their impact wouldn't be felt until even later, when I began to see how prevalent featured snippets had become.
More than a year later, I'm now more convinced than ever that most brands should be making the attainment of featured snippets a priority.
Why?
Try as they might, most sites don't stand a chance of making it to the No.1 position in the SERPs. And today, with so much priority given to ads at the top of the page, above the organic results — not to mention the fact that most people don't recognize ads from organic results — even those who do reach the coveted position have to feel as though they've secured a pyrrhic victory.
In the year-plus since the presentation, rich answers have grown significantly, as depicted by the graph below from Stone Temple Consulting:
And in the span, a number of teams and individuals have made it their charge to better decipher featured snippets, specifically regarding what seems to influence their presence for certain queries, what types of snippets there are, how to optimize your content to make it more likely that you receive one, and what Google is likely looking for when a snippet is ultimately featured.
(For in-depth background information on featured snippets, see the Related Content section at the bottom of the post.)
But not a whole lot has been written on how to keep featured snippets once your brand has one. This fact hit me like a ton of bricks during MozCon 2016, when I listened to Rob Bucci of GetStat during his presentation Taking the Top Spot: How to Earn More Featured Snippets.
This post, which is a wellspring of some comments Bucci shared near the end of his presentation, will be focused very narrowly on how to keep a Featured Snippet once your brand has been fortunate enough to receive one.
The fast five 5 Ws of featured snippets
Before we dive into that aspect, let's briefly go over a few specifics, surrounding the nuts and bolts of featured snippets.
- What are they and where do they come from? A featured snippet is the summation of an answer for a web searcher's query, typically taken from a website and includes a link to the site, the page title and the URL, according to GetStat.
- Why should you care? You shouldn't, unless you care about being the top result on the page (snark for the win). Also, since the result can come from any brand on the first page, you have the potential to occupy two positions on page 1.
- Who needs them? Any brand that desires organic reach, visibility, traffic and, yeah, uhm, conversions.
- When do they show up? Any time a query is best answered in list, table, or graph form.
For your brand or any other, however, (a) featured snippets provide you with an easy opportunity to better compete against the competition, (b) can amount to a low-investment/high-reward opportunity, and (c) can give you a leg up on the competition.
Keeping your hard-earned featured snippet
One of the main reasons to attend conferences such as MozCon in person is the potential to hear a nugget of wisdom that would be missed in a recap blog, not properly conveyed in a tweet by an attendee, or glossed over when listening to the video after the event.
For example, Dr. Pete's quote from MozCon 2015 rang clear as a bell when I heard it while sitting in the audience, but I'm not sure I would have noticed it so readily had I simply watched the videos.
During the Q&A that followed Bucci's talk, he was asked about the real value of investing in featured snippets, a particular concern given that, in most cases, Google is serving up the content with very little benefit to the brand that houses it. (Unless the user clicks on the URL at the bottom of the content and visits the website.)
Bucci did far more than answer the question before him, however.
"Let's say I was [trying to teach someone] how to make toast. The snippet is, like, step 1 put the bread in the toaster; step 2, toast the bread; step 3, eat it, right? If I added a fourth or fifth step so that it was truncated in the snippet, i.e., they didn't get the full steps to make toast, people would be more likely to click on it to get the full results. Think about how you can optimize your snippets by making it so that you don't give away the entire farm in your snippet. They have to go through your website to get the information."
This tidbit got my attention for two reasons:
- One of the biggest concerns brands have with regard to investing resources in trying to get a featured snippet is it does very little for the brand if the web searcher does not click on the URL and visit the site. Otherwise, the only entity that benefits to a significant degree is Google.
- Churn, whereby brands earn and then lose a snippet, is a very real concern, too. Research by Stone Temple Consulting found that more than 55% of the queries that showed featured snippets in January 2016 "either didn’t show a featured snippet in July 2015, OR shows a different URL for the featured snippet than it did in July 2015."
Image courtesy of Stone Temple Consulting, Google’s Featured Snippets: Automated Continuous Improvement
How to smartly invest in featured snippets
By applying the logic in Bucci's quote, your brand can employ what I call next-level thinking.
Instead of simply thinking "How do I get a featured snippet?", think "How do I keep a featured snippet?" This is especially important since, as has been reported by STC, Bucci, and others, Google is likely using engagement metrics (e.g., clicks on the URL) as a factor in determining churn.
"By crafting your snippet content in a way that encourages people to click through to your site for the full detail, you can raise your CTR on that SERP," says Bucci. "That's the key thing."
As you can see from the result below, this result, drawn from the No.1 result on the page, is unlikely to warrant a click since all the needed information is right there for the taking.
However, in the result below, the web searcher will have to click the URL and visit the owner of the content's website to see the full list.
The important point to delivering a result that's churn-resistant, says Bucci, is to think strategically.
"The biggest recommendation I made that I think people are only now starting to pay attention to how to strategically use formatting to A) win snippets and B) create great user experiences on the SERP. People were just focused on getting any old snippet, but my advice was that they should look at the query space and measure the most common snippet formats. From there, they should optimize their snippets to match those formats, because Google is clearly indicating that they want to use those formats within the give."
Bucci made a great point, highlighting how we should pay attention to the formatting and content types — not simply the queries — that consistently show up as featured snippets. This, he says, amounts to Google telling us what they're looking to reward.
Don't overthink it. Dive in.
It's exciting to see brands jump into the fray, beginning to think seriously about featured snippets and how the organic elements can impact their brands.
Dr. Pete, who has remained a passionate advocate for brands taking a serious look at how to get and keep featured snippets, says it's essential that brands build their attainment into their overall process, not use it as a one-off tactic.
"I think the first step is to think in terms of questions, and build part of your keyword research around that. In natural language search, questions are increasingly common. Which questions are part of your conversion path? Don't discount them just because they're early in the funnel or part of the research phase. Find out if those questions are showing snippets and then think about ways to use those snippets as a teaser to draw people into your content and, hopefully, your funnel. Once you're ranking on page 1, it's about shaping your content to better answer the question. I think it helps to take an 'inverted pyramid' approach — lead with your most compelling question and a summary of your content, and then dive into the details. This makes for better snippets and grabs short attention spans."
One of the best ways to get started with featured snippets is by taking a step-by-step approach so that everyone on the team knows what you're going after, why, and its likely impact to the brand.
The graphic below is as specific and as detailed as you need to be to get started.
Image source: Stone Temple Consulting
Remember, though, like all aspects of online marketing, the endeavor will be iterative. What you gain, you might lose. But the process is invaluable.
You've still created something worthwhile
Hopefully, I've shared at least one small tidbit of information that has you excited about adding the attainment of featured snippets to your content marketing strategy.
For those of you who might be on the fence, wondering if the potential reward warrants the expense, Dr. Pete's words should nudge you in the right direction.
"I think content that answers questions is naturally compelling, which is what I like about optimizing for featured snippets ... Content that answers questions succinctly provides real value and builds a base of value for your visitors, regardless of where they arrive from. Even if you lose the featured snippet, you've built something useful."
It bears repeating:
"Even if you lose the featured snippet, you've built something useful."
Dr. Pete continued:
"Think of featured snippets as much like organic ranking — they aren't something Google awards you and then lets you keep until a new winner comes along. Featured snippets are generated by the algorithm in real time, just like organic rankings. You have to keep competing for them."
Related reading:
- Ranking #0: SEO for Answers and How to Rank on Google Home, by Dr. Pete
- The Definitive Guide to Google’s Rich Answers (part of a three-post series), by Stone Temple Consulting
- How to Appear in Google's Answer Boxes, by Rand Fishkin
- How To Get More Featured Snippets, by GetStat
Has your brand experimented with featured snippets? If so, what's been the result?
Remember, Moz Pro will help you find and track featured snippets, as well as identify opportunities for acquiring them!
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!
Featured Snippets: A Dead-Simple Tactic for Making Them Stick
Posted by ronell-smith
Dr. Pete throwing down at MozCon 2015, flexin' in his retro Gatorade shirt
At MozCon 2015, Dr. Pete delivered a gem that perked up my ears when he discussed Google's featured snippets during his talk, "Surviving Google: SEO in 2020":
"Let's say you're No. 5 in a competitive query, and you're trying to get from No. 5 to No.1. That is incredibly difficult; that takes a lot of money, a lot of links, a lot of authority. You might be able to jump past No. 1 to No. 0 with this just by matching the question better. So it may actually be easier to get from No. 5 to No. 0 than it is to get from No. 5 to No. 1 ... Be a better match. Be a better answer to the question. It's good for users."
Something about those 98 words perked my ears up, especially the last two sentences.
"Be a better answer to the question. It's good for users."
Those words rolled around in my head for months, though their impact wouldn't be felt until even later, when I began to see how prevalent featured snippets had become.
More than a year later, I'm now more convinced than ever that most brands should be making the attainment of featured snippets a priority.
Why?
Try as they might, most sites don't stand a chance of making it to the No.1 position in the SERPs. And today, with so much priority given to ads at the top of the page, above the organic results — not to mention the fact that most people don't recognize ads from organic results — even those who do reach the coveted position have to feel as though they've secured a pyrrhic victory.
In the year-plus since the presentation, rich answers have grown significantly, as depicted by the graph below from Stone Temple Consulting:
And in the span, a number of teams and individuals have made it their charge to better decipher featured snippets, specifically regarding what seems to influence their presence for certain queries, what types of snippets there are, how to optimize your content to make it more likely that you receive one, and what Google is likely looking for when a snippet is ultimately featured.
(For in-depth background information on featured snippets, see the Related Content section at the bottom of the post.)
But not a whole lot has been written on how to keep featured snippets once your brand has one. This fact hit me like a ton of bricks during MozCon 2016, when I listened to Rob Bucci of GetStat during his presentation Taking the Top Spot: How to Earn More Featured Snippets.
This post, which is a wellspring of some comments Bucci shared near the end of his presentation, will be focused very narrowly on how to keep a Featured Snippet once your brand has been fortunate enough to receive one.
The fast five 5 Ws of featured snippets
Before we dive into that aspect, let's briefly go over a few specifics, surrounding the nuts and bolts of featured snippets.
- What are they and where do they come from? A featured snippet is the summation of an answer for a web searcher's query, typically taken from a website and includes a link to the site, the page title and the URL, according to GetStat.
- Why should you care? You shouldn't, unless you care about being the top result on the page (snark for the win). Also, since the result can come from any brand on the first page, you have the potential to occupy two positions on page 1.
- Who needs them? Any brand that desires organic reach, visibility, traffic and, yeah, uhm, conversions.
- When do they show up? Any time a query is best answered in list, table, or graph form.
For your brand or any other, however, (a) featured snippets provide you with an easy opportunity to better compete against the competition, (b) can amount to a low-investment/high-reward opportunity, and (c) can give you a leg up on the competition.
Keeping your hard-earned featured snippet
One of the main reasons to attend conferences such as MozCon in person is the potential to hear a nugget of wisdom that would be missed in a recap blog, not properly conveyed in a tweet by an attendee, or glossed over when listening to the video after the event.
For example, Dr. Pete's quote from MozCon 2015 rang clear as a bell when I heard it while sitting in the audience, but I'm not sure I would have noticed it so readily had I simply watched the videos.
During the Q&A that followed Bucci's talk, he was asked about the real value of investing in featured snippets, a particular concern given that, in most cases, Google is serving up the content with very little benefit to the brand that houses it. (Unless the user clicks on the URL at the bottom of the content and visits the website.)
Bucci did far more than answer the question before him, however.
"Let's say I was [trying to teach someone] how to make toast. The snippet is, like, step 1 put the bread in the toaster; step 2, toast the bread; step 3, eat it, right? If I added a fourth or fifth step so that it was truncated in the snippet, i.e., they didn't get the full steps to make toast, people would be more likely to click on it to get the full results. Think about how you can optimize your snippets by making it so that you don't give away the entire farm in your snippet. They have to go through your website to get the information."
This tidbit got my attention for two reasons:
- One of the biggest concerns brands have with regard to investing resources in trying to get a featured snippet is it does very little for the brand if the web searcher does not click on the URL and visit the site. Otherwise, the only entity that benefits to a significant degree is Google.
- Churn, whereby brands earn and then lose a snippet, is a very real concern, too. Research by Stone Temple Consulting found that more than 55% of the queries that showed featured snippets in January 2016 "either didn’t show a featured snippet in July 2015, OR shows a different URL for the featured snippet than it did in July 2015."
Image courtesy of Stone Temple Consulting, Google’s Featured Snippets: Automated Continuous Improvement
How to smartly invest in featured snippets
By applying the logic in Bucci's quote, your brand can employ what I call next-level thinking.
Instead of simply thinking "How do I get a featured snippet?", think "How do I keep a featured snippet?" This is especially important since, as has been reported by STC, Bucci, and others, Google is likely using engagement metrics (e.g., clicks on the URL) as a factor in determining churn.
"By crafting your snippet content in a way that encourages people to click through to your site for the full detail, you can raise your CTR on that SERP," says Bucci. "That's the key thing."
As you can see from the result below, this result, drawn from the No.1 result on the page, is unlikely to warrant a click since all the needed information is right there for the taking.
However, in the result below, the web searcher will have to click the URL and visit the owner of the content's website to see the full list.
The important point to delivering a result that's churn-resistant, says Bucci, is to think strategically.
"The biggest recommendation I made that I think people are only now starting to pay attention to how to strategically use formatting to A) win snippets and B) create great user experiences on the SERP. People were just focused on getting any old snippet, but my advice was that they should look at the query space and measure the most common snippet formats. From there, they should optimize their snippets to match those formats, because Google is clearly indicating that they want to use those formats within the give."
Bucci made a great point, highlighting how we should pay attention to the formatting and content types — not simply the queries — that consistently show up as featured snippets. This, he says, amounts to Google telling us what they're looking to reward.
Don't overthink it. Dive in.
It's exciting to see brands jump into the fray, beginning to think seriously about featured snippets and how the organic elements can impact their brands.
Dr. Pete, who has remained a passionate advocate for brands taking a serious look at how to get and keep featured snippets, says it's essential that brands build their attainment into their overall process, not use it as a one-off tactic.
"I think the first step is to think in terms of questions, and build part of your keyword research around that. In natural language search, questions are increasingly common. Which questions are part of your conversion path? Don't discount them just because they're early in the funnel or part of the research phase. Find out if those questions are showing snippets and then think about ways to use those snippets as a teaser to draw people into your content and, hopefully, your funnel. Once you're ranking on page 1, it's about shaping your content to better answer the question. I think it helps to take an 'inverted pyramid' approach — lead with your most compelling question and a summary of your content, and then dive into the details. This makes for better snippets and grabs short attention spans."
One of the best ways to get started with featured snippets is by taking a step-by-step approach so that everyone on the team knows what you're going after, why, and its likely impact to the brand.
The graphic below is as specific and as detailed as you need to be to get started.
Image source: Stone Temple Consulting
Remember, though, like all aspects of online marketing, the endeavor will be iterative. What you gain, you might lose. But the process is invaluable.
You've still created something worthwhile
Hopefully, I've shared at least one small tidbit of information that has you excited about adding the attainment of featured snippets to your content marketing strategy.
For those of you who might be on the fence, wondering if the potential reward warrants the expense, Dr. Pete's words should nudge you in the right direction.
"I think content that answers questions is naturally compelling, which is what I like about optimizing for featured snippets ... Content that answers questions succinctly provides real value and builds a base of value for your visitors, regardless of where they arrive from. Even if you lose the featured snippet, you've built something useful."
It bears repeating:
"Even if you lose the featured snippet, you've built something useful."
Dr. Pete continued:
"Think of featured snippets as much like organic ranking — they aren't something Google awards you and then lets you keep until a new winner comes along. Featured snippets are generated by the algorithm in real time, just like organic rankings. You have to keep competing for them."
Related reading:
- Ranking #0: SEO for Answers and How to Rank on Google Home, by Dr. Pete
- The Definitive Guide to Google’s Rich Answers (part of a three-post series), by Stone Temple Consulting
- How to Appear in Google's Answer Boxes, by Rand Fishkin
- How To Get More Featured Snippets, by GetStat
Has your brand experimented with featured snippets? If so, what's been the result?
Remember, Moz Pro will help you find and track featured snippets, as well as identify opportunities for acquiring them!
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!
Tuesday, 22 November 2016
Automating Technical Reporting for SEO
Posted by petewailes
As the web gets more complex, with JavaScript framework and library front ends on websites, progressive web apps, single-page apps, JSON-LD, and so on, we're increasingly seeing an ever-greater surface area for things to go wrong. When all you've got is HTML and CSS and links, there's only so much you can mess up. However, in today's world of dynamically generated websites with universal JS interfaces, there's a lot of room for errors to creep in.
The second problem we face with much of this is that it's hard to know when something's gone wrong, or when Google's changed how they're handling something. This is only compounded when you account for situations like site migrations or redesigns, where you might suddenly archive a lot of old content, or re-map a URL structure. How do we address these challenges then?
The old way
Historically, the way you'd analyze things like this is through looking at your log files using Excel or, if you're hardcore, Log Parser. Those are great, but they require you to know you've got an issue, or that you're looking and happen to grab a section of logs that have the issues you need to address in them. Not impossible, and we've written about doing this fairly extensively both in our blog and our log file analysis guide.
The problem with this, though, is fairly obvious. It requires that you look, rather than making you aware that there's something to look for. With that in mind, I thought I'd spend some time investigating whether there's something that could be done to make the whole process take less time and act as an early warning system.
A helping hand
The first thing we need to do is to set our server to send log files somewhere. My standard solution to this has become using log rotation. Depending on your server, you'll use different methods to achieve this, but on Nginx it looks like this:
# time_iso8601 looks like this: 2016-08-10T14:53:00+01:00
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
}
<span class="redactor-invisible-space">
</span>access_log /var/log/nginx/$year-$month-$day-access.log;
This allows you to view logs for any specific date or set of dates by simply pulling the data from files relating to that period. Having set up log rotation, we can then set up a script, which we'll run at midnight using Cron, to pull the log file that relates to yesterday's data and analyze it. Should you want to, you can look several times a day, or once a week, or at whatever interval best suits your level of data volume.
The next question is: What would we want to look for? Well, once we've got the logs for the day, this is what I get my system to report on:
30* status codes
Generate a list of all pages hit by users that resulted in a redirection. If the page linking to that resource is on your site, redirect it to the actual end point. Otherwise, get in touch with whomever is linking to you and get them to sort the link to where it should go.
404 status codes
Similar story. Any 404ing resources should be checked to make sure they're supposed to be missing. Anything that should be there can be investigated for why it's not resolving, and links to anything actually missing can be treated in the same way as a 301/302 code.
50* status codes
Something bad has happened and you're not going to have a good day if you're seeing many 50* codes. Your server is dying on requests to specific resources, or possibly your entire site, depending on exactly how bad this is.
Crawl budget
A list of every resource Google crawled, how many times it was requested, how many bytes were transferred, and time taken to resolve those requests. Compare this with your site map to find pages that Google won't crawl, or that it's hammering, and fix as needed.
Top/least-requested resources
Similar to the above, but detailing the most and least requested things by search engines.
Bad actors
Many bots looking for vulnerabilities will make requests to things like wp_admin, wp_login, 404s, config.php, and other similar common resource URLs. Any IP address that makes repeated requests to these sorts of URLs can be added automatically to an IP blacklist.
Pattern-matched URL reporting
It's simple to use regex to match requested URLs against pre-defined patterns, to report on specific areas of your site or types of pages. For example, you could report on image requests, Javascript files being called, pagination, form submissions (via looking for POST requests), escaped fragments, query parameters, or virtually anything else. Provided it's in a URL or HTTP request, you can set it up as a segment to be reported on.
Spiky search crawl behavior
Log the number of requests made by Googlebot every day. If it increases by more than x%, that's something of interest. As a side note, with most number series, a calculation to spot extreme outliers isn't hard to create, and is probably worth your time.
Outputting data
Depending on what the importance is of any particular section, you can then set the data up to be logged in a couple of ways. Firstly, large amounts of 40* and 50* status codes or bad actor requests would be worth triggering an email for. This can let you know in a hurry if something's happening which potentially indicates a large issue. You can then get on top of whatever that may be and resolve it as a matter of priority.
The data as a whole can also be set up to be reported on via a dashboard. If you don't have that much data in your logs on a daily basis, you may simply want to query the files at runtime and generate the report fresh each time you view it. On the other hand, sites with a lot of traffic and thus larger log files may want to cache the output of each day to a separate file, so the data doesn't have to be computed. Obviously the type of approach you use to do that depends a lot on the scale you'll be operating at and how powerful your server hardware is.
Conclusion
Thanks to server logs and basic scripting, there's no reason you should ever have a situation where something's amiss on your site and you don't know about it. Proactive notifications of technical issues is a necessary thing in a world where Google crawls at an ever-faster rate, meaning that they could start pulling your rankings down thanks to site downtime or errors within a matter of hours.
Set up proper monitoring and make sure you're not caught short!
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!
Automating Technical Reporting for SEO
Posted by petewailes
As the web gets more complex, with JavaScript framework and library front ends on websites, progressive web apps, single-page apps, JSON-LD, and so on, we're increasingly seeing an ever-greater surface area for things to go wrong. When all you've got is HTML and CSS and links, there's only so much you can mess up. However, in today's world of dynamically generated websites with universal JS interfaces, there's a lot of room for errors to creep in.
The second problem we face with much of this is that it's hard to know when something's gone wrong, or when Google's changed how they're handling something. This is only compounded when you account for situations like site migrations or redesigns, where you might suddenly archive a lot of old content, or re-map a URL structure. How do we address these challenges then?
The old way
Historically, the way you'd analyze things like this is through looking at your log files using Excel or, if you're hardcore, Log Parser. Those are great, but they require you to know you've got an issue, or that you're looking and happen to grab a section of logs that have the issues you need to address in them. Not impossible, and we've written about doing this fairly extensively both in our blog and our log file analysis guide.
The problem with this, though, is fairly obvious. It requires that you look, rather than making you aware that there's something to look for. With that in mind, I thought I'd spend some time investigating whether there's something that could be done to make the whole process take less time and act as an early warning system.
A helping hand
The first thing we need to do is to set our server to send log files somewhere. My standard solution to this has become using log rotation. Depending on your server, you'll use different methods to achieve this, but on Nginx it looks like this:
# time_iso8601 looks like this: 2016-08-10T14:53:00+01:00
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
}
<span class="redactor-invisible-space">
</span>access_log /var/log/nginx/$year-$month-$day-access.log;
This allows you to view logs for any specific date or set of dates by simply pulling the data from files relating to that period. Having set up log rotation, we can then set up a script, which we'll run at midnight using Cron, to pull the log file that relates to yesterday's data and analyze it. Should you want to, you can look several times a day, or once a week, or at whatever interval best suits your level of data volume.
The next question is: What would we want to look for? Well, once we've got the logs for the day, this is what I get my system to report on:
30* status codes
Generate a list of all pages hit by users that resulted in a redirection. If the page linking to that resource is on your site, redirect it to the actual end point. Otherwise, get in touch with whomever is linking to you and get them to sort the link to where it should go.
404 status codes
Similar story. Any 404ing resources should be checked to make sure they're supposed to be missing. Anything that should be there can be investigated for why it's not resolving, and links to anything actually missing can be treated in the same way as a 301/302 code.
50* status codes
Something bad has happened and you're not going to have a good day if you're seeing many 50* codes. Your server is dying on requests to specific resources, or possibly your entire site, depending on exactly how bad this is.
Crawl budget
A list of every resource Google crawled, how many times it was requested, how many bytes were transferred, and time taken to resolve those requests. Compare this with your site map to find pages that Google won't crawl, or that it's hammering, and fix as needed.
Top/least-requested resources
Similar to the above, but detailing the most and least requested things by search engines.
Bad actors
Many bots looking for vulnerabilities will make requests to things like wp_admin, wp_login, 404s, config.php, and other similar common resource URLs. Any IP address that makes repeated requests to these sorts of URLs can be added automatically to an IP blacklist.
Pattern-matched URL reporting
It's simple to use regex to match requested URLs against pre-defined patterns, to report on specific areas of your site or types of pages. For example, you could report on image requests, Javascript files being called, pagination, form submissions (via looking for POST requests), escaped fragments, query parameters, or virtually anything else. Provided it's in a URL or HTTP request, you can set it up as a segment to be reported on.
Spiky search crawl behavior
Log the number of requests made by Googlebot every day. If it increases by more than x%, that's something of interest. As a side note, with most number series, a calculation to spot extreme outliers isn't hard to create, and is probably worth your time.
Outputting data
Depending on what the importance is of any particular section, you can then set the data up to be logged in a couple of ways. Firstly, large amounts of 40* and 50* status codes or bad actor requests would be worth triggering an email for. This can let you know in a hurry if something's happening which potentially indicates a large issue. You can then get on top of whatever that may be and resolve it as a matter of priority.
The data as a whole can also be set up to be reported on via a dashboard. If you don't have that much data in your logs on a daily basis, you may simply want to query the files at runtime and generate the report fresh each time you view it. On the other hand, sites with a lot of traffic and thus larger log files may want to cache the output of each day to a separate file, so the data doesn't have to be computed. Obviously the type of approach you use to do that depends a lot on the scale you'll be operating at and how powerful your server hardware is.
Conclusion
Thanks to server logs and basic scripting, there's no reason you should ever have a situation where something's amiss on your site and you don't know about it. Proactive notifications of technical issues is a necessary thing in a world where Google crawls at an ever-faster rate, meaning that they could start pulling your rankings down thanks to site downtime or errors within a matter of hours.
Set up proper monitoring and make sure you're not caught short!
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!
Monday, 21 November 2016
Untitled Project
Watch on YouTube here: Untitled Project
Via https://www.youtube.com/channel/UCcZImr64sN4bR4BB_JA82vg/videos
Subscribe to:
Posts (Atom)