Atomic Media

SEO grew up, a lot of SEOs didn’t

Saturday, October 5th, 2024

When we were young, we all wanted to sit at the adult’s table. But we couldn’t, often, because of our behavior.

Growing up, we often thought we were cool, but we weren’t. Look back at your past. There are probably photos or things you did that you find questionable today.

This is exactly how we SEOs should be looking at our work. SEO has grown up a lot; we SEOs didn’t. 

Our mindset is a problem when tackling challenges like needing a strong brand or satisfying users the most.

What we do:

• Faking first-hand experience and looking for things we can leave out “because Google cannot measure that.”
• Buying (terrible) links (no one clicks), a practice that will be sketchy.

This is the wrong approach. 

What we should be doing: 

• Proving our experience, expertise, authoritativeness and trustworthiness.
• Creating the best content possible that is worth linking to or mentioning.

To finally grow up, we have to look into the mirror. We have to change.

If you are pressed on time, here’s a quick rundown of what I suggest you should start doing today:

• Embrace change as an opportunity to grow by making yourself familiar with it and turn obstacles on their head. 
• Do what is rational, logical and repeatable, study the Search Quality Rater Guidelines (SQRG) to understand what Google wants and invest in preparation, not prediction.
• Stop cheating or manipulating and start building something extraordinary. Start taking responsibility for your actions and stop blaming others for your failures. Save your energy to fight for positive outcomes, instead of wasting your energy on a negative candy rush.
• Communicate, understand and execute SEO the right way: As a growth engine, leading to limitless growth, not marginal improvements. Favor building over optimizing and fixing. Find your real competitive advantage and don’t buy into snake oil, cookie-cutter strategies or get-rich-quick schemes.

Disclaimer: This is not a purely tactical guide. My goal is to make you think. Motivate you to ask yourself tough questions and provide you with the necessary mindset to fill out the shoes of adolescence. If you get one good thought from this I’m more than happy.

Let’s get into the trenches: This is how we grow up!

Embrace change as an opportunity to grow

When we were children, we didn’t want to accept that change was inevitable. As we grow up, we realize it’s much easier to accept the world for what it is than to strive for what it should be. 

“You don’t get to dictate the course of events. And the paradoxical reward for accepting reality’s constraints is that they no longer feel so constraining.”

– Oliver Burkeman, “Four Thousand Weeks”

The same should apply to our SEO mindset.

Change can be old tactics no longer working (like putting white text on a white background; I only mention this because some website owners still do this today) or new things appearing on the horizon, like generative AI.

Many of our SEO playbooks are outdated, and forces are pulling on us to change. Here are two examples:

We often don’t have a crawl budget problem, but an indexing problem

Indexing is becoming harder. 

From what we know, the size of Google’s index seems to be more or less static (about 400 billion documents). Due to AI, we are:

• Producing more (good and bad) content.
• The bar for content quality has a new, higher baseline.

More content of a higher quality baseline vs. same index size = it’s harder to get indexed. 

Google is allergic to technical issues and low-quality content.

Google’s fierce changes in ecommerce searches

Google is under pressure to become a shopping engine.

• Around 50% of product searches start on Amazon, not Google.
• 61-74% of European shoppers search products on Amazon, not Google.

As a result, Google renovated the SERPs for commercial queries. 

In 2023, Google started to push a new feature: product grids.

Categorized as “Merchant Listings,” these grids appear in commercial searches at Position 1 more and more often. 

Based on my research for German SERPs, depending on the industry, product grids appear in position 1 between 15-45% of the time.

Thanks to some research done by Kevin Indig for the U.S., we know Google changed the rate of them appearing in Position 1 more often than in Position 3 around March 2024.

If your domain doesn’t play a role in product grids but your competitors do, you are in for a tough time.

The solution: Reframing problems as opportunities

Problems are what make life worth living. A problem can be reframed as a challenge or an opportunity. The cards are shuffled again, doors are opening up, and we are all new to these things. 

Here’s a practical reframing example from a talk by Carrie Rose I heard this year: 

• You are an ecommerce shop and lack authority in your niche
• The ranking domains are strong digital publishers, duh
• Instead of putting your head into the sand, you can put the obstacle on its head
• Don’t try to beat them, join them
• How? By giving them content they don’t have that is worth linking to
• Example: Internal searches showed an extreme increase in searches for White Vans
• They figured out faster than everyone else it was due to the series Squid Game on Netflix and pitched an article to news outlets
• Result: High-quality backlinks, traffic and additional revenue

Main takeaways: 

• Embrace change, make yourself familiar with new developments, tools and technology (start with an hour a week). 
• Strive to deliver “best in the world content,” not just “fine content.”
• Reframe problems into opportunities.

Double down on things that never change

On the one hand, we don’t want to change our behavior. On the other hand, our brain craves novelty, constantly looking for a dopamine hit. 

I know it’s very easy to be sucked in. New trends like AI chatbots, AI phones, AI toothbrushes. All good. But you have to avoid the shiny toy syndrome and embrace what will always matter.

If you focus on the things that never change, you can predict the future.

There lies great power in what you can’t measure

It will always pay off to do what is logical and rational. 

Google of 2004 is not Google of 2024. Just because Google cannot measure something right now doesn’t mean it cannot in the future. One example is having author bios and pages.

Due to the leak, we have our answers now, but having author bios and pages just because Google “wants” them didn’t make sense.

Things that cannot be measured are often underestimated and easily ignored. Fun fact: A metric is often more useful if it is harder to measure.

Prefer the repeatability in the present over the luck of the past

Luck is something positive that is not predictable. Repeatable means puzzle pieces falling into place like they used to. They are not the same.

In SEO, we love to look at what others did and you should. But focus on what is repeatable, not what was lucky.

You cannot follow in the footsteps of Amazon or HubSpot, as their operating conditions fundamentally changed. However, there are things worth learning from them.

• You can learn from HubSpot that it’s worth investing strategically in new channels/formats. 
• You can learn from Amazon that it’s worth investing in what customers will always value: Good prices and fast delivery.

Do what is repeatable; don’t try to emulate what was lucky.

Some more things that will never change

To draw on the Amazon example, Google will always need users to be satisfied with their search product. At least as long as this is their biggest revenue source.

One of your goals, in this case, should always be to have “best in the world content,” not good content.

In the SQRG, Google tells us exactly what they want. They have been doing so for years:

• Have author pages when it makes sense for your target audience.
• Don’t just copy the content of others, as high content quality is signaled by effort and originality.
• Auto-generated content (think AI or any programmatic plays) has more obstacles in its way to being rated Highest in terms of page quality.

Make no mistake: The next update is coming. The next disruption of the SERPs you operate in is coming as well.

It’s much better to invest in preparation over prediction. What we can’t see coming hits us the hardest.

Main takeaways:

• Do what is rational and logical, even if you or Google can’t measure it (yet).
• Copy what is repeatable today, not what was lucky yesterday.
• Study the SQRG to see what Google wants.
• Invest in preparation, not prediction.

---

Stop manipulating, start taking responsibility

Do you remember that when you were a kid, you often tried to manipulate and cheat in games?

In Germany, there is a game called “Mensch ärger dich nicht,” which roughly translates to “man, don’t be angry.”

I took chances to roll the dice again because someone “distracted me,” for example. This also happens often in golf and is known as a “mulligan,” so you get a pass and can take another shot.

At times, SEOs are like children who don’t want to accept the rules, cheat when no one is looking and love to blame others for our failures.

Examples of SEO manipulation practices

SEO is a breeding ground for manipulative tactics, and everyone knows the good old tales. Here are a few classics and recent ones:

• Terminated agencies updating the disallow file, so the work of the new agency gets tanked
• Content stealing, like the SEO heist
  • Purely AI-generated content is neither a competitive advantage nor a strategy. Even if it does work short-term, it is not something you can rely on in the future. We advocate investing in a long-term channel, but we are so greedy and in a hurry that we fall for get-rich-quick schemes like Gollum in his pursuit of his precious.
• Link buying, which sounds like exchanging drugs on a backyard
  • In some countries this is not just against Google’s official guidelines, but even problematic due to the law (in Germany we have the “Wettbewerbsrecht” for example).
• Affiliate link spam
  • Someone created thousands of domains, all just containing affiliate links to all kinds of ecommerce sites in Germany to earn a quick affiliate buck. This is literally a resource waste which makes the internet a huge garbage hole. Please stop.
  • Here’s an example domain that got spammed with these affiliate domains:

If you want to be taken seriously in SEO, don’t take shortcuts. Like Sonia Simone said, they take too long.

Why Google can’t tell us the truth

It’s easy to point fingers at Google. To call them liars and whatnot. Anyone with a clear mind has to understand that they cannot tell us exactly what is going on behind the curtain. If they did, at least a small group of SEO goblins would do everything in their power to ruin the game for everyone else.

SEO is the perfect example of the tragedy of commons. 

“Each man is locked into a system that compels him to increase his herd without limit – in a world that is limited. Ruin is the destination toward which all men rush, each pursuing his own best interest in a society that believes in the freedom of the commons.”

– Garrett Hardin, “The Tragedy of the Commons”

Some of us just can’t behave properly and break the system for everyone else.

Yet here we are, proud of engaging in our questionable get-rich-quick schemes, which no one would publicly endorse as employees of a real brand or big company.

Incentives beat intentions, unfortunately

Recently a guide on how to manipulate Reddit got a lot of traction. I understand what the intentions of the author were: Make things better. But I don’t agree with the approach and methods used to do it.

This is not an ad hominem case as this is not an individual problem.

Firstly, Google is blamed for creating incentives to spam Reddit. However, the article itself promoted incentives to do the same with the actual manual on a silver platter.

Publicly sharing the techniques to spam Reddit is promoting to spam Reddit even more, no matter how you put it. And it was never wise to fight fire with fire.

Also, people generally like Reddit and find the answers helpful. It’s no coincidence that they have these numbers: 

• There were 32 billion search queries in 2023 containing “reddit”.
• There were 3,102 billion search queries worldwide in 2023.
• = Reddit appears in 1% of all search queries.

I’m not saying there is no spam. But we don’t know the denominator here. If you seek a needle in a hay stack you will find one.

Secondly, Google is blamed 100% for the consequences of the Reddit spam. I don’t agree.

If you would look at the chain of responsibility through the eyes of several great philosophers, like Kant, Aristotle or Sartre, you would come to the conclusion that users taking advantage of the spam techniques are to blame first, then the platform (= Reddit) and then Google (= the middleman).

FYI: Others cheating doesn’t give you permission to do the same. Enabling and incentivizing these tactics is not a free ticket to cheat, either.

Look into the mirror: If spammers wouldn’t spam there would be no problem, so the root cause is our sometimes unbearable human nature.

The blame is (also) on us, not (just) the others

It’s more comfortable to blame others than to check on ourselves. 

“Google got worse” is one on the trendiest topics of 2023 and 2024. 

“Before Content Marketing was a thing, idiots did not publish content. You wouldn’t write encyclopedia articles without knowing anything. Now, we created a perverse incentive for any idiot to write about anything. We sit in a mountain of garbage.”

– Peep Laja, CEO, Wynter

What if Google didn’t get worse, but the ratio of good to bad content shifted?

If you fill a glass with more water (bad content) than wine (good content), the relative amount of wine in the glass decreases, even if the quality of the wine itself is good. It becomes harder to serve good content.

Google is responsible for their search results, but we are responsible for the mountains of garbage we produce.

A German study presumably claimed Google got worse. Their argument, only focusing on a small query subset in a specific niche, is insufficient to make such claims. It’s not even what they said but what people want to believe.

According to Statista, users are once again slightly more satisfied with Google search. And yes, according to the 286-pager on Google being a monopoly, Google tried to devaluate search quality to test the impact on revenue. But that test only lasted three months. 

No one can predict if there wouldn’t be a negative impact on revenue in the long term, which is all that matters.

Let’s assume for a moment that this was true: Google got worse and our domains were demoted in favor of some big digital publishers. How would the confirmation of this bias actually help me?

It doesn’t.

Yes, I can and should be vocal about it. But a lot of time and energy goes into being negative. Negativity is like a candy rush. It distracts us, so you need to avoid it.

Author Ryan Holiday hit the nail on the head with this quote: 

“In our own lives, we aren’t content to deal with things as they happen. We have to dive endlessly into what everything “means,” whether something is “fair” or not, what’s “behind” this or that and what everyone else is doing. Then we wonder why we don’t have the energy to actually deal with our problems.”

We need this energy wasted being negative in working on achieving positive outcomes, like crafting the best content out there or being the most helpful resource for our target audience.

Loopholes are risky short-term arbitrage opportunities, not long-term safe bets

A loophole is not a real competitive advantage, but a short-term arbitrage technique that brings a lot of risk with it. If revealed to the outside, there can be grave consequences. 

I loved this from Alex Birkett recently LinkedIn:

“Shortcuts in SEO often bring a sugar high, but they also come with a crash. […] If you treat it like a get-rich-quick scheme, you’ll need to ‘fix the plumbing’ later.”

Some things, like reputation, are not worth risiking, no matter how much there is to gain. Think of Sports Illustrated for example. 

Building a good reputation takes years. Setting it on fire can happen in seconds. 

Main takeaways: 

• Stop cheating or manipulating and start building something extraordinary.
• Start taking responsibility for your actions and stop blaming others for your failures.
• Save your energy to fight for positive outcomes, instead of wasting your energy on a negative candy rush.

Communicate und understand SEO as a growth engine, not as routine maintenance/polishing the edges

The grand finale: SEO has gotten a lot bigger.

Keeping SEO small and limited might be a way to avoid change. Could this be why many SEOs were reluctant to admit that Google uses user signals in their ranking algorithms?

Less change = SEO is smaller = more comfortable + less risky.

As outlined at the beginning, change is an opportunity. We walk into the fire of discomfort only to step out of it stronger, wiser and better.

SEO in 2024 is nothing like it was in 2004 or 2014. The fundamenta principles are the same, but we are driving a much different vehicle now with much more horsepower under the hood.

SEO is the wrong word for what we are actually doing

Digital publishers often get two-thirds or more of their traffic through SEO. A lot of companies rely heavily on organic traffic.

Some examples like Hardbacon had to file for bankruptcy as a result of the HCU and other updates. Some are or were on the cusp of it, like HouseFresh, Retro Dodo and Healthy Framework.

SEO stands for search engine optimization. But 70% or more of traffic share vs. other channels doesn’t sound like optimizing to me.

Optimization sounds like squeezing the last 5-10% out of what you already have. Limited and marginal.

The problem is that we all have different understandings of SEO, so we are not talking with each other but past each other.

To some, SEO means fixing mistakes/bugs. To others, myself included, SEO means (almost) limitless growth.

Fixing and optimizing is not enough:

• Fixing = keeping the bare minimum in place and unlocking the existing potential.
• Optimizing = using the full existing potential.
• Building = unlocking new growth = increasing the potential.

To visualize the idea further, see the following graphic:

• Fixing is like you are repairing a broken window in your one-room apartment. 
• Placing nicer furniture in that room is optimizing, you make it more appealing.
• The missing piece, then, is building new rooms if you actually want to live in a 10-room mansion.

Here are some things you need to communicate, understand and execute SEO as a growth engine:

• A business mindset, the right metrics and language a C-level member will understand.
• An updated SEO mindset, as ranking is not the end but a means to an end (= creating a high-value touchpoint with your target audience).
• The ability to play well with others, to make them embrace SEO, not despise it because you let them starve all the time by being greedy and only taking, never giving.
• Finding your (real) competitive advantage.

Finding your (real) competitive advantage

The last bullet point is especially important. It’s something that is missing quite often, from my experience.

Why should I buy from you? Why is your content the best in the world? The answer has to be the opposite of the “jumping the line” techniques criticized earlier.

The obvious question is how you get or find your real competitive advantage. It’s part of a good strategy. A strategy is always unique to a company. 

In “Good Strategy, Bad Strategy,” Richard Rumelt says the kernel of a strategy involves three pieces:

• Diagnosis.
• Guiding policy.
• Coherent actions.

To find a competitive advantage, you need to ask the right questions, like:

• What are my unique abilities/assets?
• Which of these abilities/assets matter to my target audience?
• Which of these remaining abilities/assets differentiate me from my competitors?

A SWOT analysis can be a helpful tool here. An alternative to getting started is to list all the abilities and assets that make your brand you and then answer the last two questions.

Examples of abilities or assets could be that you:

• Have a higher editorial output (quantity/quality).
• Employ SMEs in several fields, while others only cover one.
• Are uniquely fast in execution in relation to your company size.
• Have proprietary data that you can use for original research and data journalism.
• Are the manufacturer of a product (= having technical knowledge others don’t have).

Main takeaways:

• SEO is not just “a channel” it’s often the growth engine for most companies.
• Communicate it the right way: Limitless growth > marginal improvements.
• Favor building than optimizing and fixing.
• Come up with a unique competitive advantage that you can leverage in your SEO strategy.

We have work to do

I hope a lot of what I said is something you heard at least once already. But like Christian Morgenstern said (translation by me): 

“Sometimes you see something 100 or 1,000 times until you really see it for the first time.”

• We don’t want to change. Change is inevitable, though.
• We don’t always learn from the things that never change. But they let us predict the future.
• We like to skip the line and to go faster than is actually possible. Too fast often means fragile. You don’t want your SEO to be fragile, but to be unbreakable.
• We want it as easy as possible. Some things, however, are not easy. Like author James Clear said, “The cheat code is the work you’re avoiding.”

We have work to do.

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

The SEO impact of interstitials, before and after

Saturday, October 5th, 2024

Recently, an SEO client of mine lost 82% of desktop organic traffic to their homepage in one month.

Then, there was another 97% decline in desktop organic traffic in the second month. 

Why? 

The client launched a mobile and desktop pop-up for all United States visitors. Against my advice, the client hoped the conversions would outweigh the decline in organic traffic.

Here is a breakdown of the intrusive interstitial and its impact on organic traffic. 

The impact of before and after an interstitial on mobile and desktop

The homepage pop-up was related to a promotion and not triggered based on a time delay or scroll. 

Here is an example of a mobile pop-up: 

In the example above, the pop-up also negatively impacts the responsive design. 

Here is the example on desktop: 

In addition to the drop in organic traffic, the homepage lost 97% of its keyword rankings on desktop. 

And saw a 96% decline in mobile rankings for the homepage.

Most of the impact was on branded terms, indirectly impacting the reduction in PPC branded spend. 

It also negatively impacted our page experience and page speed. 

The website also saw a 10-second decline in page speed. 

Before the launch of the pop-up, the website averaged a 3-second load time.

And a 13-second load time after the launch of the pop-up. 

You can see the negative impact across page experience with the pop-up with a large number of layout shifts. 

The homepage saw a 97% decline in organic traffic cost value overall. 

There is no interstitial penalty or manual action

The intrusive interstitial effects rankings, not indexing. 

However, other areas of impact may occur, such as soft 404 errors, server speed and the site’s overall page experience. 

Remember, Google crawls from the U.S., so if those overlays are only present in European countries, Google will not detect them. 

The negative impact of interstitials is a demotion in search

When Google rolled out the Google mobile interstitial penalty on Jan. 10, 2017, it had a shockingly quiet impact. A few days later, SEO professionals started seeing pages lose 10 or more positions in rankings.  

When the Google page experience hit desktop sites in February 2022, intrusive interstitials were included as a signal. 

It’s important to remember that interstitials do not require manual action (at least not yet), but they still negatively impact your page experience. 

Google’s John Mueller stated in an SEO Office Hours on Jan. 22, 2021:

• “This is kind of like something where we would slightly demote the website in search. Sometimes, the tricky part is these slight demotions. It’s not the case that we’ll remove the site from search or move it to page 100. But if it’s relevant content, we may still show it on the first page of the search results, just not as highly as possible.”

---

4 examples of bad interstitials

1. Checking to see if you’re a spammy bot is bad

Gary Illyes from Google posted on LinkedIn, mentioning:

• “Checking if the site connection is secure interstitials you see on some sites, some of the time, is the last search-friendly thing you can do.”

Example: 

In short, there are other ways to avoid spammy bots from hitting your site.

If you detect a bot, Mueller recommended serving a 5xx status code to manage the robot detection interstitial. 

If you run a hosting service and have a “you might be a bot” interstitial, make sure it uses a non-200 HTTP result code (perhaps 503) and that it doesn’t have a noindex on it. Serving a noindex+200 will result in pages being dropped from search, if search engines see it. pic.twitter.com/LFGQcq2dzf

— John … (@JohnMu) January 17, 2022

2. Redirect to interstitial page 

If you show a user page A in the SERPs but redirect it to page B after the click and page B includes a pop-up or interstitial page, search engines will not detect content outside the redirected page. 

3. Only displays interstitial 

Here is an example of a pop-up display that covers the body copy: 

4. Store locator pop-ups

Store locators are really hit or miss on pop-ups. 

Here is an example of a good location locator pop-up: 

Instead of a pop-up, Xero opts for a banner ad. 

Once you click the “Change region” button, you’re brought to this option to select a region. 

4 examples of good interstitials

1. Displaying a cookies policy interstitial 

Legal interstitials are OK, just as long as search engines can index the content without doing anything special, like asking search engines to click on something to load the background content. 

This means GDPR and cookie policies are fine.

Here is an example of a cookies policy built with a pop-up window:

Here is another example of a cookies policy built with a banner at the bottom (my preferred method):

Mueller mentioned in Google Webmaster Office Hours that Google is trying to recognize these legal banners as separate from advertising banners.

2. Requesting age verification

Age verification classifies as a legal pop-up, so there are no issues with these.

Here is an example of an age pop-up done well:

3. Leveraging banners at the top and bottom

Here is an example of Samsung’s promotional ad banner:

4. Exit intent pop-ups that are time or scroll-based

If you’re going to launch a pop-up, I recommend using an exit intent pop-up based on time on the page (20 seconds or more) or scroll depth.

Stay tuned for more updates on the recovery of the interstitial pop-up

I’ll be updating this article as I get more data on the impact of removing the interstitial and recovery.

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Decoding Googlebot crawl stats data in Google Search Console

Friday, October 4th, 2024

Tucked away in the Settings section of Google Search Console is a report few SEO professionals discuss, but I like to monitor.

These reports are known as Crawl Stats.

Here, you’ll find an interesting set of metrics on Googlebot crawl activity. These metrics are especially useful for websites with thousands or millions of pages.

Googlebot ‘Crawl stats’ 

Long ago, Google Search Console had easy-to-find metrics on Googlebot crawl activity. Then, it seemingly disappeared.

In reality, it was tucked away in the Settings section.

How to access the crawl stats reports:

• Click on Settings at the bottom of the left navigation.
• Go to the Crawl stats section.
• Click Open report.

About the crawl stats data

As Googlebot crawls your site, Google tracks and reports on various aspects of Googlebot’s activity and reports on it in Google crawl stats.

This is where you’ll find high-level statistics about Google’s crawling history on your website. 

Google says this data is for advanced users

The Googlebot Crawl Stats data is not for the technical SEO rookies.

Google specifically says this data is aimed at “advanced users” with thousands of pages on their site, which may be why it’s located in such an unusual location, unseen by many in the SEO community.

One reason Google may perceive this as an advanced report is that so many things can influence these metrics, including network issues and cloud delivery services such as Akamai.

Who will find crawl stats most useful?

I find the Crawl Stats reports less of an “advanced” set of reports but something that’s more useful to enterprise SEOs without crawler monitoring tools such as Lumar and Botify. 

When doing SEO on an enterprise website with thousands or millions of pages, crawler optimization is a vital task, and crawler activity metrics provide important insight for defining action items.

Smaller sites likely do not need to worry too much about crawler activity because there is probably enough crawl budget allocated to your site to crawl at an appropriate pace. 

On the other hand, enterprise sites tend to have far more pages that need to be crawled, discovered, and/or refreshed than Google crawls their site each day.

For this reason, they must optimize for crawler activity, which is a tool to help guide next steps.

What to look for in this data

After years of reviewing this data across many sites, I have one primary rule:

• Do not spend a lot of time here unless you see fluctuations and correlations.

Often these reports are interesting but not actionable.

Example fluctuations that I tend to investigate: 

• HTML requests decreased (or spiked) at the same time Bytes of JavaScript downloaded increased (or spiked).
• Average response time increased (or spiked) at the same time the number of HTML requests went down (or sudden fall).
• Total download size increased (or spiked), but the number of HTML requests did not change.
• The percent of requests for discovery (to discover new URLs) increases and the percent of requests for refresh goes down; however, you did not launch new URLs on the site.

When to look at this crawl stats

Crawl stats are good to peruse (and log) at least once a month.

They are especially good to monitor after major releases, such as a platform migration or major redesign. This will help you quickly understand how Google is responding to your newly launched changes.

Remember: If you have a bot monitoring tool such as Lumar or Botify, then this data isn’t as useful as you’ll find in the bot monitoring data provided by these tools.

Caveats about the crawl stats data

Many things can influence Google’s crawl stats metrics beyond a normal dev release.

This means the SEO team, product manager(s) and developer(s) must think outside the box when evaluating the fluctuations.

You must consider what could have caused an increase or decrease in Googlebot crawl activity, not only in your release but also within the network and tech stack.

Changes to something such as Akamai could potentially impact this data. 

Log the crawl stats data in a spreadsheet

This is data I like to archive because Google reports such a small window of time.

A great example of this is a challenge I’m facing now with a client. What is reported in GSC right now looks like things are improving:

However, because I have metrics from six months ago, I can say that these metrics are 40% higher than they were six months ago.

While they’re trending down, they’re still worse than they were in the past. The client’s challenge is that development has no idea why this is happening (unfortunately, solving that problem is beyond the scope of this article).

You may think to just grab a screenshot. However, it makes it very hard to compare over time.

Notice there is no left axis in the chart. You really cannot tell what the lines reflect. (Note: Numbers do appear on the left/right axis when you are only viewing two metrics in the chart)

Instead, drop this data into a spreadsheet. Then, you have actual data that can be charted over time, calculated and used to compare with other metrics, such as visits. 

Having the historical data in one place is often useful when discussing major changes with development to show how much better the metrics were 4-6+ months ago. 

Remember, development likes hard, specific data, so charts with actual numbers on the left/right axis (or worse, no numbers on the x-axis at all) will be more useful to you than charts with varying numbers on the x-axis.

Remember, the reports boxes are paginated

Though the most important metrics you’ll need are likely visible in the default view, many of the report sections are paginated – and they’re easy to miss! 

---

Which metrics to monitor and why

Let’s get into the primary metrics to look (very quickly) each month, along with a few tips to take away action items from the data:

Total crawl requests

• View this report in Google Search Console (located in the top chart).
• Google definition: “The total number of crawl requests issued for URLs on your site, whether successful or not.”
• If this metric goes up or down, compare it with average response time and total download size (bytes).
  • An obvious reason for this metric could go up if you change a lot of code or launch a lot of new pages. However, that is by no means the only cause.

Total download size (byte)

• View this report in Google Search Console (located in the top chart).
• Google definition: “Total number of bytes downloaded from your site during crawling, for the specified time period.”
• If this metric goes up or down, compare it with average response time
• An obvious cause for this metric to increase is adding a lot of code across thousands of pages or launching a lot of new pages. However, that is by no means the only cause.

Average response time (ms)

• Google Search Console Report (located in the top chart).
• Google definition: “Average response time for all resources fetched from your site during the specified time period.”
• If this metric goes up or down, compare with with total crawl requests and total download size (bytes).

Crawl requests breakdown by response 

• View this report in Google Search Console (located below the top chart).
• Google definition: “This table shows the responses that Google received when crawling your site, grouped by response type, as a percentage of all crawl responses…”
• Common responses:
  • OK (200).
  • Moved permanently (302).
  • Server error (5xx).
  • Other client error (4xx).
  • Not found (404).
  • Not modified (304).
  • Page timeout.
  • Robots.txt not available.
  • Redirect error.

Crawl requests breakdown by file type

• View this report in Google Search Console. 
• Google definition: “The file type returned by the request. Percentage value for each type is the percentage of responses of that type, not the percentage of of bytes retrieved of that type.”
• Common responses:
  • JSON.
  • HTML.
  • JavaScript.
  • Image.
  • PDF.
  • CSS.
  • Syndication.
  • Other XML.
  • Video.
  • Other file type.
  • Unknown (failed requests).

Crawl requests breakdown by crawl purpose

• View this report in Google Search Console.
• Two purposes:
  • Refresh.
  • Discovery.
• This is an interesting metric for presentations; however, it only has a few useful use cases. For example:
  • If the percent of Googlebot activity that is for Discovery suddenly increases, but we’re not adding URLs to the site, then you have an action item to figure out what is being crawled that shouldn’t be crawled. 
  • If the percent of Googlebot activity that is for Refresh decreases significantly, but you didn’t remove pages from the site, then you have an action item to figure out why fewer existing pages are being crawled.

Crawl requests breakdown by Googlebot type

• View this report in Google Search Console.
• Google definition: “The type of user agent used to make the crawl request. Google has a number of user agents that crawl for different reasons and have different behaviors.”
• It’s an interesting metric, but not very useful. It just shows Google is still using their desktop crawler. Honestly, I usually ignore these metrics.

You can click into each metric for more data

Often when you present any SEO concern to product managers and developers, they often want to see example URLs. You can click on any of the metrics listed in this report and get example URLs.

An interesting metric to look at is “other file types” because it’s not clear what’s in the “other file types” category (often it’s font files).

The screenshot below shows the examples report for “other file type.” Every file listed is a font file (blurred out for confidentiality reasons).

In this report of examples, each row reflects one crawl request. This means if a page is crawled multiple times it could be listed more than once in the “examples.” 

As with all Google Search Console reports, this is a data sample and not every request from Googlebot.

Do you share these metrics with developers and product managers?

These metrics will typically generate one of two thoughts: 

• “There’s nothing to look at here.” 
• “What could have caused that?” 

In my experience, the answers to “what caused that” tend to require the input of product managers and/or developers.

When presenting the data and your questions about potential causes for issues, remember to clearly explain that these metrics are not user activity and solely represent Googlebot’s activity and experience on the website.

I find product managers and developers often get a bit confused when discussing this data, especially if it doesn’t match up with other metrics they have seen or facts they know about the site. 

By the way, this often happens for most Google Search Console data conversations.

If there are no Crawl Stats fluctuations or correlations to be concerned about, don’t bring it up to development, nor product management. It just becomes noise and prevents them from focusing on more critical metrics.

What’s next? 

Check out your crawl stats to make sure there are no spikes or correlations that are concerning. 

Then, determine how often you want to look at these and set up systems that prompt you to check these and other Google Search Console metrics in a systematic, analytical method each month.

While you check out your Googlebot Crawl Stats, I’ll write Part 4 in this series that will talk about how to know which URLs you should focus on for technical SEO improvements and in particular, Core Web Vitals metrics.

Dig deeper

This is the third article in a series recapping my SMX Advanced presentation on how to turn SEO metrics into action items. Below are links to the first two articles:

• Evaluating technical SEO data: Key segmentation approaches
• Key Google Search Console metrics to monitor every month

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

How AI is advancing advertising and changing consumer engagement

Friday, October 4th, 2024

Artificial intelligence (AI) is revolutionizing consumer engagement and transforming the landscape of search, according to James Murray of Microsoft Advertising, who spoke at SMX Advanced.

Here are the key points he raised in his presentation, exploring the paradigm shift in search technology and its implications for advertisers and marketers.

The expanded search box: A gateway to nuanced queries

Murray highlighted a seemingly small but significant change in search: the expansion of the search box: 

• “What have we done or what has changed? Well, a lot has changed. In fact, we think that we have completely shifted the paradigm of what search means and what is possible with search.
• “And I’d say one of the biggest and most fundamental changes through AI is that we’ve changed search from this to this. We’ve made the box bigger.”

From:

To:

This physical enlargement represents a fundamental shift in how users can interact with search engines.

No longer confined to short keyword phrases, users can now express complex, nuanced queries that more accurately reflect their true intentions.

For example, instead of searching for “cheap holiday Rome,” users can now input detailed requests like:

“Could you provide a Rome weekend itinerary for a family of 2 adults and 2 children aged 6 and 4. We want a mix of historical culture and fun activities, and the kids want to eat as much pizza as possible.”

This level of detail allows AI-powered search engines like Microsoft’s Copilot to understand context and provide more relevant, comprehensive responses.

Murray said:

• “Up until very recently, it has just not been possible in the sense that whilst we have always had the capability to put in these very large, very nuanced searches, the results that we get would often be confused or or just unintelligible.
• “Now, we’re in a situation where search through AI, through these new conversational capabilities, can actually not only understand a very complex query but can give us a meaningful reply that actually answers what we’re looking for.
• “So this is an example within the Microsoft ecosystem of Copilot. And so you can see that it has taken what I’ve given it, and it’s actually started to put together an itinerary, going to the Colosseum, going to see various bits of what Rome has to offer.”

AI’s Impact on search capabilities

Murray then explored the key advancement of integration of AI into search:

• Synthesis and summarization: AI can quickly process and synthesize information from multiple sources, providing concise, relevant answers to complex queries.

Murray explained:

• “If we look at that example which I just gave (trip to Rome), we start with this very dense, rich and sort of multifaceted, query on the left.
• “Copilot is able to simulate all of that information. But in the background, this would have taken us previously, in normal search, multiple iterations, multiple searches (see right) to pull together that same information. Whether it’s looking for family activities or what to see, history tours, the best place to get pizza in Rome, and so on.
• “And so what we’re seeing is, that from a consumer perspective, this does a number of things.
• “One is that it enables people to say what’s on their mind, but it enables them to get to answers quicker.”

Understanding true intent

AI-powered search engines aim to understand the underlying question behind a query, going beyond literal interpretation to grasp context and user intent.

Current search engines often provide broad, surface-level answers, but struggle to understand the deeper intent behind user queries.

For example, when someone searches for “Cancun weather in March,” the real underlying question might be about what to pack for a trip.

Murray said that search engines should evolve to better grasp and address the true, more nuanced intent behind users’ searches, rather than just offering basic information and related suggestions:

• “Wouldn’t it just be easier if we could say, ‘can you just pack, and make me a packing list and organize it by which room in the house I’m most likely to find those items?’
• “And to showcase how that can come to life and the reality of how we can get to the underlying question where we don’t have to read between the lines anymore because when people can actually ask what’s truly on their mind, and we can give them a sensible answer.
• “It allows that freedom to be able to most relevantly match to what people are actually asking for.

• “And so, really, what we’re seeing is this evolution of people being freed to ask the questions that they’ve always wanted to but, you know, perhaps haven’t felt comfortable of being able to do and the search engines being able to respond with much deeper, and much more nuanced and relevant answers.”

Evolution of AI search usage

Murray outlined three stages in the adoption of AI search tools:

1. Initial curiosity (2023): Users explored the capabilities of AI tools, testing their limits and discovering new opportunities.
2. Knowledge and action: As users became more familiar with AI tools, they began to find new use cases and develop greater proficiency.
3. AI in action (2024 and beyond): Users are becoming increasingly comfortable with AI tools and are extracting tangible value from them in their daily lives and work.

New possibilities for users and marketers

He then took us on the journey of how the advent of AI-powered search has opened up new avenues for users and marketers:

Expanded capabilities

Users can now perform tasks that were previously impossible or time-consuming, such as generating content, analyzing data and writing code.

• “Looking at the yellow bars here, we’re also seeing an incredible amount of new things that just previously weren’t possible.
• “So, coming up with new ideas and inspiration, summarizing content, which we’ve just seen, being able to actually generate, write and draft content. But not only that, analyzing data and writing code.
• “Actually, the list of these capabilities is expanding all the time. And we talked about that idea of value as we’ve moved from knowledge to action as the prompts that we put into some of these conversational experiences and the generative AI tools are getting more precise, we’re getting to faster task completion, and people are really starting to extract value.”

Increased efficiency

Studies show that users can complete tasks and reach decisions up to 50% faster using AI-powered search compared to traditional search methods.

• “Because we’re seeing that we’re able to ask those more complex questions, more nuanced questions, and we’re able to get to the answers quicker, what we [Microsoft] found is that on average, users take about half the time to reach a decision on Copilot compared to traditional search, and this matches up with other, tools as well.”

These are driving forward new ways of thinking and more efficient ways of being able to, you know, use engines for what we need them to do, which is ultimately get to the answers that I need, get that relevant information.

Enhanced advertising opportunities

Marketers can leverage these new search capabilities to create more targeted, relevant ad experiences for users in several ways.

• Ads have already been incorporated in AI Search: Because of AI tools rapidly growing in importance, the search tools are already AI powered.

Ads were built into the Copilot experience from day 1. Murray said if you’re optimizing for your core search experience, you’re also optimizing at the same time for the Copilot experience.

• Best Practices: Advertisers should focus on creativity, especially visually speaking ads, image extensions, product feeds and multimedia ad formats to improve click-through rates. 

What Microsoft are doing is taking the same ad creatives and content that advertisers create for search, and are replicating them within the conversational experience as well.

Murray highlighted the searcher has been transformed to deeper, more rich, engaging, conversations.

The more that advertisers can make ads visual, whether that’s using image extensions or some of the product feeds, or, new sort of formats like multimedia ads.

The more visual you can make them, the more they will likely stand out and then, the better they’re likely to get in terms of your click through rate.

Examples of marketers leveraging AI

• Content simplification: AI is regularly used for rewriting complex product summaries for easier understanding. It also useful for simplifying technical information from engineers.
• Tone adjustment: AI can help change the tone of ad copy to make it more approachable making serious content about cat trees more playful and fun. AI can generate multiple headline options, showcasing an understanding of context and even wordplay (e.g., cat puns).
• Strategic analysis: Marketers can input their brand and competitors into Copilot, to get insights. AI can then perform complex tasks like SWOT analysis. This saves time and provides external perspectives on strengths and weaknesses.
• Inclusive advertising: AI can help create more inclusive ad strategies such as analyzing customer challenges in car insurance. AI can suggest adjectives to address specific customer concerns and create a sense of safety.

Implications for marketers

Murray highlighted three key implications for marketers in this new AI-driven search landscape:

1. Quality and visibility

Ensuring high-quality, crawlable content is crucial for visibility in AI-powered search results. Tools like IndexNow can help keep content fresh and discoverable.

Murray suggests “revisiting and thinking about your SEO strategy, making sure that all of your content is crawlable.”

2. Hyper-personalization

The detailed nature of AI search queries allows for unprecedented levels of personalization in ad targeting and content delivery.

Murray clarifies here that this isn’t a result of data collection, saying that being able to be detailed with answering questions is “not because we are being invasive in terms of how much data we can stack and track on what people are, doing and and the demographics of who they are.”

• “This is simply by giving people the opportunity to explain what it is that they really want in detail. Through that conversation, the richer signals that we’re able to get, the sort of depth that we are able to understand means that when we serve an ad, it is absolutely the right ad at the right moment for the right person.”

3. Creativity and efficiency

AI tools can assist marketers in various tasks, from writing headlines to conducting SWOT analysis, fostering creativity and improving efficiency. It can even “showcase some blind spots that you might have in your creative campaign,” Murray said.

The future: Multimodal AI

Looking ahead, Murray touched on the concept of multimodal AI, which can process and generate content across various formats (text, image, video, audio).

This advancement promises even more exciting possibilities for search and content creation.

• “Where we’re moving to is where we can have any output from any input,” Murray said.

He shared an example – a picture of a rocket launch. By feeding just the image and asking AI about it, AI was able to tell us the specific event depicted in the image.

Unlock new levels of creativity, efficiency and productivity

The evolution of search in the age of AI represents a significant shift in how users interact with information and how marketers can reach their audience.

Murray encourage us to notice the small things and, quoting Liz Vassey (an American actress):

• “The rewards are inversely proportional.”

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Change Who Pays: New Google Ads process for client account transfers

Friday, October 4th, 2024

Google is introducing a new process – Change Who Pays – for transferring your client accounts to a new agency’s manager account.

Details. Google Ads sent an email sent to users about the change:

• “We’re happy to introduce a new process for transferring client accounts to a different paying manager. You may now initiate this change directly within your Ads manager account yourself, without needing to reach out to our support team. This process is called ‘Change Who Pays.”

What to do. You will need two details to complete a transfer:

• The Ads manager account ID of the new payer.
• The transfer date.

Google Ads support page. Transfer your client account to another agency

Why we care. As Georgi Zayakov put it on LinkedIn: “Finally, advertisers may now initiate this change directly within the Google Ads manager account themselves, without needing to reach out to the support team.”

The email. Here’s a screenshot of the email, shared by Zayakov:

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Google rolls out new AI-organized search results, AI Overview links

Friday, October 4th, 2024

Google is starting to roll out AI-organized search results today, as well as the new links and citations format in the Google AI Overviews.

AI-organized search results. The new AI-organized search results, as we covered in May, are now rolling out in the U.S. Google wrote:

• “We’re rolling out search results pages organized with AI in the U.S. — beginning with recipes and meal inspiration on mobile.”

This includes search results organized by web, including articles, videos, forums and more.

Google said in its testing, searchers “have found AI-organized search results pages more helpful. And with AI-organized search results pages, we’re bringing people more diverse content formats and sites, creating even more opportunities for content to be discovered.”

Here is what it looks like:

New links and citations in AI Overviews. In August, Google began testing new links and citations in the AI Overviews – and now that new format is going live.

Google told us they are now rolling out this new design globally to all countries where AI Overviews are available.

Google said in its tests they’ve “seen that this improved experience has driven an increase in traffic to supporting websites compared to the previous design, and people are finding it easier to visit sites that interest them.”

Here is what they may look like:

More AI news. Google also announced other news about AI and Search. Here are the highlights:

• AI Overview Ads. Ads are finally available for mobile users in the U.S. Dig deeper in Google AI Overview ads launch on mobile in U.S.
• New multimodal Lens features. Google is launching the ability to search with video in Lens and ask complex questions about moving images (available through Search Labs) and voice input in Lens, so you can ask questions out loud while snapping a photo (or recording video) to more easily search what you see.
• Shop easily with Lens. Google is now showing more helpful product information when using Lens for shopping, including a product’s price across retailers, reviews or where to buy. Dig deeper in Google Lens to add Shopping Ads this year.

Why we care. As Google and other search engines continue to innovate around AI and search, it is important to continue to watch the ongoing tests and launches of these features. AI Overviews have continued to evolve since it was announced as Search Generative Experience in May 2023. Plus, AI has been added to countless areas of Google Search and other products.

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Microsoft is about to change how Copilot ads appear and trigger

Thursday, October 3rd, 2024

Microsoft Advertising will soon change the way ads in Copilot appear to users, as well as how ads trigger. Advertisers are also about to get two new pilot features.

New Copilot ad experience. Copilot ads will appear below Copilot’s (organic) answers in a feature Microsoft is calling “ad voice.” This will include text referencing the conversation within Copilot and acts as a transitionary message from the advertiser. Fewer ad annotations and extensions will appear.

What it looks like. Here’s a screenshot Microsoft provided of the new look experience:

Timeline. This change will first come to copilot.microsoft.com this month and expand to more Copilot experiences – including Bing Search – “over time,” Microsoft said

New Microsoft advertising features. Microsoft will also launch two new generative AI features in pilot this month:

• Diagnostics. This “inspect campaign setup, assess account health, diagnose where attention is needed, and propose what to do next, all activated via simple conversational engagement.”
• Performance snapshot. You “can use natural language to ask Copilot in the Microsoft Advertising Platform to obtain an account or campaign-specific performance overview with a summary of key insights, trends, and anomalies.”

Why we care. Microsoft, citing internal data, said Copilot ads have “click through rates that are 69% stronger, and conversion rates are 76% higher when comparing lower funnel ad types to traditional search.” While these stats are quite vague and open to our interpretation of what they actually mean – ultimately, any changes Microsoft makes that could impact your advertising performance and ROI is one to watch and know about.

Microsoft’s blog post. Transforming audience engagement with generative AI.

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Case study: August 2024 Google core update and a recovery plan

Thursday, October 3rd, 2024

In August, Google rolled out one of its most significant updates of the year. Many SEOs believed it was a partial rollback of the controversial Helpful Content Update – a potential mea culpa from Google. 

While some sites have benefited from the changes, others have experienced sharp declines in traffic.

This case study dives into how the update impacted our client’s site, the key metrics affected and the strategies we’re implementing to recover lost visibility and boost SEO performance.

Diagnosing the drop

I recently attended the SEO Office Hours podcast, where a participant asked:

• “We haven’t been affected by the August update, but the indexing bug that happened around the same time has decreased our traffic. How do we solve this?”

My first thought was, what confidence! How sure are you that it was the indexing error if you are looking just at traffic? 

The error was resolved in a few days so how come there was no recovery of the traffic and rankings? 

Yes, we should ignore drops that happened in those two days during the bug, but if they continue after this, might it impact the core update? 

Diagnosing what actually happened is not always easy. We see it all the time. 

Sometimes, clients don’t know where to start. Other times, they simply don’t have time to investigate closely enough. The latter was true for our new client. 

Their excellent SEO team needed help auditing the website and a sounding board for solving any issues. 

After a Google core update rollout is completed, you could examine multiple metrics.

Like everything in SEO, these depend on several factors, such as your key performance indicators (KPIs), the type of website you manage and the countries you target. 

To diagnose the impact of the Google core update for our client, we used a simple, three-step approach:

• Understand what has happened sitewide.
• Use the sitewide indicators to segment the data for deeper analysis.
• Deep dive into the most affected pages and sections of the website.

Understand what happened sitewide

Our client has a large site. Digging into each page would take too much time. We needed to understand what happened overall to segment and prioritize. 

For this, we looked at: 

• Traffic and conversion trends.
• Overall content health.
• Link profile. 
• Technical SEO. 
• Wild card: persona-based sitewide signals.

Traffic and conversion trends

Two of the main KPIs for our client were traffic and conversions, which made it clear that our analysis needed to begin in these areas. 

We examined their Google Analytics 4 (GA4) and Google Search Console (GSC) data to assess organic trends and overall traffic patterns. 

This holistic approach is crucial because it establishes a baseline for our analysis. For instance, if we observe a significant drop in overall traffic, we can infer that the decline is likely not solely attributed to organic search issues.

We noticed a slight decrease in sessions overall and no impact on conversions. This was particularly noticeable in August (the timing of the Google core update), suggesting the update may have played a role and the culprit was likely organic.

The hypothesis was soon confirmed by digging into organic data in both GA4 and GSC.

Traffic was down significantly, both looking at GSC and GA4.

 Interestingly, just like overall traffic, the organic conversions were much more stable.

It was clear that the issue wasn’t the website’s relevance to its audience; those who visited it were converting. However, the overall number of users hitting the site had significantly decreased.

Before reviewing the rankings, we segmented the data by country to identify where the impact was most pronounced.

This prompted us to ask an essential question: do we care?

In our case, one of the countries most affected was India. While it would have been easy to dive into auditing this market for rankings, we recognized that the devil is in the details.

iOS is not the dominant operating system in India. In 2023, Android had a 95.17% market share, while iOS only had 3.98%.

Since our client’s solution only works with iOS, India is not the right market to focus on. Instead, we should concentrate on the U.S., which is their most relevant and larger market.

Overall content health

The second sitewide signal we explored initially is overall content health.

This gave us an initial understanding of possible issues with content at scale. We used Screaming Frog content analysis for this. 

To use any tool effectively, it’s crucial to identify what truly matters in the vast amount of data you’re presented with.

In this case, it would be easy to focus on the wrong metric, like readability.

While some pages may have readability challenges, this aligns with our technical audience’s expectations.

The more pressing issue wasn’t readability but the significant number of near duplicates. Most of the affected pages were in the template directory. This was a key insight for segmenting the data based on website structure.

Note: This analysis differs from evaluating what Google defines as helpful content. There are ways to evaluate content at scale for helpfulness, mainly using Google’s NLP API to pick out entities and analyze sentiment. Something we currently have in the pipeline.

Link profile 

The client had already said they wanted to enhance their internal linking at Stage 2 of the project.

For now, our focus was on understanding the situation, which we accomplished by using Screaming Frog.

Overall, the links looked healthy. We found valuable insights for improving our SEO strategies.

For example, when we examined the issue of internal outlinks with no anchor text, we discovered it originated from a specific template. While it’s not a top priority, it’s an easy fix at the template level.

Technical SEO 

Not every technical fix will affect rankings, no matter how interesting they may be. 

By this point in the analysis, I suspected the real issue was in the content and changes in the SERPs. 

I’ve seen many websites struggle after the latest Google updates, and we suspected that Reddit and AI Overviews were taking over the search results.

From what we’ve observed, AIO functions similarly to a featured snippet. Therefore, one area we should focus on is structured data markup.

This Screaming Frog analysis helped us identify missing structured data at the template level and reveal any obvious gaps.

Wild card: Persona-based sitewide signals

Creating well-defined personas can be beneficial for SEO and overall marketing.

The client expressed concern that their recent expansion of personas, as they shifted focus to a different market segment, may have diluted their website’s authority.

While I don’t believe proprietary metrics such as DA or AS are a ranking factor, the recent Google API leak did indicate that Google has its own understanding of domain authority. 

I can see why a company might think that expanding into a new market could make Google question its target audience. 

From a technical perspective, this concern aligns with how semantic SEO works – websites must focus on delivering closely related content.

To determine if our client faced challenges with this new vertical, we used a combination of tools: 

• Ahrefs to export the keywords and pages ranking in the U.S.
• A Python script that employed NLP to analyze the pages and assign a persona.

The findings were intriguing: most pages were assigned to both personas, indicating the content was closely aligned. 

This suggested that the website didn’t have the problem of straying too far from its core topic.

Use the sitewide indicators to segment the data for deeper analysis 

So far, everything suggests that the website remains valuable and that the traffic drop is not due to sitewide issues. 

• We already identified some challenges linked to specific page sections, such as the template directory.
• We know that most traffic comes from the blog section. 

However, with thousands of pages involved, it raises the question: where do you even start?

Using exports from various sources (GA4, GSC and the Ahrefs Top Pages report), we mapped the data to analyze the impact at the keyword level. 

This was the labor-intensive part of the process. It involved extensive data wrangling with index-match functions and pivot tables.

Although it was tricky, this approach enabled us to identify priority areas for deeper analysis, including:

• Creating a label based on % drop following the core update
  • For pages with a drop in session >= 20% or conversions >=20% – High
  • For pages with a drop in session >= 20% or conversions >=20% – Medium
  • For pages with a drop in session >= 20% or conversions >=20% – Low/None
• Key event + keyword volume 
  • Priority based on conversions >10 + volume

While not foolproof, this analysis provided us with an initial list of 42 pages to examine more closely.

It’s important to note that all the metrics involved depend on various factors, one of which is the publication date of the posts.

Newer blogs may not have built up enough authority, and they could have been initially boosted after publishing but later declined due to a lack of user signals, such as click behavior – something we suspect is related to NavBoost.

Conversely, knowing which pages were older helped us identify those that needed refreshing.

By default, Screaming Frog does not extract blog publish dates, so we needed to set up a custom extraction. This involved copying the selector from the date element using a tool like Google Chrome’s Inspect feature.

Once we had the publication dates alongside our list of pages, we could conduct a more in-depth analysis.

Deep dive into the most affected pages and sections of the website

This is where the process becomes manual. We examined the affected pages to identify any patterns, and it quickly became apparent that one existed.

Unfortunately, our suspicions were confirmed: the primary cause of the traffic drop was the change in the SERPs in the U.S. and U.K., driven by the rise of Reddit and the inclusion of AI Overviews.

Reddit emerged as a clear winner from the recent core update, significantly impacting many of the client’s keywords. 

Additionally, the expansion of AIO to cover even more queries – particularly in the client’s sector, where research indicates over 20% coverage – has worsened the situation.

AIO tends to impact informational queries more than commercial ones, which explains the loss of visibility in the blog sections.

Like many others, our client is losing valuable traffic due to Reddit and AIO, not because their website is any less helpful. Unfortunately, this is not an easy fix. 

Another area we examined closely was the templates directory. Although this section wasn’t driving much traffic or conversions, it was worth investigating for two reasons: 

• The client had invested significant time in deploying it.
• The initial analysis indicated issues with its implementation.

Similar to the priority blog pages, we noticed a pattern after reviewing a few template pages. 

These pages conflicted with search intent, and their content could be improved. 

While search intent is typically categorized into four main groups, I believe in taking a broader and more nuanced approach to understanding it.

Someone searching for a template typically wants to see several options to choose from. They intend to find a few templates, pick one they like and use it.

Google recognizes this. For instance, if I search for “Instagram template” or “Instagram templates,” the top results come from companies offering a collection of options.

Unfortunately, this wasn’t taken into full consideration before the pages were launched. 

Rather than optimizing collection pages that speak to the intent, the client optimized the individual template pages.

Additionally, the individual pages had very sparse information for users. Not that tweaking this would make a difference in situations where the intent is not aligned.

---

Fixing the ‘unfixable’

It was clear that our initial recommendations to the client needed a holistic approach, as there was no quick fix for the issues we identified.

To address these challenges, we recommended:

• A few quick wins and tests.
• Developing an improved strategy that considers these shifts.
• Closely measuring AIO and rethinking SEO KPIs.

Quick wins and tests 

Everyone seeks the ideal “low effort, high impact” solution. So, before tackling the complex issues, let’s focus on the quick wins. 

The analysis revealed some interesting tests we could try, specific areas for improvement and several technical recommendations. 

We suggested that the client optimize an initial content piece by reviewing and adjusting it to better align with E-E-A-T guidelines. 

I’ve also become increasingly interested in the concept of information gain as a factor in Google’s ranking system.

This makes sense, especially in the age of AI, where new data for training models is highly valuable. We will consider all of these elements when running the test.

The audit clearly showed that the template directory needed a rethink. With a few adjustments, it could become a valuable source of traffic and conversions. 

We first suggested revisiting the keyword strategy to better align with user intent, which would set the stage for optimizing those pages.

The audit also identified a few quick wins on the technical side, particularly in Schema markup. 

Rather than marking up everything on the site, being strategic about which Schema to implement is key. 

For instance, if you’re a SaaS product with excellent reviews, using Review Schema is essential. Our client had several opportunities to use this Schema to enhance their content.

Developing an improved strategy that considers these shifts 

The client needs to rethink their strategy, though the details aren’t fully clear yet, despite some initial ideas.

One option is focusing on user-generated content (UGC). Reddit thrives on UGC and Google seems to favor it. 

However, UGC is not enough for the client to compete with Reddit. It should be part of the strategy, but not the main tactic.

Another option is diversifying the keyword strategy. While expanding into less competitive search terms where Reddit and AI-generated content don’t dominate might help, it’s a short-term fix. 

Even though I’m skeptical about AI hype, it’s clear AIO isn’t going away and may actually become more prominent.

The real question is whether this shift is good or bad for users and companies. Ultimately, any strategy must account for how search is evolving. 

As AI plays a bigger role in the early stages of the user journey, our client – and others – will need to rethink the kind of content they produce.

Whether AIO guides searchers or they directly use chatbots like ChatGPT, many will only reach your website during the decision stage. This changes how we approach strategy and SEO overall. 

While targeting commercial keywords remains important, the focus will shift more toward building a brand and creating diverse content, like video, to rank in video SERPs and on YouTube. 

Some of the content that’s lost rankings for our client could perform well in video format.

I don’t see these changes as negative. Relying solely on Google was never a good strategy. 

Now, strategies must go beyond SEO and integrate with broader digital marketing efforts. As someone who values full-stack marketers, I believe this shift opens up exciting opportunities.

Dig deeper: Search everywhere optimization: 7 platforms SEOs need to optimize for beyond Google

Closely measuring AIO and rethinking SEO KPIs

Our client already ranks in AIO for some terms, but we can’t measure the impact on CTR since it’s not trackable in GSC or GA4 – and likely never will be.

According to Liz Reid, the Head of Google Search:

• “And we see that the links included in AI Overviews get more clicks than if the page had appeared as a traditional web listing for that query.”

This may or may not be true overall. We know our client was losing clicks and that AIO clicks are, in any case, a different user experience. 

Measuring AIO clicks has been a gray area, but it’s crucial to address. 

We recommended expanding the client’s SEO tech stack to include platforms that help with tracking. 

While they currently use Ahrefs, which doesn’t yet support AIO tracking (though it likely will soon), switching to a tool like Semrush would be a logistical challenge for such a large site. Instead, we suggested supplementing with ZipTie.

However, measurement goes beyond tools – it’s about rethinking KPIs. 

While traffic, conversions and revenue remain important, brand KPIs and micro conversions are needed now more than ever.  

With search and data privacy changes, tracking every click isn’t feasible. 

Building brand resilience and focusing on smaller touchpoints, like newsletter signups and engagement rates, can help fill the gap. As the market evolves, so must our KPIs.

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

19 keyword mining tools and strategies for PPC campaign growth

Thursday, October 3rd, 2024

Are you having trouble finding relevant keywords for your paid search campaigns? It’s time to move beyond only using the Google Keyword Planner and uncover fresh, relevant keywords.

Traditional keyword research tools frequently lack depth, offering only limited insights into the complexities of user search behavior. As a result, PPC managers often struggle to uncover new keyword opportunities.

This article explores 19 tools and strategies for keyword mining to help you discover new search terms and enhance your campaigns. 

By the end, you’ll have an arsenal of techniques to optimize your campaigns and gain a competitive edge in your growth strategy.

Finding new opportunities beyond Google’s Keyword Planner 

Although the Google Keyword Planner is valuable, exploring alternative research methods is crucial for discovering a wider array of relevant searches. 

Increasing the volume of relevant searches is an effective strategy to expand your campaigns and business. A higher volume of qualified and converting traffic can lead to more sales or leads.

Here are some effective tools and strategies you can use when looking to expand your campaigns.

1. Dynamic search ads (DSAs)

These ads leverage website content to automatically generate ads and the associated keywords.

By analyzing the search terms that trigger your DSAs, you can identify valuable keywords to incorporate into your standard search campaigns.

Fellow PPC expert Nils Rooijmans shared on LinkedIn that he likes using DSA campaigns, moving successful terms to revenue-generating campaigns. He also uses ChatGPT for new keyword ideas.

2. Google Search Console

GSC lets you see the search queries that led to clicks on your website. You can use these insights to identify new keywords and optimize your ads accordingly.

Dig deeper: Using search query reports to optimize PPC campaigns

3. Related searches on Google (People also search for)

When you enter a search term into Google, scroll down to the bottom of the page for a list of related searches. These can give you ideas for new keywords or variations of existing ones.

In the example above, I was searching for dance costumes for my daughter and came across some excellent search term suggestions. 

If I were managing a paid search account focused on costumes in these categories, these terms would be a fantastic addition to enhance the account’s effectiveness.

4. Competitor websites

Take inspiration from your competitors’ websites by analyzing their content and identifying relevant keywords they may be using in their paid search campaigns.

5. Google Autocomplete

Pay attention to the suggestions that appear as you type in search queries. These autocomplete options often reveal popular search terms and long-tail variations.

6. Microsoft Advertising

While Google Ads may be the dominant platform, Microsoft Ads search term reports can provide new keyword ideas.

Analyze the search terms that trigger your ads on this platform to discover additional keyword opportunities that can be added to your campaigns in Google Ads.

You can also add terms found in the Google Search Query Report and add them into your Microsoft campaigns.

PPC specialist Petya Yildizgoren validated this tactic on LinkedIn and introduced Amazon Ads as another source for new keywords.

7. Social monitoring tools

Monitor social media platforms to identify industry-specific trends, popular topics and commonly used phrases. 

These insights can help you uncover relevant keywords that align with your target audience’s interests. 

For example, while managing an account, I noticed in the comments that customers favored products made in the U.S.

I added related keywords to key terms, which helped drive additional non-brand traffic for the advertiser.

8. Customer conversations

Listen closely to the conversations your customers have with your support team or sales representatives.

Capture the keywords and phrases they use to describe your products or services.

For example, while working with a client offering a call center service, I discovered that leads used “answering service” instead of “call center” when monitoring their calls.

9. Website survey

Conduct surveys on your website to collect direct feedback from your customers. Focus on the open-ended responses to identify the terms used to describe respondents’ needs and goals.

Many survey tools allow users to analyze and generate reports based on the words in surveys at scale.

10. Social comments

Analyze the comments on your social media posts, as the language used by your audience can provide valuable insights into their search intent.

---

11. Broad match type

Develop campaigns using broad match type terms without negative keywords to find new keywords from search term reports. 

A valuable tip is to separate these campaigns to ensure your entire budget isn’t consumed by irrelevant keywords.

Dig deeper: Broad match keywords: When to test and when to avoid

12. Keyword research tools

Don’t limit yourself to one keyword research tool. 

Consider using a combination of tools, such as Semrush, Ahrefs or Moz Keyword Explorer, to uncover different perspectives and insights on potential keywords for your campaigns.

13. Google Trends

Use Google Trends to identify popular search terms and their search volume over time. 

This tool can also show related queries and geographic data, giving you a more comprehensive understanding of user behavior.

14. Analytics data

Dive into your website analytics data to see which organic keywords are driving traffic to your site. This can give you a good starting point for identifying relevant keywords for your paid search campaigns.

Don’t rely solely on Google Analytics. Explore other tools like Adobe Analytics, too.

15. Customer persona research

Create customer personas based on demographic, psychographic and behavioral factors.

Use these personas to guide your search for new keywords that align with their interests and behaviors.

16. Customer demographics analysis

Many customers will search in their native language and still make purchases from a website in the local language. In the U.S., for example, you’ll notice many searches in Spanish. 

Bilingual customers are often tolerant of conducting business across languages. Don’t let the absence of a fully translated website prevent you from bidding on keywords in your native language. 

I worked with a healthcare client who added Cantonese and Punjabi keywords and ads, which performed quite well despite the website being in English.

17. Customer reviews

Pay attention to the language used in customer reviews to identify commonly used keywords and phrases related to your products or services. 

This can provide additional insights into your customers’ search behavior and help you uncover new keyword opportunities.

18. Collaborate with your team

Don’t overlook the valuable input from your team members, whether it’s colleagues in different departments or external agencies. They may have unique perspectives and ideas for relevant keywords that you may not have considered.

Dig deeper: How to maximize PPC and SEO data with co-optimization audits

19. Stay up-to-date

Regularly review industry news, updates and  trends to stay current on any emerging topics or terms that could potentially be relevant for your campaigns. 

This can help you stay ahead of the competition and continue to refine your keyword strategy.

Dig deeper: Top 5 ways to stay up to date on paid search trends

Leveling up your PPC query mining strategy

There are many techniques and tools available to conduct keyword research and discover new opportunities for paid search campaigns. 

Additional strategies include:

• Semantic keyword research, which explores related terms and synonyms to expand your keyword list.
• Topic modeling, which uses advanced tools to identify underlying themes in your website content or customer feedback. 
• Seasonal keyword research, which allows you to adjust your strategy for trends like “Black Friday” or “Cyber Monday” and capture additional traffic during peak times.

Suggestions from the paid search community include using Performance Max search categories (as noted by Guillermo Pareja) and tools like Answer the Public for question-based queries (recommended by Neelam Shahzadi).

Many experts also suggest leveraging AI tools like ChatGPT to generate new keyword ideas by seeding prompts with well-performing terms.

Traditional keyword research methods are valuable, but they only scratch the surface of potential opportunities.

By combining these techniques with AI tools, you can uncover hidden gems and optimize your campaigns for maximum ROI.

Whether you’re looking to expand or refine your campaigns, these strategies will help you create a comprehensive keyword approach that goes beyond the limitations of traditional planners and drives business growth.

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Amazon Advertising guide: How to reach customers and sell more

Thursday, October 3rd, 2024

Amazon Ads has become a powerful platform for businesses of all sizes, both on and off the Amazon marketplace.

This article explores the various types of Amazon Ads, their features and how they can benefit different businesses.

Whether you’re a small local business or a large brand, understanding Amazon’s advertising options can help you reach new customers and grow your business.

Understanding Amazon advertising today

When most people think of Amazon advertising, they typically envision promoting products sold on Amazon through Seller Central or Vendor Central. 

However, Amazon’s advertising ecosystem has expanded significantly, now offering options for service and local businesses that don’t sell physical products on the Amazon marketplace.

The scope and capabilities of Amazon ads have been growing at an exponential rate. 

Amazon has been actively developing the functionality and products in its advertising suite, enhancing targeting options, expanding eligibility criteria and introducing new features like audience targeting and customer retargeting.

Amazon advertising can be broadly divided into two main categories:

• Sponsored ads
  • This includes:
    • Sponsored Products.
    • Sponsored Brands. 
    • Sponsored Display. 
  • These ad types are specifically designed for products sold through Vendor Central, Seller Central and Kindle Direct Publishing.
• Display ads
  • This category comprises two main products:
    • Amazon DSP (demand-side platform).
    • Sponsored Display for All Business (formerly called Non-Endemic Sponsored Display).
  • Both of these display ad types are available for businesses not selling goods on the marketplace. 
  • These ads appear not only on Amazon-owned sites like Amazon.com, Twitch and IMDb but also on other sites with which Amazon has direct partnerships.

As we delve deeper into each ad type, we’ll explore their unique features, targeting options and best practices to help you navigate the diverse landscape of Amazon advertising.

Sponsored ads

Amazon has been aggressively expanding the targeting options and products available within its Sponsored Ads suite. Advertisers on Amazon’s marketplace have more options than ever.

Let’s walk through the primary ad types used within Amazon’s self-managed ad service for its marketplaces worldwide. 

Sponsored Products

Available platforms:

• Vendor Central. 
• Seller Central. 
• Kindle Direct Publishing.

Eligibility requirements:

• Products must be in one or more eligible categories, have inventory available and be eligible for the Featured Offer to advertise. 

Targeting available:

• Keyword. 
• Category. 
• Product.

Placement locations:

• Amazon Marketplace search results. 
• Product detail pages.

Creative required:

• All creative is derived from the product listing on Amazon’s marketplace. 

Sponsored Products should be the starting point for almost every Amazon ad strategy for products sold through the Amazon marketplace. It is not uncommon for this ad type to make up as much as 80% of an advertiser’s budget.

One of the primary placements for this ad type is the search engine results page, which can provide the most significant lift in sales and organic ranking.

Using keyword targeting for Sponsored Products is generally one of the primary ways advertisers increase conversions for a keyword phrase to improve the organic ranking of their products.

Advertising on Amazon directly impacts organic results, especially for keyword-driven searches.

Sponsored Products have minimal availability for creativity, so it’s essential to optimize your product’s title, structured data and primary image for both organic search and ad conversion.

Sponsored Products campaigns have no monthly or upfront fees and are always on a cost-per-click basis.

Your ads may be displayed at the top of, alongside, or within shopping results and on product pages, on desktop and mobile. Ads are shown to shoppers based on relevant keywords or products.

At this time, adult products, political items, used products, refurbished products and products in closed categories are not eligible for advertising.

Sponsored Brand Ads

Available platforms:

• Vendor Central.
• Seller Central.
• Kindle Direct Publishing (only when you have at least three unique titles under a single pen name).

Eligibility requirements:

• Brand Registry is required to run this ad type. However, you do not need the Buy Box for it to run. 

Targeting available:

• Keyword.
• Category. 
• Product.

Placement locations:

• Amazon Marketplace search results. 
• Product detail pages.

Creative required:

• Creative must consist of:
  • 50-character headline.
  • Brand logo.
  • Custom image.
  • Product detail page components.
• Creative will show custom image on left with products on right that images and titles pull from detail page. 

Sponsored Brands can direct customers to either product detail pages or store pages. 

One advantage of these ads is that you don’t need to be the merchant holding the Buy Box for them to run, making them especially beneficial for brands that lack exclusivity on featured offers.

Typically, Sponsored Brand Ads account for 10-20% of a brand’s marketplace advertising budget and operate on a cost-per-click basis.

Originally, you could only run Sponsored Brand Ads if you had at least three products. Now, this requirement has been removed, allowing brands with just one product to use these ads.

Sponsored Brand videos are an effective way to capture visibility in search results and are among the most popular methods for utilizing Sponsored Brands. These videos yield a high return, particularly for products that fold, transform or are new to the category.

Sponsored Brands are available for vendors, book vendors, Kindle Direct Publishing (KDP) authors, agencies and professional sellers enrolled in Amazon Brand Registry.

Dig deeper: Amazon Ads: Maximizing Sponsored Brands campaigns

Sponsored Display (Marketplace)

Available platforms:

• Vendor Central. 
• Seller Central.

Eligibility requirements:

• Brand Registry is required to run this ad type. However, you do not need the Buy Box for it to run. 

Targeting available:

• Category. 
• Product. 
• Audiences.

Placement locations:

• Detail page. 
• Non-native placement in search results.

Creative required:

• Creative must consist of:
  • 50-character headline.
  • Brand logo. 
  • Lifestyle image.
  • Product detail page components.

Sponsored Display is the newest type of sponsored product ad. 

Over the last two years, the ability to target and retarget with sponsored display has drastically increased. This makes it a powerful tool to use Amazon audiences and retarget views and purchases of specific ASINs for your products and competitors.

This ad type usually accounts for 5-15% of the overall ad budget. However, there can be a greater variance for this ad type, with some brands using it heavily and others not. 

Sponsored Display supports cost-per-click (CPC) and cost-per-thousand viewable impression (vCPM) pricing options without minimum campaign spend requirements. 

Ad placement can be found across the Amazon store and thousands of apps and websites.

Sponsored Display allows you to target Amazon’s first-party audiences and customers who have viewed or purchased specific products or product categories. This capability makes it particularly effective for retargeting customers.

Keep an eye on ‘Rufus’

Amazon has also rolled out an AI shopping assistant called Rufus, but there is much debate on its usefulness and customer adoption. 

At the time of this writing, Amazon specifically states that there is no reporting to show Rufus’s efficacy or even control placement.

Sponsored product ads: Best practices

• Maximize performance and budget: Maximize performance and budget with Sponsored Ad products before moving to the display and programmatic ads below.
• ROI and performance: Sponsored Ads tend to be much better for ROI and performance-based campaigns. An exception would be if you’re looking to create demand for new-to-category products.
• Funnel positioning: Sponsored Ads are generally at the bottom of the funnel and are designed to capture existing demand rather than create it. However, there are use cases for Sponsored Brands and Sponsored Display to help drive new customer awareness and acquisition.
• Full funnel approach: Amazon has increased its focus on pushing advertisers to a fuller-funnel approach. They are also working to expand Amazon Marketing Cloud (AMC) to include Sponsored Ads, giving advertisers more insight into the overall customer journey.

Dig deeper: Amazon Ads: How to boost efficiency and reduce wasted spend

---

Display and programmatic advertising

Until recently, the only option for display ads on Amazon was Amazon DSP. 

While this has been a widely used tool for both companies with and without physical products, the high minimum spend was a deterrent for most small brands. 

Amazon recently brought Sponsored Display for All Businesses (formerly called Non-Endemic Sponsored Display) out of beta as an option for brands with smaller budgets. 

Amazon DSP

Amazon DSP has been a key player in programmatic advertising for many years. It offers an extensive suite of tools for analyzing customer conversion and targeting first-party audiences.

While it can effectively grow brand awareness, it tends to be less ROI-focused than Sponsored Ads.

One major drawback is the minimum ad spend requirement, which ranges from $10,000 to $35,000 per month, making it prohibitive for many small businesses.

Additionally, managed DSP services with Amazon require a minimum spend of $50,000 per month. 

To assist advertisers in navigating customer journeys, Amazon provides tools like Amazon Marketing Cloud to enhance understanding and optimize strategies.

Sponsored Display for all businesses (SDFAB)

Sponsored Display for all businesses is a new product designed for small and local service-based businesses.

It allows them to leverage Amazon’s audience targeting capabilities and the views and purchases of specific products for their display ads.

Advertisers can target Amazon’s first-party audiences or focus on customers based on product views and purchases.

SDFAB does not allow targeting at the ASIN or product level; instead, targeting is available at the sub-category level.

In some sub-categories, you can further refine your targeting by factors such as age ranges, brand, reviews or pricing.

Additionally, you can target by location and optimize for specific goals, including page views, page visits or the newly launched lead form.

This service does not require a minimum spend, allowing businesses of all sizes to use the power of Amazon’s customer data to reach highly targeted potential audiences.

It is important to note that as this is Amazon’s newest ad product, it still has some growing pains. This is something to watch as the abilities and features of this ad product continues to grow.

Dig deeper: Amazon Ads for lead generation: What advertisers need to know

Mastering Amazon Ads: Strategies for sellers and non-sellers

Amazon advertising is continually evolving, enhancing its capacity to convert and expanding the channels through which businesses can reach customers.

With its in-depth understanding of customer behavior, Amazon provides advertisers with valuable data insights that facilitate a comprehensive approach to their advertising strategies.

This evolution allows businesses to effectively engage with their target audience and optimize their marketing efforts for greater success.

Dig deeper: 5 reasons Amazon Ads is better than Google Ads for ecommerce

Courtesy of Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing