Robots meta tag missing snippet and preview directives

This issue reads the <meta name="robots"> tag a page declares and checks that it says what its author meant it to say. It covers four things, in order of how much damage each does:

1. Directives that suppress the page — noindex, none, max-snippet:0. These keep the page out of search results or strip its snippet entirely. That is sometimes deliberate, so the finding names the directive rather than assuming a mistake. (A page-wide nofollow is reported separately, as "Page tells search engines not to follow its links", so it is not counted here as well.)
2. Directives no search engine defines — no-index, nofollows, max-image-preview:huge, or two directives separated by a space instead of a comma. A crawler discards these silently, so the tag reads as deliberate in the markup and has no effect at all.
3. Directives that contradict each other — index, noindex. Search engines obey the more restrictive half, so the permissive one does nothing.
4. The preview directives — max-snippet, max-image-preview, max-video-preview. A tag that omits them leaves the page on the platform defaults: a truncated text snippet and a small image thumbnail.

A passing implementation declares all three preview directives with values their directive accepts, and nothing that suppresses the page:

<meta
  name="robots"
  content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
/>

A page with no <meta name="robots"> tag at all is not a finding. No tag means the default, index, follow, which is the correct configuration for an ordinary page — reporting its absence would ask every site to add markup that changes nothing, on every page. The check applies to pages that have made a robots declaration, because a declaration has asked to be read.

The meta robots tag directly impacts how your pages appear in search engine results pages (SERPs):

• Rich snippets: The max-snippet directive controls how much text content can appear in search snippets. Setting it to -1 allows search engines to show as much as they deem relevant.

• Visual appeal: The max-image-preview:large directive enables large thumbnail images next to your search results, which can significantly increase click-through rates.

• Video visibility: The max-video-preview directive controls whether video content can show preview clips in search results, helping video content stand out.

• CTR improvement: Properly configured meta robots tags can make your search listings more visually appealing and informative, leading to higher click-through rates.

Resolving this issue improves your SEO health score by ensuring your content has the best possible presentation in search results, which can lead to increased organic traffic.

Start from what the finding says about your tag — it names the directives it found and why each was reported.

1. If a directive suppresses the page (noindex, none, max-snippet:0): decide whether that is intended. A staging copy, a thank-you page or a login screen is meant to be hidden, and the finding can be dismissed. A page meant to rank should have the directive removed — this is usually the single reason it does not appear in search.

2. If a directive was not recognised: fix the spelling or the value. The usual causes are no-index for noindex, a directive written without its value, and a value outside the accepted set:

   Directive	Accepts
   max-snippet	a number, where -1 means no limit
   max-image-preview	none, standard or large
   max-video-preview	a number of seconds, where -1 means no limit
   unavailable_after	a date in RFC 822, RFC 850 or ISO 8601 form

   Directives are separated by commas, not spaces: content="index follow" is one unrecognised token, content="index, follow" is two directives.

3. If two directives contradict each other (index, noindex): remove the one you did not intend. Search engines obey the more restrictive half, so the page behaves as if only that half were there. A contradiction usually means two templates each appended a directive; find the one that should not run on this page type.

4. If a preview directive is unset: add it. The recommended set for an indexable page is

   <meta
     name="robots"
     content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
   />

   You can narrow these deliberately — max-snippet:50 to cap snippet length, max-image-preview:standard for pages with sensitive imagery — and any accepted value passes the check. What it reports is a directive left unset, not a value you chose.

5. Verify implementation: use View Page Source or browser developer tools to confirm the tag appears in the rendered HTML, then Google Search Console's URL Inspection tool to see how Google interprets it.

Note: a page with no <meta name="robots"> tag is not reported. No tag means index, follow, which is correct for an ordinary page — the preview directives above are still worth adding, but their absence is not treated as a defect.

Example 1: Correct Implementation

Scenario: A page that declares the full preview directive set.

Correct State (Passes):

<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta
    name="robots"
    content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

Example 2: No Meta Robots Tag

Scenario: Page has no meta robots tag.

State (Passes):

<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>Page Title</title>
</head>

Why it passes: No tag means the default, index, follow. The page is indexable and followable, which is the correct configuration for an ordinary page, so there is nothing to fix. Adding the tag with the preview directives is still worthwhile — see Example 3 — but its absence is not reported.

Example 3: Incomplete Meta Robots Tag

Scenario: The tag is present but sets none of the preview directives.

Problematic State (Fails):

<head>
  <meta name="robots" content="index, follow" />
  <title>Page Title</title>
</head>

Why it fails: The tag declares only the defaults it would already have. max-snippet, max-image-preview and max-video-preview are unset, so the page is left with a truncated snippet and a small thumbnail in results.

Corrected State (Passes):

<head>
  <meta
    name="robots"
    content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

Example 4: A Directive No Search Engine Defines

Scenario: The tag looks deliberate and does nothing.

Problematic State (Fails):

<head>
  <meta
    name="robots"
    content="no-index, max-image-preview:huge, max-snippet:-1"
  />
  <title>Page Title</title>
</head>

Why it fails: no-index is not a directive — the correct spelling is noindex, and a crawler discards the hyphenated form in silence, so a page the author believes is hidden from search is fully indexed. max-image-preview:huge is rejected the same way: the directive exists, but it accepts only none, standard or large, so the page falls back to the default thumbnail size.

Corrected State (Passes) — assuming the page IS meant to be indexed:

<head>
  <meta
    name="robots"
    content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

Example 5: Directives That Suppress the Page

Scenario: Page carries directives that keep it out of search results.

Problematic State (Fails):

<head>
  <meta name="robots" content="noindex, nofollow, max-snippet:0" />
  <title>Page Title</title>
</head>

Why it fails: noindex keeps the page out of the index entirely and max-snippet:0 forbids any text snippet. (nofollow, which stops the page's links being followed, is reported separately as "Page tells search engines not to follow its links".) If the page is a staging copy, a thank-you page or a login screen this is correct and the finding can be dismissed. If it is a page meant to rank, this is the reason it does not.

Corrected State (Passes):

<head>
  <meta
    name="robots"
    content="max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

Example 6: Contradictory Directives

Scenario: A template that concatenates directives from two places.

Problematic State (Fails):

<head>
  <meta name="robots" content="index, follow, noindex" />
  <title>Page Title</title>
</head>

Why it fails: index and noindex cannot both apply. Search engines obey the more restrictive half, so the page is deindexed despite the index that precedes it — usually a sign that one template appended a directive the other had already set.

Corrected State (Passes):

<head>
  <meta
    name="robots"
    content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1"
  />
  <title>Page Title</title>
</head>

PixyScan performs the following checks to detect this issue:

1. HTML parsing: The crawler extracts the <head> section from the page's HTML content.

2. Meta tag search: It looks for a <meta> tag with the attribute name="robots". If there is no such tag, the page passes and nothing further runs — no tag means the default index, follow, which is correct for an ordinary page.

3. Directive parsing: The content attribute is split on commas, and each token is split on its first colon into a directive name and a value. Splitting on the first colon only is what lets unavailable_after: 2026-12-31T23:59:59Z parse as one directive rather than three unrecognised ones. Names and values are compared case-insensitively and surrounding whitespace is ignored.

4. Directive validation: Each parsed directive is checked against the vocabulary the search engines define.

   Directives that stand alone, with no value: all, index, noindex, follow, nofollow, none, noarchive, nosnippet, noimageindex, nocache, notranslate, nositelinkssearchbox, nopagereadaloud, indexifembedded, noodp, noydir.

   Directives that take a value:

   Directive	Accepts
   max-snippet	a number, where -1 means no limit
   max-image-preview	none, standard or large
   max-video-preview	a number of seconds, where -1 means no limit
   unavailable_after	a date in RFC 822, RFC 850 or ISO 8601 form

   Anything else — an unknown name, a valued directive written without its value, a valueless directive written with one, or a value outside the accepted set — is recorded as not recognised, because that is exactly how a crawler treats it.

5. Findings: One issue is raised per page, carrying everything wrong with the tag:

   Detail	Contents
   harmfulDirectives	noindex, none, max-snippet:0 — whichever are present (nofollow is #154's finding, not this one's)
   invalidDirectives	each unrecognised token, with the reason it was rejected
   conflictingDirectives	pairs that cannot both apply, e.g. index + noindex
   missingDirectives	which of the three preview directives the tag does not set
   directives	every directive parsed from the tag, normalised

6. Pass/Fail determination:

   • Passes: the page has no robots tag, or has one that sets all three preview directives to accepted values and declares nothing harmful, unrecognised or contradictory.
   • Fails: the tag suppresses the page, contains a directive no engine understands, contradicts itself, leaves a preview directive unset, or carries an empty content attribute.

A rejected value does not count as its directive being set: max-image-preview:huge is reported both as unrecognised and as max-image-preview missing, because that is the state the page is actually in.

Storage Level

Page Level — This issue is evaluated for each individual URL.

Database Table / Prisma Model

PageHtmlHeadAudit

Stored Fields

Detection Dependencies

• The following data sources are required to evaluate this issue:
• HTML Document — The crawler parses the HTML head section to find the robots meta tag
• Meta Tag Extraction — The <meta name="robots"> tag and its content are extracted
• Directive Parsing — Individual directives (max-snippet, max-image-preview, max-video-preview) are parsed from the content