How to handle page triggers on pages without pagination?

[jannisborgers]

The basic problem: Page triggers (e.G. example.com/some-page/page/2) are available on all pages of a site, even if none of the templates used in the request use a paginate tag.

A client’s Sistrix report said that the meta description was the same for the URLs /about-us, /about-us/page/1, /about-us/page/2, etc. I checked and indeed, the URLs are all available in the Google index, even though they were never available as clickable links in the site or in a sitemap.xml.

This site has a /news entry that shows a paginated list of news posts, and a simple content page like /about-us that does not paginate anything. Both /news/page/2 and /about-us/page/2 are valid URLs that can be visited.

My first instinct was to insert <link rel="canonical" href="[entry-url]" /> to all pages expect the news pages.

Is there another solution? Given that there are matrix blocks with relational fields that can be optionally paginated in my content-builder matrix field, the solution above doesn't feel very comfortable. Pages that include a paginated list of items should be available as separate pages to search engines, while pages without paginated items should all point to the same canonical URL.

[brandonkelly]

How is the Sistrix report finding out about these other URIs to begin with? Is something liking to them? Theoretically if nothing links to them and they’re not part of a site map, they shouldn’t be a problem.

My first instinct was to insert <link rel="canonical" href="[entry-url]" /> to all pages expect the news pages.

You should be doing that regardless. There are multiple ways to access the same generated page with distinct URLs in Craft, beyond adding a page number (index.php?p=about-us, about-us?foo=bar, etc.). SEOmatic, and presumably other SEO plugins, will add the canonical tag for you, along with JSON-LD data and other SEO tuning.

[jannisborgers]

Thanks Brandon.

There isn’t anything in the available revisions of that page. There might have been a paginated list placed by an editor at some point in the past and due to the URLs being publicly available, it was never removed from the index after the list was removed.

I‘ll use the canonical tag and see if it fixes the indexing issues.

[rob-c-baker]

We have this issue too. While this is not a high priority issue for us, it seems to make sense that an invalid pagination URL should generate an error.

The canonical tags helps in most cases, but it is not something all indexing bots adhere to - we still have external links pointing to some of these types of URLs that are well over a decade old. Some URLs in the past have had pagination, some just seem to be random URLs that, presumably, scraping bots tried and got a 200 status code. These links do get picked up by indexing bots that still hit the URLs.

[brandonkelly]

it seems to make sense that an invalid pagination URL should generate an error.

Craft has no way of knowing for sure whether pages could be paginated. (Maybe your pagination logic is behind a conditional, or that area of the template is cached with {% cache %} tags, or you’ve implemented custom pagination navigation logic.)

[mmikkel]

If you really want to prevent the 200 status code for "invalid" pagination URLs, you can use a little bit of Twig to manually return a 404 whenever there's a pageTrigger in the URL.

Something like this could go at the top of your page template (or even in your Twig layout, if your site doesn't use pagination anywhere):

{% if craft.app.response.statusCode == 200 and craft.app.request.absoluteUrl matches "/#{craft.app.config.general.pageTrigger}[\\d\\.]+(\\/.*)?$/" %}
    {% exit 404 %}
{% endif %}

Basically what that does is to use a regular expression to test if the requested URL ends with the page trigger followed by a number (and an optional trailing slash); terminating the request if that's the case.

If your page trigger is configured to use a query parameter, something like this would probably be more appropriate:

{% if craft.app.response.statusCode == 200 and craft.app.request.queryParams[craft.app.config.general.pageTrigger|trim('?')]|default %}
    {% exit 404 %}
{% endif %}

Finally, if your page template does use pagination, but you want to throw a 404 if the requested URL has a page trigger higher than the total amount of pages, you could do something like this:

{% paginate query as pageInfo, entries %}

{% if craft.app.request.pageNum > pageInfo.totalPages %}
    {% exit 404 %}
{% endif %}