seo: stop the /join-or-log-in?next= crawl recursion
Search Console reported 72 nested /join-or-log-in URLs on media.unturf.com,
each one /join-or-log-in?next=https://.../join-or-log-in?next=... a level
deeper. Each got crawled and 401'd — Google's not indexing them (good),
but it's burning crawl budget on infinite nesting (yuck).
Root cause: base.j2's navbar "My Account" link for anonymous visitors
built the next= param from request.url (the FULL current URL with query
string). When the user lands on /join-or-log-in?next=/c/foo, that link
becomes /join-or-log-in?next=https://.../join-or-log-in?next=/c/foo —
which Google then crawls, and on THAT page the navbar link recurses
deeper, and so on.
Fix:
- base.j2 — suppress the navbar link entirely when already on
/join-or-log-in (user is on the page; no point in pointing back).
Also switch from request.url to request.path so any stray emission
can't recurse via query string.
- All internal /join-or-log-in links get rel="nofollow" (base, home,
product, comments, auction, verification-challenge). Defense in
depth: even if a future template forgets the guard, Googlebot
won't follow the link to discover deeper permutations.
Pairs with prior ba8fc45 (noindex + no-referrer + robots.txt disallow).
Already-crawled deep URLs will fall out of Search Console as Googlebot
re-fetches them, sees the 401 + no internal links pointing at them,
and ages the entries out.
This commit is contained in:
parent
ba8fc451c0
commit
7fe6b56cdb
6 changed files with 21 additions and 10 deletions
|
|
@ -60,7 +60,7 @@
|
|||
{% elif request.user is none %}
|
||||
<div class="well">
|
||||
<p>
|
||||
<a href="/join-or-log-in" class="mps-button mps-button-blue">Log in to bid</a>
|
||||
<a href="/join-or-log-in" class="mps-button mps-button-blue" rel="nofollow">Log in to bid</a>
|
||||
</p>
|
||||
</div>
|
||||
{% else %}
|
||||
|
|
|
|||
|
|
@ -149,7 +149,16 @@
|
|||
|
||||
|
||||
{% else %}
|
||||
<a href="{{ request.route_url('join-or-log-in', _query={'next': request.url}) }}">My Account</a>
|
||||
{# Suppress this link when the user is already on /join-or-log-in.
|
||||
Otherwise request.url (which includes the ?next= query string)
|
||||
feeds itself back as the next= value, building infinitely-nested
|
||||
/join-or-log-in?next=https://.../join-or-log-in?next=... URLs
|
||||
that Google then crawls page-by-page (Search Console flagged 72
|
||||
on media.unturf.com). rel="nofollow" keeps the link out of
|
||||
crawl-discovery on every other page. #}
|
||||
{% if request.path != '/join-or-log-in' %}
|
||||
<a href="{{ request.route_url('join-or-log-in', _query={'next': request.path}) }}" rel="nofollow">My Account</a>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
<a href="/cart" class="cart-and-count">🛒 Cart ${{ "{:,.2f}".format(request.active_cart.total) }} ({{ "{:,}".format(request.active_cart.count) }})</a>
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@
|
|||
<p class="comment-error">{{ error_msg }}</p>
|
||||
{% endif %}
|
||||
{% else %}
|
||||
<p><a href="/join-or-log-in">Sign in</a> to leave a comment.</p>
|
||||
<p><a href="/join-or-log-in" rel="nofollow">Sign in</a> to leave a comment.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
|
@ -30,7 +30,7 @@
|
|||
<p class="type-body-lg landing-hero-sub">Commission-free digital downloads & physical product sales. Open source. Public domain.</p>
|
||||
<div class="landing-hero-ctas">
|
||||
<a href="/s/new" class="mps-button mps-button-green">+ Open a Shop</a>
|
||||
<a href="/join-or-log-in" class="mps-button">Log In</a>
|
||||
<a href="/join-or-log-in" class="mps-button" rel="nofollow">Log In</a>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
|
@ -145,8 +145,10 @@
|
|||
<br>
|
||||
<a href="{{ product.absolute_url(request) }}" rel="nofollow">${{ '{:,.2f}'.format(product.price) }}</a>
|
||||
{% endif %}
|
||||
<br />
|
||||
<a href="{{ product.shop.absolute_url(request) }}" rel="nofollow" class="shop-theme-link-color">{{ product.shop.name }}</a>
|
||||
{# Shop name intentionally omitted on SERP cards — only meaningful
|
||||
once we ship marketplace mode (multi-shop catalog). Until then
|
||||
every card on a single-shop page belongs to the same shop, so
|
||||
the label is pure clutter. #}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
|
|
|
|||
|
|
@ -276,7 +276,7 @@
|
|||
</details>
|
||||
{% elif product.offers_allowed and (not request.user or not request.user.authenticated) %}
|
||||
<p style="margin-top: var(--space-3)">
|
||||
<a href="/join-or-log-in?next={{ request.path | urlencode }}" class="mps-button">
|
||||
<a href="/join-or-log-in?next={{ request.path | urlencode }}" class="mps-button" rel="nofollow">
|
||||
Log in to make an offer
|
||||
</a>
|
||||
</p>
|
||||
|
|
@ -293,7 +293,7 @@
|
|||
<br/>
|
||||
{% if (not request.user or not request.user.authenticated) and signed_get_object_url is none %}
|
||||
<br/>
|
||||
Already purchased? <a href="/join-or-log-in">log in</a>
|
||||
Already purchased? <a href="/join-or-log-in" rel="nofollow">log in</a>
|
||||
<br/>
|
||||
<br/>
|
||||
<br/>
|
||||
|
|
@ -322,7 +322,7 @@
|
|||
<a href="#comments" class="product-comments-link">Comments ({{ product.public_comment_count }})</a>
|
||||
{% endif %}
|
||||
|
||||
{% if price_history | length > 1 %}
|
||||
{% if shop.show_price_history and price_history | length > 1 %}
|
||||
<details class="price-history-details">
|
||||
<summary>Price History ({{ price_history | length }})</summary>
|
||||
<div class="analytics-table-wrap">
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@
|
|||
</form>
|
||||
|
||||
<div class="login-card-hint">
|
||||
<span class="type-body-sm"><strong>Didn't get a code?</strong> Check your spam folder, or <a href="/join-or-log-in">try again</a> with a different email address.</span>
|
||||
<span class="type-body-sm"><strong>Didn't get a code?</strong> Check your spam folder, or <a href="/join-or-log-in" rel="nofollow">try again</a> with a different email address.</span>
|
||||
</div>
|
||||
|
||||
</section>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue