Why does your link checker report 403 errors on pages that open fine in a browser?

What is really happening behind a 403

A 403 means the server understood the request and refused to fulfill it. When a human can open the same page, the refusal is about who is asking rather than what is being asked for. A bot management layer scored the request and rejected it, a web application firewall matched a rule on the user agent or on missing headers, the site blocks datacenter address ranges outright, or a rate limit tripped after your checker requested too many pages from that host in a short burst. Related: What a 404 Does to a Visitor, and How to Soften It

Some hosts and content delivery layers return 403 to anything that has not completed a JavaScript challenge, which a plain HTTP client can never pass. Others reject HEAD requests specifically while answering GET normally. In every one of these situations the page is fine for visitors and the link is not broken. The report is describing your checker's relationship with that server, not the state of the destination, and it should be read that way. Related: How to Find Broken Links at Scale

Keep reading: Why Broken Links Quietly Hurt Your SEO, Internal vs External Broken Links: Which to Fix First, How to Find Broken Links at Scale. See how LinkRotly helps you continuous broken link monitoring for websites.

Separating false positives from real blocks

The diagnostics are quick. Fetch the URL with a browser-like user agent and standard Accept and Accept-Language headers. Try GET instead of HEAD. Try from a different network or region. Slow down and retry after a pause. If any of those returns a 200, you have a checker-side problem, not a broken link. Read the response body as well, because bot challenge pages carry telltale markup such as a challenge form or a message about verifying your browser, even when the status is 403. Related: How Often Should You Audit Your Site for Broken Links

Real 403s look different. A page moved behind a login, a paywall, a members area, or a geographic restriction will return 403 or a login page regardless of how politely you ask. If every method still fails and the body is an access denied or sign in page, the link is effectively broken for any of your readers who lack access. At that point the decision becomes editorial: keep the link with a note about access, replace it with an open source, or remove it.

Configuring a checker to be a good citizen

Identify the checker honestly with a descriptive user agent that includes a URL where site owners can learn what it is and how to contact you. Send realistic Accept and Accept-Language headers. Respect robots.txt disallow rules and any crawl delay directive. Rate limit per host with a small number of concurrent requests and a pause between them. When a host answers 403 or 429, retry later rather than immediately, and back off further if the second attempt also fails.

Resist the temptation to impersonate a browser to force through a block. It may work for a while, but it defeats the intent of the site's protection and can get your address range banned entirely, which then breaks every other check against that host and possibly others behind the same provider. If a destination persistently blocks all automated access, classify it as unverifiable instead of broken, and put it on a short list for occasional manual review. Related: Why Broken Links Quietly Hurt Your SEO

Reporting 403s usefully

Give 403s their own category with subtypes so a reviewer knows what to do at a glance: likely bot block when challenge markup is detected, likely rate limit when the failure followed a burst or came with a 429, access restricted when a login page is detected, and unknown for the rest. Include a snippet of the response body and the headers that matter, such as the Server header or challenge related headers, so the reviewer can act without fetching the page again.

Reduce noise deliberately. Do not alert on a first-seen 403 from a host already known to challenge bots. Alert only when a page that was previously reachable flips to 403 and stays there across several runs, since that pattern is what a genuine access change looks like. Re-test hosts on the unverifiable list periodically, because protection rules change over time and some sites later allow well-behaved checkers through.