WP Newsify

Multi-CDN failover sent users to a non-HTTPS origin causing mixed-content errors and the failover healthcheck that enforced TLS

As the web grows more complex and performance expectations rise, using multiple Content Delivery Networks (CDNs) has become a common strategy for increasing availability, improving load times, and guarding against outages. However, when not implemented carefully, multi-CDN setups can introduce new problems—ones that unexpectedly impact user experience and compromise security. One such incident involved a failover scenario in a multi-CDN configuration that began routing users to a HTTP origin despite the primary site being HTTPS-only. This led to widespread mixed-content errors and client-side asset failures.

TL;DR

When a healthcheck-triggered failover in a multi-CDN configuration switched traffic to a secondary CDN, it routed users to a non-HTTPS (HTTP-only) origin. This caused browsers to reject insecure content, triggering mixed-content warnings and breaking critical site functionality. Surprisingly, the failover healthcheck criteria only validated TLS from the CDN to the origin, not end-user HTTPS access. Addressing the issue required a review of origin health monitoring, stricter TLS enforcement, and content validation rules.

The Rise of Multi-CDN Architectures

Deploying a multi-CDN architecture has become a best practice in today’s global internet landscape. Organizations use two or more CDNs to improve:

This kind of approach works well in theory, but its complexity leaves room for misconfiguration. If one of your CDNs is misaligned—especially in how it communicates with the origin or serves secure content—you may unknowingly expose your users to errors and even security risks.

The Incident: When Failover Meets HTTP

During a routine failover initiated due to high latency detected on CDN1, a global e-commerce platform began routing traffic through CDN2. However, without thorough testing of how CDN2 fetched resources from the origin, something crucial was overlooked: CDN2 was serving assets via a non-HTTPS origin.

This wouldn’t have been a problem for standalone HTTP sites, but in this case, the platform’s primary domain enforced HTTPS across all pages using a strict Content Security Policy (CSP). As soon as CDN2’s HTTP-based resources were served in an HTTPS environment, browsers began throwing mixed-content errors and blocking essential content such as JavaScript files, CSS stylesheets, and fonts.

The result? Users couldn’t interact with the platform: buttons didn’t respond, drop-downs didn’t function, and payment pages failed to load.

Understanding Mixed-Content Errors

Mixed content occurs when a secure website (loaded over HTTPS) loads any asset (like scripts, images, or videos) over an insecure HTTP connection. Most modern browsers cancel such requests to protect users, leading to breakdowns in how pages are displayed or function.

There are two types of mixed-content:

  1. Active mixed content: Includes scripts, iframes or other code that can interact with the document. These are blocked completely.
  2. Passive mixed content: Includes media elements like images or videos. Browsers might load these but often flag warnings.

In this case, it was heavy with active mixed content: everything from React bundles to authentication scripts failed to load, effectively turning a robust e-commerce platform into a static HTML tombstone.

The Healthcheck That Missed the Mark

Most CDNs employ healthchecks to determine if an origin is healthy and can serve traffic. These healthchecks typically include HTTP requests at regular intervals to check if an expected response is received. But in this scenario, an important nuance was missed: the failover was triggered by latency metrics from CDN1, and once routed to CDN2, CDN2’s healthchecks confirmed the origin was “healthy.” But how?

As it turns out, the healthcheck validated the origin by accessing a TLS-terminated endpoint—but only between the CDN and origin server. It didn’t validate how the origin responded to user-facing URLs or whether HTTPS was properly configured for content accessible to real end-users.

Key misconfiguration issues included:

This oversight led to a situation where the origin passed the healthcheck but delivered HTTP content incompatible with the site’s security requirements.

Detecting and Diagnosing the Problem

The issue didn’t manifest immediately. Initially, CDN2 showed green lights—origin connections succeeded, responses were timely, and there were no internal CDN alarms. However, within minutes, error reports began to surge:

Teams quickly examined browser Developer Tools and noticed a flood of Mixed content: The page was loaded over HTTPS, but requested an insecure script errors. The logs, rich in 403 and 404 responses for asset files, pointed to HTTP origins being force-fed into HTTPS pages.

Mitigation and the Fixes Applied

Resolving the issue involved multiple tactical and strategic fixes.

Tactical Fixes:

Strategic Changes:

What Can We Learn From This?

This type of failover fiasco highlights the dangerous assumptions often made in distributed environments:

Takeaway: Any component in a failover chain should be treated as live production and be tested accordingly—including security configurations, protocols, and content delivery behavior. CDN redundancy is valuable—but only when all parts conform to the same expectations.

Final Thoughts

Failover systems are meant to improve stability, not compromise it. In this story, a seemingly intelligent, automated system created chaos because origin validation didn’t take HTTPS and user security into account. By refining healthcheck criteria and enforcing stricter protocols, companies can avoid this pitfall and deliver the reliable, secure experience users expect.

The battle for high availability doesn’t end by adding more nodes; it ends by ensuring every node, path, and byte respects the security standards of your application—especially when adversarial conditions strike.

Follow Us
Exit mobile version