Visa outage of 1 June 2018: a partial switch failure that failover did not catch

What happened on 1 June 2018, from Visa's own account

This post is a retrospective, published on the eighth anniversary of the incident. The primary source is the letter Visa Europe sent to the UK Treasury Committee on 15 June 2018, signed by Charlotte Hogg, then Chief Executive Officer of Visa Europe. We re-read it on 22 September 2026. All times are British Summer Time, as the letter gives them.

The letter gives these figures for the incident window:

• 51.2 million Visa transactions were sent to Visa's European systems, and 5.2 million (10%) "failed to process correctly".
• For UK-issued cards, 27.6 million transactions were sent and 2.4 million (9%) failed to process.
• During the two peaks, "an average of 35 percent of transactions failed to process for UK cardholders". Outside the peaks, the UK failure rate "was closer to 7 percent on average".
• Many cardholders retried, and taking retries into account, the failure rate "falls by approximately half".

Visa defined "failed to process" as either a timeout, when the system "did not respond within required time limits (generally 10-15 seconds)", or a transaction "improperly declined due to a Visa system error". The letter states that "at no time during the incident was there a full system outage".

The failure class: a partial fault that failover logic did not detect

The architecture in the letter was redundant on paper. Visa operated two UK data centres in an "active-active" arrangement, each able to "independently handle 100 percent of the transactions for Visa in Europe". Each centre had two core switches, primary and secondary, and "if the primary switch fails, in normal operation the backup switch would take over".

The letter describes what went wrong in one sentence: "a component within a switch in our primary data centre suffered a very rare partial failure which prevented the backup switch from activating." The half-working primary site then kept talking to the healthy one. In the letter's words, "the malfunctioning system at the primary data centre continued to try to synchronise messages with the secondary site. This created a backlog of messages at the secondary data centre, which, in turn, slowed down that site's ability to process incoming transactions."

We call this pattern a grey failure: a component that is degraded without being dead, so health checks built for a clean outage keep treating it as available. The letter's remediation list confirms the detection gap. The switch manufacturer recommended software that "will continuously review key components in the switch to track their availability" and "automatically take the component or switch out of operation" when the same errors recur. Visa also cited its planned migration to VisaNet, which "can isolate and remove a failing component with one command".

In its January 2019 written evidence to the Treasury Committee, Visa stated that VisaNet "now removes our reliance on the core infrastructure that led to this failure". The same document cautions that unique events "can happen regardless of the level of investment that a business has made or the amount of testing that has been applied". We take that as the honest limit of any test plan, and we do not claim a test would have prevented this specific hardware fault.

Why the Visa outage matters for payment testing

A failover test that powers off a node proves the easy case. The 1 June failure had three properties that such a test never exercises:

• The faulty component stayed partly alive. The partial failure "prevented the backup switch from activating", so the redundancy inside the primary data centre never engaged.
• The fault spread through replication. The healthy site slowed down under a backlog of synchronisation messages from the unhealthy one.
• The failures looked like ordinary declines and timeouts. The letter counts timeouts and improper declines as failures, and notes that cardholders see declines at any time for fraud checks, expired cards, or insufficient funds. The UK failure rate averaged about 7% outside the peaks and 35% during them.

The downstream money effects are in the letter. Some merchants "decided temporarily not to accept any cards and move to 'cash only'". Issuers placed holds for pending transactions that had failed, which Visa and issuers then had to clear. Visa waived fees to UK acquirers for failed transactions and said it was establishing a programme "to mitigate processing-related fees" that UK acquiring clients and merchants "may have incurred through their acquirers". For an acquirer or payment facilitator, each of those is a reconciliation item and a support queue; our acquiring segment page covers where authorisation and settlement break.

Visa's January 2019 evidence also describes its stand-in processing for issuers, which "provides an authorisation transaction response, based on the issuer's rules, in the event that the issuer has an authorisation processing or capacity issue". Visa reported handling 18.9 million transactions for UK issuers through stand-in in 2018. Issuer stand-in is designed for an issuer outage. On 1 June the degradation sat inside Visa's own processing, so the tests below focus on merchant and acquirer fallback.

What to test: degraded components, replication backlog, merchant fallback

These are the tests we run for this failure class. They are our practice, and no scheme rule prescribes them.

We inject the degraded state at the network layer rather than by stopping processes. On a Linux test host, tc with the netem discipline (iproute2) produces partial loss and jitter on the replication interface:

# Degrade the replication link: 200 ms delay with 50 ms jitter, 30% packet loss
tc qdisc add dev eth1 root netem delay 200ms 50ms loss 30%

# Run the authorisation load and the failover drill, then remove the fault
tc qdisc del dev eth1 root

The interface name and values are test parameters; we choose them so the component still passes a plain liveness check, because a fault that trips the liveness check tests the clean outage again.

Three points decide whether the evidence is useful:

1. Measure the fleet, not one request. A 7% failure rate hides inside normal decline noise. Our performance testing runs record the decline and timeout rate per response code during the drill, against a baseline taken before the fault.
2. Test duplicate holds on the merchant side. Retries were common on 1 June. The payment idempotency guide has the request patterns we use to prove a retry creates one hold and one capture.
3. Test offline and cash-only procedures as flows. A merchant that switches to offline acceptance needs defined limits, a queue, and a clean end-of-day batch. Our POS segment page lists the terminal and batch-close cases.

Our operational resilience testing service runs these drills with transactions in flight and hands over the failover timings, the fault parameters, and the reconciliation result for each run.