2025년 11월 18일 Cloudflare Global Network 오류(비교)

보기 RAW Blame 되돌리기 비교

 r68 vs r69
 ...
 This post is an in-depth recount of exactly what happened and what systems and processes failed. It is also the beginning, though not the end, of what we plan to do in order to make sure an outage like this will not happen again.
-{{{+5 The outage}}}
+'''{{{+5 The outage}}}'''
 The chart below shows the volume of 5xx error HTTP status codes served by the Cloudflare network. Normally this should be very low, and it was right up until the start of the outage.
 ...
 The following services were impacted:
-틀
+{{{#!wiki style="margin: -5px 0px -10px 0px"
+||<tablewidth=100%><tablebordercolor=#000,#fff><tablebgcolor=#ffffff,#1c1d1f><colcolor=#000,#fff><width=40%> '''Service / Product''' ||<width=60%><colcolor=#000,#fff> '''{{{#fff,#fff Impact description}}}''' ||
+|| Core CDN and security services || HTTP 5xx status codes. The screenshot at the top of this post shows a typical error page delivered to end users. ||
+|| Turnstile || Turnstile failed to load. ||
+|| Workers KV || Workers KV returned a significantly elevated level of HTTP 5xx errors as requests to KV’s “front end” gateway failed due to the core proxy failing. ||
+|| Dashboard || While the dashboard was mostly operational, most users were unable to log in due to Turnstile being unavailable on the login page. ||
+|| Email Security || While email processing and delivery were unaffected, we observed a temporary loss of access to an IP reputation source which reduced spam-detection accuracy and prevented some new-domain-age detections from triggering, with no critical customer impact observed. We also saw failures in some Auto Move actions; all affected messages have been reviewed and remediated. ||
+|| Access || Authentication failures were widespread for most users, beginning at the start of the incident and continuing until the rollback was initiated at 13:05. Any existing Access sessions were unaffected.
+All failed authentication attempts resulted in an error page, meaning none of these users ever reached the target application while authentication was failing. Successful logins during this period were correctly logged during this incident.
+Any Access configuration updates attempted at that time would have either failed outright or propagated very slowly. All configuration updates are now recovered. ||
+}}}
 As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information.
-{{{+5 How Cloudflare processes requests, and how this went wrong today}}}
+'''{{{+5 How Cloudflare processes requests, and how this went wrong today}}}'''
 Every request to Cloudflare takes a well-defined path through our network. It could be from a browser loading a webpage, a mobile app calling an API, or automated traffic from another service. These requests first terminate at our HTTP and TLS layer, then flow into our core proxy system (which we call FL for “Frontline”), and finally through Pingora, which performs cache lookups or fetches data from the origin if needed.
 ...
 [[파일:클플 오류6.png|width=100%]]
-{{{+5 The query behaviour change}}}
+'''{{{+5 The query behaviour change}}}'''
 I mentioned above that a change in the underlying query behaviour resulted in the feature file containing a large number of duplicate rows. The database system in question uses ClickHouse’s software.
 ...
 However, as part of the additional permissions that were granted to the user, the response now contained all the metadata of the r0 schema effectively more than doubling the rows in the response ultimately affecting the number of rows (i.e. features) in the final file output.
-{{{+5 Memory preallocation}}}
+'''{{{+5 Memory preallocation}}}'''
 Each module running on our proxy service has a number of limits in place to avoid unbounded memory consumption and to preallocate memory as a performance optimization. In this specific instance, the Bot Management system has a limit on the number of machine learning features that can be used at runtime. Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features.
 ...
 {{{thread fl2_worker_thread panicked: called Result::unwrap() on an Err value}}}
-{{{+5 Other impact during the incident}}}
+'''{{{+5 Other impact during the incident}}}'''
 Other systems that rely on our core proxy were impacted during the incident. This included Workers KV and Cloudflare Access. The team was able to reduce the impact to these systems at 13:04, when a patch was made to Workers KV to bypass the core proxy. Subsequently, all downstream systems that rely on Workers KV (such as Access itself) observed a reduced error rate.
 ...
 The first period, from 11:30 to 13:10, was due to the impact to Workers KV, which some control plane and dashboard functions rely upon. This was restored at 13:10, when Workers KV bypassed the core proxy system. The second period of impact to the dashboard occurred after restoring the feature configuration data. A backlog of login attempts began to overwhelm the dashboard. This backlog, in combination with retry attempts, resulted in elevated latency, reducing dashboard availability. Scaling control plane concurrency restored availability at approximately 15:30.
-{{{+5 Remediation and follow-up steps}}}
+'''{{{+5 Remediation and follow-up steps}}}'''
 Now that our systems are back online and functioning normally, work has already begun on how we will harden them against failures like this in the future. In particular we are:
 ...