| r69 vs r70 | ||
|---|---|---|
| ... | ... | |
| 346 | 346 | |
| 347 | 347 | On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today. |
| 348 | 348 | |
| 349 | ||
| 349 | {{{#!wiki style="margin: -5px 0px -10px 0px" | |
| 350 | ||<tablewidth=100%><tablebordercolor=#000,#fff><tablebgcolor=#ffffff,#1c1d1f><colcolor=#000,#fff><width=10%> '''Time(UTC)''' ||<width=45%><colcolor=#000,#fff> '''Status''' ||<width=45%><colcolor=#000,#fff> '''Description''' || | |
| 351 | || 11:05 || Normal. || Database access control change deployed. || | |
| 352 | || 11:28 || Impact starts. || Deployment reaches customer environments, first errors observed on customer HTTP traffic. || | |
| 353 | || 11:32-13:05 || The team investigated elevated traffic levels and errors to Workers KV service.|| The initial symptom appeared to be degraded Workers KV response rate causing downstream impact on other Cloudflare services. | |
| 350 | 354 | |
| 355 | Mitigations such as traffic manipulation and account limiting were attempted to bring the Workers KV service back to normal operating levels. | |
| 356 | ||
| 357 | The first automated test detected the issue at 11:31 and manual investigation started at 11:32. The incident call was created at 11:35. || | |
| 358 | || 13:05 || Workers KV and Cloudflare Access bypass implemented — impact reduced. || During investigation, we used internal system bypasses for Workers KV and Cloudflare Access so they fell back to a prior version of our core proxy. Although the issue was also present in prior versions of our proxy, the impact was smaller as described below. | |
| 359 | || | |
| 360 | || 13:37 || Work focused on rollback of the Bot Management configuration file to a last-known-good version. || We were confident that the Bot Management configuration file was the trigger for the incident. Teams worked on ways to repair the service in multiple workstreams, with the fastest workstream a restore of a previous version of the file. || | |
| 361 | || 14:24 || Stopped creation and propagation of new Bot Management configuration files. || We identified that the Bot Management module was the source of the 500 errors and that this was caused by a bad configuration file. We stopped automatic deployment of new Bot Management configuration files. || | |
| 362 | || 14:24 || Test of new file complete. || We observed successful recovery using the old version of the configuration file and then focused on accelerating the fix globally. || | |
| 363 | || 14:30 || Main impact resolved. Downstream impacted services started observing reduced errors. || A correct Bot Management configuration file was deployed globally and most services started operating correctly. || | |
| 364 | || 17:06 || All services resolved. Impact ends. || All downstream services restarted and all operations fully restored. || | |
| 365 | }}} | |
| 366 | ||
| 351 | 367 | Cloudflare's connectivity cloud protects [[https://www.cloudflare.com/ko-kr/network-services/|entire corporate networks]], helps customers build [[https://workers.cloudflare.com/|Internet-scale applications efficiently]], accelerates any [[https://www.cloudflare.com/ko-kr/application-services/products/|website or Internet application]], [[https://www.cloudflare.com/ko-kr/ddos/|wards off DDoS attacks]], keeps [[https://www.cloudflare.com/ko-kr/application-services/products/|hackers at bay]], and can help you on [[https://www.cloudflare.com/ko-kr/zero-trust/|your journey to Zero Trust.]] |
| 352 | 368 | |
| 353 | 369 | Visit [[https://one.one.one.one/|1.1.1.1]] from any device to get started with our free app that makes your Internet faster and safer. |
| ... | ... |