Comment by montebicyclelo
2 days ago
TLDR, unexpected blank fields
> policy change was inserted into the regional Spanner tables
> This policy data contained unintended blank fields
> Service Control... pulled in blank fields... hit null pointer causing the binaries to go into a crash loop
Really whatever null condition caused a crash is mostly irrelevant. The big problem is instantaneous global replication of policy changes . All code can fail unexpectedly, the gradual rollout of the code was pointless since it doesn't take effect until any policy changes. Yet policy changes are near instantaneous.