Full outage
All services (excluding stats) should be operational.
Stats will be undergoing a rewrite before it’s returned to service.
A blog post will be coming (it’s already written), however given current world events this should not be anyone’s focus and we’ll be publishing it in a few weeks.
download portal + mirror is restored.
gerrit is restored.
mail, www, wiki, and some internal services are restored.
it’s 3am in Seattle, the rest of this will be done tomorrow.
Around 8PM PST on May 2nd, 2020 an attacker used a CVE in our saltstack master to gain access to our infrastructure (please see https://docs.saltstack.com/en/latest/topics/releases/3000.2.html for more information).
We are able to verify that:
- Signing keys are unaffected - these hosts are entirely separate from our main infrastructure.
- Builds are unaffected - builds have been paused due to an unrelated issue since April 30th.
All services are offline - we’re working on re-provisioning servers.