May 30 SSL incident

On May 30, 2020, two root certification authorities in the Public Key Infrastructure of the Internet expired, causing some unexpected issues for Algolia's API. The company's Site Reliability Engineering (SRE) team was notified about a certificate problem with their service, but initial verifications showed that everything seemed normal and the API was responding correctly. However, further investigation revealed an underlying issue where some clients were unable to verify the certificate and connect to the API. The SRE team discovered that there was a valid path to the USERTrust RSA Certification Authority, as well as an expired path. The browser was able to find the valid chain, but other tools like curl were not able to find it. After updating their certificate configuration and removing the expired certificates from the certificate chain served by their servers, traffic levels recovered for most customers. However, a small group of customers still reported issues connecting to the API. The team found that some clients had outdated certificate stores that needed a much older root certificate. They also discovered an OpenSSL bug from 2014 which caused back end implementations of some customers to be impacted. Algolia is now reaching out to affected customers, improving their certificate checking tool, and supporting the work of the OpenSSL team through donations.