At Algolia, they consider several key aspects when selecting a critical SaaS provider, including top-notch support with 24/7 response times and SLA requirements, verification of incident management tools and escalation policies, high infrastructure availability, trustworthiness and transparency of the company, security of the solution, stable and scalable APIs, understanding of the underlying infrastructure, and potential fallback solutions to mitigate risks.