Control Planes for Database-Per-User in Neon

Neon is a Postgres service with a custom-built storage engine that separates compute from persistence, making it suitable for building multi-tenant, database-per-user applications. In this article, the focus is on designing an effective control plane to manage such systems at scale while maintaining operational efficiency and security. The Neon object hierarchy consists of projects, branches, and databases. When talking about database-per-user architectures in Neon, a project-per-user design is usually considered. A control plane is a crucial component for managing database-per-user systems. It extends beyond tools like the Neon console and `neonctl` to include other systems such as CI/CD, observability platforms, and schema migration frameworks. The control plane grows with an organization's system and its needs. The catalog database serves as a centralized hub for tracking and managing all Neon projects and databases. It is essential for effectively managing projects at scale by storing critical operational data such as schema version and customer billing status. Designing the catalog database schema involves linking tables, using appropriate data types, capturing critical operational data, indexing, and starting with the essentials while planning for extensions. The control plane has various uses, including onboarding new customers, billing and invoicing, updating the catalog database, storing initial information in a user's database, provisioning application instances, and monitoring and observability. Automating tasks like onboarding or schema migrations can make them faster and safer as an organization scales. Key takeaways for designing an effective control plane include starting simple and scaling thoughtfully, treating the control plane like an internal product, centralizing metadata, picking battles wisely, and using the catalog database for insights. The control plane integrates and standardizes operations on the data plane, making it legible and comprehensive, thus minimizing risks of technical operations.