Understanding Pulsar Message TTL, Backlog, and Retention

Apache Pulsar manages message life cycle using several mechanisms, including backlog quota and retention policy. In an ideal scenario with unlimited storage space, Pulsar retains unacknowledged messages in a backlog until they are acknowledged for deletion. However, real-world scenarios require additional measures to prevent the unlimited growth of message backlogs. To manage this, Pulsar uses time-to-live (TTL) parameters and subscription backlog quotas. TTL automatically moves a message to the acknowledged state after its expiration time, while the backlog quota enforces a hard limit on the logical size of backlogs in a topic. Pulsar offers three policies to prevent backlog overflow: producer_request_hold, consumer_backlog_eviction, and clear_backlog. The default broker option is producer_request_hold, which relies on consumers to drain the backlog. Additionally, Pulsar's retention policy helps retain acknowledged messages and messages on a topic with no subscription based on two configuration parameters: defaultRetentionTimeInMinutes and defaultRetentionTimeInSize in the broker.conf file. Understanding and managing message retention in Pulsar involves considering both backlog quota and retention policy configurations, as well as adjusting retention policies for any extra storage provided by features like tiered storage.