Practical MQTT patterns

MQTT message replay after an offline client reconnects

Learn how MQTT session queues differ from a replayable log, and how Sagüin resumes each durable consumer from a stored position.

MQTT can preserve a client session between connections, but a persistent session is not the same thing as an independently replayable event history. The distinction matters when an edge device, gateway, or downstream service can be offline long enough to exhaust a bounded session queue.

Sagüin adds an append channel to ordinary MQTT topics. The channel stores each accepted record once and keeps a position for every durable consumer. A returning consumer resumes from that position instead of requiring a private copy of the complete backlog.

Session recovery and replay solve different problems

A persistent MQTT session remembers subscriptions and unacknowledged QoS 1 or QoS 2 deliveries. That is the right tool for a temporary network interruption. It does not create an application-controlled event log with an independent cursor for every reader.

An append channel is useful when the history itself matters:

  • telemetry becomes evidence for billing, maintenance, or safety;
  • a consumer may be offline for days rather than seconds;
  • a second consumer needs to read the same records at a different pace;
  • a consumer must deliberately seek back to an earlier point;
  • retention should be defined for the stream rather than by each client's private queue.

Sagüin stores a record once, then stores a position for each durable consumer: the same idea as a Kafka consumer group's committed offset, without the cluster. Ten consumers do not require ten copies of every stored record.

Configure a replayable MQTT topic range

An append channel claims an ordinary MQTT topic filter:

channels:
  fleet-telemetry:
    type: append
    filter: fleet/+/telemetry/#
    retention_period: 7d

retention_period and retention_bytes bound how long and how much history the channel keeps; left out, they take broker.storage's defaults.

Publishers continue to use normal MQTT topics such as fleet/truck-17/telemetry/temperature. Subscribers use normal MQTT subscriptions. The storage and replay behaviour comes from the channel that owns that part of the topic space, not from a new wire protocol.

Topics outside configured channels go to $broadcast, a reserved append channel that follows MQTT: every message on it is stored, whatever its QoS, and each session keeps one position on it, stored if the session is durable.

What happens during an outage

Consider a service consuming fleet/+/telemetry/#:

  1. The service connects with a stable client identifier, Clean Start off and a Session Expiry Interval above zero, so its session outlives the connection.
  2. It consumes records at QoS 1 and acknowledges them.
  3. Sagüin advances its position past acknowledged records and saves it with the next commit.
  4. The service loses its network connection while publishers continue writing records.
  5. On reconnect, the service resumes from its stored position.

Delivery is at least once at QoS 1 and 2. A record can be delivered again, after a crash or a reconnect, so the consuming application must be idempotent. Sagüin does not claim exactly-once processing, and does not deduplicate by message id: every delivery carries the record's saguin-id, which is what a consumer deduplicates on.

Retention still has a boundary

A replayable log is deliberately bounded. Retention deletes a channel's oldest records by age and size, whoever still needed them, and the operator sets it for the consumers the channel must serve. If retention passes records a consumer was owed, the consumer continues from the oldest surviving record, the floor, and the loss is counted: in saguin_missed_total, and per reader in the operations route position-lost.

The gap is visible before anyone looks at a chart. This expression fires when retention has passed a consumer's stored position:

saguin_channel_floor_offset > saguin_channel_position_min

A deliberate replay is held to the same standard: a seek to a time retention has already passed answers below-floor rather than quietly serving the oldest surviving record.

When not to use an append channel

Use ordinary MQTT broadcast when only currently connected subscribers matter. Use a retained message or Sagüin latest channel when a subscriber needs the current value rather than every transition. Use a queue channel when each record should be processed by one worker rather than by every subscriber.

The four behaviours are complementary:

RequirementBehaviour
Ordinary MQTT publish and subscribe, retained messages$broadcast
Replay every stored record per consumerAppend
Read the current value per topicLatest
Give each job to one worker at a timeQueue

Verify the contract

The detailed rules for positions, acknowledgements and reconnects are in Delivery, retention and missed records in Words, and seeking in $saguin topics. What survives a restart in each storage mode is in Store.

For a hands-on check, clone the Sagüin repository, install the Python examples as its README shows, and run make demo. The guided tour starts its own broker and walks through broadcast, append, latest, queue and storage; its append section covers independent consumers, replay, durable positions and seeking by offset.