Safe Distributed Locks for Backends: Leases, Fencing Tokens, and Lost Ownership

A distributed lock is not merely a key with a TTL, and it cannot turn a network into a perfectly synchronous environment. A process can acquire a lock, pause during garbage collection or a virtual-machine stall, let its lease expire, and later resume as if it still owned the resource. Another process may have acquired a new lease and written newer data meanwhile. Unless the destination can distinguish ownership generations, the stale process can overwrite the new result even though the lock service honored its expiration contract.

This guide develops a practical distributed-lock design for backends: decide whether a lock is necessary, separate mutual exclusion from authority over the protected resource, assign distinct roles to leases and ownership tokens, add monotonically increasing fencing tokens, bound waiting and execution, resolve unknown outcomes, and test failures such as pauses, partitions, and failed renewals. The focus is not one library but the invariants that must hold when processes, containers, or network zones compete for a shared resource.

1. First ask whether a distributed lock is necessary

Locks are often introduced too early to hide a problem that the system of record can solve more directly. A unique constraint or idempotency key is more reliable for preventing duplicate orders. An atomic conditional update protects inventory better than a read-lock-write workflow. A broker's partition assignment is usually clearer than a custom mutex when only one consumer should process a partition.

Distributed locks are most suitable for coarse coordination, short ownership periods, and bounded contention: leader election, shard handoff, a singleton maintenance operation, or access to a legacy system without a suitable atomic primitive. They should not become a general transaction manager around every request.

2. Four failures break the intuition “I still hold the lock”

A local mutex is tied to threads and shared memory. A distributed lock service sees only requests across a network. It cannot directly know whether an application is running, paused, partitioned, or dead. A design must account for at least four cases:

1. Process pause: the holder stops longer than its TTL, then resumes with its old stack and data.
2. Network partition: the client cannot reach the lock service but can still reach the database or protected device.
3. Lost response: acquire or renew committed at the server, but the response timed out, leaving the client uncertain.
4. Clock and scheduling effects: wall clocks can jump, while timers, event loops, and threads can be delayed without a crash.

“The process is alive” does not prove the lease is valid, and “the request timed out” does not prove acquisition failed. Ownership must be treated as temporary state that can disappear at any time. Every important side effect needs an independent barrier at the protected resource.

A lease tells a client when it may attempt work; a fencing token tells the destination which request is obsolete.

3. Leases, ownership tokens, and fencing tokens have different jobs

A lease is time-bounded ownership. Its TTL allows the system to make progress after a holder dies without unlocking. An ownership token is a random value tied to one acquisition, ensuring that a client renews or releases only its own lease. A fencing token is a monotonically increasing number assigned to each ownership generation; the destination rejects operations carrying a token below the highest token it has accepted.

acquire("invoice-export")
  -> lease_id: "random-owner-value"
  -> fencing_token: 731
  -> expires_in_ms: 15000

These values are not substitutes. A TTL cannot prevent an old holder from continuing. A random ownership token stops A from deleting B's newer lock but does not tell storage that A is stale. A fencing token protects ordering only when the system performing the side effect stores and checks it atomically.

4. The stale-holder scenario and destination-side fencing

Suppose worker A receives fencing token 41 and starts building a report. A pauses for 30 seconds, exceeding a 15-second lease. Worker B acquires the lock, receives token 42, writes a new file, and the storage gateway records last_fence = 42. A then resumes and submits token 41. Storage that checks only credentials and path may let A overwrite B. Storage that enforces fencing rejects A.

UPDATE export_targets
SET object_version = :new_version,
    last_fence = :token,
    updated_at = CURRENT_TIMESTAMP
WHERE id = :target_id
  AND last_fence < :token;

The caller must inspect the affected-row count. Zero means that the token is stale or the target is missing; it is not automatically an idempotent success. If object storage or a device cannot evaluate a fence directly, route writes through one gateway that checks tokens, or create immutable versioned objects and update the current pointer with compare-and-set.

A fencing token must increase in the order that the coordination service confirms ownership. A database sequence, a revision from a linearizable coordination store, or an atomically allocated counter may provide it. A client wall-clock timestamp is not a trustworthy fence because it can collide, move backward, or differ across machines.

5. Acquisition and release need atomic contracts

At minimum, acquisition must create a lock only when absent and attach the TTL and ownership token in one atomic action. A check-then-set sequence allows two clients to observe an empty lock. Release must delete only when the stored token matches the holder. An unconditional delete can remove B's new lease when A's delayed release arrives after A's lease expired.

owner = secureRandom()
acquired = putIfAbsent(
  key = "locks/report:monthly",
  value = owner,
  ttl = 15 seconds
)

releaseAtomicallyIfValueEquals(key, owner)

Renewal must likewise be an atomic compare-and-renew. A mismatch means ownership has been lost and the client must begin cancellation; it must not recreate the key and continue the same work. Limit renewal count or total hold duration so a live but defective holder cannot monopolize a resource forever.

6. Choose TTL, heartbeat, and execution budgets together

A short TTL expires during ordinary pauses; a long TTL delays recovery after a dead holder. Measure acquisition time, network latency, runtime pauses, and execution percentiles instead of selecting a round number. Work with no execution bound cannot be made safe by choosing one TTL.

• Give the operation an overall deadline and stop beginning new steps as lease expiry approaches.
• Renew well before expiry with room for jitter and one bounded retry.
• Use a monotonic clock to measure local duration; do not infer authority from wall time.
• Mark the context as ownership lost as soon as renewal fails or confirmation no longer fits in the safety margin.
• Attach the fencing token to the final side effect even when every heartbeat appeared healthy.

For example, a 30-second lease might heartbeat every 8 seconds and abandon ownership unless renewal is confirmed before a defined safety margin. Those values must come from workload telemetry and failure assumptions; they are not universal defaults.

7. An unknown outcome is not a definite failure

When acquisition times out, the server may have granted the lease and lost only the response. Blind retry can create another session or make the client compete with itself. A coordination API should support a stable request ID, a way to inspect session state, or idempotent acquisition within the same session. If ambiguity cannot be resolved, the client may need to let the suspected lease expire before retrying under a conservative policy.

A timeout while writing a resource also does not prove the side effect failed. Combine an idempotent operation ID with the fencing token. The fence answers “which ownership generation is authorized?” while the operation ID answers “is this a replay of the same action?” They cover different failure dimensions.

8. Model the holder as a state machine

state = ACQUIRING
lease = acquire(deadline)
if lease.failed: return BUSY

state = ACTIVE
start heartbeat(lease)
try:
  while hasMoreSteps():
    require state == ACTIVE
    require lease.remaining() > safetyMargin
    performBoundedStep()
  commitSideEffect(fence = lease.fencingToken,
                   operationId = command.id)
finally:
  state = RELEASING
  releaseIfOwner(lease.ownerToken)

A background heartbeat must report lost ownership to the main execution path. Logging a renewal error while work continues is a correctness bug. Long steps should support cancellation or be split so state can be checked between them. When an external call cannot be canceled, its result must still pass a fencing check before becoming current state.

Release is a best-effort latency optimization, not the only safety mechanism. The TTL releases ownership after a crash. If release times out, the client must not assume the lock is gone and start another operation for the same resource within the process.

9. Lock scope and contention determine scalability

A global key such as locks/import is simple but turns every tenant into one queue. Scope the key to the smallest invariant that needs serialization, such as locks/customer:{id}:billing-sync. Excessively fine keys, however, may fail to cover an invariant spanning multiple resources. Define the aggregate and a fixed acquisition order if an operation truly needs several keys.

• Use a global lock order to reduce deadlocks when multiple locks are unavoidable.
• Bound acquisition wait and return busy or enqueue work instead of holding a request indefinitely.
• Retry with exponential backoff and jitter so contenders do not wake together.
• Do not hold a lease while waiting for a person or an unbounded dependency.
• When contention remains high, a queue partitioned by resource key is often clearer.

10. Select a coordination service by guarantees

Do not select a technology merely because the stack already has Redis, a database, or another key-value store. Read the product's exact consistency, failover, session, lease, and revision contract. A consensus-backed service may offer increasing revisions and compare transactions suitable for fencing. An asynchronously replicated cache can lose acknowledged lock state during failover, which may be unacceptable for correctness-critical side effects.

Do not recreate consensus or a multi-node locking algorithm from a few snippets. Use a maintained client and pin compatible versions, while still fencing the protected resource because no library can infer the application's side effects.

11. Observe more than successful acquisitions

A useful dashboard includes acquisition wait, hold duration, renewal count, renewal failures, lease expiry while a holder is still running, fencing rejections, and operations that finished after ownership loss. Separate timeout, busy, session-lost, and coordination-service error outcomes.

Structured logs can include a normalized lock key, operation ID, fencing token, state transition, duration, and outcome. Avoid sensitive payloads and unbounded IDs as metric labels. Alert when p99 hold duration approaches TTL, fencing rejections increase, or one hot key dominates waiting time.

12. Test failures that the happy path cannot reveal

The essential test proves that an old holder cannot commit after a newer holder. Pause A beyond its TTL, let B acquire and commit with a higher token, then resume A; storage must reject A. Also simulate a lost acquire response, renewal timeout, delayed release, process crash, test-environment clock changes, coordination failover, and a slow destination.

• Two contenders start together: only one receives the current generation.
• A holder dies without release: another contender eventually progresses after expiry.
• The stale holder resumes: its side effect is rejected by fencing.
• An old release arrives late: it cannot delete the new lease because ownership tokens differ.
• Acquisition has an unknown outcome: retry does not create two active workflows.
• A hot key is overloaded: waiting remains bounded and retries are desynchronized.

Production checklist

• The invariant and resource scope are explicit, and simpler alternatives were considered first.
• Acquire, renew, and release are atomic with respect to the ownership token.
• The lease has a deadline, a safety margin, and a finite maximum hold duration.
• Renewal loss stops new work and never silently reacquires authority.
• Every critical side effect carries a fence that the destination enforces.
• An operation ID protects retries from duplicate effects and is not confused with a fence.
• Contention, hold time, session loss, and fencing rejection are observable.
• Fault tests cover pauses, partitions, unknown outcomes, failover, and delayed releases.

A safe distributed lock does not end with answering who owns a key. The design must accept that clients can lose authority without knowing, timeouts create uncertainty, and stale holders can still send commands. Leases enable recovery, ownership tokens bind operations to a session, and fencing tokens turn ownership into a condition the destination can enforce. If that barrier cannot be implemented, reduce the stated guarantee or redesign the workflow instead of treating a TTL as absolute proof.

References

• Redis Documentation: Distributed Locks
• etcd Documentation: How to create locks
• Google Research: The Chubby Lock Service