Renew Vault Leases in Long-Running Workers

A short-lived request handler authenticates to Vault, reads a secret, and exits — nothing to renew. A long-running worker is different: its token and its dynamic database lease both expire on a TTL, and if it never renews them it starts throwing permission errors an hour into the job. This page keeps a HashiCorp Vault session alive with hvac, complementing the dynamic database credentials and AppRole workflows.

The reason workers need their own page is that they hold two independent leases with different rules, and both must be maintained. The token lease authorises talking to Vault at all; the secret lease keeps the database user alive. Renewing one and forgetting the other produces a worker that can still reach Vault but has no valid database credential, or one with a live credential and no way to obtain the next.

Problem 1: authenticate once, never renew

# ANTI-PATTERN
client.auth.approle.login(role_id=RID, secret_id=SID)   # token expires in 1h
while True:
    do_work(client)   # PermissionError once the token TTL lapses

The token silently expires mid-job and every subsequent call fails. What makes this particularly awkward in a worker is that the failure lands in the middle of processing rather than at startup: a message is dequeued, the work begins, the credential dies, and the job fails after having done part of its side effects. Whether that is safe depends entirely on whether the work was idempotent, which is rarely something anyone decided deliberately.

The while True shape also means no natural restart point. A request handler that fails gets a new process eventually; a worker loop keeps running, failing every iteration, consuming messages and dead-lettering them at speed. A worker that cannot renew should stop rather than spin — a crash is recoverable by the orchestrator, a fast failure loop drains a queue into the dead-letter.

Problem 2: ignoring the database lease

Even with a live token, the dynamic database credential has its own lease. When it expires the database rejects the connection, even though Vault itself is still reachable. This is the confusing variant: Vault health checks pass, the token is valid, vault status is green, and the worker still cannot query anything — because the thing that expired was never the token.

Distinguishing the two at the moment of failure saves real time. A Vault call returning 403 means the token; a database connection rejecting authentication means the lease. Logging which one failed, rather than a generic “credential error”, turns a confusing incident into a one-line diagnosis.

Secure implementation

Renew the token and the lease before expiry, and re-authenticate on failure.

# vault_session.py
import time
import hvac


class VaultSession:
    def __init__(self, url, role_id, secret_id):
        self.url, self.role_id, self.secret_id = url, role_id, secret_id
        self.client = hvac.Client(url=url)
        self._login()

    def _login(self):
        self.client.auth.approle.login(role_id=self.role_id, secret_id=self.secret_id)

    def maintain(self, lease_id: str):
        """Call periodically from the worker loop (e.g. every 60s)."""
        try:
            self.client.auth.token.renew_self()          # extend the token TTL
            self.client.sys.renew_lease(lease_id=lease_id)  # extend the secret lease
        except hvac.exceptions.InvalidRequest:
            self._login()                                # token/lease gone -> re-auth
            return "reauthenticated"                     # caller should refetch + reconnect
        return "renewed"

Renew at half the TTL from the worker loop; on InvalidRequest, re-authenticate, fetch a fresh credential, and reconnect the database client.

The return value is the interface that matters. maintain deliberately does not reconnect anything itself — it reports what happened and leaves the consequences to the caller, because only the caller knows which engines, clients, and pools were built from the old credential. A maintenance routine that tried to rebuild those would need a reference to every one of them, which is exactly the coupling that makes such code impossible to test.

Calling maintain from the worker loop rather than from a background thread is a deliberate simplification worth defending. A renewal thread introduces the usual questions — what happens if the main loop is blocked, is the client thread-safe, how does the thread exit cleanly on shutdown — for the benefit of renewing during long blocking operations. If your work items are short, in-loop renewal is simpler and has no concurrency surface at all. If a single work item can run longer than the TTL, a thread becomes necessary, and then hvac.Client access needs its own lock.

Jitter, backoff, and the thundering herd

A fleet of workers started by the same deployment authenticates within seconds of each other, so their leases expire within seconds of each other, so they all renew at the same moment — forever. That synchronisation is harmless at three workers and a genuine problem at three hundred, where every renewal interval delivers a spike of requests to Vault.

# worker.py — jittered maintenance with bounded backoff
import random, time

BASE_INTERVAL = 30.0          # seconds between maintenance attempts

def next_delay(failures: int) -> float:
    if failures == 0:
        return BASE_INTERVAL * random.uniform(0.8, 1.2)     # ±20% jitter
    backoff = min(BASE_INTERVAL * 2 ** failures, 300.0)     # cap at 5 minutes
    return backoff * random.uniform(0.5, 1.0)               # full jitter on retry

failures = 0
while running:
    process_one_item()
    try:
        session.maintain(lease_id)
        failures = 0
    except Exception:
        failures += 1
        if failures > 5:
            raise SystemExit("vault: maintenance failing repeatedly; exiting")
    time.sleep(next_delay(failures))

The jitter on the healthy path spreads renewals across a window instead of a spike. The jitter on the retry path matters more: without it, a brief Vault outage synchronises the entire fleet — every worker fails at the same instant, backs off by the same amount, and retries in unison, which is precisely the load pattern a recovering Vault handles worst.

The failure counter and the eventual exit are the other half. A worker that cannot renew after several attempts is going to fail its next database operation anyway, so exiting lets the orchestrator restart it — which performs a fresh login with a fresh secret_id and often resolves the situation entirely. What it must not do is continue consuming work it cannot complete.

Renewals are also worth alarming on. A worker that re-authenticates frequently is telling you its TTL is too short for its workload, and a fleet whose renewal failure rate is climbing is telling you about Vault before your users find out. Both are cheap counters and both are more informative than a dashboard of Vault’s own health, because they measure the thing that actually affects the work.

Shutting down without stranding leases

The other half of a worker’s lifecycle is its ending, and it is routinely ignored. A worker that is scaled down, redeployed, or killed by the scheduler leaves its token and its secret lease alive in Vault until they expire on their own. With a one-hour TTL and a deployment cadence of several times a day, a fleet accumulates a steady population of orphaned leases belonging to processes that no longer exist.

That matters for three reasons. Each orphaned database lease holds a user that may still have open connections, consuming connection slots. Vault’s lease count grows, and lease storage is not free. And most importantly, a credential that outlives its process is exactly the kind of thing that should not exist in a system built around short-lived credentials — the whole point is that access ends when it is no longer needed.

# worker.py — revoke on the way out, including on SIGTERM
import signal

running = True

def _stop(signum, frame):
    global running
    running = False                       # let the loop finish its current item

signal.signal(signal.SIGTERM, _stop)
signal.signal(signal.SIGINT, _stop)

try:
    while running:
        process_one_item()
        session.maintain(lease_id)
finally:
    try:
        session.client.sys.revoke_lease(lease_id=lease_id)   # drop the DB user now
        session.client.auth.token.revoke_self()              # drop the token now
    except Exception:
        pass                              # shutting down anyway; TTL is the backstop

Handling SIGTERM is what makes this work in practice, because that is what orchestrators send before a container is stopped. Setting a flag rather than raising means the current work item finishes rather than being abandoned half-done, which matters for anything non-idempotent. The grace period the platform allows — typically thirty seconds by default — bounds how long that finishing can take, so a worker whose items routinely run longer needs that period raised or its items made resumable.

The except Exception: pass around revocation is deliberate rather than lazy. The process is exiting; a failed revocation is not worth turning a clean shutdown into a crash, and the TTL still guarantees the credential dies. Revocation is an optimisation that closes the window early, not a correctness requirement — which is precisely why it is safe to attempt on a best-effort basis.

Worth knowing: revoking the token also revokes every lease created by that token, so revoke_self() alone usually suffices. Revoking the database lease first is belt-and-braces for the case where the token was already gone, and it makes the intent explicit to anyone reading the shutdown path.

Gotchas & version-specific behaviour

• A lease has a max TTL; past it, renewal fails and you must fetch a new dynamic credential.
• renew_self extends the token but not any secret leases — renew both.
• Recreate the database engine/connection pool after refetching credentials; old connections keep the expired password.
• Add jitter to the renewal interval so a fleet of workers does not renew in lockstep.
• Handle Vault being briefly unreachable with retries; do not crash the worker on one failed renew.
• Renewal returns the new lease_duration, which may be shorter than requested when the max TTL is close — read it rather than assuming the interval is unchanged.

That last point turns a smooth renewal loop into a nasty surprise at the end of a lease’s life. As a lease approaches its max TTL, Vault grants shorter and shorter extensions, so a worker renewing every thirty seconds on the assumption of an hourly TTL will eventually be renewing a lease with ninety seconds left. Reading the returned duration and shortening the interval accordingly is the difference between noticing the ceiling and hitting it.

Production parity checklist

• Renew token and lease at roughly half their TTL from the worker loop.
• Re-authenticate and reconnect on InvalidRequest rather than crashing.
• Rebuild the connection pool when the dynamic credential is replaced.
• Add jitter to renewal timing across the worker fleet.
• Alarm on repeated re-authentications, which signal a TTL that is too short.
• Exit after a bounded number of consecutive maintenance failures rather than looping.
• Handle SIGTERM and revoke both the token and the secret lease on the way out.

Verifying the shutdown path is easy to skip and worth ten minutes. Send the worker a SIGTERM in a staging environment, then check that Vault reports no remaining lease for that instance and the database shows the dynamic user gone. If either survives, the handler is not wired up, and you will not find that out from any log line — an orphaned lease is silent by nature.

Key takeaways

A long-running worker must renew both its Vault token and its secret lease before expiry, and re-authenticate then rebuild its connections when a lease reaches its maximum TTL. The two leases fail differently and are diagnosed differently — a 403 from Vault is the token, a rejected database login is the lease — so logging which one lapsed converts a confusing incident into an obvious one.

Everything else is operational hygiene that costs a few lines: jitter so a fleet does not renew in lockstep, bounded backoff so a Vault outage does not turn into a retry storm, a failure counter so a worker that cannot renew exits rather than draining its queue into the dead-letter, and reading the returned lease_duration so the approaching max TTL is visible before it arrives.