Running it

What an operator needs to know: the shared machinery every region relies on, how several applications share one Redis safely, how to install Valkey on premise or point at Azure, and the checklist before a region goes to production.

On this page

    Shared machinery

    Cross-pod invalidation

    Every eviction publishes {region, key} on one pub/sub channel (dac:cache:channel:invalidate) that all applications on the Redis subscribe to. A pod that receives a message drops the entry from its L1; the next read goes to L2 or the source. The channel is shared on purpose: an eviction of a shared domain region by indc-core must reach alert-be-app's L1 too.

    Leader election

    Some work must run once across all pods — loading a static version, the scheduled refresh, flushing write-behind queues. Pods compete for a short Redis lock ({root}:lock:{role}, lease 30 s, re-extended at a third of the lease). If the leader stops, another pod takes the lock within a lease. Nothing waits on the leader: readers keep serving the current version.

    Refresh events

    Other systems tell the cache that a row changed by appending {region, key} to the Redis Stream {root}:channel:refresh. Pods read it as a consumer group: exactly one pod reloads that key from the source and the invalidation bus clears the others. An event whose consumer failed is re-claimed after claim-min-idle; a failing reload is retried up to max-deliveries times. The producer that never appends the event is the one case the cache cannot see — use an outbox when commit and event must succeed together.

    Write-behind

    A region with WRITE_BEHIND queues its writes on {root}:{region}:stream; the leader flushes them in batches, coalesced by key, through the region's CacheWriter. Failed writes land on :dlq. Watch dac.cache.writebehind.{enqueued,flushed,failed}.

    Serialisation

    Values are stored as JSON through the engine's own mapper. Records with plain fields round-trip without configuration. A third-party DTO (as the CAST UserDto case demonstrated) must be round-trip tested before its region is switched on.

    Cluster broadcast

    Asking every pod at once — the consolidated /actuator/dachealth check and @DacBroadcast functions — has moved to its own page: Cluster broadcast & health. The cluster coordination primitives (locks, semaphores, latches) are on Locks, semaphores & latches.

    One Redis, many applications

    Two rules make sharing safe:

    1. Every application has its own root — {spring.application.name}:cache unless dac.cache.root says otherwise. Two applications can both have a region called transactions without touching each other.
    2. Shared domains pin their own namespace — dac:cast, dac:idm-user… — so the same region is the same keys from every application. The contract fingerprint refuses a second application that declares the region with a different shape.

    Redis key layout

    {root}:{region}:current                   which snapshot version is live (static regions)
{root}:{region}:contract                  region shape fingerprint — the collision guard
{root}:{region}:{vN}:id:{id}              one snapshot, JSON
{root}:{region}:{vN}:idx:{name}:{value}   index entry (plain key or SET)
{root}:{region}:stream / :dlq             write-behind queue + dead letters
{root}:lock:{role}                        leader lease
{root}:channel:refresh                    refresh-event stream
dac:cache:channel:invalidate              eviction pub/sub (shared by everyone)

    Ground rules on a shared backbone

    Installing Valkey on premise

    Local development

    docker run -d -p 6379:6379 valkey/valkey:8.0

    Then spring.data.redis.host: localhost. The live initializer needs even less: its embedded backbone runs the engine in-process with no Redis at all.

    Kubernetes — the high-availability chart

    dac/pkgs/valkey-sentinel-chart is the chart the CNH on-premise environments use. It gives three Valkey nodes (one primary, two replicas), Sentinel failover, an append-only journal for queued writes and refresh events, and optional HAProxy and a key-inspection UI.

    helm install valkey dac/pkgs/valkey-sentinel-chart \
  --namespace valkey --create-namespace \
  --set auth.password=<a-strong-password> \
  --set persistence.size=2Gi
    valuedefaultmeaning
    replicaCount3nodes (each pod = one Valkey + one Sentinel); keep it odd, minimum 3
    auth.password / auth.existingSecret—the password, inline or from an existing Secret
    sentinel.quorum2Sentinels that must agree before a failover
    persistence.size2Gidisk per node for the journal
    valkey.extraConfig—extra server settings, e.g. maxmemory 256mb
    primaryProxy.enabledfalseHAProxy for clients outside the cluster (they then use host/port, not Sentinel)

    Applications connect through Sentinel so they follow a failover:

    spring:
  data:
    redis:
      sentinel:
        master: indc-master
        nodes: valkey-sentinel.valkey.svc:26379
      password: ${VALKEY_PASSWORD}

    Production checks

    Azure Managed Redis notes

    topicguidance
    ConnectionPlain host + port 10000 + TLS + key — no Sentinel block; Azure handles HA. Same engine, simpler config.
    LatencyEvery L2 hit crosses ExpressRoute (single-digit ms). L1 serves the hot set in-process, static regions live entirely in L1 after preload, warming fills L2 before readers ask.
    ACLsOne Redis user per application from day one.
    Evictionnoeviction; size the tier.
    Streams stay localCache regions move to Azure happily. Work queues — priority lanes, write-behind — stay on in-cluster Valkey; a blocking XREADGROUP over a WAN pays latency for nothing. An app using both defines two connection factories.
    SizingFull master-data sets (small) + active-user transactional entries × payload × TTL window (the real driver). Estimate before the tier is chosen.

    Observing it

    Safety principles

    Pre-production checklist