Cluster broadcast & health

Ask every pod at once and get one consolidated answer back — the pattern behind the built-in cluster health check, available for any function you annotate.

On this page

    The pattern

    Feature

    Some questions are about the cluster, not about one pod behind the load balancer: is every pod healthy, what does each pod hold in its near cache, which config did each pod actually load? The broadcast bus answers them with scatter-gather over the backbone: one request is published on {root}:channel:broadcast, Redis reports how many pods received it, every pod (the caller included) runs the function and pushes its reply onto a short-lived list, and the caller consolidates the replies into one result. No pod list, no service discovery — the subscriber count is the pod count.

    The consolidated health check

    The built-in health function is registered on every pod automatically. Expose the endpoint (management.endpoints.web.exposure.include: …,dachealth) and one HTTP call to any pod checks all of them:

    {
      "status": "UP",                    // UP · DOWN (a pod reported a problem) · UNKNOWN (replies missing)
      "expectedPods": 3,
      "respondedPods": 3,
      "nodes": [
        { "node": "5f2c…", "ok": true, "details": { "application": "indc-core", "status": "UP" } },
        { "node": "a91b…", "ok": true, "details": { "application": "indc-core", "status": "UP" } },
        { "node": "c07d…", "ok": true, "details": { "application": "indc-core", "status": "UP" } }
      ]
    }

    Each pod's status comes from its actuator HealthEndpoint when the application has one — so a pod whose database check fails reports DOWN here too — and is reported as UP otherwise (a pod that answers a broadcast is at least alive).

    Broadcasting your own functions

    Any bean method becomes cluster-callable with one annotation. The method takes no parameter or one String payload; it returns a String or any JSON-serialisable object:

    @DacBroadcast("l1-stats")
    public Map<String, Long> l1Stats() {          // runs on EVERY pod when broadcast
        return cacheManager.regions().stream()
                .collect(toMap(CacheRegion::name, r -> r.stats().l1Size()));
    }
    @Autowired Broadcaster broadcaster;
    
    BroadcastResult result = broadcaster.broadcast("l1-stats", "", Duration.ofSeconds(5));
    result.replies().forEach(reply ->
            log.info("pod {} -> {}", reply.nodeId(), reply.body()));
    // result.complete()  — did every pod that received the request answer in time?
    // result.allOk()     — did every reply succeed?

    Rules and guarantees

    aspectbehaviour
    scopeAll pods sharing one keyspace root — normally the pods of one application. Pods mounted on a shared domain root broadcast across every application on that domain.
    completenessexpectedReplies comes from the pub/sub subscriber count at publish time. A pod that fails mid-call simply never replies: the result returns at the timeout with complete() == false.
    unknown functionA pod without the function still replies — with an error — so the caller is never left waiting for a counted subscriber.
    failuresA function that throws becomes an error reply for that pod only; allOk() turns false, the other replies are unaffected.
    executionFunctions run on a dedicated single thread per pod, never on the pub/sub listener thread — a slow handler cannot delay invalidation delivery.
    hygieneReply lists live under {root}:broadcast:reply:{id}, are deleted after consolidation and expire after 60 s regardless. Keep payloads and replies small and free of client data — they cross the shared backbone like every other key.
    what it is notNot a work queue and not guaranteed delivery — it is a live question to whoever is up right now. For "exactly one pod must act" use refresh events; for "must survive a restart" use the write-behind stream.
    More information

    Switches and timeouts live under dac.cache.broadcast — see the configuration reference. For waiting on distributed work instead of asking about it, see Locks, semaphores & latches.