Set up a relay for CosmicAC

Set up a relay so CosmicAC reaches its job agents on a worker network that blocks direct connections.

Run a relay and point CosmicAC at it, so job agents reach your deployment on a worker network that blocks direct connections.

Most deployments never need one. CosmicAC's components normally connect to each other directly over the Holepunch peer-to-peer stack. Set up a relay only when you hit either of these:

Communication failure: the Docker agent, cosmicac-wrk-agent-inference or cosmicac-wrk-agent-instance, can't communicate with cosmicac-wrk-server-k8s-nvidia. Job statuses stop updating, and a job reports that its last communication is minutes old. CosmicAC can also create a job successfully, and the job then stays in a creating or starting state.
Access issues: users report that they can't access GPU containers on a specific worker network.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
A host that runs Node.js and npm, reachable from your deployment host and from the job virtual machines on your cluster.
An open UDP port on that host, with no firewall in front of it. The relay listens on 49737 unless you set another port.
At most 200 MB of RAM and minimal CPU on that host, which is what a relay uses in production.

Install the relay service

On the relay host, install blind-relay-service globally:

npm install -g blind-relay-service

Start the relay

Run the relay, giving it a permanent directory for its state:

blind-relay --storage /var/lib/blind-relay

To bind a different UDP port, add --port <num>.

Keep the storage directory

The relay derives its key from the corestore in this directory. A relay that starts without that directory generates a new key, which no longer matches the key you set in the next step, and jobs stop reaching it. Back the directory up along with the rest of your deployment state. Without --storage, the relay uses ./corestore in the directory you run the command from.

The relay prints its storage path and the key it listens on:

Using corestore storage at /var/lib/blind-relay
Server listening on <relay-key>

Copy the key from the second line. The next step needs it.

Set the relay key

On your deployment host, set the key in .env so cosmicac-wrk-server-k8s-nvidia passes it to the jobs it creates:

K8S_COMMON_CONFIG__interconnect__relays='["<relay-key>"]'

The value is a JSON array, so it takes more than one key if you run more than one relay. For the variable form, see Config overrides.

Apply the change and restart the worker

Write the value into the worker's config, then restart the worker so it picks the value up:

task apply-wrk-server-k8s-nvidia-common-config
task restart SERVICES="cosmicac-wrk-server-k8s-nvidia"

Existing jobs keep the settings they started with

Only jobs created after this restart use the relay. The worker passes the relay key into a job when it creates that job, so a job that is already running never receives it. Delete each job that was stuck with cosmicac jobs delete <jobId>, then create it again.

Confirm a new job reports its status

Create a job, then list your jobs:

cosmicac jobs list

The new job leaves the creating state, and its status keeps updating.

Help and troubleshooting

A job that was stuck before the restart hasn't recovered

That job was created before the relay existed, so it never received the relay key. Delete it with cosmicac jobs delete <jobId>, then create it again.