Tiles, devices and the network
One par block can run on many devices at once. The runtime cuts it into tiles and
hands them to every device that can take one: the CPU’s cores, each local GPU whatever its
vendor, and GPUs on other machines. A remote GPU is one more worker in the same queue, so
the program does not change when the network does: transparent, scalable network
tiling.
The compiler writes each array’s footprint per item into the block, so the runtime knows which slice of each array a tile reads and writes, and moves only that slice to a GPU across a bus or a network.
The tile size follows the devices: large enough that a GPU’s per-tile cost is a small share of the run, small enough that every worker gets several tiles, and, for a remote GPU, large enough that the link’s latency stays a small fraction of a tile.
Workers and scheduling
Section titled “Workers and scheduling”| Worker | Tiles in flight |
|---|---|
| each CUDA card | 3 |
| a Vulkan GPU, any vendor | 3 |
| the Metal GPU | 3 |
| a remote GPU | the depth its daemon offers |
| the CPU pool, every core | one tile at a time, split across the cores |
Each tile goes to the worker with the earliest estimated finish, from the rate it has shown so far; a tile may wait for a faster worker rather than go to a slower free one. Rates start from the run-time measurement. CUDA cards, a Vulkan GPU from another vendor, Metal, the CPU and remote daemons all serve one queue in the same run, so an NVIDIA card through CUDA and another through Vulkan work on the same block together.
Results are deterministic. Reductions from each tile are folded in tile order, whichever device ran it, so the answer does not depend on which devices took part. A lost tile is re-issued to another worker before any new tile.
xcd: GPUs over the network
Section titled “xcd: GPUs over the network”xcd is a daemon that serves a machine’s GPUs to programs elsewhere on the network.
- Discovery: named by address, or found on the local network by mDNS
(
_xcd._tcp), within a named pool and the same compiler version. - Security: every connection is TLS; the program pins the daemon’s certificate on first use and refuses a changed one, and a shared secret admits a program.
- Compatibility: program and daemon agree the protocol, the compiler version, and the data layout (byte order and pointer width) before any work moves.
- Caching: kernels are sent once per connection; arrays and footprint slices are named by hash, and the daemon keeps them, so a second pass over the same data sends almost nothing. Slices it wrote stay cached under their new hash.
- Fitting: each tile is checked against the card’s free memory, and a daemon whose card is full is given no more tiles.
- Checking: for integer kernels, the first tile from each daemon and every sixteenth after it are re-run on the local CPU and compared; a daemon that disagrees is dropped and the local result kept.
- Liveness: a heartbeat every second; a daemon that stops answering is dropped and its tiles go to other workers.
The run-time measurement includes remote GPUs as a candidate of their own, so a program uses the network only where it is faster. See Performance: networked for figures, including a Raspberry Pi 5 using a remote RTX 5080.