SSH fleet
Same train folder, remote boxes over SSH. Path: local → one place → pool. Change --on; jobs keep the same verbs. Places live in ~/.aquin/places.json. Remote work under <remoteDir>/jobs/<id>/.
1. Add a place
Register an SSH host. aq probes cpu/ram/gpu and stores capacity on the place.
$aq add ssh temp
$aq places
$aq places --probe
2. Launch (or sync)
aq launch rsyncs the folder, sets up aq on the box, and opens an SSH shell. aq go reattaches. aq sync pushes without a shell. aq shutdown stops remote jobs and clears the session.
$cd my-run
$aq launch --on temp
$aq go temp
$aq sync --on temp
$aq shutdown
$aq shutdown --wipe
3. Remote jobs
Background work on a place. aq jobs train is aq jobs run -- aq train. After start, aq jobs wait <id> polls status and log snapshots until exit (agents should use this — not bare logs -f).
$aq jobs train --on temp
$aq jobs run --on temp -- sleep 30
$aq jobs
$aq jobs wait <id>
$aq jobs status <id>
$aq jobs logs <id> --once
$aq jobs logs <id> -f
$aq jobs pull <id>
# job log/spec → jobs-pull/<id>/ + remote artifacts/ → ./artifacts/
$aq jobs down <id>
from aquin import Place
p = Place("temp")
j = p.train() # or p.eval() / p.serve() / p.run(["sleep", "10"])
print(j.id, j.status()["status"])
print(j.logs())
j.pull()
4. Pools
A pool is a named set of SSH places. --on <pool> picks a free member (reachable, capacity, lowest live-job load).
$aq add pool gpus box1 box2
$aq jobs train --on gpus
$aq jobs train --on gpus --nodes 2
--nodes N (N>1) starts the same command on N boxes with RANK / WORLD_SIZE / MASTER_ADDR / MASTER_PORT (torchrun-friendly). One job id covers the gang.
Recover (spot / preempt)
Restart the same job id after host or process death.
$aq jobs recover <id>
$aq jobs recover <id> --next
$aq jobs recover <id> --on gpus
Managed watch
Auto-recover after flake or preempt. Pair --manage with aq jobs watch.
$aq jobs run --on gpus --manage --retry 3 --prefer next -- aq train
$aq jobs watch
$aq jobs manage <id> --retry 5 --prefer same
Sweeps
Fan out many independent jobs. Use --shard N and/or --grid k=a,b. Command args may use {i} {n} {shard} {shards} and each grid key. Jobs are tagged sweep=<id>.
$aq jobs sweep --shard 8 --on gpus -- aq train --shard {i}/{n}
$aq jobs sweep --grid lr=1e-3,1e-4 --grid wd=0,1e-4 --on gpus -- python train.py --lr {lr} --wd {wd}
$aq jobs list --tag sweep=<id>
from aquin import Place
p = Place("gpus")
out = p.sweep(["aq", "train", "--shard", "{i}/{n}"], shard=8, manage=True)
# out["sweep"], out["jobs"], out["total"]
Env on each job: AQ_SWEEP, AQ_SHARD, AQ_SHARDS, AQ_SWEEP_I, AQ_SWEEP_N, and AQ_<GRIDKEY>.
Queues
Pending work claimed onto SSH (priority). Worker claims and starts jobs on the place/pool.
$aq queue add gpus --on gpus
$aq queue push gpus --priority 10 -- aq train
$aq queue worker gpus --once
from aquin import Queue
j = Queue("gpus").submit(["aq", "train"], priority=10)
Ports, tags, GPUs
$aq port 8000 --on temp --bg
$aq jobs run --on temp --port 8000 -- python -m http.server 8000
$aq launch --on temp --port 6006
$aq tag place temp team=ml
$aq tag job <id> exp=baseline
$aq places --tag team=ml
$aq jobs list --tag exp=baseline
$aq jobs run --on box --gpu 1 -- python train.py
$aq jobs run --on box --devices 0,2 -- …
$aq places
--gpu N claims N free devices (CUDA_VISIBLE_DEVICES). aq places shows live load and free GPUs.
Flags cheat sheet
| Flag | Where | Meaning |
|---|---|---|
| --on <place|pool> | launch, jobs, sync, port, queue | Target box or pool pick |
| --nodes N | jobs run/train/… | Multi-node gang on a pool |
| --gpu N / --devices | jobs | Claim or pin GPUs |
| --port N | launch, go, jobs | SSH -L; sets AQ_PORT / PORT |
| --tag k=v | jobs, places list | Labels / filters |
| --manage / --retry / --prefer | jobs run, sweep | Auto-recover policy |
| --shard / --grid | jobs sweep | Fan-out variants |
| --json | jobs, recover, sweep | Machine-readable (SDK) |