AquinLabs

SSH fleet

Same train folder, remote boxes over SSH. Path: local → one place → pool. Change --on; jobs keep the same verbs. Places live in ~/.aquin/places.json. Remote work under <remoteDir>/jobs/<id>/.

Prerequisiteaq installed · SSH key access to the box · a local train folder (or any folder you sync)

1. Add a place

Register an SSH host. aq probes cpu/ram/gpu and stores capacity on the place.

shell
$aq add ssh temp
$aq places
$aq places --probe

2. Launch (or sync)

aq launch rsyncs the folder, sets up aq on the box, and opens an SSH shell. aq go reattaches. aq sync pushes without a shell. aq shutdown stops remote jobs and clears the session.

shell
$cd my-run
$aq launch --on temp
$aq go temp
$aq sync --on temp
$aq shutdown
$aq shutdown --wipe

3. Remote jobs

Background work on a place. aq jobs train is aq jobs run -- aq train. After start, aq jobs wait <id> polls status and log snapshots until exit (agents should use this — not bare logs -f).

shell
$aq jobs train --on temp
$aq jobs run --on temp -- sleep 30
$aq jobs
$aq jobs wait <id>
$aq jobs status <id>
$aq jobs logs <id> --once
$aq jobs logs <id> -f
$aq jobs pull <id>
 # job log/spec → jobs-pull/<id>/  +  remote artifacts/ → ./artifacts/
$aq jobs down <id>
python
from aquin import Place
p = Place("temp")
j = p.train()                 # or p.eval() / p.serve() / p.run(["sleep", "10"])
print(j.id, j.status()["status"])
print(j.logs())
j.pull()

4. Pools

A pool is a named set of SSH places. --on <pool> picks a free member (reachable, capacity, lowest live-job load).

shell
$aq add pool gpus box1 box2
$aq jobs train --on gpus
$aq jobs train --on gpus --nodes 2

--nodes N (N>1) starts the same command on N boxes with RANK / WORLD_SIZE / MASTER_ADDR / MASTER_PORT (torchrun-friendly). One job id covers the gang.

Recover (spot / preempt)

Restart the same job id after host or process death.

shell
$aq jobs recover <id>
$aq jobs recover <id> --next
$aq jobs recover <id> --on gpus

Managed watch

Auto-recover after flake or preempt. Pair --manage with aq jobs watch.

shell
$aq jobs run --on gpus --manage --retry 3 --prefer next -- aq train
$aq jobs watch
$aq jobs manage <id> --retry 5 --prefer same

Sweeps

Fan out many independent jobs. Use --shard N and/or --grid k=a,b. Command args may use {i} {n} {shard} {shards} and each grid key. Jobs are tagged sweep=<id>.

shell
$aq jobs sweep --shard 8 --on gpus -- aq train --shard {i}/{n}
$aq jobs sweep --grid lr=1e-3,1e-4 --grid wd=0,1e-4 --on gpus -- python train.py --lr {lr} --wd {wd}
$aq jobs list --tag sweep=<id>
python
from aquin import Place
p = Place("gpus")
out = p.sweep(["aq", "train", "--shard", "{i}/{n}"], shard=8, manage=True)
# out["sweep"], out["jobs"], out["total"]

Env on each job: AQ_SWEEP, AQ_SHARD, AQ_SHARDS, AQ_SWEEP_I, AQ_SWEEP_N, and AQ_<GRIDKEY>.

Queues

Pending work claimed onto SSH (priority). Worker claims and starts jobs on the place/pool.

shell
$aq queue add gpus --on gpus
$aq queue push gpus --priority 10 -- aq train
$aq queue worker gpus --once
python
from aquin import Queue
j = Queue("gpus").submit(["aq", "train"], priority=10)

Ports, tags, GPUs

ports · SSH -L tunnel
$aq port 8000 --on temp --bg
$aq jobs run --on temp --port 8000 -- python -m http.server 8000
$aq launch --on temp --port 6006
tags
$aq tag place temp team=ml
$aq tag job <id> exp=baseline
$aq places --tag team=ml
$aq jobs list --tag exp=baseline
shared GPUs
$aq jobs run --on box --gpu 1 -- python train.py
$aq jobs run --on box --devices 0,2 -- …
$aq places

--gpu N claims N free devices (CUDA_VISIBLE_DEVICES). aq places shows live load and free GPUs.

Flags cheat sheet

FlagWhereMeaning
--on <place|pool>launch, jobs, sync, port, queueTarget box or pool pick
--nodes Njobs run/train/…Multi-node gang on a pool
--gpu N / --devicesjobsClaim or pin GPUs
--port Nlaunch, go, jobsSSH -L; sets AQ_PORT / PORT
--tag k=vjobs, places listLabels / filters
--manage / --retry / --preferjobs run, sweepAuto-recover policy
--shard / --gridjobs sweepFan-out variants
--jsonjobs, recover, sweepMachine-readable (SDK)

Where to go next