# Capabilities (your own programs)

A **capability** describes your software once: what hardware it needs, which files to download, how to install and start it, and what you are willing to pay. After that, renting a machine for it, or fifty, is one call with its id, on whichever provider has the cheapest machine that fits.

GPU Forge never interprets your commands or files. They run as root on your own machine.

| Endpoint | Purpose |
|---|---|
| `POST /v1/capabilities` | Create a capability |
| `GET /v1/capabilities` | List yours |
| `GET /v1/capabilities/{id}` | Read one |
| `PATCH /v1/capabilities/{id}` | Change settings, the spec, secrets, on/off, parked machines |
| `DELETE /v1/capabilities/{id}` | Delete one |
| `POST /v1/catalog/{program_id}/add` | Add one of our ready programs to your account as a capability |

## Create a capability

```
POST /v1/capabilities
```

| Field | Required | Description |
|---|---|---|
| `name` | yes | Your name for it. Free text |
| `user_id` | yes | Your account id (`GET /v1/cli/whoami`) |
| `description` | no | Free text |
| `spec` | yes | What it needs and how to run it (below) |
| `secrets` | no | Credentials and secret environment variables (below). Never readable back |

```bash
curl -s "https://api.gpuforge.io/v1/capabilities" \
  -H "Authorization: Bearer $GF_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "my-inference-server",
    "user_id": "usr_...",
    "spec": {
      "version": "1.0.0",
      "min_vram_gb": 48,
      "min_gpu_family": "ada_lovelace",
      "disk_gb": 100,
      "assets": [
        {"url": "https://my-bucket.example.com/server-1.0.0.tar.zst", "dest": "/opt/app", "unpack": true,
         "sha256": "9f2c..."},
        {"url": "https://my-bucket.example.com/weights.safetensors", "dest": "/opt/app/weights.safetensors",
         "sha256": "41ab..."}
      ],
      "install": "cd /opt/app && ./install.sh",
      "start": "cd /opt/app && exec ./serve --port 8000",
      "port": 8000,
      "ready_check": {"path": "/health/ready", "timeout_seconds": 900},
      "max_price_per_hour": 1.5,
      "idle_ttl_seconds": 900
    }
  }'
```

`201` returns the capability with its `capability_id` (`cap_...`). Nothing is rented yet.

## The spec

Required fields: `min_vram_gb`, `disk_gb`, `min_gpu_family`, `start`, `port`. To rent with `deploy` or `scale`, the spec also needs `max_price_per_hour`.

### Hardware

| Field | Default | Description |
|---|---|---|
| `min_vram_gb` | required | GPU memory per GPU, GB |
| `min_gpu_family` | required | `ada_lovelace`, `hopper` or `blackwell`: that generation or newer |
| `disk_gb` | required | The whole disk your machine needs, GB, rounded up to the next 50 GB. The container image uses part of it too. Files are downloaded all at once and then unpacked in order (each archive is deleted once extracted), so plan for *everything unpacked + every archive*, plus the image |
| `gpu_count` | `1` | GPUs on one machine, exact, 1–8. `max_price_per_hour` is for the whole machine |
| `min_ram_gb` | `0` | System RAM, GB. 0 = no filter |
| `min_cpu_ghz` | | Minimum CPU speed reported by the provider |
| `min_cuda_version` | | Lowest host driver CUDA version, e.g. `"12.8"`. Set it if your build needs a recent CUDA: an older host fails only after the machine is rented and your files are downloaded |
| `only_gpus` | | Card names to rent only, e.g. `["RTX PRO 6000"]`. Matched on words: case, order and vendor words are ignored |
| `excluded_gpus` | | Card names never to rent, same matching. `"rtx 5090"` blocks the RTX 5090 everywhere |

### Price and spending

| Field | Default | Description |
|---|---|---|
| `max_price_per_hour` | | The most one machine may cost you per hour. Required for `deploy` and `scale`. Checked before any provider is called |
| `surge_price_per_hour` | | A higher limit used only when nothing fits, or every machine refused, under `max_price_per_hour`. With `price_reduction` on, a surge-priced machine is moved to a cheaper one as soon as one appears |
| `daily_budget_usd` | | The most this capability may cost per UTC day (running and parked machines). Once reached, its machines are parked (or ended where parking is not possible) and new rentals and wakes are refused until 00:00 UTC. Spend is metered every minute, so it can pass the budget by about a minute of running cost |
| `interruptible` | | `true`: rent Vast.ai bid machines, 40–75% cheaper, but a higher bidder can pause the machine and the work on it is lost. For batch and development work, never for something a person is waiting on |
| `deploy_lane` | `standard` | How far above the cheapest fit an automatic deploy may go for a machine on a faster network line: `standard` 15%, `priority` 50%. A dearer machine is chosen only if it saves at least 60 seconds of downloading your files |
| `price_reduction` | off | `{"enabled": true, "min_saving_pct": 20, "min_saving_usd_hr": 0.13, "drain_max_seconds": 360}`: every few minutes, a running machine is compared with the market; if a machine that fits is cheaper by both amounts, it is rented, and once it is serving the old one is marked `retiring_at` (send it no new work) and ended when idle or after `drain_max_seconds` |

### Software

| Field | Default | Description |
|---|---|---|
| `version` | | Your label, e.g. `1.1` or a git commit. Each machine shows the version it was rented with as `capability_version` |
| `image` | provider default | A Docker image. Leave it unset for most programs (see "Images" below) |
| `assets` | | Files to download before install: `[{url, dest, sha256?, unpack?}]` (below) |
| `origins` + `files` | | The same files from a bucket per region (below). Use either `assets` or `origins` + `files` |
| `env` | | Environment variables for `install` and `start` (not secret; use `secrets.env` for secrets) |
| `install` | | Shell command run once after the files land, as root. Make it safe to run again. A wake skips it when it already succeeded on this disk for this version |
| `start` | required | Shell command that starts your program. It should stay in the foreground; GPU Forge runs and supervises it |
| `stop` | | Optional graceful shutdown command. Without it, the process group is ended |
| `port` | required | The port your program listens on inside the container. `capability_url` points here |
| `additional_ports` | | Other ports that must be reachable from outside. At most 4 ports in total, `port` included. A port you do not declare cannot be reached |
| `ready_check` | | How GPU Forge can tell your program really works (below). Without it, `ready` means only that the process started (`started_unverified`) |
| `busy_check` | | A JSON field on your port that counts running jobs: `{"path": "/status", "field": "jobs.running"}`. Shown as `busy` on each machine; used to drain a machine before it is replaced |
| `report` | | Values your machine publishes about itself, relayed on every fleet row under `report`: `{"path": "/status", "labels": {"model": "model.name"}, "numbers": {"queue": "queue.length"}}` |

### Idle and access

| Field | Default | Description |
|---|---|---|
| `idle_ttl_seconds` | `600` | Seconds without activity before a machine parks or ends. Cannot be switched off: 0 is stored as 3600 |
| `idle_by` | `traffic` | What counts as activity: `traffic` (network bytes or GPU use) or `gpu` (GPU use only, so an open SSH session alone does not keep a machine billing). A `busy_check` decides over both |
| `ssh_public_keys` | | OpenSSH public keys for root SSH on Vast.ai, RunPod and Nebius. Never send a private key. See [Machines](machines.md#ssh-access) |

### Files: `assets`

| Field | Required | Description |
|---|---|---|
| `url` | yes | Any HTTP(S) URL the machine can reach, usually your bucket. Machines download directly; the bytes never pass through GPU Forge |
| `dest` | yes | Absolute path where the file lands. Do not use `/workspace`: on some providers it is shared network storage, not your disk |
| `sha256` | no | Fingerprint, checked after download; install stops on a mismatch. Strongly recommended, and required for `update-capability` |
| `unpack` | no | `true`: extract a tar archive (optionally zstd or gzip) into `dest`, then delete the archive |

Files are downloaded in parallel. Several medium-sized files are easier to cache and republish than one huge file.

### Files: `origins` and `files`

One file list, served from a bucket per region. The machine's country picks the origin at rent time.

```json
{
  "origins": [
    {"name": "eu", "zones": ["europe"], "base_url": "https://eu-bucket.example.com/app/"},
    {"name": "us", "zones": ["north_america", "south_america", "any"], "base_url": "https://us-bucket.example.com/app/"}
  ],
  "files": [
    {"path": "server-1.0.0.tar.zst", "dest": "/opt/app", "unpack": true, "sha256": "9f2c..."}
  ]
}
```

Zones: `north_america`, `south_america`, `europe`, `asia` (including South-East Asia and Oceania), and `any` (the fallback). Each zone may appear in one origin only; a machine in a zone no origin covers is skipped unless one origin is `any`. `base_url` must end with `/`; a file's `path` must not start with `/`.

### `ready_check`

| Field | Default | Description |
|---|---|---|
| `path` | required | A path on your port that proves the program really works. Point it at something that exercises the real work, not a static health page |
| `method` | `GET` | HTTP method |
| `body` | | Request body, sent as is |
| `expect_status` | `200` | The status that counts as ready |
| `timeout_seconds` | `600` | How long after start to keep trying before calling it failed. Loading large weights can take minutes |

### Images

Leave `image` unset unless you need your own. The provider defaults (a Vast.ai base image and a RunPod PyTorch image) currently provide Ubuntu 24.04, CUDA 13 and Python 3.12, but they are not identical, so **bring everything your program needs as files** and treat the image as a floor. The one thing you cannot ship is the CUDA driver: use `min_cuda_version` for that.

On RunPod, GPU Forge's start-up replaces a custom image's own start command, so a custom image's entrypoint does not run unless your `start` runs it.

### What your program is told

These environment variables are set for `install` and `start`:

| Variable | Value |
|---|---|
| `GPUFORGE_MACHINE_ID` | The machine's `pod_id`. Use it as your worker id; never hard-code one |
| `GPUFORGE_CAPABILITY` | The capability's name |
| `GPUFORGE_PROVIDER` | The provider the machine is on |
| `GPUFORGE_INSTANCE_ID` | The provider's own id for the machine |

## Secrets

Secrets are kept **beside** the spec, never inside it. They are stored encrypted, never returned by any API call (a read shows only their names), and reach only your own machines.

| Field | Description |
|---|---|
| `secrets.env` | Secret environment variables, e.g. `{"HF_TOKEN": "..."}`. They are merged into `env` on the machine and win over a plain `env` value of the same name |
| `secrets.asset_auth` | A read-only credential for a **private** bucket: `{"type": "s3", "access_key_id": "...", "secret_access_key": "...", "region": "auto", "expires_seconds": 3600}`. Works with Amazon S3, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces, MinIO and Google Cloud Storage (HMAC). Use `region: "auto"` for R2 and the bucket's real region for S3. Add `session_token` only for a temporary credential |

How the bucket credential works: when a machine asks for its files, GPU Forge turns each file URL into a signed link valid for `expires_seconds` (default one hour; size it to your download). The credential itself never leaves GPU Forge. Give a read-only credential scoped to the one bucket. Prefer a permanent key: a temporary one (with `session_token`) expires, and then new machines cannot download their files.

Without `asset_auth`, a private bucket answers 403 to every download, after the machine is already rented and billing. A capability whose secrets cannot be read is refused before anything is rented.

Changing secrets:

- Leave `secrets` out of a `PATCH` and they stay as they are; a spec change never touches them.
- Sending `env` replaces all of `env` and keeps `asset_auth`, and the other way round.
- `{}` clears all secrets. Clearing is refused while machines depend on them.
- Running machines read secrets fresh: `POST /v1/fleet/{pod_id}/restart-capability` applies a rotated secret to a running machine.

A read returns a summary only:

```json
"secrets": {"set": true, "env_keys": ["HF_TOKEN"], "asset_auth": "s3", "asset_auth_temporary": false}
```

## Change a capability

```
PATCH /v1/capabilities/{capability_id}
```

Send only what you want to change:

| Field | Description |
|---|---|
| `settings` | Change single settings; the rest of the spec stays. Allowed: `idle_ttl_seconds`, `max_price_per_hour`, `surge_price_per_hour`, `daily_budget_usd`, `interruptible`, `only_gpus`, `excluded_gpus`, `min_gpu_family`, `idle_by`, `price_reduction`, `ssh_public_keys` |
| `spec` | The **whole** spec. It **replaces** the stored one; anything you leave out is deleted. Read the capability, change what you need, send all of it back. Not together with `settings` |
| `name`, `description` | Change on their own |
| `secrets` | See above |
| `active` | `false` switches the capability off: every new rental and wake is refused (`409`). Running machines continue until they idle out or you end them |
| `dormant_pods` | How many idle machines to keep **parked** instead of ending them (Vast.ai, Jarvis Labs, Nebius). New capabilities start at 1. Extra parked machines are ended |
| `skip_machines` | Vast.ai machine ids this capability must never use (a machine's `datacenter_id`). Sends the whole list; `[]` clears it |

```bash
curl -s -X PATCH "https://api.gpuforge.io/v1/capabilities/cap_ab12cd34" \
  -H "Authorization: Bearer $GF_KEY" -H "Content-Type: application/json" \
  -d '{"settings": {"idle_ttl_seconds": 1200, "max_price_per_hour": 2.5}}'
```

**Running machines are not changed.** A change applies to the next machine rented (and, for idle limits, to woken ones). To move a running machine onto your new spec, use `POST /v1/fleet/{pod_id}/update-capability`; to change one machine's idle limit, use keepalive. See [Machines](machines.md).

A change while a rental for the same capability is in progress answers `409`; try again in a moment.

## Delete a capability

```
DELETE /v1/capabilities/{capability_id}
```

Refused (`409`) while any machine runs it: destroy those first. There is no undo.

## Add a ready program to your account

```
POST /v1/catalog/{program_id}/add
```

Makes **your copy** of one of our ready programs: a capability on your account that runs our built program, starts from machines that already hold it, and bills your balance like any capability of yours.

```bash
curl -s -X POST "https://api.gpuforge.io/v1/catalog/<program id>/add" \
  -H "Authorization: Bearer $GF_KEY" -H "Content-Type: application/json" -d '{}'
```

`201`:

| Field | Description |
|---|---|
| `capability_id` | Your copy's id. Use it from now on |
| `name`, `version` | The program and its current version |
| `gf_api_key` | Your **program key** (`gfv_...`), **shown once**. Every request to the program on your machine needs `Authorization: Bearer <gf_api_key>` |
| `note`, `next` | What to do next |

Adding the same program twice answers `409` with your existing copy's id.

How a copy differs from your own capability:

- You choose its settings (price limits, daily budget, idle limit, GPUs, interruptible, price reduction), `active` and `dormant_pods`. Its files, commands and environment stay ours; a full `spec` you send keeps them as they are.
- The only secret you can set is the program key, `{"secrets": {"env": {"GF_API_KEY": "..."}}}`. Machines started after the change use the new key.
- New versions of the program reach your copy by themselves, keeping your settings. Parked machines take the new version at their next wake.
- A new copy keeps no parked machines (`dormant_pods: 0`): an idle machine is let go. Parking is your choice (`POST /v1/fleet/{pod_id}/park`).
- Parked machines of a ready program are shared between customers, and each start or wake empties the working folder. See [How GPU Forge behaves](principles.md#ready-programs-share-parked-machines).
- There is no SSH; use the program's own API.