# Machines

A **machine** is a rented GPU computer running one program: one of our ready programs or one of your own capabilities. The API calls a machine a **pod**; each one has a unique `pod_id` such as `my-worker-3f9a1c`.

Every machine runs a program. To rent a plain machine for interactive work, make a capability whose `start` keeps running (for example `sleep infinity`) and rent that; see [Capabilities](capabilities.md).

| Endpoint | Purpose | Costs money |
|---|---|---|
| `POST /v1/offers/search` | Find machines that fit | No |
| `POST /v1/capabilities/{id}/deploy` | Rent the cheapest machine that fits a capability | **Yes** |
| `POST /v1/capabilities/{id}/scale` | Run N machines of a capability | **Yes** |
| `POST /v1/deploy` | Rent one offer you picked | **Yes** |
| `GET /v1/fleet` | Your machines | No |
| `GET /v1/fleet/{pod_id}` | One machine | No |
| `GET /v1/fleet/{pod_id}/logs` | Startup output, events and the last error | No |
| `GET /v1/fleet/{pod_id}/output` | Full program output (ready programs only) | No |
| `GET /v1/fleet/{pod_id}/stop-choices` | What parking would cost per day | No |
| `POST /v1/fleet/{pod_id}/park` | Stop it, keep its disk | Storage only |
| `POST /v1/fleet/{pod_id}/wake` | Start a parked machine | **Yes** |
| `POST /v1/fleet/{pod_id}/release` | Done for now: park if possible, else end | No |
| `POST /v1/fleet/{pod_id}/destroy` | End it, disk included | No |
| `POST /v1/fleet/{pod_id}/keepalive` | Change its idle limit and restart the countdown | Keeps billing |
| `POST /v1/fleet/{pod_id}/restart-capability` | Restart your software in place | No |
| `POST /v1/fleet/{pod_id}/update-capability` | Roll your latest capability onto it in place | No |
| `POST /v1/fleet/{pod_id}/reboot` | Restart the container in place (Vast.ai) | No |
| `GET /v1/fleet/{pod_id}/ssh` | SSH status and command | No |
| `POST /v1/fleet/{pod_id}/ssh/keys` | Add an SSH public key | No |
| `DELETE /v1/fleet/{pod_id}/ssh/keys/{key_id}` | Remove an SSH key | No |
| `GET /v1/gpu-families` | GPU generations and what they include | No |
| `GET /v1/clusters/options`, `POST /v1/clusters`, ... | Multi-node clusters | **Yes** |

Units everywhere: sizes in **gigabytes**, prices in **US dollars per hour**.

## Search for machines

```
POST /v1/offers/search
```

Searches every provider at once and returns every machine that fits, cheapest first. Searching is free and rents nothing.

You can describe the hardware directly, or send a `capability_id` and the requirements are read from that capability. Anything you send explicitly wins over the capability.

### Request fields

| Field | Default | Description |
|---|---|---|
| `capability_id` | | Take the requirements from this capability. Give this **or** the three fields below |
| `min_vram_gb` | | GPU memory **per GPU**, GB. A card's listed memory is rounded up to the next whole GB before comparing |
| `disk_gb` | | Working disk beyond the container image, GB. Rounded up to the next 50 GB step |
| `min_gpu_family` | | Oldest GPU generation: `ada_lovelace`, `hopper` or `blackwell`. You get that generation and newer. Older generations and non-NVIDIA cards are never returned |
| `cluster_size` | `1` | **Exact** number of GPUs on one machine, 1–8 |
| `min_ram_gb` | `0` | System RAM, GB. 0 = no filter |
| `min_cpu_ghz` | | Minimum CPU speed reported by the provider |
| `max_price_per_hour` | | Most you will pay for the whole machine |
| `only_gpus` | | Card names to return only, e.g. `["RTX PRO 6000"]`. Matched on words, so case, order and vendor words do not matter |
| `excluded_gpus` | | Card names never to return, same matching |
| `country_codes` | | ISO country codes, e.g. `["US", "DE"]` |
| `min_cuda_version` | | Lowest host driver CUDA version, e.g. `"12.8"` |
| `min_reliability` | | Host reliability floor 0–1 (Vast.ai only) |
| `providers` | all | Any of `vast`, `runpod`, `nebius`, `jarvis` |
| `secure_cloud_only` | `true` | Only datacenter-grade hosts. `false` also returns community hosts, which are cheaper and less accountable |
| `interruptible` | `false` | Vast.ai bid machines: much cheaper, but can be paused by a higher bidder and the running work lost |
| `sort` | `price` | `price`, `price_per_vram`, `vram` or `reliability` |
| `limit` | `20` | 1–200 |
| `include_candidates` | `true` | Also return machines with a requirement the provider cannot confirm before boot |
| `ssh_public_keys` | | Only return providers that support managed SSH |

```bash
curl -s "https://api.gpuforge.io/v1/offers/search" \
  -H "Authorization: Bearer $GF_KEY" -H "Content-Type: application/json" \
  -d '{"min_vram_gb": 80, "disk_gb": 100, "min_gpu_family": "hopper", "limit": 5}'
```

### Response

| Field | Description |
|---|---|
| `offers` | Machines where **every** requirement was verified |
| `candidates` | Machines that match everything the provider can prove; each lists its `unverified_constraints` (for example RAM on providers that do not publish it). Verify those after boot |
| `total`, `total_candidates` | Counts |
| `truncated` | More machines fit than are listed (the dearer ones were dropped) |
| `expires_at` | When the `offer_id`s in this answer stop working (about 15 minutes) |
| `provisioned_disk` | The disk size actually used after rounding |
| `provider_errors` | Providers that failed. You still get the others' results |

Each offer has `offer_id`, `provider`, `gpu` (`name`, `family`, `count`, `vram_gb_per_gpu`, `vram_gb_total`), `host` (`cpu_cores`, `ram_gb`, `disk_available_gb`, `cuda_version`), `location` (`country_code`), `price`, `availability`, `secure_cloud`, `reliability`, `inet_down_mbps` and `interruptible`.

**Price:** `price.customer_usd_hr` is what you pay per hour for that machine, locked when you rent it. When it is `null`, you pay `price.total_usd_hr`. Your `max_price_per_hour` is compared with the price you pay.

A `null` field means the provider does not publish that value, not that it is zero.

## Rent the cheapest machine for a capability

```
POST /v1/capabilities/{capability_id}/deploy
```

The capability already says what it needs and what it is worth per hour, so no body is needed. GPU Forge first **wakes a parked machine** of this program if one is free (seconds). Otherwise it searches every provider and rents the cheapest machine that fits under the price limit, moving to the next cheapest if a provider refuses.

| Field (all optional) | Description |
|---|---|
| `max_price_per_hour` | Overrides the capability's limit for this rental. Without either, the deploy is refused (`400`) |
| `name` | Label for the machine. Default: the capability's name |
| `providers` | Restrict to these providers (skips waking a parked machine) |
| `country_codes` | Restrict to these countries (skips waking a parked machine) |
| `idle_ttl_seconds` | Overrides the capability's idle limit for this machine |
| `ssh_public_keys` | Public keys for SSH (your own capabilities only) |

```bash
curl -s -X POST "https://api.gpuforge.io/v1/capabilities/cap_ab12cd34/deploy" -H "Authorization: Bearer $GF_KEY"
```

```json
{
  "pod_id": "hello-gpu-3f9a1c",
  "capability_id": "cap_ab12cd34",
  "provider": "vast",
  "price_usd_hr": 0.48,
  "gpu_name": "RTX 4090",
  "country_code": "DE",
  "offers_considered": 37,
  "attempts": [ ... ],
  "note": "..."
}
```

The answer comes as soon as the machine is **rented**, not when it is ready. If the capability has a `surge_price_per_hour` and every machine under its normal limit refused, the deploy tries once more up to the surge price.

## Run N machines

```
POST /v1/capabilities/{capability_id}/scale
```

`desired_pods` is a **target**, not an increment: asking for 3 when 1 is running starts 2; asking again starts nothing. Parked machines are woken first, and only the rest is rented.

| Field | Description |
|---|---|
| `desired_pods` | Required. How many machines you want running in total (0–500) |
| `min_pods` | The fewest worth having. If fewer can be had, what was rented is ended again (and billed for the seconds it lived) |
| `max_price_per_hour` | Overrides the capability's limit for every machine in this call |
| `country_codes` | Restrict to these countries |
| `idle_ttl_seconds` | Overrides the capability's idle limit |
| `ssh_public_keys` | Public keys for SSH |

The answer has `running_before`, `newly_rented`, `running_now`, `short_by` (how many the providers could not supply), `pods` (each new machine), `woken` and `woken_pods`.

`scale` never ends machines: a target below what is running rents nothing and ends nothing, because a machine in the middle of a request is not GPU Forge's to stop. Machines leave on their idle limit, or when you release or destroy them. A short fill is normal when providers run out: send the same target again later and it fills in. Only one scale or deploy per capability runs at a time; a second one at the same moment answers `409` (try again in a moment).

## Rent an offer you picked

```
POST /v1/deploy
```

| Field | Required | Description |
|---|---|---|
| `offer_id` | yes | Copied exactly from a search answer. Expires about 15 minutes after the search (`400` after that: search again) |
| `capability_id` | yes | The capability this machine will run |
| `max_price_per_hour` | recommended | Checked against the offer's price before any provider is called; above it answers `409` and nothing is rented |
| `name` | no | Label; a random suffix is added to make the `pod_id` |
| `idle_ttl_seconds` | no | Idle limit for this machine (default 600) |
| `ssh_public_keys` | no | Public keys for SSH |
| `env` | no | Extra environment variables. These reach the provider; use capability secrets for anything sensitive |

The answer has `pod_id`, `provider`, `instance_id`, `price_usd_hr`, `image` and notes on what still has to happen.

## Watch your machines

```
GET /v1/fleet
GET /v1/fleet/{pod_id}
```

`GET /v1/fleet` lists everything that could be costing you money right now, newest first: running, starting, stopping and parked machines. Filters: `capability_id`, `state`, `provider`. `?include=all` also returns ended machines; `?include=telemetry` adds each machine's last heartbeat (combine with a comma).

`GET /v1/fleet/{pod_id}` returns one machine with its last heartbeat in full (`beat`: GPU use, temperature, disks and more).

### Fields you will use most

| Field | Description |
|---|---|
| `pod_id` | The machine's id |
| `capability_id`, `capability_name`, `capability_version` | What it runs (name and version as they were at rent time) |
| `state` | `deploying`, `active`, `stopped` (with `parked_at` set: parked; without it: stopped by the provider, kept as it is until you start it again or delete it, and you get a notification), `failed` or `terminated` |
| `capability_state` | Your program: `fetching` → `downloading` → `installing` → `starting` → `ready`, or `failed`. `started_unverified` = launched but the capability has no `ready_check` |
| `status` | One sentence for a person: what the machine is doing now and what to expect |
| `progress` | Six steps that only move forward: `renting`, `booting`, `downloading`, `unpacking`, `installing`, `starting` |
| `capability_error` | The failure in full, only when `capability_state` is `failed` |
| `capability_url` | Where your program answers (HTTPS when available) |
| `capability_urls` | One plain HTTP address per declared port |
| `capability_reachable` | Why `capability_url` can or cannot be relied on yet |
| `health` | `ok`, `warning` (missed two heartbeats) or `investigating` (silent for about two minutes; GPU Forge is asking the provider) |
| `idle_ttl_seconds` | The machine parks or ends itself after this many seconds with no activity |
| `provider`, `country_code`, `rented_gpu_name`, `gpu_name`, `gpu_count`, `vram_gb_per_gpu`, `disk_free_gb` | Where it runs and what it has (`gpu_name` and disk are measured on the machine) |
| `sell_usd_hr` | What you pay per hour while it runs, locked at rent. When `null`, you pay `price_usd_hr` |
| `parked_at` | Set while the machine is parked |
| `created_at`, `capability_ready_at`, `destroyed_at`, `destroyed_reason` | Its timeline. `destroyed_reason` is one word, e.g. `requested`, `idle` |

What each machine costs you right now is also in `GET /v1/billing` → `machines[].cost_usd_hr`.

### Waiting for ready

Poll `GET /v1/fleet/{pod_id}` every 10 seconds until `capability_state` is `ready`, and pass `status` on to your user while you wait.

- A machine that already holds the program: usually 1–3 minutes. A woken parked machine: about 20–60 seconds plus the program's start.
- A fresh machine: download and install time, up to 15–20 minutes for large programs. Wait at least 20 minutes before giving up.
- `ready` is re-checked on every heartbeat: a program that exits later moves to `failed`.
- If `capability_state` is `failed` or the machine is gone (`destroyed_at` set), read the logs, then deploy again.

### Logs and output

```
GET /v1/fleet/{pod_id}/logs
```

Returns `last_seen_at`, `events`, `error` and `startup` (recent output of your program, separate from downloads). It is the last report the machine sent, readable even after the machine has ended; it is not a live stream.

```
GET /v1/fleet/{pod_id}/output?since={time}&limit={n}
```

For machines running our ready programs: everything the program printed, kept 30 days. Returns `pieces` (`at`, `run`, `text`) and `next` (the `since` value for the next page).

## Stop, park, wake

| Action | What happens | Disk | Billing after |
|---|---|---|---|
| `POST .../park` | Stops the machine now and keeps its disk | Kept | Storage price only |
| `POST .../wake` | Starts a parked machine | Kept | Full price |
| `POST .../release` | Done for now: parks it if its program keeps parked machines (`dormant_pods`) and has a free slot, otherwise destroys it | Kept or deleted | Storage or nothing |
| `POST .../destroy` | Ends the rental for good | **Deleted** | Nothing |

Send these as `POST` with an empty JSON body `{}`. Destroy is `POST /v1/fleet/{pod_id}/destroy`, not `DELETE`.

### Park

```bash
curl -s "https://api.gpuforge.io/v1/fleet/<pod_id>/stop-choices" -H "Authorization: Bearer $GF_KEY"
```

```json
{
  "pod_id": "hello-gpu-3f9a1c",
  "park": {"allowed": true, "refusal": null, "usd_hr": 0.012, "usd_day": 0.288, "released_after_unused_days": 3},
  "release": {"usd_day": 0.0, "note": "The machine and its disk end now; the next start rents a fresh one."}
}
```

```bash
curl -s -X POST "https://api.gpuforge.io/v1/fleet/<pod_id>/park" -H "Authorization: Bearer $GF_KEY" \
  -H "Content-Type: application/json" -d '{}'
```

- Parking is possible on Vast.ai, Jarvis Labs and Nebius, when the program is switched on and the machine runs the program's current version and disk size. Otherwise it answers `409` with the reason.
- Parking by hand raises the program's `dormant_pods` by one if there was no free slot.
- The provider can take up to a minute to finish stopping the machine.
- A parked machine unused for 3 days is released; you are notified a day before.

### Wake

```bash
curl -s -X POST "https://api.gpuforge.io/v1/fleet/<pod_id>/wake" -H "Authorization: Bearer $GF_KEY" \
  -H "Content-Type: application/json" -d '{}'
```

You rarely need this: `deploy` and `scale` wake parked machines first. A wake that produces no heartbeat within 240 seconds is parked again; after two failed wakes the machine is ended. Read `capability_url` again after every wake.

### Destroy

```bash
curl -s -X POST "https://api.gpuforge.io/v1/fleet/<pod_id>/destroy" -H "Authorization: Bearer $GF_KEY" \
  -H "Content-Type: application/json" -d '{}'
```

GPU Forge asks the provider to end the machine and records it gone only once the provider confirms. Until then the answer is `502`, the request is retried automatically, and the machine stays visible in your fleet, because it may still be billing. Copy anything you need off the disk first.

## Idle shutdown and keepalive

Every machine has an idle limit (`idle_ttl_seconds`, default 600 seconds). After that long without activity it parks (if its program keeps parked machines and the provider can park) or ends.

- Activity is network traffic or GPU use, measured on the machine. With the capability setting `idle_by: "gpu"`, only GPU use counts, so an open SSH session alone does not keep a machine running.
- The clock starts only when your program is up.
- The limit cannot be switched off; a value of 0 is stored as at least one hour.

To keep one running machine longer, or restart its countdown:

```bash
curl -s -X POST "https://api.gpuforge.io/v1/fleet/<pod_id>/keepalive" \
  -H "Authorization: Bearer $GF_KEY" -H "Content-Type: application/json" \
  -d '{"idle_ttl_seconds": 3600, "reason": "long training run"}'
```

```json
{"pod_id": "hello-gpu-3f9a1c", "idle_ttl_seconds": 3600, "previous_idle_ttl_seconds": 600, "idle_reset_at": "...", "note": "..."}
```

The machine keeps billing for as long as it runs. Changing a capability's idle limit affects new and woken machines, not running ones; keepalive is how you change a running one. Answers `409` if the machine is already being shut down or is stopped.

## Fix or update a running machine

These keep the same rental, price and disk. Each order is picked up on the machine's next heartbeat (a few seconds), so they work only on a machine that is still reporting.

| Endpoint | Use it when | What happens |
|---|---|---|
| `POST .../restart-capability` | Your program crashed but the machine is fine | Your program is stopped and started again from what is on disk. Secrets and env are re-read, so this also applies a rotated secret |
| `POST .../update-capability` | You changed your capability and want this machine on the new version | The machine takes the current capability, downloads only files whose `sha256` changed, re-runs `install` and starts. Every file needs a `sha256` (`422` otherwise). In-flight requests are cut: update one machine at a time |
| `POST .../reboot` | The machine is stuck | The container restarts on the same machine; only files not already verified on disk are downloaded. Vast.ai only |

Your own capabilities only; machines of our ready programs are updated by GPU Forge.

## SSH access

SSH is available for your own capabilities on Vast.ai, RunPod and Nebius. It is not available on machines of our ready programs.

1. **Rent with your public key.** Put it in the capability's `ssh_public_keys`, or send `ssh_public_keys` on deploy, scale or `POST /v1/deploy`. Never send a private key. A machine rented without keys has no managed SSH port.
2. **Read the connection details:**

   ```bash
   curl -s "https://api.gpuforge.io/v1/fleet/<pod_id>/ssh" -H "Authorization: Bearer $GF_KEY"
   ```

   ```json
   {
     "pod_id": "hello-gpu-3f9a1c",
     "state": "configured",
     "host": "203.0.113.10",
     "port": 41022,
     "username": "root",
     "command": "ssh -p 41022 root@203.0.113.10",
     "host_public_key": "ssh-ed25519 AAAA...",
     "host_key_fingerprint": "SHA256:...",
     "keys": [{"key_id": "<64 hex characters>", "public_key": "ssh-ed25519 AAAA... you@laptop"}],
     "generation": "...",
     "note": "Key changes affect new connections; already-open SSH sessions remain open. ..."
   }
   ```

   `state` is `disabled` (rented without keys), `pending` (being applied; poll again), `configured`, `error` or `unavailable` (the machine is stopped, ended or silent). Pin `host_public_key` in your `known_hosts` to verify the machine.

3. **Add or remove keys** on a running machine:

   ```bash
   curl -s -X POST "https://api.gpuforge.io/v1/fleet/<pod_id>/ssh/keys" \
     -H "Authorization: Bearer $GF_KEY" -H "Content-Type: application/json" \
     -d '{"public_key": "ssh-ed25519 AAAA... teammate@laptop"}'

   curl -s -X DELETE "https://api.gpuforge.io/v1/fleet/<pod_id>/ssh/keys/<key_id>" -H "Authorization: Bearer $GF_KEY"
   ```

   Changes apply to new connections once `state` is `configured` again. Sessions already open stay open.

The CLI does all of this with `gpuforge ssh <pod_id>`.

## What survives what

| Event | GPU billing | Files on disk | Installed software |
|---|---|---|---|
| Park | Stops (storage price only) | Kept (ready programs: working folder emptied at the next start or wake) | Kept |
| Wake | Resumes | Kept | Kept; `install` is skipped when it already succeeded for this version |
| Idle limit reached | Stops | Kept if parked, deleted if ended | Same |
| `restart-capability` | Continues | Kept | Kept |
| `update-capability` | Continues | Kept; changed files replaced | Re-installed |
| `reboot` (Vast.ai) | Continues | Kept | Re-installed from verified files |
| Release | Stops | Kept if parked, deleted if ended | Same |
| Destroy | Stops | **Deleted** | Deleted |
| Balance at $0 past the grace period | Stops | **Deleted** | Deleted |
| Parked and unused for 3 days | Stops | **Deleted** | Deleted |
| Provider takes the machine back (interruptible, or a failure) | Stops | Lost | Lost |
| Parked machine's GPU given to someone else | – | The wake fails and a new machine is rented | – |

Keep your own copy of anything that matters. Machines are rented from GPU marketplaces and can fail or be reclaimed.

## GPU families

```
GET /v1/gpu-families
```

Lists the supported generations (`ada_lovelace`, `hopper`, `blackwell`), the minimum CUDA compute capability of each, and which generations satisfy each minimum. The order is a policy line, not a speed ranking.

## Several GPUs and clusters

- **Up to 8 GPUs on one machine:** set `gpu_count` (1–8) on the capability, or `cluster_size` on a search. It is exact, not a minimum, and `max_price_per_hour` is for the whole machine. Prefer this to a cluster.
- **Multi-node clusters** (training across machines, RunPod): `GET /v1/clusters/options` lists cards available for a cluster now with an estimated price. `POST /v1/clusters` with `capability_id`, `gpu_type`, `nodes` (2–8), `gpus_per_node` (1–8) and `max_price_per_hour` (for the whole cluster) rents all nodes at once. Each node runs the capability's `start` with `NODE_RANK`, `NUM_NODES`, `MASTER_ADDR`, `MASTER_PORT` and `WORLD_SIZE` set, and appears in your fleet as `<cluster_id>-n<rank>`. `GET /v1/clusters` and `GET /v1/clusters/{cluster_id}` show them; `DELETE /v1/clusters/{cluster_id}` ends every node. A cluster also ends once every node has been idle past its idle limit.