# CLAUDE
Source: https://docs.usechamber.io/CLAUDE
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
This is the documentation repository for Chamber, a GPU capacity control plane for enterprise AI infrastructure. It uses Mintlify as the documentation framework with MDX formatting.
* **Product**: Chamber ([https://usechamber.com](https://usechamber.com))
* **Dashboard**: [https://app.usechamber.io](https://app.usechamber.io)
* **Framework**: Mintlify (MDX-based documentation)
## Development Commands
```bash theme={null}
# Install Mintlify CLI (one-time setup)
npm i -g mintlify
# Start local development server with hot reload
mintlify dev
# View at http://localhost:3000
```
## Deployment
Changes pushed to `main` are automatically deployed via Mintlify's GitHub integration. No manual build step required.
## Architecture
### Content Organization
* `concepts/` - Core concepts (teams, capacity pools, reservations, workload classes, scheduling)
* `platform/` - Platform UI guides (dashboard, team management, capacity management, workloads)
* `agent/` - Kubernetes agent documentation (installation, troubleshooting, upgrading)
* `api-reference/` - API documentation
* `cli/` - CLI documentation
### Configuration
* `docs.json` - Main Mintlify config (navigation structure, theme, colors, navbar)
* Navigation is defined in `docs.json` under `navigation.tabs`
### Adding New Pages
1. Create `.mdx` file in appropriate directory
2. Add frontmatter with `title` and `description`
3. Add page path to `docs.json` navigation section
## MDX Components
Available Mintlify components for documentation pages:
```mdx theme={null}
Content
...
Content
Content
Hidden content
Tip text
Info text
Warning text
Note text
```
## Frontmatter Format
Every `.mdx` page requires:
```mdx theme={null}
---
title: "Page Title"
description: "Brief description of the page content"
---
```
# Installation
Source: https://docs.usechamber.io/agent/installation
Install the Chamber agent in your Kubernetes cluster
This guide covers installing the Chamber agent using Helm.
For a guided setup experience, use the [installation wizard](https://app.usechamber.io/pools/new) in the Chamber dashboard.
This guide installs only the Chamber agent on an **existing** Kubernetes cluster. To deploy a complete GPU-ready cluster (VPC, Kubernetes, GPU autoscaling, GPU drivers, and the Chamber agent), see [Cluster Deployment](/cluster-deployment/overview).
## Prerequisites
Before installing, ensure you have:
Verify with `kubectl version`. The agent supports Kubernetes 1.24 and later.
Verify with `helm version`. Install from [helm.sh](https://helm.sh/docs/intro/install/) if needed.
Nodes must have NVIDIA drivers installed. Verify with `nvidia-smi` on GPU nodes.
The [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin) must be deployed. Verify:
```bash theme={null}
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
```
[DCGM-Exporter](https://github.com/NVIDIA/dcgm-exporter) is required for GPU metrics collection. Verify:
```bash theme={null}
kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter
```
If not installed, deploy via the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) or standalone:
```bash theme={null}
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install dcgm-exporter nvidia/dcgm-exporter -n gpu-operator --create-namespace
```
For the GPU Usage metric, DCGM profiling metrics must be enabled. See [GPU Usage Metric Not Available](/agent/troubleshooting#gpu-usage-metric-not-available) for configuration steps.
Get this from the Chamber dashboard (see below).
## Quick Install
```bash theme={null}
# Install the agent (OCI registry - no helm repo add needed)
helm install chamber-agent oci://public.ecr.aws/chamber/chamber-agent-chart \
--version 0.8.96 \
--namespace chamber-system \
--create-namespace \
--set saas.url="wss://controlplane-api.usechamber.io/agent" \
--set saas.token="" \
--set saas.clusterId="your-gpu-cluster"
```
Your cluster will appear in the Chamber dashboard within **30 seconds**.
Always specify `--version` to ensure reproducible deployments. Check the [releases page](https://github.com/ChamberOrg/chamber-agent/releases) for the latest version.
## Getting a Cluster Token
Go to [app.usechamber.io](https://app.usechamber.io) and sign in.
Click **Settings** in the sidebar, then **Security** tab.
Click **New Token**, give it a name, and copy immediately.
Store your token securely. It's only shown once and cannot be retrieved later.
## Verifying Installation
### Check Pod Status
```bash theme={null}
kubectl get pods -n chamber-system -l app.kubernetes.io/name=chamber-agent
```
Expected output:
```
NAME READY STATUS RESTARTS AGE
chamber-agent-5d4f6b7c8d-abc12 1/1 Running 0 30s
```
### Check Logs
```bash theme={null}
kubectl logs -n chamber-system -l app.kubernetes.io/name=chamber-agent --tail=20
```
Look for:
```
INFO WebSocket connected successfully
INFO Metadata collected: 4 nodes, 32 GPUs
```
### Check Version
```bash theme={null}
helm list -n chamber-system
```
Shows the installed chart version and app version.
### Verify in Dashboard
Within 30 seconds of installation:
1. Go to [app.usechamber.io](https://app.usechamber.io)
2. Navigate to **Capacity Pools**
3. Your cluster should appear with the correct GPU count
If GPUs show as 0, verify the NVIDIA device plugin is running and nodes have `nvidia.com/gpu` resources.
## Network Requirements
The agent requires outbound HTTPS access to:
| Host | Port | Purpose |
| -------------------------------- | ---- | --------------------------- |
| `controlplane-api.usechamber.io` | 443 | Control plane communication |
The agent only makes **outbound** connections. No inbound firewall rules are required.
## Uninstalling
```bash theme={null}
helm uninstall chamber-agent -n chamber-system
```
Uninstalling disconnects your cluster from Chamber. Running workloads will continue to completion but won't be tracked.
## Next Steps
Common issues and solutions
Upgrade to the latest version
# Agent Overview
Source: https://docs.usechamber.io/agent/overview
Understanding the Chamber Kubernetes agent
The Chamber agent is a lightweight Kubernetes controller that connects your GPU clusters to Chamber.
## What the Agent Does
The agent runs in your Kubernetes cluster and:
Discovers GPU nodes and reports capacity to Chamber
Creates and monitors GPU workloads on your cluster
Gathers GPU utilization for dashboards
Implements scheduling decisions from Chamber
## Architecture
```mermaid theme={null}
flowchart LR
subgraph cluster["Your Kubernetes Cluster"]
Agent["Chamber Agent"]
GPU1["GPU Nodes"]
end
Agent -->|Secure connection\n outbound only| Chamber["Chamber"]
Agent --> GPU1
```
The agent maintains a secure connection to Chamber and executes scheduling decisions locally.
## Resilience
**Workloads continue running even if Chamber is temporarily unavailable.** The agent operates independently once workloads are scheduled. If the connection is lost, running workloads are unaffected and the agent automatically reconnects.
## Communication
### Outbound Only
The agent only makes **outbound** connections:
* No inbound ports required
* Works behind corporate firewalls
* Encrypted with TLS
### Network Requirements
| Host | Port | Purpose |
| ----------------- | ---- | --------------------- |
| `*.usechamber.io` | 443 | Chamber communication |
## Resource Requirements
| Resource | Requirement |
| -------- | -------------------------- |
| CPU | 100m request, 200m limit |
| Memory | 128Mi request, 256Mi limit |
The agent has a minimal footprint. These defaults work for most clusters.
## Security
### Permissions
The agent requires these Kubernetes permissions:
```yaml theme={null}
rules:
# Core resources - read cluster state
- apiGroups: [""]
resources: ["nodes", "pods", "namespaces", "events"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["services", "endpoints"] # DCGM-Exporter discovery
verbs: ["get", "list"]
# Workload management
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "delete", "get", "list", "watch"]
# Metrics
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
- apiGroups: ["scheduling.run.ai"]
resources: ["queues", "podgroups"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
```
The agent does **not** require cluster-admin. Permissions are scoped to GPU workload management.
### Data Sent to Chamber
| Data | Purpose |
| --------------------------------------------- | ------------------------ |
| Node inventory (GPU count, type) | Capacity tracking |
| Workload metadata (name, status, GPU request) | Workload management |
| GPU utilization metrics | Dashboards and reporting |
The agent does **not** send container logs, environment variables, or secrets.
## Next Steps
Install the agent via Helm
Common issues and solutions
Deploy a full GPU cluster with Terraform
# Configuration
Source: https://docs.usechamber.io/agent/standalone/configuration
Environment variables, workload labels, upgrading, and uninstalling the Standalone Agent
The Standalone Agent is in **beta**. Configuration options may change.
The agent is configured via environment variables in `/etc/chamber/agent.env`. After making changes, restart the agent:
```bash theme={null}
sudo vi /etc/chamber/agent.env
sudo systemctl restart chamber-agent-standalone
```
## Required Settings
| Variable | Description |
| ----------------------- | ---------------------------------------------------------------------------------------------------- |
| `CHAMBER_CLUSTER_TOKEN` | API token for authenticating with the Chamber control plane. Generated in **Settings > API Tokens**. |
The `CHAMBER_SAAS_URL` is set automatically by the installer and should not need to be changed.
## Common Settings
| Variable | Default | Description |
| -------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `CHAMBER_CLUSTER_ID` | hostname | Identifier for this host in the Chamber dashboard. Set this to a human-readable name if the hostname is opaque (e.g., `ip-10-0-1-42`). |
| `CHAMBER_HOSTNAME` | hostname | Override the reported hostname. |
| `CHAMBER_DCGM_URL` | `http://localhost:9400/metrics` | DCGM Exporter endpoint. Change if running DCGM Exporter on a non-default port. |
## Metrics Settings
| Variable | Default | Description |
| ------------------------------------- | -------- | --------------------------------------------------- |
| `CHAMBER_METRICS_ENABLED` | `true` | Enable or disable all metric collection. |
| `CHAMBER_METRICS_COLLECTION_INTERVAL` | `30` | Seconds between metric collection cycles. |
| `CHAMBER_METRICS_BATCH_INTERVAL` | `60` | Seconds between batch uploads to Chamber. |
| `CHAMBER_METRICS_BUFFER_SIZE` | `100000` | Maximum metrics buffered before oldest are dropped. |
## GPU Usage Settings
| Variable | Default | Description |
| ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `CHAMBER_GPU_USAGE_ENABLED` | `true` | Enable or disable the composite GPU Usage metric. |
| `CHAMBER_GPU_USAGE_POWER_THRESHOLD` | `0.1` | Power ratio (0.0–1.0) below which GPU is considered idle. At the default, GPUs drawing less than 10% of their power limit report GPU Usage as 0. |
## Discovery Settings
| Variable | Default | Description |
| --------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
| `CHAMBER_DISCOVERY_ENABLED` | `true` | Enable or disable automatic GPU workload discovery. |
| `CHAMBER_DISCOVERY_SCAN_INTERVAL` | `15` | Seconds between GPU process scans. |
| `CHAMBER_DISCOVERY_STABILIZATION_DELAY` | `5` | Seconds to wait before classifying a newly detected process. Prevents transient processes from appearing as workloads. |
| `CHAMBER_DISCOVERY_EVENT_BUFFER_SIZE` | `500` | Maximum discovery events buffered. |
## OTLP Settings
| Variable | Default | Description |
| --------------------------------- | --------- | ----------------------------------------- |
| `CHAMBER_OTLP_GRPC_ENABLED` | `true` | Enable or disable the OTLP gRPC receiver. |
| `CHAMBER_OTLP_GRPC_PORT` | `4317` | gRPC server port. |
| `CHAMBER_OTLP_GRPC_HOST` | `0.0.0.0` | gRPC server bind address. |
| `CHAMBER_OTLP_METRICS_QUEUE_SIZE` | `10000` | Maximum queued metric batches. |
## Connection Settings
| Variable | Default | Description |
| ------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `CHAMBER_HEARTBEAT_INTERVAL` | `30` | Seconds between WebSocket heartbeats. |
| `CHAMBER_METADATA_INTERVAL` | `60` | Seconds between host metadata reports. |
| `CHAMBER_RECONNECT_BACKOFF` | `5` | Initial reconnection delay (seconds) after a dropped WebSocket connection. |
| `CHAMBER_MAX_RECONNECT_BACKOFF` | `30` | Maximum reconnection delay (seconds). The agent uses exponential backoff with jitter between these bounds. |
## Workload Labels
Add environment variables to your training jobs to enrich workload metadata in the dashboard:
```bash theme={null}
CHAMBER_JOB_NAME="llama2-finetune-v3" \
CHAMBER_TEAM_NAME="ml-platform" \
torchrun --nproc_per_node=4 train.py
```
| Variable | Description |
| ------------------- | ------------------------------------------------------------------------------------------ |
| `CHAMBER_JOB_NAME` | Override the auto-detected workload name. Useful when the process command line is generic. |
| `CHAMBER_TEAM_NAME` | Set a team label for workload attribution. Groups workloads by team in the dashboard. |
These environment variables are read from the process environment at discovery time. They are not part of `/etc/chamber/agent.env` — set them on your training process directly.
## Upgrading
Re-run the installer to upgrade to the latest version. Your configuration in `/etc/chamber/agent.env` is preserved:
```bash theme={null}
curl -fsSL https://chamber-agent-releases.s3.amazonaws.com/install.sh | sudo bash
```
To pin to a specific version (e.g., for reproducibility):
```bash theme={null}
curl -fsSL https://chamber-agent-releases.s3.amazonaws.com/install.sh \
| sudo bash -s -- --version
```
## Uninstalling
```bash theme={null}
sudo systemctl stop chamber-agent-standalone
sudo systemctl disable chamber-agent-standalone
sudo rm /etc/systemd/system/chamber-agent-standalone.service
sudo systemctl daemon-reload
sudo rm -rf /opt/chamber-standalone
sudo rm -rf /etc/chamber
sudo userdel chamber-agent
```
This removes the agent, its configuration, and the system user. It does not remove Docker, Python, or DCGM Exporter.
## Next Steps
Common issues and solutions
Full list of metrics collected
# Installation
Source: https://docs.usechamber.io/agent/standalone/installation
Install the Chamber Standalone Agent on a GPU host
The Standalone Agent is in **beta**. Features and configuration may change.
This guide covers installing the Chamber Standalone Agent on a Linux GPU host (bare metal, cloud VM, or on-prem server).
## Prerequisites
| Requirement | Details |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Chamber account** | Sign up at [app.usechamber.io/signup](https://app.usechamber.io/signup) and contact the Chamber team to enable standalone agent access. Then create an API token from **Settings > API Tokens**. |
| **GPU host** | Linux host with one or more NVIDIA GPUs (GCP, AWS, Azure, or bare metal) |
| **NVIDIA driver** | Pre-installed on most cloud GPU images. Verify with `nvidia-smi`. |
| **Root access** | The installer requires `sudo` |
Python 3.11+, Docker, and DCGM Exporter are installed automatically by the installer if not already present.
## Step 1: Get Your Cluster Token
Go to [app.usechamber.io](https://app.usechamber.io) and navigate to **Settings**.
Click **New Token**, give it a name, and copy the generated token immediately.
Store your token securely. It is only shown once and cannot be retrieved later.
## Step 2: Install the Agent
SSH into your GPU host and run:
```bash theme={null}
curl -fsSL https://chamber-agent-releases.s3.amazonaws.com/install.sh \
| sudo CHAMBER_CLUSTER_TOKEN= bash
```
Replace `` with the token from Step 1.
The installer will:
* Install Python 3.11+, Docker, and NVIDIA Container Toolkit (if not already present)
* Set up DCGM Exporter for GPU profiling metrics
* Install and configure the Chamber Standalone Agent
* Start the agent as a `systemd` service
The full process takes 2–5 minutes depending on what is already installed.
## Step 3: Verify
Your host should appear in the Chamber dashboard within 60 seconds. You can also verify locally:
```bash theme={null}
# Check service status
sudo systemctl status chamber-agent-standalone
# View recent logs
sudo journalctl -u chamber-agent-standalone --no-pager -n 20
```
You should see log lines indicating a successful WebSocket connection to the control plane.
### Verify in Dashboard
1. Go to [app.usechamber.io/dashboard?tab=services](https://app.usechamber.io/dashboard?tab=services)
2. Your host should appear with GPU count and metrics flowing
If the host doesn't appear within 60 seconds, see [Troubleshooting](/agent/standalone/troubleshooting).
## Common Operations
```bash theme={null}
# View live logs
sudo journalctl -u chamber-agent-standalone -f
# Restart the agent
sudo systemctl restart chamber-agent-standalone
# Stop the agent
sudo systemctl stop chamber-agent-standalone
# Check agent version
/opt/chamber-standalone/venv/bin/chamber-standalone version
# View detected GPU and host info
/opt/chamber-standalone/venv/bin/chamber-standalone host-info
```
## Network Requirements
The agent requires outbound HTTPS/WSS access only. No inbound ports need to be opened.
| Host | Port | Purpose |
| -------------------------------- | ---- | --------------------------------------- |
| `controlplane-api.usechamber.io` | 443 | Control plane communication (WebSocket) |
If you are running behind a corporate firewall, ensure outbound WebSocket connections to port 443 are allowed.
## Next Steps
Understand the metrics collected and what they mean
Customize agent behavior with environment variables
Send your application metrics to Chamber
Common issues and solutions
# Metrics Reference
Source: https://docs.usechamber.io/agent/standalone/metrics
GPU, host, and workload metrics collected by the Standalone Agent and how to use them
The Standalone Agent is in **beta**. Metric names and labels may change.
The Standalone Agent collects GPU metrics (via DCGM Exporter), host metrics (via psutil), per-process GPU metrics (via NVML), and application-level service metrics (via the built-in OTLP receiver). All metrics are viewable on the **Dashboard > Services** page at [app.usechamber.io/dashboard?tab=services](https://app.usechamber.io/dashboard?tab=services).
## GPU Metrics
These metrics are scraped from DCGM Exporter every 30 seconds.
### Core GPU Metrics
| Metric | Unit | Description | What to look for |
| ---------------------------------------- | ----- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `chamber_gpu_utilization_percent` | % | Percentage of time the GPU had at least one kernel running. | Sustained 0% may indicate idle GPUs wasting capacity. Sustained 100% is normal for training workloads. |
| `chamber_gpu_memory_used_bytes` | bytes | GPU memory (VRAM) in use. | Approaching total memory indicates risk of OOM errors. |
| `chamber_gpu_memory_utilization_percent` | % | GPU memory in use as a percentage of total. | Above 90% for extended periods suggests memory pressure. |
| `chamber_gpu_temperature_celsius` | °C | GPU die temperature. | Consistently above 80°C may indicate thermal throttling. Check airflow and cooling. |
| `chamber_gpu_power_usage_watts` | watts | Current power draw. | At or near the power limit can cause power throttling, reducing clock speeds and throughput. |
### GPU Usage (Composite Metric)
| Metric | Unit | Description |
| ------------------- | ---- | ------------------------------------------------------------ |
| `chamber_gpu_usage` | % | A composite metric reflecting actual GPU compute efficiency. |
Standard GPU utilization (`chamber_gpu_utilization_percent`) can be misleading — a single-threaded kernel running on 1 of 132 streaming multiprocessors (SMs) still reports 100% utilization. GPU Usage provides a more accurate picture.
**How it's calculated:**
```
GPU Usage = max(SM Active, Tensor Active, DRAM Active) × 100
```
| Input | DCGM Source | What it measures |
| ------------- | --------------------------------- | ------------------------------------------------------------- |
| SM Active | `DCGM_FI_PROF_SM_ACTIVE` | Fraction of SMs that are actively executing a warp (0.0–1.0) |
| Tensor Active | `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` | Fraction of time the tensor cores are active (0.0–1.0) |
| DRAM Active | `DCGM_FI_PROF_DRAM_ACTIVE` | Fraction of time GPU memory (HBM) is being accessed (0.0–1.0) |
**Power gate:** If the GPU's power draw is below 10% of its power limit, GPU Usage is reported as 0 regardless of the profiling metrics. This filters out noise from idle GPUs.
**Why it matters:** GPU Usage tells you whether your workload is actually keeping the GPU busy. A training job might show 100% standard utilization but only 30% GPU Usage, meaning most SMs are idle and you could optimize kernel launch, batch size, or parallelism.
GPU Usage requires DCGM profiling metrics to be enabled. See [Ensuring DCGM Profiling Metrics](#ensuring-dcgm-profiling-metrics-are-collected) below.
### Profiling Metrics
These are the raw profiling metrics behind the GPU Usage composite. They are useful for advanced debugging.
| Metric | Unit | Description |
| ------------------- | --------------- | -------------------------------------------------------------------------------- |
| `gpu_sm_active` | ratio (0.0–1.0) | Streaming Multiprocessor activity — fraction of SMs executing at least one warp |
| `gpu_tensor_active` | ratio (0.0–1.0) | Tensor core pipeline activity — high during mixed-precision training (FP16/BF16) |
| `gpu_dram_active` | ratio (0.0–1.0) | GPU memory (HBM) bus activity — high during data-loading-bound workloads |
## Host Metrics
Collected via psutil every 30 seconds.
| Metric | Unit | Description | What to look for |
| ------------------------------------ | ----- | ---------------------------------- | --------------------------------------------------------------------- |
| `chamber_node_cpu_usage_percent` | % | Host CPU utilization | High CPU can bottleneck data loading pipelines, starving GPUs of data |
| `chamber_node_cpu_usage_cores` | cores | Number of CPU cores in use (float) | Compare against total cores to gauge headroom |
| `chamber_node_cpu_capacity_cores` | cores | Total logical CPU count | — |
| `chamber_node_memory_usage_bytes` | bytes | Host memory (RAM) in use | Near capacity can cause OOM kills by the Linux kernel |
| `chamber_node_memory_capacity_bytes` | bytes | Total host memory | — |
| `chamber_node_memory_usage_percent` | % | Host memory utilization | Above 90% sustained warrants investigation |
## Per-Process GPU Metrics
Collected via NVML for each process using a GPU. These are attributed to discovered workloads in the dashboard.
| Metric | Unit | Description |
| ------------------------------------------------ | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `chamber_gpu_process_memory_bytes` | bytes | GPU memory allocated by a specific process |
| `chamber_gpu_process_memory_utilization_percent` | % | Process GPU memory as a percentage of total GPU memory |
| `chamber_gpu_process_sm_utilization_percent` | % | Process-level SM utilization (requires NVIDIA driver 450+ and Volta or newer GPUs; the agent skips this metric gracefully on unsupported hardware) |
## Service-Level Metrics
In addition to infrastructure metrics, the Standalone Agent tracks **services** — applications running on your GPU hosts that are identified by the `service_name` label. This is what powers the **Services** tab on the dashboard, giving you a per-service view of resource consumption and application health.
### How Services Are Identified
The agent associates a `service_name` with metrics through three mechanisms, in priority order:
1. **`OTEL_SERVICE_NAME` environment variable** — If your GPU process has `OTEL_SERVICE_NAME` set, the agent reads it directly and uses it to tag all GPU and per-process metrics for that workload. This is the most reliable method. For Docker containers where the process environment may not be directly readable, the agent automatically falls back to querying the Docker Engine API via `/var/run/docker.sock` to read container environment variables — no configuration needed beyond ensuring the agent user has Docker socket access (the installer handles this by default).
2. **OTLP `service.name` resource attribute** — If your application exports OpenTelemetry metrics to the agent's OTLP receiver (port 4317), the `service.name` resource attribute is extracted and attached to every metric in that export. When exactly one service is actively sending OTLP metrics, the agent also uses it to label GPU infrastructure metrics for the running workload.
3. **Workload discovery** — The agent classifies GPU processes by inspecting command lines and environment variables to identify ML frameworks (PyTorch, TensorFlow, vLLM, etc.) and launchers (torchrun, deepspeed, accelerate). This provides `job_name` and `workload_type` labels even without `OTEL_SERVICE_NAME`.
```bash theme={null}
# Option 1: Set on your training process
OTEL_SERVICE_NAME="llama2-finetune" torchrun --nproc_per_node=4 train.py
# Option 2: Set in your OTel SDK resource (see OTLP Integration page)
Resource.create({"service.name": "llama2-finetune"})
```
### Application Metrics via OTLP
Any metrics your application exports over OpenTelemetry are forwarded to the Chamber dashboard with the `service_name` label. These appear alongside GPU and host metrics on the Services tab, letting you correlate application behavior with infrastructure state.
Common examples of application metrics you can send:
| Metric | What it tells you |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| `training_loss` | Whether the model is converging — a sudden spike may indicate a data issue or corrupted checkpoint |
| `training_throughput_samples_total` | Samples processed over time — a drop signals a bottleneck (data loading, GPU contention, network) |
| `inference_request_duration_seconds` | End-to-end latency per request — rising latency under stable load may point to memory pressure or thermal throttling |
| `inference_requests_total` | Request volume — correlate with GPU utilization to understand per-request cost |
| `model_load_time_seconds` | Time to load model weights — useful for tracking cold-start performance across deployments |
| `batch_queue_depth` | Pending items in your processing queue — growing depth means your GPUs can't keep up with inbound work |
These are examples — you can send any metric your application produces. See [OTLP Integration](/agent/standalone/otel-integration) for setup instructions.
### What You See on the Services Dashboard
The **Services** tab ([app.usechamber.io/dashboard?tab=services](https://app.usechamber.io/dashboard?tab=services)) groups metrics by `service_name`, giving you a unified view per service:
* **GPU utilization and GPU Usage** attributed to each service — see which services are efficiently using their GPUs and which are underutilizing
* **GPU memory per service** — identify which service is consuming the most VRAM and whether it's at risk of OOM
* **Per-process breakdowns** — drill into individual worker processes within a service (e.g., each rank in a distributed training job)
* **Application metrics** — any custom OTLP metrics your service exports, displayed alongside the infrastructure data
* **Host CPU and memory** — spot cases where a service is bottlenecked on CPU or host memory rather than GPU
This per-service view makes it straightforward to answer questions like: *"Is my training job actually using the GPUs I allocated?"* or *"Why did inference latency spike at 2 PM?"* — without having to cross-reference multiple monitoring systems.
For the best experience on the Services dashboard, set `OTEL_SERVICE_NAME` on every GPU process. This ensures GPU metrics are correctly attributed even when multiple services share a host.
## Workload Discovery
The agent automatically discovers GPU-using processes and classifies them by framework:
* **PyTorch** (including torchrun, torch.distributed)
* **DeepSpeed**
* **vLLM**
* **Other GPU processes**
Discovered workloads appear in the dashboard with:
* Process name and command line
* Framework type
* GPU memory usage per process
* Lifecycle state (running, terminated)
* Duration
For distributed training (e.g., `torchrun --nproc_per_node=4`), the agent groups worker processes under a single workload entry.
You can enrich workload metadata with environment variables. See [Workload Labels](/agent/standalone/configuration#workload-labels).
## Ensuring DCGM Profiling Metrics Are Collected
The GPU Usage metric and profiling metrics (`gpu_sm_active`, `gpu_tensor_active`, `gpu_dram_active`) require DCGM Exporter to expose profiling-level fields. The installer sets this up automatically, but if you are running DCGM Exporter manually or troubleshooting missing metrics, verify the following:
### 1. DCGM Exporter Must Be Running
```bash theme={null}
# Check if DCGM Exporter is serving metrics
curl -s http://localhost:9400/metrics | head -20
```
You should see Prometheus-formatted metrics including lines starting with `DCGM_FI_`. If this returns nothing, DCGM Exporter is not running or not reachable.
### 2. Profiling Fields Must Be Enabled
DCGM Exporter is configured by a CSV file that lists which fields to export. The profiling fields required are:
| DCGM Field ID | Field Name | Required For |
| ------------- | --------------------------------- | ------------------------- |
| 1001 | `DCGM_FI_PROF_SM_ACTIVE` | GPU Usage (SM Active) |
| 1004 | `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` | GPU Usage (Tensor Active) |
| 1005 | `DCGM_FI_PROF_DRAM_ACTIVE` | GPU Usage (DRAM Active) |
Verify these fields are in the DCGM Exporter counters file (commonly `/etc/dcgm-exporter/default-counters.csv` or passed via the `-f` flag):
```bash theme={null}
# Check if profiling fields are configured
grep -E "DCGM_FI_PROF_SM_ACTIVE|DCGM_FI_PROF_PIPE_TENSOR_ACTIVE|DCGM_FI_PROF_DRAM_ACTIVE" \
/etc/dcgm-exporter/default-counters.csv
```
If these entries are missing, add them:
```csv theme={null}
DCGM_FI_PROF_SM_ACTIVE, gauge
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge
DCGM_FI_PROF_DRAM_ACTIVE, gauge
```
Then restart DCGM Exporter.
### 3. Profiling Must Be Supported by Your GPU
DCGM profiling metrics require:
* NVIDIA driver 450+ (most modern drivers)
* A GPU that supports profiling (Volta, Turing, Ampere, Hopper, Blackwell — i.e., V100, T4, A100, H100, B200, etc.)
* No other profiling tool (e.g., Nsight Systems) actively holding a profiling session
If profiling metrics are unavailable, the core GPU metrics (utilization, memory, temperature, power) still collect normally. Only the GPU Usage composite metric will be missing.
### 4. Verify Profiling Metrics Are Flowing
```bash theme={null}
curl -s http://localhost:9400/metrics | grep DCGM_FI_PROF
```
You should see lines like:
```
DCGM_FI_PROF_SM_ACTIVE{gpu="0",...} 0.45
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0",...} 0.38
DCGM_FI_PROF_DRAM_ACTIVE{gpu="0",...} 0.22
```
If these lines are absent or all values are 0 while a workload is running, check the troubleshooting steps above.
## Metric Labels
All metrics include these standard labels:
| Label | Description |
| ----------------- | ------------------------------------------------------------- |
| `cluster_id` | Identifier for this host (defaults to hostname, configurable) |
| `node_name` | Hostname of the machine |
| `organization_id` | Your Chamber organization ID |
| `managed_by` | Always `external` for standalone agents |
GPU metrics additionally include:
| Label | Description |
| ----------- | ---------------------------------------------- |
| `gpu_index` | GPU index (0, 1, 2, ...) |
| `gpu_type` | GPU model name (e.g., `NVIDIA A100-SXM4-80GB`) |
Process and service metrics additionally include:
| Label | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `job_id` | Unique workload identifier |
| `job_name` | Workload name (auto-detected or from `CHAMBER_JOB_NAME`) |
| `pod_name` | Process identifier (`process:pid-`) |
| `team_name` | Team label (from `CHAMBER_TEAM_NAME` or default) |
| `service_name` | Service identifier (from `OTEL_SERVICE_NAME` env var or OTLP `service.name` resource attribute). Used to group metrics on the Services dashboard. |
## What to Look For on the Dashboard
The **Services** tab ([app.usechamber.io/dashboard?tab=services](https://app.usechamber.io/dashboard?tab=services)) gives you a per-host and per-GPU breakdown. Here are common patterns to watch for:
| Pattern | What it means | Action |
| -------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| High utilization but low GPU Usage | Kernels are running but underutilizing the GPU's compute capacity | Profile with Nsight, increase batch size, or check for serialization bottlenecks |
| GPU memory near 100% | Risk of OOM errors that crash training | Reduce batch size, enable gradient checkpointing, or use a larger GPU |
| High CPU, low GPU utilization | Data loading is the bottleneck — GPUs are starved | Add more DataLoader workers, use faster storage, or pre-process data |
| Temperature consistently above 80°C | Thermal throttling may be reducing clock speeds | Check cooling, airflow, or ambient temperature |
| Power at limit | Power throttling is reducing clock speeds | Expected under heavy load, but sustained power capping may indicate undersized infrastructure |
| GPU idle (0% utilization, 0 GPU Usage) | GPU is allocated but not in use | Check for crashed jobs, queued-but-not-started workloads, or misconfigured resource requests |
## Next Steps
Send your own application metrics to Chamber
Tune collection intervals and other settings
# OTel Integration
Source: https://docs.usechamber.io/agent/standalone/otel-integration
Send OpenTelemetry metrics from your applications to Chamber via the Standalone Agent
The Standalone Agent is in **beta**. OTLP integration behavior may change.
The Standalone Agent includes a built-in OpenTelemetry (OTLP) gRPC receiver. Any application instrumented with the OpenTelemetry SDK can send metrics directly to the agent, which forwards them to the Chamber dashboard alongside GPU and host metrics.
This lets you correlate application-level metrics (training loss, throughput, queue depth, request latency) with infrastructure metrics in a single view.
## How It Works
```mermaid theme={null}
flowchart LR
App["Your App
(OTel SDK)"] -->|"gRPC :4317"| Agent["Chamber
Standalone Agent"]
Agent -->|batch upload| Chamber["Chamber
Dashboard"]
```
1. Your application exports OTLP metrics over gRPC to `localhost:4317`
2. The agent receives, transforms, and buffers the metrics
3. Metrics are uploaded to Chamber in batches every 60 seconds
4. They appear on the **Services** dashboard alongside infrastructure metrics
## Configuration
The OTLP receiver is **enabled by default** on port 4317. No additional configuration is required for most setups.
| Variable | Default | Description |
| --------------------------------- | --------- | --------------------------------------------- |
| `CHAMBER_OTLP_GRPC_ENABLED` | `true` | Enable or disable the OTLP gRPC receiver |
| `CHAMBER_OTLP_GRPC_PORT` | `4317` | Port the gRPC server listens on |
| `CHAMBER_OTLP_GRPC_HOST` | `0.0.0.0` | Bind address for the gRPC server |
| `CHAMBER_OTLP_METRICS_QUEUE_SIZE` | `10000` | Maximum queued metric batches before dropping |
To disable the receiver, set `CHAMBER_OTLP_GRPC_ENABLED=false` in `/etc/chamber/agent.env` and restart the agent.
## Sending Metrics from Your Application
Point your application's OTLP exporter to the agent's gRPC endpoint. Here is an example using the Python OpenTelemetry SDK:
```python theme={null}
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
# Configure exporter to send to the local Chamber agent
exporter = OTLPMetricExporter(
endpoint="localhost:4317",
insecure=True, # Local connection, no TLS needed
)
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30000)
provider = MeterProvider(
resource=Resource.create({"service.name": "my-training-job"}),
metric_readers=[reader],
)
metrics.set_meter_provider(provider)
meter = metrics.get_meter("my-training-job")
# Create metrics
loss_gauge = meter.create_gauge("training.loss", description="Current training loss")
throughput_counter = meter.create_counter(
"training.samples.processed",
unit="1",
description="Total training samples processed",
)
# Record metrics in your training loop
loss_gauge.set(0.42)
throughput_counter.add(batch_size)
```
Set `service.name` in your resource attributes — it appears as the service name label on all metrics in the Chamber dashboard.
## Metric Name Transformation
The agent transforms OTLP metric names to a Prometheus-compatible format before uploading:
| Transformation | Example |
| ------------------------------------- | ---------------------------------------------------------------- |
| Dots and hyphens become underscores | `training.loss` → `training_loss` |
| Unit suffixes are appended | `request_duration` (unit: `s`) → `request_duration_seconds` |
| Monotonic sums get `_total` suffix | `samples.processed` (sum, monotonic) → `samples_processed_total` |
| Consecutive underscores are collapsed | `my__metric` → `my_metric` |
### Unit Suffix Mapping
| OTel Unit | Suffix |
| --------- | --------------- |
| `s` | `_seconds` |
| `ms` | `_milliseconds` |
| `By` | `_bytes` |
| `KBy` | `_kilobytes` |
| `MBy` | `_megabytes` |
| `GBy` | `_gigabytes` |
| `1` | `_ratio` |
## Histogram Support
OTLP histograms are decomposed into Prometheus-style metrics:
| Generated Metric | Description |
| ---------------- | -------------------------------------------------------------- |
| `{name}_bucket` | Cumulative bucket counts with `le` (less-than-or-equal) labels |
| `{name}_count` | Total number of observations |
| `{name}_sum` | Sum of all observed values |
## Supported Metric Types
| OTel Type | Supported |
| --------------------------------- | --------- |
| Gauge | Yes |
| Sum (monotonic and non-monotonic) | Yes |
| Histogram | Yes |
| ExponentialHistogram | No |
| Summary | No |
## Verifying Metrics Are Flowing
After configuring your application, check the agent logs:
```bash theme={null}
sudo journalctl -u chamber-agent-standalone -f | grep -i otlp
```
You should see log lines indicating metrics received. Then check the **Services** tab in the dashboard — your application metrics will appear with the `service_name` label you configured.
## Next Steps
Full list of infrastructure metrics collected
All environment variables and settings
# Standalone Agent
Source: https://docs.usechamber.io/agent/standalone/overview
Monitor GPU utilization and workloads on bare-metal or VM-based GPU hosts without Kubernetes
The Standalone Agent is currently in **beta** and supports **metric collection only**. Features and configuration may change.
The Chamber Standalone Agent runs on individual GPU hosts (bare metal, cloud VMs, or on-prem servers) and reports GPU, host, and application-level metrics to the Chamber dashboard. It is designed for environments where GPUs are not managed by Kubernetes.
## When to Use the Standalone Agent
| Environment | Agent |
| --------------------------------- | ----------------------------------- |
| Kubernetes cluster with GPU nodes | [Kubernetes Agent](/agent/overview) |
| Bare-metal GPU servers | **Standalone Agent** |
| Cloud GPU VMs (GCP, AWS, Azure) | **Standalone Agent** |
| On-prem GPU workstations | **Standalone Agent** |
## What It Does
Collects utilization, memory, temperature, power draw, and profiling-level metrics from every GPU on the host via DCGM Exporter
Reports CPU usage, memory usage, and capacity so you can see the full picture alongside GPU data
Automatically detects GPU processes (PyTorch, DeepSpeed, vLLM, etc.), groups distributed training workers, and tracks their lifecycle
Accepts OpenTelemetry metrics from your applications over gRPC, forwarding them to the Chamber dashboard alongside infrastructure metrics
## Architecture
```mermaid theme={null}
flowchart LR
subgraph host["Your GPU Host"]
Agent["Chamber
Standalone Agent"]
DCGM["DCGM Exporter"]
GPU["NVIDIA GPUs"]
App["Your App
(OTLP)"]
end
DCGM -->|scrape metrics| Agent
GPU --> DCGM
App -->|"gRPC :4317"| Agent
Agent -->|"WSS :443
outbound only"| Chamber["Chamber
Control Plane"]
```
The agent connects outbound to Chamber over a secure WebSocket. No inbound ports or firewall rules are required for Chamber communication.
## What You'll See in the Dashboard
Once the agent is running, navigate to **Dashboard > Services** ([app.usechamber.io/dashboard?tab=services](https://app.usechamber.io/dashboard?tab=services)) to see:
* **GPU utilization and GPU Usage** per host and per GPU — identify underutilized or overloaded GPUs
* **GPU memory consumption** — track memory pressure across your fleet
* **Temperature and power draw** — spot thermal throttling or power-capped GPUs before they impact training
* **Host CPU and memory** — correlate host-level bottlenecks with GPU performance
* **Active workloads** — see which processes are running on each GPU, how long they've been active, and how much GPU memory they're consuming
* **Application metrics** — if you instrument your app with OpenTelemetry, custom metrics appear alongside infrastructure data
Metrics are collected every 30 seconds and uploaded in batches every 60 seconds. Your host should appear in the dashboard within 60 seconds of the agent starting.
## Beta Limitations
* **Metric collection only** — the standalone agent does not support scheduling, capacity pool management, or workload orchestration
* **No auto-remediation** — unlike the Kubernetes agent, the standalone agent does not restart or reschedule failed workloads
* **Features may change** — configuration, metric names, and dashboard views are subject to change during the beta period
## Next Steps
Install the agent on a GPU host
Understand the metrics collected and what they mean
Send application-level OpenTelemetry metrics to Chamber
Environment variables, workload labels, upgrading, and uninstalling
# Troubleshooting
Source: https://docs.usechamber.io/agent/standalone/troubleshooting
Common issues and solutions for the Chamber Standalone Agent
The Standalone Agent is in **beta**. If you encounter issues not covered here, contact [support@usechamber.io](mailto:support@usechamber.io).
## Agent Fails to Start
Check the logs for error details:
```bash theme={null}
sudo journalctl -u chamber-agent-standalone -n 50 --no-pager
```
**Common causes:**
| Symptom | Cause | Fix |
| -------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `Authentication failed` or `403` | Invalid or expired cluster token | Verify `CHAMBER_CLUSTER_TOKEN` in `/etc/chamber/agent.env`. Generate a new token in **Settings > API Tokens** if needed. |
| `Connection refused` or timeout | Firewall blocking outbound WebSocket on port 443 | Ensure outbound connections to `controlplane-api.usechamber.io:443` are allowed. |
| `Organization deactivated` | Your Chamber organization has been deactivated | Contact [support@usechamber.io](mailto:support@usechamber.io). |
| `Superseded by new instance` | Another agent instance connected with the same token | Only one agent instance should run per host. Check for duplicate services. |
## Host Not Appearing in Dashboard
1. Verify the agent is running:
```bash theme={null}
sudo systemctl status chamber-agent-standalone
```
2. Check for a successful WebSocket connection in the logs:
```bash theme={null}
sudo journalctl -u chamber-agent-standalone --no-pager -n 20 | grep -i websocket
```
3. Verify the token is correct:
```bash theme={null}
grep CHAMBER_CLUSTER_TOKEN /etc/chamber/agent.env
```
Ensure it matches a valid token from **Settings > API Tokens** in the Chamber dashboard.
4. Check network connectivity:
```bash theme={null}
curl -s -o /dev/null -w "%{http_code}" https://controlplane-api.usechamber.io/health
```
A `200` response confirms network access.
## No GPU Metrics
### NVIDIA Driver Not Detected
```bash theme={null}
nvidia-smi
```
If this fails, the NVIDIA driver is not installed or not loaded. Install the driver appropriate for your GPU and OS.
### DCGM Exporter Not Running
```bash theme={null}
curl -s http://localhost:9400/metrics | head -20
```
If this returns nothing or errors, DCGM Exporter is not running. Check its status:
```bash theme={null}
# If running as a Docker container
docker ps | grep dcgm
# If running as a systemd service
sudo systemctl status nvidia-dcgm-exporter
```
Restart if needed, or re-run the installer which will set up DCGM Exporter automatically.
### DCGM Exporter on a Non-Default Port
If DCGM Exporter is running on a port other than 9400, update the agent configuration:
```bash theme={null}
# In /etc/chamber/agent.env
CHAMBER_DCGM_URL=http://localhost:/metrics
```
Then restart the agent: `sudo systemctl restart chamber-agent-standalone`
## GPU Usage Metric Not Available
The GPU Usage metric requires DCGM profiling metrics. If it's missing:
1. **Check profiling metrics are exported:**
```bash theme={null}
curl -s http://localhost:9400/metrics | grep DCGM_FI_PROF
```
You should see `DCGM_FI_PROF_SM_ACTIVE`, `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`, and `DCGM_FI_PROF_DRAM_ACTIVE`.
2. **If absent, add profiling fields** to your DCGM Exporter counters file. See [Ensuring DCGM Profiling Metrics Are Collected](/agent/standalone/metrics#ensuring-dcgm-profiling-metrics-are-collected).
3. **Check GPU compatibility:** Profiling requires Volta-generation GPUs or newer (V100, T4, A100, H100, B200, etc.).
4. **Check for conflicting profiling sessions:** Another tool (e.g., Nsight Systems, Nsight Compute) holding a profiling session prevents DCGM from collecting profiling metrics. Close the conflicting tool and restart DCGM Exporter.
## OTLP Receiver Issues
### Application Can't Connect to Port 4317
1. Verify the OTLP receiver is enabled:
```bash theme={null}
grep CHAMBER_OTLP_GRPC_ENABLED /etc/chamber/agent.env
```
It should be `true` (or absent, since `true` is the default).
2. Check the agent is listening:
```bash theme={null}
ss -tlnp | grep 4317
```
3. If another process is using port 4317, change the agent's OTLP port:
```bash theme={null}
# In /etc/chamber/agent.env
CHAMBER_OTLP_GRPC_PORT=4318
```
Then restart the agent and update your application's exporter endpoint.
### Metrics Sent but Not Appearing in Dashboard
* Metrics are batched and uploaded every 60 seconds. Wait at least 2 minutes after sending.
* Check agent logs for OTLP-related errors:
```bash theme={null}
sudo journalctl -u chamber-agent-standalone -f | grep -i otlp
```
* Verify your application sets `service.name` in its resource attributes. Metrics without a service name may be harder to find in the dashboard.
## Agent Keeps Restarting
Check for repeated crash logs:
```bash theme={null}
sudo journalctl -u chamber-agent-standalone --no-pager -n 100
```
Common causes:
* **Python version too old:** The agent requires Python 3.11+. Check with `/opt/chamber-standalone/venv/bin/python --version`.
* **Corrupted installation:** Re-run the installer to repair: `curl -fsSL https://chamber-agent-releases.s3.amazonaws.com/install.sh | sudo bash`
## Health and Liveness Probes
The agent provides CLI commands for health and liveness checks, useful for container orchestration or custom monitoring:
```bash theme={null}
# Readiness: succeeds when the agent is connected and healthy
chamber-standalone health
# Liveness: succeeds when the agent process is alive and responsive
chamber-standalone liveness
```
Both commands check for a recent heartbeat file (within the last 120 seconds) and exit with code 0 (healthy) or 1 (unhealthy). The systemd service uses the liveness probe as a watchdog automatically.
If you're running the agent in a container, configure probes like:
```yaml theme={null}
livenessProbe:
exec:
command: ["chamber-standalone", "liveness"]
periodSeconds: 30
readinessProbe:
exec:
command: ["chamber-standalone", "health"]
periodSeconds: 30
```
## Support
If these steps don't resolve your issue, contact [support@usechamber.io](mailto:support@usechamber.io) with:
* Agent logs: `sudo journalctl -u chamber-agent-standalone --no-pager -n 200`
* Agent version: `/opt/chamber-standalone/venv/bin/chamber-standalone version`
* Host info: `/opt/chamber-standalone/venv/bin/chamber-standalone host-info`
* GPU info: `nvidia-smi`
# Troubleshooting
Source: https://docs.usechamber.io/agent/troubleshooting
Common issues and solutions for the Chamber agent
This guide covers common issues with the Chamber agent and how to resolve them.
## Quick Checks
```bash theme={null}
# Check agent pod status
kubectl get pods -n chamber-system -l app.kubernetes.io/name=chamber-agent
# View recent logs
kubectl logs -n chamber-system -l app.kubernetes.io/name=chamber-agent --tail=50
# Check installed version
helm list -n chamber-system
```
## Cluster Not Appearing in Dashboard
**Symptoms**: Agent pod is running but cluster doesn't appear in Chamber
Verify the token is correct and not expired:
```bash theme={null}
kubectl logs -l app.kubernetes.io/name=chamber-agent | grep -i "token\|auth"
```
If you see authentication errors, generate a new token from **Settings > Security > API Tokens -> New Token** in the Chamber dashboard.
The agent needs outbound HTTPS access to Chamber:
```bash theme={null}
kubectl logs -l app.kubernetes.io/name=chamber-agent | grep -i "connect\|websocket"
```
If behind a corporate firewall, you may need to configure proxy settings.
Verify the cluster name was set correctly during installation:
```bash theme={null}
kubectl logs -l app.kubernetes.io/name=chamber-agent | grep -i "cluster"
```
## GPUs Not Detected
**Symptoms**: Cluster appears but shows 0 GPUs
The NVIDIA device plugin must be running:
```bash theme={null}
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
```
If not running, install it from [NVIDIA's documentation](https://github.com/NVIDIA/k8s-device-plugin).
Check that nodes report GPU resources:
```bash theme={null}
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia\\.com/gpu
```
Nodes should show a GPU count. If showing ``, the NVIDIA drivers or device plugin may not be configured correctly.
## Workloads Not Tracked
**Symptoms**: Workloads run but don't appear in Chamber
Workloads must have the team label to be tracked:
```yaml theme={null}
metadata:
labels:
chamber.io/team: your-team-slug
```
If you configured `watchNamespaces`, verify your workload's namespace is included. By default, the agent watches all namespaces.
## Agent Not Starting
**Symptoms**: Pod in CrashLoopBackOff or Error state
```bash theme={null}
# Check logs from crashed pod
kubectl logs -l app.kubernetes.io/name=chamber-agent --previous
```
| Error | Solution |
| ----------------- | ----------------------------------------- |
| Token/auth errors | Generate new token from Chamber dashboard |
| Connection errors | Check firewall allows outbound HTTPS |
| Permission errors | Reinstall agent to fix RBAC |
## GPU Metrics Not Appearing
**Symptoms**: Dashboard shows no GPU utilization data
The agent discovers GPU metrics via DCGM-Exporter:
```bash theme={null}
kubectl logs -n chamber-system -l app.kubernetes.io/name=chamber-agent | grep -i dcgm
```
Look for "DCGM-Exporter detected" messages.
Check that metrics are being sent:
```bash theme={null}
kubectl logs -n chamber-system -l app.kubernetes.io/name=chamber-agent | grep -i "metrics"
```
Look for "Sent X metrics to Control Plane" messages.
If DCGM-Exporter is not installed, the agent can't collect GPU metrics. Install via NVIDIA GPU Operator or standalone:
```bash theme={null}
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install dcgm-exporter nvidia/dcgm-exporter -n gpu-operator
```
## GPU Usage Metric Not Available
**Symptoms**: Basic GPU metrics (utilization, memory, temperature) appear in the dashboard, but the GPU Usage metric is missing.
Chamber's GPU Usage metric requires the following five DCGM metrics to be collected:
* **Profiling metrics**: `DCGM_FI_PROF_SM_ACTIVE`, `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`, `DCGM_FI_PROF_DRAM_ACTIVE` — not enabled in the default DCGM-Exporter configuration and must be added explicitly.
* **Power metrics**: `DCGM_FI_DEV_POWER_USAGE`, `DCGM_FI_DEV_POWER_MGMT_LIMIT` — included in the default DCGM-Exporter configuration but may be missing if your cluster uses a custom metrics configuration.
All five metrics must be present for GPU Usage to be computed. If any are missing, the agent logs a warning indicating which metrics are unavailable.
If you already have a custom DCGM-Exporter metrics ConfigMap, ensure all five metrics above are included in your existing configuration. A common issue is that custom configurations may omit `DCGM_FI_DEV_POWER_MGMT_LIMIT`, which is required for the power gate calculation.
Create a ConfigMap that includes both standard and profiling metrics:
```bash theme={null}
kubectl apply -f - <
If DCGM-Exporter was installed via the NVIDIA GPU Operator, upgrade with the ConfigMap reference:
```bash theme={null}
helm upgrade gpu-operator nvidia/gpu-operator \
-n gpu-operator \
--reuse-values \
--set dcgmExporter.config.name=dcgm-exporter-metrics
```
The GPU Operator automatically restarts DCGM-Exporter pods and handles the volume mount configuration. This approach is persistent across future upgrades.
Alternatively, patch the DCGM-Exporter daemonset directly:
```bash theme={null}
kubectl patch daemonset nvidia-dcgm-exporter -n gpu-operator --type='json' -p='[
{
"op": "replace",
"path": "/spec/template/spec/containers/0/env/2/value",
"value": "/etc/dcgm-exporter/dcgm-metrics.csv"
},
{
"op": "add",
"path": "/spec/template/spec/containers/0/volumeMounts/-",
"value": {
"name": "metrics-config",
"mountPath": "/etc/dcgm-exporter",
"readOnly": true
}
},
{
"op": "add",
"path": "/spec/template/spec/volumes/-",
"value": {
"name": "metrics-config",
"configMap": {
"name": "dcgm-exporter-metrics"
}
}
}
]'
```
Wait for the rollout to complete:
```bash theme={null}
kubectl rollout status daemonset/nvidia-dcgm-exporter -n gpu-operator --timeout=120s
```
The patch command uses a hardcoded path that assumes `DCGM_EXPORTER_COLLECTORS` is at env var index 2. If the patch fails with "path not found", check the actual index:
```bash theme={null}
kubectl get daemonset nvidia-dcgm-exporter -n gpu-operator \
-o jsonpath='{.spec.template.spec.containers[0].env}' | jq .
```
Find the index of `DCGM_EXPORTER_COLLECTORS` and adjust the patch path accordingly. This patch may also be overwritten if the GPU Operator is upgraded later.
Confirm the profiling metrics are being collected:
```bash theme={null}
# Get a DCGM-Exporter pod name
DCGM_POD=$(kubectl get pods -n gpu-operator -l app=nvidia-dcgm-exporter \
-o jsonpath='{.items[0].metadata.name}')
# Port-forward and check metrics
kubectl port-forward -n gpu-operator pod/$DCGM_POD 9400:9400 &
PF_PID=$!
sleep 3
curl -s localhost:9400/metrics | grep -E "DCGM_FI_PROF_(SM|PIPE_TENSOR|DRAM)_ACTIVE|DCGM_FI_DEV_POWER_MGMT_LIMIT"
kill $PF_PID 2>/dev/null
```
You should see all four metrics in the output:
* `DCGM_FI_PROF_SM_ACTIVE`
* `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
* `DCGM_FI_PROF_DRAM_ACTIVE`
* `DCGM_FI_DEV_POWER_MGMT_LIMIT`
If any are missing, update your DCGM-Exporter metrics ConfigMap to include them. The GPU Usage metric will appear in the Chamber dashboard shortly after all required metrics are available.
Profiling metrics require NVIDIA Turing architecture or newer GPUs (T4, V100, A100, H100, etc.). Consumer GPUs (GeForce series) may not support profiling metrics.
## Proxy Configuration
If your cluster uses an HTTP proxy:
```yaml theme={null}
# values.yaml
env:
- name: HTTPS_PROXY
value: "http://proxy.example.com:8080"
- name: NO_PROXY
value: "10.0.0.0/8,172.16.0.0/12,.cluster.local"
```
Then upgrade the agent:
```bash theme={null}
helm upgrade chamber-agent oci://public.ecr.aws/chamber/chamber-agent-chart \
-n chamber-system -f values.yaml
```
## Getting Help
If you're still having issues:
1. Collect logs: `kubectl logs -l app.kubernetes.io/name=chamber-agent > agent-logs.txt`
2. Contact support with logs and your agent version
## Related Pages
Install the agent via Helm
Upgrade to the latest version
# Upgrading
Source: https://docs.usechamber.io/agent/upgrading
Upgrade the Chamber agent to newer versions
This guide covers upgrading the Chamber agent to newer versions.
## Check Current Version
```bash theme={null}
helm list -n chamber-system
```
## Check Available Versions
Visit the [releases page](https://github.com/ChamberOrg/chamber-agent/releases) for the latest version and changelog.
## Upgrade to Latest
```bash theme={null}
helm upgrade chamber-agent oci://public.ecr.aws/chamber/chamber-agent-chart \
-n chamber-system \
--reuse-values
```
`--reuse-values` preserves your existing configuration (token, cluster ID, etc.).
## Upgrade to Specific Version
```bash theme={null}
helm upgrade chamber-agent oci://public.ecr.aws/chamber/chamber-agent-chart \
-n chamber-system \
--version 0.8.96 \
--reuse-values
```
Pinning to a specific version ensures reproducible deployments across environments.
## Verify Upgrade
```bash theme={null}
# Check rollout status
kubectl rollout status deployment/chamber-agent -n chamber-system
# Check the new version
helm list -n chamber-system
# Check logs for errors
kubectl logs -n chamber-system -l app.kubernetes.io/name=chamber-agent --tail=50
```
Look for successful connection messages:
```
INFO WebSocket connected successfully
INFO Metadata collected: X nodes, Y GPUs
```
## Rollback
If issues occur after upgrade:
```bash theme={null}
# View upgrade history
helm history chamber-agent -n chamber-system
# Rollback to previous version
helm rollback chamber-agent -n chamber-system
# Or rollback to specific revision
helm rollback chamber-agent 2 -n chamber-system
```
## Related Pages
Fresh installation guide
Fix post-upgrade issues
# Supported Scenarios
Source: https://docs.usechamber.io/ai-ops/capabilities
The GPU workload failures and operational scenarios the AI Ops Agent detects and resolves
The Chamber AI Ops Agent handles over 200 GPU infrastructure scenarios across training failures, hardware faults, scheduling issues, distributed computing problems, and capacity planning. This page covers the major categories and how the agent addresses each one.
## Scenario Overview
| Category | What It Covers | Agent Response |
| ------------------------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------- |
| [GPU Memory (OOM)](#gpu-memory-oom) | Out-of-memory errors during training | Auto-fix configuration, resubmit |
| [Distributed Training](#distributed-training) | NCCL timeouts, rank failures, communication errors | Diagnose network/config issues, resubmit with fixes |
| [Hardware Faults](#hardware-faults) | ECC errors, PCIe failures, thermal throttling | Isolate faulty hardware, migrate jobs, escalate for repair |
| [Scheduling & Preemption](#scheduling-and-preemption) | Capacity exhaustion, spot reclamation, taint mismatches | Resubmit, adjust priority, or wait for capacity |
| [Ray Workloads](#ray-workloads) | Ray Train, Serve, Data, and cluster-level failures | Framework-aware diagnosis and remediation |
| [Stuck and Cascading Failures](#stuck-and-cascading-failures) | Hanging jobs, multi-step failures, dependency chains | Multi-turn investigation, correlated diagnosis |
| [Capacity Planning](#capacity-planning) | Oversubscription, demand forecasting, utilization gaps | Analysis and recommendations for team leads |
## GPU Memory (OOM)
Out-of-memory errors are the most common GPU training failure. The agent doesn't just detect them — it identifies the specific cause and applies the right fix.
**Signal:** Job fails with `CUDA out of memory` during forward pass. GPU memory at 98%+ utilization.
**Diagnosis:** Batch size too large for model size and available GPU memory.
**Action:** Reduce batch size, enable gradient checkpointing, resubmit with corrected configuration.
**Example:** LLaMA 7B fine-tuning on A100-80GB with batch\_size=64 fails. Agent reduces to batch\_size=16 with gradient checkpointing enabled and resubmits.
**Signal:** OOM occurs during data loading, not model forward pass. Container memory limit hit.
**Diagnosis:** Too many DataLoader workers consuming host memory.
**Action:** Reduce `num_workers`, resubmit.
**Example:** 16 DataLoader workers on a node with limited host memory. Agent reduces to 4 workers and resubmits.
**Signal:** `RuntimeError: DataLoader worker is killed` with shared memory errors.
**Diagnosis:** Default `/dev/shm` size (64MB) insufficient for multi-worker data loading.
**Action:** Add `emptyDir` volume mount for `/dev/shm` with adequate size, resubmit.
**Signal:** OOM during backward pass with large sequence lengths.
**Diagnosis:** Activation memory exceeds GPU capacity without checkpointing.
**Action:** Enable gradient checkpointing and mixed precision training (`--fp16`), resubmit.
## Distributed Training
Multi-GPU and multi-node training introduces communication complexity. The agent diagnoses network, synchronization, and hardware issues across distributed training setups.
**Signal:** NCCL AllReduce timeout. Training hangs after initialization.
**Diagnosis:** NCCL defaulting to loopback interface instead of the cluster network.
**Action:** Set `NCCL_SOCKET_IFNAME=eth0` environment variable, enable NCCL debug logging, resubmit.
**Signal:** Training hangs at a specific step. Some ranks complete while others wait.
**Diagnosis:** Conditional code paths causing rank divergence in DDP training — not all ranks execute the same collective operations.
**Action:** Identify the divergent code path, recommend fix, escalate to developer if code change required.
**Signal:** NCCL falls back to TCP with 100x slower communication. Training continues but at unacceptable speed.
**Diagnosis:** InfiniBand link failure on a specific node.
**Action:** Cancel job, resubmit with node exclusion, escalate for IB hardware repair.
**Signal:** Distributed training slows progressively. NCCL timeouts occur after hours of training.
**Diagnosis:** GPU on one node throttling due to overheating (92C+), creating a straggler that slows all ranks.
**Action:** Cancel job, resubmit excluding the overheating node, escalate for cooling system inspection.
## Hardware Faults
GPU hardware issues require quick detection and isolation to prevent cascading failures.
**Signal:** CUDA errors during computation. Xid errors in kernel logs.
**Diagnosis:** ECC uncorrectable memory errors indicating GPU memory degradation.
**Action:** Cancel affected jobs, migrate to healthy GPUs, mark node for maintenance, escalate for hardware replacement.
**Signal:** `Xid 79: GPU has fallen off the bus` in system logs.
**Diagnosis:** PCIe link failure — GPU is no longer communicating with the host.
**Action:** Drain node, resubmit affected jobs to other nodes, escalate for hardware inspection.
**Signal:** Degraded GPU-to-GPU communication bandwidth on multi-GPU nodes.
**Diagnosis:** NVLink bridge errors causing fallback to PCIe for inter-GPU communication.
**Action:** Migrate multi-GPU workloads to healthy nodes, schedule node maintenance.
## Scheduling and Preemption
The agent handles the full lifecycle of workload scheduling issues.
**Signal:** Job stuck in PENDING state. No available GPUs matching the request.
**Diagnosis:** All matching GPUs allocated. Checks elastic workloads that could be preempted.
**Action:** If reserved workload: preempt eligible elastic workloads. If elastic: queue with estimated wait time, notify team.
**Signal:** Job preempted due to cloud provider spot instance reclamation.
**Diagnosis:** Spot interruption notice followed by pod termination.
**Action:** Resubmit job with same configuration. If frequent reclamation: recommend switching to reserved capacity.
**Signal:** Pod unschedulable despite available GPU resources.
**Diagnosis:** Node taints don't match pod tolerations in the workload manifest.
**Action:** Add correct tolerations to manifest, resubmit.
**Signal:** `ImagePullBackOff` or `ErrImagePull` in Kubernetes events.
**Diagnosis:** Registry authentication expired or image tag not found.
**Action:** If auth issue: escalate (requires credential update). If tag issue: recommend correct tag.
## Ray Workloads
First-class support for the Ray distributed computing framework, covering Ray Train, Ray Serve, Ray Data, and cluster management.
**Signal:** Ray worker killed by memory monitor during distributed training.
**Diagnosis:** Worker memory limit too low for model size, or missing memory optimization (DeepSpeed ZeRO, gradient checkpointing).
**Action:** Increase worker memory allocation or enable memory optimization, resubmit RayJob.
**Signal:** Ray Serve replica fails to start. Model loading exceeds replica memory.
**Diagnosis:** Model too large for configured replica resources.
**Action:** Increase replica GPU memory, or configure model parallelism across replicas.
**Signal:** RayJob CR created but Ray cluster never reaches running state.
**Diagnosis:** nodeSelector targeting wrong level (e.g., pod-level instead of node-level) or autoscaler deadlock.
**Action:** Fix YAML structure, resubmit. For autoscaler issues: diagnose scaling constraints and recommend configuration changes.
**Signal:** All Ray workers disconnect. Cluster becomes unresponsive.
**Diagnosis:** Global Control Store (GCS) crash on head node, typically from head node OOM.
**Action:** If GCS fault tolerance enabled: wait for recovery. If not: cancel and resubmit with GCS FT configuration and larger head node memory.
**Signal:** GPU utilization drops to near zero despite data pipeline running.
**Diagnosis:** Ray Data operator producing data faster than GPU training can consume, filling object store.
**Action:** Configure backpressure limits, adjust data pipeline parallelism to match GPU throughput.
## Stuck and Cascading Failures
Some failures require multi-step investigation across multiple data sources and time windows.
**Signal:** Job status is RUNNING but loss metrics haven't updated in 30+ minutes.
**Diagnosis:** Agent checks GPU utilization (if near 0%, likely deadlock or I/O stall), network metrics (if communication stalled), and recent log entries to pinpoint where the job is stuck.
**Action:** Based on diagnosis — may fix I/O bottleneck, restart communication, or cancel and resubmit with corrected config.
**Signal:** Multiple jobs fail simultaneously with different error messages.
**Diagnosis:** Agent correlates the failures to a single node going offline, rather than treating each as an independent issue.
**Action:** Identify the root cause (node crash), resubmit all affected jobs with node exclusion, escalate for node repair.
**Signal:** Training job blocked, but the error traces back through preprocessing to an expired credential.
**Diagnosis:** Multi-hop investigation: training blocked by preprocessing failure, preprocessing blocked by expired S3 credentials.
**Action:** Escalate credential refresh, then resubmit the full pipeline.
**Signal:** Same job fails repeatedly with different error messages at roughly the same training step.
**Diagnosis:** Agent correlates across attempts — discovers the underlying cause (e.g., thermal throttling causing GPU clock reduction) is the same despite different surface errors.
**Action:** Address the root cause rather than chasing individual error messages.
## Capacity Planning
Beyond reactive incident handling, the agent provides proactive capacity intelligence.
**Signal:** Team consistently hitting queue limits during peak hours.
**Diagnosis:** Analyzes submission patterns over time — identifies peak demand windows, utilization gaps, and oversubscription trends.
**Recommendations:** GPU allocation adjustments, time-slot scheduling for peak hours, priority queuing for short-duration jobs. Escalates to team leads for capacity decisions.
**Signal:** Heartbeat detects GPUs sitting idle while jobs are queued in other teams.
**Diagnosis:** Identifies underutilized allocations and teams that could benefit from elastic burst capacity.
**Recommendations:** Rebalancing suggestions, elastic capacity sharing policies, right-sizing team allocations.
## Next Steps
How the agent keeps your infrastructure safe with five layers of guardrails
Enable AI Ops for your organization
# Getting Started
Source: https://docs.usechamber.io/ai-ops/getting-started
Enable the AI Ops Agent for your organization and configure it for your team's workflow
**Private Beta** — The AI Ops Agent is currently in private beta and available to select customers. To request access, [contact us](mailto:support@usechamber.io) or reach out to your account representative.
Get the Chamber AI Ops Agent running for your organization in minutes. The agent connects to your existing Chamber deployment — no additional cluster-side components required.
## Prerequisites
* A Chamber account with admin access
* At least one Kubernetes cluster with the [Chamber agent installed](/agent/installation)
* A Slack workspace for your team
## Setup
Navigate to **Settings > AI Ops Agent** in the [Chamber dashboard](https://app.usechamber.io). Toggle **Enable AI Ops Agent** to on.
Click **Add to Slack** to authorize the AI Ops Agent in your workspace. This creates the `@chambie-ops` bot in your Slack — separate from any existing Chamber integrations.
Select the channels where the agent should post:
* **Alerts channel** — real-time incident notifications (e.g., `#ml-ops-alerts`)
* **Daily reports channel** — scheduled summaries (e.g., `#ml-ops-daily`)
* **Approvals channel** — approval requests for high-risk actions (e.g., `#ml-ops-approvals`)
* **Escalation channel** — unresolved issues that need human attention (e.g., `#ml-ops-escalation`)
Set the operational boundaries for the agent. The default policy allows read-only operations and requires approval for all write actions — a safe starting point.
You can customize:
| Setting | Default | Description |
| ---------------------------- | ---------- | --------------------------------------------------- |
| **Heartbeat interval** | 5 minutes | How often the agent checks your infrastructure |
| **Daily budget limit** | \$500 | Maximum GPU spend the agent can authorize per day |
| **Monthly budget limit** | \$10,000 | Maximum GPU spend per month |
| **Max GPUs per auto-submit** | 4 | Jobs up to this size can be auto-submitted (Tier 2) |
| **Tier 3 approval channel** | — | Slack channel for high-risk approval requests |
| **Tier 4 approvers** | — | Specific users who can approve destructive actions |
| **Approval timeout** | 30 minutes | How long before unanswered approvals escalate |
Once enabled, the agent starts its first heartbeat check within 5 minutes. You'll see a message in your alerts channel:
```
@chambie-ops connected
Monitoring: 3 capacity pools, 48 GPUs, 12 active workloads
Policy: Tier 0-2 auto, Tier 3+ approval required
Heartbeat: every 5 minutes
Daily report: 9:00 AM to #ml-ops-daily
All systems healthy. HEARTBEAT_OK
```
If everything is healthy, the agent will be silent until something needs attention.
Start with the default policy (read-only auto, approval for writes) and gradually expand autonomous permissions as you build confidence in the agent's behavior.
## What Happens Next
Once running, the AI Ops Agent immediately begins:
1. **Monitoring** your active workloads, GPU health, and capacity utilization
2. **Posting daily cost reports** to your configured channel
3. **Responding to failures** as they occur — diagnosing, recommending, and acting within your policy
4. **Building its knowledge base** — learning the patterns specific to your infrastructure
The AI Ops Agent is a separate Slack bot (`@chambie-ops`) from Chamber's conversational assistant (`@chambie`). The conversational assistant handles ad-hoc questions and manual tasks. The AI Ops Agent handles autonomous monitoring and remediation.
## Adjusting the Policy Over Time
As the agent proves itself, you can gradually expand its autonomous capabilities:
| Phase | Recommended Policy | What Changes |
| ------------ | ------------------------------------------------------- | ------------------------------------------------------- |
| **Week 1** | Read-only auto, all writes need approval | Observe how the agent diagnoses issues |
| **Week 2-4** | Auto-submit small elastic jobs (up to 4 GPUs) | Agent handles routine OOM resubmissions automatically |
| **Month 2+** | Auto-submit up to 16 GPUs, auto-resubmit preempted jobs | Agent handles the majority of routine incidents |
| **Ongoing** | Tune based on your comfort level | Review the audit trail and expand or restrict as needed |
## Next Steps
Learn what the AI Ops Agent does and why it matters
See the full range of failures the agent can handle
Deep dive into the five-layer safety model
Configure your existing Slack integration
# How It Works
Source: https://docs.usechamber.io/ai-ops/how-it-works
Three-trigger proactivity, intelligent diagnosis, and a learning system that compounds expertise over time
The Chamber AI Ops Agent combines proactive monitoring, intelligent diagnosis, and safe remediation into a continuous operational loop. This page explains the architecture behind it.
## Three-Trigger Proactivity Model
Most monitoring tools are reactive — they wait for an alert, then notify you. The AI Ops Agent is proactive. It uses three complementary trigger mechanisms to ensure nothing slips through the cracks.
```mermaid theme={null}
flowchart TD
HB["Heartbeat
(every 5 min)"]
CR["Scheduled Tasks
(cron)"]
EV["Event-Driven
(real-time)"]
HB -->|"Anything wrong?"| Agent["AI Ops Agent"]
CR -->|"Generate report"| Agent
EV -->|"Job failed"| Agent
Agent --> Diagnose["Diagnose"]
Diagnose --> Act["Act or Escalate"]
Act --> Verify["Verify"]
Verify --> Learn["Learn"]
```
### Heartbeat — Catching What Alerts Miss
Every 5 minutes, the agent checks in on your infrastructure and asks: *Is anything wrong?*
This catches the failures that don't produce alerts — GPUs sitting idle, training jobs running slower than expected, costs drifting above budget, or utilization gradually degrading.
If everything is healthy, the heartbeat is silent. No noise, no unnecessary messages.
The heartbeat interval is configurable per organization. 5 minutes is the default — you can set it anywhere from 1 to 30 minutes.
### Scheduled Tasks — Reports and Reviews on Your Schedule
Recurring operational tasks run on a schedule you define:
| Task | Default Schedule | What It Does |
| ---------------------- | ---------------- | ----------------------------------------------------------------- |
| Daily cost report | 9:00 AM | GPU spend breakdown, trends, anomalies |
| Weekly capacity review | Monday 10:00 AM | Utilization trends, demand forecast, optimization recommendations |
| Training loop check | Every 5 minutes | Monitors active training jobs for failures or stalls |
Each scheduled task runs in an isolated session — they don't interfere with each other or with real-time incident handling.
### Event-Driven — Real-Time Response
When something happens in your cluster, the agent responds immediately:
| Event | Response Time | Agent Action |
| ---------------------- | ------------- | --------------------------------------------------------------- |
| Job failure | \< 30 seconds | Diagnose root cause, attempt auto-fix or escalate |
| GPU hardware fault | \< 30 seconds | Assess blast radius, migrate affected jobs, request maintenance |
| Budget threshold (80%) | \< 1 minute | Summarize spending, alert team, recommend adjustments |
| Pod eviction | \< 30 seconds | Determine cause, resubmit if appropriate |
## Diagnosis Flow
When the agent detects an issue — whether from a heartbeat, scheduled check, or real-time event — it follows a structured diagnosis flow.
```mermaid theme={null}
flowchart LR
O["Observe
(collect evidence)"] --> D["Diagnose
(root cause)"]
D --> R["Recommend
(choose action)"]
R --> A["Act
(execute safely)"]
A --> V["Verify
(confirm fix)"]
V --> L["Learn
(bank pattern)"]
```
### Observe
The agent collects evidence before making any decisions:
* **Workload details** — job configuration, GPU type, resource requests
* **Logs** — recent error messages and stack traces
* **Metrics** — GPU memory, utilization, temperature over time
* **Kubernetes events** — pod status, node conditions, scheduling events
### Diagnose
Using the collected evidence, the agent performs root-cause analysis — not just pattern matching on error strings. It correlates across data sources to identify the actual cause.
For example, an "NCCL timeout" error might be caused by:
* A network interface misconfiguration (fix: set `NCCL_SOCKET_IFNAME`)
* An InfiniBand hardware failure (fix: migrate to healthy nodes)
* A thermal-throttled GPU creating a straggler (fix: exclude overheating node)
The agent distinguishes between these by checking network metrics, hardware status, and thermal data — then applies the right fix.
### Recommend, Act, Verify
Based on the diagnosis and the action's [risk tier](/ai-ops/safety), the agent either:
* **Auto-executes** low-risk fixes (e.g., resubmit with corrected batch size)
* **Requests approval** for high-risk actions (e.g., cancel a running job)
* **Escalates** when human judgment is needed (e.g., hardware replacement)
After acting, the agent monitors the fix to verify it worked — and notifies your team of the outcome.
## Tiered Intelligence
The agent uses different AI model tiers depending on the complexity of the task, optimizing for both speed and cost:
| Tier | Used For | Examples |
| ------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **Fast** | Heartbeat checks, event triage, status queries | "Are all jobs healthy?" — answered in milliseconds |
| **Reasoning** | Root-cause analysis, pattern matching, planning | "Why did this distributed training job fail?" — deep multi-source analysis |
| **Critical** | Destructive decisions, novel failures, complex multi-step remediation | "Should we cancel this 64-GPU job and reallocate?" — careful reasoning with full context |
Fast-tier checks keep the agent's operating cost low during quiet periods. The agent only escalates to more powerful reasoning when the situation demands it.
## Memory and Learning
The AI Ops Agent doesn't just fix problems — it learns from them. Over time, it builds a knowledge base specific to your infrastructure.
### Pattern Bank
Every resolved incident becomes a reusable pattern:
```
Signal: Job fails with "CUDA out of memory" on A100 80GB, batch_size > 32
Cause: Mixed precision not enabled, gradient accumulation not configured
Fix: Enable --fp16, set gradient_accumulation_steps = batch_size / 8
Trap: Don't just halve batch_size — throughput drops 50%. Fix root cause.
```
When the agent encounters a matching signal in the future, it applies the known fix immediately — no re-diagnosis needed.
### Failure Journal
When a fix doesn't work, the agent records what went wrong and adds a prevention rule:
```
What happened: Resubmitted OOM job but didn't verify node selector.
Job landed on T4 instead of A100, failed with different error.
Prevention: Always verify GPU type matches original job before resubmission.
```
### How Expertise Compounds
```mermaid theme={null}
flowchart LR
E["Encounter
(new failure)"] --> D["Diagnose"]
D --> F["Fix"]
F --> O["Outcome"]
O -->|Success| B["Bank Pattern"]
O -->|Failure| J["Journal + Escalate"]
B --> E
J --> E
```
After several months of operation, the pattern bank becomes a comprehensive runbook specific to your infrastructure — covering the failure modes, GPU types, and workload patterns unique to your environment.
The agent's memory is human-readable and auditable. Your team can review, edit, and version-control the pattern bank alongside your infrastructure code.
## Notification Philosophy
The agent communicates through Slack with a clear philosophy: **inform without overwhelming**.
* **3-5 messages per incident**, not dozens
* **Batched updates** — status changes are grouped, not streamed
* **Silent when healthy** — heartbeats produce no output when everything is fine
* **Structured reports** — daily and weekly summaries go to designated channels
* **Approval requests** — interactive Slack buttons for actions that need human sign-off
## Next Steps
See the full range of failures the agent detects and resolves
Learn about the five-layer safety model and risk-tiered actions
# AI Ops Agent
Source: https://docs.usechamber.io/ai-ops/overview
Your always-on AI operations teammate that monitors, diagnoses, and resolves GPU workload issues
Chamber's AI Ops Agent is an autonomous operations teammate that watches over your GPU infrastructure 24/7. It detects failures in seconds, diagnoses root causes, and resolves routine issues automatically — so your ML engineers can focus on building models, not firefighting infrastructure.
Proactive heartbeat checks every 5 minutes catch silent failures, idle GPUs, and drifting costs — without waiting for alerts
Root-cause analysis powered by AI that correlates logs, metrics, and Kubernetes events to pinpoint exactly what went wrong
Resolves routine issues like OOM errors, preempted jobs, and scheduling conflicts automatically with safe, verified fixes
Builds a knowledge base specific to your infrastructure — getting faster and more accurate with every incident it handles
## How Chamber AI Ops Fits In
The AI Ops Agent extends Chamber's control plane with autonomous operational intelligence. It works alongside the Kubernetes agent you already have deployed — no additional cluster-side components required.
```mermaid theme={null}
flowchart LR
subgraph cluster["Your GPU Cluster"]
KAgent["Chamber
Kubernetes Agent"]
GPUs["GPU Nodes"]
end
subgraph chamber["Chamber Platform"]
CP["Control Plane"]
AI["AI Ops Agent"]
end
subgraph team["Your Team"]
Slack["Slack"]
end
KAgent -->|metrics, events| CP
CP -->|real-time events| AI
AI -->|diagnose, remediate| CP
CP -->|schedule, manage| KAgent
KAgent --> GPUs
AI -->|"alerts, approvals,
reports"| Slack
```
## What Makes It Different
Doesn't wait for alerts. Continuously checks your infrastructure and catches issues before they escalate — including silent failures that don't produce events.
Purpose-built for ML infrastructure. Knows the difference between an OOM from a large batch size and an OOM from a memory leak. Fixes the root cause, not just the symptom.
Five layers of safety guardrails. Risk-tiered actions. Human approval for anything destructive. Full audit trail. Your infrastructure, your rules.
## A Day in the Life
Here's what the AI Ops Agent does for your team on a typical day:
| Time | What Happens | How You See It |
| ----------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| **6:02 AM** | Heartbeat detects a training job stuck for 45 minutes | Slack alert with diagnosis and recommended fix |
| **6:03 AM** | Identifies OOM error, reduces batch size, resubmits job | Slack notification: "Resubmitted job with fix. Monitoring." |
| **6:18 AM** | Verifies resubmitted job is training successfully | Slack update: "Job progressing normally. Loss decreasing." |
| **9:00 AM** | Daily cost report generated | Posted to #ml-ops-daily with spend breakdown and trends |
| **2:15 PM** | GPU hardware fault detected on a node | Slack alert: "ECC errors on node-gpu-07. Migrated affected jobs. Requesting maintenance." |
| **2:16 PM** | Sends approval request for node isolation | Slack approval buttons — your team decides |
The AI Ops Agent sends 3-5 messages per incident, not dozens. It batches updates and only notifies on meaningful state changes.
## Next Steps
Deep dive into the three-trigger proactivity model, diagnosis flow, and learning system
Explore the 200+ failure scenarios the agent can detect and resolve
Understand the five-layer safety model and enterprise governance controls
Enable AI Ops for your organization
# Safety & Governance
Source: https://docs.usechamber.io/ai-ops/safety
Five-layer defense-in-depth safety model, risk-tiered actions, and enterprise governance controls
Autonomous agents operating on GPU infrastructure must be safe by design. The Chamber AI Ops Agent uses a five-layer defense-in-depth model — every action passes through multiple independent safety checks before execution. No single layer's failure can result in an unauthorized action.
## Five-Layer Defense-in-Depth
```mermaid theme={null}
flowchart TB
L5["Layer 5: Server Guardrails
(Chamber API — ultimate safety net)"]
L4["Layer 4: Human Approval
(Slack approval for high-risk actions)"]
L3["Layer 3: CLI Guardrails
(budget checks, scope validation)"]
L2["Layer 2: Tenant Policy
(deny-by-default YAML policy)"]
L1["Layer 1: Credential Isolation
(per-org scoped credentials)"]
L1 --> L2 --> L3 --> L4 --> L5
```
| Layer | What It Does | Cannot Be Bypassed By |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| **Layer 1: Credential Isolation** | Each organization's agent runs with its own scoped credentials. No cross-tenant access is possible at the infrastructure level. | Any client-side code |
| **Layer 2: Tenant Policy** | A deny-by-default policy defines exactly which actions the agent can take for your organization. Anything not explicitly allowed is blocked. | Agent reasoning or prompts |
| **Layer 3: CLI Guardrails** | Budget checks, scope validation, and rate limits built into the Chamber CLI. Every write action is validated before execution. | Policy misconfiguration |
| **Layer 4: Human Approval** | High-risk actions require explicit human approval via Slack before execution. Approvals are cryptographically bound and single-use. | Automated processes |
| **Layer 5: Server Guardrails** | Chamber's API server enforces final budget limits, scope validation, and rate limits. This is the ultimate safety net — even if all client-side layers fail, the server rejects unauthorized actions. | Nothing — this is the final gate |
Layers 1, 2, and 5 are always active and cannot be disabled. Layers 3 and 4 provide additional defense for write operations.
## Risk-Tiered Action Model
Every action the agent can take is classified into a risk tier. Higher tiers require progressively more safety checks.
| Tier | Category | Examples | Policy |
| ------------------- | --------------------- | -------------------------------------------------------------------------- | --------------------------------------------- |
| **0 — Observe** | Read-only queries | List workloads, get metrics, check GPU status, read logs | Auto-execute |
| **1 — Inform** | Analysis and insights | Explain failures, analyze capacity trends, generate reports | Auto-execute, notify team |
| **2 — Low Write** | Safe mutations | Submit small jobs (up to 4 GPUs), create allocations (up to 8 instances) | **Dry-run first**, then execute |
| **3 — High Write** | Significant mutations | Submit large jobs (over 4 GPUs), transfer allocations, create reservations | **Human approval required** |
| **4 — Destructive** | Irreversible actions | Cancel running jobs, release allocations | **Dual approval required**, mandatory audit |
| **5 — Prohibited** | Dangerous operations | Delete pools, destroy infrastructure | **Hard blocked** — not available to the agent |
### Dry-Run Enforcement
For Tier 2 actions, the agent always performs a dry-run first. The dry-run shows exactly what will happen — resources consumed, configuration applied, estimated cost — before any mutation occurs. Only after the dry-run passes validation does the agent execute.
### Human Approval Workflow
For Tier 3 and Tier 4 actions, the agent sends an approval request to your team via Slack:
```
@chambie-ops requests approval:
Action: Cancel job llama-finetune-v3 (job-a1b2c3d4)
Reason: GPU ECC errors detected on assigned node. Job producing
corrupted gradients. Recommend cancel and resubmit on
healthy hardware.
Risk Tier: 4 (Destructive)
Evidence: 12 uncorrectable ECC errors in last 10 minutes.
Loss spiked to NaN at step 4,521.
[Approve] [Deny]
Expires in 30 minutes. If no response, escalates to #ml-ops-escalation.
```
**Approval security:**
* Approval buttons are **cryptographically signed** — they cannot be forged or replayed
* Each approval is **single-use** — clicking Approve twice has no effect
* Approvals **expire** after 30 minutes (configurable) — unresolved requests escalate automatically
* **Approver identity is verified** — only authorized team members can approve Tier 4 actions
## Deny-by-Default Tenant Policy
Your organization's tenant policy defines exactly what the agent is allowed to do. Everything else is denied.
```yaml theme={null}
# Example tenant policy
version: "1.0"
allowed_tools:
# Read-only operations — always allowed
tier_0:
- workloads.list
- workloads.get
- metrics.get
- failure_analysis.diagnose
- gpus.status
- logs.get
# Analysis — allowed with notification
tier_1:
- workloads.explain
- insights.analyze
- capacity.plan
# Low-risk writes — allowed with constraints
tier_2:
- workloads.submit:
max_gpus: 4
job_type: [elastic]
- allocations.create:
max_instances: 8
# High-risk writes — require approval
tier_3:
- workloads.submit:
max_gpus: 64
- allocations.transfer
- reservations.create
# Destructive — require dual approval
tier_4:
- workloads.cancel
- allocations.release
budget:
daily_limit_usd: 500
monthly_limit_usd: 10000
approval:
tier_3_channel: "#ml-ops-approvals"
tier_4_approvers: ["@alice", "@bob"]
timeout_minutes: 30
escalation_channel: "#ml-ops-escalation"
```
The tenant policy is version-controlled and auditable. Changes to the policy require manual configuration — the agent cannot modify its own permissions.
## Audit Trail
Every action the agent takes is logged with full context:
| Field | Description |
| ------------- | -------------------------------------------------------- |
| **Action** | What the agent did (e.g., `workloads.submit`) |
| **Risk Tier** | The action's risk classification |
| **Reason** | Why the agent took this action (root-cause description) |
| **Evidence** | The logs, metrics, and events that informed the decision |
| **Approver** | Who approved the action (for Tier 3+) |
| **Outcome** | Whether the action succeeded and its effect |
| **Timestamp** | When the action occurred |
Audit records are retained for compliance review and can be exported for integration with your existing governance tools.
## Incident Deduplication
When multiple signals detect the same issue — for example, a real-time event and the next heartbeat both notice a failed job — the agent deduplicates them. Only one remediation runs per incident.
If two team members try to address the same issue from different Slack threads, the second is told the issue is already being handled and is offered updates on the resolution.
## Guardrail Enforcement Examples
A resubmission would exceed the daily budget limit. The agent blocks the action, notifies the team, and suggests waiting until the next budget cycle or requesting a budget increase.
A job requests 128 GPUs but the tenant policy caps individual jobs at 64. The agent blocks the request locally — it never reaches the API.
Releasing an allocation that has running workloads is a Tier 4 action. The agent sends a dual-approval request showing which workloads would be affected. Both designated approvers must approve before execution.
The agent encounters a failure it hasn't seen before and cannot confidently diagnose. Rather than guessing, it escalates to your team with the evidence it has collected — logs, metrics, and its best hypothesis — for human review.
## Next Steps
Enable AI Ops and configure your tenant policy
Learn about the three-trigger proactivity model and diagnosis flow
# Capacity
Source: https://docs.usechamber.io/api-reference/capacity
Check budget allocations and remaining GPU hours
## Get Capacity
Retrieve capacity budget and allocation information for your organization.
```bash theme={null}
GET /v1/capacity
```
### Query Parameters
| Parameter | Type | Description |
| --------------- | ------ | -------------------------------- |
| `initiative_id` | string | Filter to specific team |
| `pool_id` | string | Filter to specific capacity pool |
### Example Request
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/capacity" \
-H "Authorization: Bearer " \
-H "X-Organization-Id: "
```
### Example Response
```json theme={null}
{
"data": {
"organization_id": "org_abc123",
"summary": {
"total_allocated_gpu_hours": 10000,
"total_used_gpu_hours": 6500,
"total_remaining_gpu_hours": 3500,
"current_active_gpus": 24
},
"pools": [
{
"pool_id": "pool_prod",
"name": "Production Pool",
"allocated_gpu_hours": 6000,
"used_gpu_hours": 4200,
"remaining_gpu_hours": 1800,
"instance_types": ["p4d.24xlarge", "p5.48xlarge"]
},
{
"pool_id": "pool_dev",
"name": "Development Pool",
"allocated_gpu_hours": 4000,
"used_gpu_hours": 2300,
"remaining_gpu_hours": 1700,
"instance_types": ["g5.xlarge", "g5.2xlarge"]
}
],
"initiatives": [
{
"initiative_id": "init_ml_team",
"name": "ML Platform Team",
"allocated_gpu_hours": 5000,
"used_gpu_hours": 3200,
"remaining_gpu_hours": 1800
},
{
"initiative_id": "init_research",
"name": "Research Team",
"allocated_gpu_hours": 5000,
"used_gpu_hours": 3300,
"remaining_gpu_hours": 1700
}
]
},
"request_id": "req_abc123"
}
```
### Response Fields
| Field | Description |
| --------------------------- | --------------------------------------------- |
| `total_allocated_gpu_hours` | Total GPU hours allocated to the organization |
| `total_used_gpu_hours` | GPU hours consumed across all workloads |
| `total_remaining_gpu_hours` | Available GPU hours remaining |
| `current_active_gpus` | Number of GPUs currently in use |
| `pools` | Breakdown by capacity pool |
| `initiatives` | Breakdown by team |
# Health
Source: https://docs.usechamber.io/api-reference/health
Service health check endpoint
## Health Check
Check the health status of the Chamber API service.
This endpoint does not require authentication and can be used for monitoring and load balancer health checks.
```bash theme={null}
GET /v1/health
```
### Example Request
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/health"
```
### Example Response
```json theme={null}
{
"data": {
"status": "healthy",
"version": "1.0.0",
"timestamp": "2024-01-15T10:30:00Z"
}
}
```
### Response Fields
| Field | Description |
| ----------- | ----------------------------------------------- |
| `status` | Service health status: `healthy` or `unhealthy` |
| `version` | API version number |
| `timestamp` | Current server timestamp (ISO 8601) |
# API Reference
Source: https://docs.usechamber.io/api-reference/introduction
Chamber REST API for programmatic access to GPU capacity and workload management
The Chamber API provides programmatic access to manage GPU capacity, monitor workloads, and query metrics across your organization.
## Base URL
```
https://api.usechamber.io/v1
```
## Authentication
All API requests (except `/v1/health`) require authentication using a Bearer token.
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/workloads" \
-H "Authorization: Bearer " \
-H "X-Organization-Id: "
```
### Required Headers
| Header | Required | Description |
| ------------------- | -------- | ------------------------------------------------------ |
| `Authorization` | Yes | Bearer token (JWT from Cognito or API token) |
| `X-Organization-Id` | Yes\* | Your organization ID (\*not required for `/v1/health`) |
| `X-Request-Id` | No | Optional request ID for tracing |
Contact [support@usechamber.io](mailto:support@usechamber.io) to obtain API credentials for your organization.
## Response Format
All successful responses follow a consistent structure:
```json theme={null}
{
"data": {
// Response content
},
"metadata": {
"next_token": "...",
"total_count": 100
},
"request_id": "req_abc123"
}
```
### Error Responses
Errors return an appropriate HTTP status code with details:
```json theme={null}
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Invalid parameter value",
"details": [...]
},
"request_id": "req_abc123"
}
```
## Authorization Model
Access to resources is controlled by your role within the organization:
| Role | Access Level |
| ----------- | ------------------------------------------------- |
| `ORG_ADMIN` | Full organization access |
| `TEAM_LEAD` | Team-specific access with management capabilities |
| `MEMBER` | Team-specific read access |
## Endpoints
List, retrieve, and get statistics for GPU workloads
Query GPU utilization, memory, temperature, and power metrics
Check budget allocations and remaining GPU hours
Service health check endpoint
# Metrics
Source: https://docs.usechamber.io/api-reference/metrics
Query GPU utilization, memory, temperature, and power metrics
## Get Workload Metrics
Retrieve GPU metrics for a specific workload.
```bash theme={null}
GET /v1/workloads/{workload_id}/metrics
```
### Path Parameters
| Parameter | Type | Description |
| ------------- | ------ | ------------------------------ |
| `workload_id` | string | The unique workload identifier |
### Query Parameters
| Parameter | Type | Description |
| ------------ | ------ | --------------------------------------------------------------------------------- |
| `time_range` | string | Time range: `last_1h`, `last_6h`, `last_24h`, `job_lifetime` (default: `last_1h`) |
| `metrics` | string | Comma-separated list of metrics to include |
### Available Metrics
| Metric | Description |
| -------------------- | ---------------------------------- |
| `gpu_utilization` | GPU compute utilization percentage |
| `memory_utilization` | GPU memory utilization percentage |
| `temperature` | GPU temperature in Celsius |
| `power_usage` | Power consumption in Watts |
| `gpu_usage` | Overall GPU usage metric |
### Example Request
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/workloads/wl_abc123/metrics?time_range=last_6h&metrics=gpu_utilization,memory_utilization" \
-H "Authorization: Bearer " \
-H "X-Organization-Id: "
```
### Example Response
```json theme={null}
{
"data": {
"workload_id": "wl_abc123",
"time_range": "last_6h",
"metrics": {
"gpu_utilization": {
"avg": 85.2,
"min": 45.0,
"max": 99.8,
"current": 92.1
},
"memory_utilization": {
"avg": 78.5,
"min": 60.2,
"max": 89.4,
"current": 82.0
}
}
},
"request_id": "req_abc123"
}
```
***
## Get Ranked Workload Metrics
Retrieve metrics across workloads, ranked by utilization or consumption.
```bash theme={null}
GET /v1/workloads/metrics
```
### Query Parameters
| Parameter | Type | Description |
| --------------- | ------- | -------------------------------------------------------------------------------- |
| `sort_by` | string | Sort metric: `gpu_utilization`, `memory_utilization`, `power_usage`, `gpu_hours` |
| `order` | string | Sort order: `asc`, `desc` (default: `desc`) |
| `limit` | integer | Number of results (1-100, default: 10) |
| `initiative_id` | string | Filter to specific team |
| `status` | string | Filter by workload status |
### Example Request
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/workloads/metrics?sort_by=gpu_utilization&order=desc&limit=5" \
-H "Authorization: Bearer " \
-H "X-Organization-Id: "
```
### Example Response
```json theme={null}
{
"data": {
"workloads": [
{
"workload_id": "wl_abc123",
"name": "training-large-model",
"gpu_utilization": {
"avg": 95.2,
"current": 97.1
},
"initiative_id": "init_ml_team"
},
{
"workload_id": "wl_def456",
"name": "inference-batch",
"gpu_utilization": {
"avg": 88.7,
"current": 91.0
},
"initiative_id": "init_research"
}
],
"scope_average": {
"gpu_utilization": 72.5
}
},
"metadata": {
"total_count": 45
},
"request_id": "req_abc123"
}
```
# Workloads
Source: https://docs.usechamber.io/api-reference/workloads
List, retrieve, and get statistics for GPU workloads
## List Workloads
Retrieve a paginated list of workloads with optional filtering.
Results are automatically filtered based on your role and team access.
```bash theme={null}
GET /v1/workloads
```
### Query Parameters
| Parameter | Type | Description |
| ----------------- | ------- | -------------------------------------------------------------------------- |
| `organization_id` | string | Filter by organization |
| `initiative_id` | string | Filter by team |
| `status` | string | Filter by status: `pending`, `running`, `completed`, `failed`, `cancelled` |
| `submitted_by` | string | Filter by user who submitted the workload |
| `is_managed` | boolean | Filter managed vs unmanaged workloads |
| `start_date` | string | Filter workloads submitted after this date (ISO 8601) |
| `end_date` | string | Filter workloads submitted before this date (ISO 8601) |
| `limit` | integer | Number of results per page (1-100, default: 20) |
| `next_token` | string | Pagination cursor from previous response |
### Example Request
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/workloads?status=running&limit=10" \
-H "Authorization: Bearer " \
-H "X-Organization-Id: "
```
### Example Response
```json theme={null}
{
"data": {
"workloads": [
{
"workload_id": "wl_abc123",
"name": "training-job-001",
"status": "running",
"submitted_by": "user@example.com",
"initiative_id": "init_xyz",
"instance_type": "p4d.24xlarge",
"gpu_count": 8,
"submitted_at": "2024-01-15T10:30:00Z",
"started_at": "2024-01-15T10:32:00Z"
}
]
},
"metadata": {
"next_token": "eyJ...",
"total_count": 45
},
"request_id": "req_abc123"
}
```
***
## Get Workload
Retrieve details for a specific workload.
```bash theme={null}
GET /v1/workloads/{workload_id}
```
### Path Parameters
| Parameter | Type | Description |
| ------------- | ------ | ------------------------------ |
| `workload_id` | string | The unique workload identifier |
### Example Request
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/workloads/wl_abc123" \
-H "Authorization: Bearer " \
-H "X-Organization-Id: "
```
### Example Response
```json theme={null}
{
"data": {
"workload_id": "wl_abc123",
"name": "training-job-001",
"status": "running",
"submitted_by": "user@example.com",
"initiative_id": "init_xyz",
"instance_type": "p4d.24xlarge",
"gpu_count": 8,
"submitted_at": "2024-01-15T10:30:00Z",
"started_at": "2024-01-15T10:32:00Z",
"completed_at": null,
"duration_seconds": 3600,
"gpu_hours_used": 8.0,
"exit_code": null,
"is_managed": true,
"cluster_id": "cluster_prod1",
"namespace": "ml-training"
},
"request_id": "req_abc123"
}
```
***
## Get Workload Statistics
Retrieve aggregated statistics across workloads.
```bash theme={null}
GET /v1/workloads/stats
```
### Query Parameters
| Parameter | Type | Description |
| --------------- | ------ | --------------------------------------------------------------------------------------------- |
| `time_range` | string | Predefined range: `today`, `this_week`, `this_month`, `last_7_days`, `last_30_days`, `custom` |
| `start_date` | string | Start date for custom range (ISO 8601) |
| `end_date` | string | End date for custom range (ISO 8601) |
| `group_by` | string | Group results by: `submitted_by`, `initiative_id`, `instance_type` |
| `initiative_id` | string | Filter to specific team |
### Example Request
```bash theme={null}
curl -X GET "https://api.usechamber.io/v1/workloads/stats?time_range=last_7_days&group_by=initiative_id" \
-H "Authorization: Bearer " \
-H "X-Organization-Id: "
```
### Example Response
```json theme={null}
{
"data": {
"summary": {
"total_workloads": 156,
"total_gpu_hours": 2450.5,
"average_duration_seconds": 7200,
"success_rate": 0.94
},
"grouped": [
{
"initiative_id": "init_ml_team",
"workload_count": 89,
"gpu_hours": 1200.0,
"success_rate": 0.96
},
{
"initiative_id": "init_research",
"workload_count": 67,
"gpu_hours": 1250.5,
"success_rate": 0.91
}
]
},
"request_id": "req_abc123"
}
```
# AI Assistant
Source: https://docs.usechamber.io/cli/chat
Ask questions about your GPU infrastructure, workloads, and capacity directly from the terminal
# AI Assistant
**Get answers without leaving your terminal.** The `chamber chat` command connects you to Chamber's AI assistant — ask about workload status, GPU utilization, capacity planning, and more in natural language.
```bash theme={null}
chamber chat "how many GPUs are running right now?"
```
AI Chat requires browser-based authentication. Run `chamber login` (without `--token`) to authenticate. API tokens are not supported for chat.
## Modes
### Quick Question
Pass your question as an argument for a single response:
```bash theme={null}
chamber chat "what's the status of my latest training job?"
```
```bash theme={null}
chamber chat "which teams have the highest GPU utilization this week?"
```
### Interactive Conversation
Run `chamber chat` without arguments to start a multi-turn conversation:
```bash theme={null}
$ chamber chat
Chamber AI Assistant
Type /help for commands, /quit to exit
> what jobs are currently running?
You have 3 jobs running:
• gpt-finetune-v2 — 8x H100, running for 2h 15m
• embedding-train — 4x A100, running for 45m
• llm-eval — 2x H100, running for 12m
> how much capacity is left in the H100 pool?
The us-west-h100 pool has 16 of 64 GPUs available (75% utilized).
Your team has 12 GPU-hours remaining in this billing period.
> /quit
```
Interactive commands:
| Command | Action |
| ------------------ | --------------------------- |
| `/quit` or `/exit` | End the session |
| `/new` | Start a new conversation |
| `/help` | Show available commands |
| `Esc` or `Ctrl+C` | Cancel the current response |
### Named Conversations
Use `-c` to name a conversation and resume it later:
```bash theme={null}
# Start a named conversation
chamber chat -c debug-job "my training job failed, can you help?"
# Come back later and continue where you left off
chamber chat -c debug-job "I tried your suggestion, still seeing OOM errors"
```
### Pipe Input
Pipe logs, config files, or other data into chat for analysis:
```bash theme={null}
# Analyze logs
kubectl logs my-training-pod | chamber chat "what errors are in these logs?"
# Review a config
cat .chamber.yaml | chamber chat "is this config optimized for distributed training?"
```
When stdout is piped (e.g., `chamber chat "question" | pbcopy`), streaming is automatically disabled and the full response is buffered before output.
### Image Attachments
Attach an image for visual analysis:
```bash theme={null}
chamber chat -i screenshot.png "what does this GPU utilization graph tell me?"
```
Supported formats: PNG, JPG, GIF, WebP.
### List Conversations
View your recent chat history:
```bash theme={null}
chamber chat --list
```
## What You Can Ask
The AI assistant has access to your organization's live infrastructure data. Example questions:
* "What jobs are running right now?"
* "Why did my training job fail?"
* "Show me jobs submitted by my team this week"
* "What's the average queue wait time?"
* "How much GPU capacity is available?"
* "When does our H100 reservation expire?"
* "Which pools have spare capacity?"
* "Are we at risk of hitting our budget?"
* "What's our GPU utilization this week?"
* "Which teams are underutilizing their allocations?"
* "Show me utilization trends for the last 30 days"
* "Are there any idle GPUs I can use?"
* "How can I improve my training job's GPU efficiency?"
* "Should I use reserved or elastic for this workload?"
* "What's the best GPU type for my use case?"
* "Help me right-size my resource requests"
The assistant also understands Chamber CLI commands and can suggest the right command for what you're trying to do — and execute it for you with a single confirmation.
## Command Execution
When the AI assistant determines that a CLI command can fulfill your request, it will suggest the command and ask for confirmation before running it locally:
```bash theme={null}
$ chamber chat "run my training script in ./demo on 4 H100s"
Chambie AI Assistant
Thinking... ✓
I'll set that up for you.
┌ Suggested command:
│ chamber run ./demo --gpus 4 --gpu-type H100 --team abc123
│
│ Run training in ./demo on 4x H100 GPUs
└ Execute this command? [Y/n]:
```
Press **Enter** or type **y** to execute the command. Type **n** to skip it.
This works in all modes — quick questions, interactive conversations, and piped input. A few things to know:
* **Confirmation required** — The assistant always asks before executing. You stay in control.
* **Local execution** — Commands run on your machine using the same `chamber` binary you're already using.
* **Non-TTY mode** — When input is piped (non-interactive), suggested commands are displayed but not executed, so scripts remain safe.
* **Ctrl+C** — You can cancel a running command at any time.
Try asking the assistant to perform actions: "cancel workload abc123", "submit a job with 8 H100s", or "list my team's running jobs". It will translate your request into the right CLI command.
## Command Reference
```bash theme={null}
chamber chat [message] [flags]
```
| Flag | Description |
| -------------------- | ----------------------------------------------------------- |
| `-c, --conversation` | Named conversation thread (resume later with the same name) |
| `-m, --model` | AI model selection |
| `-i, --image` | Attach an image file (PNG, JPG, GIF, WebP) |
| `--no-stream` | Disable streaming (buffer full response before output) |
| `--raw` | Disable markdown rendering in terminal |
| `-l, --list` | List recent conversations |
## Output
When running in a terminal, responses are rendered with formatted markdown including syntax highlighting, bold/italic text, and clickable links. Use `--raw` to disable formatting if needed.
When piped to another command, streaming is disabled automatically and raw text is output:
```bash theme={null}
# Copy response to clipboard
chamber chat "list my running jobs" | pbcopy
# Save to file
chamber chat "summarize this week's utilization" > report.txt
```
# Commands Reference
Source: https://docs.usechamber.io/cli/commands
Complete reference for all Chamber CLI commands
# Commands Reference
This page documents all available Chamber CLI commands and their options.
## Global Flags
These flags are available on all commands:
| Flag | Description |
| -------------- | ------------------------------------------------ |
| `-o, --output` | Output format: `table` (default), `json`, `yaml` |
| `-h, --help` | Show help for any command |
## AI Chat
### `chamber chat`
Ask questions about your GPU infrastructure using natural language. The assistant can also suggest and execute CLI commands on your behalf with confirmation. See [AI Assistant](/cli/chat) for the complete guide.
```bash theme={null}
# Quick question
chamber chat "what jobs are running right now?"
# Interactive conversation
chamber chat
# Named conversation (resume later)
chamber chat -c my-thread "help me debug this failure"
# Pipe input for analysis
kubectl logs my-pod | chamber chat "what errors are in these logs?"
# Attach an image
chamber chat -i gpu-metrics.png "what does this graph show?"
# List recent conversations
chamber chat --list
```
| Flag | Description |
| -------------------- | ---------------------------------------------------------------------------- |
| `-c, --conversation` | Named conversation thread (alphanumeric, hyphens, underscores, max 64 chars) |
| `-m, --model` | AI model selection |
| `-i, --image` | Attach image file (PNG, JPG, GIF, WebP) — single-shot and piped modes only |
| `--no-stream` | Buffer full response (auto-enabled when stdout is piped) |
| `--raw` | Disable markdown rendering |
| `-l, --list` | List recent conversations |
Chat requires browser-based login (`chamber login`). API tokens (`--token`) are not supported for chat.
***
## Authentication
### `chamber login`
Authenticate with Chamber.
```bash theme={null}
# Browser-based login (interactive)
chamber login
# Token-based login (headless/scripted)
chamber login --token ch.your-api-token
```
| Flag | Description |
| ------------- | ---------------------------- |
| `-t, --token` | API token for authentication |
### `chamber logout`
Remove stored credentials.
```bash theme={null}
chamber logout
```
### `chamber whoami`
Display current user information.
```bash theme={null}
chamber whoami
```
Example output:
```
Email: user@example.com
User ID: 550e8400-e29b-41d4-a716-446655440000
Token expires: 2024-01-15T10:30:00Z
Organization: my-org-id
```
***
## Run (Auto-Containerize)
The `run` command automatically containerizes and submits your training code. See [Auto-Containerize with Run](/cli/run) for complete documentation.
### `chamber run`
Auto-containerize and submit a GPU workload.
```bash theme={null}
# Preview what will be generated
chamber run ./my-project --gpus 4 --gpu-type H100 --team --dry-run
# Build, push, and submit
chamber run ./my-project --gpus 4 --gpu-type H100 --team
# With custom entrypoint
chamber run ./my-project \
--gpus 8 \
--gpu-type H100 \
--team \
--entrypoint train.py \
--entrypoint-args "--config config.yaml"
# Use reserved capacity
chamber run ./my-project --gpus 4 --gpu-type H100 --team --class RESERVED
```
| Flag | Description |
| ------------------- | --------------------------------------- |
| `--team, -t` | Team ID (required) |
| `--gpus, -g` | Number of GPUs |
| `--gpu-type` | GPU type: `H100`, `A100`, `L40S`, etc. |
| `--class, -c` | Workload class: `RESERVED` or `ELASTIC` |
| `--name, -n` | Workload name |
| `--entrypoint` | Override detected Python entrypoint |
| `--entrypoint-args` | Arguments for the entrypoint |
| `--registry` | Container registry URL |
| `--dockerfile` | Use existing Dockerfile |
| `--dry-run` | Preview without building/submitting |
| `--save-dockerfile` | Save generated Dockerfile |
| `--save-manifest` | Save generated K8s manifest |
| `--no-cache` | Force Docker rebuild |
Use `--dry-run` to preview the generated Dockerfile and manifest before submitting.
**Prerequisites:** Docker, AWS CLI (for ECR). See [Auto-Containerize with Run](/cli/run) for setup instructions.
***
## Workloads
Manage GPU workloads. Aliases: `wl`, `jobs`
### `chamber workloads list`
List workloads with optional filters.
```bash theme={null}
# List all workloads
chamber workloads list
# Filter by status
chamber workloads list --status RUNNING
# Filter by initiative
chamber workloads list --initiative abc123
# Limit results
chamber workloads list --limit 50
# Output as JSON
chamber workloads list -o json
```
| Flag | Description |
| ------------------ | ------------------------------------------------------------------------------------------------- |
| `-s, --status` | Filter by status: `PENDING`, `QUEUED`, `RUNNING`, `COMPLETED`, `FAILED`, `CANCELLED`, `PREEMPTED` |
| `-i, --initiative` | Filter by team ID |
| `--submitted-by` | Filter by submitter user ID |
| `-l, --limit` | Maximum results (1-100, default: 20) |
Example output:
```
ID Name Status GPUs Type Class Submitted
-------- ---------------------- -------- ---- ---- -------- ----------------
a1b2c3d4 gpt-finetune-v2 RUNNING 8 H100 RESERVED 2024-01-15T09:30
e5f6g7h8 embedding-training QUEUED 4 A100 ELASTIC 2024-01-15T09:15
```
### `chamber workloads get`
Get detailed information about a specific workload.
```bash theme={null}
chamber workloads get
# Output as JSON
chamber workloads get -o json
```
Example output:
```
ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890
Name: gpt-finetune-v2
Status: RUNNING
Workload Class: RESERVED
Priority: 75
GPUs: 8 x H100
Initiative: ml-training-team
Submitted By: user@example.com
Submitted At: 2024-01-15T09:30:00Z
Started At: 2024-01-15T09:32:15Z
```
### `chamber workloads stats`
Get aggregated workload statistics.
```bash theme={null}
# Default: last 7 days
chamber workloads stats
# Custom time range
chamber workloads stats --time-range last_30_days
# Group by submitter
chamber workloads stats --group-by submitted_by
```
| Flag | Description |
| -------------- | --------------------------------------------------------------------------------------- |
| `--time-range` | Time range: `today`, `this_week`, `this_month`, `last_7_days` (default), `last_30_days` |
| `--group-by` | Group results by: `submitted_by`, `initiative_id`, `instance_type` |
### `chamber workloads metrics`
Get GPU metrics for a running workload.
```bash theme={null}
chamber workloads metrics
```
Example output:
```
GPU Metrics:
Metric Current Avg Min Max
--------------- ------- ---- ---- ----
Utilization (%) 92.5 88.3 45.0 98.0
Memory (%) 78.2 75.1 60.0 82.0
Temperature (C) 72 68 55 76
Power (W) 420 385 250 450
```
### `chamber workloads cancel`
Cancel a running or queued workload.
```bash theme={null}
chamber workloads cancel
```
***
## Submit Workloads
### `chamber submit`
Submit a new GPU workload.
```bash theme={null}
# Interactive mode
chamber submit -i
# With flags
chamber submit \
--name "my-training-job" \
--gpus 8 \
--gpu-type H100 \
--class RESERVED \
--initiative abc123
# With Kubernetes manifest
chamber submit \
--name "custom-job" \
--gpus 4 \
--gpu-type A100 \
--class ELASTIC \
--initiative abc123 \
--manifest job.yaml
# Using a template
chamber submit \
--name "my-job" \
--initiative abc123 \
--template template-id
```
| Flag | Description |
| ------------------- | --------------------------------------------------------------- |
| `-n, --name` | Workload name (required) |
| `--initiative` | Team ID (required) |
| `-g, --gpus` | Number of GPUs |
| `--gpu-type` | GPU type: `H100`, `A100`, etc. (required unless using template) |
| `-c, --class` | Workload class: `RESERVED` or `ELASTIC` (default: `ELASTIC`) |
| `-p, --priority` | Priority: `critical`, `high`, `normal`, `low`, `background` |
| `--external-id` | Your own workload identifier |
| `-m, --manifest` | Path to Kubernetes manifest YAML |
| `--template` | Use a template instead of specifying flags |
| `--scaling-mode` | `gang` or `elastic` (default: `gang`) |
| `--pods` | Number of pods for distributed training |
| `--gpus-per-pod` | GPUs per pod (manual distribution) |
| `--min-pods` | Minimum pods for elastic scaling |
| `-i, --interactive` | Interactive mode with prompts |
Use `--class RESERVED` for guaranteed capacity that won't be preempted. Use `--class ELASTIC` for lower-priority workloads that can run on spare capacity.
***
## Capacity
### `chamber capacity`
View capacity pools and budget information.
```bash theme={null}
chamber capacity
# Output as JSON
chamber capacity -o json
```
Example output:
```
Budget:
Allocated: 10000 GPU-hours
Used: 4500 GPU-hours
Available: 5500 GPU-hours
Capacity Pools:
ID Name GPU Type Total Used Available
-------- ---------------- -------- ----- ---- ---------
pool-001 us-west-h100 H100 64 48 16
pool-002 us-east-a100 A100 128 96 32
```
***
## Teams
Manage teams. Aliases: `team`, `initiatives`
### `chamber teams list`
List all teams you have access to.
```bash theme={null}
# List all teams
chamber teams list
# Output as JSON
chamber teams list -o json
```
Example output:
```
ID Name Description Parent
-------- ------------------- --------------------------- --------
team-001 ML Research Main ML research team -
team-002 NLP Projects Natural language processing team-001
team-003 Computer Vision CV and image processing team-001
```
### `chamber teams get`
Get detailed information about a specific team.
```bash theme={null}
chamber teams get
```
### `chamber teams create`
Create a new team or sub-team.
```bash theme={null}
# Create a root team (requires ORG_ADMIN)
chamber teams create --name "New Team" --description "Team description"
# Create a sub-team under an existing team
chamber teams create --name "Sub Team" --parent parent-team-id
```
| Flag | Description |
| --------------- | --------------------------------------- |
| `--name` | Team name (required) |
| `--description` | Team description |
| `--parent` | Parent team ID (for creating sub-teams) |
Creating root-level teams requires ORG\_ADMIN permissions. Sub-teams can be created by team members.
***
## Allocations
Manage capacity allocations for teams. Aliases: `alloc`
### `chamber allocations list`
List allocations for a team.
```bash theme={null}
chamber allocations list --initiative
```
| Flag | Description |
| -------------- | ------------------ |
| `--initiative` | Team ID (required) |
Example output:
```
ID Reservation Instances Status Expires
---------- ------------- --------- ------- ----------------
alloc-001 res-h100-west 4 ACTIVE 2024-06-30T23:59
alloc-002 res-a100-east 8 ACTIVE 2024-12-31T23:59
```
### `chamber allocations get`
Get detailed information about a specific allocation.
```bash theme={null}
chamber allocations get
```
### `chamber allocations create`
Create a new allocation for a team. Requires ORG\_ADMIN permissions.
```bash theme={null}
chamber allocations create \
--initiative \
--reservation \
--instances 4
```
| Flag | Description |
| --------------- | ------------------------------------------ |
| `--initiative` | Team ID (required) |
| `--reservation` | Reservation ID (required) |
| `--instances` | Number of instances to allocate (required) |
***
## Templates
Manage reusable workload templates. Aliases: `template`, `tpl`
### `chamber templates list`
List available templates.
```bash theme={null}
# List all templates
chamber templates list
# Filter by name
chamber templates list --name "training"
# Filter by scope
chamber templates list --scope ORGANIZATION
# Include system templates
chamber templates list --include-system
```
| Flag | Description |
| ------------------ | ------------------------------------------------------ |
| `--name` | Filter by template name |
| `--scope` | Filter by scope: `PERSONAL`, `PROJECT`, `ORGANIZATION` |
| `--include-system` | Include system-provided templates |
Example output:
```
ID Name Scope GPU Type GPUs
----------- -------------------- ------------ -------- ----
tpl-001 standard-training ORGANIZATION H100 8
tpl-002 inference-job ORGANIZATION A100 4
tpl-003 my-custom-template PERSONAL H100 16
```
### `chamber templates get`
Get detailed information about a specific template.
```bash theme={null}
chamber templates get
```
***
## Artifacts (Container Images)
Manage container images and artifact repositories. Aliases: `artifact`, `images`
Chamber supports multiple registry types:
* **Doogle** - Doogle Artifact Repository (default, managed by Chamber)
* **ECR** - AWS Elastic Container Registry
* **GCR** - Google Container Registry
* **ACR** - Azure Container Registry
### `chamber artifacts repos list`
List artifact repositories in your organization.
```bash theme={null}
# List all repositories
chamber artifacts repos list
# Filter by registry type
chamber artifacts repos list --type doogle
# Output as JSON
chamber artifacts repos list -o json
```
| Flag | Description |
| ------------- | ------------------------------------------------------ |
| `--type` | Filter by registry type: `doogle`, `ecr`, `gcr`, `acr` |
| `-l, --limit` | Maximum results (default: 20) |
Example output:
```
ID Name Type URI Images Created
-------- ---------------- ------ ------------------------------------- ------ ----------------
repo-001 ml-training doogle doogle.usechamber.io/org123/ml-train 12 2024-01-15T09:30
repo-002 inference-models ecr 123456.dkr.ecr.us-east-1.amazonaws.com 5 2024-01-10T14:20
```
### `chamber artifacts repos get`
Get detailed information about a specific repository.
```bash theme={null}
chamber artifacts repos get
chamber artifacts repos get -o json
```
### `chamber artifacts repos create`
Create a new artifact repository.
```bash theme={null}
# Create a Doogle repository (recommended)
chamber artifacts repos create --name my-models
# Create with description
chamber artifacts repos create --name training-images --description "Training container images"
# Create ECR repository
chamber artifacts repos create --name my-ecr-repo --type ecr
```
| Flag | Description |
| ------------------- | ------------------------------------------------------ |
| `-n, --name` | Repository name (required) |
| `-d, --description` | Repository description |
| `--type` | Registry type: `doogle` (default), `ecr`, `gcr`, `acr` |
Use Doogle Artifact Repository for a fully managed experience with no additional cloud setup required.
### `chamber artifacts repos delete`
Delete an artifact repository.
```bash theme={null}
chamber artifacts repos delete
```
This will delete all images in the repository. This action cannot be undone.
### `chamber artifacts list`
List container images in a repository.
```bash theme={null}
# List images in a repository
chamber artifacts list --repo
# Filter by tag
chamber artifacts list --repo --tag latest
# Output as JSON
chamber artifacts list --repo -o json
```
| Flag | Description |
| ------------- | ----------------------------- |
| `-r, --repo` | Repository ID (required) |
| `-t, --tag` | Filter by image tag |
| `-l, --limit` | Maximum results (default: 20) |
Example output:
```
Digest Tags Size Pushed At
-------------------- ------------ -------- ----------------
sha256:a1b2c3d4... latest, v1.2 1.2 GB 2024-01-15T10:30
sha256:e5f6g7h8... v1.1 1.1 GB 2024-01-14T09:15
sha256:i9j0k1l2... 980 MB 2024-01-13T14:45
```
### `chamber artifacts get`
Get detailed information about a specific artifact (image).
```bash theme={null}
chamber artifacts get
chamber artifacts get -o json
```
### `chamber artifacts delete`
Delete an artifact (image) from a repository.
```bash theme={null}
chamber artifacts delete
```
### `chamber artifacts login`
Get Docker login credentials for pushing/pulling images.
```bash theme={null}
# Get credentials
chamber artifacts login
# Get credentials for a specific repository
chamber artifacts login
# Output docker login command (can be piped to sh)
chamber artifacts login --docker
# Login directly with Docker
chamber artifacts login --docker | sh
```
| Flag | Description |
| ---------- | -------------------------------- |
| `--docker` | Output full docker login command |
Example output:
```
Registry: doogle.usechamber.io
Username: chamber
Password: eyJhbGc...
Expires At: 2024-01-15T16:30:00Z
To login with Docker:
docker login -u chamber -p doogle.usechamber.io
```
***
## Metrics
### `chamber metrics`
View organization-wide GPU metrics.
```bash theme={null}
# Default: last 24 hours
chamber metrics
# Filter by team
chamber metrics --initiative
# Custom time range
chamber metrics --time-range last_7d
# Group by team
chamber metrics --group-by team
# Output as JSON
chamber metrics -o json
```
| Flag | Description |
| -------------- | ----------------------------------------------------------------------------- |
| `--initiative` | Filter by team ID |
| `--time-range` | Time range: `last_1h`, `last_6h`, `last_24h` (default), `last_7d`, `last_30d` |
| `--group-by` | Group metrics by: `team` |
***
## Configuration
### `chamber config show`
Display current configuration.
```bash theme={null}
chamber config show
```
### `chamber config get`
Get a specific configuration value.
```bash theme={null}
chamber config get organization_id
chamber config get api_url
```
### `chamber config set`
Set a configuration value.
```bash theme={null}
chamber config set organization_id your-org-id
chamber config set api_url https://api.usechamber.io/v1
```
| Key | Description |
| ----------------- | ------------------------------------- |
| `organization_id` | Default organization for API requests |
| `api_url` | Chamber API URL |
***
## System
### `chamber health`
Check API health status.
```bash theme={null}
chamber health
```
### `chamber completion`
Generate shell completion scripts.
```bash theme={null}
chamber completion bash
chamber completion zsh
chamber completion fish
```
***
## JSON Output
All commands support `-o json` for machine-readable output:
```bash theme={null}
# Get workloads as JSON
chamber workloads list -o json
# Pipe to jq for filtering
chamber workloads list -o json | jq '.[] | select(.status == "RUNNING")'
# Use in scripts
WORKLOAD_ID=$(chamber workloads list -o json | jq -r '.[0].id')
```
**Early Access** — The Chamber CLI is currently in early access. Contact [support@usechamber.io](mailto:support@usechamber.io) to get started.
# Configuration
Source: https://docs.usechamber.io/cli/configuration
Configure the Chamber CLI settings and credentials
# Configuration
The Chamber CLI stores configuration and credentials in `~/.chamber/`.
## Configuration Files
| File | Purpose |
| ------------------------- | --------------------------------------------------------- |
| `~/.chamber/config.json` | CLI settings (API URL, organization ID, default registry) |
| `~/.chamber/token.json` | Authentication credentials |
| `~/.chamber/chat_history` | AI chat readline history (up to 500 entries) |
The `token.json` file contains sensitive credentials. It's created with `0600` permissions (owner read/write only). Do not share or commit this file.
## Settings
### Organization ID
Set your default organization for API requests:
```bash theme={null}
chamber config set organization_id your-org-id
```
Most commands require an organization context. Setting this avoids having to specify it on every command.
Find your organization ID in the [Chamber Dashboard](https://app.usechamber.io) under **Settings > Organization**.
### API URL
By default, the CLI connects to `https://api.usechamber.io/v1`. Override this for development or private deployments:
```bash theme={null}
chamber config set api_url https://your-api-endpoint.com/v1
```
### Default Registry
Set your container registry for the `chamber run` command to avoid specifying it on every run:
```bash theme={null}
chamber config set default_registry 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo
```
When using ECR, `chamber run` automatically authenticates using your AWS CLI credentials and creates the repository if it doesn't exist.
## View Configuration
See all current settings:
```bash theme={null}
chamber config show
```
Output:
```
Current configuration:
API URL: https://api.usechamber.io/v1
Organization ID: my-org-id
Default Registry: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo
Config directory: /Users/you/.chamber
```
Get a specific value:
```bash theme={null}
chamber config get organization_id
```
## Environment Variables
You can override configuration using environment variables:
| Variable | Description |
| ----------------- | ---------------------------------- |
| `CHAMBER_API_URL` | Override API URL |
| `CHAMBER_ORG_ID` | Override organization ID |
| `CHAMBER_TOKEN` | Override authentication token |
| `CHAMBER_DEBUG` | Set to `1` for verbose API logging |
Example:
```bash theme={null}
CHAMBER_ORG_ID=other-org chamber workloads list
```
## Multiple Profiles
For working with multiple organizations or environments, you can use shell aliases:
```bash theme={null}
# In ~/.bashrc or ~/.zshrc
alias chamber-prod="CHAMBER_ORG_ID=prod-org chamber"
alias chamber-dev="CHAMBER_ORG_ID=dev-org CHAMBER_API_URL=https://dev-api.example.com/v1 chamber"
```
Usage:
```bash theme={null}
chamber-prod workloads list
chamber-dev workloads list
```
## Reset Configuration
To reset all configuration and credentials:
```bash theme={null}
rm -rf ~/.chamber
```
Then re-authenticate:
```bash theme={null}
chamber login --token ch.your-api-token
chamber config set organization_id your-org-id
```
## Troubleshooting
### "Not logged in" error
```
Error: not logged in - run 'chamber login' first
```
Your token may have expired or been deleted. Re-authenticate:
```bash theme={null}
chamber login --token ch.your-api-token
```
### "No organization configured" error
```
Error: no organization configured - run 'chamber config set organization_id '
```
Set your organization:
```bash theme={null}
chamber config set organization_id your-org-id
```
### Check your setup
Verify everything is configured correctly:
```bash theme={null}
# Check health
chamber health
# Check authentication
chamber whoami
# Check configuration
chamber config show
```
**Early Access** — The Chamber CLI is currently in early access. Contact [support@usechamber.io](mailto:support@usechamber.io) to get started.
# Installation
Source: https://docs.usechamber.io/cli/installation
Install Chamber CLI in seconds and submit your first GPU workload
# Installation
The Chamber CLI is currently in early access. Contact [support@usechamber.io](mailto:support@usechamber.io) to get access.
## First-Time Setup
**Chamber guides you through everything.** Just run `chamber login` and follow the prompts. No manual configuration required.
### Step 1: Login
```bash theme={null}
chamber login
```
This opens your browser to authenticate. After logging in, you're automatically configured.
For CI/CD or headless servers, use token-based auth:
```bash theme={null}
chamber login --token ch.your-api-token
```
Generate tokens in the [Chamber Dashboard](https://app.usechamber.io) under **Settings > API Tokens**.
The [AI Assistant](/cli/chat) (`chamber chat`) requires browser-based login and does not support API tokens.
### Step 2: Verify
```bash theme={null}
chamber whoami
```
That's it! You're ready to submit workloads.
## Submit Your First Workload
Navigate to your training code and run:
```bash theme={null}
chamber run ./my-training-project --gpus 4 --team --dry-run
```
**First time?** The `--dry-run` flag shows you exactly what Chamber will generate without actually building or submitting. Remove it when you're ready to run for real.
### What happens next
If it's your first time, Chamber will interactively guide you through:
1. **Registry setup** — Choose Google Artifact Registry, AWS ECR, or another registry
2. **Docker installation** — If Docker isn't installed, Chamber offers to help
3. **Cloud CLI setup** — If AWS CLI or gcloud isn't configured, Chamber walks you through it
```
$ chamber run ./my-project --gpus 4 --team abc123
No container registry configured
Chamber needs a container registry to store your Docker images.
Select your registry type:
[1] Google Artifact Registry (recommended for GCP users)
[2] AWS ECR (recommended for AWS users)
[3] Other Docker registry
Select an option [1]:
```
After the one-time setup, future runs are instant:
```bash theme={null}
# This just works now
chamber run ./my-project --gpus 4 --team abc123
```
## Optional: Set Your Organization
If you belong to multiple organizations, set a default:
```bash theme={null}
chamber config set organization_id your-org-id
```
Find your organization ID in the [Chamber Dashboard](https://app.usechamber.io) under **Settings > Organization**.
## Shell Completion
Enable tab completion for faster workflows:
```bash theme={null}
# Add to ~/.bashrc
source <(chamber completion bash)
```
```bash theme={null}
# Add to ~/.zshrc
source <(chamber completion zsh)
```
```bash theme={null}
chamber completion fish | source
```
## Next Steps
Ask about workloads, capacity, and more
Submit training jobs with `chamber run`
Explore all available commands
# CLI Overview
Source: https://docs.usechamber.io/cli/introduction
Submit GPU workloads in seconds — no Docker or Kubernetes expertise required
# Chamber CLI
**Go from Python code to running GPU workload in one command.** The Chamber CLI eliminates the complexity of containerization, registry management, and Kubernetes — so ML engineers and data scientists can focus on what matters: training models.
**No Docker expertise required.** Chamber auto-detects your project, generates optimized Dockerfiles, handles registry authentication, and submits your workload — all interactively guided.
## AI Assistant — Ask Your Infrastructure Anything
Query your GPU infrastructure in natural language, directly from the terminal:
```bash theme={null}
chamber chat "how many GPUs are running right now?"
```
Start an interactive conversation for back-and-forth troubleshooting, capacity planning, or workload analysis:
```bash theme={null}
chamber chat
```
Interactive conversations, command execution, piped input, image analysis, and more
Deep dive into `chamber run`
## The Fastest Path to GPU Training
```bash theme={null}
chamber run ./my-training-project --gpus 4 --team my-team
```
That's it. One command. Chamber handles everything else:
Automatically identifies PyTorch, TensorFlow, or JAX. Finds your entrypoint and requirements.
Creates a GPU-optimized container with CUDA, cuDNN, and your dependencies.
Missing Docker? No registry configured? Chamber walks you through each step interactively.
Handles authentication, builds your image, pushes to your registry, and submits to Chamber.
Install and run your first workload
## Interactive Setup — No Prior Configuration Needed
Chamber CLI guides you through everything. Don't have Docker? It'll help you install it. No registry configured? It'll walk you through setting one up:
```
$ chamber run ./my-training --gpus 4 --team abc123
No container registry configured
Chamber needs a container registry to store your Docker images.
You can use Google Artifact Registry, AWS ECR, or any Docker-compatible registry.
Select your registry type:
[1] Google Artifact Registry (recommended for GCP users)
Example: us-central1-docker.pkg.dev/my-project/ml-images
[2] AWS ECR (recommended for AWS users)
Example: 123456789012.dkr.ecr.us-east-1.amazonaws.com/ml-images
[3] Other Docker registry
Select an option [1]: 1
Google Artifact Registry Setup
Enter your GAR registry URL: us-central1-docker.pkg.dev/my-project/ml-images
Save as default registry? (y/n) [y]: y
Default registry saved to ~/.chamber/config.json
[detect] Analyzing project structure...
[detect] Framework: PyTorch
[detect] Entrypoint: train.py
[setup] Checking prerequisites...
```
**First-time users:** Just run `chamber run` on your project. The CLI will guide you through every step — from installing Docker to configuring your registry to submitting your workload.
## Automatic Prerequisite Detection
Missing a tool? Chamber detects it and offers to help:
```
$ chamber run ./my-project --gpus 4 --team abc123
⚠ Docker is not installed
Docker is required to build and push container images.
Installation options:
[1] Quick install (recommended)
brew install --cask docker
[2] Open installation guide in browser
[3] Show manual installation instructions
[4] Skip (continue anyway)
Select an option [1]:
```
The same guided experience applies to:
* **Docker** — Required for building images
* **AWS CLI** — For ECR authentication
* **gcloud CLI** — For Google Artifact Registry authentication
* **Azure CLI** — For ACR authentication
## Why ML Engineers Love Chamber CLI
Run `chamber run` and follow the prompts. No YAML files, no Docker knowledge needed.
Auto-detects frameworks, entrypoints, and optimal GPU configurations.
Configure once, run forever. Settings are saved for future use.
Use `--dry-run` to see exactly what will be generated before building.
Single binary with no dependencies. SSH-friendly for remote workstations.
JSON output mode for CI/CD pipelines and automation.
## Quick Command Reference
| What you want to do | Command |
| ------------------------------ | ----------------------------------------------- |
| **Ask a question** | `chamber chat "what GPUs are available?"` |
| **Start a conversation** | `chamber chat` |
| **Submit a training workload** | `chamber run ./my-project --gpus 4 --team ` |
| Preview before submitting | `chamber run ./my-project --gpus 4 --dry-run` |
| List your workloads | `chamber workloads list` |
| Check GPU capacity | `chamber capacity` |
| View workload status | `chamber workloads get ` |
| Cancel a workload | `chamber workloads cancel ` |
## System Requirements
* **macOS** (Intel or Apple Silicon) or **Linux** (x86\_64 or ARM64)
* A Chamber account
**For auto-containerization (`chamber run`):**
* Docker — *Chamber will help you install it*
* Cloud CLI (gcloud/aws/az) — *Chamber will help you install it*
Install Chamber CLI and run your first workload
# Auto-Containerize & Run
Source: https://docs.usechamber.io/cli/run
Submit GPU workloads with one command — no Docker or Kubernetes expertise needed
**Coming Soon — Fully Managed GPU Infrastructure**
We're building automated infrastructure management across cloud providers so your
research and MLE teams can run GPU workloads directly in their own cloud accounts
without ever thinking about infrastructure. If you'd like early access or want to
learn more, reach out at [support@usechamber.io](mailto:support@usechamber.io).
# Auto-Containerize with `chamber run`
**One command. That's all it takes.** Point `chamber run` at your training code and it handles everything: containerization, registry authentication, image building, and workload submission.
```bash theme={null}
chamber run ./my-training-project --gpus 4 --team my-team
```
**No Docker knowledge required.** Chamber auto-detects your project structure, generates optimized Dockerfiles, and guides you through any missing prerequisites.
## What Chamber Does For You
Scans for PyTorch, TensorFlow, or JAX. Finds `train.py`, `main.py`, or your entrypoint. Reads `requirements.txt` or `pyproject.toml`.
Creates a GPU-optimized container with the right CUDA version, cuDNN, and your dependencies pre-installed.
Auto-authenticates with Google Artifact Registry or AWS ECR using your existing cloud credentials.
Uses `docker buildx build --push` to build and push in a single efficient step. Automatically pulls the `:latest` tag to seed the layer cache so teammates get fast rebuilds. Uses content-addressed image tags — if the image already exists, build and push are skipped entirely.
Creates and submits a Kubernetes job to Chamber with your GPU and resource requirements.
## Quick Start
```bash theme={null}
# Preview what will be generated (recommended first step)
chamber run ./my-training-project --gpus 4 --team --dry-run
# Build, push, and submit the workload
chamber run ./my-training-project --gpus 4 --team
```
**Always use `--dry-run` first** to preview the generated Dockerfile and Kubernetes manifest before building.
## Interactive Setup (First-Time Users)
Chamber guides you through everything. Missing something? Chamber will help:
### No Registry Configured?
```
$ chamber run ./my-project --gpus 4 --team abc123
No container registry configured
Chamber needs a container registry to store your Docker images.
You can use Google Artifact Registry, AWS ECR, or any Docker-compatible registry.
Select your registry type:
[1] Google Artifact Registry (recommended for GCP users)
Example: us-central1-docker.pkg.dev/my-project/ml-images
[2] AWS ECR (recommended for AWS users)
Example: 123456789012.dkr.ecr.us-east-1.amazonaws.com/ml-images
[3] Other Docker registry
Select an option [1]: 1
Google Artifact Registry Setup
Enter your GAR registry URL: us-central1-docker.pkg.dev/my-project/ml-images
Save as default registry? (y/n) [y]: y
Default registry saved to ~/.chamber/config.json
```
### Docker Not Installed?
```
⚠ Docker is not installed
Docker is required to build and push container images.
Installation options:
[1] Quick install (recommended)
brew install --cask docker
[2] Open installation guide in browser
[3] Show manual installation instructions
[4] Skip (continue anyway)
Select an option [1]:
```
### AWS CLI Not Configured?
```
⚠ AWS CLI is not authenticated
You need AWS credentials to push images to ECR.
Options:
[1] Run 'aws configure' interactively (recommended)
[2] Set environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
[3] Show me how to get credentials
[4] Skip (I'll handle authentication myself)
Select an option [1]:
```
**One-time setup.** After your first successful run, these settings are saved. Future runs work instantly without prompts.
## Supported Container Registries
Chamber automatically handles authentication for major cloud registries:
| Registry | URL Pattern | Auto-Auth |
| ---------------------------- | ------------------------------------------ | ------------------ |
| **Google Artifact Registry** | `{region}-docker.pkg.dev/{project}/{repo}` | gcloud CLI |
| **AWS ECR** | `{account}.dkr.ecr.{region}.amazonaws.com` | AWS CLI |
| **Other registries** | Any Docker-compatible registry | Via `docker login` |
## Project Detection
Chamber automatically detects your project configuration:
| What | How |
| ------------------------ | -------------------------------------------------------- |
| **Framework** | Scans `requirements.txt` for PyTorch, TensorFlow, or JAX |
| **Entrypoint** | Looks for `train.py`, `main.py`, `run.py`, or `app.py` |
| **Python Version** | Checks `.python-version` or `pyproject.toml` |
| **Distributed Training** | Detects Accelerate, DeepSpeed, Ray, or Horovod |
| **Requirements** | Uses `requirements.txt`, `pyproject.toml`, or `setup.py` |
## Project Configuration
Create a `.chamber.yaml` file for persistent settings (optional):
```yaml theme={null}
# .chamber.yaml
name: "llm-finetune"
entrypoint: "train.py"
entrypoint_args: "--config config.yaml --epochs 100"
# Resources
gpu_type: "H100"
gpus: 4
job_class: "ELASTIC"
# Environment variables injected into the container
env:
WANDB_PROJECT: "my-project"
NCCL_DEBUG: "INFO"
# Forward these local env vars to container (if set)
forward_env:
- WANDB_API_KEY
- HF_TOKEN
# Files to exclude from Docker build context
ignore:
- "output/"
- "checkpoints/"
- "*.bin"
- "*.safetensors"
```
CLI flags always override `.chamber.yaml` values, so you can set defaults while still customizing per-run.
## Command Reference
```bash theme={null}
chamber run [flags]
```
### Required Flags
| Flag | Description |
| ------------ | ------------------------ |
| `--team, -t` | Team ID for the workload |
### Resource Flags
| Flag | Default | Description |
| ------------- | ------------------- | --------------------------------------- |
| `--gpus, -g` | From config or 1 | Number of GPUs |
| `--gpu-type` | From config or H100 | GPU type (H100, A100, L40S, etc.) |
| `--class, -c` | ELASTIC | Workload class: `RESERVED` or `ELASTIC` |
### Container Flags
| Flag | Description |
| -------------- | --------------------------------------------------------- |
| `--registry` | Container registry URL (or use `default_registry` config) |
| `--base-image` | Override the auto-detected base image |
| `--dockerfile` | Use an existing Dockerfile instead of generating one |
| `--no-cache` | Force Docker rebuild without cache |
### Entrypoint Flags
| Flag | Description |
| ------------------- | ------------------------------------------------------ |
| `--name, -n` | Workload name (default: from config or directory name) |
| `--entrypoint` | Override detected Python entrypoint |
| `--entrypoint-args` | Arguments to pass to the entrypoint |
| `--env` | Environment variable KEY=VALUE (repeatable) |
### Workflow Flags
| Flag | Description |
| ------------------- | ------------------------------------------------- |
| `--dry-run` | Preview generated artifacts without executing |
| `--save-dockerfile` | Write generated Dockerfile to project directory |
| `--save-manifest` | Write generated K8s manifest to project directory |
## Examples
### Basic Training Workload
```bash theme={null}
chamber run ./my-project --gpus 4 --gpu-type H100 --team abc123
```
### Custom Entrypoint with Arguments
```bash theme={null}
chamber run ./my-project \
--gpus 8 \
--gpu-type H100 \
--team abc123 \
--entrypoint train.py \
--entrypoint-args "--config config.yaml --lr 1e-4 --batch-size 32"
```
### With Environment Variables
```bash theme={null}
chamber run ./my-project \
--gpus 4 \
--gpu-type H100 \
--team abc123 \
--env WANDB_API_KEY=$WANDB_API_KEY \
--env HF_TOKEN=$HF_TOKEN
```
### Reserved (Non-Preemptible) Capacity
```bash theme={null}
chamber run ./my-project \
--gpus 4 \
--gpu-type H100 \
--team abc123 \
--class RESERVED
```
### Using Your Own Dockerfile
```bash theme={null}
chamber run ./my-project \
--gpus 4 \
--gpu-type H100 \
--team abc123 \
--dockerfile Dockerfile.custom
```
## Generated Artifacts
### Example Dockerfile
For a PyTorch project with Accelerate, Chamber generates:
```dockerfile theme={null}
# syntax=docker/dockerfile:1
# Auto-generated by Chamber CLI
FROM nvcr.io/nvidia/pytorch:24.04-py3
WORKDIR /workspace
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
COPY . .
CMD ["accelerate", "launch", "train.py", "--config", "config.yaml"]
```
Chamber uses [BuildKit](https://docs.docker.com/build/buildkit/) with pip cache mounts, so repeated builds reuse cached Python packages instead of re-downloading them.
### Example Kubernetes Manifest
```yaml theme={null}
apiVersion: batch/v1
kind: Job
metadata:
name: llm-finetune-a1b2c3d4
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: training
image: us-central1-docker.pkg.dev/my-project/ml-images:abc123
resources:
requests:
nvidia.com/gpu: "4"
limits:
nvidia.com/gpu: "4"
env:
- name: "WANDB_PROJECT"
value: "my-project"
```
## Troubleshooting
Chamber will prompt you with installation options:
```
⚠ Docker is not installed
Installation options:
[1] Quick install (recommended)
[2] Open installation guide in browser
[3] Show manual installation instructions
```
Select an option and follow the prompts.
If auto-authentication fails:
**For Google Artifact Registry:**
```bash theme={null}
gcloud auth login
gcloud auth configure-docker {region}-docker.pkg.dev
```
**For AWS ECR:**
```bash theme={null}
aws configure
# Or set environment variables:
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
```
Check your `requirements.txt` is complete. Use `--dry-run` to preview the Dockerfile:
```bash theme={null}
chamber run ./my-project --dry-run
```
If needed, use `--save-dockerfile` to inspect and modify:
```bash theme={null}
chamber run ./my-project --dry-run --save-dockerfile
# Edit Dockerfile.chamber
chamber run ./my-project --dockerfile Dockerfile.chamber
```
## Docker Build Optimizations
Chamber automatically applies several optimizations to make builds fast:
| Optimization | What it does |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Single build+push** | Uses `docker buildx build --push` to build and push in one step. This is significantly faster than separate build+push because buildkit pushes layers directly as they complete, avoiding Docker daemon/containerd manifest duplication issues |
| **BuildKit** | Enabled by default (`DOCKER_BUILDKIT=1`) for parallel build stages and advanced caching |
| **Pip cache mounts** | Uses `--mount=type=cache,target=/root/.cache/pip` so pip packages are cached across builds instead of re-downloaded |
| **Remote layer caching** | Pulls the `:latest` tag from your registry before building to seed the local layer cache. After a successful build, tags as `:latest` using `docker buildx imagetools create` (fast manifest aliasing, no layer re-upload) so future builds benefit |
| **Content-addressed tags** | Images are tagged with a content hash of your project. If the image already exists in the registry, build and push are skipped entirely |
| **Platform targeting** | Explicitly builds for `linux/amd64` to ensure consistent images regardless of your local architecture |
| **Reduced metadata** | Uses `--provenance=false --sbom=false` to skip unnecessary metadata generation that slows down builds |
| **Context size warnings** | Reports build context size and warns if it exceeds 500MB, helping you identify large files that should be in `.dockerignore` |
| **Streaming output** | Build output is streamed to your terminal in real-time so you can follow progress |
**Why single build+push matters:** With Docker Desktop's containerd image store, a separate `docker push` can push ALL manifests from multi-platform base images (e.g., 6 manifests for NVIDIA images), causing each layer to be checked 6 times. The `buildx build --push` approach only pushes what it built — making pushes dramatically faster, especially for large ML images.
Use `--no-cache` to force a full Docker rebuild (this does not force re-push if the image already exists):
```bash theme={null}
chamber run ./my-project --gpus 4 --team abc123 --no-cache
```
## Best Practices
Always preview generated artifacts before building to catch issues early.
Store project-specific settings to avoid repeating flags on every run.
Configure your registry once: `chamber config set default_registry `
Securely inject API keys without hardcoding them in your config.
# AWS EKS
Source: https://docs.usechamber.io/cluster-deployment/aws-eks
Deploy a GPU-ready EKS cluster with Chamber on AWS
The [`terraform-aws-chamber-eks`](https://github.com/ChamberOrg/terraform-aws-chamber-eks) module deploys a production-ready Amazon EKS cluster with GPU autoscaling, NVIDIA drivers, and the Chamber Agent — all in a single `terraform apply`.
## Prerequisites
Install from [developer.hashicorp.com/terraform/install](https://developer.hashicorp.com/terraform/install). Verify with `terraform version`.
The AWS provider uses your local credentials. Verify with `aws sts get-caller-identity`.
You need a cluster token and cluster ID from the [Chamber Console](https://app.usechamber.io). See [Getting a Cluster Token](/agent/installation#getting-a-cluster-token) for instructions.
## Quick Start
Create a new directory for your Terraform configuration and add a `main.tf` file:
```hcl theme={null}
provider "aws" {
region = var.aws_region
}
module "chamber_eks" {
source = "github.com/ChamberOrg/terraform-aws-chamber-eks"
cluster_name = "my-gpu-cluster"
aws_region = var.aws_region
chamber_cluster_token = var.chamber_cluster_token
chamber_cluster_id = var.chamber_cluster_id
}
variable "aws_region" {
type = string
}
variable "chamber_cluster_token" {
type = string
sensitive = true
}
variable "chamber_cluster_id" {
type = string
}
output "configure_kubectl" {
value = module.chamber_eks.configure_kubectl
}
```
```hcl theme={null}
aws_region = "us-west-2"
chamber_cluster_token = "your-token-here"
chamber_cluster_id = "your-cluster-id"
```
Do not commit `terraform.tfvars` to version control. Add it to `.gitignore`. For CI/CD pipelines, use environment variables: `TF_VAR_chamber_cluster_token`.
```bash theme={null}
terraform init
terraform plan
terraform apply
```
Deployment takes approximately **15-20 minutes**.
```bash theme={null}
$(terraform output -raw configure_kubectl)
```
```bash theme={null}
# Verify system nodes are ready
kubectl get nodes
# Verify Karpenter is running
kubectl get pods -n karpenter
# Verify Chamber Agent is connected
kubectl get pods -n chamber-system
```
Your cluster should appear in the [Chamber Console](https://app.usechamber.io) under **Capacity Pools**.
## Using an Existing VPC
To deploy into an existing VPC instead of creating a new one:
```hcl theme={null}
module "chamber_eks" {
source = "github.com/ChamberOrg/terraform-aws-chamber-eks"
cluster_name = "my-gpu-cluster"
aws_region = var.aws_region
chamber_cluster_token = var.chamber_cluster_token
chamber_cluster_id = var.chamber_cluster_id
create_vpc = false
vpc_id = "vpc-xxxxxxxx"
private_subnet_ids = ["subnet-aaa", "subnet-bbb", "subnet-ccc"]
}
```
Private subnets must have outbound internet access (via NAT Gateway) for nodes to pull container images and connect to the Chamber control plane.
## Key Variables
The table below covers the most commonly configured variables. For the complete list, see the [module README on GitHub](https://github.com/ChamberOrg/terraform-aws-chamber-eks#inputs).
### Required
| Variable | Description | Type |
| ----------------------- | ---------------------------------- | -------- |
| `cluster_name` | Name of the EKS cluster | `string` |
| `chamber_cluster_token` | Cluster token from Chamber Console | `string` |
| `chamber_cluster_id` | Cluster ID from Chamber Console | `string` |
### AWS
| Variable | Description | Default |
| ------------ | ------------------------------ | ------------- |
| `aws_region` | AWS region for the EKS cluster | `"us-west-2"` |
### VPC
| Variable | Description | Default |
| -------------------- | ---------------------------------------------------------------- | --------------- |
| `create_vpc` | Create a new VPC or use existing | `true` |
| `vpc_id` | Existing VPC ID (required when `create_vpc = false`) | `null` |
| `private_subnet_ids` | Existing private subnet IDs (required when `create_vpc = false`) | `[]` |
| `vpc_cidr` | CIDR block for new VPC | `"10.0.0.0/16"` |
| `single_nat_gateway` | Use a single NAT gateway (cost savings for dev/staging) | `false` |
### EKS
| Variable | Description | Default |
| ---------------------------- | ------------------------------------ | ---------------------------------------- |
| `cluster_version` | Kubernetes version | `"1.32"` |
| `system_node_instance_types` | Instance types for system node group | `["m5.large", "m5a.large", "m6i.large"]` |
### GPU
| Variable | Description | Default |
| ----------------------------- | ------------------------------------- | --------------------------- |
| `create_default_gpu_nodepool` | Create Terraform-managed GPU NodePool | `false` |
| `gpu_instance_families` | GPU instance families for NodePool | `["g5", "g6", "p4d", "p5"]` |
| `capacity_types` | Capacity types (`on-demand`, `spot`) | `["on-demand", "spot"]` |
| `gpu_limits` | Maximum GPUs for NodePool | `100` |
### Chamber
| Variable | Description | Default |
| ----------------------- | ----------------------------------- | ---------- |
| `chamber_agent_version` | Chamber Agent version | `"latest"` |
| `enable_kai_scheduler` | Enable KAI fractional GPU scheduler | `true` |
## Key Outputs
| Output | Description |
| ------------------------- | --------------------------------- |
| `cluster_name` | Name of the EKS cluster |
| `cluster_endpoint` | EKS API server endpoint |
| `vpc_id` | VPC ID |
| `configure_kubectl` | Command to configure kubectl |
| `verification_commands` | Commands to verify the deployment |
| `karpenter_node_role_arn` | Karpenter node IAM role ARN |
For all outputs, see the [module README on GitHub](https://github.com/ChamberOrg/terraform-aws-chamber-eks#outputs).
## GPU Pool Management
After deployment, you need GPU pools for Karpenter to know which GPU nodes to provision. There are two approaches:
Manage GPU pools through the [Chamber Console](https://app.usechamber.io):
1. Go to **Capacity Pools** > **Create Dynamic Pool**
2. Select your cluster and configure GPU type, limits, and capacity types
3. The pool syncs to your cluster automatically — Karpenter provisions GPU nodes on demand
This is the recommended approach for most teams. It allows per-GPU-type management with real-time limit adjustments.
To manage GPU pools entirely through Terraform, set `create_default_gpu_nodepool = true`:
```hcl theme={null}
module "chamber_eks" {
source = "github.com/ChamberOrg/terraform-aws-chamber-eks"
# ... required variables ...
create_default_gpu_nodepool = true
gpu_instance_families = ["g5", "g6"]
capacity_types = ["on-demand", "spot"]
gpu_limits = 50
}
```
This creates a single broad NodePool covering the specified instance families.
## Troubleshooting
Check the Chamber Agent logs:
```bash theme={null}
kubectl logs -n chamber-system -l app=chamber-agent --tail=100
```
Verify your `chamber_cluster_token` and `chamber_cluster_id` are correct.
1. Verify a GPU pool exists:
```bash theme={null}
kubectl get nodepool
```
If none exists, create one via the Chamber Console or set `create_default_gpu_nodepool = true`.
2. Check Karpenter logs:
```bash theme={null}
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=100
```
3. Check for capacity errors:
```bash theme={null}
kubectl get events --field-selector reason=FailedProvisioning
```
Your AWS account lacks GPU Spot quota. Either request a quota increase via the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas), or set `capacity_types = ["on-demand"]`.
This is expected when no GPU nodes exist yet. GPU Operator DaemonSets start automatically when Karpenter provisions GPU nodes in response to a workload.
## Cleanup
```bash theme={null}
terraform destroy
```
Ensure all GPU workloads are terminated before destroying to avoid orphaned resources.
## Next Steps
Submit your first GPU workload
Configure capacity pools and reservations
Detailed troubleshooting guide
Full source, examples, and changelog
# Google Cloud GKE
Source: https://docs.usechamber.io/cluster-deployment/google-gke
Deploy a GPU-ready GKE cluster with Chamber on Google Cloud
The [`terraform-google-chamber-gke`](https://github.com/ChamberOrg/terraform-google-chamber-gke) module deploys a production-ready Google GKE cluster with GPU autoscaling, NVIDIA drivers, and the Chamber Agent — all in a single `terraform apply`.
## Prerequisites
Install from [developer.hashicorp.com/terraform/install](https://developer.hashicorp.com/terraform/install). Verify with `terraform version`.
Install the [gcloud CLI](https://cloud.google.com/sdk/docs/install) and authenticate:
```bash theme={null}
gcloud auth login
gcloud auth application-default login
```
Enable the required APIs in your project:
```bash theme={null}
gcloud services enable \
container.googleapis.com \
compute.googleapis.com \
iam.googleapis.com \
--project=YOUR_PROJECT_ID
```
You need a cluster token and cluster ID from the [Chamber Console](https://app.usechamber.io). See [Getting a Cluster Token](/agent/installation#getting-a-cluster-token) for instructions.
## Quick Start
The GKE module requires explicit provider configuration because the `kubernetes`, `helm`, and `kubectl` providers need the cluster endpoint and credentials from the module outputs. The full configuration is shown below.
Create a new directory for your Terraform configuration and add a `main.tf` file:
```hcl theme={null}
provider "google" { project = var.gcp_project_id; region = var.gcp_region }
provider "google-beta" { project = var.gcp_project_id; region = var.gcp_region }
data "google_client_config" "default" {}
provider "kubernetes" {
host = "https://${module.chamber_gke.cluster_endpoint}"
token = data.google_client_config.default.access_token
cluster_ca_certificate = base64decode(module.chamber_gke.cluster_ca_certificate)
}
provider "helm" {
kubernetes {
host = "https://${module.chamber_gke.cluster_endpoint}"
token = data.google_client_config.default.access_token
cluster_ca_certificate = base64decode(module.chamber_gke.cluster_ca_certificate)
}
}
provider "kubectl" {
host = "https://${module.chamber_gke.cluster_endpoint}"
token = data.google_client_config.default.access_token
cluster_ca_certificate = base64decode(module.chamber_gke.cluster_ca_certificate)
load_config_file = false
}
module "chamber_gke" {
source = "github.com/ChamberOrg/terraform-google-chamber-gke"
gcp_project_id = var.gcp_project_id
gcp_region = var.gcp_region
cluster_name = "my-gpu-cluster"
chamber_cluster_token = var.chamber_cluster_token
chamber_cluster_id = var.chamber_cluster_id
}
variable "gcp_project_id" {
type = string
}
variable "gcp_region" {
type = string
}
variable "chamber_cluster_token" {
type = string
sensitive = true
}
variable "chamber_cluster_id" {
type = string
}
output "configure_kubectl" {
value = module.chamber_gke.configure_kubectl
}
```
```hcl theme={null}
gcp_project_id = "my-gcp-project"
gcp_region = "us-central1"
chamber_cluster_token = "your-token-here"
chamber_cluster_id = "your-cluster-id"
```
Do not commit `terraform.tfvars` to version control. Add it to `.gitignore`. For CI/CD pipelines, use environment variables: `TF_VAR_chamber_cluster_token`.
```bash theme={null}
terraform init
terraform plan
terraform apply
```
Deployment takes approximately **15-20 minutes**.
```bash theme={null}
$(terraform output -raw configure_kubectl)
```
```bash theme={null}
# Verify system nodes are ready
kubectl get nodes -l purpose=system
# Verify Karpenter is running
kubectl get pods -n karpenter
# Verify Chamber Agent is connected
kubectl get pods -n chamber-system
```
Your cluster should appear in the [Chamber Console](https://app.usechamber.io) under **Capacity Pools**.
## Using an Existing VPC
To deploy into an existing VPC instead of creating a new one:
```hcl theme={null}
module "chamber_gke" {
source = "github.com/ChamberOrg/terraform-google-chamber-gke"
gcp_project_id = var.gcp_project_id
gcp_region = var.gcp_region
cluster_name = "my-gpu-cluster"
chamber_cluster_token = var.chamber_cluster_token
chamber_cluster_id = var.chamber_cluster_id
create_vpc = false
network_name = "my-existing-vpc"
subnetwork_name = "my-existing-subnet"
ip_range_pods = "pods"
ip_range_services = "services"
}
```
The existing subnet must have secondary IP ranges named for pods and services. Cloud NAT must be configured for private node egress.
## Key Variables
The table below covers the most commonly configured variables. For the complete list, see the [module README on GitHub](https://github.com/ChamberOrg/terraform-google-chamber-gke#inputs).
### Required
| Variable | Description | Type |
| ----------------------- | ---------------------------------- | -------- |
| `gcp_project_id` | GCP project ID | `string` |
| `cluster_name` | Name of the GKE cluster | `string` |
| `chamber_cluster_token` | Cluster token from Chamber Console | `string` |
| `chamber_cluster_id` | Cluster ID from Chamber Console | `string` |
### GCP
| Variable | Description | Default |
| ------------ | --------------------------------------------- | --------------- |
| `gcp_region` | GCP region for the GKE cluster | `"us-central1"` |
| `gcp_zones` | Zones within the region (defaults to first 3) | `[]` |
### VPC
| Variable | Description | Default |
| ----------------- | --------------------------------------------------------- | --------------- |
| `create_vpc` | Create a new VPC or use existing | `true` |
| `network_name` | Existing VPC name (required when `create_vpc = false`) | `null` |
| `subnetwork_name` | Existing subnet name (required when `create_vpc = false`) | `null` |
| `vpc_cidr` | CIDR block for new VPC primary subnet | `"10.0.0.0/16"` |
### GKE
| Variable | Description | Default |
| -------------------------- | -------------------------------------------- | ----------------- |
| `cluster_version` | Kubernetes version | `"1.32"` |
| `system_node_machine_type` | Machine type for system node pool | `"e2-standard-4"` |
| `enable_private_endpoint` | Private endpoint only (no public API access) | `false` |
### GPU
| Variable | Description | Default |
| ----------------------------- | ------------------------------------- | ------------------------------------------------------- |
| `create_default_gpu_nodepool` | Create Terraform-managed GPU NodePool | `false` |
| `gpu_machine_families` | GPU machine families for NodePool | `["a2", "g2", "a3"]` |
| `gpu_accelerator_types` | GPU accelerator types | `["nvidia-l4", "nvidia-a100-80gb", "nvidia-h100-80gb"]` |
| `capacity_types` | Capacity types (`on-demand`, `spot`) | `["on-demand", "spot"]` |
| `gpu_limits` | Maximum GPUs for NodePool | `100` |
### Chamber
| Variable | Description | Default |
| ----------------------- | ----------------------------------- | ---------- |
| `chamber_agent_version` | Chamber Agent version | `"latest"` |
| `enable_kai_scheduler` | Enable KAI fractional GPU scheduler | `true` |
## Key Outputs
| Output | Description |
| --------------------------------- | ------------------------------------------ |
| `cluster_name` | GKE cluster name |
| `cluster_endpoint` | GKE API server endpoint |
| `network_name` | VPC network name |
| `configure_kubectl` | gcloud command to configure kubectl |
| `verification_commands` | Commands to verify the deployment |
| `karpenter_service_account_email` | Karpenter controller service account email |
For all outputs, see the [module README on GitHub](https://github.com/ChamberOrg/terraform-google-chamber-gke#outputs).
## GPU Pool Management
After deployment, you need GPU pools for Karpenter to know which GPU nodes to provision. There are two approaches:
Manage GPU pools through the [Chamber Console](https://app.usechamber.io):
1. Go to **Capacity Pools** > **Create Dynamic Pool**
2. Select your cluster and configure GPU type, limits, and capacity types
3. The pool syncs to your cluster automatically — Karpenter provisions GPU nodes on demand
This is the recommended approach for most teams. It allows per-GPU-type management with real-time limit adjustments.
To manage GPU pools entirely through Terraform, set `create_default_gpu_nodepool = true`:
```hcl theme={null}
module "chamber_gke" {
source = "github.com/ChamberOrg/terraform-google-chamber-gke"
# ... required variables ...
create_default_gpu_nodepool = true
gpu_machine_families = ["a2", "g2"]
gpu_accelerator_types = ["nvidia-l4", "nvidia-a100-80gb"]
capacity_types = ["on-demand", "spot"]
gpu_limits = 50
}
```
This creates a single broad NodePool covering the specified machine families and accelerator types.
## Troubleshooting
Check the Chamber Agent logs:
```bash theme={null}
kubectl logs -n chamber-system -l app.kubernetes.io/name=chamber-agent --tail=50
```
Verify your `chamber_cluster_token` and `chamber_cluster_id` are correct.
1. Verify a GPU pool exists:
```bash theme={null}
kubectl get nodepools.karpenter.sh
```
If none exists, create one via the Chamber Console or set `create_default_gpu_nodepool = true`.
2. Check Karpenter logs:
```bash theme={null}
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=100
```
3. Verify GCENodeClass:
```bash theme={null}
kubectl get gcenodeclasses
```
Check the KAI Scheduler logs and pod events:
```bash theme={null}
kubectl logs -n chamber-system -l app=kai-scheduler --tail=50
kubectl describe pod
```
## Cleanup
```bash theme={null}
terraform destroy
```
Ensure all GPU workloads are terminated before destroying to avoid orphaned resources.
## Next Steps
Submit your first GPU workload
Configure capacity pools and reservations
Detailed troubleshooting guide
Full source, examples, and changelog
# Cluster Deployment
Source: https://docs.usechamber.io/cluster-deployment/overview
Deploy GPU-ready Kubernetes clusters with Terraform
Chamber provides Terraform modules that deploy complete GPU-ready Kubernetes clusters on AWS and Google Cloud. A single `terraform apply` creates the entire infrastructure stack — from networking to GPU workload orchestration.
## What Gets Deployed
Each module provisions the full stack your team needs to run GPU workloads with Chamber:
```mermaid theme={null}
flowchart TD
VPC["VPC / Network"] --> K8s["Kubernetes Cluster\n(EKS or GKE)"]
K8s --> System["System Node Pool"]
K8s --> Karpenter["Karpenter\n(GPU Autoscaling)"]
K8s --> NVIDIA["NVIDIA GPU Operator\n(Drivers & Metrics)"]
K8s --> KAI["KAI Scheduler\n(Fractional GPU)"]
K8s --> Agent["Chamber Agent"]
Agent -->|Secure connection| Chamber["Chamber Control Plane"]
```
## Terraform Modules vs. Helm Agent
Chamber offers two ways to connect a cluster. Choose based on your starting point:
| | Terraform Modules | Helm Agent Only |
| ------------------- | ------------------------------------ | ------------------------------------- |
| **What it creates** | Full infrastructure stack | Chamber agent only |
| **Prerequisites** | Cloud provider account + Terraform | Existing Kubernetes cluster with GPUs |
| **Time to deploy** | \~15-20 minutes | \~30 seconds |
| **GPU autoscaling** | Included (Karpenter) | Not included |
| **GPU drivers** | Included (NVIDIA GPU Operator) | Must be pre-installed |
| **Best for** | New clusters, greenfield deployments | Existing clusters |
Already have a Kubernetes cluster with GPU nodes? Skip the Terraform modules and [install the Chamber agent directly via Helm](/agent/installation).
## Choose Your Cloud
Deploy a GPU-ready EKS cluster on Amazon Web Services
Deploy a GPU-ready GKE cluster on Google Cloud Platform
## Components
Both modules deploy the same core components:
[Karpenter](https://karpenter.sh/) automatically provisions and deprovisions GPU nodes based on workload demand. When a GPU job is submitted, Karpenter launches the right instance type. When nodes sit idle, Karpenter consolidates or terminates them to reduce costs.
The [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) installs GPU drivers, the device plugin (enabling `nvidia.com/gpu` resource requests), DCGM-Exporter (GPU metrics for Chamber dashboards), and GPU Feature Discovery (node labeling by GPU type).
The KAI Scheduler enables GPU time-sharing, allowing multiple containers to share a single GPU. This improves utilization and reduces costs for workloads that don't need a full GPU.
The [Chamber Agent](/agent/overview) connects your cluster to the Chamber control plane, reporting capacity, managing workloads, and collecting GPU metrics.
# Capacity Pools
Source: https://docs.usechamber.io/concepts/capacity-pools
GPU resource pools representing your clusters
A capacity pool is a logical grouping of GPU resources.
## Key Points
* Pools are created automatically when you install the Chamber agent
* A single Kubernetes cluster can have multiple pools
* The Chamber dashboard shows pool utilization and availability
## Related Concepts
Setting up the cluster agent
# Overview
Source: https://docs.usechamber.io/concepts/overview
How Chamber manages GPU capacity across your organization
Chamber is a GPU capacity control plane designed to maximize utilization while ensuring teams have guaranteed access to the resources they need.
## Architecture
Chamber operates as a SaaS platform with lightweight agents deployed in your Kubernetes clusters:
```mermaid theme={null}
flowchart TB
subgraph control["Chamber (Cloud)"]
Dashboard["Dashboard"]
end
control <-->|Secure connection| A
control <-->|Secure connection| B
control <-->|Secure connection| C
subgraph A["Cluster A"]
AgentA["Agent"]
GPUA["GPU Nodes"]
end
subgraph B["Cluster B"]
AgentB["Agent"]
GPUB["GPU Nodes"]
end
subgraph C["Cluster C"]
AgentC["Agent"]
GPUC["GPU Nodes"]
end
```
## How It Works
1. **Agents report capacity**: Lightweight agents in each cluster report available GPU resources
2. **Chamber schedules workloads**: When teams submit workloads, Chamber decides where to run them
3. **Agents execute locally**: Agents create and manage workloads on your clusters
4. **Dashboard provides visibility**: View utilization, manage teams, and track workloads
The agent only makes **outbound** connections to Chamber. No inbound firewall rules are required.
## Data Model
Chamber's data model reflects enterprise organizational structures:
```mermaid theme={null}
flowchart TD
Org["Organization"]
Org --> Pools["Capacity Pools\n (GPU clusters)"]
Org --> Teams["Teams"]
Teams --> Child Teams["Child Teams"]
Teams --> Res2["Reservations"]
Pools --> Res1["Reservations"]
Res1 <--> Res2
Res2 --> Workloads["Workloads"]
```
### Key Relationships
| Relationship | Description |
| -------------------- | --------------------------------------- |
| Organization → Teams | An org contains a hierarchy of teams |
| Organization → Pools | An org can have multiple capacity pools |
| Team ↔ Pool | Many-to-many via reservations |
| Team → Workloads | Workloads belong to exactly one team |
## Workload Classes
| Class | Capacity Source | Preemptible | Use Case |
| ------------ | --------------------------------------- | ----------- | --------------------------------------- |
| **Reserved** | Team's reservation | No | Production training, SLA workloads |
| **Elastic** | Reserved capacity first, then idle pool | Yes | Experiments, batch processing, dev work |
Elastic workloads maximize cluster utilization by using capacity that would otherwise sit idle, while still guaranteeing reserved workloads always have resources.
## Next Steps
Understand the organizational hierarchy
Learn about GPU resource pools
How capacity is allocated to teams
Deep dive into the scheduler
# Reservations
Source: https://docs.usechamber.io/concepts/reservations
Allocating GPU capacity from pools to teams
Reservations are the link between capacity pools and teams. They guarantee a specific amount of GPU capacity for a team.
## What is a Reservation?
A reservation:
* Allocates GPUs from a pool to a team
* Guarantees capacity for reserved workloads
* Can be adjusted or removed as needs change
* Supports the many-to-many relationship between pools and teams
## Many-to-Many Relationships
Teams and pools have a many-to-many relationship through reservations:
### One Team, Multiple Pools
A team can have reservations in multiple pools:
```mermaid theme={null}
flowchart LR
Team["Team: ML Platform\n Total: 112 GPUs across 3 pools"]
Team --> West["us-west-cluster\n 64 GPUs"]
Team --> East["us-east-cluster\n 32 GPUs"]
Team --> EU["eu-cluster\n 16 GPUs"]
```
This enables:
* Geographic distribution of workloads
* Access to different GPU types
* Redundancy across clusters
### One Pool, Multiple Teams
A pool can have reservations from multiple teams:
```mermaid theme={null}
flowchart LR
Pool["Pool: us-west-cluster\n 256 GPUs total"]
Pool --> ML["ML Platform\n 64 GPUs"]
Pool --> CV["CV Team\n 48 GPUs"]
Pool --> NLP["NLP Team\n 32 GPUs"]
Pool --> Unreserved["Unreserved\n 112 GPUs (elastic)"]
```
## Capacity Guarantees
### Reserved Capacity
When a team has a reservation:
* Reserved workloads from that team are **guaranteed** those GPUs
* Workloads can start immediately if capacity is available within the reservation
* Other teams cannot use this capacity for reserved workloads
### Elastic Usage
Unreserved pool capacity is available for elastic workloads from any team:
```mermaid theme={null}
flowchart TD
subgraph pool["Pool: 100 GPUs"]
Reserved["Reserved: 60 GPUs"]
Unreserved["Unreserved: 40 GPUs"]
end
subgraph runtime["At Runtime"]
Running["Reserved running: 45 GPUs"]
Idle["Reserved idle: 15 GPUs\n ← Elastic can use this!"]
Elastic["Elastic running: 40 GPUs"]
end
pool --> runtime
```
Elastic workloads can use idle reserved capacity, but will be preempted if a reserved workload needs those resources.
## Best Practices
Reserve what teams actually need, not what they might need. Unused reserved capacity can't serve elastic workloads from other teams.
Teams with unpredictable demand should use smaller reservations plus elastic capacity rather than large reservations.
Audit reservations quarterly. Teams' needs change; reservations should too.
Configure alerts for low utilization (below 30%) and high queue depth to catch misconfigurations.
## Related Concepts
The teams that receive reservations
How workloads use reserved vs elastic capacity
# Scheduling
Source: https://docs.usechamber.io/concepts/scheduling
How Chamber schedules GPU workloads across your organization
Chamber's scheduling engine decides where and when your GPU workloads run — enforcing capacity guarantees, hierarchical bursting, and preemption automatically.
## How a Workload Gets Scheduled
```mermaid theme={null}
flowchart LR
User([You]) -->|submit| API[API]
API -->|Pending| Scheduler[Scheduler]
Scheduler -->|Queued| Agent[Chamber Agent]
Agent -->|Starting| K8s[Kubernetes]
K8s -->|Running| GPUs[GPUs]
Agent -.->|report status| API
```
You submit a workload via the API, CLI, or dashboard. Chamber records it and marks it **Pending**.
The scheduler assigns an allocation — picking the best fit based on your team's reservations and available capacity. The workload moves to **Queued**.
The workload is dispatched to the Chamber Agent running in your Kubernetes cluster.
The in-cluster scheduler admits the workload based on your team's quota and current demand. When GPUs are available, it places the pods. The workload moves to **Starting**, then **Running**.
The agent reports status back — you see real-time updates as your workload progresses through to **Completed**, or is **Preempted** / **Failed** if something changes.
## Reserved vs Elastic Workloads
Every workload you submit is either **Reserved** or **Elastic**. This determines how it accesses GPU capacity and whether it can be preempted.
| | Reserved | Elastic |
| ------------------------ | ----------------------------------------------------- | ---------------------------------------------- |
| **What it means** | Guaranteed capacity backed by your team's reservation | Uses spare GPUs when available |
| **Can it be preempted?** | Never | Yes — when reserved work needs the GPUs |
| **Priority level** | Non-preemptible (priority ≥ 100) | Preemptible (priority \< 100) |
| **Best for** | Production inference, critical training runs | Experiments, hyperparameter sweeps, batch jobs |
Use **Reserved** for work that can't be interrupted. Use **Elastic** to take advantage of idle GPUs across the cluster — you get free capacity, but accept the trade-off that it can be reclaimed.
## How Scheduling Decisions Are Made
When you submit a workload:
1. **Chamber picks the best allocation** for your workload based on available capacity across your team's reservations.
2. **Reserved workloads are admitted first.** If your team's allocation has room, the workload starts. Reserved capacity is guaranteed and protected.
3. **Elastic workloads use surplus capacity.** They run on idle GPUs — either from your team's unused reservation or from spare capacity elsewhere in the pool. Elastic workloads can be reclaimed at any time when reserved work needs the GPUs.
4. **The scheduler evaluates continuously.** As workloads complete and GPUs free up, waiting workloads are admitted automatically.
### Key Terms
| Term | What It Means |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Bursting** | When a team's elastic workloads use GPUs beyond their reserved allocation. Your team "bursts" into idle capacity from the pool. |
| **Burst limit** | The maximum amount a team can burst beyond its reservation. For example, a 50% burst limit on a 32-GPU reservation means the team can use up to 48 GPUs total (32 reserved + 16 burst). Set to 0 to disable bursting entirely. |
| **Bursting priority** | A weight (1–10) that controls how surplus GPUs are divided when multiple teams are bursting at the same time. Higher priority means a larger share of the available surplus. |
| **Preemption** | When the scheduler stops an elastic workload to free GPUs for higher-priority work. Only elastic workloads can be preempted — reserved workloads are always protected. |
## Hierarchical Bursting
Your organization's teams are arranged in a tree. Chamber respects this hierarchy when distributing surplus capacity:
```
Acme Corp
├── Research (bursting priority: 3)
│ ├── NLP Team (bursting priority: 2)
│ └── Vision Team (bursting priority: 1)
└── Production (bursting priority: 2)
├── Inference (bursting priority: 1)
└── Training (bursting priority: 1)
```
* Each team has a **bursting priority** that controls its share of surplus GPUs when multiple teams are bursting.
* Surplus capacity flows down the tree proportionally — Research gets 60% of excess (priority 3 out of 5), Production gets 40% (priority 2 out of 5).
* Within Research, NLP gets twice the surplus share of Vision (priority 2 vs 1).
* Teams don't compete in a flat pool. The hierarchy ensures organizational priorities are respected even when capacity is tight.
* Each team can also have a **burst limit** that caps how far beyond its reservation it can go — preventing any single team from consuming the entire pool.
### Example: Hierarchical Bursting in Action
A pool has 100 GPUs. Research has 48 reserved, Production has 32 reserved. Research is only using 20.
| Team | Reservation | In Use | Idle | What Happens |
| ---------- | ----------- | ------ | ---- | --------------------------------------------------------- |
| Research | 48 | 20 | 28 | 28 idle GPUs available for bursting |
| Production | 32 | 32 | 0 | Elastic workloads can burst into Research's idle capacity |
| Pool total | 100 | 52 | — | 48 surplus GPUs distributed by bursting priority |
Production's elastic workloads can temporarily burst into Research's idle GPUs. But the moment Research submits reserved work that needs those GPUs back, Production's elastic workloads are preempted.
## Preemption
Preemption is how Chamber reclaims GPUs for higher-priority work. It's designed to be predictable and minimize disruption.
### Rules
1. **Reserved workloads are never preempted.** The scheduler will not reclaim them under any circumstances.
2. **Elastic workloads are always reclaimable.** The scheduler can reclaim their resources when reserved workloads need capacity.
3. **Higher-priority elastic workloads displace lower-priority ones.** Among elastic workloads, priority determines who stays when the pool is full.
4. **Cheapest workloads are reclaimed first.** The scheduler picks victims in this order:
* **Queued** workloads (haven't started — zero cost)
* **Starting** workloads (pods being created — minimal cost)
* **Running** workloads, lowest priority first (most expensive, last resort)
5. **Only enough workloads are reclaimed** to free the capacity needed — no more.
### Preemption Examples
NLP Team has a **32-GPU reservation**. They're running 20 GPUs of reserved inference and 12 GPUs of elastic experiments — fully utilizing their allocation. A researcher submits a **12-GPU reserved training workload**.
```mermaid theme={null}
flowchart LR
subgraph Before["Before — NLP Team: 32 / 32 GPUs"]
B1["Reserved inference - 20 GPUs"]
B2["Elastic experiments - 12 GPUs"]
end
E{{"New 12-GPU reserved workload"}}
subgraph After["After — NLP Team: 32 / 32 GPUs"]
A1["Reserved inference - 20 GPUs"]
A2["Reserved training - 12 GPUs ✓"]
end
Before --> E --> After
```
The 12 elastic experiments are preempted to make room. Reserved always wins over elastic within the same allocation.
The pool is fully utilized. Team A is running an **8-GPU elastic sweep at priority 50**. Team B submits an **8-GPU elastic workload at priority 75**.
```mermaid theme={null}
flowchart LR
subgraph Before["Before — Pool: fully utilized"]
B1["Team A elastic sweep - 8 GPUs · priority 50"]
B2["Team B - waiting"]
end
E{{"Team B submits priority-75 workload"}}
subgraph After["After — Pool: fully utilized"]
A1["Team A - preempted"]
A2["Team B elastic workload - 8 GPUs · priority 75 ✓"]
end
Before --> E --> After
```
Both are elastic, but priority determines who stays. The lower-priority sweep is reclaimed.
Vision Team has a **16-GPU reservation** but is only using 4 GPUs. A researcher submits an **8-GPU elastic workload**.
```mermaid theme={null}
flowchart LR
subgraph Before["Before — Vision Team: 4/16 GPUs"]
B1["Reserved workload - 4 GPUs"]
B2["Idle - 12 GPUs"]
end
E{{"New 8-GPU elastic workload"}}
subgraph After["After — Vision Team: 12/16 GPUs"]
A1["Reserved workload - 4 GPUs"]
A2["Elastic workload - 8 GPUs ✓"]
end
Before --> E --> After
```
Plenty of idle capacity — the elastic workload starts immediately with no preemption. If reserved work later needs those GPUs, this elastic workload would be the first to go.
Research has **48 GPUs reserved** but is only using 20. Production's elastic workloads have expanded into **28 of Research's idle GPUs**. A Research team member submits a **24-GPU reserved workload**.
```mermaid theme={null}
flowchart LR
subgraph Before["Before — Pool: 100 GPUs"]
B1["Research - 20 reserved"]
B2["Production - 32 reserved + 28 elastic"]
end
E{{"Research submits 24-GPU reserved workload"}}
subgraph After["After — Pool: 100 GPUs"]
A1["Research - 44 reserved ✓"]
A2["Production - 32 reserved + 4 elastic"]
end
Before --> E --> After
```
Research's reservation can support the new workload (48 − 20 = 28 free slots ≥ 24 needed). The scheduler preempts 24 of Production's 28 elastic workloads — just enough to free the physical GPUs. Production's remaining 4 elastic workloads and all reserved workloads are untouched.
## Fractional GPUs
Not every workload needs a full GPU. You can request fractions in 0.25 increments:
| Request | Use Case |
| ---------- | ------------------------------------------- |
| `0.25` GPU | Lightweight inference, notebooks, debugging |
| `0.5` GPU | Moderate inference, small model fine-tuning |
| `0.75` GPU | Heavier single-GPU tasks |
| `1.0+` GPU | Standard training and inference |
Multiple fractional workloads share a single physical GPU via memory-based isolation and time-slicing. You can specify either a GPU fraction (e.g., `0.5`) or an explicit GPU memory limit in MiB.
## Distributed Training
For multi-GPU workloads that span multiple pods, Chamber supports two scheduling modes.
### Gang Scheduling
All pods must start together or none do. No partial starts, no stragglers. This is the default behavior.
**Example:** You submit an 8-node distributed training workload. The cluster only has 6 GPUs free. Chamber waits until all 8 are available, then starts them simultaneously. Your training framework sees all workers from the first step.
### Topology-Aware Placement
For latency-sensitive distributed workloads, Chamber co-locates pods within network topology boundaries:
* **Required placement** — All pods must land within a specific topology boundary (e.g., same zone). If placement isn't possible, the workload waits.
* **Preferred placement** — Pods are placed within a tighter boundary (e.g., same rack) when possible, but the constraint is relaxed if needed.
This is useful for distributed training workloads that benefit from NVLink or high-bandwidth interconnects between nodes.
Topology-aware placement requires that your cluster nodes are labeled with the appropriate topology keys (e.g., `topology.kubernetes.io/zone`, rack labels). A topology definition must also be configured on the cluster. Contact your cluster administrator to ensure nodes are properly tagged before using this feature.
## Workload Status Reference
| Status | What It Means | What You Should Do |
| ------------- | --------------------------------------------- | ---------------------------------------------------------- |
| **Pending** | Submitted, waiting for allocation assignment | Wait — Chamber is finding the best allocation |
| **Queued** | Dispatched to cluster, waiting for GPUs | Wait — the scheduler will admit when capacity is available |
| **Starting** | Admitted by the scheduler, pods being created | Wait — images are pulling, almost there |
| **Running** | All pods are running | Monitor your workload |
| **Completed** | Finished successfully | Collect results |
| **Failed** | Something went wrong | Check the failure reason |
| **Preempted** | Reclaimed for a higher-priority workload | Resubmit — or switch to Reserved if this keeps happening |
| **Cancelled** | You cancelled it | No action needed |
If your elastic workloads are frequently preempted, consider increasing their priority, switching critical work to Reserved, asking your admin to raise your team's bursting priority, or scheduling during off-peak hours when more surplus capacity is available.
## Related Concepts
Reserved vs elastic workload types
How capacity is allocated to teams
Organizational hierarchy and bursting priorities
GPU resource pools representing your clusters
# Teams
Source: https://docs.usechamber.io/concepts/teams
Hierarchical organization of teams and projects
Teams are the core organizational unit in Chamber. They represent teams, projects, or business units that consume GPU resources.
## What is a Team?
A team is a named entity that:
* Belongs to an organization
* Can have parent and child teams (forming a hierarchy)
* Receives capacity reservations from pools
* Contains workloads
Think of teams as your org chart mapped to GPU resource allocation.
## Hierarchy
Teams form a tree structure that mirrors your organization:
```mermaid theme={null}
flowchart TD
Research["Research (root team)"]
Research --> CV["Computer Vision"]
Research --> NLP
Research --> Robotics
CV --> OD["Object Detection"]
CV --> IG["Image Generation"]
NLP --> LLM["LLM Training"]
NLP --> SR["Speech Recognition"]
```
### Why Hierarchies Matter
Child teams (aka Projects) can use parent capacity when their own is exhausted
Managers see aggregate utilization for their entire subtree
Team leads manage their subtree without org-wide access
Restructure teams without recreating reservations
## Capacity and Workloads
### Reservations
Each team can have reservations in one or more capacity pools:
```mermaid theme={null}
flowchart LR
Team["Team: Computer Vision"]
Team --> West["us-west-cluster\n 64 GPUs reserved"]
Team --> East["us-east-cluster\n 32 GPUs reserved"]
```
### Workloads
Workloads are associated with exactly one team via labels:
```yaml theme={null}
metadata:
labels:
chamber.io/team: computer-vision
```
Workloads without a team label will be rejected by the Chamber scheduler.
## Best Practices
Begin with a flat structure (one team per major team). Add hierarchy only when you need finer-grained capacity control.
Slugs appear in workload labels and CLI commands. Use short, descriptive slugs like `cv-team` instead of `computer-vision-research-team-2024`.
Structure teams to match how you want to see utilization reports. If you report by project, make projects teams.
## Related Concepts
How capacity is allocated to teams
Reserved vs elastic workloads
# Workload Classes
Source: https://docs.usechamber.io/concepts/workload-classes
Reserved vs elastic workloads
Chamber supports two workload classes that determine how workloads access GPU capacity: **Reserved** and **Elastic**.
## Overview
| Class | Capacity Source | Preemptible | Best For |
| ------------ | --------------------------------------- | ----------- | --------------------------------------- |
| **Reserved** | Team's reservation | No | Production training, SLA workloads |
| **Elastic** | Reserved capacity first, then idle pool | Yes | Experiments, batch processing, dev work |
## Reserved Workloads
Reserved workloads use capacity from the team's reservation. They are guaranteed resources and cannot be preempted.
**Use reserved for:**
* Long-running training that can't be interrupted
* Workloads with delivery commitments or deadlines
* ML pipelines that must complete reliably
## Elastic Workloads
Elastic workloads use idle capacity. They maximize utilization but can be preempted when reserved workloads need resources.
**Use elastic for:**
* Experiments and exploratory work
* Development and debugging
* Batch workloads that can checkpoint and resume
Design elastic workloads to checkpoint periodically so they can resume if preempted.
## Related Concepts
How Chamber schedules workloads
# Introduction
Source: https://docs.usechamber.io/introduction
The AI Ops agent for ML teams
Chamber is the AI Ops agent for ML teams. Chamber helps teams monitor, debug, root cause, and resolve GPU workload issues automatically without manual intervention. Gain out of the box insights into your GPU cluster resources, utilization, cost and team/workload level breakdowns automatically. Use AI-powered insights and autonomous agents to resolve failures, and optimize usage to accelerate development, reduce costs, and ship faster.
## What Chamber Does
Autonomous agent that monitors, diagnoses, and resolves GPU workload issues 24/7
Dashboard with GPU utilization metrics, workload status, and team performance insights
Allocate GPU resources through hierarchical teams with guaranteed reservations
Two-phase scheduling with reserved and elastic workload classes for optimal resource utilization
Manage capacity across multiple Kubernetes clusters from a single control plane
*See Chamber AI Ops agent in action*
## How It Works
Chamber operates as a SaaS control plane with a lightweight agent deployed in your Kubernetes clusters:
Install the Chamber agent in your GPU clusters via Helm chart. The agent syncs workload state with the control plane.
Chamber automatically discovers cluster resources and running workloads, collects infrastructure and application metrics, and surfaces AI-powered insights on the dashboard — with no manual configuration.
Assign GPU reservations from your capacity pools to teams. Teams get guaranteed access to their reserved capacity.
Workloads submitted to your cluster are scheduled by Chamber. Reserved workloads get guaranteed resources; elastic workloads use idle capacity.
## Getting Started
Get up and running with Chamber in minutes
Deep dive into how Chamber works
# Capacity Management
Source: https://docs.usechamber.io/platform/capacity-management
Manage capacity pools across your clusters
The Clusters page lets you manage capacity pools across your Kubernetes clusters.
![Clusters]()
## Overview Metrics
| Metric | Description |
| ------------------ | ---------------------------------- |
| **Total Clusters** | Connected Kubernetes clusters |
| **Total Pools** | Capacity pools across all clusters |
| **Total GPUs** | GPU resources across all clusters |
## Cluster Details
Each cluster card shows:
* Cluster name and region
* Kubernetes version
* Node count and GPU count
* Status (active/inactive)
* Number of capacity pools
Click **Sync New Cluster** to connect a new cluster using the installation wizard.
# Cost Explorer
Source: https://docs.usechamber.io/platform/cost-explorer
Analyze, forecast, and optimize GPU spend across your organization
Know exactly where your GPU dollars go. The Cost Explorer gives you full visibility into spend across teams, clusters, GPU types, and workload classes — with built-in forecasting, waste detection, and optimization recommendations to help you get more out of every GPU-hour.
Navigate to **Cost Explorer** in the Chamber dashboard sidebar to get started.
## At a Glance
Cost Explorer is built to answer the questions every GPU infrastructure team asks:
* **How much are we spending?** Total and per-team GPU costs with trend analysis
* **Where is the money going?** Breakdown by team, user, cluster, GPU type, and workload class
* **What will we spend next?** Month-end forecasts with confidence intervals
* **What are we wasting?** Idle capacity, failed workload costs, and underutilized resources
* **How do we optimize?** Actionable recommendations for spot instances, rightsizing, and failure reduction
## Key Metrics
Four headline numbers are displayed at the top of every Cost Explorer view:
| Metric | Description |
| ----------------------- | ----------------------------------------------------------------- |
| **Total Spend** | GPU cost from completed workloads in the selected time range |
| **Month to Date** | Cumulative GPU spend so far this calendar month |
| **Projected Month-End** | Where you'll land by end of month, based on current trajectory |
| **Trend vs. Prior** | Percentage change comparing the recent period to the prior period |
## Filtering and Grouping
### Filters
Slice your data by any combination of:
* **Team** — One or more teams
* **Cluster** — Specific capacity pools
* **GPU Type** — Hardware type (A100, H100, etc.)
* **Workload Class** — Type of workload (training, inference, interactive, etc.)
Filters apply across every chart, metric, and table on the page. Active filters appear as removable pills below the filter bar.
### Grouping
Change how the spend chart and cost breakdown are organized:
* Team / User / Cluster / GPU Type / Workload Class
Switch grouping at any time to see costs from a completely different angle.
### Time Range
Pick from presets or define your own window:
* Last 7 / 30 / 90 days
* This month / Last month
* **Custom range** with a calendar picker
Filters and time range are saved in the URL — bookmark or share a specific cost view with your team.
## Spend Over Time
A time-series chart shows how your spend trends across the selected period:
* **Granularity** — Toggle between daily, weekly, or monthly aggregation
* **Stacked series** — Top contributors are shown individually; smaller ones grouped as "Other"
* **Forecast overlay** — Project costs forward with a shaded 95% confidence band
### Forecasting
Turn on the forecast overlay to see where spend is heading:
* Uses linear regression fitted to your daily historical data
* Shows a projected line with upper and lower confidence bounds
* Defaults to an end-of-month projection, but you can pick any date up to 365 days ahead
* **Trend detection** automatically classifies your spend as increasing (over 10% growth), decreasing (over 10% decline), or stable
## Capacity Utilization
This section compares what you're paying for (provisioned GPU capacity) against what's actually being used by workloads:
| Metric | Description |
| ----------------------- | ----------------------------------------------------- |
| **Total Capacity Cost** | Cost of all provisioned GPUs across selected clusters |
| **Used Cost** | Cost attributable to completed workloads |
| **Wasted Cost** | Idle capacity — provisioned but sitting unused |
| **Utilization %** | Percentage of provisioned capacity actively used |
Utilization is color-coded so you can spot problems at a glance:
* **Green** (70%+) — Healthy
* **Orange** (40-70%) — Worth reviewing
* **Red** (under 40%) — Significant optimization opportunity
Capacity metrics come from cluster telemetry. Team and Workload Class filters only affect the "Used Cost" number, since provisioned capacity isn't tied to specific workloads.
## Cost Breakdown
A donut chart shows the proportional distribution of spend by your selected dimension. Items contributing less than 1% are grouped as "Other" — hover to expand the full list.
## Cost Events Table
Dig into the details with a full table of individual cost events:
| Column | Description |
| -------------- | ----------------------------- |
| Date | When the cost was incurred |
| Team | Team name |
| User | Who submitted the workload |
| Cluster | Capacity pool |
| GPU Type | Hardware type |
| Workload Class | Type of workload |
| Hours | GPU instance-hours consumed |
| Rate | Hourly rate for this GPU type |
| Cost | Total cost for this event |
### Pagination and Export
* Navigate results with cursor-based pagination
* Set your page size: 10, 25, 50, or 100 rows
* **Export to CSV** to pull data into spreadsheets or external tools (up to 100K rows)
## GPU Rate Configuration
Cost calculations rely on per-GPU-type hourly rates. Org Admins can set these up:
1. Click the **settings icon** in the Cost Explorer header
2. Set the hourly rate for each GPU type in your organization
3. Rates apply retroactively to all cost calculations
## Waste Analysis
Chamber automatically surfaces wasted GPU spend from three sources:
| Waste Category | Description |
| ---------------------------- | ------------------------------------------------------------------------------------ |
| **Failed Workloads** | GPU time consumed by workloads that ultimately failed |
| **Preempted Workloads** | GPU time from workloads preempted before completion |
| **Short Runtime, High Cost** | Workloads under 5 minutes that cost more than \$1 — often a sign of misconfiguration |
## Optimization Recommendations
Chamber analyzes your usage patterns and surfaces actionable next steps:
Identifies workloads that could run on spot or preemptible instances, with estimated monthly savings.
Flags teams or GPU types with failure rates above 20% and calculates the cost impact.
# Dashboard
Source: https://docs.usechamber.io/platform/dashboard
Real-time GPU utilization, spend efficiency, and AI-powered insights
The Chamber dashboard is your command center for GPU infrastructure. It has two views: the **Executive GPU Usage Dashboard** for real-time operational metrics, and the **GPU Spend & Efficiency Dashboard** for cost analysis and optimization. Switch between them using the tabs at the top of the page.
![Executive GPU Usage Dashboard]()
## Executive GPU Usage Dashboard
Get a live pulse on your GPU fleet.
### Key Metrics
| Metric | Description |
| -------------------- | ----------------------------------- |
| **Total GPUs** | Total GPU count across all clusters |
| **Running GPUs** | GPUs currently executing workloads |
| **Unused GPUs** | GPUs not currently in use |
| **Active Workloads** | Running and pending workloads |
### Features
* **Cluster filtering** — View metrics for specific clusters or all clusters at once
* **Unused GPUs by Chip Type** — Breakdown of idle GPUs by GPU model
* **Active Workloads chart** — Running vs. queued workloads over the last 48 hours
* **Utilization trends** — Track GPU usage patterns over time
## GPU Spend & Efficiency Dashboard
See the financial picture behind your GPU usage — where the money goes, what's being wasted, and where to optimize.
### Headline Metrics
Five KPIs give you the high-level story:
| Metric | Description |
| ------------------------- | ------------------------------------------------------------- |
| **Total GPU Spend** | Organization-wide GPU cost from workload runtime |
| **Effective Utilization** | Percentage of provisioned capacity actively used by workloads |
| **Recoverable Waste** | Cost from failed, preempted, or short-runtime workloads |
| **Idle Cost** | Provisioned GPU capacity that went unused |
| **Forecast Month-End** | Projected total spend by end of month with trend indicator |
### Spend Breakdown
Three side-by-side cards show your top cost drivers:
* **By GPU Type** — Which hardware is costing the most
* **By Cluster** — Which capacity pools have the highest spend
* **By Team** — Team rankings with utilization efficiency bars
Each card shows cost, percentage of total, utilization rate, and waste percentage for the top 3 contributors.
### Spend by Utilization Band
A horizontal bar chart groups your spend into utilization ranges so you can immediately see where efficiency breaks down:
| Band | Utilization | What It Means |
| ----------------------- | ----------- | ---------------------------------------- |
| Severe underutilization | 0-20% | GPUs mostly idle during workload runtime |
| Poor utilization | 20-40% | Significant optimization opportunity |
| Acceptable | 40-60% | Room for improvement |
| Good | 60-80% | Healthy usage |
| Excellent | 80-100% | Near-optimal GPU efficiency |
If more than 40% of your spend falls below 40% utilization, Chamber flags this as a recoverable waste opportunity with specific recommendations.
### AI Insights
Chamber generates AI-powered analysis of your spend patterns, surfacing:
* **Critical waste alerts** — Highlights the percentage of spend with low utilization and what to do about it
* **Workload distribution analysis** — Shows how workloads cluster across utilization bands
* **Recommended actions** — Specific next steps like GPU type prioritization, team-specific rightsizing, and auto-scaling policy suggestions
### Spend Trends
Three time-series charts track how costs evolve:
* **Daily Spend** — Total cost per day
* **Spend by GPU Type** — Stacked area chart breaking out hardware cost distribution
* **Idle vs. Used Cost** — Stacked area showing provisioned capacity vs. actual workload usage
### Forecasts
See where spend is heading, broken down by:
* **Cluster** — Projected cost per capacity pool
* **GPU Type** — Projected cost per hardware type
* **Team** — Projected cost per team
### Team Leaderboard
A ranked table showing every team's cost efficiency:
| Column | Description |
| ----------------- | ------------------------------------------------------------------- |
| **Spend** | Total GPU cost for the team |
| **% of Total** | Team's share of organization-wide spend |
| **Utilization** | GPU efficiency percentage (color-coded) |
| **Est. Waste** | Estimated spend lost to utilization below the 70% efficiency target |
| **Cost/Workload** | Average GPU cost per completed workload |
Click any team to jump to their detailed project view.
Need to dig deeper with custom filters, CSV exports, or longer forecasts? Head to the [Cost Explorer](/platform/cost-explorer).
# Email
Source: https://docs.usechamber.io/platform/integrations/email
Automated utilization digests, workload notifications, and AI-powered email from Chamber
Chamber delivers GPU utilization digests, workload status alerts, and AI-composed messages directly to your team's inbox — so everyone stays informed without needing to open the dashboard.
## Why Email?
* **Stakeholder visibility** — Weekly digests give leadership and finance teams GPU cost and utilization data without needing a Chamber login
* **Personal workload alerts** — Get notified the moment your workloads start, complete, fail, or get preempted
* **Flexible delivery** — Send to all org members, target specific roles, or add external email aliases
* **AI-powered outreach** — Let the Chamber AI assistant draft and send professional emails on your behalf, backed by real usage data
## Weekly Utilization Digest
The weekly digest is a comprehensive GPU usage report delivered to your team every week. It's enabled by default for all organizations.
### What's Included
Each digest covers the previous 7 days and includes:
* **Utilization Summary** — Overall GPU usage percentage with week-over-week trend
* **Team Breakdown** — Per-team GPU allocation vs. actual usage
* **Top Performing Workloads** — Workloads with the highest GPU efficiency
* **Improvement Opportunities** — Underutilized workloads that could be optimized or rightsized
* **Cost Insights** — Estimated GPU costs, potential savings, and cost-per-workload metrics
### Configure the Weekly Digest
Digest configuration requires **Org Admin** permissions.
Navigate to **Settings → Notifications** in the Chamber dashboard.
Choose which **day of the week** (default: Monday) and **time** (default: 8:00 AM UTC) the digest goes out.
Select who receives the digest:
* **All Members** — Every org member receives it (individuals can opt out)
* **Subscribers Only** — Only members who explicitly subscribe receive it
Restrict delivery to specific roles: Platform Admin, Org Admin, Team Lead, or Member.
Add external email addresses (e.g., `finance-team@company.com`) that should also receive the digest.
Use **All Members** mode with role filtering to automatically reach all Team Leads and Admins, while individual members can still opt in on their own.
### Send a Test Digest
Want to see what the email looks like before rolling it out? Admins can send a test digest at any time using real data from the previous week. Go to **Settings → Notifications** and click **Send Test Digest** — it's sent only to you.
Test digests are rate-limited to once every 5 minutes per user.
## Daily Digest
For teams that need more frequent updates, Chamber also offers a daily utilization digest covering the previous 24 hours with the same metrics as the weekly version.
The daily digest is **disabled by default**. To enable it:
1. Go to **Settings → Notifications**
2. Toggle **Daily Digest** on
3. Set the delivery time (UTC)
4. Choose the recipient mode and roles, just like the weekly digest
Daily digests support their own separate set of email aliases, so you can route daily reports to an ops team while weekly reports go to leadership.
## Workload Notifications
Get an email the moment one of your workloads changes status. Each team member configures their own preferences independently.
### Supported Workload Events
| Event | Description |
| ------------- | ----------------------------------------------------------------- |
| **Running** | Your workload has started executing on a GPU |
| **Completed** | Your workload finished successfully |
| **Failed** | Your workload encountered an error |
| **Preempted** | Your elastic workload was preempted by a higher-priority workload |
| **Queued** | Your workload entered the scheduling queue |
| **Cancelled** | Your workload was cancelled |
### Configure Workload Notifications
1. Go to **Settings → Notifications**
2. Under **Workload Notifications**, toggle on the events you want emails for
3. Changes take effect immediately
Workload notifications are sent to the email associated with your Chamber account. Only events for **your own workloads** trigger emails.
## Managing Your Subscription
### Opt In / Opt Out
Depending on your organization's recipient mode:
* **All Members mode** — You're subscribed by default. Go to **Settings → Notifications** and click **Unsubscribe** to opt out of the weekly or daily digest.
* **Subscribers Only mode** — You start unsubscribed. Go to **Settings → Notifications** and click **Subscribe** to opt in.
### Email Aliases
Org Admins can add external email addresses that receive digests alongside org members. Great for:
* Distribution lists (`gpu-reports@company.com`)
* Finance or leadership teams who don't have Chamber accounts
* External stakeholders who need visibility into GPU costs
Aliases can be configured separately for weekly and daily digests.
## AI-Powered Email
The Chamber AI assistant (available via the Slack chatbot or dashboard chat) can compose and send emails on your behalf using live Chamber data.
### How It Works
Describe who you want to reach and what you want to say — for example, "Email the top 5 teams by GPU spend with their utilization stats."
The AI identifies the right people, pulls relevant usage stats, workload details, or capacity info, and drafts a professional message.
You review the draft and can request edits. Nothing sends without your approval.
The email is delivered with clear attribution showing it came from you via Chamber.
### Recipient Types
| Type | Description |
| ------------------ | ------------------------------------------- |
| **Top Teams** | Leads of the N teams with highest GPU usage |
| **Team Lead** | The lead of a specific team |
| **Workload Owner** | The user who submitted a specific workload |
| **Yourself** | Send a report to your own inbox |
| **Specific Users** | Choose users by name or email address |
### Tone Options
Match your communication style — choose from **formal**, **friendly**, **urgent**, or **informational**.
### Data You Can Include
* GPU usage statistics
* Workload details and history
* Capacity and allocation info
* Optimization recommendations
Try: "Email the top 3 teams by GPU spend this month with their utilization stats and suggestions for improvement" — the AI handles everything from recipient resolution to drafting.
# Integrations
Source: https://docs.usechamber.io/platform/integrations/overview
Connect Chamber with your tools for notifications, alerts, and reports
Stop context-switching between tools. Chamber integrates directly with the platforms your team already lives in, so GPU alerts, usage reports, and AI-powered insights show up right where you work.
![Integrations]()
## Available Integrations
Real-time workload and capacity alerts in your channels, an AI chatbot for natural-language queries, per-user notification preferences, and automated team reports on a schedule you choose.
Weekly and daily GPU utilization digests delivered to any inbox, per-user workload notifications, support for external email aliases, and AI-composed emails backed by real usage data.
| Integration | Status | Highlights |
| ------------- | ----------- | ------------------------------------------------------------- |
| **Slack** | Available | Notifications, AI chatbot, channel routing, scheduled reports |
| **Email** | Available | Utilization digests, workload alerts, AI-powered email |
| **PagerDuty** | Coming Soon | Incident creation for critical capacity and workload issues |
| **Webhooks** | Coming Soon | HTTP callbacks for custom automation |
## What You Can Get Notified About
Between Slack and email, Chamber keeps your team informed across every stage of the GPU lifecycle:
| Category | Events | Slack | Email |
| ---------------------- | ------------------------------------------------------------- | --------------------- | -------------------- |
| **Workload Lifecycle** | Started, completed, failed, preempted, queued, cancelled | Per-user preferences | Per-user preferences |
| **Capacity Alerts** | Utilization warnings, pool capacity low, reservation expiring | Org-level toggles | — |
| **Clawback** | Reclaimable underutilized reserved capacity | Org-level toggle | — |
| **Reports** | Daily and weekly utilization digests | Scheduled to channels | Scheduled to inboxes |
| **AI Outreach** | AI-composed messages with real usage data | Via chatbot | AI-drafted emails |
## Getting Started
Head to **Settings → Integrations** in the Chamber dashboard to connect Slack, or **Settings → Notifications** to configure email preferences.
Need an integration we don't support yet? Let us know!
# Slack
Source: https://docs.usechamber.io/platform/integrations/slack
Real-time notifications, AI chatbot, and scheduled reports in your Slack workspace
Connect Chamber to your Slack workspace and never miss a beat — get real-time workload notifications, capacity alerts, AI-powered answers, and scheduled utilization reports delivered straight to the channels your team already uses.
## Why Slack?
* **Instant awareness** — Know the moment a workload fails, gets preempted, or completes without watching the dashboard
* **Smart channel routing** — Route different notification types to different channels so the right people see the right alerts
* **Personal control** — Every team member decides what they get and how (channel post or DM)
* **AI chatbot** — Ask Chamber questions in plain English right from Slack and get answers about workloads, GPUs, capacity, and costs
* **Automated reports** — Daily or weekly utilization digests posted to any channel, on your schedule
## Connect Your Workspace
Connecting Slack requires **Org Admin** permissions.
Navigate to **Settings → Integrations** in the Chamber dashboard.
Click the **Connect Slack** button. You'll be redirected to Slack's authorization page.
Review the requested permissions and click **Allow**. Chamber needs access to read channels, post messages, and look up users.
Pick a default channel for notifications. You can override this per notification category later.
Click **Send Test Message** to confirm everything works. You should see a test notification in your selected channel.
## Notification Types
Org Admins control which notification types are active at the organization level. Here's everything Chamber can alert you about.
### Workload Notifications
| Event | Description | Default |
| --------------------------- | --------------------------------------------------------- | ------- |
| **Workload Status Updates** | Notifies when workloads start, complete, or are cancelled | On |
| **Workload Failures** | Alerts when a workload fails with error details | On |
| **Preemption Alerts** | Notifies when elastic workloads are preempted | On |
| **Queue Position Updates** | Updates when a queued workload's position changes | Off |
### Capacity Alerts
| Event | Description | Default |
| -------------------------- | --------------------------------------------------- | ------- |
| **Utilization Alerts** | Warns when GPU utilization is unusually low or high | On |
| **Reservation Expiration** | Notifies before a capacity reservation expires | On |
| **Pool Capacity Warnings** | Alerts when a capacity pool is running low | On |
| **Clawback Opportunities** | Flags reclaimable underutilized reserved capacity | Off |
### Reports
| Event | Description | Default |
| ---------------------------- | ------------------------------------ | ------- |
| **Daily Utilization Report** | Daily summary of GPU usage and costs | Off |
| **Weekly Cost Report** | Weekly cost breakdown with trends | Off |
## Channel Routing
By default, all notifications go to the channel you selected during setup. Override this per category to make sure alerts reach the right audience.
**Available categories:**
* **Workload Notifications** — Workload events (started, completed, failed, preempted, queue changes)
* **Capacity Alerts** — Utilization warnings, reservation expirations, pool capacity, clawback
* **Reports** — Daily and weekly scheduled digests
Navigate to **Settings → Integrations → Slack → Channel Overrides**.
Select a notification category and pick the target channel from the dropdown. Chamber lists all public and private channels it has access to.
Click the remove button next to any override to revert that category back to the default channel.
A common setup: route **Capacity Alerts** to `#infra-alerts` for platform engineers, and **Workload Notifications** to `#ml-workloads` for ML engineers.
## User Linking
Chamber automatically maps your team members to Slack users by matching email addresses. This powers @mention notifications so individuals get pinged directly when their workloads complete or fail.
To trigger a sync:
1. Go to **Settings → Integrations → Slack**
2. Click **Sync Users**
3. Chamber matches emails between your Chamber organization and Slack workspace
After syncing, you'll see a count of linked users. Anyone without a matching email can be linked manually by an admin.
## Personal Notification Preferences
Every team member can customize their own Slack notifications without affecting anyone else.
Navigate to **Settings → Integrations → Slack → My Preferences** to configure:
| Setting | Options | Default |
| -------------------------- | --------------------------------- | ------- |
| **Enabled** | On / Off | On |
| **Delivery Method** | Channel post / Direct message | Channel |
| **Mention Me** | Include @mention in channel posts | On |
| **Workload Failed** | Notify on failures | On |
| **Workload Preempted** | Notify on preemptions | On |
| **Workload Started** | Notify on starts | Off |
| **Workload Completed** | Notify on completions | Off |
| **Workload Submitted** | Notify on submissions | Off |
| **Workload Queued** | Notify on queue entry | Off |
| **Queue Position Changes** | Notify when position changes | Off |
Personal preferences only apply to **your own workloads**. Org-level notifications like capacity alerts and reports are controlled by admins.
## AI Chatbot
The Chamber Slack chatbot lets your team talk to Chamber directly from Slack. Ask questions in natural language and get instant answers about workloads, utilization, capacity, costs, and more.
The AI chatbot must be enabled by an Org Admin. Go to **Settings → Integrations → Slack** and toggle **AI Chat** on.
### How to Use
* **Mention the app** in any channel: `@Chamber what's the GPU utilization for the training team?`
* **Send a DM** to the Chamber app for private queries
* **Reply in threads** to continue a conversation with full context
### Example Queries
* "What workloads are currently running?"
* "Show me GPU utilization for the last 24 hours"
* "Which team has the highest GPU spend this week?"
* "What's the queue depth for the research cluster?"
* "Why did workload `train-resnet-42` fail?"
### What the Chatbot Can Do
* Look up workload status, history, and failure details
* Report on GPU utilization and capacity across clusters
* Surface team and user activity
* Provide cost and spend breakdowns
* Draft and send emails on your behalf (with preview before sending)
* Link you directly to the relevant dashboard page
Thread replies maintain full conversation context — ask follow-ups like "and what about last week?" without repeating yourself.
## Scheduled Team Reports
Set up automated utilization reports for specific teams, posted to any Slack channel on a daily or weekly cadence.
### Create a Report
Go to **Settings → Integrations → Slack → Team Reports**.
Select the team you want to report on.
Choose **Daily** or **Weekly**, set the delivery time, and for weekly reports, pick the day of the week.
Choose which Slack channel receives the report.
Toggle which metrics to include in each report.
### Available Report Metrics
| Metric | Description |
| ------------------------- | ------------------------------------------- |
| **GPU Utilization** | Team GPU usage percentage and trends |
| **Workload Success Rate** | Ratio of completed vs. failed workloads |
| **Cost Breakdown** | GPU spend by type, cluster, and user |
| **Top Users** | Most active team members by GPU usage |
| **Queue Wait Times** | Average and peak queue wait durations |
| **Capacity Trends** | Allocation and usage over the report period |
## Troubleshooting
1. Verify the integration status shows **Connected** in Settings → Integrations
2. Check that the notification type is enabled at the org level
3. Ensure Chamber has been invited to the target channel (for private channels)
4. Click **Send Test Message** to verify the connection
1. Run **Sync Users** to refresh email matching
2. Verify the user's Chamber email matches their Slack email
3. Check the user's personal preferences — **Mention Me** must be enabled
1. Confirm **AI Chat** is toggled on in the Slack integration settings
2. Ensure the user has been linked via email sync
3. The user must be a member of your Chamber organization
4. Check that you're mentioning the app correctly (`@Chamber`) or DMing it directly
1. Check the report status in **Team Reports** for error messages
2. Verify the target channel still exists and Chamber has access
3. Ensure the report is toggled **Enabled**
4. Reports run at the configured UTC time — double-check your timezone settings
# Managing Teams
Source: https://docs.usechamber.io/platform/managing-teams
Create and organize your team hierarchy
Teams represent teams, projects, or business units in Chamber.
![Teams]()
## Key Metrics
| Metric | Description |
| ----------------------------- | ---------------------------------------- |
| **Total Allocated Instances** | GPU instances allocated across all teams |
| **Total Projects** | Projects in your organization |
| **Teams** | Top-level teams |
## Features
* **Hierarchical structure**: Organize teams with parent-child relationships
* **Permission management**: Control access with Owner permissions
* **Instance tracking**: See GPU instance counts per team
* **Expand/collapse**: Navigate the team hierarchy easily
## Team Types
| Type | Description |
| ----------- | -------------------------------- |
| **Team** | Top-level organizational unit |
| **Project** | Sub-team or project under a team |
## Managing Teams
* Click **New Team / Project** to create a new team or project
* Click on a team row to view details and manage members
* Use **Expand All** to see the full hierarchy
## Related Pages
View cluster capacity
Manage organization members
# Settings
Source: https://docs.usechamber.io/platform/settings
Configure API tokens, GPU pricing, labels, and cloud integrations
The Settings area in Chamber is where Org Admins configure organization-wide options — from API authentication and GPU cost rates to label management and cloud provider integrations. Access it from the **Settings** tab in the sidebar.
## API Tokens
API tokens authenticate the Chamber Kubernetes agent and any direct API integrations with your organization. Manage them under **Settings → Security**.
### Create a Token
Give the token a descriptive name (e.g., "production-cluster-agent") and an optional description.
Tokens can expire up to 1 year from creation. If left blank, the token does not expire.
The full token is shown only once. Copy it immediately — you won't be able to view it again.
Treat API tokens like passwords. Revoking a token immediately disconnects any agents or integrations using it.
### Manage Existing Tokens
The token list shows each token's:
* **Name** and description
* **Status** — Active, expired, or revoked
* **Last used** — When the token was last used to authenticate
* **Expiration** — When the token expires (if set)
Sort by any column to quickly find tokens. Click **Revoke** to permanently disable a token.
## GPU Pricing
GPU pricing configuration determines how Chamber calculates costs across the Cost Explorer, dashboard, email digests, and AI insights. Set it up under **Settings → GPU Pricing**.
### Configure Rates
The **GPU Pricing** tab lists every GPU type detected in your fleet. For each type, you can set:
| Field | Description |
| ------------------------ | --------------------------------------------- |
| **On-Demand Price/Hour** | Hourly cost for on-demand GPU instances |
| **Spot Price/Hour** | Hourly cost for spot or preemptible instances |
| **Reserved Price/Hour** | Hourly cost for reserved capacity (optional) |
| **Currency** | Pricing currency (default: USD) |
| **Notes** | Free-text field for internal reference |
Rates apply retroactively to all cost calculations, so historical data in the Cost Explorer updates immediately when you change a rate.
### GPU Type Mappings
The **Type Mappings** tab handles GPU name standardization. Different clusters may report the same hardware under different names (e.g., `NVIDIA A100-SXM4-80GB` vs. `a100`). Type mappings let you consolidate these into a single canonical name so cost and utilization data rolls up correctly.
## Labels
Labels let you tag workloads with custom metadata for filtering, searching, and grouping. By default, all labels support exact-match search. Under **Settings → Labels**, you can promote specific labels to unlock additional capabilities.
### Promoted Labels
Promoting a label key enables:
* **Faceted search** — Filter workloads by label values in the UI
* **Grouping** — Group workloads by label in tables and charts
* **Sorting** — Sort workloads by label values
### Manage the Allowlist
Enter the label key you want to promote (e.g., `team`, `environment`, `model-type`). Keys can be up to 255 characters.
The UI shows your current usage against the limit (default: 20 promoted labels). Plan which labels matter most.
Remove a label from the allowlist to revert it to exact-match-only search.
Promoting a label applies to new workloads going forward. Existing workloads are not affected until they are re-indexed.
## ODCR (AWS Capacity Reservations)
If you use AWS On-Demand Capacity Reservations (ODCRs), Chamber can automatically track reserved instance utilization alongside your capacity pools. Configure this under **Settings → ODCR**.
### How It Works
Chamber assumes a read-only IAM role in your AWS account to describe EC2 instances associated with your ODCRs. This lets Chamber map reserved capacity to your clusters and track utilization accurately.
### Setup
Click **Launch Stack** to open a pre-configured CloudFormation template in your AWS console. The stack creates the IAM role with the minimum required permissions. Once deployed, paste the Role ARN back into Chamber.
Create an IAM role manually with the following configuration:
**Trust Policy** — Allows Chamber's AWS account to assume the role with an external ID:
* Principal: Chamber's AWS account (shown in the UI)
* Condition: External ID matching `chamber-odcr-{environment}`
**Permissions Policy** — Read-only access:
* `ec2:DescribeInstances`
Paste the resulting Role ARN into Chamber.
The IAM role is strictly read-only — Chamber can only describe EC2 instances. It cannot modify, create, or terminate any resources in your AWS account.
# User Management
Source: https://docs.usechamber.io/platform/user-management
Manage users, roles, and permissions
Manage organization members and their access levels in Chamber.
![User Management]()
## Key Metrics
| Metric | Description |
| ----------------- | --------------------------- |
| **Total Members** | Organization members |
| **Active Users** | Currently active users |
| **Pending** | Pending invitation requests |
## Features
* **Search and filter**: Find members by name or email
* **Role management**: Assign Admin or Member roles
* **Invitation tracking**: View pending invitations in the Pending tab
* **Member details**: See when each member joined
## Roles
| Role | Description |
| ---------- | -------------------------------------------------- |
| **Admin** | Full access to manage teams, members, and settings |
| **Member** | Access based on team membership |
## Managing Members
* Navigate to **Settings > Users** to view all members
* Click on a member to view or edit their details
* Use the **Pending** tab to manage invitations
## Related Pages
Organize your team hierarchy
View cluster capacity
# Workloads
Source: https://docs.usechamber.io/platform/workloads
View and manage GPU workloads
The Workloads page lets you track and manage all GPU workloads across your organization.
![Workloads]()
## Key Metrics
| Metric | Description |
| ------------------- | ------------------------------------------ |
| **Running** | Active workloads currently executing |
| **Pending** | Workloads queued for execution |
| **Completed** | Successfully finished workloads |
| **Total Workloads** | All submissions in the selected time range |
## Features
* **Search and filter**: Find workloads by name, status, project, or workload class
* **My Submissions vs Browse All**: Toggle between your workloads and all organization workloads
* **Status tabs**: Quick filter by Running, Pending, or Completed
* **Workload details**: Click any workload to see detailed information, events, and metrics
## Workload States
| State | Description |
| --------- | --------------------------------------- |
| Running | Actively executing on GPU resources |
| Pending | Waiting in queue for available capacity |
| Completed | Finished successfully |
| Failed | Terminated with error |
## Related Pages
Reserved vs elastic workloads
How workloads are scheduled
# Quickstart
Source: https://docs.usechamber.io/quickstart
Get started with Chamber in under 10 minutes
This guide walks you through setting up Chamber for your organization, from requesting access to submitting your first workload.
## Prerequisites
* A Kubernetes cluster with GPU nodes
* `kubectl` access to your cluster
* `helm` v3 installed
Don't have a GPU Kubernetes cluster yet? Use our [Terraform modules](/cluster-deployment/overview) to deploy a GPU-ready cluster on AWS or GCP in \~15 minutes.
## Step 1: Create Your Organization
Contact us at [support@usechamber.io](mailto:support@usechamber.io) to request access to Chamber.
After signing in, you'll be prompted to create an organization. This is your top-level tenant in Chamber.
Navigate to **Settings > Members** to invite colleagues to your organization.
## Step 2: Connect Your Cluster
Install the Chamber agent in your Kubernetes cluster to sync workload state with the control plane.
```bash theme={null}
# Get your cluster token from the Chamber dashboard
# Navigate to Settings > Security > API Tokens -> New Token
# Install the agent
helm install chamber-agent oci://public.ecr.aws/chamber/chamber-agent-chart \
--version 0.8.96 \
--namespace chamber-system \
--create-namespace \
--set saas.url="wss://controlplane-api.usechamber.io/agent" \
--set saas.token="" \
--set saas.clusterId="your-gpu-cluster"
```
Verify the agent is running:
```bash theme={null}
kubectl get pods -n chamber-system -l app.kubernetes.io/name=chamber-agent
```
## Step 3: Create Your First Team
Teams represent teams or projects in your organization. They form a hierarchy for organizing capacity allocation.
In the Chamber dashboard, click **Teams** in the sidebar.
Click **Create Team** and enter:
* **Name**: e.g., "ML Platform"
* **Description**: Brief description of the team/project
Create sub-teams for different teams or projects under your root team.
## Step 4: Allocate Capacity
Reserve GPU capacity for your team from your connected cluster.
Navigate to **Capacity Pools** to see your connected clusters.
Click on a pool, then **Create Reservation**:
* Select the team to receive capacity
* Specify the number of GPUs to reserve
Start with a small reservation and adjust based on utilization metrics. You can always increase or redistribute capacity later.
## Step 5: Submit a Workload
With capacity reserved, your team can now submit GPU workloads.
```yaml Example Workload theme={null}
apiVersion: batch/v1
kind: Job
metadata:
name: training-workload
labels:
chamber.io/team: ml-platform # Links workload to team
chamber.io/workload-class: reserved # Use reserved capacity
spec:
template:
spec:
containers:
- name: trainer
image: your-training-image:latest
resources:
limits:
nvidia.com/gpu: 4
restartPolicy: Never
```
Workloads without the `chamber.io/workload-class` label default to **elastic** and will use idle capacity.
## Next Steps
Learn how to structure your organization hierarchy
Understand reserved vs elastic workloads
Monitor utilization and workload status
# API Reference
Source: https://docs.usechamber.io/sdk/python/api-reference
Complete reference for the Chamber Python SDK
## ChamberClient
The main class for interacting with the Chamber API.
### Initialization
```python theme={null}
from chamber_sdk import ChamberClient
# From environment variable (CHAMBER_TOKEN)
client = ChamberClient()
# From CLI config (~/.chamber/token.json)
client = ChamberClient.from_config()
# Direct token
client = ChamberClient(
token="ch.your-api-token",
organization_id="org-123", # Optional: for multi-org users
api_url="https://custom.api.example.com/v1", # Optional: override endpoint
timeout=60 # Optional: request timeout in seconds
)
```
**Default API URL:** `https://api.usechamber.io/v1`
***
## Auto-Containerize & Run
### run()
Auto-containerize and submit a GPU workload in one call. See the [full guide](/sdk/python/run) for detailed documentation.
```python theme={null}
# Configure registries once
ChamberClient.add_registry("prod", "us-east1-docker.pkg.dev/my-project/prod", set_default=True)
ChamberClient.add_registry("dev", "us-east1-docker.pkg.dev/my-project/dev")
# Submit using default registry
job = client.run("./my-project", gpus=4, team="ml-research")
# Submit to specific registry by name
job = client.run("./my-project", gpus=4, team="ml-research", registry="dev")
# Full URLs still work
job = client.run("./my-project", registry="123456.dkr.ecr.us-east-1.amazonaws.com", team="ml-research")
# Dry run (preview without executing)
result = client.run("./my-project", dry_run=True)
print(result.dockerfile)
print(result.manifest)
```
Requires installation with `pip install chamber-sdk[run]`
**Key Parameters:**
| Parameter | Type | Description |
| ------------- | -------- | ------------------------------------------------------------------- |
| `directory` | str | Path to project directory (required) |
| `gpus` | int | Number of GPUs (default: 1) |
| `gpu_type` | str | GPU type (default: "H100") |
| `team` | str | Team ID (required for submission) |
| `registry` | str | Registry name (e.g., "prod") or URL. Uses default if not specified. |
| `distributed` | str | "auto", "ray", "deepspeed", or "none" |
| `dry_run` | bool | Preview without executing |
| `wait` | bool | Block until workload completes |
| `on_progress` | callable | Progress callback(stage, message) |
**Returns:** `Workload` object (or `DryRunResult` if `dry_run=True`)
See [Auto-Containerize & Run](/sdk/python/run) for the complete parameter reference.
### Registry Management
Static methods for managing container registries. Configuration is persisted to `~/.chamber/config.json`.
#### list\_registries()
List all configured registries.
```python theme={null}
registries = ChamberClient.list_registries()
# {'prod': 'us-east1-docker.pkg.dev/my-project/prod', 'dev': '...'}
```
**Returns:** `dict[str, str]` mapping names to URLs
#### add\_registry()
Add or update a named registry.
```python theme={null}
ChamberClient.add_registry(
"prod",
"us-east1-docker.pkg.dev/my-project/prod",
set_default=True # Optional: also set as default
)
```
**Parameters:**
| Parameter | Type | Description |
| ------------- | ---- | --------------------------------------------- |
| `name` | str | Registry name (e.g., "prod", "dev") |
| `url` | str | Registry URL |
| `set_default` | bool | If True, also set as default (default: False) |
#### set\_default\_registry()
Set the default registry by name.
```python theme={null}
ChamberClient.set_default_registry("dev")
```
**Raises:** `ValueError` if the registry name doesn't exist
#### get\_default\_registry()
Get the current default registry.
```python theme={null}
name, url = ChamberClient.get_default_registry()
# ('prod', 'us-east1-docker.pkg.dev/my-project/prod')
```
**Returns:** `tuple[str, str]` of (name, url) or `None` if not configured
***
## Workload Submission
### submit\_job()
Submit a new GPU workload.
```python theme={null}
job = client.submit_job(
name="training-job",
initiative_id="team-id",
gpu_type="H100",
requested_gpus=8,
job_class=JobClass.RESERVED,
priority=50,
tags={"experiment": "v1", "owner": "ml-team"},
external_id="my-tracking-id"
)
```
**Parameters:**
| Parameter | Type | Required | Description |
| ---------------- | --------- | -------- | ------------------------------------------------- |
| `name` | str | Yes | Human-readable workload name (1-255 chars) |
| `initiative_id` | str | Yes | Team ID |
| `gpu_type` | str | Yes | GPU model (e.g., "H100", "A100") |
| `requested_gpus` | int/float | No | Number of GPUs (optional if using template) |
| `job_class` | JobClass | No | RESERVED or ELASTIC (default: RESERVED) |
| `priority` | int | No | 0-100 (higher = more important) |
| `tags` | dict | No | Key-value pairs for tagging |
| `metadata` | dict | No | Custom metadata dictionary |
| `external_id` | str | No | Your own tracking ID (max 255 chars) |
| `k8s_manifest` | str | No | Custom Kubernetes manifest YAML (max 65536 chars) |
| `template_id` | str | No | Template ID to use for configuration |
| `allocation_id` | str | No | Force specific capacity allocation |
**Returns:** `Workload` object
### Distributed Training
For multi-node distributed training:
```python theme={null}
job = client.submit_job(
name="distributed-training",
initiative_id="team-id",
gpu_type="H100",
requested_gpus=32,
gpus_per_pod=8,
requested_pods=4,
scaling_mode=ScalingMode.GANG, # All-or-nothing scheduling
distribution_mode="auto"
)
```
**Additional Parameters:**
| Parameter | Type | Description |
| ------------------- | ----------- | -------------------------------- |
| `gpus_per_pod` | int | GPUs per pod |
| `requested_pods` | int | Number of pods |
| `scaling_mode` | ScalingMode | GANG or ELASTIC |
| `min_pods` | int | Minimum pods for elastic scaling |
| `distribution_mode` | str | "auto" or "manual" |
***
## Workload Management
### get\_workload()
Retrieve details of a specific workload.
```python theme={null}
job = client.get_workload("workload-id")
print(f"Status: {job.status}")
print(f"GPUs: {job.requested_gpus}")
```
**Returns:** `Workload` object
### list\_workloads()
List workloads with optional filters.
```python theme={null}
response = client.list_workloads(
status=JobStatus.RUNNING,
initiative_id="team-id",
submitted_by="user-123",
limit=50
)
for job in response.items:
print(f"{job.name}: {job.status}")
```
**Parameters:**
| Parameter | Type | Description |
| ------------------ | --------- | ------------------------------------------------ |
| `status` | JobStatus | Filter by status |
| `initiative_id` | str | Filter by team |
| `submitted_by` | str | Filter by user |
| `is_managed` | bool | True for Chamber workloads, False for discovered |
| `submitted_after` | str | ISO8601 datetime filter |
| `submitted_before` | str | ISO8601 datetime filter |
| `sort_order` | str | "asc" or "desc" |
| `limit` | int | Max results (1-100, default: 20) |
| `next_token` | str | Pagination cursor |
**Returns:** `PaginatedResponse` with `items`, `next_token`, `total_count`
### iter\_workloads()
Iterate through all workloads with automatic pagination.
```python theme={null}
for job in client.iter_workloads(status=JobStatus.COMPLETED):
print(f"{job.name}: {job.requested_gpus} GPUs")
```
### cancel\_workload()
Cancel a running or pending workload.
```python theme={null}
cancelled_job = client.cancel_workload("workload-id")
print(f"Status: {cancelled_job.status}") # CANCELLED
```
**Returns:** `Workload` object with updated status
### search\_workloads()
Search workloads with advanced filtering.
```python theme={null}
results = client.search_workloads(
status=["RUNNING", "PENDING"],
gpu_type=["H100"],
priority_min=50,
submitted_from="2024-01-01T00:00:00Z",
query="training",
sort_by="submitted_at",
sort_order="desc",
page_size=25
)
print(f"Found {results.total_count} workloads")
for job in results.items:
print(f"{job.name}: {job.status}")
# Paginate through results
if results.has_more:
next_page = client.search_workloads(cursor=results.next_cursor)
```
**Parameters:**
| Parameter | Type | Description |
| -------------------- | ---------- | ------------------------------------- |
| `status` | list\[str] | Filter by status(es) |
| `job_class` | list\[str] | Filter by workload class(es) |
| `gpu_type` | list\[str] | Filter by GPU type(s) |
| `initiative_id` | list\[str] | Filter by team ID(s) |
| `submitted_by` | list\[str] | Filter by user ID(s) |
| `priority_min` | int | Minimum priority |
| `priority_max` | int | Maximum priority |
| `requested_gpus_min` | int | Minimum GPU count |
| `requested_gpus_max` | int | Maximum GPU count |
| `submitted_from` | str | Start date (ISO8601) |
| `submitted_to` | str | End date (ISO8601) |
| `query` | str | Full-text search query |
| `sort_by` | str | Sort field (default: submitted\_at) |
| `sort_order` | str | "asc" or "desc" (default: desc) |
| `page_size` | int | Results per page (1-100, default: 25) |
| `cursor` | str | Pagination cursor |
**Returns:** `WorkloadSearchResult` with `items`, `total_count`, `has_more`, `next_cursor`
### get\_workload\_aggregations()
Get workload counts grouped by a dimension.
```python theme={null}
agg = client.get_workload_aggregations(
dimension="status",
initiative_id=["team-ml"]
)
print(f"Total: {agg.total}")
for bucket in agg.buckets:
print(f" {bucket.key}: {bucket.count}")
```
**Parameters:**
| Parameter | Type | Description |
| ---------------- | ---------- | ------------------------------------------------------------------------------------------ |
| `dimension` | str | Group by: status, job\_class, gpu\_type, initiative\_id, submitted\_by, capacity\_pool\_id |
| `status` | list\[str] | Filter by status(es) |
| `job_class` | list\[str] | Filter by workload class(es) |
| `initiative_id` | list\[str] | Filter by team ID(s) |
| `submitted_from` | str | Start date (ISO8601) |
| `submitted_to` | str | End date (ISO8601) |
**Returns:** `AggregationResult` with `dimension`, `buckets`, `total`
### wait\_for\_completion()
Block until a workload reaches a terminal status.
```python theme={null}
result = client.wait_for_completion(
workload_id="job-id",
poll_interval=30, # Seconds between checks
timeout=3600 # Max wait time in seconds
)
```
**Returns:** `Workload` object with final status
***
## Metrics and Statistics
### get\_workload\_metrics()
Retrieve GPU metrics for a specific workload.
```python theme={null}
metrics = client.get_workload_metrics(
"workload-id",
time_range="job_lifetime"
)
print(f"GPU Utilization: {metrics.gpu_utilization.avg:.1f}%")
print(f"Memory: {metrics.memory_utilization.avg:.1f}%")
print(f"Temperature: {metrics.temperature.avg:.1f}C")
print(f"Power: {metrics.power_usage.avg:.1f} W")
```
**Parameters:**
| Parameter | Type | Description |
| ------------- | ---------- | ----------------------------------------------- |
| `workload_id` | str | The workload ID |
| `time_range` | str | last\_1h, last\_6h, last\_24h, or job\_lifetime |
| `metrics` | list\[str] | Specific metrics to retrieve |
**Returns:** `WorkloadMetrics` with `gpu_utilization`, `memory_utilization`, `temperature`, `power_usage`
### get\_global\_metrics()
Get aggregated metrics for your organization.
```python theme={null}
metrics = client.get_global_metrics(
time_range="last_24h",
initiative_id="team-ml"
)
print(f"Active workloads: {metrics.active_workloads}")
print(f"Total GPU hours: {metrics.total_gpu_hours}")
print(f"Avg GPU utilization: {metrics.gpu_utilization.avg:.1f}%")
```
**Parameters:**
| Parameter | Type | Description |
| --------------- | ---- | -------------------------------------------------- |
| `time_range` | str | last\_1h, last\_6h, last\_24h, last\_7d, last\_30d |
| `initiative_id` | str | Filter by team |
| `cluster_id` | str | Filter by cluster |
| `group_by` | str | Group by field (e.g., "initiative\_id") |
**Returns:** `GlobalMetrics` with aggregated data
### get\_batch\_workload\_metrics()
Get metrics for multiple workloads at once.
```python theme={null}
data = client.get_batch_workload_metrics(
status=["RUNNING"],
time_range="last_24h",
sort_by="gpu_utilization",
limit=10
)
```
**Returns:** Dict with workload metrics ranked by the specified metric
### get\_workload\_stats()
Get aggregated workload statistics.
```python theme={null}
stats = client.get_workload_stats(
time_range="last_7_days",
initiative_id="team-id"
)
print(f"Total workloads: {stats.total}")
for status, count in stats.by_status.items():
print(f" {status}: {count}")
```
**Returns:** `WorkloadStats` with `total`, `by_status`, `by_job_class`
***
## Teams
### list\_teams()
List all teams accessible to the current user.
```python theme={null}
teams = client.list_teams()
for team in teams:
print(f"{team.name} ({team.id})")
```
**Returns:** List of `Team` objects
### create\_team()
Create a new team.
```python theme={null}
team = client.create_team(
name="ML Research",
description="Machine learning research team",
tags={"department": "engineering"}
)
print(f"Created team: {team.id}")
```
**Parameters:**
| Parameter | Type | Description |
| ------------- | ---- | --------------------------------- |
| `name` | str | Team name (1-255 chars) |
| `description` | str | Team description (max 1000 chars) |
| `parent_id` | str | Parent team ID for sub-teams |
| `tags` | dict | Key-value pairs for tagging |
**Returns:** `Team` object
### get\_team()
Get details of a specific team.
```python theme={null}
team = client.get_team("team-id")
print(f"Team: {team.name}")
print(f"Description: {team.description}")
```
**Returns:** `Team` object
***
## Templates
### list\_templates()
List available workload templates.
```python theme={null}
templates = client.list_templates(
scope="ORGANIZATION",
include_system=True
)
for template in templates:
print(f"{template.name}: {template.gpu_type}")
```
**Parameters:**
| Parameter | Type | Description |
| ---------------- | ---- | ---------------------------------------- |
| `name` | str | Filter by name (partial match) |
| `scope` | str | PERSONAL, PROJECT, or ORGANIZATION |
| `include_system` | bool | Include system templates (default: True) |
**Returns:** List of `Template` objects
### get\_template()
Get details of a specific template.
```python theme={null}
template = client.get_template("template-id")
print(f"Template: {template.name}")
print(f"GPU Type: {template.gpu_type}")
print(f"GPUs: {template.requested_gpus}")
```
**Returns:** `Template` object
***
## Allocations
### list\_allocations()
List capacity allocations for a team.
```python theme={null}
allocations = client.list_allocations(initiative_id="team-id")
for alloc in allocations:
print(f"{alloc.id}: {alloc.allocated_instances} instances ({alloc.status})")
```
**Returns:** List of `Allocation` objects
### create\_allocation()
Create a new capacity allocation.
```python theme={null}
allocation = client.create_allocation(
initiative_id="team-id",
reservation_id="reservation-123",
allocated_instances=4
)
print(f"Allocated: {allocation.id}")
```
**Returns:** `Allocation` object
### get\_allocation()
Get details of a specific allocation.
```python theme={null}
allocation = client.get_allocation("allocation-id")
print(f"Status: {allocation.status}")
print(f"Instances: {allocation.allocated_instances}")
```
**Returns:** `Allocation` object
***
## Capacity
### get\_capacity()
Check available GPU capacity and budget.
```python theme={null}
capacity = client.get_capacity()
# Budget info
print(f"Allocated: {capacity.budget.allocated} GPU-hours")
print(f"Used: {capacity.budget.used} GPU-hours")
print(f"Available: {capacity.budget.available} GPU-hours")
# Capacity pools
for pool in capacity.pools:
print(f"{pool.name} ({pool.gpu_type}):")
print(f" Available: {pool.available_gpus}/{pool.total_gpus} GPUs")
```
### health()
Check API health status.
```python theme={null}
health = client.health()
print(f"Status: {health.status}")
print(f"Version: {health.version}")
```
***
## Enums
### JobStatus
```python theme={null}
from chamber_sdk import JobStatus
JobStatus.PENDING # Submitted, awaiting scheduling
JobStatus.QUEUED # Scheduled, waiting for resources
JobStatus.STARTING # Resources allocated, starting
JobStatus.RUNNING # Currently executing
JobStatus.COMPLETED # Finished successfully
JobStatus.FAILED # Workload failed
JobStatus.PREEMPTED # Preempted (elastic jobs)
JobStatus.CANCELLED # Cancelled by user
```
### JobClass
```python theme={null}
from chamber_sdk import JobClass
JobClass.RESERVED # Guaranteed capacity, non-preemptible
JobClass.ELASTIC # Uses idle capacity, can be preempted
JobClass.DISCOVERED # External workload discovered by Chamber
```
### ScalingMode
```python theme={null}
from chamber_sdk import ScalingMode
ScalingMode.GANG # All-or-nothing scheduling
ScalingMode.ELASTIC # Can scale between min/max pods
```
### AllocationStatus
```python theme={null}
from chamber_sdk import AllocationStatus
AllocationStatus.ACTIVE # Currently active
AllocationStatus.RELEASED # Released by user
AllocationStatus.EXPIRED # Expired
```
***
## Data Models
### Workload
| Field | Type | Description |
| ---------------- | --------- | -------------------------------- |
| `id` | str | Unique workload ID |
| `name` | str | Workload name |
| `status` | JobStatus | Current status |
| `requested_gpus` | float | Number of GPUs |
| `gpu_type` | str | GPU model |
| `job_class` | JobClass | RESERVED, ELASTIC, or DISCOVERED |
| `priority` | int | Priority level |
| `tags` | dict | Key-value tags |
| `metadata` | dict | Custom metadata |
| `template_id` | str | Template ID if used |
| `initiative_id` | str | Team ID |
| `submitted_by` | str | User ID |
| `submitted_at` | datetime | Submission timestamp |
| `started_at` | datetime | Start timestamp |
| `completed_at` | datetime | Completion timestamp |
| `failure_reason` | str | Failure message (if failed) |
### Team
| Field | Type | Description |
| ------------- | -------- | ------------------ |
| `id` | str | Team ID |
| `name` | str | Team name |
| `description` | str | Team description |
| `parent_id` | str | Parent team ID |
| `tags` | dict | Key-value tags |
| `created_at` | datetime | Creation timestamp |
### Template
| Field | Type | Description |
| ---------------- | ---- | ---------------------------------- |
| `id` | str | Template ID |
| `name` | str | Template name |
| `scope` | str | PERSONAL, PROJECT, or ORGANIZATION |
| `description` | str | Template description |
| `gpu_type` | str | GPU model |
| `requested_gpus` | int | Number of GPUs |
| `k8s_manifest` | str | Kubernetes manifest |
| `configuration` | dict | Additional configuration |
### Allocation
| Field | Type | Description |
| --------------------- | ---------------- | ------------------- |
| `id` | str | Allocation ID |
| `initiative_id` | str | Team ID |
| `reservation_id` | str | Reservation ID |
| `allocated_instances` | int | Number of instances |
| `status` | AllocationStatus | Allocation status |
| `created_at` | datetime | Creation timestamp |
***
## Exceptions
All exceptions inherit from `ChamberError`:
```python theme={null}
from chamber_sdk import (
ChamberError,
AuthenticationError,
AuthorizationError,
NotFoundError,
ValidationError,
RateLimitError,
ServerError,
DockerError, # For run() errors
)
from chamber_sdk.run import RegistryAuthError # Specific registry auth failures
try:
job = client.get_workload("invalid-id")
except NotFoundError:
print("Workload not found")
except AuthenticationError:
print("Invalid or expired token")
except RateLimitError as e:
print(f"Rate limited: {e.message}")
except ChamberError as e:
print(f"API error: {e}")
```
| Exception | HTTP Code | Description |
| --------------------- | --------- | ------------------------------------------------------------------------- |
| `AuthenticationError` | 401 | Invalid or expired token |
| `AuthorizationError` | 403 | Insufficient permissions |
| `NotFoundError` | 404 | Resource not found |
| `ValidationError` | 400 | Invalid request parameters |
| `RateLimitError` | 429 | Rate limit exceeded |
| `ServerError` | 5xx | Server-side error |
| `DockerError` | N/A | Docker build/push error (for `run()`) |
| `RegistryAuthError` | N/A | Registry authentication/authorization failure (subclass of `DockerError`) |
### Handling `run()` Errors
```python theme={null}
from chamber_sdk import ChamberClient, DockerError
from chamber_sdk.run import RegistryAuthError
try:
job = client.run("./my-project", registry="...", team="...")
except RegistryAuthError as e:
# Registry authentication failed (401/403 from registry)
# e.registry contains the registry URL
# e.detail contains the error details
print(f"Registry auth failed for {e.registry}: {e.detail}")
except DockerError as e:
# Docker not installed, daemon not running, build failed, or push failed
print(f"Docker error: {e}")
except FileNotFoundError as e:
# No Python entrypoint found in project
print(f"Project error: {e}")
except ValueError as e:
# Missing required parameters (registry, team)
print(f"Configuration error: {e}")
```
# Authentication
Source: https://docs.usechamber.io/sdk/python/authentication
Configure API credentials for the Chamber Python SDK
The Chamber SDK supports multiple authentication methods. Choose the one that best fits your workflow.
## Authentication Methods
Set the `CHAMBER_TOKEN` environment variable:
```bash theme={null}
export CHAMBER_TOKEN="ch.your-api-token"
```
Then initialize the client without arguments:
```python theme={null}
from chamber_sdk import ChamberClient
client = ChamberClient()
```
If you have the [Chamber CLI](/cli/installation) installed and authenticated, the SDK can use its credentials:
```python theme={null}
from chamber_sdk import ChamberClient
client = ChamberClient.from_config()
```
This reads from `~/.chamber/token.json`.
Pass the token directly to the client:
```python theme={null}
from chamber_sdk import ChamberClient
client = ChamberClient(token="ch.your-api-token")
```
Avoid hardcoding tokens in source code. Use environment variables or secret management for production.
## Getting an API Token
1. Log in to the [Chamber Dashboard](https://app.usechamber.io)
2. Navigate to **Settings** > **API Tokens**
3. Click **Create Token** and copy the generated token
API tokens start with the prefix `ch.` followed by a unique identifier.
## Multi-Organization Users
If you belong to multiple organizations, specify which one to use:
```python theme={null}
client = ChamberClient(
token="ch.your-api-token",
organization_id="org-123456"
)
```
## Configuration Options
| Parameter | Description | Default |
| ----------------- | ----------------------------------- | ------------------------------ |
| `token` | API token | From env or config |
| `organization_id` | Organization ID for multi-org users | None |
| `api_url` | Override API endpoint | `https://api.usechamber.io/v1` |
| `timeout` | Request timeout in seconds | 30 |
## Custom API Endpoint
For testing or custom deployments, you can override the default API URL:
```python theme={null}
client = ChamberClient(
token="ch.your-api-token",
api_url="https://custom.api.example.com/v1"
)
```
The default API endpoint is `https://api.usechamber.io/v1`.
## Verifying Authentication
Test your credentials by checking the API health:
```python theme={null}
client = ChamberClient()
health = client.health()
print(f"API Status: {health.status}")
print(f"API Version: {health.version}")
```
## Token Precedence
The SDK looks for authentication tokens in this order:
1. **Direct token parameter** - `ChamberClient(token="...")`
2. **Environment variable** - `CHAMBER_TOKEN`
3. **Config file** - `~/.chamber/token.json`
If no token is found, an `AuthenticationError` is raised.
## Next Steps
With authentication configured, follow the [Quickstart guide](/sdk/python/quickstart) to submit your first workload.
# Installation
Source: https://docs.usechamber.io/sdk/python/installation
Install the Chamber Python SDK
The Chamber Python SDK is currently in early access. Contact [support@usechamber.io](mailto:support@usechamber.io) to get access.
## Next Steps
Once you have access, [configure authentication](/sdk/python/authentication) to connect to the Chamber API.
# Python SDK
Source: https://docs.usechamber.io/sdk/python/overview
Official Python client library for the Chamber GPU platform
The Chamber Python SDK provides a simple, intuitive interface for submitting and managing GPU workloads programmatically.
## Features
Auto-containerize and submit workloads without Docker or Kubernetes expertise
Submit GPU workloads with full control over resources, priority, and scheduling
Track workload status, retrieve GPU metrics, and get workload statistics
Query available GPU capacity, budgets, and manage allocations
Support for multi-node distributed training with gang and elastic scheduling
Manage teams and use workload templates for consistent configurations
## Quick Examples
### One-Liner: Auto-Containerize & Submit
The fastest way to get your training code running on GPUs. No Dockerfile or Kubernetes knowledge required:
```python theme={null}
from chamber_sdk import ChamberClient
client = ChamberClient.from_config()
# Configure registries once (persisted to ~/.chamber/config.json)
ChamberClient.add_registry("prod", "us-east1-docker.pkg.dev/my-project/prod", set_default=True)
ChamberClient.add_registry("dev", "us-east1-docker.pkg.dev/my-project/dev")
# Submit to default registry
job = client.run("./my-project", gpus=4, team="ml-research")
# Or specify a registry by name
job = client.run("./my-project", gpus=4, team="ml-research", registry="dev")
print(f"Submitted: {job.id}")
```
Install with `pip install chamber-sdk[run]` to use this feature. See the [Auto-Containerize & Run guide](/sdk/python/run) for details.
### Standard: Full Control
For production workloads where you need complete control:
```python theme={null}
from chamber_sdk import ChamberClient, JobClass
# Initialize client
client = ChamberClient.from_config()
# Submit a training workload
job = client.submit_job(
name="llm-fine-tuning",
initiative_id="team-ml",
gpu_type="H100",
requested_gpus=8,
job_class=JobClass.RESERVED,
tags={"experiment": "gpt-finetune-v1"}
)
print(f"Submitted workload: {job.id}")
# Wait for completion
result = client.wait_for_completion(job.id, timeout=3600)
print(f"Workload finished with status: {result.status}")
# Get metrics
if result.status.value == "COMPLETED":
metrics = client.get_workload_metrics(job.id)
print(f"GPU utilization: {metrics.gpu_utilization.avg:.1f}%")
```
## API Endpoint
The SDK connects to the Chamber API at `https://api.usechamber.io/v1` by default. You can override this for custom deployments:
```python theme={null}
client = ChamberClient(
token="ch.your-token",
api_url="https://custom.api.example.com/v1"
)
```
## Requirements
* Python 3.8 or higher
* `requests` library (installed automatically)
## Security
The SDK is designed with security as a priority:
* **Input Validation** - All user inputs are validated before being passed to external commands or APIs
* **Credential Protection** - Tokens and credentials are never logged or exposed in error messages
* **Safe Command Execution** - Shell injection is prevented through strict input sanitization
* **Path Traversal Protection** - File operations are restricted to intended directories
Credentials loaded from `~/.chamber/token.json` or environment variables are handled securely and never included in debug output or exception messages.
## Supported Container Registries
Push container images to your preferred registry with automatic authentication:
| Registry | Auto-Auth | Auto-Create Repo |
| ---------------------------- | ---------------- | ---------------- |
| **Google Artifact Registry** | ✅ via gcloud CLI | ✅ |
| **AWS ECR** | ✅ via AWS CLI | ✅ |
Just provide your registry URL—the SDK automatically detects the type and handles authentication. No manual `docker login` required.
## What's New
* **Seamless Multi-Cloud Registries** - Full support for Google Artifact Registry and AWS ECR with auto-authentication
* **Named Registries** - Configure registries once, use by name (e.g., `registry="prod"`)
* **Auto-Containerize & Run** - Submit workloads in one line without Docker/K8s expertise
* **Framework Detection** - Auto-detect PyTorch, TensorFlow, JAX and select optimal base images
* **Distributed Training** - Auto-detect DeepSpeed, Accelerate, Ray and configure appropriately
* **Workload Search** - Advanced filtering with full-text search
* **Aggregations** - Get workload counts by status, GPU type, team, and more
* **Teams API** - List, create, and manage teams
* **Templates** - Use predefined workload templates
## Next Steps
Install the SDK via pip
One-liner workload submission
Configure your API credentials
Complete method and class documentation
# Quickstart
Source: https://docs.usechamber.io/sdk/python/quickstart
Submit your first GPU workload with the Chamber Python SDK
This guide walks you through submitting and monitoring a GPU workload using the Python SDK.
**New to containerization?** Try [`client.run()`](/sdk/python/run) to auto-containerize and submit your training project in one line, no Docker or Kubernetes expertise required.
## Prerequisites
* [Chamber SDK installed](/sdk/python/installation)
* [Authentication configured](/sdk/python/authentication)
* A team ID from your Chamber organization
## Submit a Workload
```python theme={null}
from chamber_sdk import ChamberClient, JobClass, JobStatus
```
```python theme={null}
client = ChamberClient.from_config()
# Or: client = ChamberClient() # Uses CHAMBER_TOKEN env var
```
```python theme={null}
job = client.submit_job(
name="my-training-job",
initiative_id="your-team-id",
gpu_type="H100",
requested_gpus=4,
job_class=JobClass.RESERVED,
priority=50,
tags={
"model": "llama-7b",
"dataset": "custom-v1"
}
)
print(f"Workload ID: {job.id}")
print(f"Status: {job.status}")
```
```python theme={null}
# Poll for updates
job = client.get_workload(job.id)
print(f"Current status: {job.status}")
# Or wait for completion
result = client.wait_for_completion(
job.id,
poll_interval=30, # Check every 30 seconds
timeout=7200 # Timeout after 2 hours
)
if result.status == JobStatus.COMPLETED:
print("Workload completed successfully!")
else:
print(f"Workload ended with status: {result.status}")
```
## Complete Example
```python theme={null}
from chamber_sdk import ChamberClient, JobClass, JobStatus
def run_training():
# Initialize
client = ChamberClient.from_config()
# Check capacity first
capacity = client.get_capacity()
print(f"Available GPU hours: {capacity.budget.available}")
# Submit workload
job = client.submit_job(
name="transformer-training",
initiative_id="team-ml",
gpu_type="H100",
requested_gpus=8,
job_class=JobClass.RESERVED,
priority=75,
tags={
"experiment": "transformer-v2",
"hyperparams": "lr=1e-4,batch=32"
}
)
print(f"Submitted: {job.id}")
# Wait for completion
result = client.wait_for_completion(job.id, timeout=3600)
# Get metrics
if result.status == JobStatus.COMPLETED:
metrics = client.get_workload_metrics(job.id)
print(f"Avg GPU utilization: {metrics.gpu_utilization.avg:.1f}%")
print(f"Peak memory: {metrics.memory_utilization.max:.1f}%")
return result
if __name__ == "__main__":
run_training()
```
## Using Templates
If your organization has workload templates configured, you can use them to simplify workload submission:
```python theme={null}
# List available templates
templates = client.list_templates(scope="ORGANIZATION")
for t in templates:
print(f"{t.name}: {t.gpu_type} x {t.requested_gpus}")
# Submit using a template
job = client.submit_job(
name="templated-job",
initiative_id="team-ml",
gpu_type="H100",
template_id="template-abc123",
tags={"experiment": "v1"}
)
```
## Cancelling a Workload
```python theme={null}
# Cancel a running or pending workload
cancelled = client.cancel_workload(job.id)
print(f"Workload cancelled: {cancelled.status}") # CANCELLED
```
## Workload Classes
| Class | Description | Use Case |
| ---------- | ------------------------------------ | --------------------------------------------- |
| `RESERVED` | Guaranteed capacity, non-preemptible | Production training, time-sensitive workloads |
| `ELASTIC` | Uses idle capacity, can be preempted | Experiments, fault-tolerant workloads |
## Common Parameters
| Parameter | Type | Description |
| ---------------- | --------- | ------------------------------------------ |
| `name` | str | Human-readable workload name (1-255 chars) |
| `initiative_id` | str | Team ID |
| `gpu_type` | str | GPU model (e.g., "H100", "A100") |
| `requested_gpus` | int/float | Number of GPUs (can be fractional) |
| `job_class` | JobClass | RESERVED or ELASTIC (default: RESERVED) |
| `priority` | int | 0-100, higher = more important |
| `tags` | dict | Key-value pairs for organizing workloads |
## Next Steps
One-liner workload submission
Query and filter your workloads
Run multi-node distributed workloads
Complete SDK documentation
# Auto-Containerize & Run
Source: https://docs.usechamber.io/sdk/python/run
Submit GPU workloads in one line without Docker or Kubernetes expertise
**Coming Soon — Fully Managed GPU Infrastructure**
We're building automated infrastructure management across cloud providers so your
research and MLE teams can run GPU workloads directly in their own cloud accounts
without ever thinking about infrastructure. If you'd like early access or want to
learn more, reach out at [support@usechamber.io](mailto:support@usechamber.io).
The `client.run()` method is designed for scientists and ML practitioners who want to submit GPU workloads **without needing expertise in Docker or Kubernetes**. Point it at your training project and let the SDK handle the rest.
**Install with the `run` extra** to use this feature:
```bash theme={null}
pip install chamber-sdk[run]
```
## Why Use `client.run()`?
Traditional GPU workload submission requires:
1. Writing a Dockerfile optimized for your ML framework
2. Choosing the right base image (CUDA version, framework version, etc.)
3. Building and pushing the container image
4. Authenticating to your container registry
5. Writing a Kubernetes manifest
6. Submitting via the API
**With `client.run()`, all of this happens automatically:**
```python theme={null}
from chamber_sdk import ChamberClient
client = ChamberClient.from_config()
# Configure your registries once
ChamberClient.add_registry("prod", "us-east1-docker.pkg.dev/my-project/prod", set_default=True)
ChamberClient.add_registry("dev", "us-east1-docker.pkg.dev/my-project/dev")
# Then submit with just a name
job = client.run(
"./my-training-project",
gpus=4,
gpu_type="H100",
team="ml-research",
registry="prod", # Uses named registry
)
print(f"Submitted: {job.id}")
```
## What It Does Automatically
Analyzes your `requirements.txt` to identify PyTorch, TensorFlow, JAX, or other frameworks and selects the optimal NVIDIA NGC base image.
Creates an optimized Dockerfile with proper CUDA configuration, dependency installation, and entrypoint setup.
Uses `docker buildx build --push` to build and push in a single efficient step. For AWS ECR and Google Artifact Registry, authentication and repository creation are handled automatically. Pulls the `:latest` tag to seed the layer cache for faster rebuilds.
Generates the appropriate manifest (Job or RayJob) with correct GPU resource requests, environment variables, and volume mounts.
Submits the workload to Chamber with proper workload class, priority, and team assignment.
## Supported Container Registries
Chamber automatically handles authentication and repository creation for major cloud registries. Just provide your registry URL and the SDK does the rest.
Full auto-authentication via gcloud CLI. Repositories are created automatically if they don't exist.
Full auto-authentication via AWS CLI. Repositories are created automatically if they don't exist.
| Registry | URL Pattern | Auto-Auth | Auto-Create Repo |
| ---------------------------- | ------------------------------------------ | --------- | ---------------- |
| **Google Artifact Registry** | `{region}-docker.pkg.dev/{project}/{repo}` | ✅ | ✅ |
| **AWS ECR** | `{account}.dkr.ecr.{region}.amazonaws.com` | ✅ | ✅ |
| **Other registries** | Any Docker-compatible registry | Manual | Manual |
## Prerequisites
Docker must be installed and running on your machine. The SDK uses Docker to build and push images.
For Google Artifact Registry. Install and run `gcloud auth login`.
For AWS ECR. Install and run `aws configure`.
**Setup:**
```bash theme={null}
# Install gcloud CLI: https://cloud.google.com/sdk/docs/install
gcloud auth login
```
**Required IAM Permissions:**
* `artifactregistry.repositories.get`
* `artifactregistry.repositories.create`
* `artifactregistry.repositories.uploadArtifacts`
**Setup:**
```bash theme={null}
# Install AWS CLI: https://aws.amazon.com/cli/
aws configure
```
**Required IAM Permissions:**
* `ecr:GetAuthorizationToken`
* `ecr:CreateRepository`
* `ecr:BatchCheckLayerAvailability`
* `ecr:PutImage`
* `ecr:InitiateLayerUpload`
* `ecr:UploadLayerPart`
* `ecr:CompleteLayerUpload`
## Basic Usage
### Minimal Example
```python theme={null}
from chamber_sdk import ChamberClient
client = ChamberClient.from_config()
job = client.run(
"./my-training-project",
gpus=4,
gpu_type="H100",
team="my-team-id",
registry="us-central1-docker.pkg.dev/my-project/ml-images",
)
```
```python theme={null}
from chamber_sdk import ChamberClient
client = ChamberClient.from_config()
job = client.run(
"./my-training-project",
gpus=4,
gpu_type="H100",
team="my-team-id",
registry="123456789012.dkr.ecr.us-east-1.amazonaws.com",
)
```
### With Progress Callbacks
Monitor each stage of the pipeline:
```python theme={null}
def on_progress(stage: str, message: str):
print(f"[{stage}] {message}")
job = client.run(
"./my-project",
gpus=4,
gpu_type="H100",
team="my-team",
registry="us-central1-docker.pkg.dev/my-project/ml-images",
on_progress=on_progress,
)
```
```
[config] Resolving configuration...
[detect] Detecting project...
[detect] Framework: pytorch
[detect] Entrypoint: train.py
[dockerfile] Generating Dockerfile...
[manifest] Generating K8s manifest...
[docker] Authenticating to Google Artifact Registry (us-central1)...
[docker] GAR authentication successful
[docker] Creating repository ml-images...
[docker] Repository ready
[docker] Pulling cache image: us-central1-docker.pkg.dev/my-project/ml-images/my-project:latest
[docker] Cache image pulled successfully
[docker] Build context size: 42.5 MB
[docker] Building and pushing image for linux/amd64 (x86_64)
[docker] Image tag: us-central1-docker.pkg.dev/my-project/ml-images/my-project:a1b2c3d4
[docker] Build and push complete
[submit] Submitting workload...
[submit] Workload submitted: wl_abc123
```
```
[config] Resolving configuration...
[detect] Detecting project...
[detect] Framework: pytorch
[detect] Entrypoint: train.py
[dockerfile] Generating Dockerfile...
[manifest] Generating K8s manifest...
[docker] Authenticating to ECR (us-east-1)...
[docker] ECR authentication successful
[docker] Pulling cache image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-project:latest
[docker] Cache image pulled successfully
[docker] Build context size: 42.5 MB
[docker] Building and pushing image for linux/amd64 (x86_64)
[docker] Image tag: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-project:a1b2c3d4
[docker] Build and push complete
[submit] Submitting workload...
[submit] Workload submitted: wl_abc123
```
### Wait for Completion
Block until the workload finishes:
```python theme={null}
job = client.run(
"./my-project",
gpus=4,
gpu_type="H100",
team="my-team",
registry="123456.dkr.ecr.us-east-1.amazonaws.com",
wait=True,
poll_interval=30,
timeout=7200, # 2 hours
)
print(f"Final status: {job.status}")
```
## Registry Configuration
Configure your container registries once and reference them by name. This makes it easy to switch between dev/staging/prod environments.
### Setting Up Named Registries
Add registries to `~/.chamber/config.json`:
```json theme={null}
{
"registries": {
"prod": "us-east1-docker.pkg.dev/my-project/prod",
"dev": "us-east1-docker.pkg.dev/my-project/dev",
"staging": "us-central1-docker.pkg.dev/my-project/staging",
"ecr": "123456789012.dkr.ecr.us-east-1.amazonaws.com"
},
"default_registry": "prod"
}
```
Or configure programmatically:
```python theme={null}
from chamber_sdk import ChamberClient
# Add registries (persisted to ~/.chamber/config.json)
ChamberClient.add_registry("prod", "us-east1-docker.pkg.dev/my-project/prod", set_default=True)
ChamberClient.add_registry("dev", "us-east1-docker.pkg.dev/my-project/dev")
ChamberClient.add_registry("ecr", "123456789012.dkr.ecr.us-east-1.amazonaws.com")
# List configured registries
print(ChamberClient.list_registries())
# {'prod': 'us-east1-docker.pkg.dev/my-project/prod', 'dev': '...', 'ecr': '...'}
# Check default
print(ChamberClient.get_default_registry())
# ('prod', 'us-east1-docker.pkg.dev/my-project/prod')
# Change default
ChamberClient.set_default_registry("dev")
```
### Using Named Registries
Once configured, use registries by name:
```python theme={null}
# Uses default registry (no registry parameter needed)
job = client.run("./my-project", gpus=4, team="ml-team")
# Use a specific registry by name
job = client.run("./my-project", gpus=4, team="ml-team", registry="dev")
job = client.run("./my-project", gpus=4, team="ml-team", registry="ecr")
# Full URLs still work
job = client.run("./my-project", gpus=4, team="ml-team",
registry="us-west1-docker.pkg.dev/other-project/images")
```
## Configuration File
Create a `.chamber.yaml` file in your project directory to avoid repeating parameters:
```yaml theme={null}
# .chamber.yaml
name: my-training-job
gpus: 4
gpu_type: H100
team: my-team-id
registry: prod # Use named registry, or full URL
# Python entrypoint
entrypoint: train.py
entrypoint_args: --batch-size 32 --epochs 10
# Distributed training mode
distributed: auto # "auto", "ray", "deepspeed", or "none"
# Workload class
job_class: ELASTIC # or "RESERVED"
# Environment variables
env:
CUDA_VISIBLE_DEVICES: "0,1,2,3"
NCCL_DEBUG: INFO
# Forward secrets from local environment
forward_env:
- WANDB_API_KEY
- HF_TOKEN
# Additional packages
extra_pip_packages:
- wandb
- tensorboard
extra_apt_packages:
- ffmpeg
# Build customization
pre_build_commands:
- pip install flash-attn --no-build-isolation
post_build_commands:
- python -c "import torch; print(torch.cuda.is_available())"
# Resource overrides
cpu: "24"
memory: 200Gi
shm_size: 64Gi
# Files to exclude from container
ignore:
- "*.ckpt"
- "wandb/"
- "outputs/"
```
Then submit with minimal arguments:
```python theme={null}
job = client.run("./my-project") # Uses .chamber.yaml settings
```
## Dry Run (Preview Mode)
Preview exactly what will happen without building or submitting:
```python theme={null}
result = client.run("./my-project", dry_run=True)
print("=== Detected Profile ===")
print(f"Framework: {result.profile.framework}")
print(f"Entrypoint: {result.profile.entrypoint}")
print(f"Python version: {result.profile.python_version}")
print("\n=== Generated Dockerfile ===")
print(result.dockerfile)
print("\n=== K8s Manifest ===")
print(result.manifest)
print("\n=== Submit Payload ===")
for key, value in result.submit_payload.items():
if key != "k8s_manifest":
print(f" {key}: {value}")
```
## Save Generated Files
Inspect the generated Dockerfile and manifest:
```python theme={null}
job = client.run(
"./my-project",
gpus=4,
gpu_type="H100",
team="my-team",
registry="123456.dkr.ecr.us-east-1.amazonaws.com",
save_dockerfile=True, # Writes Dockerfile.chamber
save_manifest=True, # Writes manifest.chamber.yaml
)
```
## Framework Detection
The SDK automatically detects your ML framework from `requirements.txt` and selects the optimal base image:
| Framework | Detected From | Base Image |
| ---------- | --------------------- | ---------------------------------------------- |
| PyTorch | `torch`, `pytorch` | `nvcr.io/nvidia/pytorch:24.04-py3` |
| TensorFlow | `tensorflow`, `keras` | `nvcr.io/nvidia/tensorflow:24.04-tf2-py3` |
| JAX | `jax`, `jaxlib` | `nvcr.io/nvidia/jax:24.04-py3` |
| Generic | (fallback) | `nvcr.io/nvidia/cuda:12.4.1-devel-ubuntu22.04` |
Override the base image if needed:
```python theme={null}
job = client.run(
"./my-project",
base_image="nvcr.io/nvidia/pytorch:24.01-py3",
# ... other args
)
```
## Distributed Training
The SDK auto-detects distributed training frameworks and configures the appropriate launch command:
| Framework | Detected From | Launch Command |
| ---------- | ---------------------------- | --------------------------------- |
| DeepSpeed | `deepspeed` in requirements | `deepspeed --num_gpus N train.py` |
| Accelerate | `accelerate` in requirements | `accelerate launch train.py` |
| Ray | `ray` in requirements | Creates RayJob K8s manifest |
| Horovod | `horovod` in requirements | Uses standard launcher |
Force a specific distributed mode:
```python theme={null}
# DeepSpeed
job = client.run(
"./deepspeed-project",
gpus=8,
distributed="deepspeed",
# ...
)
# Ray (creates RayJob for large-scale distributed training)
job = client.run(
"./ray-project",
gpus=32,
distributed="ray",
# ...
)
# Disable distributed detection
job = client.run(
"./single-gpu-project",
gpus=1,
distributed="none",
# ...
)
```
## Using an Existing Dockerfile
Skip Dockerfile generation and use your own:
```python theme={null}
job = client.run(
"./my-project",
dockerfile="./Dockerfile.custom",
# ...
)
```
## Registry Auto-Authentication
The SDK automatically detects your registry type and handles authentication seamlessly. No manual `docker login` required.
When using Google Artifact Registry, the SDK automatically:
1. **Detects** GAR registry URLs (pattern: `{region}-docker.pkg.dev/{project}/{repo}`)
2. **Authenticates** Docker using your gcloud CLI credentials
3. **Creates** the repository if it doesn't exist
```python theme={null}
# GAR authentication happens automatically!
job = client.run(
"./my-project",
registry="us-central1-docker.pkg.dev/my-gcp-project/ml-images",
team="ml-team",
gpus=4,
gpu_type="H100",
)
```
**Supported regions:** All GCP regions (us-central1, us-east1, europe-west1, asia-northeast1, etc.)
Ensure your gcloud CLI is authenticated (`gcloud auth login`) before running.
When using AWS ECR, the SDK automatically:
1. **Detects** ECR registry URLs (pattern: `{account}.dkr.ecr.{region}.amazonaws.com`)
2. **Authenticates** Docker using your AWS CLI credentials
3. **Creates** the ECR repository if it doesn't exist
```python theme={null}
# ECR authentication happens automatically!
job = client.run(
"./my-project",
registry="123456789012.dkr.ecr.us-east-1.amazonaws.com",
team="ml-team",
gpus=4,
gpu_type="H100",
)
```
**Supported regions:** All AWS regions (us-east-1, us-west-2, eu-west-1, ap-northeast-1, etc.)
Ensure your AWS CLI is configured (`aws configure`) before running.
### How It Works
When you call `client.run()`, the SDK:
```mermaid theme={null}
flowchart LR
A[Detect Registry Type] --> B{GAR or ECR?}
B -->|GAR| C[gcloud auth configure-docker]
B -->|ECR| D[aws ecr get-login-password]
C --> E[Create Repo if Needed]
D --> E
E --> F[Build & Push Image]
F --> G[Submit Workload]
```
This means you can switch between registries just by changing the URL—no code changes required.
## All Parameters
| Parameter | Type | Default | Description |
| ----------------- | -------- | -------------- | ------------------------------------------------------------------- |
| `directory` | str | required | Path to project directory |
| `gpus` | int | 1 | Number of GPUs |
| `gpu_type` | str | "H100" | GPU type (H100, A100, L40S, etc.) |
| `team` | str | None | Team ID (required) |
| `name` | str | directory name | Workload name |
| `entrypoint` | str | auto-detect | Python entry point |
| `entrypoint_args` | str | None | CLI arguments for entrypoint |
| `registry` | str | None | Registry name (e.g., "prod") or URL. Uses default if not specified. |
| `base_image` | str | auto-select | Override base Docker image |
| `dockerfile` | str | None | Path to existing Dockerfile |
| `distributed` | str | "auto" | "auto", "ray", "deepspeed", or "none" |
| `job_class` | str | "ELASTIC" | "RESERVED" or "ELASTIC" |
| `env` | dict | None | Environment variables |
| `no_cache` | bool | False | Force rebuild even if image exists |
| `dry_run` | bool | False | Preview without executing |
| `save_dockerfile` | bool | False | Save generated Dockerfile |
| `save_manifest` | bool | False | Save generated K8s manifest |
| `wait` | bool | False | Wait for workload completion |
| `poll_interval` | float | 10.0 | Seconds between status checks |
| `timeout` | float | None | Max wait time in seconds |
| `on_progress` | callable | None | Progress callback(stage, message) |
## Error Handling
```python theme={null}
from chamber_sdk import ChamberClient, DockerError
from chamber_sdk.run import RegistryAuthError
client = ChamberClient.from_config()
try:
job = client.run(
"./my-project",
registry="123456.dkr.ecr.us-east-1.amazonaws.com",
team="my-team",
gpus=4,
gpu_type="H100",
)
except RegistryAuthError as e:
# Registry authentication failed (401/403 from registry)
# Useful for detecting expired ECR tokens (12-hour validity)
print(f"Registry auth failed for {e.registry}")
print(f"Details: {e.detail}")
except DockerError as e:
# Docker not installed, daemon not running, build failed, or push failed
print(f"Docker error: {e}")
except FileNotFoundError as e:
# No Python entrypoint found in project
print(f"Project error: {e}")
except ValueError as e:
# Missing required parameters (registry, team)
print(f"Configuration error: {e}")
```
## Configuration Priority
Settings are resolved in this order (highest priority first):
1. **Function arguments** — `client.run(..., gpus=8)` overrides everything
2. **Project `.chamber.yaml`** — Project-specific settings
3. **Global `~/.chamber/config.json`** — Default registry and other global settings
4. **Auto-detection** — Framework, entrypoint, Python version
### Registry Resolution
The `registry` parameter can be a name or URL:
1. If it looks like a URL (contains `.` or `/`), it's used as-is
2. Otherwise, it's looked up in `~/.chamber/config.json` under `registries`
3. If not specified, the `default_registry` from config is used
## Caching
The SDK computes a content hash of your project and uses it as the image tag. If the image already exists in the registry, the build and push steps are skipped:
```
[docker] Image already exists: 123456.dkr.ecr.us-east-1.amazonaws.com/my-project:a1b2c3d4 (skipping build & push)
```
Force a rebuild with `no_cache=True`:
```python theme={null}
job = client.run(
"./my-project",
no_cache=True, # Always rebuild
# ...
)
```
## Docker Build Optimizations
The SDK automatically applies several optimizations to make builds fast:
| Optimization | What it does |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Single build+push** | Uses `docker buildx build --push` to build and push in one step. This is significantly faster than separate build+push because buildkit pushes layers directly as they complete |
| **BuildKit** | Enabled by default (`DOCKER_BUILDKIT=1`) for parallel build stages and advanced caching |
| **Pip cache mounts** | Uses `--mount=type=cache,target=/root/.cache/pip` so pip packages are cached across builds |
| **Remote layer caching** | Pulls the `:latest` tag before building to seed the layer cache. After a successful build, creates a `:latest` tag alias using `docker buildx imagetools create` (fast manifest aliasing, no layer re-upload) |
| **Content-addressed tags** | Images are tagged with a content hash. If the image already exists, build and push are skipped entirely |
| **Platform targeting** | Explicitly builds for `linux/amd64` to ensure consistent images |
| **Reduced metadata** | Uses `--provenance=false --sbom=false` to skip unnecessary metadata generation |
| **Context size warnings** | Reports build context size and warns if it exceeds 500MB |
**Why single build+push matters:** With Docker Desktop's containerd image store, a separate `docker push` can push ALL manifests from multi-platform base images (e.g., 6 manifests for NVIDIA images), causing each layer to be checked 6 times. The `buildx build --push` approach only pushes what it built — making pushes dramatically faster, especially for large ML images.
### Low-Level Build Functions
For advanced use cases, you can access the optimized build functions directly:
```python theme={null}
from chamber_sdk.run import (
build_and_push_image,
pull_image_for_cache,
create_image_tag,
latest_tag,
compute_context_size,
)
# Check context size before building
size = compute_context_size("./my-project")
print(f"Build context: {size / 1024 / 1024:.1f} MB")
# Pull cache image
cache_from = None
latest = latest_tag("registry.example.com/my-project:abc123")
if pull_image_for_cache(latest):
cache_from = latest
# Build and push in one step
build_and_push_image(
context_dir="./my-project",
tag="registry.example.com/my-project:abc123",
dockerfile_content=dockerfile,
dockerignore_content=dockerignore,
cache_from=cache_from,
on_progress=lambda msg: print(f"[docker] {msg}"),
)
# Create :latest alias (fast, no layer re-upload)
create_image_tag("registry.example.com/my-project:abc123", latest)
```