> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usechamber.io/llms.txt
> Use this file to discover all available pages before exploring further.

# OTel Integration

> Send OpenTelemetry metrics from your applications to Chamber via the Standalone Agent

<Warning>
  The Standalone Agent is in **beta**. OTLP integration behavior may change.
</Warning>

The Standalone Agent includes a built-in OpenTelemetry (OTLP) gRPC receiver. Any application instrumented with the OpenTelemetry SDK can send metrics directly to the agent, which forwards them to the Chamber dashboard alongside GPU and host metrics.

This lets you correlate application-level metrics (training loss, throughput, queue depth, request latency) with infrastructure metrics in a single view.

## How It Works

```mermaid theme={null}
flowchart LR
    App["Your App<br/>(OTel SDK)"] -->|"gRPC :4317"| Agent["Chamber<br/>Standalone Agent"]
    Agent -->|batch upload| Chamber["Chamber<br/>Dashboard"]
```

1. Your application exports OTLP metrics over gRPC to `localhost:4317`
2. The agent receives, transforms, and buffers the metrics
3. Metrics are uploaded to Chamber in batches every 60 seconds
4. They appear on the **Services** dashboard alongside infrastructure metrics

## Configuration

The OTLP receiver is **enabled by default** on port 4317. No additional configuration is required for most setups.

| Variable                          | Default   | Description                                   |
| --------------------------------- | --------- | --------------------------------------------- |
| `CHAMBER_OTLP_GRPC_ENABLED`       | `true`    | Enable or disable the OTLP gRPC receiver      |
| `CHAMBER_OTLP_GRPC_PORT`          | `4317`    | Port the gRPC server listens on               |
| `CHAMBER_OTLP_GRPC_HOST`          | `0.0.0.0` | Bind address for the gRPC server              |
| `CHAMBER_OTLP_METRICS_QUEUE_SIZE` | `10000`   | Maximum queued metric batches before dropping |

To disable the receiver, set `CHAMBER_OTLP_GRPC_ENABLED=false` in `/etc/chamber/agent.env` and restart the agent.

## Sending Metrics from Your Application

Point your application's OTLP exporter to the agent's gRPC endpoint. Here is an example using the Python OpenTelemetry SDK:

```python theme={null}
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource

# Configure exporter to send to the local Chamber agent
exporter = OTLPMetricExporter(
    endpoint="localhost:4317",
    insecure=True,  # Local connection, no TLS needed
)

reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30000)
provider = MeterProvider(
    resource=Resource.create({"service.name": "my-training-job"}),
    metric_readers=[reader],
)
metrics.set_meter_provider(provider)

meter = metrics.get_meter("my-training-job")

# Create metrics
loss_gauge = meter.create_gauge("training.loss", description="Current training loss")
throughput_counter = meter.create_counter(
    "training.samples.processed",
    unit="1",
    description="Total training samples processed",
)

# Record metrics in your training loop
loss_gauge.set(0.42)
throughput_counter.add(batch_size)
```

<Info>
  Set `service.name` in your resource attributes — it appears as the service name label on all metrics in the Chamber dashboard.
</Info>

## Metric Name Transformation

The agent transforms OTLP metric names to a Prometheus-compatible format before uploading:

| Transformation                        | Example                                                          |
| ------------------------------------- | ---------------------------------------------------------------- |
| Dots and hyphens become underscores   | `training.loss` → `training_loss`                                |
| Unit suffixes are appended            | `request_duration` (unit: `s`) → `request_duration_seconds`      |
| Monotonic sums get `_total` suffix    | `samples.processed` (sum, monotonic) → `samples_processed_total` |
| Consecutive underscores are collapsed | `my__metric` → `my_metric`                                       |

### Unit Suffix Mapping

| OTel Unit | Suffix          |
| --------- | --------------- |
| `s`       | `_seconds`      |
| `ms`      | `_milliseconds` |
| `By`      | `_bytes`        |
| `KBy`     | `_kilobytes`    |
| `MBy`     | `_megabytes`    |
| `GBy`     | `_gigabytes`    |
| `1`       | `_ratio`        |

## Histogram Support

OTLP histograms are decomposed into Prometheus-style metrics:

| Generated Metric | Description                                                    |
| ---------------- | -------------------------------------------------------------- |
| `{name}_bucket`  | Cumulative bucket counts with `le` (less-than-or-equal) labels |
| `{name}_count`   | Total number of observations                                   |
| `{name}_sum`     | Sum of all observed values                                     |

## Supported Metric Types

| OTel Type                         | Supported |
| --------------------------------- | --------- |
| Gauge                             | Yes       |
| Sum (monotonic and non-monotonic) | Yes       |
| Histogram                         | Yes       |
| ExponentialHistogram              | No        |
| Summary                           | No        |

## Verifying Metrics Are Flowing

After configuring your application, check the agent logs:

```bash theme={null}
sudo journalctl -u chamber-agent-standalone -f | grep -i otlp
```

You should see log lines indicating metrics received. Then check the **Services** tab in the dashboard — your application metrics will appear with the `service_name` label you configured.

## Next Steps

<CardGroup cols={2}>
  <Card title="Metrics Reference" icon="chart-line" href="/agent/standalone/metrics">
    Full list of infrastructure metrics collected
  </Card>

  <Card title="Configuration" icon="gear" href="/agent/standalone/configuration">
    All environment variables and settings
  </Card>
</CardGroup>