OTel Integration

The Standalone Agent is in beta. OTLP integration behavior may change.

The Standalone Agent includes a built-in OpenTelemetry (OTLP) gRPC receiver. Any application instrumented with the OpenTelemetry SDK can send metrics directly to the agent, which forwards them to the Chamber dashboard alongside GPU and host metrics. This lets you correlate application-level metrics (training loss, throughput, queue depth, request latency) with infrastructure metrics in a single view.

How It Works

Your application exports OTLP metrics over gRPC to localhost:4317
The agent receives, transforms, and buffers the metrics
Metrics are uploaded to Chamber in batches every 60 seconds
They appear on the Services dashboard alongside infrastructure metrics

Configuration

The OTLP receiver is enabled by default on port 4317. No additional configuration is required for most setups.

Variable	Default	Description
`CHAMBER_OTLP_GRPC_ENABLED`	`true`	Enable or disable the OTLP gRPC receiver
`CHAMBER_OTLP_GRPC_PORT`	`4317`	Port the gRPC server listens on
`CHAMBER_OTLP_GRPC_HOST`	`0.0.0.0`	Bind address for the gRPC server
`CHAMBER_OTLP_METRICS_QUEUE_SIZE`	`10000`	Maximum queued metric batches before dropping

To disable the receiver, set CHAMBER_OTLP_GRPC_ENABLED=false in /etc/chamber/agent.env and restart the agent.

Sending Metrics from Your Application

Point your application’s OTLP exporter to the agent’s gRPC endpoint. Here is an example using the Python OpenTelemetry SDK:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource

# Configure exporter to send to the local Chamber agent
exporter = OTLPMetricExporter(
    endpoint="localhost:4317",
    insecure=True,  # Local connection, no TLS needed
)

reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30000)
provider = MeterProvider(
    resource=Resource.create({"service.name": "my-training-job"}),
    metric_readers=[reader],
)
metrics.set_meter_provider(provider)

meter = metrics.get_meter("my-training-job")

# Create metrics
loss_gauge = meter.create_gauge("training.loss", description="Current training loss")
throughput_counter = meter.create_counter(
    "training.samples.processed",
    unit="1",
    description="Total training samples processed",
)

# Record metrics in your training loop
loss_gauge.set(0.42)
throughput_counter.add(batch_size)

Set service.name in your resource attributes — it appears as the service name label on all metrics in the Chamber dashboard.

Metric Name Transformation

The agent transforms OTLP metric names to a Prometheus-compatible format before uploading:

Transformation	Example
Dots and hyphens become underscores	`training.loss` → `training_loss`
Unit suffixes are appended	`request_duration` (unit: `s`) → `request_duration_seconds`
Monotonic sums get `_total` suffix	`samples.processed` (sum, monotonic) → `samples_processed_total`
Consecutive underscores are collapsed	`my__metric` → `my_metric`

Unit Suffix Mapping

OTel Unit	Suffix
`s`	`_seconds`
`ms`	`_milliseconds`
`By`	`_bytes`
`KBy`	`_kilobytes`
`MBy`	`_megabytes`
`GBy`	`_gigabytes`
`1`	`_ratio`

Histogram Support

OTLP histograms are decomposed into Prometheus-style metrics:

Generated Metric	Description
`{name}_bucket`	Cumulative bucket counts with `le` (less-than-or-equal) labels
`{name}_count`	Total number of observations
`{name}_sum`	Sum of all observed values

Supported Metric Types

OTel Type	Supported
Gauge	Yes
Sum (monotonic and non-monotonic)	Yes
Histogram	Yes
ExponentialHistogram	No
Summary	No

Verifying Metrics Are Flowing

After configuring your application, check the agent logs:

sudo journalctl -u chamber-agent-standalone -f | grep -i otlp

You should see log lines indicating metrics received. Then check the Services tab in the dashboard — your application metrics will appear with the service_name label you configured.

Documentation

OTel Integration

How It Works

Configuration

Sending Metrics from Your Application

Metric Name Transformation

Unit Suffix Mapping

Histogram Support

Supported Metric Types

Verifying Metrics Are Flowing

Next Steps

Metrics Reference

Configuration

​How It Works

​Configuration

​Sending Metrics from Your Application

​Metric Name Transformation

​Unit Suffix Mapping

​Histogram Support

​Supported Metric Types

​Verifying Metrics Are Flowing

​Next Steps

Metrics Reference

Configuration

How It Works

Configuration

Sending Metrics from Your Application

Metric Name Transformation

Unit Suffix Mapping

Histogram Support

Supported Metric Types

Verifying Metrics Are Flowing

Next Steps