# Metrics (Carbon 2.0)

> Every metric Carbon core and each datasource emit, the metrics primitives, and the LogMetrics and PrometheusMetrics exporters.

This page lists every built-in metric in Carbon 2.0.0, the primitives in [`carbon_core::metrics`](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/index.html), and the two exporter crates. For how metrics fit into a pipeline, see [Metrics](https://carbon.sevenlabs.org/concepts/metrics.md); to scrape them, see [Prometheus](https://carbon.sevenlabs.org/guides/prometheus.md).

## How metrics are collected

- Every metric is a `static` [`Counter`](#counter), [`Gauge`](#gauge) or [`Histogram`](#histogram) registered in one process-wide [`MetricsRegistry`](#metricsregistry).
- Metrics have no labels. Each crate prefixes its own names instead (`carbon_`, `yellowstone_grpc_`, `transaction_crawler_` and so on).
- Pipeline metrics are registered by [`PipelineBuilder::build`](https://carbon.sevenlabs.org/reference/pipeline-builder.md#build). With the carbon-core `postgres` feature enabled, `build` also registers the Postgres metrics, whether or not you use a Postgres processor.
- Datasource metrics are registered when the datasource's `consume` starts, so they appear in snapshots only after [`Pipeline::run`](https://carbon.sevenlabs.org/reference/pipeline-builder.md#run) has started that datasource.
- `Pipeline::run` itself exports a snapshot only once, at shutdown. Periodic output comes from the exporter: [`LogMetrics`](#logmetrics) runs its own timer, and [`PrometheusMetrics`](#prometheusmetrics) renders the registry on each scrape.

> **Warning: Registration is not deduplicated.** `MetricsRegistry::register_*` appends without checking for duplicates. Calling `build()` on two pipelines in one process, or running two instances of the same datasource crate, registers the same metrics twice. Snapshots then contain each entry twice, and the Prometheus output repeats the `# TYPE` line for those names, which Prometheus rejects as an invalid scrape. Run one pipeline per process, with at most one instance of each datasource crate, if you scrape with Prometheus.

## Built-in metrics

Histogram buckets are listed as their upper bounds. Each histogram also has an implicit overflow bucket (`+Inf`).

### carbon-core pipeline

Defined in [`pipeline.rs`](https://github.com/sevenlabs-hq/carbon/blob/v2.0.0/crates/core/src/pipeline.rs). Updated by `Pipeline::run` once per update it receives.

| Name                                         | Type      | Unit    | Meaning                                                                                                                                                                                             |
| -------------------------------------------- | --------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `carbon_updates_received_total`              | counter   | updates | Updates taken off the datasource channel.                                                                                                                                                           |
| `carbon_updates_processed_total`             | counter   | updates | Updates the pipeline finished handling, successfully or not.                                                                                                                                        |
| `carbon_updates_successful_total`            | counter   | updates | Updates for which every matching pipe returned `Ok`.                                                                                                                                                |
| `carbon_updates_failed_total`                | counter   | updates | Updates for which a pipe, decoder conversion or processor returned `Err`. The error is logged and the pipeline continues. Pipes after the failing one do not run for that update.                   |
| `carbon_updates_queued`                      | gauge     | updates | Channel length after the last processed update: how many updates are waiting. Compare with [`channel_buffer_size`](https://carbon.sevenlabs.org/reference/pipeline-builder.md#channel_buffer_size). |
| `carbon_account_updates_processed_total`     | counter   | updates | `Update::Account` updates that went through all account pipes without error.                                                                                                                        |
| `carbon_transaction_updates_processed_total` | counter   | updates | `Update::Transaction` updates that went through all instruction and transaction pipes without error.                                                                                                |
| `carbon_account_deletions_processed_total`   | counter   | updates | `Update::AccountDeletion` updates that went through all deletion pipes without error.                                                                                                               |
| `carbon_block_details_processed_total`       | counter   | updates | `Update::BlockDetails` updates that went through all block-details pipes without error.                                                                                                             |
| `carbon_updates_process_time_nanoseconds`    | histogram | ns      | Time to run one update through every pipe. Buckets: 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9.                                                                                                              |
| `carbon_updates_process_time_milliseconds`   | histogram | ms      | Same measurement in whole milliseconds (truncated, so sub-millisecond updates record `0`). Buckets: 1, 5, 10, 25, 50, 100, 250, 500, 1000.                                                          |

### carbon-core `postgres` feature

Defined in [`postgres/processors.rs`](https://github.com/sevenlabs-hq/carbon/blob/v2.0.0/crates/core/src/postgres/processors.rs). Updated by `PostgresAccountProcessor`, `PostgresJsonAccountProcessor`, `PostgresInstructionProcessor` and `PostgresJsonInstructionProcessor` (see the [Postgres reference](https://carbon.sevenlabs.org/reference/postgres.md)). The counter names do not end in `_total`.

| Name                                                 | Type      | Unit | Meaning                                                                                     |
| ---------------------------------------------------- | --------- | ---- | ------------------------------------------------------------------------------------------- |
| `postgres_accounts_upsert_upserted`                  | counter   | rows | Account upserts that succeeded.                                                             |
| `postgres_accounts_upsert_failed`                    | counter   | rows | Account upserts that returned an error.                                                     |
| `postgres_accounts_upsert_duration_milliseconds`     | histogram | ms   | Duration of successful account upserts. Buckets: 1, 5, 10, 25, 50, 100, 250, 500, 1000.     |
| `postgres_instructions_upsert_upserted`              | counter   | rows | Instruction upserts that succeeded.                                                         |
| `postgres_instructions_upsert_failed`                | counter   | rows | Instruction upserts that returned an error.                                                 |
| `postgres_instructions_upsert_duration_milliseconds` | histogram | ms   | Duration of successful instruction upserts. Buckets: 1, 5, 10, 25, 50, 100, 250, 500, 1000. |

### Datasource histogram buckets

Most datasource histograms use one of three bucket sets:

| Set      | Buckets                                              |
| -------- | ---------------------------------------------------- |
| ns       | 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9 (nanoseconds)      |
| ms-short | 1, 5, 10, 25, 50, 100, 250, 500, 1000 (milliseconds) |
| ms-long  | 1, 10, 50, 100, 500, 1000, 5000 (milliseconds)       |

### carbon-yellowstone-grpc-datasource

See [Yellowstone gRPC](https://carbon.sevenlabs.org/datasources/yellowstone-grpc.md). Updates are queued with a non-blocking `try_send`. When the pipeline channel is full, the update is dropped and an error is logged.

| Name                                                         | Type      | Unit        | Meaning                                                                                                               |
| ------------------------------------------------------------ | --------- | ----------- | --------------------------------------------------------------------------------------------------------------------- |
| `yellowstone_grpc_account_updates_received_total`            | counter   | updates     | Account notifications converted to `Update::Account`. Counted even when `try_send` failed and the update was dropped. |
| `yellowstone_grpc_account_deletions_received_total`          | counter   | updates     | Zero-lamport account notifications converted to `Update::AccountDeletion`. Counted even when the send failed.         |
| `yellowstone_grpc_transaction_updates_received_total`        | counter   | updates     | Transactions queued to the pipeline. Not counted when the send failed.                                                |
| `yellowstone_grpc_account_process_time_nanoseconds`          | histogram | ns (ns set) | Time to convert one account notification and attempt the send.                                                        |
| `yellowstone_grpc_account_deletion_process_time_nanoseconds` | histogram | ns (ns set) | Same, for deletions.                                                                                                  |
| `yellowstone_grpc_transaction_process_time_nanoseconds`      | histogram | ns (ns set) | Time to convert one protobuf transaction and queue it.                                                                |

### carbon-helius-laserstream-datasource

See [Helius LaserStream](https://carbon.sevenlabs.org/datasources/helius-laserstream.md). Same behaviour as Yellowstone, including `try_send`.

| Name                                                    | Type      | Unit        | Meaning                                                                                               |
| ------------------------------------------------------- | --------- | ----------- | ----------------------------------------------------------------------------------------------------- |
| `laserstream_account_updates_received_total`            | counter   | updates     | Account notifications converted to `Update::Account`, counted even when the send failed.              |
| `laserstream_account_deletions_received_total`          | counter   | updates     | Zero-lamport notifications converted to `Update::AccountDeletion`, counted even when the send failed. |
| `laserstream_transaction_updates_received_total`        | counter   | updates     | Transactions queued to the pipeline.                                                                  |
| `laserstream_account_process_time_nanoseconds`          | histogram | ns (ns set) | Time to convert one account notification and attempt the send.                                        |
| `laserstream_account_deletion_process_time_nanoseconds` | histogram | ns (ns set) | Same, for deletions.                                                                                  |
| `laserstream_transaction_process_time_nanoseconds`      | histogram | ns (ns set) | Time to convert one transaction and queue it.                                                         |

### carbon-stream-message-datasource

See [Stream message](https://carbon.sevenlabs.org/datasources/stream-message.md). Names use the `agave_grpc_` prefix. Sends wait for channel capacity, so updates are not dropped when the pipeline is slow, and the process-time histograms include that wait.

| Name                                                   | Type      | Unit        | Meaning                                                                       |
| ------------------------------------------------------ | --------- | ----------- | ----------------------------------------------------------------------------- |
| `agave_grpc_account_updates_received_total`            | counter   | updates     | Account messages forwarded as `Update::Account`.                              |
| `agave_grpc_account_deletions_received_total`          | counter   | updates     | Zero-lamport account messages forwarded as `Update::AccountDeletion`.         |
| `agave_grpc_transaction_updates_received_total`        | counter   | updates     | Transactions forwarded to the pipeline.                                       |
| `agave_grpc_account_process_time_nanoseconds`          | histogram | ns (ns set) | Time to forward one account message, including the wait for channel capacity. |
| `agave_grpc_account_deletion_process_time_nanoseconds` | histogram | ns (ns set) | Same, for deletions.                                                          |
| `agave_grpc_transaction_process_time_nanoseconds`      | histogram | ns (ns set) | Same, for transactions.                                                       |

### carbon-helius-atlas-ws-datasource

See [Helius Atlas WebSocket](https://carbon.sevenlabs.org/datasources/helius-atlas-ws.md). Counters are incremented before the non-blocking `try_send`, so they include dropped updates.

| Name                                                        | Type      | Unit        | Meaning                                                                                                                                              |
| ----------------------------------------------------------- | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `helius_atlas_ws_account_updates_received_total`            | counter   | updates     | Account notifications converted to `Update::Account`.                                                                                                |
| `helius_atlas_ws_account_deletions_received_total`          | counter   | updates     | Zero-lamport notifications converted to `Update::AccountDeletion`.                                                                                   |
| `helius_atlas_ws_transaction_updates_received_total`        | counter   | updates     | Transaction notifications converted to `Update::Transaction`.                                                                                        |
| `helius_atlas_ws_account_process_time_nanoseconds`          | histogram | ns (ns set) | Time to convert one account notification.                                                                                                            |
| `helius_atlas_ws_account_deletion_process_time_nanoseconds` | histogram | ns (ns set) | Same, for deletions.                                                                                                                                 |
| `helius_atlas_ws_transaction_process_time_nanoseconds`      | histogram | ns (ns set) | Time to convert one transaction notification.                                                                                                        |
| `helius_atlas_ws_clock_process_time_nanoseconds`            | histogram | ns (ns set) | Time to deserialize one Clock sysvar notification. The datasource watches the Clock sysvar to detect stalls and slot gaps and to trigger reconnects. |

### carbon-rpc-block-subscribe-datasource

See [RPC block subscribe](https://carbon.sevenlabs.org/datasources/rpc-block-subscribe.md). Failed transactions (non-`Ok` status) are skipped and not counted. If `try_send` fails, the remaining transactions of that block are dropped.

| Name                                                   | Type      | Unit         | Meaning                                                                   |
| ------------------------------------------------------ | --------- | ------------ | ------------------------------------------------------------------------- |
| `block_subscribe_transactions_processed_total`         | counter   | transactions | Transactions converted to `Update::Transaction`, counted before the send. |
| `block_subscribe_blocks_received_total`                | counter   | blocks       | Block notifications fully handled.                                        |
| `block_subscribe_transaction_process_time_nanoseconds` | histogram | ns (ns set)  | Time to decode and convert one transaction.                               |
| `block_subscribe_block_process_time_nanoseconds`       | histogram | ns (ns set)  | Time to handle one block notification, all transactions included.         |

### carbon-rpc-block-crawler-datasource

See [RPC block crawler](https://carbon.sevenlabs.org/datasources/rpc-block-crawler.md). Failed transactions are skipped and not counted. If `try_send` fails, the remaining transactions of that block are dropped.

| Name                                                 | Type      | Unit             | Meaning                                                                                                                                                 |
| ---------------------------------------------------- | --------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `block_crawler_blocks_fetched_total`                 | counter   | blocks           | Successful `getBlock` calls.                                                                                                                            |
| `block_crawler_blocks_skipped_total`                 | counter   | blocks           | `getBlock` calls that failed with RPC error `-32004`, `-32007` or `-32009` (skipped slot or block not available). Other errors are logged, not counted. |
| `block_crawler_blocks_received_total`                | counter   | blocks           | Fetched blocks picked up by the processing task.                                                                                                        |
| `block_crawler_blocks_processed_total`               | counter   | blocks           | Blocks whose transactions were all handled.                                                                                                             |
| `block_crawler_transactions_processed_total`         | counter   | transactions     | Transactions converted to `Update::Transaction`, counted before the send.                                                                               |
| `block_crawler_blocks_fetch_times_milliseconds`      | histogram | ms (ms-long set) | Duration of each successful `getBlock` call.                                                                                                            |
| `block_crawler_transaction_process_time_nanoseconds` | histogram | ns (ns set)      | Time to decode and convert one transaction.                                                                                                             |
| `block_crawler_block_process_time_nanoseconds`       | histogram | ns (ns set)      | Time to handle one block, all transactions included.                                                                                                    |

### carbon-rpc-transaction-crawler-datasource

See [RPC transaction crawler](https://carbon.sevenlabs.org/datasources/rpc-transaction-crawler.md).

| Name                                                        | Type      | Unit             | Meaning                                                                                                                  |
| ----------------------------------------------------------- | --------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `transaction_crawler_signatures_fetched_total`              | counter   | signatures       | Signatures returned by `getSignaturesForAddress`, summed over pages.                                                     |
| `transaction_crawler_transactions_fetched_total`            | counter   | transactions     | `getTransaction` requests that completed, including those that still failed after all retries.                           |
| `transaction_crawler_signatures_fetch_times_milliseconds`   | histogram | ms (ms-long set) | Time per `getSignaturesForAddress` page, including handing the signatures to the fetch stage.                            |
| `transaction_crawler_transaction_fetch_times_milliseconds`  | histogram | ms (ms-long set) | Time per successful `getTransaction`, measured from the first attempt, so retries and backoff are included.              |
| `transaction_crawler_transaction_process_time_milliseconds` | histogram | ms (ms-long set) | Time to convert one fetched transaction into an `Update::Transaction`. Failed transactions are skipped and not recorded. |

### carbon-rpc-gpa-datasource

See [RPC getProgramAccounts](https://carbon.sevenlabs.org/datasources/rpc-gpa.md).

| Name                                  | Type      | Unit              | Meaning                                           |
| ------------------------------------- | --------- | ----------------- | ------------------------------------------------- |
| `rpc_gpa_fetch_duration_milliseconds` | histogram | ms (ms-short set) | Duration of the single `getProgramAccounts` call. |
| `rpc_gpa_accounts_processed_total`    | counter   | accounts          | Accounts successfully sent to the pipeline.       |

### carbon-rpc-program-subscribe-datasource

See [RPC program subscribe](https://carbon.sevenlabs.org/datasources/rpc-program-subscribe.md). If `try_send` fails, the subscription loop exits.

| Name                                                 | Type      | Unit        | Meaning                                                                                               |
| ---------------------------------------------------- | --------- | ----------- | ----------------------------------------------------------------------------------------------------- |
| `program_subscribe_accounts_processed_total`         | counter   | accounts    | `programSubscribe` notifications converted to an account update or deletion, counted before the send. |
| `program_subscribe_account_process_time_nanoseconds` | histogram | ns (ns set) | Time to decode and convert one notification.                                                          |

### carbon-helius-gpa-v2-datasource

See [Helius getProgramAccountsV2](https://carbon.sevenlabs.org/datasources/helius-gpa-v2.md).

| Name                                        | Type      | Unit              | Meaning                                     |
| ------------------------------------------- | --------- | ----------------- | ------------------------------------------- |
| `helius_gpa_v2_pages_fetched_total`         | counter   | pages             | Pages fetched from the paginated API.       |
| `helius_gpa_v2_accounts_processed_total`    | counter   | accounts          | Accounts successfully sent to the pipeline. |
| `helius_gpa_v2_fetch_duration_milliseconds` | histogram | ms (ms-short set) | Time to fetch one page.                     |

### carbon-helius-gtfa-datasource

See [Helius getTransactionsForAddress](https://carbon.sevenlabs.org/datasources/helius-gtfa.md).

| Name                                       | Type      | Unit              | Meaning                                         |
| ------------------------------------------ | --------- | ----------------- | ----------------------------------------------- |
| `helius_gtfa_pages_fetched_total`          | counter   | pages             | Pages fetched from the API.                     |
| `helius_gtfa_transactions_processed_total` | counter   | transactions      | Transactions successfully sent to the pipeline. |
| `helius_gtfa_fetch_duration_milliseconds`  | histogram | ms (ms-short set) | Time to fetch one page.                         |

### carbon-validator-snapshot-datasource

See [Validator snapshot](https://carbon.sevenlabs.org/datasources/validator-snapshot.md). Histogram buckets: 1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 10000 ms. A snapshot load usually takes longer than 10 s, so expect it in the `+Inf` bucket.

| Name                                            | Type      | Unit     | Meaning                                                     |
| ----------------------------------------------- | --------- | -------- | ----------------------------------------------------------- |
| `validator_snapshot_load_duration_milliseconds` | histogram | ms       | Time to load the snapshot from a local path or download it. |
| `validator_snapshot_scan_duration_milliseconds` | histogram | ms       | Time to scan every account after loading.                   |
| `validator_snapshot_accounts_processed_total`   | counter   | accounts | Accounts successfully sent to the pipeline.                 |

### carbon-jetstreamer-datasource (Carbon 1.x)

`carbon-jetstreamer-datasource` is published at 1.0.0 on Carbon 1.x and is not compatible with carbon-core 2.0.0. Listed for completeness.

| Name                                          | Type    | Unit         | Meaning                                                   |
| --------------------------------------------- | ------- | ------------ | --------------------------------------------------------- |
| `jetstreamer_blocks_sent_total`               | counter | blocks       | Block details sent to the pipeline.                       |
| `jetstreamer_transactions_sent_total`         | counter | transactions | Transactions sent to the pipeline.                        |
| `jetstreamer_transactions_filtered_in_total`  | counter | transactions | Transactions that matched the datasource filters.         |
| `jetstreamer_transactions_filtered_out_total` | counter | transactions | Transactions that did not match the filters.              |
| `jetstreamer_internal_slots_processed`        | gauge   | slots        | Slots processed by the firehose, from its internal stats. |
| `jetstreamer_internal_blocks_processed`       | gauge   | blocks       | Blocks processed by the firehose.                         |
| `jetstreamer_internal_transactions_processed` | gauge   | transactions | Transactions processed by the firehose.                   |

### carbon-jito-shredstream-grpc-datasource (Carbon 1.x)

Also published at 1.0.0 on Carbon 1.x only.

| Name                                                   | Type      | Unit        | Meaning                               |
| ------------------------------------------------------ | --------- | ----------- | ------------------------------------- |
| `jito_shredstream_grpc_entry_updates_received_total`   | counter   | entries     | Entries received, summed per message. |
| `jito_shredstream_grpc_duplicate_entries_total`        | counter   | entries     | Duplicate entries skipped.            |
| `jito_shredstream_grpc_entry_process_time_nanoseconds` | histogram | ns (ns set) | Time to process one entries message.  |

## `Metric`

Name and help text shared by `Counter`, `Gauge` and `Histogram`.

```rust
pub trait Metric: Send + Sync {
    fn name(&self) -> &'static str;
    fn help(&self) -> &'static str { "" }
}
```

**Details.** Implemented by the three primitives. The registry stores the concrete types, not `dyn Metric`, so implementing `Metric` on your own type does not make it exportable.

**Example.**

```rust
use carbon_core::metrics::{Counter, Metric};

static SEEN: Counter = Counter::new("my_seen_total", "Things seen");
assert_eq!(SEEN.name(), "my_seen_total");
```

**See also.** [`Counter`](#counter), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/trait.Metric.html).

## `Counter`

A monotonically increasing `u64`.

```rust
pub struct Counter { /* private */ }

impl Counter {
    pub const fn new(name: &'static str, help: &'static str) -> Self;
    pub fn inc(&self);
    pub fn inc_by(&self, v: u64);
    pub fn get(&self) -> u64;
}
```

**Details.** `new` is `const`, so you can declare a counter as a plain `static`. `inc` and `inc_by` are a single relaxed atomic add, cheap enough for the hot path. A counter is exported only after you pass it to [`MetricsRegistry::register_counter`](#metricsregistry).

| Method            | Description                                                                                        |
| ----------------- | -------------------------------------------------------------------------------------------------- |
| `new(name, help)` | Creates a counter at 0. Use Prometheus-style `snake_case` names, ending in `_total` by convention. |
| `inc()`           | Adds 1.                                                                                            |
| `inc_by(v)`       | Adds `v`.                                                                                          |
| `get()`           | Returns the current value.                                                                         |

**Example.** See [Register custom metrics](#register-custom-metrics).

**See also.** [`Gauge`](#gauge), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/struct.Counter.html).

## `Gauge`

An `f64` value that can go up and down.

```rust
pub struct Gauge { /* private */ }

impl Gauge {
    pub const fn new(name: &'static str, help: &'static str) -> Self;
    pub fn set(&self, value: f64);
    pub fn add(&self, delta: f64);
    pub fn get(&self) -> f64;
}
```

**Details.** Stored as the bit pattern of an `f64` in an `AtomicU64`. Starts at `0.0`. `add` accepts negative deltas.

| Method            | Description                                             |
| ----------------- | ------------------------------------------------------- |
| `new(name, help)` | Creates a gauge at 0. `const`, so usable in a `static`. |
| `set(value)`      | Replaces the value.                                     |
| `add(delta)`      | Adds `delta` atomically (compare-and-swap loop).        |
| `get()`           | Returns the current value.                              |

**Example.** See [Register custom metrics](#register-custom-metrics).

**See also.** [`Counter`](#counter), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/struct.Gauge.html).

## `Histogram`

A bucketed distribution of `f64` observations with fixed upper bounds.

```rust
pub struct Histogram { /* private */ }

impl Histogram {
    pub fn new(name: &'static str, help: &'static str, boundaries: Vec<f64>) -> Self;
    pub fn record(&self, value: f64);
    pub fn get(&self) -> HistogramSnapshot;
}
```

**Details.** `boundaries` are inclusive upper bounds and must be sorted ascending. The histogram keeps `boundaries.len() + 1` buckets: a value goes into the first bucket whose bound is `>=` the value, and values above the last bound go into the final overflow bucket. `record` also adds the value to a running sum. Bucket counts are per bucket, not cumulative; `PrometheusMetrics` makes them cumulative when it renders them.

`new` allocates, so it is not `const`. Wrap a static histogram in `std::sync::LazyLock`.

| Method                        | Description                                          |
| ----------------------------- | ---------------------------------------------------- |
| `new(name, help, boundaries)` | Creates an empty histogram.                          |
| `record(value)`               | Counts `value` in its bucket and adds it to the sum. |
| `get()`                       | Returns a [`HistogramSnapshot`](#histogramsnapshot). |

**Example.** See [Register custom metrics](#register-custom-metrics).

**See also.** [`HistogramSnapshot`](#histogramsnapshot), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/struct.Histogram.html).

## `HistogramSnapshot`

Point-in-time copy of a histogram.

```rust
pub struct HistogramSnapshot {
    pub counts: Vec<u64>,
    pub boundaries: Vec<f64>,
    pub sum: f64,
}
```

**Details.**

| Field        | Type       | Meaning                                                                                                       |
| ------------ | ---------- | ------------------------------------------------------------------------------------------------------------- |
| `counts`     | `Vec<u64>` | Observations per bucket, not cumulative. Has `boundaries.len() + 1` entries; the last is the overflow bucket. |
| `boundaries` | `Vec<f64>` | Bucket upper bounds, as passed to `Histogram::new`.                                                           |
| `sum`        | `f64`      | Sum of all recorded values.                                                                                   |

The total count is `counts.iter().sum::<u64>()`.

**Example.**

```rust
let h = SWAP_SIZE.get();
let total: u64 = h.counts.iter().sum();
println!("{} observations, mean {}", total, h.sum / total.max(1) as f64);
```

**See also.** [`MetricsSnapshot`](#metricssnapshot), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/struct.HistogramSnapshot.html).

## `MetricsSnapshot`

Point-in-time values of every registered metric, passed to [`MetricsExporter::export`](#metricsexporter).

```rust
pub struct MetricsSnapshot {
    pub counters: Vec<(&'static str, &'static str, u64)>,
    pub gauges: Vec<(&'static str, &'static str, f64)>,
    pub histograms: Vec<(&'static str, &'static str, HistogramSnapshot)>,
}
```

**Details.** Each tuple is `(name, help, value)`. Entries are in registration order. A metric registered twice appears twice.

**Example.**

```rust
let snapshot = carbon_core::metrics::MetricsRegistry::global().snapshot();
let received = snapshot
    .counters
    .iter()
    .find(|(name, _, _)| *name == "carbon_updates_received_total")
    .map(|(_, _, v)| *v)
    .unwrap_or(0);
```

**See also.** [`MetricsRegistry`](#metricsregistry), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/struct.MetricsSnapshot.html).

## `MetricsRegistry`

The process-wide registry that exporters read from.

```rust
pub struct MetricsRegistry { /* private */ }

impl MetricsRegistry {
    pub fn global() -> &'static MetricsRegistry;
    pub fn register_counter(&self, counter: &'static Counter);
    pub fn register_gauge(&self, gauge: &'static Gauge);
    pub fn register_histogram(&self, histogram: &'static Histogram);
    pub fn snapshot(&self) -> MetricsSnapshot;
}
```

**Details.** There is one registry per process, returned by `global()`; you cannot create another. It holds `&'static` references, so metrics must live in `static`s. Registration appends without deduplication (see the [warning above](#how-metrics-are-collected)). There is no way to unregister a metric. The internal locks panic with `metrics counters lock poisoned` (or `gauges`, `histograms`) if a thread panicked while holding them.

**Example.** See [Register custom metrics](#register-custom-metrics).

**See also.** [`MetricsSnapshot`](#metricssnapshot), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/struct.MetricsRegistry.html).

### Register custom metrics

Declare metrics as statics, register them once at startup (before `Pipeline::run`), and update them from your processors. Every exporter picks them up with the built-in metrics.

`src/main.rs`:

```rust
use {
    carbon_core::metrics::{Counter, Gauge, Histogram, MetricsRegistry},
    std::sync::LazyLock,
};

static SWAPS_SEEN: Counter = Counter::new("my_swaps_seen_total", "Swaps decoded by my processor");
static LAST_SLOT: Gauge = Gauge::new("my_last_slot", "Slot of the last processed swap");
static SWAP_SIZE: LazyLock<Histogram> = LazyLock::new(|| {
    Histogram::new(
        "my_swap_amount_in",
        "Swap input amount in lamports",
        vec![1_000.0, 1_000_000.0, 1_000_000_000.0],
    )
});

fn main() {
    let registry = MetricsRegistry::global();
    registry.register_counter(&SWAPS_SEEN);
    registry.register_gauge(&LAST_SLOT);
    registry.register_histogram(&SWAP_SIZE);

    SWAPS_SEEN.inc();
    SWAPS_SEEN.inc_by(2);
    LAST_SLOT.set(312_000_000.0);
    SWAP_SIZE.record(500.0);
    SWAP_SIZE.record(2_500_000.0);

    let snapshot = registry.snapshot();
    for (name, _help, value) in &snapshot.counters {
        println!("counter {name} = {value}");
    }
    for (name, _help, value) in &snapshot.gauges {
        println!("gauge {name} = {value}");
    }
    for (name, _help, h) in &snapshot.histograms {
        println!("histogram {name}: counts={:?} boundaries={:?} sum={}", h.counts, h.boundaries, h.sum);
    }
}
```

Output:

```text
counter my_swaps_seen_total = 3
gauge my_last_slot = 312000000
histogram my_swap_amount_in: counts=[1, 0, 1, 0] boundaries=[1000.0, 1000000.0, 1000000000.0] sum=2500500
```

This program needs only `carbon-core = "2.0.0"`.

## `MetricsExporter`

A sink for metrics snapshots, registered with [`PipelineBuilder::metrics`](https://carbon.sevenlabs.org/reference/pipeline-builder.md#metrics).

```rust
pub trait MetricsExporter: Send + Sync {
    fn initialize(self: Arc<Self>) -> CarbonResult<()> { Ok(()) }
    fn export(&self, snapshot: &MetricsSnapshot) -> CarbonResult<()>;
    fn shutdown(&self) -> CarbonResult<()> { Ok(()) }
}
```

**Details.** `Pipeline::run` calls these methods at fixed points:

| Method       | When the pipeline calls it                                                                        | Notes                                                                                                                                                   |
| ------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `initialize` | Once per exporter, at the start of `run`, inside the Tokio runtime, before any datasource starts. | Spawn periodic work here. Returning `Err` makes `run` return that error.                                                                                |
| `export`     | Once per exporter, at shutdown, with a final snapshot.                                            | The pipeline does not call `export` per update or on a timer. The `carbon-core` doc comment saying it runs "after each processed update" is inaccurate. |
| `shutdown`   | Once per exporter, after the final `export`.                                                      | Stop background tasks here.                                                                                                                             |

The final `export` and `shutdown` run on every shutdown path: Ctrl-C, cancellation through [`datasource_cancellation_token`](https://carbon.sevenlabs.org/reference/pipeline-builder.md#datasource_cancellation_token), and all datasources finishing. An `Err` from either makes `run` return early with that error.

**Example.** An exporter that prints counters every 10 seconds and once more at shutdown:

`src/exporter.rs`:

```rust
use {
    carbon_core::{
        error::CarbonResult,
        metrics::{MetricsExporter, MetricsRegistry, MetricsSnapshot},
    },
    std::{sync::Arc, time::Duration},
};

/// Prints every counter to stdout every 10 seconds and once more at shutdown.
struct StdoutExporter;

impl MetricsExporter for StdoutExporter {
    fn initialize(self: Arc<Self>) -> CarbonResult<()> {
        // The pipeline only calls `export` at shutdown, so run your own loop
        // for periodic exports.
        tokio::spawn(async move {
            let mut interval = tokio::time::interval(Duration::from_secs(10));
            loop {
                interval.tick().await;
                let _ = self.export(&MetricsRegistry::global().snapshot());
            }
        });
        Ok(())
    }

    fn export(&self, snapshot: &MetricsSnapshot) -> CarbonResult<()> {
        for (name, _help, value) in &snapshot.counters {
            println!("{name} {value}");
        }
        Ok(())
    }
}
```

Register it with `.metrics(Arc::new(StdoutExporter))`.

**See also.** [`LogMetrics`](#logmetrics), [`PrometheusMetrics`](#prometheusmetrics), [Metrics](https://carbon.sevenlabs.org/concepts/metrics.md), [docs.rs](https://docs.rs/carbon-core/2.0.0/carbon_core/metrics/trait.MetricsExporter.html).

## `LogMetrics`

An exporter that writes a summary and every metric to the [`log`](https://docs.rs/log) crate at `info` level. Crate: `carbon-log-metrics = "2.0.0"`.

```rust
pub struct LogMetrics { /* private */ }

impl LogMetrics {
    pub fn new() -> Self;
    pub fn new_with_flush_interval(interval_secs: u64) -> Self;
}

impl Default for LogMetrics { /* same as new() */ }
impl MetricsExporter for LogMetrics { /* … */ }
```

**Details.**

| Constructor                                          | Flush interval                                                                                                           |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `LogMetrics::new()` / `LogMetrics::default()`        | 5 seconds                                                                                                                |
| `LogMetrics::new_with_flush_interval(interval_secs)` | `interval_secs` seconds. Must be at least 1: a value of `0` makes the background task panic, and periodic logging stops. |

- `initialize` spawns a Tokio task that logs a snapshot every interval. The first tick fires immediately, so you see one block at startup.
- The pipeline's shutdown `export` logs one last block, then `shutdown` cancels the task.
- Output goes through `log::info!` with target `carbon_log_metrics`. Nothing is printed unless you install a logger (for example `env_logger`) with `info` enabled for that target.
- Each block starts with a summary line: `HH:MM:SS (+time since last flush) | N processed (P%), N successful, N failed (F%), Q in queue`. The processed percentage is `carbon_updates_processed_total` divided by `carbon_updates_received_total` plus `carbon_updates_queued`; the failed percentage is relative to processed.
- Then one line per counter and gauge (`  name: value`) and one per histogram (`  name: count=N`; buckets and sum are not logged).

**Example.** This program runs a pipeline with a datasource that emits three empty blocks and then returns, which closes the channel and ends the pipeline.

`src/main.rs`:

```rust
use {
    async_trait::async_trait,
    carbon_core::{
        datasource::{BlockDetails, Datasource, DatasourceId, Update, UpdateType},
        error::CarbonResult,
        pipeline::Pipeline,
        processor::Processor,
    },
    carbon_log_metrics::LogMetrics,
    std::sync::Arc,
    tokio::sync::mpsc::Sender,
    tokio_util::sync::CancellationToken,
};

/// Emits three empty blocks, then returns, which ends the pipeline.
struct ThreeBlocks;

#[async_trait]
impl Datasource for ThreeBlocks {
    async fn consume(
        &self,
        id: DatasourceId,
        sender: Sender<(Update, DatasourceId)>,
        _cancellation_token: CancellationToken,
    ) -> CarbonResult<()> {
        for slot in 1..=3 {
            let block = BlockDetails {
                slot,
                block_hash: None,
                previous_block_hash: None,
                rewards: None,
                num_reward_partitions: None,
                block_time: None,
                block_height: None,
            };
            let _ = sender.send((Update::BlockDetails(block), id.clone())).await;
        }
        Ok(())
    }

    fn update_types(&self) -> Vec<UpdateType> {
        vec![UpdateType::BlockDetails]
    }
}

struct Noop;

impl Processor<BlockDetails> for Noop {
    async fn process(&mut self, _data: &BlockDetails) -> CarbonResult<()> {
        Ok(())
    }
}

#[tokio::main]
async fn main() -> CarbonResult<()> {
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();

    Pipeline::builder()
        .datasource(ThreeBlocks)
        .block_details(Noop)
        .metrics(Arc::new(LogMetrics::new()))
        .build()?
        .run()
        .await
}
```

`Cargo.toml`:

```toml
[dependencies]
async-trait = "0.1.89"
carbon-core = "2.0.0"
carbon-log-metrics = "2.0.0"
env_logger = "0.11.8"
tokio = { version = "1.48.0", features = ["full"] }
tokio-util = "0.7.17"
```

```sh
cargo run
```

Output:

```text
[2026-11-08T23:53:11Z INFO  carbon_core::pipeline] starting pipeline. num_datasources: 1, num_exporters: 1, num_account_pipes: 0, num_account_deletion_pipes: 0, num_instruction_pipes: 0, num_transaction_pipes: 0
[2026-11-08T23:53:11Z INFO  carbon_core::pipeline] update_receiver closed, shutting down.
[2026-11-08T23:53:11Z INFO  carbon_log_metrics] 00:00:00 (+4.145248ms) | 3 processed (100%), 3 successful, 0 failed (0%), 0 in queue
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_updates_received_total: 3
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_updates_processed_total: 3
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_updates_successful_total: 3
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_updates_failed_total: 0
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_account_updates_processed_total: 0
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_transaction_updates_processed_total: 0
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_account_deletions_processed_total: 0
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_block_details_processed_total: 3
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_updates_queued: 0
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_updates_process_time_nanoseconds: count=3
[2026-11-08T23:53:11Z INFO  carbon_log_metrics]   carbon_updates_process_time_milliseconds: count=3
[2026-11-08T23:53:11Z INFO  carbon_core::pipeline] pipeline shutdown complete.
```

The pipeline ended before the first periodic tick ran, so only the shutdown export appears. With the carbon-core `postgres` feature enabled, the six `postgres_*` metrics are listed too.

**See also.** [`MetricsExporter`](#metricsexporter), [docs.rs](https://docs.rs/carbon-log-metrics/2.0.0/carbon_log_metrics/struct.LogMetrics.html).

## `PrometheusMetrics`

An exporter that serves the registry in the Prometheus text format over HTTP. Crate: `carbon-prometheus-metrics = "2.0.0"`.

```rust
pub struct PrometheusMetrics { /* private */ }

impl PrometheusMetrics {
    pub fn new() -> Self;
    pub fn with_server(config: PrometheusServerConfig) -> Self;              // feature "http-server"
    pub fn start_server(
        config: PrometheusServerConfig,
    ) -> tokio::task::JoinHandle<Result<(), Box<dyn std::error::Error + Send + Sync>>>; // feature "http-server"
    pub fn prometheus_text() -> String;
    pub fn format_snapshot(snapshot: &MetricsSnapshot) -> String;
}

impl Default for PrometheusMetrics { /* same as new() */ }
impl MetricsExporter for PrometheusMetrics { /* … */ }
```

**Details.**

| Item                        | Behaviour                                                                                                                                                                        |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `new()` / `default()`       | An exporter that does nothing: no server is started and `export` is a no-op. Use it only if you serve `prometheus_text()` yourself or call `start_server`.                       |
| `with_server(config)`       | When the pipeline calls `initialize`, spawns an HTTP server on `config.listen_addr`. Bind errors are logged (`Prometheus metrics server error: …`) and do not stop the pipeline. |
| `start_server(config)`      | Spawns the same server immediately on the current Tokio runtime and returns its `JoinHandle`. Independent of any pipeline.                                                       |
| `prometheus_text()`         | Renders a fresh snapshot of the global registry.                                                                                                                                 |
| `format_snapshot(snapshot)` | Renders a given snapshot.                                                                                                                                                        |
| `export`                    | No-op. Values are read live on each scrape, so the shutdown export changes nothing.                                                                                              |
| `shutdown`                  | Default no-op. The server stops on its own when the process receives Ctrl-C; it is not stopped by a cancellation token.                                                          |

The server:

- Serves `GET /metrics` only, with `content-type: text/plain; charset=utf-8; version=0.0.4` and `cache-control: no-cache, no-store, must-revalidate`.
- Applies a request timeout (`408 Request Timeout` after `request_timeout_secs`) and, when `enable_cors` is set, a permissive CORS layer.
- Logs `Prometheus metrics server starting on http://<addr>/metrics` at startup.

Rendering:

- Characters other than ASCII letters, digits, `_` and `:` in names are replaced with `_`. All built-in names are already valid.
- `# HELP` is written when the help text is non-empty, then `# TYPE` (`counter`, `gauge` or `histogram`).
- Histograms are written as cumulative `<name>_bucket{le="…"}` lines, a `le="+Inf"` bucket, `<name>_count` and `<name>_sum`. There are no other labels.

**Example.** Serve metrics on `127.0.0.1:9464` for the lifetime of the pipeline. The datasource emits three blocks and then waits for Ctrl-C.

`src/main.rs`:

```rust
use {
    async_trait::async_trait,
    carbon_core::{
        datasource::{BlockDetails, Datasource, DatasourceId, Update, UpdateType},
        error::CarbonResult,
        pipeline::Pipeline,
        processor::Processor,
    },
    carbon_prometheus_metrics::{PrometheusMetrics, PrometheusServerConfig},
    std::sync::Arc,
    tokio::sync::mpsc::Sender,
    tokio_util::sync::CancellationToken,
};

/// Emits three empty blocks, then waits for shutdown.
struct ThreeBlocks;

#[async_trait]
impl Datasource for ThreeBlocks {
    async fn consume(
        &self,
        id: DatasourceId,
        sender: Sender<(Update, DatasourceId)>,
        cancellation_token: CancellationToken,
    ) -> CarbonResult<()> {
        for slot in 1..=3 {
            let block = BlockDetails {
                slot,
                block_hash: None,
                previous_block_hash: None,
                rewards: None,
                num_reward_partitions: None,
                block_time: None,
                block_height: None,
            };
            let _ = sender.send((Update::BlockDetails(block), id.clone())).await;
        }
        cancellation_token.cancelled().await;
        Ok(())
    }

    fn update_types(&self) -> Vec<UpdateType> {
        vec![UpdateType::BlockDetails]
    }
}

struct Noop;

impl Processor<BlockDetails> for Noop {
    async fn process(&mut self, _data: &BlockDetails) -> CarbonResult<()> {
        Ok(())
    }
}

#[tokio::main]
async fn main() -> CarbonResult<()> {
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();

    let config = PrometheusServerConfig::new().listen_addr("127.0.0.1:9464".parse().unwrap());

    Pipeline::builder()
        .datasource(ThreeBlocks)
        .block_details(Noop)
        .metrics(Arc::new(PrometheusMetrics::with_server(config)))
        .build()?
        .run()
        .await
}
```

Add `carbon-prometheus-metrics = "2.0.0"` to the dependencies from the [`LogMetrics` example](#logmetrics). Start it with `cargo run`, then scrape it from another terminal:

```sh
curl -s http://127.0.0.1:9464/metrics
```

Output:

```text
# HELP carbon_updates_received_total Total updates pulled from datasources
# TYPE carbon_updates_received_total counter
carbon_updates_received_total 3
# HELP carbon_updates_processed_total Total updates processed by the pipeline
# TYPE carbon_updates_processed_total counter
carbon_updates_processed_total 3
…
# HELP carbon_updates_queued Current number of updates waiting in queue
# TYPE carbon_updates_queued gauge
carbon_updates_queued 0
# HELP carbon_updates_process_time_nanoseconds Time taken to process updates in nanoseconds
# TYPE carbon_updates_process_time_nanoseconds histogram
carbon_updates_process_time_nanoseconds_bucket{le="1000"} 2
carbon_updates_process_time_nanoseconds_bucket{le="10000"} 3
carbon_updates_process_time_nanoseconds_bucket{le="100000"} 3
carbon_updates_process_time_nanoseconds_bucket{le="1000000"} 3
carbon_updates_process_time_nanoseconds_bucket{le="10000000"} 3
carbon_updates_process_time_nanoseconds_bucket{le="100000000"} 3
carbon_updates_process_time_nanoseconds_bucket{le="1000000000"} 3
carbon_updates_process_time_nanoseconds_bucket{le="+Inf"} 3
carbon_updates_process_time_nanoseconds_count 3
carbon_updates_process_time_nanoseconds_sum 7030
…
```

Press Ctrl-C to stop both the pipeline and the server.

**See also.** [`PrometheusServerConfig`](#prometheusserverconfig), [Prometheus](https://carbon.sevenlabs.org/guides/prometheus.md), [docs.rs](https://docs.rs/carbon-prometheus-metrics/2.0.0/carbon_prometheus_metrics/struct.PrometheusMetrics.html).

## `PrometheusServerConfig`

Configuration for the `/metrics` HTTP server. Requires the `http-server` feature (on by default).

```rust
#[derive(Debug, Clone)]
pub struct PrometheusServerConfig {
    pub listen_addr: std::net::SocketAddr,
    pub enable_cors: bool,
    pub request_timeout_secs: u64,
}

impl PrometheusServerConfig {
    pub fn new() -> Self;
    pub fn listen_addr(self, addr: std::net::SocketAddr) -> Self;
    pub fn enable_cors(self, enable: bool) -> Self;
    pub fn request_timeout_secs(self, timeout: u64) -> Self;
}
```

**Details.** `new()` and `Default` give these values; each builder method replaces one.

| Field                  | Type         | Default        | Description                                                      |
| ---------------------- | ------------ | -------------- | ---------------------------------------------------------------- |
| `listen_addr`          | `SocketAddr` | `0.0.0.0:9464` | Address and port to bind. The default listens on all interfaces. |
| `enable_cors`          | `bool`       | `true`         | Adds a permissive CORS layer (`access-control-allow-origin: *`). |
| `request_timeout_secs` | `u64`        | `30`           | Requests running longer get `408 Request Timeout`.               |

**Example.** Serve metrics yourself, without a pipeline exporter:

`src/main.rs`:

```rust
use carbon_prometheus_metrics::{PrometheusMetrics, PrometheusServerConfig};

#[tokio::main]
async fn main() {
    // Serve /metrics yourself instead of through the exporter.
    let server = PrometheusMetrics::start_server(
        PrometheusServerConfig::new()
            .listen_addr("0.0.0.0:9100".parse().unwrap())
            .enable_cors(false)
            .request_timeout_secs(10),
    );

    // Or render the exposition text directly, e.g. from your own HTTP handler.
    let text: String = PrometheusMetrics::prometheus_text();
    println!("{text}");

    server.abort();
}
```

**See also.** [`run_metrics_server`](#run_metrics_server), [docs.rs](https://docs.rs/carbon-prometheus-metrics/2.0.0/carbon_prometheus_metrics/struct.PrometheusServerConfig.html).

## `run_metrics_server`

Free functions that run the `/metrics` server on the current task until Ctrl-C. Require the `http-server` feature.

```rust
pub async fn run_metrics_server(
    listen_addr: std::net::SocketAddr,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>>;

pub async fn run_default_metrics_server() -> Result<(), Box<dyn std::error::Error + Send + Sync>>;

pub async fn run_metrics_server_with_config(
    config: PrometheusServerConfig,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>>;
```

**Details.** `run_metrics_server` uses `listen_addr` with the other defaults; `run_default_metrics_server` uses `PrometheusServerConfig::default()` (`0.0.0.0:9464`). They return `Err` only if binding fails. A server error after startup is logged and the function returns `Ok(())`. Each function also returns `Ok(())` after Ctrl-C. `PrometheusMetrics::with_server` and `start_server` call `run_metrics_server_with_config` inside `tokio::spawn`.

**Example.**

```rust
tokio::spawn(carbon_prometheus_metrics::run_metrics_server(
    "127.0.0.1:9464".parse().unwrap(),
));
```

**See also.** [`PrometheusMetrics`](#prometheusmetrics), [docs.rs](https://docs.rs/carbon-prometheus-metrics/2.0.0/carbon_prometheus_metrics/fn.run_metrics_server.html).

---

Carbon 2.0.0 documentation · Web: https://carbon.sevenlabs.org/reference/metrics · Last updated 2026-11-09 · Index of all pages: https://carbon.sevenlabs.org/llms.txt