Monitor an indexer with Prometheus

carbon-prometheus-metrics serves every metric in Carbon's registry (pipeline counters, datasource metrics, and your own) as Prometheus text on http://<host>:9464/metrics. This guide adds it to a pipeline, shows real output from Solana mainnet, and gives a scrape config, queries and alert rules you can copy.

You need: Rust 1.96.1 or newer and a Carbon pipeline. The example uses the public RPC endpoint, so no account is needed. To run the scrape and alert steps you also need Prometheus (any 2.x or 3.x release; tested with 3.5.0).

For what each metric means, see Metrics and the metrics reference.

cargo add carbon-prometheus-metrics@2.0.0

[dependencies]
carbon-core = "2.0.0"
carbon-meteora-dlmm-decoder = "2.0.0"
carbon-prometheus-metrics = "2.0.0"
carbon-rpc-transaction-crawler-datasource = "2.0.0"
env_logger = "0.11.8"
log = "0.4.28"
tokio = { version = "1.48.0", features = ["full"] }

use {
    carbon_core::{
        error::CarbonResult,
        instruction::InstructionProcessorInputType,
        metrics::{Counter, MetricsRegistry},
        pipeline::Pipeline,
        processor::Processor,
    },
    carbon_meteora_dlmm_decoder::{
        instructions::MeteoraDlmmInstruction, MeteoraDlmmDecoder, PROGRAM_ID,
    },
    carbon_prometheus_metrics::{PrometheusMetrics, PrometheusServerConfig},
    carbon_rpc_transaction_crawler_datasource::{ConnectionConfig, Filters, RpcTransactionCrawler},
    std::sync::Arc,
};

// Your own metric, served next to Carbon's on the same endpoint.
static SWAPS_DECODED: Counter = Counter::new(
    "dlmm_swaps_decoded_total",
    "Meteora DLMM swap instructions decoded",
);

#[tokio::main]
async fn main() -> CarbonResult<()> {
    env_logger::init();

    MetricsRegistry::global().register_counter(&SWAPS_DECODED);

    let datasource = RpcTransactionCrawler::new(
        "https://api.mainnet-beta.solana.com".to_string(),
        PROGRAM_ID,
        ConnectionConfig::default(),
        Filters::new(None, None, None),
        None,
    );

    // Serves http://0.0.0.0:9464/metrics while the pipeline runs.
    let prometheus = PrometheusMetrics::with_server(PrometheusServerConfig::default());

    Pipeline::builder()
        .datasource(datasource)
        .metrics(Arc::new(prometheus))
        .instruction(MeteoraDlmmDecoder, SwapCounter)
        .build()?
        .run()
        .await
}

struct SwapCounter;

impl Processor<InstructionProcessorInputType<'_, MeteoraDlmmInstruction>> for SwapCounter {
    async fn process(
        &mut self,
        input: &InstructionProcessorInputType<'_, MeteoraDlmmInstruction>,
    ) -> CarbonResult<()> {
        if let MeteoraDlmmInstruction::Swap { .. } | MeteoraDlmmInstruction::Swap2 { .. } =
            input.decoded_instruction
        {
            SWAPS_DECODED.inc();
        }
        Ok(())
    }
}

let config = PrometheusServerConfig::new()
    .listen_addr("127.0.0.1:9100".parse().unwrap())
    .enable_cors(false)
    .request_timeout_secs(10);

RUST_LOG=info cargo run

[… INFO  carbon_core::pipeline] starting pipeline. num_datasources: 1, num_exporters: 1, num_account_pipes: 0, num_account_deletion_pipes: 0, num_instruction_pipes: 1, num_transaction_pipes: 0
[… INFO  carbon_prometheus_metrics] Prometheus metrics server starting on http://0.0.0.0:9464/metrics

curl -s localhost:9464/metrics

# HELP dlmm_swaps_decoded_total Meteora DLMM swap instructions decoded
# TYPE dlmm_swaps_decoded_total counter
dlmm_swaps_decoded_total 6
# HELP carbon_updates_received_total Total updates pulled from datasources
# TYPE carbon_updates_received_total counter
carbon_updates_received_total 41
# HELP carbon_updates_processed_total Total updates processed by the pipeline
# TYPE carbon_updates_processed_total counter
carbon_updates_processed_total 41
# HELP carbon_updates_successful_total Updates processed without error
# TYPE carbon_updates_successful_total counter
carbon_updates_successful_total 41
# HELP carbon_updates_failed_total Updates that errored during processing
# TYPE carbon_updates_failed_total counter
carbon_updates_failed_total 0
…
# HELP transaction_crawler_transactions_fetched_total Transactions fetched by transaction crawler
# TYPE transaction_crawler_transactions_fetched_total counter
transaction_crawler_transactions_fetched_total 62
# HELP carbon_updates_queued Current number of updates waiting in queue
# TYPE carbon_updates_queued gauge
carbon_updates_queued 0
# HELP carbon_updates_process_time_nanoseconds Time taken to process updates in nanoseconds
# TYPE carbon_updates_process_time_nanoseconds histogram
carbon_updates_process_time_nanoseconds_bucket{le="1000"} 0
carbon_updates_process_time_nanoseconds_bucket{le="10000"} 0
carbon_updates_process_time_nanoseconds_bucket{le="100000"} 12
carbon_updates_process_time_nanoseconds_bucket{le="1000000"} 39
carbon_updates_process_time_nanoseconds_bucket{le="10000000"} 41
…
carbon_updates_process_time_nanoseconds_bucket{le="+Inf"} 41
carbon_updates_process_time_nanoseconds_count 41
carbon_updates_process_time_nanoseconds_sum 9060388
…
transaction_crawler_transaction_fetch_times_milliseconds_bucket{le="5000"} 39
transaction_crawler_transaction_fetch_times_milliseconds_bucket{le="+Inf"} 58
transaction_crawler_transaction_fetch_times_milliseconds_count 58
transaction_crawler_transaction_fetch_times_milliseconds_sum 371211

Scrape it with Prometheus#

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - carbon-alerts.yml

scrape_configs:
  - job_name: carbon
    static_configs:
      - targets: ["localhost:9464"]
        labels:
          indexer: dlmm-swaps

Give each indexer its own indexer label (or its own job) so the queries below can tell them apart. Check the file and start Prometheus:

promtool check config prometheus.yml
prometheus --config.file=prometheus.yml

Checking prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: prometheus.yml is valid prometheus config file syntax

Open http://localhost:9090/targets: the carbon target should be UP. promtool check metrics on the endpoint's output warns that the _milliseconds and _nanoseconds histograms don't use seconds as their base unit. That is a naming lint; Prometheus ingests them normally.

Useful queries#

These are real results from Prometheus 3.5.0 scraping the program above every 5 seconds, after about three minutes:

up{job="carbon"}                                                         1
rate(carbon_updates_processed_total[1m])                                 0.509
histogram_quantile(0.99, rate(carbon_updates_process_time_milliseconds_bucket[2m]))   0.99
carbon_updates_queued                                                    0
rate(carbon_updates_failed_total[1m]) / clamp_min(rate(carbon_updates_processed_total[1m]), 1e-9)   0
increase(dlmm_swaps_decoded_total[2m])                                   15.65

The millisecond histogram records whole milliseconds, truncated, so every update faster than 1 ms lands in the le="1" bucket and quantiles below 1 ms are interpolated. Use the nanosecond histogram for fast processors and the millisecond one for thresholds of 5 ms and above. Both measure the time to run one update through all pipes, including your processors' database writes.

Alert rules#

groups:
  - name: carbon
    rules:
      - alert: CarbonIndexerDown
        expr: up{job="carbon"} == 0
        for: 2m
        annotations:
          summary: "{{ $labels.instance }} metrics endpoint is unreachable"

      - alert: CarbonNoUpdates
        expr: rate(carbon_updates_received_total[5m]) == 0
        for: 5m
        annotations:
          summary: "{{ $labels.indexer }} received no updates for 10 minutes"

      - alert: CarbonProcessingErrors
        expr: |
          rate(carbon_updates_failed_total[5m])
            / clamp_min(rate(carbon_updates_processed_total[5m]), 1e-9) > 0.01
        for: 10m
        annotations:
          summary: "More than 1% of updates fail in {{ $labels.indexer }}"

      - alert: CarbonQueueFilling
        # 1000 is the default channel_buffer_size; use your own value.
        expr: carbon_updates_queued > 0.8 * 1000
        for: 5m
        annotations:
          summary: "{{ $labels.indexer }} is falling behind; streaming datasources drop updates when the queue is full"

      - alert: CarbonSlowUpdates
        expr: |
          histogram_quantile(0.99, rate(carbon_updates_process_time_milliseconds_bucket[5m])) > 250
        for: 10m
        annotations:
          summary: "p99 time per update in {{ $labels.indexer }} is above 250 ms"

promtool check rules carbon-alerts.yml

Checking carbon-alerts.yml
  SUCCESS: 5 rules found

What each alert catches:

• CarbonIndexerDown: the process exited or the port isn't reachable. Carbon's pipeline also returns when every datasource stops, so a crashed datasource ends in this alert.
• CarbonNoUpdates: the endpoint is up but nothing arrives: a dead upstream, a filter that matches nothing, or a datasource that gave up reconnecting. Tune the window to your program's normal traffic; a quiet program can legitimately have minutes without transactions.
• CarbonProcessingErrors: processors or decoders return Err. The pipeline logs each error and continues, so this is often the only signal. See Error handling.
• CarbonQueueFilling: the processors are slower than the datasource. Most streaming datasources (Yellowstone, LaserStream, RPC program and block subscribe) use a non-blocking send and drop updates when the channel is full. Set the threshold from your channel_buffer_size.
• CarbonSlowUpdates: per-update latency, usually your database.

For gaps after reconnects, count missed slots in your own counter from DatasourceDisconnection events; Run Carbon in production has the code.

Common problems#

curl: (7) Failed to connect to localhost port 9464#

The exporter was created with PrometheusMetrics::new(), which starts no server. Use PrometheusMetrics::with_server(PrometheusServerConfig::default()). Projects generated with carbon scaffold -m prometheus use new(); change that line.

Prometheus metrics server error: Address already in use (os error 98)#

Another process (often a previous run of the indexer) holds port 9464. The pipeline keeps running without an endpoint. Stop the other process or set a different listen_addr.

Datasource metrics are missing#

Datasources register their metrics when they start consuming, so they appear a moment after run() begins. A datasource that failed to start never registers them.

text format parsing error … second HELP line for metric name#

A metric was registered more than once. Each register_* call adds it again and the endpoint prints it twice, which the Prometheus text parser rejects. Register your own metrics once at startup, not inside a processor or a datasource's consume. Calling build() on more than one pipeline in a process registers Carbon's pipeline metrics once per call, so run one pipeline per process.

The target is DOWN from another host#

listen_addr is 127.0.0.1, or a firewall blocks the port. Bind 0.0.0.0 (the default) and open TCP 9464 to the Prometheus server only. The endpoint has no authentication.

Next steps#