# Monitor an indexer with Prometheus (Carbon 2.0) > Serve Carbon's pipeline and datasource metrics on a Prometheus endpoint, scrape them, and alert on stalls, failures, queue depth and slow updates. `carbon-prometheus-metrics` serves every metric in Carbon's registry (pipeline counters, datasource metrics, and your own) as Prometheus text on `http://:9464/metrics`. This guide adds it to a pipeline, shows real output from Solana mainnet, and gives a scrape config, queries and alert rules you can copy. **You need:** Rust 1.96.1 or newer and a Carbon pipeline. The example uses the public RPC endpoint, so no account is needed. To run the scrape and alert steps you also need [Prometheus](https://prometheus.io/download/) (any 2.x or 3.x release; tested with 3.5.0). For what each metric means, see [Metrics](https://carbon.sevenlabs.org/concepts/metrics.md) and the [metrics reference](https://carbon.sevenlabs.org/reference/metrics.md#built-in-metrics). ### Step 1: Add the crate In an existing Carbon project, or a new one from the [Quickstart](https://carbon.sevenlabs.org/quickstart.md): ```sh cargo add carbon-prometheus-metrics@2.0.0 ``` `Cargo.toml`: ```toml [dependencies] carbon-core = "2.0.0" carbon-meteora-dlmm-decoder = "2.0.0" carbon-prometheus-metrics = "2.0.0" carbon-rpc-transaction-crawler-datasource = "2.0.0" env_logger = "0.11.8" log = "0.4.28" tokio = { version = "1.48.0", features = ["full"] } ``` The crate's default `http-server` feature provides the HTTP endpoint. Keep it on. ### Step 2: Register the exporter with a server Use `PrometheusMetrics::with_server`. `PrometheusMetrics::new()` creates an exporter that **starts no server**, so nothing listens on port 9464. This program indexes Meteora DLMM and also defines its own counter, `dlmm_swaps_decoded_total`. Any `Counter`, `Gauge` or `Histogram` you register in `MetricsRegistry::global()` is served on the same endpoint. `src/main.rs`: ```rust use { carbon_core::{ error::CarbonResult, instruction::InstructionProcessorInputType, metrics::{Counter, MetricsRegistry}, pipeline::Pipeline, processor::Processor, }, carbon_meteora_dlmm_decoder::{ instructions::MeteoraDlmmInstruction, MeteoraDlmmDecoder, PROGRAM_ID, }, carbon_prometheus_metrics::{PrometheusMetrics, PrometheusServerConfig}, carbon_rpc_transaction_crawler_datasource::{ConnectionConfig, Filters, RpcTransactionCrawler}, std::sync::Arc, }; // Your own metric, served next to Carbon's on the same endpoint. static SWAPS_DECODED: Counter = Counter::new( "dlmm_swaps_decoded_total", "Meteora DLMM swap instructions decoded", ); #[tokio::main] async fn main() -> CarbonResult<()> { env_logger::init(); MetricsRegistry::global().register_counter(&SWAPS_DECODED); let datasource = RpcTransactionCrawler::new( "https://api.mainnet-beta.solana.com".to_string(), PROGRAM_ID, ConnectionConfig::default(), Filters::new(None, None, None), None, ); // Serves http://0.0.0.0:9464/metrics while the pipeline runs. let prometheus = PrometheusMetrics::with_server(PrometheusServerConfig::default()); Pipeline::builder() .datasource(datasource) .metrics(Arc::new(prometheus)) .instruction(MeteoraDlmmDecoder, SwapCounter) .build()? .run() .await } struct SwapCounter; impl Processor> for SwapCounter { async fn process( &mut self, input: &InstructionProcessorInputType<'_, MeteoraDlmmInstruction>, ) -> CarbonResult<()> { if let MeteoraDlmmInstruction::Swap { .. } | MeteoraDlmmInstruction::Swap2 { .. } = input.decoded_instruction { SWAPS_DECODED.inc(); } Ok(()) } } ``` `PrometheusServerConfig::default()` listens on `0.0.0.0:9464`, allows any CORS origin and times requests out after 30 seconds. Change them with the builder methods: ```rust let config = PrometheusServerConfig::new() .listen_addr("127.0.0.1:9100".parse().unwrap()) .enable_cors(false) .request_timeout_secs(10); ``` ### Step 3: Run the pipeline ```sh RUST_LOG=info cargo run ``` Output: ```text [… INFO carbon_core::pipeline] starting pipeline. num_datasources: 1, num_exporters: 1, num_account_pipes: 0, num_account_deletion_pipes: 0, num_instruction_pipes: 1, num_transaction_pipes: 0 [… INFO carbon_prometheus_metrics] Prometheus metrics server starting on http://0.0.0.0:9464/metrics ``` The server starts when `run()` begins and stops when the process receives ctrl-C. ### Step 4: Read the endpoint In another terminal: ```sh curl -s localhost:9464/metrics ``` Output after about two minutes on the public RPC (trimmed): Output: ```text # HELP dlmm_swaps_decoded_total Meteora DLMM swap instructions decoded # TYPE dlmm_swaps_decoded_total counter dlmm_swaps_decoded_total 6 # HELP carbon_updates_received_total Total updates pulled from datasources # TYPE carbon_updates_received_total counter carbon_updates_received_total 41 # HELP carbon_updates_processed_total Total updates processed by the pipeline # TYPE carbon_updates_processed_total counter carbon_updates_processed_total 41 # HELP carbon_updates_successful_total Updates processed without error # TYPE carbon_updates_successful_total counter carbon_updates_successful_total 41 # HELP carbon_updates_failed_total Updates that errored during processing # TYPE carbon_updates_failed_total counter carbon_updates_failed_total 0 … # HELP transaction_crawler_transactions_fetched_total Transactions fetched by transaction crawler # TYPE transaction_crawler_transactions_fetched_total counter transaction_crawler_transactions_fetched_total 62 # HELP carbon_updates_queued Current number of updates waiting in queue # TYPE carbon_updates_queued gauge carbon_updates_queued 0 # HELP carbon_updates_process_time_nanoseconds Time taken to process updates in nanoseconds # TYPE carbon_updates_process_time_nanoseconds histogram carbon_updates_process_time_nanoseconds_bucket{le="1000"} 0 carbon_updates_process_time_nanoseconds_bucket{le="10000"} 0 carbon_updates_process_time_nanoseconds_bucket{le="100000"} 12 carbon_updates_process_time_nanoseconds_bucket{le="1000000"} 39 carbon_updates_process_time_nanoseconds_bucket{le="10000000"} 41 … carbon_updates_process_time_nanoseconds_bucket{le="+Inf"} 41 carbon_updates_process_time_nanoseconds_count 41 carbon_updates_process_time_nanoseconds_sum 9060388 … transaction_crawler_transaction_fetch_times_milliseconds_bucket{le="5000"} 39 transaction_crawler_transaction_fetch_times_milliseconds_bucket{le="+Inf"} 58 transaction_crawler_transaction_fetch_times_milliseconds_count 58 transaction_crawler_transaction_fetch_times_milliseconds_sum 371211 ``` Datasource metrics (here `transaction_crawler_*`) appear once the datasource starts. Metrics carry no labels; Prometheus adds `job` and `instance` when it scrapes, and you can add your own in the scrape config. The 19 transaction fetches above 5 seconds are the public endpoint rate-limiting requests. ## Scrape it with Prometheus `prometheus.yml`: ```yaml global: scrape_interval: 15s evaluation_interval: 15s rule_files: - carbon-alerts.yml scrape_configs: - job_name: carbon static_configs: - targets: ["localhost:9464"] labels: indexer: dlmm-swaps ``` Give each indexer its own `indexer` label (or its own job) so the queries below can tell them apart. Check the file and start Prometheus: ```sh promtool check config prometheus.yml prometheus --config.file=prometheus.yml ``` Output: ```text Checking prometheus.yml SUCCESS: 1 rule files found SUCCESS: prometheus.yml is valid prometheus config file syntax ``` Open `http://localhost:9090/targets`: the `carbon` target should be `UP`. `promtool check metrics` on the endpoint's output warns that the `_milliseconds` and `_nanoseconds` histograms don't use seconds as their base unit. That is a naming lint; Prometheus ingests them normally. ## Useful queries | Question | PromQL | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Is the endpoint up? | `up{job="carbon"}` | | Updates per second (throughput) | `rate(carbon_updates_processed_total[5m])` | | Transactions or accounts per second | `rate(carbon_transaction_updates_processed_total[5m])`, `rate(carbon_account_updates_processed_total[5m])` | | Failure ratio | `rate(carbon_updates_failed_total[5m]) / clamp_min(rate(carbon_updates_processed_total[5m]), 1e-9)` | | Queue depth | `carbon_updates_queued` | | p99 time per update, ms | `histogram_quantile(0.99, rate(carbon_updates_process_time_milliseconds_bucket[5m]))` | | p50 time per update, µs | `histogram_quantile(0.5, rate(carbon_updates_process_time_nanoseconds_bucket[5m])) / 1e3` | | Mean time per update, ms | `rate(carbon_updates_process_time_nanoseconds_sum[5m]) / rate(carbon_updates_process_time_nanoseconds_count[5m]) / 1e6` | | Your own counter | `increase(dlmm_swaps_decoded_total[1h])` | | Postgres write failures | `rate(postgres_instructions_upsert_failed[5m])` (these counters have no `_total` suffix) | These are real results from Prometheus 3.5.0 scraping the program above every 5 seconds, after about three minutes: Output: ```text up{job="carbon"} 1 rate(carbon_updates_processed_total[1m]) 0.509 histogram_quantile(0.99, rate(carbon_updates_process_time_milliseconds_bucket[2m])) 0.99 carbon_updates_queued 0 rate(carbon_updates_failed_total[1m]) / clamp_min(rate(carbon_updates_processed_total[1m]), 1e-9) 0 increase(dlmm_swaps_decoded_total[2m]) 15.65 ``` The millisecond histogram records whole milliseconds, truncated, so every update faster than 1 ms lands in the `le="1"` bucket and quantiles below 1 ms are interpolated. Use the nanosecond histogram for fast processors and the millisecond one for thresholds of 5 ms and above. Both measure the time to run one update through **all** pipes, including your processors' database writes. ## Alert rules `carbon-alerts.yml`: ```yaml groups: - name: carbon rules: - alert: CarbonIndexerDown expr: up{job="carbon"} == 0 for: 2m annotations: summary: "{{ $labels.instance }} metrics endpoint is unreachable" - alert: CarbonNoUpdates expr: rate(carbon_updates_received_total[5m]) == 0 for: 5m annotations: summary: "{{ $labels.indexer }} received no updates for 10 minutes" - alert: CarbonProcessingErrors expr: | rate(carbon_updates_failed_total[5m]) / clamp_min(rate(carbon_updates_processed_total[5m]), 1e-9) > 0.01 for: 10m annotations: summary: "More than 1% of updates fail in {{ $labels.indexer }}" - alert: CarbonQueueFilling # 1000 is the default channel_buffer_size; use your own value. expr: carbon_updates_queued > 0.8 * 1000 for: 5m annotations: summary: "{{ $labels.indexer }} is falling behind; streaming datasources drop updates when the queue is full" - alert: CarbonSlowUpdates expr: | histogram_quantile(0.99, rate(carbon_updates_process_time_milliseconds_bucket[5m])) > 250 for: 10m annotations: summary: "p99 time per update in {{ $labels.indexer }} is above 250 ms" ``` ```sh promtool check rules carbon-alerts.yml ``` Output: ```text Checking carbon-alerts.yml SUCCESS: 5 rules found ``` What each alert catches: - **`CarbonIndexerDown`**: the process exited or the port isn't reachable. Carbon's pipeline also returns when every datasource stops, so a crashed datasource ends in this alert. - **`CarbonNoUpdates`**: the endpoint is up but nothing arrives: a dead upstream, a filter that matches nothing, or a datasource that gave up reconnecting. Tune the window to your program's normal traffic; a quiet program can legitimately have minutes without transactions. - **`CarbonProcessingErrors`**: processors or decoders return `Err`. The pipeline logs each error and continues, so this is often the only signal. See [Error handling](https://carbon.sevenlabs.org/concepts/error-handling.md). - **`CarbonQueueFilling`**: the processors are slower than the datasource. Most streaming datasources (Yellowstone, LaserStream, RPC program and block subscribe) use a non-blocking send and **drop** updates when the channel is full. Set the threshold from your [`channel_buffer_size`](https://carbon.sevenlabs.org/reference/pipeline-builder.md#channel_buffer_size). - **`CarbonSlowUpdates`**: per-update latency, usually your database. For gaps after reconnects, count missed slots in your own counter from `DatasourceDisconnection` events; [Run Carbon in production](https://carbon.sevenlabs.org/guides/production.md#detect-gaps-after-reconnects) has the code. ## Common problems ### `curl: (7) Failed to connect to localhost port 9464` The exporter was created with `PrometheusMetrics::new()`, which starts no server. Use `PrometheusMetrics::with_server(PrometheusServerConfig::default())`. Projects generated with `carbon scaffold -m prometheus` use `new()`; change that line. ### `Prometheus metrics server error: Address already in use (os error 98)` Another process (often a previous run of the indexer) holds port 9464. The pipeline keeps running without an endpoint. Stop the other process or set a different `listen_addr`. ### Datasource metrics are missing Datasources register their metrics when they start consuming, so they appear a moment after `run()` begins. A datasource that failed to start never registers them. ### `text format parsing error … second HELP line for metric name` A metric was registered more than once. Each `register_*` call adds it again and the endpoint prints it twice, which the Prometheus text parser rejects. Register your own metrics once at startup, not inside a processor or a datasource's `consume`. Calling `build()` on more than one pipeline in a process registers Carbon's pipeline metrics once per call, so run one pipeline per process. ### The target is `DOWN` from another host `listen_addr` is `127.0.0.1`, or a firewall blocks the port. Bind `0.0.0.0` (the default) and open TCP 9464 to the Prometheus server only. The endpoint has no authentication. ## Next steps - [Metrics](https://carbon.sevenlabs.org/concepts/metrics.md): How the registry and exporters work. - [Metrics reference](https://carbon.sevenlabs.org/reference/metrics.md): Every built-in metric, per datasource. - [Run Carbon in production](https://carbon.sevenlabs.org/guides/production.md): Backpressure, reconnects, shutdown and sizing. --- Carbon 2.0.0 documentation · Web: https://carbon.sevenlabs.org/guides/prometheus · Last updated 2026-11-09 · Index of all pages: https://carbon.sevenlabs.org/llms.txt