Troubleshooting

Problems are grouped by the stage where they show up. Each heading is the symptom or the message you see; the error text quoted under it was produced by Carbon 2.0.0. For the variants of Carbon's own Error enum, see the errors reference.

Installing and compiling#

rustc 1.95.0 is not supported by the following packages#

error: rustc 1.95.0 is not supported by the following packages:
  carbon-core@2.0.0 requires rustc 1.96.1
  carbon-macros@2.0.0 requires rustc 1.96.1
  carbon-proc-macros@2.0.0 requires rustc 1.96.1

Carbon 2.0 sets rust-version = "1.96.1". Older Cargo versions word the same error as cannot be built because it requires rustc 1.96.1 or newer. Update the toolchain and build again:

rustup update stable
rustc --version

If the project has a rust-toolchain.toml that pins an older version, raise it to at least 1.96.1.

cargo add adds Carbon 1.0.0 instead of 2.0.0#

On a toolchain older than 1.96.1, Cargo's resolver picks the newest version your compiler supports, and says so in passing:

     Locking 276 packages to latest Rust 1.95.0 compatible versions
      Adding carbon-core v1.0.0 (available: v2.0.0, requires Rust 1.96.1)

You end up on Carbon 1 without an error, and the code on this site doesn't compile against it. Update Rust, then pin the version when you add crates: cargo add carbon-core@2.0.0. Check what resolved with cargo tree -i carbon-core.

InstructionDecoder is not implemented for a decoder#

error[E0277]: the trait bound `for<'a> MeteoraDlmmDecoder: InstructionDecoder<'a>` is not satisfied
...
note: there are multiple different versions of crate `carbon_core` in the dependency graph
   --> ~/.cargo/registry/src/index.crates.io-…/carbon-core-2.0.0/src/instruction.rs:187:1
    |
187 | pub trait InstructionDecoder<'a> {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this is the expected trait
    |
   ::: ~/.cargo/registry/src/index.crates.io-…/carbon-core-1.0.0/src/instruction.rs:188:1
    |
188 | pub trait InstructionDecoder<'a> {
    | -------------------------------- this is the found trait

A Carbon 1 crate is in the same build as Carbon 2: here carbon-meteora-dlmm-decoder = "1" next to carbon-core = "2". Each pulls its own carbon-core, and their traits are unrelated types. The same happens with Datasource or AccountDecoder. Set every carbon-* dependency to 2.0.0 and run cargo update. carbon-jetstreamer-datasource and carbon-jito-shredstream-grpc-datasource have no 2.0 release, so they can't be used with Carbon 2 at all. See Installation.

expected Address, found Pubkey#

error[E0308]: mismatched types
   --> src/main.rs:20:9
    |
 18 |     let datasource = RpcTransactionCrawler::new(
    |                      -------------------------- arguments to this function are incorrect
 19 |         "https://api.mainnet-beta.solana.com".to_string(),
 20 |         solana_pubkey::Pubkey::from_str_const("LBUZKhRxPF3XUpBCjp4YzTKgLccjZhTSDM9YuVaPwxo"),
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Address`, found `Pubkey`

Your crate depends on a Solana crate from an older cohort (here solana-pubkey = "2"; solana-sdk 2.x does the same) and passes its type to Carbon, which uses the Agave 4.2 cohort. Variants of this error read expected Pubkey, found Pubkey or mention Signature, Hash or VersionedTransaction. Use the versions Carbon's workspace pins, for example solana-pubkey = "4.2.1", solana-signature = "3.4.1", solana-client = "=4.2.2", solana-transaction-status = "=4.2.2" and solana-commitment-config = "3.1.1", or use the re-exported types from the Carbon crate you're calling. Find the duplicates with:

cargo tree --duplicates --edges normal | grep solana

The full list of pinned versions is in the workspace Cargo.toml.

Build fails in clang-sys: couldn't find any valid shared libraries#

error: failed to run custom build command for `clang-sys v1.8.1`
...
  called `Result::unwrap()` on an `Err` value: "couldn't find any valid shared libraries matching: ['libclang.so', 'libclang-*.so'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"

carbon-validator-snapshot-datasource depends on solana-ledger, which builds RocksDB with bindgen, and bindgen needs libclang. It's the only Carbon 2 crate that does; you also hit this building the whole Carbon workspace. Install the development package, which provides libclang.so:

# Debian and Ubuntu
sudo apt-get install libclang-dev
# macOS
brew install llvm

If libclang is installed somewhere non-standard, point LIBCLANG_PATH at the directory that contains libclang.so.

the crate carbon-…-datasource@2.0.0 could not be found in registry index#

error: the crate `carbon-jito-shredstream-grpc-datasource@2.0.0` could not be found in registry index.

Jito ShredStream and Jetstreamer datasources stayed on Carbon 1 (they don't support Transaction V1 and the Solana 4 crates yet). The datasources page lists the twelve that exist for 2.0.

Connecting to a datasource#

Method not found (-32601) from blockSubscribe#

[2026-11-08T23:51:57Z ERROR carbon_rpc_block_subscribe_datasource] Failed to subscribe to block updates: SubscribeFailed { reason: "Method not found (-32601)", message: "{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32601,\"message\":\"Method not found\"},\"id\":1}" }
...
[2026-11-08T23:52:37Z ERROR carbon_core::pipeline] error consuming datasource: Custom("Failed to subscribe after 10 attempts: subscribe failed: Method not found (-32601)")

The endpoint doesn't enable blockSubscribe. Solana's public endpoint (wss://api.mainnet-beta.solana.com) never does, and many providers enable it only on paid plans. RpcBlockSubscribe retries every three seconds and gives up after ten failures, after which the pipeline stops. Use a provider that supports the method, or switch datasource: RpcProgramSubscribe works on the public endpoint for account updates, RpcTransactionCrawler for transactions, and Yellowstone gRPC for production streaming.

HTTP 429 Too Many Requests from the RPC endpoint#

Solana's public endpoint limits requests per IP and per method. getTransaction is the tightest: the response headers on a throttled call show x-ratelimit-method-limit: 10 and retry-after: 10. The Solana RPC client inside Carbon's RPC datasources retries a 429 up to five times on its own, sleeping for the retry-after time, and logs this only at debug level:

[2026-11-08T23:49:57Z DEBUG solana_rpc_client::http_sender] Too many requests: server responded with Response { url: "https://api.mainnet-beta.solana.com/", status: 429, headers: {… "retry-after": "10", … "x-ratelimit-tier": "free", "x-ratelimit-method-limit": "10", "x-ratelimit-method-remaining": "-1", …} }, 4 retries left, pausing for 10s

When those retries run out, the call fails with Status(429, …) and the datasource's own retry policy takes over. For the transaction crawler that's RetryConfig (three retries with backoff by default); after it, the crawler logs Failed to fetch transaction <signature> after 3 retries: … and skips that transaction. Fixes, in order of effect:

• Use your own RPC provider. Every RPC datasource takes the URL as its first argument.
• Lower the request rate: for the crawler, reduce max_concurrent_requests and batch_limit in ConnectionConfig.
• To see the throttling, add the RPC client's logger: RUST_LOG=info,solana_rpc_client=debug.

Status(413, … "Request Entity Too Large") from getSignaturesForAddress#

[2026-11-08T23:48:38Z ERROR carbon_rpc_transaction_crawler_datasource] Failed to fetch signatures after 0 retries: Error { request: Some(GetSignaturesForAddress), kind: Reqwest(reqwest::Error { kind: Status(413, Some(ReasonPhrase(b"Request Entity Too Large"))), url: "https://api.mainnet-beta.solana.com/" }) }

The public endpoint sometimes rejects getSignaturesForAddress for busy programs with HTTP 413; the same request succeeds a few seconds later. With RetryConfig::no_retry() (as in the transaction-crawler-rpc example) the crawler logs it and tries again on the next poll, so it's harmless. With RetryConfig::default() it backs off and retries up to three times first. A dedicated RPC provider doesn't do this.

gRPC transport error: transport error and the pipeline stops#

[2026-11-09T00:05:46Z ERROR carbon_core::pipeline] error consuming datasource: FailedToConsumeDatasource("gRPC transport error: transport error")
[2026-11-09T00:05:46Z INFO  carbon_core::pipeline] update_receiver closed, shutting down.

YellowstoneGrpcGeyserClient couldn't open the first connection: the host doesn't resolve, the port is closed, TLS failed, or the URL has no scheme. The first connection isn't retried, so the pipeline stops at once. Check that the URL is the full gRPC endpoint your provider gave you, with scheme and port (https://your-endpoint:443), and that X_TOKEN is set if the provider needs one. Reconnection after a connection has been established is configured separately; see Yellowstone gRPC. Error variant: FailedToConsumeDatasource.

Failed to receive updates(…)#

This is the message of Error::FailedToReceiveUpdates. No datasource or pipeline code in Carbon 2.0.0 returns it, so if it's in your log, it comes from a custom datasource or a third-party crate; the text in parentheses is that code's own reason. Search your code for FailedToReceiveUpdates. Carbon's own datasources report connection failures as FailedToConsumeDatasource or Custom, logged as error consuming datasource: ….

Geyser stream error … 501 Not Implemented, many times per second#

[2026-11-09T00:05:47Z ERROR carbon_yellowstone_grpc_datasource] Geyser stream error: Status { code: Internal, message: "protocol error: received message with invalid compression flag: 123 (valid flags are 0 and 1) while receiving response with status: 501 Not Implemented", source: None }
[2026-11-09T00:05:47Z ERROR carbon_yellowstone_grpc_datasource] Disconnected at slot 0

The URL points at a server that accepts the connection but doesn't speak Yellowstone gRPC, typically a JSON-RPC URL such as https://api.mainnet-beta.solana.com put in GEYSER_URL. After a stream error the datasource resubscribes immediately with no delay, so the error repeats about twenty times a second until you stop the process. Use your provider's gRPC endpoint, which is a different host or port from its RPC URL.

Transaction version (1) is not supported by the requesting client#

[2026-11-09T00:12:01Z ERROR carbon_rpc_block_crawler_datasource] Error fetching block at slot 467111665: Error { request: Some(GetBlock), kind: RpcError(RpcResponseError { code: -32015, message: "Transaction version (1) is not supported by the requesting client. Please try the request again with the following configuration parameter: \"maxSupportedTransactionVersion\": 1", data: Empty }) }

Your RpcBlockConfig (for RpcBlockCrawler) or RpcBlockSubscribeConfig (for RpcBlockSubscribe) leaves max_supported_transaction_version unset, so the node refuses blocks that contain versioned transactions, which is almost every mainnet block. The block is skipped. Set max_supported_transaction_version: Some(1), as shown in Transactions are logged as Failed to decode transaction. The transaction crawler and Helius GTFA datasources set it for you.

… must be set: NotPresent although the variable is in .env#

thread 'main' panicked at examples/yellowstone-grpc/src/variants.rs:16:32:
GEYSER_URL must be set: NotPresent

dotenv::dotenv() reads .env from the current directory and its parents, not from the crate being run. Running an example from the repository root with cargo run -p … doesn't load examples/<name>/.env. Run from the example's directory, or set the variables in the shell.

Decoding#

Transactions are logged as Failed to decode transaction#

[2026-11-09T00:12:18Z ERROR carbon_rpc_block_crawler_datasource] Failed to decode transaction: EncodedTransactionWithStatusMeta { transaction: Json(UiTransaction { signatures: ["4JBnX1dGq94vApHPA8MsjcGb8akiw8WwcuokdMWBaZP2z2JktVdtv5wYro8e6v8kJJgMN8a4Z66V9S7EjRhxV8n8"], message: Raw(UiRawMessage { header: MessageHeader { …

The datasource asked the node for transactions in Json or JsonParsed encoding. Carbon needs the original binary transaction to rebuild instructions and accounts, so it skips every transaction it can't decode (all 251 in the block above). Request Base64 and Transaction V1 in the config you pass to the datasource:

RpcBlockConfig {
    encoding: Some(UiTransactionEncoding::Base64),
    transaction_details: Some(TransactionDetails::Full),
    max_supported_transaction_version: Some(1),
    ..RpcBlockConfig::default()
}

RpcBlockSubscribeConfig takes the same three fields. For Helius Atlas TransactionSubscribeOptions, set encoding: Some(UiEnhancedTransactionEncoding::Base64) and max_supported_transaction_version: Some(1); its SDK defaults to JSON-parsed. A custom datasource that calls getTransaction must also request UiTransactionEncoding::Base64. See Migrate to 2.0.

Looking for transactions that failed on chain rather than ones Carbon couldn't decode? See Decode or skip failed transactions.

The processor never runs for a program's instructions#

The pipeline receives transactions (carbon_updates_received_total grows) but your instruction processor is never called. Causes, most common first:

• Wrong program ID. Every generated decoder starts with if instruction.program_id != PROGRAM_ID { return None; }. A decoder for one program ignores a fork or redeployment of it at another address, and devnet deployments often use different IDs. Compare the decoder's PROGRAM_ID with the program in an explorer. For a program at another address, generate a decoder from its IDL with carbon-cli parse --program-id.
• The datasource doesn't deliver that program's transactions. Check the datasource filter: the crawler's account, Yellowstone's account_include / account_required, blockSubscribe's MentionsAccountOrProgram.
• The transactions failed. RPC datasources drop transactions whose meta.status is an error before they reach the pipeline; Yellowstone does so when the filter sets failed: Some(false).
• A filter rejects them, for example a SlotRangeFilter with the wrong bounds.

To see what the decoder returns, log every variant with {other:?} in a catch-all match arm and run with RUST_LOG=debug.

Some instructions stop decoding after a program upgrade#

After a program upgrade, a decoder generated from the old IDL returns None for instructions it can't match, and Carbon skips them silently. A generated decoder rejects an instruction when:

• its discriminator is new (an instruction added in the upgrade);
• its data is shorter than the old layout (a field removed or narrowed);
• it has fewer accounts than the old IDL lists.

Data that is longer than the old layout still decodes: the extra bytes are ignored, so new trailing fields are dropped without an error. Regenerate the decoder from the current IDL (Generate a decoder). If you index history from before the upgrade too, register both versions with slot filters: Versioned decoders.

Closed accounts arrive as AccountDeletion, not as account updates#

In Carbon 2.0, every account update with zero lamports becomes Update::AccountDeletion, carrying the account's final state. Account processors registered with .account(…) don't see it. Register .account_deletions(processor) as well. See Updates.

Processing#

Nothing is printed#

The pipeline runs, no error appears, and nothing is printed. Carbon, its datasources, LogMetrics and the examples all write through the log crate, which prints nothing unless a logger is initialized and enabled:

1. Call env_logger::init() (or initialize another logger) at the start of main.
2. Set RUST_LOG when you run. env_logger shows only errors when it's unset, and Carbon's progress messages are info:

RUST_LOG=info cargo run

If you use println! in a processor and still see nothing, the processor isn't being called; see The processor never runs for a program's instructions.

error processing update in the log#

The line is logged by carbon_core::pipeline at ERROR and reads error processing update (<the update>): <the error>, with the full Debug output of the update. A processor or a decoder step returned an error for it. The pipeline logs it with the whole update, increments carbon_updates_failed_total, and moves on to the next update; it doesn't stop. The error also ends processing of that update: pipes after the failing one, and the remaining instructions of the same transaction, aren't run. Return Ok(()) for conditions you can tolerate and log them yourself; return Err only when skipping the rest of the update is what you want. The error variants are in the errors reference; Error handling covers the patterns.

The processor runs several times for one transaction#

An instruction processor is called once per matching instruction, including instructions nested inside other programs' calls (CPIs) and the event logs that programs such as Meteora DLMM and Jupiter emit through self-CPI. A single Jupiter swap can produce a top-level route, several inner instructions and an event. Check input.metadata.stack_height (1 for top-level instructions) or match only the variants you need. To handle a transaction once as a whole, use a transaction processor. See Instructions.

Performance#

Crawler prints nothing for seconds at a time#

RpcTransactionCrawler on the public endpoint fetches a thousand signatures in a few seconds, then stalls on getTransaction: throttled calls sleep for ten seconds inside the RPC client without logging at info (see HTTP 429). In a 60-second run against Meteora DLMM, the LogMetrics line read:

[2026-11-08T23:49:36Z INFO  carbon_log_metrics]   transaction_crawler_signatures_fetched_total: 1000
[2026-11-08T23:49:36Z INFO  carbon_log_metrics]   transaction_crawler_transactions_fetched_total: 26

Failed transactions are fetched but skipped, so a busy stretch of failed bot transactions also prints nothing. Use your own RPC endpoint for real backfills, or Helius GTFA for history with server-side filters.

Full(..) send errors and missing updates#

[2026-11-09T00:12:19Z ERROR carbon_rpc_block_crawler_datasource] Error sending transaction update: "Full(..)"

The channel between the datasources and the pipeline is full, and the datasource dropped the update instead of waiting. Almost every Carbon 2.0 datasource sends with try_send, so a full channel means lost data, each with its own message:

The channel fills when processors are slower than the datasource or the buffer is small: with channel_buffer_size(10), the block crawler above delivered 10 of a block's 251 transactions. Fixes:

• Keep the default buffer of 1,000 or raise it with channel_buffer_size. A bigger buffer absorbs bursts, not a sustained gap.
• For the transaction crawler, set the last ConnectionConfig::new argument, blocking_send, to true, so it waits for space instead of dropping.
• Make processors faster (next entry), and watch carbon_updates_queued in metrics.

in queue keeps growing in the LogMetrics line#

LogMetrics prints … | N processed (…), N successful, 0 failed (0%), M in queue every five seconds. If M keeps growing, processors are slower than the datasource. The pipeline handles one update at a time and runs every pipe for it before taking the next, so a slow processor delays everything. Once the queue reaches the channel size, updates are dropped (previous entry). Move slow work out of the processing path: batch database writes, hand work to a spawned task through a channel, and filter at the datasource so fewer updates arrive. See Production.

Shutdown#

Pipeline exits immediately after starting#

[2026-11-08T23:48:31Z INFO  carbon_core::pipeline] update_receiver closed, shutting down.
[2026-11-08T23:48:31Z INFO  carbon_core::pipeline] pipeline shutdown complete.

run() returns when every datasource has finished and dropped its sender. Some datasources are finite by design: GpaDatasource, HeliusGpaV2Datasource, SnapshotDatasource, HeliusGtfaDatasource and RpcBlockCrawler with an end_slot stop when they've sent everything. A custom datasource whose consume returns without keeping a sender alive also ends the pipeline. If you expected the pipeline to keep running, pair the finite source with a streaming one in the same pipeline (Multiple datasources), or check the log for an error immediately before these lines (next entry). See Finite datasources.

Pipeline exits with status 0 after a datasource error#

[2026-11-09T00:14:58Z ERROR carbon_core::pipeline] error consuming datasource: FailedToConsumeDatasource("Failed to download https://api.mainnet-beta.solana.com/snapshot.tar.bz2/genesis.tar.bz2 after 3 retries: Failed to consume datasource (HTTP error: 405 Method Not Allowed)")
[2026-11-09T00:14:58Z INFO  carbon_core::pipeline] update_receiver closed, shutting down.
[2026-11-09T00:14:58Z INFO  carbon_core::pipeline] pipeline shutdown complete.
[2026-11-09T00:14:58Z INFO  snapshot_validator_carbon_example] snapshot bootstrap complete

When a datasource's consume returns an error, the pipeline only logs it. Once no datasource is left, run() returns Ok(()), so your code after it runs as if the work succeeded and the process exits with status 0. If a supervisor or a job runner must see the failure, check a counter you increment in your processors after run() returns, or wrap the datasource in your own Datasource that records the error before returning it. Error variant: FailedToConsumeDatasource.

The transaction crawler never exits, even with until_signature#

RpcTransactionCrawler walks backwards until it reaches until_signature (or the oldest signature the node returns), then keeps polling for new signatures every polling_interval. Its consume never finishes, so the pipeline runs until it's cancelled. To backfill a fixed range and stop, cancel the pipeline yourself (Stop with a cancellation token) when your processor sees the slot or signature you want, or use Helius GTFA with a slot range, which exits when drained.

Ctrl-C does nothing the first time#

The pipeline only notices Ctrl-C while it waits for the next update. A Ctrl-C that arrives while a processor is running is lost, and the process doesn't exit, because Tokio has replaced the default signal handler. In a test with a processor that takes 100 ms per update, the first Ctrl-C was ignored and the pipeline processed all 100 queued updates; a second Ctrl-C, sent while it was idle, stopped it. Handle Ctrl-C in your own task and cancel the pipeline's token from there: Ctrl-C.

Pending updates are lost with ShutdownStrategy::ProcessPending#

ProcessPending is meant to drain the queue on Ctrl-C, but in 2.0.0 the pipeline loop exits as soon as it sees the cancellation it triggered, so queued updates are usually dropped. Don't rely on it to drain; see ShutdownStrategy for the details and alternatives.

Failed to send signature: SendError { .. } when stopping#

[2026-11-08T23:49:36Z ERROR carbon_rpc_transaction_crawler_datasource] Failed to send signature: SendError { .. }

The crawler's internal tasks were still sending when the pipeline closed its channels. It's logged during shutdown only and is harmless.

Still stuck#

Run with RUST_LOG=debug to see decoder and datasource detail, and compare your setup with the runnable examples. If you think you've found a bug in Carbon, open an issue on GitHub with the Carbon version, your Cargo.toml, and the log.