# RPC transaction crawler (Carbon 2.0) > Backfill and then poll every transaction that mentions an address, using getSignaturesForAddress and getTransaction on any Solana RPC. `RpcTransactionCrawler` pages through `getSignaturesForAddress` for one address, fetches each transaction with `getTransaction`, and emits it as a transaction update. It walks history backwards from the newest signature (or from a signature you choose), then keeps polling for new ones. Use it to backfill a program's or wallet's history, or to index a low-traffic program from any RPC endpoint, including the free public one. | | | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Crate | `carbon-rpc-transaction-crawler-datasource` 2.0.0 | | Install | `cargo add carbon-rpc-transaction-crawler-datasource@2.0.0` | | Emits | `Transaction` | | Mode | Historical backfill, then polling for new transactions. Never finishes on its own. | | Provider | Any Solana JSON-RPC HTTP endpoint. The public `https://api.mainnet-beta.solana.com` works, slowly. | | API docs | [docs.rs](https://docs.rs/carbon-rpc-transaction-crawler-datasource/2.0.0/carbon_rpc_transaction_crawler_datasource/) | | Source | [`datasources/rpc-transaction-crawler-datasource`](https://github.com/sevenlabs-hq/carbon/blob/v2.0.0/datasources/rpc-transaction-crawler-datasource/src/lib.rs) | ## Quick example This program crawls the Orca Whirlpool program on the public RPC and prints every decoded swap. ```sh cargo new crawler && cd crawler cargo add carbon-core@2.0.0 carbon-log-metrics@2.0.0 carbon-orca-whirlpool-decoder@2.0.0 carbon-rpc-transaction-crawler-datasource@2.0.0 cargo add tokio@1 --features full cargo add env_logger@0.11 log@0.4 ``` `src/main.rs`: ```rust use { carbon_core::{ error::CarbonResult, instruction::InstructionProcessorInputType, pipeline::Pipeline, processor::Processor, }, carbon_log_metrics::LogMetrics, carbon_orca_whirlpool_decoder::{ instructions::OrcaWhirlpoolInstruction, OrcaWhirlpoolDecoder, PROGRAM_ID, }, carbon_rpc_transaction_crawler_datasource::{ConnectionConfig, Filters, RpcTransactionCrawler}, std::sync::Arc, }; #[tokio::main] async fn main() -> CarbonResult<()> { env_logger::init(); let datasource = RpcTransactionCrawler::new( "https://api.mainnet-beta.solana.com".to_string(), PROGRAM_ID, // crawl signatures that mention the Orca Whirlpool program ConnectionConfig::default(), // 100 signatures per page, 5 concurrent getTransaction calls Filters::new(None, None, None), None, // commitment: defaults to confirmed ); Pipeline::builder() .datasource(datasource) .metrics(Arc::new(LogMetrics::new())) .instruction(OrcaWhirlpoolDecoder, SwapLogger) .build()? .run() .await } struct SwapLogger; impl Processor> for SwapLogger { async fn process( &mut self, input: &InstructionProcessorInputType<'_, OrcaWhirlpoolInstruction>, ) -> CarbonResult<()> { let tx = &input.metadata.transaction_metadata; match input.decoded_instruction { OrcaWhirlpoolInstruction::Swap { data, accounts, .. } => { println!("slot {} swap pool={} amount={}", tx.slot, accounts.whirlpool, data.amount); } OrcaWhirlpoolInstruction::SwapV2 { data, accounts, .. } => { println!("slot {} swapV2 pool={} amount={}", tx.slot, accounts.whirlpool, data.amount); } _ => {} } Ok(()) } } ``` ```sh cargo run ``` Output: ```text slot 467115758 swap pool=4v8ufj8Hj7UvFgtofQJAtzUud5xomwZfEqfCTHZ4wM72 amount=19999 slot 467115758 swapV2 pool=FpCMFDFGYotvufJ7HrFHsWEiiQCGbkLCtwHiDnh7o28Q amount=7639572 slot 467115758 swap pool=Czfq3xZZDmsdGdUyrNLtRhGc47cXcZtLG4crryfu44zE amount=2476330084 slot 467115758 swap pool=FpCMFDFGYotvufJ7HrFHsWEiiQCGbkLCtwHiDnh7o28Q amount=168756526 ``` The public endpoint rate-limits `getTransaction` per IP (its `x-ratelimit-method-limit` header reported 10 when this page was written), so expect a few transactions per second at most. Run with `RUST_LOG=info` to see the 429 retries and the pipeline counters. The program runs until you press Ctrl-C. ## Configuration ### `RpcTransactionCrawler::new` ```rust pub const fn new( rpc_url: String, account: Pubkey, connection_config: ConnectionConfig, filters: Filters, commitment: Option, ) -> Self ``` | Parameter | Type | Description | | ------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------- | | `rpc_url` | `String` | HTTP JSON-RPC endpoint. | | `account` | `Pubkey` | Address passed to `getSignaturesForAddress`: a program ID, a wallet, a pool, any account. | | `connection_config` | `ConnectionConfig` | Paging, concurrency, retries and channel sizes. See below. | | `filters` | `Filters` | Optional account filter and signature range. See below. | | `commitment` | `Option` | Commitment for both RPC calls. `None` means `confirmed`. `processed` is not accepted by these RPC methods. | All fields of `RpcTransactionCrawler` are public, so you can also build it with a struct literal. ### `Filters` ```rust pub const fn new( accounts: Option>, before_signature: Option, until_signature: Option, ) -> Self ``` | Field | Type | Default | Meaning | | ------------------ | --------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `accounts` | `Option>` | `None` | Keep only transactions whose account keys (static keys plus addresses loaded from lookup tables) include at least one of these. Applied after the transaction is fetched, so it saves processing, not RPC calls. | | `before_signature` | `Option` | `None` | Start the backfill immediately before this signature (exclusive) instead of at the newest one. | | `until_signature` | `Option` | `None` | Stop the backfill when this signature is reached (exclusive). | ### `ConnectionConfig` `ConnectionConfig::default()` is an inherent `const fn`, not the `Default` trait. Override single fields with `ConnectionConfig { batch_limit: 1000, ..ConnectionConfig::default() }`. The positional constructor takes the arguments in a different order from the fields: ```rust pub const fn new( batch_limit: usize, polling_interval: Duration, max_concurrent_requests: usize, retry_config: RetryConfig, max_signature_channel_size: Option, max_transaction_channel_size: Option, blocking_send: bool, ) -> Self ``` | Field | Type | Default | Meaning | | ------------------------------ | --------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `batch_limit` | `usize` | `100` | `limit` for each `getSignaturesForAddress` page. RPC nodes cap it at 1000. | | `polling_interval` | `Duration` | 5 s | Sleep after a page comes back empty, before polling for new signatures. | | `max_concurrent_requests` | `usize` | `5` | `getTransaction` calls in flight at once. | | `retry_config` | `RetryConfig` | `RetryConfig::default()` | Backoff for both RPC calls. | | `max_signature_channel_size` | `Option` | `None` (1000) | Buffer between the signature fetcher and the transaction fetcher. | | `max_transaction_channel_size` | `Option` | `None` (1000) | Buffer between the transaction fetcher and the step that sends to the pipeline. | | `blocking_send` | `bool` | `false` | `false` uses `try_send` into the pipeline channel and **drops** the update when it is full. `true` waits for space. Set `true` for backfills that must be complete. | ### `RetryConfig` ```rust pub const fn new(max_retries: u32, initial_backoff_ms: u64, max_backoff_ms: u64, backoff_multiplier: f64) -> Self pub const fn default() -> Self // 3, 1000, 10000, 2.0 pub const fn no_retry() -> Self // 0, 0, 0, 0.0 ``` | Field | Type | Default | Meaning | | -------------------- | ----- | ------- | --------------------------------------------- | | `max_retries` | `u32` | `3` | Retries after the first failed attempt. | | `initial_backoff_ms` | `u64` | `1000` | Delay before the first retry. | | `max_backoff_ms` | `u64` | `10000` | Upper bound on the delay. | | `backoff_multiplier` | `f64` | `2.0` | Factor applied to the delay after each retry. | ## Behaviour - **Pipeline.** Three tasks run in sequence: a signature fetcher, a transaction fetcher that runs up to `max_concurrent_requests` `getTransaction` calls (Base64 encoding, `maxSupportedTransactionVersion: 1`), and a processor that builds the update and sends it to the pipeline. `consume` returns as soon as the tasks are spawned. - **Lifecycle.** The first pass pages backwards from `before_signature` (or the newest signature) until the RPC returns an empty page or `until_signature` is reached. The crawler then sleeps `polling_interval` and polls from the tip down to the newest signature it has seen, forever. It never finishes, so a pipeline that only has this datasource runs until Ctrl-C or until you cancel it (see [shutdown](https://carbon.sevenlabs.org/concepts/shutdown.md)). After the first pass it also picks up everything newer than `before_signature`. - **Ordering.** Signatures arrive newest first, and transactions are emitted in the order their `getTransaction` calls complete. Updates are not in slot order. Sort by `slot` downstream if order matters. - **What is skipped.** Failed transactions (non-`Ok` status) and transactions without metadata are dropped. Every update has `is_vote: false`, `index: None` and `block_hash: None`; `block_time` comes from the RPC. - **Retries.** Each call retries with exponential backoff from `RetryConfig`. A signature page that still fails is retried from scratch on the next loop, so the crawler never skips a page. A transaction that still fails is logged with `Failed to fetch transaction … after N retries` and skipped. On top of that, Solana's `RpcClient` retries HTTP 429 responses up to 5 times, honouring `Retry-After`, and only logs at `debug` level. - **Backpressure.** With `blocking_send: false`, updates are dropped with `Failed to send update` when the pipeline channel (`channel_buffer_size`, default 1000) is full. - **Metrics.** `transaction_crawler_signatures_fetched_total`, `transaction_crawler_transactions_fetched_total` (counters), `transaction_crawler_signatures_fetch_times_milliseconds`, `transaction_crawler_transaction_fetch_times_milliseconds`, `transaction_crawler_transaction_process_time_milliseconds` (histograms). See [metrics](https://carbon.sevenlabs.org/concepts/metrics.md). ## Troubleshooting ### Signatures are fetched but nothing is processed `transaction_crawler_signatures_fetched_total` grows while `transaction_crawler_transactions_fetched_total` stays at 0. At first nothing is logged, because `RpcClient` retries 429 responses silently; once those retries run out, `RUST_LOG=info` shows `Failed to fetch transaction … Status(429, None)`. The RPC is rate limiting `getTransaction`. On the public endpoint the limit is shared by everyone on your IP. Lower `max_concurrent_requests` to 1 or 2, or use a dedicated RPC provider. ### `Method does not support commitment below confirmed` You passed `CommitmentConfig::processed()`. `getSignaturesForAddress` and `getTransaction` only accept `confirmed` and `finalized`; the signature fetcher retries the failing call forever. ### Transactions are missing from a backfill Check the logs for `Failed to send update` (pipeline channel full) and `Failed to fetch transaction … after N retries` (RPC errors). Set `blocking_send: true`, raise `retry_config.max_retries`, and raise the pipeline's [`channel_buffer_size`](https://carbon.sevenlabs.org/reference/pipeline-builder.md) if your processors are slow. Failed transactions are skipped by design. ### Transactions arrive out of order Expected: `getTransaction` calls complete in any order. Use `slot` (and your own ordering) in the processor, or set `max_concurrent_requests: 1` for strictly newest-first order. ### The program never exits after the backfill Expected: the crawler switches to polling. To stop at the end of a range, cancel the pipeline from your code with [`datasource_cancellation_token`](https://carbon.sevenlabs.org/concepts/shutdown.md), or use [`RpcBlockCrawler`](https://carbon.sevenlabs.org/datasources/rpc-block-crawler.md) with an `end_slot`. ## See also - [Backfill transactions](https://carbon.sevenlabs.org/guides/backfill-transactions.md) - [Stream with RPC](https://carbon.sevenlabs.org/guides/stream-with-rpc.md) - [Helius `getTransactionsForAddress`](https://carbon.sevenlabs.org/datasources/helius-gtfa.md): the same job in fewer calls, on Helius. - [Datasources concept](https://carbon.sevenlabs.org/concepts/datasources.md) and [update types](https://carbon.sevenlabs.org/concepts/updates.md) --- Carbon 2.0.0 documentation · Web: https://carbon.sevenlabs.org/datasources/rpc-transaction-crawler · Last updated 2026-11-09 · Index of all pages: https://carbon.sevenlabs.org/llms.txt