# Validator snapshot (Carbon 2.0) > Load every account owned by a program, or a list of accounts, from a Solana validator snapshot on disk or downloaded from a node. `SnapshotDatasource` loads a full validator snapshot with Agave's own loader, scans every account in it, and emits the ones owned by programs you list (or with addresses you list) as account updates at the snapshot's slot. Use it to bootstrap complete account state for a large program without `getProgramAccounts` limits, when you can afford the disk, memory and time a mainnet snapshot needs. | | | | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Crate | `carbon-validator-snapshot-datasource` 2.0.0 | | Install | `cargo add carbon-validator-snapshot-datasource@2.0.0` | | Emits | `AccountUpdate` | | Mode | Point-in-time state at the snapshot slot. Finishes after the scan. | | Provider | None: a snapshot archive on disk, or a node that serves `/genesis.tar.bz2` and `/snapshot.tar.bz2` (Agave RPC nodes do, including `api.mainnet-beta.solana.com`). | | API docs | [docs.rs](https://docs.rs/carbon-validator-snapshot-datasource/2.0.0/carbon_validator_snapshot_datasource/) | | Source | [`datasources/validator-snapshot-datasource`](https://github.com/sevenlabs-hq/carbon/blob/v2.0.0/datasources/validator-snapshot-datasource/src/lib.rs) | > **Warning: Heavy prerequisites.** The crate links Agave's ledger and RocksDB, so it needs a C++ compiler and libclang to build (Debian and Ubuntu: `sudo apt install build-essential clang libclang-dev`); without libclang the build fails in `clang-sys`. A mainnet full snapshot was 126 GB compressed when this page was written, and loading it needs several times that in disk and tens of GB of RAM. ## Quick example This program loads every Orca Whirlpool pool account from a snapshot and prints its liquidity. It compiles as shown; to run it, set `SNAPSHOT_PATH` to a ledger directory or `SNAPSHOT_URL` to a node. ```sh cargo new snapshot && cd snapshot cargo add carbon-core@2.0.0 carbon-log-metrics@2.0.0 carbon-orca-whirlpool-decoder@2.0.0 carbon-validator-snapshot-datasource@2.0.0 cargo add tokio@1 --features full cargo add env_logger@0.11 log@0.4 ``` `src/main.rs`: ```rust use { carbon_core::{ account::AccountProcessorInputType, error::CarbonResult, pipeline::Pipeline, processor::Processor, }, carbon_log_metrics::LogMetrics, carbon_orca_whirlpool_decoder::{ accounts::OrcaWhirlpoolAccount, OrcaWhirlpoolDecoder, PROGRAM_ID, }, carbon_validator_snapshot_datasource::{SnapshotDatasource, SnapshotSource}, std::{env, sync::Arc}, }; #[tokio::main] async fn main() -> CarbonResult<()> { env_logger::init(); let source = match env::var("SNAPSHOT_PATH") { // A ledger directory with genesis.tar.bz2 and snapshots/ Ok(path) => SnapshotSource::LocalPath(path.into()), // A node that serves /genesis.tar.bz2 and /snapshot.tar.bz2, e.g. http://1.2.3.4:8899 Err(_) => SnapshotSource::Remote { url: env::var("SNAPSHOT_URL").expect("set SNAPSHOT_PATH or SNAPSHOT_URL"), }, }; let datasource = SnapshotDatasource::new( source, vec![PROGRAM_ID], // owners: every account owned by the Orca Whirlpool program vec![], // accounts: specific pubkeys, whatever their owner ); Pipeline::builder() .datasource(datasource) .metrics(Arc::new(LogMetrics::new())) .account(OrcaWhirlpoolDecoder, PoolLogger) .build()? .run() .await } struct PoolLogger; impl Processor> for PoolLogger { async fn process( &mut self, input: &AccountProcessorInputType<'_, OrcaWhirlpoolAccount>, ) -> CarbonResult<()> { if let OrcaWhirlpoolAccount::Whirlpool(pool) = &input.decoded_account.data { println!( "slot {} pool {} liquidity={}", input.metadata.slot, input.metadata.pubkey, pool.liquidity ); } Ok(()) } } ``` ```sh cargo build --release SNAPSHOT_PATH=/data/ledger RUST_LOG=info ./target/release/snapshot ``` Build in release mode: loading a snapshot in a debug build is much slower. With `RUST_LOG=info` you see `Snapshot loaded at slot … in …s, starting account scan`, one line per pool, then `Snapshot processing completed`. Pointed at a directory without a genesis archive, it stops at once: Output: ```text [ERROR carbon_core::pipeline] error consuming datasource: FailedToConsumeDatasource("Genesis archive not found: \"/data/ledger/genesis.tar.bz2\". Expected genesis.tar.bz2") ``` ## Configuration ### `SnapshotDatasource::new` ```rust pub fn new(source: SnapshotSource, owners: Vec, accounts: Vec) -> Self ``` | Parameter | Type | Meaning | | ---------- | ---------------- | -------------------------------------------------- | | `source` | `SnapshotSource` | Where the snapshot comes from. | | `owners` | `Vec` | Emit every account owned by one of these programs. | | `accounts` | `Vec` | Also emit these accounts, whatever their owner. | If both lists are empty, the datasource logs `No owners or accounts specified, skipping account scan` and emits nothing. ### `SnapshotSource` ```rust pub enum SnapshotSource { LocalPath(PathBuf), Remote { url: String }, } ``` | Variant | Meaning | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `LocalPath(path)` | A ledger directory containing `genesis.tar.bz2` and a `snapshots/` directory with a full snapshot archive (`snapshot--.tar.zst`; an incremental archive next to it is also loaded). The loader unpacks accounts into `/accounts`. | | `Remote { url }` | A base URL. The datasource downloads `/genesis.tar.bz2` and `/snapshot.tar.bz2`, following up to 10 redirects, into `./ledger` under the current working directory, then loads it like a local path. Pass the node's address, not the archive URL. | ## Behaviour - **Downloading.** Each file is tried 3 times, 2 seconds apart. The saved file name comes from the `Content-Disposition` header or the final redirect URL, so a node's redirect to `snapshot--.tar.zst` keeps the name Agave expects. Progress is logged every 50 MB and 5 seconds. Only the full snapshot is downloaded. - **Loading.** Agave's `bank_forks_utils` loads the snapshot in load-only mode, skipping the initial accounts hash. A loaded bank at slot 0 is rejected as invalid. Loading runs on the async task that called `consume` and can't be cancelled. - **Scanning.** Every account is visited once on a blocking thread. Accounts with zero lamports are skipped. Matches are emitted as `AccountUpdate` with the snapshot slot and `transaction_signature: None`; no deletions are emitted. - **Ordering.** The order of the accounts database, not by pubkey or slot. - **Backpressure.** Updates are sent with `blocking_send`, so the scan waits for the pipeline and nothing is dropped. - **Finishing.** `consume` returns after the scan. A pipeline with only this datasource then shuts down; next to a streaming datasource, the pipeline keeps running. - **Errors.** Missing directories or archives, a failed download, or a snapshot Agave can't load return `FailedToConsumeDatasource` with the reason. - **Metrics.** `validator_snapshot_accounts_processed_total` (counter), `validator_snapshot_load_duration_milliseconds`, `validator_snapshot_scan_duration_milliseconds` (histograms). ## Troubleshooting ### Build fails in `clang-sys`: `couldn't find any valid shared libraries matching: ['libclang.so', …]` Install libclang (`sudo apt install clang libclang-dev` on Debian and Ubuntu), or point `LIBCLANG_PATH` at the directory that contains `libclang.so`. ### `Genesis archive not found: …/genesis.tar.bz2` `LocalPath` must point at the ledger directory itself, with `genesis.tar.bz2` at its top level. ### `No loadable full snapshot archive found in …/snapshots` Put the full snapshot archive in `/snapshots/` and keep its original `snapshot--.tar.zst` name; Agave finds archives by name. ### `Failed to download …/genesis.tar.bz2 after 3 retries` The URL is not a node's base address, or the node doesn't serve snapshots. `Remote { url }` appends `/genesis.tar.bz2` itself: use `https://api.mainnet-beta.solana.com`, not `…/snapshot.tar.bz2`. ### The process is killed during loading Out of memory or disk. Mainnet snapshots need a machine sized like an RPC node. For a single program, [`GpaDatasource`](https://carbon.sevenlabs.org/datasources/rpc-gpa.md) or [Helius gPA v2](https://carbon.sevenlabs.org/datasources/helius-gpa-v2.md) is far cheaper. ## See also - [Load a snapshot](https://carbon.sevenlabs.org/guides/load-a-snapshot.md) - [Load account state](https://carbon.sevenlabs.org/guides/load-account-state.md) - [RPC getProgramAccounts](https://carbon.sevenlabs.org/datasources/rpc-gpa.md) - [Datasources concept](https://carbon.sevenlabs.org/concepts/datasources.md) and [update types](https://carbon.sevenlabs.org/concepts/updates.md) --- Carbon 2.0.0 documentation · Web: https://carbon.sevenlabs.org/datasources/validator-snapshot · Last updated 2026-11-09 · Index of all pages: https://carbon.sevenlabs.org/llms.txt