Load a snapshot

A validator snapshot is a full copy of every account on the network at one slot. carbon-validator-snapshot-datasource loads one, scans it for the accounts you ask for, and sends them through your pipeline. Use it to bootstrap state for programs too large for getProgramAccounts, without any RPC load.

You need:

• Rust 1.96.1 or newer, a C++ compiler and libclang (on Debian or Ubuntu: sudo apt install build-essential clang libclang-dev). The datasource builds Agave's ledger and runtime crates, including RocksDB.
• A snapshot: a mainnet full snapshot archive was 126 GB in November 2026. Plan for several times that in free disk space, since the accounts are unpacked next to the archive, and for a machine sized like a Solana RPC node: loading builds the full accounts index in memory.
• No account or API key: you can download the snapshot from any RPC node that serves it.

What the datasource reads#

SnapshotDatasource::new(source, owners, accounts) takes a source and two lists. It sends every account whose owner is in owners, plus every account whose address is in accounts. If both lists are empty, it logs No owners or accounts specified, skipping account scan and sends nothing.

The source is one of:

A local ledger directory looks like this:

Keep the archive's original file name: Agave reads the slot and hash from it. The datasource creates ledger/accounts/ and unpacks the accounts there. If snapshots/ holds several archives, the one with the highest slot is used.

Get a snapshot#

Agave RPC nodes serve their latest full snapshot at /snapshot.tar.bz2, which redirects to the real archive name, and the genesis at /genesis.tar.bz2. Solana's public endpoint serves both. Look up the archive's name first:

NODE=https://api.mainnet-beta.solana.com
SNAPSHOT=$(curl -sI "$NODE/snapshot.tar.bz2" | grep -i '^location:' | sed 's|.*/||; s/\r//')
echo "$SNAPSHOT"

snapshot-467081864-Hc57Ndf3cWvzMJmCgAmRyqHNeqLbrWhXXQWQaid7NCsA.tar.zst

Then download the genesis and the archive under that name:

mkdir -p ledger/snapshots
curl -L -o ledger/genesis.tar.bz2 "$NODE/genesis.tar.bz2"
curl -L -o "ledger/snapshots/$SNAPSHOT" "$NODE/$SNAPSHOT"

Any other Agave node that exposes its RPC port works the same way, and is often faster than the rate-limited public endpoint. Snapshots taken by your own validator are already in its ledger directory.

Load the accounts#

This program loads every Meteora DLMM account from a snapshot and prints its pools.

cargo new snapshot-loader
cd snapshot-loader

[package]
name = "snapshot-loader"
version = "0.1.0"
edition = "2024"

[dependencies]
carbon-core = "2.0.0"
carbon-log-metrics = "2.0.0"
carbon-meteora-dlmm-decoder = "2.0.0"
carbon-validator-snapshot-datasource = "2.0.0"
env_logger = "0.11.11"
log = "0.4.35"
tokio = { version = "1.54.1", features = ["full"] }

use {
    carbon_core::{
        account::AccountProcessorInputType, error::CarbonResult, pipeline::Pipeline,
        processor::Processor,
    },
    carbon_log_metrics::LogMetrics,
    carbon_meteora_dlmm_decoder::{accounts::MeteoraDlmmAccount, MeteoraDlmmDecoder, PROGRAM_ID},
    carbon_validator_snapshot_datasource::{SnapshotDatasource, SnapshotSource},
    std::{env, sync::Arc},
};

#[tokio::main]
async fn main() -> CarbonResult<()> {
    env_logger::init();

    let source = match env::var("LEDGER_DIR") {
        // A directory with genesis.tar.bz2 and snapshots/snapshot-<slot>-<hash>.tar.zst
        Ok(dir) => SnapshotSource::LocalPath(dir.into()),
        // A node that serves /genesis.tar.bz2 and /snapshot.tar.bz2
        Err(_) => SnapshotSource::Remote {
            url: env::var("SNAPSHOT_URL").expect("set LEDGER_DIR or SNAPSHOT_URL"),
        },
    };

    let datasource = SnapshotDatasource::new(
        source,
        vec![PROGRAM_ID], // load every account owned by these programs
        vec![],           // plus these specific accounts
    );

    Pipeline::builder()
        .datasource(datasource)
        .metrics(Arc::new(LogMetrics::new()))
        .account(MeteoraDlmmDecoder, PairCounter { pairs: 0 })
        .build()?
        .run()
        .await
}

struct PairCounter {
    pairs: u64,
}

impl Processor<AccountProcessorInputType<'_, MeteoraDlmmAccount>> for PairCounter {
    async fn process(
        &mut self,
        input: &AccountProcessorInputType<'_, MeteoraDlmmAccount>,
    ) -> CarbonResult<()> {
        if let MeteoraDlmmAccount::LbPair(pair) = &input.decoded_account.data {
            self.pairs += 1;
            println!(
                "#{} slot {} pair {}: {} / {}",
                self.pairs, input.metadata.slot, input.metadata.pubkey, pair.token_x_mint, pair.token_y_mint
            );
        }
        Ok(())
    }
}

cargo build --release

LEDGER_DIR=./ledger RUST_LOG=info cargo run --release

SNAPSHOT_URL=https://api.mainnet-beta.solana.com RUST_LOG=info cargo run --release

#1 slot 467081864 pair <pool address>: <token X mint> / <token Y mint>

How the scan behaves#

• One slot. Every account carries the snapshot's slot in input.metadata.slot, and transaction_signature is None.
• Only live accounts. Zero-lamport accounts aren't loadable and are skipped, so the datasource never emits account deletions.
• Order. Accounts arrive in the order the accounts database stores them, not sorted by address.
• Backpressure. The scan waits for the pipeline, so a slow processor slows the scan but loses nothing.
• Cancellation. Ctrl+C is checked only before each download attempt and during the scan. A download or snapshot load in progress runs to completion first; kill the process to stop it immediately.
• Downloads aren't cached. Each run with Remote downloads both files again into ./ledger. Download once, then switch to LocalPath("./ledger"). A failed download is retried 3 times, 2 seconds apart.

Follow the snapshot with live data#

A snapshot is often hours old by the time it's loaded. To keep the state current, add a stream of account changes as a second datasource, a Yellowstone account filter or rpc-program-subscribe, and apply updates only when their slot is newer than what you stored for that account. Changes between the snapshot's slot and the moment the stream starts aren't covered by either: replay them with LaserStream's from_slot if the snapshot is recent enough, or reload the changed accounts with getProgramAccounts. Multiple datasources shows the pattern with getProgramAccounts; the snapshot datasource drops into the same place.

Common problems#

Build fails: Unable to find libclang or couldn't find any valid shared libraries matching: ['libclang.so', …]#

RocksDB's bindings need libclang at build time. Install it (sudo apt install clang libclang-dev, or brew install llvm on macOS), or set LIBCLANG_PATH to the directory that contains libclang.so.

Build fails with No space left on device#

Agave's crates and RocksDB need several gigabytes in target/. Free space, or point CARGO_TARGET_DIR at a larger disk.

FailedToConsumeDatasource("Ledger directory does not exist: …")#

LEDGER_DIR points to a missing directory. Use the directory that contains genesis.tar.bz2, not the snapshots/ directory itself.

Genesis archive not found: … Expected genesis.tar.bz2#

Download genesis.tar.bz2 from the same cluster as the snapshot and put it at the root of the ledger directory.

No loadable full snapshot archive found in …/snapshots#

The snapshots/ directory has no archive with a name Agave recognizes. Keep the snapshot-<slot>-<hash>.tar.zst name the server gave it.

Failed to download …/snapshot.tar.bz2 after 3 retries: HTTP error: …#

SNAPSHOT_URL must be the node's base URL (https://api.mainnet-beta.solana.com), not the snapshot file's URL: the datasource appends /genesis.tar.bz2 and /snapshot.tar.bz2 itself. A 429 or 413 status means the node rate-limited you; download from another node.

The process is killed while loading#

The machine ran out of memory building the accounts index. Use a machine with more RAM.

Nothing is printed and the log says No owners or accounts specified#

Both lists passed to SnapshotDatasource::new are empty. Add your program ID to owners.

Next steps#