Write a decoder by hand

Decode a program that has no IDL by implementing CarbonDeserialize, ArrangeAccounts, InstructionDecoder and AccountDecoder yourself, then unit-test it.

Most decoders are generated from an IDL with the CLI. Write one by hand when the program has no Anchor or Codama IDL: a native Rust program, a program whose IDL is wrong or missing, or when you only need two instructions out of fifty.

This guide writes a decoder for Counter, a native program with a borsh-encoded, one-byte-tagged instruction set, and tests it with unit tests. The same pieces apply to any layout.

You need: Rust 1.96.1 or newer, and the program's binary layout: the bytes that identify each instruction and account (the discriminator), the field order and encoding of each, and the order of each instruction's accounts. Get it from the program's source or its client SDK. Decoders explains how decoders fit into the pipeline.

The layout#

Item	Discriminator	Payload (borsh)	Accounts, in order
`Initialize` instruction	`[0]`	`initial_count: u64`	`counter` (writable), `authority` (signer), `system_program`
`Increment` instruction	`[1]`	`amount: u64`	`counter` (writable), `authority` (signer)
`Counter` account	`[1]`	`authority: Pubkey`, `count: u64`
`CounterIncremented` event (`sol_log_data`)	`b"cinc"`	`counter: Pubkey`, `new_count: u64`

Anchor programs use 8-byte discriminators (sha256("global:<name>")[..8] for instructions, sha256("account:<Name>")[..8] for accounts). Native programs often use a one-byte enum tag, and some accounts have none at all; then the account's owner and exact data length identify it.

The pieces#

Piece	Role
`CarbonDeserialize`	`DISCRIMINATOR` constant plus `deserialize(&[u8]) -> Option<Self>`: check the prefix, decode the rest. Requires `BorshDeserialize`. `InstructionMetadata::decode_log_events::<T>()` needs it.
`Type::decode(&[u8]) -> Option<Self>`	An inherent function. `try_decode_instructions!` calls it by that name, so every instruction type needs one. Generated decoders have the same function.
`ArrangeAccounts`	Turns the instruction's positional `&[AccountMeta]` into a named struct. Return `None` when required accounts are missing.
`try_decode_instructions!`	Tries each `Enum::Variant => Type` in order and returns `Some(Enum::Variant { program_id, data, accounts })` for the first type whose `decode` and `arrange_accounts` both succeed.
`InstructionDecoder`	What `.instruction(decoder, processor)` takes. Return `None` for instructions that aren't yours.
`AccountDecoder`	What `.account(decoder, processor)` takes. Return `None` for accounts that aren't yours.

Create a library crate#

Terminal

cargo new --lib counter-decoder
cd counter-decoder
cargo add carbon-core@2.0.0 solana-account@4.3.2 solana-instruction@3.4.1
cargo add borsh --features derive
cargo add solana-pubkey@4.2.1 --features borsh
cargo add --dev base64 solana-transaction-status@=4.2.2

Cargo.toml

[package]
name = "counter-decoder"
version = "0.1.0"
edition = "2021"

[dependencies]
borsh = { version = "1.8.1", features = ["derive"] }
carbon-core = "2.0.0"
solana-account = "4.3.2"
solana-instruction = "3.4.1"
solana-pubkey = { version = "4.2.1", features = ["borsh"] }

[dev-dependencies]
base64 = "0.23.1"
solana-transaction-status = "=4.2.2"

Newer patch versions of borsh and base64 work too. carbon-core's default macros feature provides try_decode_instructions!. The borsh feature of solana-pubkey lets Pubkey fields derive BorshDeserialize (carbon-core already turns it on; declaring it keeps your crate correct on its own). Use the Solana crate versions Carbon 2.0 depends on so Pubkey, Instruction and Account are the same types Carbon uses.

Write the decoder#

src/lib.rs

//! Hand-written Carbon decoder for the Counter program.
use {
    borsh::BorshDeserialize,
    carbon_core::{
        account::{AccountDecoder, DecodedAccount},
        account_utils::next_account,
        deserialize::{ArrangeAccounts, CarbonDeserialize},
        instruction::InstructionDecoder,
    },
    solana_instruction::{AccountMeta, Instruction},
    solana_pubkey::Pubkey,
};

pub const PROGRAM_ID: Pubkey = Pubkey::from_str_const("Counter111111111111111111111111111111111111");

/// The decoder you register with the pipeline.
pub struct CounterDecoder;

/// Borsh-decode `data` after checking and removing a discriminator prefix.
/// `try_from_slice` rejects trailing bytes, so a longer layout never decodes as a shorter one.
fn decode_prefixed<T: BorshDeserialize>(discriminator: &[u8], data: &[u8]) -> Option<T> {
    let payload = data.strip_prefix(discriminator)?;
    T::try_from_slice(payload).ok()
}

// ---------- Instructions ----------

/// Instruction 0: create a counter. Data: `[0] ++ initial_count: u64`.
#[derive(Debug, Clone, PartialEq, BorshDeserialize)]
pub struct Initialize {
    pub initial_count: u64,
}

/// Instruction 1: add to a counter. Data: `[1] ++ amount: u64`.
#[derive(Debug, Clone, PartialEq, BorshDeserialize)]
pub struct Increment {
    pub amount: u64,
}

impl CarbonDeserialize for Initialize {
    const DISCRIMINATOR: &'static [u8] = &[0];

    fn deserialize(data: &[u8]) -> Option<Self> {
        decode_prefixed(Self::DISCRIMINATOR, data)
    }
}

impl CarbonDeserialize for Increment {
    const DISCRIMINATOR: &'static [u8] = &[1];

    fn deserialize(data: &[u8]) -> Option<Self> {
        decode_prefixed(Self::DISCRIMINATOR, data)
    }
}

// `try_decode_instructions!` calls `Type::decode(&[u8])`.
impl Initialize {
    pub fn decode(data: &[u8]) -> Option<Self> {
        <Self as CarbonDeserialize>::deserialize(data)
    }
}

impl Increment {
    pub fn decode(data: &[u8]) -> Option<Self> {
        <Self as CarbonDeserialize>::deserialize(data)
    }
}

#[derive(Debug, Clone, PartialEq)]
pub struct InitializeAccounts {
    pub counter: Pubkey,
    pub authority: Pubkey,
    pub system_program: Pubkey,
}

#[derive(Debug, Clone, PartialEq)]
pub struct IncrementAccounts {
    pub counter: Pubkey,
    pub authority: Pubkey,
    /// Accounts past the ones the program declares.
    pub remaining: Vec<AccountMeta>,
}

impl ArrangeAccounts for Initialize {
    type ArrangedAccounts = InitializeAccounts;

    fn arrange_accounts(accounts: &[AccountMeta]) -> Option<Self::ArrangedAccounts> {
        let mut iter = accounts.iter();
        Some(InitializeAccounts {
            counter: next_account(&mut iter)?,
            authority: next_account(&mut iter)?,
            system_program: next_account(&mut iter)?,
        })
    }
}

impl ArrangeAccounts for Increment {
    type ArrangedAccounts = IncrementAccounts;

    fn arrange_accounts(accounts: &[AccountMeta]) -> Option<Self::ArrangedAccounts> {
        let mut iter = accounts.iter();
        let counter = next_account(&mut iter)?;
        let authority = next_account(&mut iter)?;
        Some(IncrementAccounts {
            counter,
            authority,
            remaining: iter.as_slice().to_vec(),
        })
    }
}

/// One variant per instruction, in the shape `try_decode_instructions!` builds.
#[derive(Debug, Clone, PartialEq)]
pub enum CounterInstruction {
    Initialize {
        program_id: Pubkey,
        data: Initialize,
        accounts: InitializeAccounts,
    },
    Increment {
        program_id: Pubkey,
        data: Increment,
        accounts: IncrementAccounts,
    },
}

impl InstructionDecoder<'_> for CounterDecoder {
    type InstructionType = CounterInstruction;

    fn decode_instruction(&self, instruction: &Instruction) -> Option<Self::InstructionType> {
        if instruction.program_id != PROGRAM_ID {
            return None;
        }

        carbon_core::try_decode_instructions!(
            instruction,
            PROGRAM_ID,
            CounterInstruction::Initialize => Initialize,
            CounterInstruction::Increment => Increment,
        )
    }
}

// ---------- Accounts ----------

/// Counter state. Layout: `[1] ++ authority: Pubkey ++ count: u64` (41 bytes).
#[derive(Debug, Clone, PartialEq, BorshDeserialize)]
pub struct Counter {
    pub authority: Pubkey,
    pub count: u64,
}

impl CarbonDeserialize for Counter {
    const DISCRIMINATOR: &'static [u8] = &[1];

    fn deserialize(data: &[u8]) -> Option<Self> {
        decode_prefixed(Self::DISCRIMINATOR, data)
    }
}

#[derive(Debug, Clone, PartialEq)]
pub enum CounterAccount {
    Counter(Counter),
}

impl<'a> AccountDecoder<'a> for CounterDecoder {
    type AccountType = CounterAccount;

    fn decode_account(
        &self,
        account: &'a solana_account::Account,
    ) -> Option<DecodedAccount<Self::AccountType>> {
        if account.owner != PROGRAM_ID {
            return None;
        }

        let data = <Counter as CarbonDeserialize>::deserialize(&account.data)?;
        Some(DecodedAccount {
            lamports: account.lamports,
            data: CounterAccount::Counter(data),
            owner: account.owner,
            executable: account.executable,
            rent_epoch: account.rent_epoch,
        })
    }
}

// ---------- Events ----------

/// Logged with `sol_log_data` by `Increment`. Data: `b"cinc" ++ counter ++ new_count: u64`.
#[derive(Debug, Clone, PartialEq, BorshDeserialize)]
pub struct CounterIncremented {
    pub counter: Pubkey,
    pub new_count: u64,
}

impl CarbonDeserialize for CounterIncremented {
    const DISCRIMINATOR: &'static [u8] = b"cinc";

    fn deserialize(data: &[u8]) -> Option<Self> {
        decode_prefixed(Self::DISCRIMINATOR, data)
    }
}

Notes on the choices:

try_from_slice, not deserialize(&mut slice). Generated decoders read with BorshDeserialize::deserialize(&mut slice), which stops after the last field and ignores extra bytes. try_from_slice fails on extra bytes, so a payload from a longer, newer layout is rejected instead of decoded wrongly. Use the lenient form if the program appends optional trailing data.
Fully qualified calls. CarbonDeserialize::deserialize and BorshDeserialize::deserialize have the same name. With both traits in scope, call <Counter as CarbonDeserialize>::deserialize(..).
Program check first. decode_instruction compares program_id before decoding and decode_account compares the owner. Discriminators are short and collide across programs; a one-byte tag matches many unrelated instructions.
remaining. Keeping accounts past the declared ones, as generated decoders do, preserves extra accounts that some instructions take, such as remaining accounts for routing.
Order in try_decode_instructions!. The first match wins. With distinct discriminators the order doesn't matter; list the longer layout first if two types share one.

Test it#

Unit tests build instruction and account bytes by hand and check every rejection path, so a layout mistake fails here rather than silently in production. The event test builds the InstructionMetadata the pipeline would pass and checks that decode_log_events finds the event in the logs of instruction [0].

tests/decoder.rs

use {
    base64::{engine::general_purpose::STANDARD, Engine},
    carbon_core::{
        account::AccountDecoder,
        instruction::{InstructionDecoder, InstructionMetadata},
        transaction::TransactionMetadata,
    },
    counter_decoder::{
        Counter, CounterAccount, CounterDecoder, CounterIncremented, CounterInstruction, Increment,
        PROGRAM_ID,
    },
    solana_instruction::{AccountMeta, Instruction},
    solana_pubkey::Pubkey,
    solana_transaction_status::TransactionStatusMeta,
    std::sync::Arc,
};

fn increment_ix(data: Vec<u8>) -> (Instruction, Pubkey, Pubkey) {
    let counter = Pubkey::new_unique();
    let authority = Pubkey::new_unique();
    let instruction = Instruction {
        program_id: PROGRAM_ID,
        accounts: vec![
            AccountMeta::new(counter, false),
            AccountMeta::new_readonly(authority, true),
        ],
        data,
    };
    (instruction, counter, authority)
}

#[test]
fn decodes_increment() {
    let mut data = vec![1];
    data.extend_from_slice(&42u64.to_le_bytes());
    let (instruction, counter, authority) = increment_ix(data);

    let decoded = CounterDecoder.decode_instruction(&instruction).expect("decodes");

    let CounterInstruction::Increment { data, accounts, .. } = decoded else {
        panic!("expected Increment, got {decoded:?}");
    };
    assert_eq!(data, Increment { amount: 42 });
    assert_eq!(accounts.counter, counter);
    assert_eq!(accounts.authority, authority);
    assert!(accounts.remaining.is_empty());
}

#[test]
fn rejects_other_programs_and_bad_data() {
    let mut data = vec![1];
    data.extend_from_slice(&42u64.to_le_bytes());

    let (mut other_program, _, _) = increment_ix(data.clone());
    other_program.program_id = Pubkey::new_unique();
    assert!(CounterDecoder.decode_instruction(&other_program).is_none());

    let (truncated, _, _) = increment_ix(data[..5].to_vec());
    assert!(CounterDecoder.decode_instruction(&truncated).is_none());

    let (unknown, _, _) = increment_ix(vec![9, 0, 0, 0, 0, 0, 0, 0, 0]);
    assert!(CounterDecoder.decode_instruction(&unknown).is_none());

    let (mut missing_accounts, _, _) = increment_ix(data);
    missing_accounts.accounts.truncate(1);
    assert!(CounterDecoder.decode_instruction(&missing_accounts).is_none());
}

#[test]
fn decodes_counter_account() {
    let authority = Pubkey::new_unique();
    let mut data = vec![1];
    data.extend_from_slice(authority.as_ref());
    data.extend_from_slice(&7u64.to_le_bytes());

    let account = solana_account::Account {
        lamports: 1_183_200,
        data,
        owner: PROGRAM_ID,
        executable: false,
        rent_epoch: u64::MAX,
    };

    let decoded = CounterDecoder.decode_account(&account).expect("decodes");
    assert_eq!(decoded.data, CounterAccount::Counter(Counter { authority, count: 7 }));
}

#[test]
fn decodes_event_from_logs() {
    let counter = Pubkey::new_unique();
    let mut event = b"cinc".to_vec();
    event.extend_from_slice(counter.as_ref());
    event.extend_from_slice(&43u64.to_le_bytes());

    let logs = vec![
        format!("Program {PROGRAM_ID} invoke [1]"),
        format!("Program data: {}", STANDARD.encode(&event)),
        format!("Program {PROGRAM_ID} success"),
    ];
    let metadata = InstructionMetadata {
        transaction_metadata: Arc::new(TransactionMetadata {
            meta: TransactionStatusMeta {
                log_messages: Some(logs),
                ..Default::default()
            },
            ..Default::default()
        }),
        stack_height: 1,
        index: 0,
        absolute_path: vec![0],
    };

    let events = metadata.decode_log_events::<CounterIncremented>();
    assert_eq!(events, vec![CounterIncremented { counter, new_count: 43 }]);
}

Terminal

cargo test

Output

     Running tests/decoder.rs (target/debug/deps/decoder-3c61c1f19ffcfb96)

running 4 tests
test decodes_counter_account ... ok
test rejects_other_programs_and_bad_data ... ok
test decodes_increment ... ok
test decodes_event_from_logs ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Add a test with a real instruction and account from mainnet once the program is deployed; Test decoders and processors shows how to capture them as fixtures.

Use it in a pipeline#

The decoder plugs in like a generated one. A processor receives CounterInstruction and can decode the events its instruction logged:

Rust

use {
    carbon_core::{error::CarbonResult, instruction::InstructionProcessorInputType, processor::Processor},
    counter_decoder::{CounterIncremented, CounterInstruction},
};

struct CounterProcessor;

impl Processor<InstructionProcessorInputType<'_, CounterInstruction>> for CounterProcessor {
    async fn process(
        &mut self,
        input: &InstructionProcessorInputType<'_, CounterInstruction>,
    ) -> CarbonResult<()> {
        if let CounterInstruction::Increment { data, accounts, .. } = input.decoded_instruction {
            // Events this instruction logged with `sol_log_data`.
            for event in input.metadata.decode_log_events::<CounterIncremented>() {
                println!("{} +{} -> {}", accounts.counter, data.amount, event.new_count);
            }
        }
        Ok(())
    }
}

Register it with .instruction(counter_decoder::CounterDecoder, CounterProcessor), and the account side with .account(counter_decoder::CounterDecoder, processor). InstructionProcessorInputType and AccountProcessorInputType are described in Processor inputs.

decode_log_events reads Program data: <base64> log lines that the instruction at exactly this position in the call tree emitted, skips the ones whose bytes T::deserialize rejects, and returns the rest in order. Events logged by an inner instruction belong to that inner instruction, not its parent. Events emitted as a self-CPI (Anchor's emit_cpi!) are instructions, not logs: decode them with your InstructionDecoder as a separate variant.

Use it in a transaction pipe#

.transaction(..) pipes need an InstructionDecoderCollection, whose enum must be Eq + Hash + Serialize. Add #[derive(Eq, Hash, serde::Serialize)] to CounterInstruction and to every type it contains (with serde and the serde feature of solana-pubkey), then list the decoder in instruction_decoder_collection!.

Common problems#

no associated function or constant named `decode` found for struct `Increment` #

try_decode_instructions! calls Type::decode. Add impl Type { pub fn decode(data: &[u8]) -> Option<Self> { .. } } for each instruction type, delegating to CarbonDeserialize.

`multiple applicable items in scope` on `deserialize`#

Both CarbonDeserialize and BorshDeserialize are in scope. Call <Type as CarbonDeserialize>::deserialize(data).

Every instruction decodes as the wrong variant#

Two types accept the same bytes: a shared or empty discriminator, or lenient decoding that ignores trailing bytes. Give each type its own discriminator check, use try_from_slice, and put the more specific type first in try_decode_instructions!.

Instructions decode in tests but not in the pipeline#

The program ID constant doesn't match the deployed program, or arrange_accounts requires more accounts than real transactions pass. Test with a captured mainnet instruction.

Next steps#

Decoders

The decoder traits and how decoding works.

Test decoders and processors

Fixtures from real transactions and accounts.

Generate a decoder

Generate the same pieces from an Anchor or Codama IDL.

Macros reference

try_decode_instructions!, instruction_decoder_collection! and InstructionType.