Built for blazing fast persistence of terra bytes of structured data on a single machine while offering rich querying capabilities, e.g. bitcoin/blockchain data. Blockchains need rich and often analytical queries which is done through explorers because indexing speed of even embedded/in-process (not through socket) analytical db like DuckDB right on the node would be still slower than a hand-made solution on top of Redb or RocksDb.
Redbit reads struct annotations and derives code necessary for persisting and querying structured data into/from Redb using secondary indexes and dictionaries, served by axum through auto-generated REST API.
✅ Querying and ranging by secondary index
✅ Optional dictionaries for low cardinality fields + first level cache for building them without overhead
✅ One-to-One / One-to-Option / One-to-Many entities with cascade read/write/delete
✅ All goodies including intuitive data ordering without writing custom codecs
✅ All keys and all newType column types with fixed-sized value implement Copy => minimal cloning
✅ Http response streaming api with efficient querying (ie. get txs or utxos for really HOT address)
✅ Query contraints : eq, ne, lt, le, gt, ge, in with logical AND
{
"header": {
"height": { "$eq": 1 }
},
"transactions": {
"hash": { "$in": ["bar", "baz"] },
"utxo": {
"address": { "$eq": "foo" }
}
}
}✅ Column types : String, Int, Vec<u8>, [u8; N], bool, uuid::Uuid, std::time::Duration
✅ Optional column is basically One-to-Option relationship, we build a table for optional "values"
✅ Column encodings of binary columns : hex, base64, utf-8 + custom impl of ByteVecColumnSerde
✅ All types have binary (db) and human-readable (http) serde support
✅ Macro derived http rest API at http://127.0.0.1:3033/swagger-ui/ with examples
✅ Macro derived unit tests and integration tests on axum test server and benchmarks
✅ TypeScript client generated from OpenAPI spec with tests suite requesting all endpoints
✅ For other features, check the redbit-ui
❌ Root key must be newtype struct with numeric inner type (that's part of the design decision to achieve fast indexing of even whole bitcoin)
cd examples/utxo
cargo test # to let all the self-generated tests run
cargo test --features integration # to let http layer self-generated tests run
cargo bench # to run benchmarks
cargo run # to run the demo example and start the server
Check the redbit-ui for frontend dev.
The utxo example has close to 500 frontend/backend derived tests and 130 benchmarks, so that if any redbit app derived from the definition compiles, it is transparent, well tested and benched already.
Let's say we want to persist and query blockchain data using Redbit, declare annotated Structs examples/utxo/src/lib.rs:
pub use redbit::*;
// feel free to add custom #[derive(Foo, Bar)] attributes to your types, they will get merged with the ones from redbit
#[root_key] pub struct Height(pub u32);
#[pointer_key(u16)] pub struct BlockPointer(Height);
#[pointer_key(u16)] pub struct TransactionPointer(BlockPointer);
#[pointer_key(u16)] pub struct UtxoPointer(TransactionPointer);
// #[column] pub struct Time(pub chrono::DateTime<chrono::Utc>);
#[column("hex")] pub struct BlockHash(pub [u8; 32]);
#[column("hex")] pub struct TxHash(pub [u8; 32]);
#[column("base64")] pub struct Address(pub Vec<u8>);
#[column("utf-8")] pub struct AssetName(pub Vec<u8>); // String is supported but this is more efficient
#[column] pub struct Duration(pub std::time::Duration);
#[column] pub struct Weight(pub u32);
#[column] pub struct Timestamp(pub u32);
#[column]
pub struct TempInputRef {
pub tx_hash: TxHash,
pub index: u32,
}
#[entity]
pub struct Block {
#[pk]
pub height: Height,
pub header: Header,
pub transactions: Vec<Transaction>,
}
#[entity]
pub struct Header {
#[fk(one2one)]
pub height: Height,
#[column(index)]
pub hash: BlockHash,
#[column(index)]
pub prev_hash: BlockHash,
#[column(range)]
pub timestamp: Timestamp,
#[column(range)]
pub duration: Duration,
#[column]
pub nonce: u64,
#[column(transient)]
pub weight: Weight,
}
#[entity]
pub struct Transaction {
#[fk(one2many)]
pub id: BlockPointer,
#[column(index)]
pub hash: TxHash,
pub utxos: Vec<Utxo>,
pub inputs: Vec<InputRef>,
pub maybe_value: Option<MaybeValue>, // just to demonstrate option is possible
#[column(transient)]
pub transient_inputs: Vec<TempInputRef>,
}
#[entity]
pub struct Utxo {
#[fk(one2many)]
pub id: TransactionPointer,
#[column]
pub amount: u64,
#[column(dictionary(cache = 10000))]
pub address: Address,
pub assets: Vec<Asset>,
}
#[entity]
pub struct InputRef {
#[fk(one2many)]
pub id: TransactionPointer,
}
#[entity]
pub struct MaybeValue {
#[fk(one2opt)]
pub id: BlockPointer,
#[column(index)]
pub hash: BlockHash
}
#[entity]
pub struct Asset {
#[fk(one2many)]
pub id: UtxoPointer,
#[column]
pub amount: u64,
#[column(dictionary(cache = 10000))]
pub name: AssetName,
}
use chain::api::*;
pub struct BlockChain {
pub storage: Arc<Storage>,
}
impl BlockChain {
pub fn new(storage: Arc<Storage>) -> Arc<dyn BlockChainLike<Block>> {
Arc::new(BlockChain { storage })
}
fn resolve_tx_inputs(&self, read_tx: &StorageReadTx, block: &mut Block) -> Result<(), ChainError> {
for tx in &mut block.transactions {
for transient_input in tx.transient_inputs.iter_mut() {
let tx_pointers = Transaction::get_ids_by_hash(read_tx, &transient_input.tx_hash)?;
match tx_pointers.first() {
Some(tx_pointer) => tx.inputs.push(InputRef { id: TransactionPointer::from_parent(*tx_pointer, transient_input.index as u16) }),
None => tx.inputs.push(InputRef { id: TransactionPointer::from_parent(BlockPointer::from_parent(Height(0), 0), 0) }),
}
}
}
Ok(())
}
}And R/W entire instances efficiently using indexes and dictionaries examples/utxo/src/demo.rs:
use redbit::{AppError, Storage};
use std::sync::Arc;
use crate::model_v1::*;
pub async fn showcase() -> Result<(), AppError> {
let storage = Storage::temp("showcase", 1, true)?;
let blocks = Block::sample_many(2);
println!("Persisting blocks:");
let write_tx = storage.begin_write()?;
Block::store_many(&write_tx, &blocks)?;
write_tx.commit()?;
let read_tx = storage.begin_read()?;
let first_block = Block::first(&read_tx)?.unwrap();
let last_block = Block::last(&read_tx)?.unwrap();
Block::take(&read_tx, 100)?;
Block::get(&read_tx, &first_block.height)?;
Block::range(&read_tx, &first_block.height, &last_block.height, None)?;
Block::get_transactions(&read_tx, &first_block.height)?;
Block::get_header(&read_tx, &first_block.height)?;
Block::exists(&read_tx, &first_block.height)?;
Block::first(&read_tx)?;
Block::last(&read_tx)?;
Block::stream_range(storage.begin_read()?, first_block.height, last_block.height, None)?.try_collect::<Vec<Block>>().await?;
let block_infos = Block::table_info(Arc::clone(&storage))?;
println!("Block persisted with tables :");
for info in block_infos {
println!("{}", serde_json::to_string_pretty(&info).unwrap());
}
let first_block_header = Header::first(&read_tx)?.unwrap();
let last_block_header = Header::last(&read_tx)?.unwrap();
Header::get_by_hash(&read_tx, &first_block_header.hash)?;
Header::get_by_timestamp(&read_tx, &first_block_header.timestamp)?;
Header::take(&read_tx, 100)?;
Header::get(&read_tx, &first_block_header.height)?;
Header::range(&read_tx, &first_block_header.height, &last_block_header.height, None)?;
Header::range_by_timestamp(&read_tx, &first_block_header.timestamp, &last_block_header.timestamp)?;
Header::stream_by_hash(storage.begin_read()?, first_block_header.hash, None)?.try_collect::<Vec<Header>>().await?;
Header::stream_by_timestamp(storage.begin_read()?, first_block_header.timestamp, None)?.try_collect::<Vec<Header>>().await?;
Header::stream_range(storage.begin_read()?, first_block_header.height, last_block_header.height, None)?.try_collect::<Vec<Header>>().await?;
Header::stream_range_by_timestamp(storage.begin_read()?, first_block_header.timestamp, last_block_header.timestamp, None)?.try_collect::<Vec<Header>>().await?;
let block_header_infos = Header::table_info(Arc::clone(&storage))?;
println!("
Block header persisted with tables :");
for info in block_header_infos {
println!("{}", serde_json::to_string_pretty(&info).unwrap());
}
let first_transaction = Transaction::first(&read_tx)?.unwrap();
let last_transaction = Transaction::last(&read_tx)?.unwrap();
Transaction::get_ids_by_hash(&read_tx, &first_transaction.hash)?;
Transaction::get_by_hash(&read_tx, &first_transaction.hash)?;
Transaction::take(&read_tx, 100)?;
Transaction::get(&read_tx, &first_transaction.id)?;
Transaction::range(&read_tx, &first_transaction.id, &last_transaction.id, None)?;
Transaction::get_utxos(&read_tx, &first_transaction.id)?;
Transaction::get_maybe_value(&read_tx, &first_transaction.id)?;
Transaction::parent_key(&read_tx, &first_transaction.id)?;
Transaction::stream_ids_by_hash(&read_tx, &first_transaction.hash)?.try_collect::<Vec<BlockPointer>>().await?;
Transaction::stream_by_hash(storage.begin_read()?, first_transaction.hash.clone(), None)?.try_collect::<Vec<Transaction>>().await?;
Transaction::stream_range(storage.begin_read()?, first_transaction.id, last_transaction.id, None)?.try_collect::<Vec<Transaction>>().await?;
let transaction_infos = Transaction::table_info(Arc::clone(&storage))?;
println!("
Transaction persisted with tables :");
for info in transaction_infos {
println!("{}", serde_json::to_string_pretty(&info).unwrap());
}
let first_utxo = Utxo::first(&read_tx)?.unwrap();
let last_utxo = Utxo::last(&read_tx)?.unwrap();
Utxo::get_by_address(&read_tx, &first_utxo.address)?;
Utxo::get_ids_by_address(&read_tx, &first_utxo.address)?;
Utxo::take(&read_tx, 100)?;
Utxo::get(&read_tx, &first_utxo.id)?;
Utxo::range(&read_tx, &first_utxo.id, &last_utxo.id, None)?;
Utxo::get_assets(&read_tx, &first_utxo.id)?;
Utxo::parent_key(&read_tx, &first_utxo.id)?;
Utxo::stream_ids_by_address(&read_tx, &first_utxo.address)?.try_collect::<Vec<TransactionPointer>>().await?;
Utxo::stream_range(storage.begin_read()?, first_utxo.id, last_utxo.id, None)?.try_collect::<Vec<Utxo>>().await?;
Utxo::stream_by_address(storage.begin_read()?, first_utxo.address.clone(), None)?.try_collect::<Vec<Utxo>>().await?;
// even streaming parents is possible
Utxo::stream_transactions_by_address(storage.begin_read()?, first_utxo.address, None)?.try_collect::<Vec<Transaction>>().await?;
let utxo_infos = Utxo::table_info(Arc::clone(&storage))?;
println!("
Utxo persisted with tables :");
for info in utxo_infos {
println!("{}", serde_json::to_string_pretty(&info).unwrap());
}
let first_asset = Asset::first(&read_tx)?.unwrap();
let last_asset = Asset::last(&read_tx)?.unwrap();
Asset::get_by_name(&read_tx, &first_asset.name)?;
Asset::take(&read_tx, 100)?;
Asset::get(&read_tx, &first_asset.id)?;
Asset::range(&read_tx, &first_asset.id, &last_asset.id, None)?;
Asset::parent_key(&read_tx, &first_asset.id)?;
Asset::stream_by_name(storage.begin_read()?, first_asset.name.clone(), None)?.try_collect::<Vec<Asset>>().await?;
Asset::stream_range(storage.begin_read()?, first_asset.id, last_asset.id, None)?.try_collect::<Vec<Asset>>().await?;
// even streaming parents is possible
Asset::stream_utxos_by_name(storage.begin_read()?, first_asset.name, None)?.try_collect::<Vec<Utxo>>().await?;
let asset_infos = Asset::table_info(Arc::clone(&storage))?;
println!("
Asset persisted with tables :");
for info in asset_infos {
println!("{}", serde_json::to_string_pretty(&info).unwrap());
}
println!("
Deleting blocks:");
for block in blocks.iter() {
Block::delete_and_commit(Arc::clone(&storage), &block.height)?;
}
Ok(())
}The same api is accessible through http endpoints at http://127.0.0.1:3033/swagger-ui/.
Performance wise, check 🔥flamegraph. The demo example persists data into 30 tables to allow for rich querying.
cargo flamegraph --bin target/release/demo --release
The slowest block::_store_many operation in this context persists 3 blocks of 3 transactions of 1 input and 3 utxos of 3 assets, ie.
the operations writes :
- 3 blocks
- 3 * 3 = 9 transactions
- 3 * 3 = 9 inputs
- 3 * 3 * 3 = 27 utxos
- 3 * 3 * 3 * 3 = 81 assets
block::_first operation reads whole block with all its transactions, inputs, utxos and assets.
function ops/s
-------------------------------------------------------------
model_v1::block::_store 895
model_v1::block::_store_and_commit 900
model_v1::block::_store_many 924
model_v1::transaction::_store_and_commit 1421
model_v1::transaction::_store 1515
model_v1::transaction::_store_many 1554
model_v1::header::_store_many 2393
model_v1::header::_store 2405
model_v1::header::_store_and_commit 2416
model_v1::utxo::_store_many 2424
model_v1::utxo::_store 2429
model_v1::utxo::_store_and_commit 2443
model_v1::asset::_store_many 3100
model_v1::maybevalue::_store_many 3335
model_v1::maybevalue::_store_and_commit 3337
model_v1::block::_take 3534
model_v1::asset::_store_and_commit 3536
model_v1::block::_tail 3561
model_v1::maybevalue::_store 3726
model_v1::asset::_store 3809
model_v1::transaction::_delete_and_commit 3990
model_v1::inputref::_store_many 4114
model_v1::inputref::_store_and_commit 4310
model_v1::inputref::_store 4393
model_v1::block::_delete_and_commit 4704
model_v1::maybevalue::_delete_and_commit 4840
model_v1::header::_delete_and_commit 5029
model_v1::utxo::_delete_and_commit 5164
model_v1::inputref::_delete_and_commit 5404
model_v1::asset::_delete_and_commit 5993
model_v1::block::_first 6998
model_v1::block::_last 7029
model_v1::block::_get 7066
model_v1::block::_get_transactions 7369
model_v1::transaction::_tail 11484
model_v1::transaction::_take 11597
model_v1::transaction::_get_by_hash 23027
model_v1::transaction::_last 23147
model_v1::transaction::_get 23316
model_v1::transaction::_first 23463
model_v1::transaction::_get_utxos 25795
model_v1::block::_range 34342
model_v1::block::_stream_range 35609
model_v1::transaction::_stream_blocks_by_hash 38416
model_v1::block::_filter 39537
model_v1::utxo::_tail 43164
model_v1::utxo::_take 43588
model_v1::transaction::_range 50969
model_v1::transaction::_stream_range 53281
model_v1::transaction::_stream_by_hash 55123
model_v1::utxo::_stream_transactions_by_address 57961
model_v1::transaction::_filter 59201
model_v1::utxo::_get_by_address 76851
model_v1::utxo::_first 85521
model_v1::utxo::_get 85974
model_v1::utxo::_last 86876
model_v1::utxo::_range 91460
model_v1::utxo::_stream_range 92318
model_v1::utxo::_stream_by_address 93600
model_v1::asset::_stream_utxos_by_name 99357
model_v1::utxo::_filter 113435
model_v1::utxo::_get_assets 115271
model_v1::header::_tail 131106
model_v1::header::_take 133659
model_v1::header::_stream_range_by_duration 142589
model_v1::header::_stream_range_by_timestamp 144273
model_v1::asset::_tail 166354
model_v1::asset::_take 177099
model_v1::header::_range 178057
model_v1::header::_stream_range 183405
model_v1::asset::_range 200480
model_v1::asset::_stream_range 213592
model_v1::asset::_stream_by_name 215059
model_v1::header::_stream_by_hash 216169
model_v1::header::_stream_by_prev_hash 216836
model_v1::header::_stream_by_duration 220106
model_v1::header::_stream_by_timestamp 222003
model_v1::header::_range_by_duration 228665
model_v1::header::_get_by_prev_hash 236697
model_v1::block::_get_header 238202
model_v1::header::_range_by_timestamp 240079
model_v1::header::_get_by_hash 241971
model_v1::asset::_get_by_name 247323
model_v1::header::_get_by_duration 247416
model_v1::header::_get_by_timestamp 247604
model_v1::header::_filter 273563
model_v1::header::_last 273607
model_v1::header::_first 273965
model_v1::header::_get 274030
model_v1::maybevalue::_range 321227
model_v1::asset::_filter 344035
model_v1::asset::_last 349913
model_v1::asset::_first 354041
model_v1::asset::_get 354357
model_v1::maybevalue::_stream_range 361289
model_v1::maybevalue::_tail 404006
model_v1::utxo::_stream_ids_by_address 410981
model_v1::maybevalue::_take 438698
model_v1::asset::_stream_ids_by_name 481656
model_v1::maybevalue::_stream_by_hash 500080
model_v1::utxo::_get_ids_by_address 563491
model_v1::inputref::_stream_range 572872
model_v1::asset::_get_ids_by_name 630366
model_v1::maybevalue::_get_by_hash 646471
model_v1::utxo::_pk_range 674832
model_v1::inputref::_pk_range 686672
model_v1::transaction::_pk_range 689984
model_v1::transaction::_get_inputs 700894
model_v1::asset::_pk_range 710843
model_v1::transaction::_get_maybe_value 710843
model_v1::inputref::_tail 747446
model_v1::block::_pk_range 753012
model_v1::maybevalue::_pk_range 769604
model_v1::maybevalue::_filter 789092
model_v1::inputref::_range 803348
model_v1::maybevalue::_get 807487
model_v1::header::_pk_range 809029
model_v1::maybevalue::_stream_ids_by_hash 823364
model_v1::header::_stream_heights_by_hash 826160
model_v1::transaction::_stream_ids_by_hash 832868
model_v1::header::_stream_heights_by_prev_hash 840315
model_v1::header::_stream_heights_by_duration 860993
model_v1::maybevalue::_last 863275
model_v1::maybevalue::_first 863476
model_v1::header::_stream_heights_by_timestamp 897432
model_v1::inputref::_take 961252
model_v1::transaction::_get_ids_by_hash 981489
model_v1::maybevalue::_get_ids_by_hash 1037355
model_v1::header::_get_heights_by_hash 1048933
model_v1::header::_get_heights_by_prev_hash 1052521
model_v1::header::_get_heights_by_duration 1186155
model_v1::header::_get_heights_by_timestamp 1226136
model_v1::transaction::_exists 1373985
model_v1::utxo::_exists 1584058
model_v1::inputref::_exists 1628797
model_v1::inputref::_get 1638297
model_v1::inputref::_filter 1651337
model_v1::maybevalue::_exists 1656452
model_v1::block::_exists 1669672
model_v1::inputref::_last 1802614
model_v1::inputref::_first 1803427
model_v1::asset::_exists 1807370
model_v1::header::_exists 1982514
chain syncs blockchains with nodes :
Hand-made criterion benchmarks deployed.
Indexing speed in logs is the average, for example, the first ~ 100k bitcoin blocks with just one Tx have lower in/out indexing throughput because the block is indexed into ~ 24 tables in total.
If node and indexer each uses its own SSD, then the throughput reaches :
- 2.0GHz & NVMe on PCIe Gen3 :
~ 9 000 Inputs+outputs / s - 3.0GHz & NVMe on PCIe Gen4 :
~ 15 000 Inputs+outputs / s - 4.0GHz & NVMe on PCIe Gen5 :
~ 28 000 Inputs+outputs / s
In a nutshell, whole bitcoin up to height ~ 0.9M can be indexed in less than 4 days on a PCIe Gen5 SSD with 4.0GHz CPU.