Custom Indexing Framework
The custom indexing framework (sui-indexer-alt-framework) turns raw checkpoint data into database rows through interfaces that you implement. The process() interface transforms the data, commit() stores it in batches, and the optional prune() removes old records based on your retention policy.
What interfaces does the framework provide?
The framework exposes the following interfaces that you implement to define your data processing logic:
-
process(): Transform raw checkpoint data (transactions, events, object changes) into your desired database rows. This is where you extract meaningful information, filter relevant data, and format it for storage. -
commit(): Store your processed data to the database with proper transaction handling. The framework calls this with batches of processed data for efficient bulk operations. -
prune(): Clean up old data based on your retention policies (optional). Useful for managing database size by removing outdated records while preserving recent data. You enable pruning for pipelines configured with a retention policy, which sets how many of the most recent checkpoints the pipeline keeps.
Which pipeline type should you use?
Sequential and concurrent pipeline types and their trade-offs are detailed in Pipeline Architecture.
What Are Custom Indexers?
The `sui-indexer-alt-framework` is a powerful Rust framework for building high-performance, custom blockchain indexers on Sui. It provides customizable, production-ready components for data ingestion, processing, and storage.
Indexer Pipeline Architecture
The `sui-indexer-alt-framework` provides two distinct pipeline architectures. Understand the differences between the sequential and concurrent pipelines that the `sui-indexer-alt-framework` provides to decide which best suits your project needs.
Build a Custom Indexer
Build a custom indexer using the `sui-indexer-alt-framework` module. The example indexer demonstrates a sequential pipeline that extracts transaction digests from Sui checkpoints and stores them in a local PostgreSQL database, the default example storage backend.
Integrate a Custom Indexer
Add a custom indexer to an existing Rust service using the `sui-indexer-alt-framework` module. Implement the `Processor` and `Handler` traits, register a pipeline, and point the indexer at a checkpoint source.
Bring Your Own Store (BYOS)
Implement a custom storage backend for the customer indexer framework.
Integrate Data Sources
Choose and configure checkpoint data sources for a Sui custom indexer, including production-grade fullnode gRPC, full-retention Requester Pays GCS buckets, public testing endpoints, and local checkpoint files.
Optimize Runtime and Performance
Learn how to optimize Sui custom indexer performance through runtime configuration, resource monitoring, and debugging tools. Covers ingestion settings, database tuning, Tokio console debugging, Prometheus metrics, and data pruning strategies.