Integrate Custom Data Sources with a Custom Indexer
The sui-indexer-alt-framework ingests ordered Sui checkpoints from one polling-based source and can optionally add fullnode gRPC streaming for lower latency at the network tip. For production, use Google Cloud Storage (GCS) for backfill and recovery, then use a production-grade fullnode gRPC endpoint from an RPC provider for steady state.
A polling source retrieves checkpoints from a checkpoint store or fullnode, including earlier checkpoints that remain within its retention. Streaming pushes newly produced checkpoints from a fullnode with lower latency, but only near the network tip, so it always runs alongside a polling source.
Which checkpoint source should I use?
| Source | CLI configuration | Retention | Recommended use |
|---|---|---|---|
| Production-grade fullnode gRPC from an RPC provider | --rpc-api-url <URL> and --streaming-url <URL> | Depends on the fullnode | Production steady state |
| Public-good fullnode gRPC | --rpc-api-url <URL> and --streaming-url <URL> | Depends on the service | Testing only; not for production ingestion |
| Requester Pays GCS checkpoint bucket | --remote-store-gcs <BUCKET> and --remote-store-header "x-goog-user-project:<BILLING_PROJECT>" | Full | Production backfill and recovery |
| Public HTTPS checkpoint store | --remote-store-url <URL> | Most recent 30 days | Testing only; not for production ingestion |
| Local checkpoint files | --local-ingestion-path <PATH> | Files present locally | Reproducible development and tests |
| Archival Service gRPC (public or provider) | --rpc-api-url <URL> | Full historical point data; limits and pricing vary | Targeted historical ingestion; use GCS for bulk production backfill |
The Mainnet and Testnet GCS buckets are:
- Mainnet:
gs://mysten-mainnet-checkpoints-use4 - Testnet:
gs://mysten-testnet-checkpoints-use4
The public-good HTTPS checkpoint stores are https://checkpoints.mainnet.sui.io and https://checkpoints.testnet.sui.io. They are public-good testing endpoints with only the most recent 30 days of checkpoints. Do not use them as a production backfill, recovery, fallback, or steady-state source.
The public-good fullnode endpoints are https://fullnode.mainnet.sui.io:443, https://fullnode.testnet.sui.io:443, and https://fullnode.devnet.sui.io:443. They are also for testing only, not for production indexers. For production, obtain a fullnode gRPC endpoint from an RPC provider with capacity and retention appropriate for your workload.
How do I set up a production indexer?
Use separate backfill and steady-state processes. Both processes use the same database and pipeline configuration. Each pipeline records its progress in that database as a committed watermark, so the steady-state process resumes from those committed watermarks after the backfill stops.
- Backfill from the Requester Pays GCS bucket.
- After every required pipeline catches up, stop the GCS-backed process.
- Restart with fullnode gRPC polling and streaming through a production-grade endpoint from an RPC provider.
- If an outage or lag exceeds the fullnode's retention, stop the process, recover the missing range from the GCS bucket, and then switch back to fullnode gRPC.
How do I backfill from Requester Pays GCS?
Because the checkpoint buckets are Requester Pays, Google Cloud charges access to your own billing project. Configure Google credentials that can charge an enabled billing project. Pass the billing project in the x-goog-user-project header:
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcp-credentials.json
$ export GOOGLE_CLOUD_PROJECT=your-billing-project
$ cargo run -- \
--remote-store-gcs mysten-mainnet-checkpoints-use4 \
--remote-store-header "x-goog-user-project:${GOOGLE_CLOUD_PROJECT}"
For Testnet, replace the bucket with mysten-testnet-checkpoints-use4. Workload Identity or another supported Google credential source can replace the credential file, but Requester Pays still requires the billing project header.
How do I switch to fullnode gRPC at the network tip?
After every required pipeline catches up, stop the GCS-backed process and restart with fullnode gRPC polling and streaming through a production-grade endpoint from an RPC provider:
$ export FULLNODE_GRPC_URL=https://your-rpc-provider.example:443
$ cargo run -- \
--rpc-api-url "$FULLNODE_GRPC_URL" \
--streaming-url "$FULLNODE_GRPC_URL"
The polling client retrieves checkpoints missed during short streaming interruptions, subject to the fullnode's retention. If an outage or lag exceeds that retention, stop the process, recover the missing range from the full-retention GCS bucket, and then switch back to fullnode gRPC.
Push-based data source (recommended for steady state)
Streaming pushes newly produced checkpoints from a fullnode with lower latency than polling. It emits checkpoints near the network tip, so always configure a polling source as well. For normal production steady state, use the same production-grade RPC provider endpoint for both:
$ export FULLNODE_GRPC_URL=https://your-rpc-provider.example:443
$ cargo run -- \
--rpc-api-url "$FULLNODE_GRPC_URL" \
--streaming-url "$FULLNODE_GRPC_URL"
The streaming client automatically falls back to the polling client when streaming is unavailable or when it needs an earlier checkpoint that remains within the polling source's retention.
The following public-good endpoints are available for testing only:
- Mainnet:
https://fullnode.mainnet.sui.io:443 - Testnet:
https://fullnode.testnet.sui.io:443 - Devnet:
https://fullnode.devnet.sui.io:443
Do not use these public-good endpoints for a production indexer. Use a production-grade fullnode endpoint from an RPC provider instead.
Polling-based data sources
The following sections describe each polling-based source.
Requester Pays GCS checkpoint buckets
Use --remote-store-gcs for full-history backfills and recovery. Do not pass a gs:// URL to --remote-store-url; provide only the bucket name to --remote-store-gcs and include the billing header shown in the production backfill example.
See Google Cloud's Use Requester Pays documentation and Running a Remote Store for more information.
Fullnode gRPC polling
Use --rpc-api-url without --streaming-url when polling alone is sufficient. For production, configure it with an RPC provider endpoint:
$ export FULLNODE_GRPC_URL=https://your-rpc-provider.example:443
$ cargo run -- --rpc-api-url "$FULLNODE_GRPC_URL"
This is useful for Devnet, Localnet, custom networks, and steady-state workloads that do not need the lowest possible latency. The fullnode's retention limits historical availability. For testing, you can replace the provider URL with the public-good endpoint for your network.
Public HTTPS checkpoint store: testing only
Use the public HTTPS store only for development or tests against a checkpoint that is still within its 30-day window:
$ cargo run -- \
--remote-store-url https://checkpoints.testnet.sui.io \
--first-checkpoint RECENT_CHECKPOINT
A fresh pipeline starts at checkpoint 0 unless you provide --first-checkpoint. Because checkpoint 0 is no longer present in the public HTTPS store, omitting a recent starting checkpoint causes the indexer to retry missing data indefinitely. Once a pipeline has a committed watermark, it resumes from that watermark and ignores --first-checkpoint.
Local path
Local checkpoint files provide reproducible development and test inputs:
$ cargo run -- --local-ingestion-path /path/to/checkpoints
Use a local path for unit and integration tests with known checkpoint data. Do not rely on a fullnode's local ingestion directory as an operated production store because the node does not automatically clean up those files.
Archival Service gRPC
Archival Service implements the gRPC LedgerService. Configure its endpoint through --rpc-api-url. Use the public endpoints only for small tests and targeted historical ingestion because they have strict rate limits.
Provider-hosted archival endpoints can offer higher limits and production service-level agreements for point lookups and targeted historical ingestion. Even with a provider endpoint, use GCS by default when you backfill an indexer. Bulk sequential backfill through LedgerService typically runs slower and costs more than reading checkpoint files from GCS. Use the Requester Pays GCS buckets unless your provider documents sufficient bulk throughput and acceptable pricing.
See Querying Historical Data with Archival Service for endpoint details.