Integrate Custom Data Sources with a Custom Indexer

The sui-indexer-alt-framework ingests ordered Sui checkpoints from one polling-based source and can optionally add fullnode gRPC streaming for lower latency at the network tip. For production, use Google Cloud Storage (GCS) for backfill and recovery, then use a production-grade fullnode gRPC endpoint from an RPC provider for steady state.

Checkpoint data sources

Source	CLI configuration	Retention	Recommended use
Production-grade fullnode gRPC from an RPC provider	--rpc-api-url and --streaming-url	Depends on the fullnode	Production steady state
Public-good fullnode gRPC	--rpc-api-url and --streaming-url	Depends on the service	Testing only; not for production ingestion
Requester Pays GCS checkpoint bucket	--remote-store-gcs and billing header	Full	Production backfill and recovery
Public HTTPS checkpoint store	--remote-store-url	Most recent 30 days	Testing only; not for production ingestion
Local checkpoint files	--local-ingestion-path	Files present locally	Reproducible development and tests
Archival Service gRPC (public or provider)	--rpc-api-url	Full historical point data; limits and pricing vary	Targeted historical ingestion; use GCS for bulk production backfill

The Mainnet and Testnet GCS buckets are:

• Mainnet: gs://mysten-mainnet-checkpoints-use4
• Testnet: gs://mysten-testnet-checkpoints-use4

The public-good HTTPS checkpoint stores are https://checkpoints.mainnet.sui.io and https://checkpoints.testnet.sui.io. They are public-good testing endpoints with only the most recent 30 days of checkpoints. Do not use them as a production backfill, recovery, fallback, or steady-state source.

The public-good fullnode endpoints are https://fullnode.mainnet.sui.io:443, https://fullnode.testnet.sui.io:443, and https://fullnode.devnet.sui.io:443. They are also for testing only and are not intended for production indexers. For production, obtain a fullnode gRPC endpoint from an RPC provider with capacity and retention appropriate for your workload.

Recommended production flow

Use separate backfill and steady-state processes. Both processes use the same database and pipeline configuration, so the steady-state process resumes from committed watermarks after the backfill stops.

Backfill from Requester Pays GCS

Configure Google credentials that can charge an enabled billing project. Pass the billing project in the x-goog-user-project header:

$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcp-credentials.json
$ export GOOGLE_CLOUD_PROJECT=your-billing-project

$ cargo run -- \
  --remote-store-gcs mysten-mainnet-checkpoints-use4 \
  --remote-store-header "x-goog-user-project:${GOOGLE_CLOUD_PROJECT}"

For Testnet, replace the bucket with mysten-testnet-checkpoints-use4. Workload Identity or another supported Google credential source can replace the credential file, but the billing project header is still required for Requester Pays.

Switch to fullnode gRPC at the network tip

After every required pipeline catches up, stop the GCS-backed process and restart with fullnode gRPC polling and streaming through a production-grade endpoint from an RPC provider:

$ export FULLNODE_GRPC_URL=https://your-rpc-provider.example:443

$ cargo run -- \
  --rpc-api-url "$FULLNODE_GRPC_URL" \
  --streaming-url "$FULLNODE_GRPC_URL"

The polling client retrieves checkpoints missed during short streaming interruptions, subject to the fullnode's retention. If an outage or lag exceeds that retention, stop the process, recover the missing range from the full-retention GCS bucket, and then switch back to fullnode gRPC.

Push-based data source (recommended for steady state)

Streaming pushes newly produced checkpoints from a fullnode with lower latency than polling. It emits checkpoints near the network tip, so always configure a polling source as well. For normal production steady state, use the same production-grade RPC provider endpoint for both:

$ export FULLNODE_GRPC_URL=https://your-rpc-provider.example:443

$ cargo run -- \
  --rpc-api-url "$FULLNODE_GRPC_URL" \
  --streaming-url "$FULLNODE_GRPC_URL"

The streaming client automatically falls back to the polling client when streaming is unavailable or when it needs an earlier checkpoint that remains within the polling source's retention.

The following public-good endpoints are available for testing only:

• Mainnet: https://fullnode.mainnet.sui.io:443
• Testnet: https://fullnode.testnet.sui.io:443
• Devnet: https://fullnode.devnet.sui.io:443

Do not use these public-good endpoints for a production indexer. Use a production-grade fullnode endpoint from an RPC provider instead.

Polling-based data sources

Requester Pays GCS checkpoint buckets

Use --remote-store-gcs for full-history backfills and recovery. Do not pass a gs:// URL to --remote-store-url; provide only the bucket name to --remote-store-gcs and include the billing header shown in the production backfill example.

See Google Cloud's Use Requester Pays documentation and Running a Remote Store for more information.

Fullnode gRPC polling

Use --rpc-api-url without --streaming-url when polling alone is sufficient. For production, configure it with an RPC provider endpoint:

$ export FULLNODE_GRPC_URL=https://your-rpc-provider.example:443
$ cargo run -- --rpc-api-url "$FULLNODE_GRPC_URL"

This is useful for Devnet, Localnet, custom networks, and steady-state workloads that do not need the lowest possible latency. Historical availability is limited by the fullnode's retention. For testing, you can replace the provider URL with the public-good endpoint for your network.

Public HTTPS checkpoint store: testing only

Use the public HTTPS store only for development or tests against a checkpoint that is still within its 30-day window:

$ cargo run -- \
  --remote-store-url https://checkpoints.testnet.sui.io \
  --first-checkpoint RECENT_CHECKPOINT

A fresh pipeline starts at checkpoint 0 unless you provide --first-checkpoint. Because checkpoint 0 is no longer present in the public HTTPS store, omitting a recent starting checkpoint causes the indexer to retry missing data indefinitely. Once a pipeline has a committed watermark, it resumes from that watermark and ignores --first-checkpoint.

Local path

Local checkpoint files provide reproducible development and test inputs:

$ cargo run -- --local-ingestion-path /path/to/checkpoints

Use a local path for unit and integration tests with known checkpoint data. Do not rely on a fullnode's local ingestion directory as an operated production store because the node does not automatically clean up those files.

Archival Service gRPC

The Archival Service implements the gRPC LedgerService. Configure its endpoint through --rpc-api-url. Use the public endpoints only for small tests and targeted historical ingestion because they have strict rate limits.

Provider-hosted archival endpoints can offer higher limits and production service-level agreements for point lookups and targeted historical ingestion. Even with a provider endpoint, use GCS by default when you backfill an indexer. Bulk sequential backfill through LedgerService typically runs slower and costs more than reading checkpoint files from GCS. Use the Requester Pays GCS buckets unless your provider documents sufficient bulk throughput and acceptable pricing.

See Querying Historical Data with Archival Service for endpoint details.