Sui Validator Alert Reference
When you run a Sui validator or full node, monitor the health of the nodes and set up alerts on problems. Besides alerting on crashes and other common issues, you might also want to configure alerting based on the following example rules.
The following sections cover the alert queries, but you should customize the details, such as the threshold in each query and the recommended trigger duration, to suit your infrastructure.
Which alerts need immediate attention?
These alerts should receive the most immediate attention from you or your team.
Crash loop
| Key | Value |
|---|---|
| Name | Crash loop |
| Severity | Immediate attention |
| Summary | Node is crash looping |
| Duration | Recommended to trigger after 15m |
max without(version) (uptime) < 60 or absent(uptime)
Node is not staying up longer than 60s. Possible reasons:
- Binary version too old
- Incorrect configuration
- Software bug
Please notify Sui community on Discord if you cannot resolve this on your own.
Consensus proposals failure
| Key | Value |
|---|---|
| Name | Consensus proposals failure |
| Severity | Immediate attention |
| Summary | Consensus block proposal rate is low |
| Duration | Recommended to trigger after 1h |
sum without(force) (rate(consensus_proposed_blocks[5m])) < 1.0
Validators with a slow consensus proposal rate can hurt network latency and throughput. It is usually due to network, disk, or CPU performance issues.
Checkpoint execution rate is low
| Key | Value |
|---|---|
| Name | Checkpoint execution rate is low |
| Severity | Immediate attention |
| Summary | Validator is not executing checkpoints quickly enough |
| Duration | Recommended to trigger after 1h |
rate(last_executed_checkpoint[5m]) < 1.0
Validators and full nodes with slow checkpoint execution do not have up-to-date information from the network. It is usually due to network, disk, or CPU performance issues.
Safe mode during reconfiguration
| Key | Value |
|---|---|
| Name | Safe mode during reconfiguration |
| Severity | Immediate attention |
| Summary | Validator or full node failed to advance the epoch and entered safe mode |
| Duration | Recommended to trigger after 15m |
is_safe_mode > 0.5
Usually this issue is outside the control of validator or full node operators. Please notify Sui community on Discord when you observe this.
Which alerts can wait for normal maintenance?
All alerts are important, but you can address the following alerts and warnings within the normal node maintenance workflow.
System invariant violations
| Key | Value |
|---|---|
| Name | System invariant violations |
| Severity | Normal maintenance |
| Summary | The node reports an invariant violation |
| Duration | Recommended to trigger after 15m |
system_invariant_violations > 0
Usually this issue is outside the control of validator or full node operators. Please notify Sui community on Discord when you observe this.