Recovery Guide
If your node falls too far behind the rest of the network or its database becomes corrupted then you will need to recover it from the latest state snapshot. The Supra Foundation maintains the following publicly accessible snapshots for each production network:
Testnet
Node Type
Scope
Version
Snapshot Source
Default
RPC
Full Historical State
v8.0
testnet-snapshot
Y
Full Historical State
v9.0
testnet-archive-snapshot
Pruned RPC
Latest 1008 Epochs¹
Coming Soon
Coming Soon
Y
Validator
Latest 84 Epochs²
v9.0
testnet-validator-snapshot (current default for validators)
Y
¹ ≈ 3 months
² ≈ 1 week
Mainnet
Node Type
Scope
Version
Snapshot Source
Default
RPC
Full Historical State
v7.1
mainnet
Y
Pruned RPC
Latest 1008 Epochs¹
Coming Soon
Coming Soon
Y
Validator³
Latest 84 Epochs²
Coming Soon
Coming Soon
Y
¹ ≈ 3 months
² ≈ 1 week
³ Validators can be synced from RPC snapshots. Use the `mainnet` snapshot source until validator-specific snapshots are available.
The manage_supra_nodes.sh sync command can be used to retrieve the latest snapshot. This command currently defaults to the testnet-snapshot bucket for testnet RPC nodes and testnet-validator-snapshot for testnet validators, and the mainnet bucket for both types of mainnet nodes. A specific bucket can be specified via the --snapshot-source optional argument.
After selecting the snapshot source that is relevant to your node’s version and the network that it is participating in, follow the steps below to sync your node.
Syncing the Latest Snapshot
First, stop your node:
docker stop "$CONTAINER_NAME"Then sync the latest snapshot from your selected snapshot source. You can omit the --snapshot-source optional parameter if you wish to sync from the default source for the type, version and network of your node. For validators, run:
./manage_supra_nodes.sh sync --snapshot-source <snapshot_source> validator <host_supra_home> <network>And for RPC nodes, run:
./manage_supra_nodes.sh sync --snapshot-source <snapshot_source> rpc <host_supra_home> <network>After the command completes, restart your node and monitor its logs to ensure that it is syncing the latest blocks correctly. If your node produces an error, first refer to the Troubleshooting section below, then, if your issue cannot be resolved by the instructions provided there, reach out to Supra Support.
Troubleshooting Guide
Resolving Database Corruption Errors
The sync script will occasionally leave the database in an inconsistent state after terminating. You will see an error similar to the below upon starting your node when this happens:
RpcNode error: Store error: Database Change error:
DBError{Error { message: "Corruption:IO error: No such file or directory:
./configs/rpc_store/453665.sst
./configs/rpc_store/MANIFEST-446443 may be corrupted."}}This can generally be resolved by repeating the sync with the addition of the --exact-timestamps optional argument. This argument will cause the script to perform a more thorough sync that will retrieve all files from the remote snapshot that have last-modified timestamps that differ from the timestamps of the corresponding local files. This will generally take longer than a normal sync. It will also continue to sync the same files if run repeatedly due to a limitation in the third-party tools that we rely on for this process. Accordingly, we do not recommend repeatedly syncing with this argument active.
./manage_supra_nodes.sh sync --exact-timestamps --snapshot-source <snapshot_source> <node_type> <host_supra_home> <network>Should the error persist, try a standard sync (i.e. without --exact-timestamps) one more time. This should now complete quickly.
./manage_supra_nodes.sh sync --snapshot-source <snapshot_source> <node_type> <host_supra_home> <network>If the error still persists after the final sync, please contact Supra Support.
Last updated