Builders
Node Operators
Node Management
Monitoring

Node Metrics and Monitoring

The DEVIIUM Chain devium--batcher exposes a variety of metrics to help observe the health of the system and debug issues. Metrics are formatted for use with Prometheus and exposed via a metrics endpoint. The default metrics endpoint is http://localhost:7300/metrics.

To enable metrics, pass the --metrics.enabled flag to the devium--batcher. You can customize the metrics port and address via the --metrics.port and --metrics.addr flags, respectively.

Important Metrics

To monitor the health of your node, you should monitor the following metrics:

  • op_node_default_refs_number: This metric represents the devium--batcher's current L1/L2 reference block number for different sync types. If it stops increasing, it means that the node is not syncing. If it goes backwards, it means your node is reorging.
  • op_node_default_peer_count: This metric represents how many peers the devium--batcher is connected to. Without peers, the devium--batcher cannot sync unsafe blocks and your node will lag behind the sequencer as it will fall back to syncing purely from L1.
  • op_node_default_rpc_client_request_duration_seconds: This metric measures the latency of RPC requests initiated by the devium--batcher. This metric is important when debugging sync performance, as it will reveal which specific RPC calls are slowing down sync. This metric exposes one timeseries per RPC method. The most important RPC methods to monitor are:
    • engine_forkChoiceUpdatedV1, engine_getPayloadV1, and engine_newPayloadV1: These methods are used to execute blocks on devium-geth. If these methods are slow, it means that sync time is bottlenecked by either devium-geth itself or your connection to it.
    • eth_getBlockByHash, eth_getTransactionReceipt, and eth_getBlockByNumber: These methods are used by the devium--batcher to fetch transaction data from L1. If these methods are slow, it means that sync time is bottlenecked by your L1 RPC.

Available Metrics

A complete list of available metrics is below:

METRICDESCRIPTIONLABELSTYPE
devium_node_default_infoPseudo-metric tracking version and config infoversiongauge
devium_node_default_up1 if the devium node has finished starting upgauge
devium_node_default_rpc_server_requests_totalTotal requests to the RPC servermethodcounter
devium_node_default_rpc_server_request_duration_secondsHistogram of RPC server request durationsmethodhistogram
devium_node_default_rpc_client_requests_totalTotal RPC requests initiated by the opnode's RPC clientmethodcounter
devium_node_default_rpc_client_request_duration_secondsHistogram of RPC client request durationsmethodhistogram
devium_node_default_rpc_client_responses_totalTotal RPC request responses received by the opnode's RPC clientmethod,errorcounter
devium_node_default_l1_source_cache_sizeL1 Source cache cache sizetypegauge
devium_node_default_l1_source_cache_getL1 Source cache lookups, hitting or nottype,hitcounter
devium_node_default_l1_source_cache_addL1 Source cache additions, evicting previous values or nottype,evictedcounter
devium_node_default_l2_source_cache_sizeL2 Source cache cache sizetypegauge
devium_node_default_l2_source_cache_getL2 Source cache lookups, hitting or nottype,hitcounter
devium_node_default_l2_source_cache_addL2 Source cache additions, evicting previous values or nottype,evictedcounter
devium_node_default_derivation_idle1 if the derivation pipeline is idlegauge
devium_node_default_pipeline_resets_totalCount of derivation pipeline resets eventscounter
devium_node_default_last_pipeline_resets_unixTimestamp of last derivation pipeline resets eventgauge
devium_node_default_unsafe_payloads_totalCount of unsafe payloads eventscounter
devium_node_default_last_unsafe_payloads_unixTimestamp of last unsafe payloads eventgauge
devium_node_default_derivation_errors_totalCount of derivation errors eventscounter
devium_node_default_last_derivation_errors_unixTimestamp of last derivation errors eventgauge
devium_node_default_sequencing_errors_totalCount of sequencing errors eventscounter
devium_node_default_last_sequencing_errors_unixTimestamp of last sequencing errors eventgauge
devium_node_default_publishing_errors_totalCount of p2p publishing errors eventscounter
devium_node_default_last_publishing_errors_unixTimestamp of last p2p publishing errors eventgauge
devium_node_default_unsafe_payloads_buffer_lenNumber of buffered L2 unsafe payloadsgauge
devium_node_default_unsafe_payloads_buffer_mem_sizeTotal estimated memory size of buffered L2 unsafe payloadsgauge
devium_node_default_refs_numberGauge representing the different L1/L2 reference block numberslayer,typegauge
devium_node_default_refs_timeGauge representing the different L1/L2 reference block timestampslayer,typegauge
devium_node_default_refs_hashGauge representing the different L1/L2 reference block hashes truncated to float valueslayer,typegauge
devium_node_default_refs_seqnrGauge representing the different L2 reference sequence numberstypegauge
devium_node_default_refs_latencyGauge representing the different L1/L2 reference block timestamps minus current time, in secondslayer,typegauge
devium_node_default_l1_reorg_depthHistogram of L1 Reorg Depthshistogram
devium_node_default_transactions_sequenced_totalCount of total transactions sequencedgauge
devium_node_default_p2p_peer_countCount of currently connected p2p peersgauge
devium_node_default_p2p_stream_countCount of currently connected p2p streamsgauge
devium_node_default_p2p_gossip_events_totalCount of gossip events by typetypecounter
devium_node_default_p2p_bandwidth_bytes_totalP2P bandwidth by directiondirectiongauge
devium_node_default_sequencer_building_diff_secondsHistogram of Sequencer building time, minus block timehistogram
devium_node_default_sequencer_building_diff_totalNumber of sequencer block building jobscounter
devium_node_default_sequencer_sealing_secondsHistogram of Sequencer block sealing timehistogram
devium_node_default_sequencer_sealing_totalNumber of sequencer block sealing jobscounter