Set up Prometheus and Grafana for an Arc node

Arc is currently in its testnet phase. During this period, the network may experience instability or unplanned downtime. Throughout this page, all references to Arc refer specifically to the Arc Testnet.

Deploy Prometheus and Grafana in Docker on the same host as your Arc node, scrape metrics from the Execution Layer (EL) and Consensus Layer (CL), and load the pre-built Arc dashboards. The stack binds to 127.0.0.1 only and is reached over SSH port forwarding from a remote operator workstation.

Prerequisites

You have a running Arc node that meets the node requirements, set up either from the Run an Arc node tutorial or deployed as a systemd service
You have installed Docker Engine 24+ with BuildKit and Docker Compose v2 on the node host
Your node is started with --metrics 127.0.0.1:9001 on the EL and --metrics 127.0.0.1:29000 on the CL (the default in Run an Arc node)
You have cloned the arc-node repository on the host, to reuse the pre-built Grafana dashboards

Step 1: Verify the metrics endpoints

Confirm both metrics endpoints are reachable on the host before deploying the stack. The Execution Layer serves metrics at the root path, not at /metrics:

curl -s http://127.0.0.1:9001 | head
curl -s http://127.0.0.1:29000/metrics | head

Both commands return Prometheus-formatted text. If either fails, fix the node startup or --metrics flag before continuing. See Monitor a node for troubleshooting.

Step 2: Create the monitoring directory

This guide stores all monitoring files under $ARC_HOME/monitoring. Create the directory layout:

ARC_HOME="${ARC_HOME:-$HOME/.arc}"
ARC_MONITORING="${ARC_MONITORING:-$ARC_HOME/monitoring}"

mkdir -p "$ARC_MONITORING"/grafana-provisioning/datasources
mkdir -p "$ARC_MONITORING"/grafana-provisioning/dashboards
mkdir -p "$ARC_MONITORING"/prometheus-data
mkdir -p "$ARC_MONITORING"/dashboards

Step 3: Copy the pre-built Arc dashboards

The arc-node repository ships ready-to-use Grafana dashboards. From the arc-node repository root, copy them into the monitoring directory:

cp -r deployments/monitoring/config-grafana/provisioning/dashboards-data/* \
  "$ARC_MONITORING"/dashboards/

Step 4: Configure Prometheus

Write the following file to $ARC_MONITORING/prometheus.yml. The EL job scrapes the root path; the CL job scrapes /metrics:

global:
  scrape_interval: 1s

scrape_configs:
  - job_name: "arc_execution"
    metrics_path: "/"
    scrape_interval: 1s
    static_configs:
      - targets: ["127.0.0.1:9001"]
        labels:
          client_name: "reth"
          client_type: "execution"

  - job_name: "arc_consensus"
    metrics_path: "/metrics"
    scrape_interval: 1s
    static_configs:
      - targets: ["127.0.0.1:29000"]
        labels:
          client_name: "malachite"
          client_type: "consensus"

Step 5: Configure the Grafana datasource

Write the following file to $ARC_MONITORING/grafana-provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: prometheus
    uid: prometheus
    type: prometheus
    url: http://127.0.0.1:9090
    isDefault: true
    editable: true

Step 6: Configure dashboard provisioning

Write the following file to $ARC_MONITORING/grafana-provisioning/dashboards/default.yml:

apiVersion: 1

providers:
  - name: "arc"
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 10
    options:
      path: /var/lib/grafana/dashboards

Step 7: Create the Docker Compose file

Write the following file to $ARC_MONITORING/compose.yaml:

services:
  prometheus:
    image: prom/prometheus
    user: "0"
    network_mode: host
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.listen-address=127.0.0.1:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus-data:/prometheus
    restart: unless-stopped

  grafana:
    image: grafana/grafana-oss
    network_mode: host
    environment:
      GF_SERVER_HTTP_ADDR: 127.0.0.1
      GF_SERVER_HTTP_PORT: 3000
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
    restart: unless-stopped

The Arc metrics endpoints in this guide bind to 127.0.0.1 only. network_mode: host is the simplest way for Prometheus to scrape them without changing the Arc node configuration or exposing metrics publicly.

Step 8: Start the stack

cd "$ARC_MONITORING"
docker compose up -d

Confirm both containers are running:

docker compose ps

Check that Grafana and Prometheus are healthy:

curl -s http://127.0.0.1:3000/api/health
curl -s http://127.0.0.1:9090/-/ready

Step 9: Access the dashboards

The stack binds Grafana and Prometheus to 127.0.0.1 only. They are not exposed to the public internet. If you are operating on the same machine, open:

http://127.0.0.1:3000 for Grafana
http://127.0.0.1:9090 for Prometheus

If you are connecting from a remote workstation, use SSH port forwarding:

ssh -N \
  -L 3000:127.0.0.1:3000 \
  -L 9090:127.0.0.1:9090 \
  user@YOUR_SERVER_IP

Then open on your local machine:

http://localhost:3000
http://localhost:9090

Grafana default credentials from the Compose file above are username admin and password admin. Change them before exposing Grafana through any reverse proxy or shared access setup. After Prometheus scrapes both endpoints and Grafana loads the provisioned dashboards, you see metrics for execution layer activity, consensus layer activity, block height, validator telemetry, and connected peers. Newly started Prometheus instances need a short time to collect enough samples; if some panels are empty immediately after startup, wait a minute and refresh.

Troubleshooting

Browser shows `ERR_CONNECTION_REFUSED`

If you are connecting to a remote server, confirm you are opening http://localhost:3000 and http://localhost:9090 on your local machine, not http://YOUR_SERVER_IP:3000. Also confirm the SSH tunnel is still running.

Prometheus restarts with a permissions error

If Prometheus logs include an error about queries.active or write permission under /prometheus, fix ownership of the local data directory:

sudo chown -R 65534:65534 "$ARC_MONITORING"/prometheus-data
docker compose restart prometheus

Grafana is healthy but dashboards show no data

Confirm Prometheus targets are up:

curl -s http://127.0.0.1:9090/api/v1/targets

Re-check the Arc metrics endpoints:

curl -s http://127.0.0.1:9001 | head
curl -s http://127.0.0.1:29000/metrics | head

If Prometheus reaches the targets but some panels remain empty, wait for more samples to accumulate and refresh.

Check Arc node health directly

You can verify the Arc node independently of Grafana:

cast block-number --rpc-url http://127.0.0.1:8545
sudo journalctl -u arc-execution -f
sudo journalctl -u arc-consensus -f

Exchanges

On/off-ramps

Wallets

DeFi

Relayers and paymasters

Infrastructure

Run a node

Set up Prometheus and Grafana for an Arc node

Prerequisites

Step 1: Verify the metrics endpoints

Step 2: Create the monitoring directory

Step 3: Copy the pre-built Arc dashboards

Step 4: Configure Prometheus

Step 5: Configure the Grafana datasource

Step 6: Configure dashboard provisioning

Step 7: Create the Docker Compose file

Step 8: Start the stack

Step 9: Access the dashboards

Troubleshooting

Browser shows `ERR_CONNECTION_REFUSED`

Prometheus restarts with a permissions error

Grafana is healthy but dashboards show no data

Check Arc node health directly

​Prerequisites

​Step 1: Verify the metrics endpoints

​Step 2: Create the monitoring directory

​Step 3: Copy the pre-built Arc dashboards

​Step 4: Configure Prometheus

​Step 5: Configure the Grafana datasource

​Step 6: Configure dashboard provisioning

​Step 7: Create the Docker Compose file

​Step 8: Start the stack

​Step 9: Access the dashboards

​Troubleshooting

​Browser shows ERR_CONNECTION_REFUSED

​Prometheus restarts with a permissions error

​Grafana is healthy but dashboards show no data

​Check Arc node health directly

Prerequisites

Step 1: Verify the metrics endpoints

Step 2: Create the monitoring directory

Step 3: Copy the pre-built Arc dashboards

Step 4: Configure Prometheus

Step 5: Configure the Grafana datasource

Step 6: Configure dashboard provisioning

Step 7: Create the Docker Compose file

Step 8: Start the stack

Step 9: Access the dashboards

Troubleshooting

Browser shows `ERR_CONNECTION_REFUSED`

Prometheus restarts with a permissions error

Grafana is healthy but dashboards show no data

Check Arc node health directly