> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arc.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Set up Prometheus and Grafana for an Arc node

> Deploy a local Prometheus and Grafana stack on the same host as your Arc node to scrape metrics and view the pre-built dashboards.

<Info>
  Arc is currently in its testnet phase. During this period, the network may
  experience instability or unplanned downtime. Throughout this page, all
  references to Arc refer specifically to the Arc Testnet.
</Info>

Deploy Prometheus and Grafana in Docker on the same host as your Arc node,
scrape metrics from the Execution Layer (EL) and Consensus Layer (CL), and load
the pre-built Arc dashboards. The stack binds to `127.0.0.1` only and is reached
over SSH port forwarding from a remote operator workstation.

## Prerequisites

* You have a running Arc node that meets the
  [node requirements](/arc/references/node-requirements), set up either from the
  [Run an Arc node](/arc/tutorials/run-an-arc-node) tutorial or
  [deployed as a systemd service](/arc/tutorials/deploy-node-as-service)
* You have installed
  [Docker Engine 24+ with BuildKit](https://docs.docker.com/engine/install/) and
  [Docker Compose v2](https://docs.docker.com/compose/install/) on the node host
* Your node is started with `--metrics 127.0.0.1:9001` on the EL and
  `--metrics 127.0.0.1:29000` on the CL (the default in
  [Run an Arc node](/arc/tutorials/run-an-arc-node))
* You have cloned the [`arc-node`](https://github.com/circlefin/arc-node)
  repository on the host, to reuse the pre-built Grafana dashboards

## Step 1: Verify the metrics endpoints

Confirm both metrics endpoints are reachable on the host before deploying the
stack. The Execution Layer serves metrics at the root path, not at `/metrics`:

```shell theme={null}
curl -s http://127.0.0.1:9001 | head
curl -s http://127.0.0.1:29000/metrics | head
```

Both commands return Prometheus-formatted text. If either fails, fix the node
startup or `--metrics` flag before continuing. See
[Monitor a node](/arc/tutorials/monitor-a-node) for troubleshooting.

## Step 2: Create the monitoring directory

This guide stores all monitoring files under `$ARC_HOME/monitoring`. Create the
directory layout:

```shell theme={null}
ARC_HOME="${ARC_HOME:-$HOME/.arc}"
ARC_MONITORING="${ARC_MONITORING:-$ARC_HOME/monitoring}"

mkdir -p "$ARC_MONITORING"/grafana-provisioning/datasources
mkdir -p "$ARC_MONITORING"/grafana-provisioning/dashboards
mkdir -p "$ARC_MONITORING"/prometheus-data
mkdir -p "$ARC_MONITORING"/dashboards
```

## Step 3: Copy the pre-built Arc dashboards

The `arc-node` repository ships ready-to-use Grafana dashboards. From the
`arc-node` repository root, copy them into the monitoring directory:

```shell theme={null}
cp -r deployments/monitoring/config-grafana/provisioning/dashboards-data/* \
  "$ARC_MONITORING"/dashboards/
```

## Step 4: Configure Prometheus

Write the following file to `$ARC_MONITORING/prometheus.yml`. The EL job scrapes
the root path; the CL job scrapes `/metrics`:

```yaml theme={null}
global:
  scrape_interval: 1s

scrape_configs:
  - job_name: "arc_execution"
    metrics_path: "/"
    scrape_interval: 1s
    static_configs:
      - targets: ["127.0.0.1:9001"]
        labels:
          client_name: "reth"
          client_type: "execution"

  - job_name: "arc_consensus"
    metrics_path: "/metrics"
    scrape_interval: 1s
    static_configs:
      - targets: ["127.0.0.1:29000"]
        labels:
          client_name: "malachite"
          client_type: "consensus"
```

## Step 5: Configure the Grafana datasource

Write the following file to
`$ARC_MONITORING/grafana-provisioning/datasources/prometheus.yml`:

```yaml theme={null}
apiVersion: 1

datasources:
  - name: prometheus
    uid: prometheus
    type: prometheus
    url: http://127.0.0.1:9090
    isDefault: true
    editable: true
```

## Step 6: Configure dashboard provisioning

Write the following file to
`$ARC_MONITORING/grafana-provisioning/dashboards/default.yml`:

```yaml theme={null}
apiVersion: 1

providers:
  - name: "arc"
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 10
    options:
      path: /var/lib/grafana/dashboards
```

## Step 7: Create the Docker Compose file

Write the following file to `$ARC_MONITORING/compose.yaml`:

```yaml theme={null}
services:
  prometheus:
    image: prom/prometheus
    user: "0"
    network_mode: host
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.listen-address=127.0.0.1:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus-data:/prometheus
    restart: unless-stopped

  grafana:
    image: grafana/grafana-oss
    network_mode: host
    environment:
      GF_SERVER_HTTP_ADDR: 127.0.0.1
      GF_SERVER_HTTP_PORT: 3000
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
    restart: unless-stopped
```

<Note>
  The Arc metrics endpoints in this guide bind to `127.0.0.1` only.
  `network_mode: host` is the simplest way for Prometheus to scrape them without
  changing the Arc node configuration or exposing metrics publicly.
</Note>

## Step 8: Start the stack

```shell theme={null}
cd "$ARC_MONITORING"
docker compose up -d
```

Confirm both containers are running:

```shell theme={null}
docker compose ps
```

Check that Grafana and Prometheus are healthy:

```shell theme={null}
curl -s http://127.0.0.1:3000/api/health
curl -s http://127.0.0.1:9090/-/ready
```

## Step 9: Access the dashboards

The stack binds Grafana and Prometheus to `127.0.0.1` only. They are not exposed
to the public internet.

If you are operating on the same machine, open:

* `http://127.0.0.1:3000` for Grafana
* `http://127.0.0.1:9090` for Prometheus

If you are connecting from a remote workstation, use SSH port forwarding:

```shell theme={null}
ssh -N \
  -L 3000:127.0.0.1:3000 \
  -L 9090:127.0.0.1:9090 \
  user@YOUR_SERVER_IP
```

Then open on your local machine:

* `http://localhost:3000`
* `http://localhost:9090`

Grafana default credentials from the Compose file above are username `admin` and
password `admin`. Change them before exposing Grafana through any reverse proxy
or shared access setup.

After Prometheus scrapes both endpoints and Grafana loads the provisioned
dashboards, you see metrics for execution layer activity, consensus layer
activity, block height, validator telemetry, and connected peers. Newly started
Prometheus instances need a short time to collect enough samples; if some panels
are empty immediately after startup, wait a minute and refresh.

## Troubleshooting

### Browser shows `ERR_CONNECTION_REFUSED`

If you are connecting to a remote server, confirm you are opening
`http://localhost:3000` and `http://localhost:9090` on your local machine, not
`http://YOUR_SERVER_IP:3000`. Also confirm the SSH tunnel is still running.

### Prometheus restarts with a permissions error

If Prometheus logs include an error about `queries.active` or write permission
under `/prometheus`, fix ownership of the local data directory:

```shell theme={null}
sudo chown -R 65534:65534 "$ARC_MONITORING"/prometheus-data
docker compose restart prometheus
```

### Grafana is healthy but dashboards show no data

Confirm Prometheus targets are up:

```shell theme={null}
curl -s http://127.0.0.1:9090/api/v1/targets
```

Re-check the Arc metrics endpoints:

```shell theme={null}
curl -s http://127.0.0.1:9001 | head
curl -s http://127.0.0.1:29000/metrics | head
```

If Prometheus reaches the targets but some panels remain empty, wait for more
samples to accumulate and refresh.

### Check Arc node health directly

You can verify the Arc node independently of Grafana:

```shell theme={null}
cast block-number --rpc-url http://127.0.0.1:8545
sudo journalctl -u arc-execution -f
sudo journalctl -u arc-consensus -f
```