Monitoring

This section outlines how to monitor the status of the BIFROST Node and Relayer. It will be useful for debugging, adjusting, and understanding what actually happens in the BIFROST Node and Relayer.
​
Prerequisites
To follow the instructions on this page, the following conditions must be met.
Docker must be installed
You need to be running the BIFROST Node with the `—prometheus-external’ flag
(for full validators) You need to be running the BIFROST Relayer with the ‘-p’ flag
Basic operating knowledge of bash/terminal
​
Components for health checking
There are three software components to monitor BIFROST Node and Relayer:
1.Prometheus server gathers data in the form of a time-based database. This data represents the health of the BIFROST Node and Relayer.
2.Alertmanager notifies anomalies to the operator when the Prometheus server detects them by analyzing the collected information.
3.Grafana dashboard visualizes the collected data from the Prometheus server, allowing easy access.
​
Prepare docker stack
Download
The docker-compose.yml file is provided to build a monitoring system. Download the file using the command below.
$ git clone https://github.com/bifrost-platform/bifrost-node.git
Monitoring system files located in the ${projectroute}/maintenance directory.
$ cd bifrost-node/maintenance
Alertmanager
You need to decide how to be notified when the server instance goes down or when the CPU usage goes beyond a certain limit.
Please check the alertmanger/alertmanager.yml file.
global:
  resolve_timeout: 1m
​
route:
  receiver: 'gmail-notifications'
​
receivers:
  - name: 'gmail-notifications'
    email_configs:
      - to: '' # receiver email
        from: '' # sender(monitoring system) gmail
        smarthost: 'smtp.gmail.com:587'
        auth_username: '' # sender(monitoring system) gmail
        auth_identity: '' # sender(monitoring system) gmail
        auth_password: '' # sender(monitoring system) gmail's app password <https://support.google.com/mail/answer/185833?hl=en>
        send_resolved: true
The setting to receive notifications via Gmail has been preconfigured. Refer to the link to set the password and fill in the blank.
If you wish to receive notifications via other methods (Slack, Webhooks), refer to the link to write out a configuration file.
Prometheus
You can omit this process if are not a full-node validator and if the Prometheus export port is in the default setting.
Please check the prometheus/prometheus.yml file.
global:
  scrape_interval: 3s
  evaluation_interval: 3s
​
rule_files:
  - 'rules.yml'
​
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager:9093"
​
scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 3s
    static_configs:
      - targets: [ "localhost:9090" ]
  - job_name: "substrate_node"
    scrape_interval: 3s
    static_configs:
      - targets: [ "host.docker.internal:9615" ]
  - job_name: "node_exporter"
    scrape_interval: 3s
    static_configs:
      - targets: [ "node-exporter:9100" ]
#  - job_name: "relayer"
#    scrape_interval: 3s
#    static_configs:
#      - targets: [ "host.docker.internal:8000" ]
If you changed the export port of the BIFROST Node, change the number 9615 in targets: [ "host.docker.internal:9615" ] to the port number you have set.
If you are a full-node validator, delete all “#”s to uncomment the relayer part. If you changed the export port of the relayer, change 8000 in host.docker.internal:8000 to the port you have set.
Start containers
If you have resolved all processes above, return to the maintenance pathway and execute the following command.
$ docker compose up -d
You have succeeded in setup if you see the following message.
[+] Running 10/10
 ⠿ Network maintenance_default             Created                                                                                                 0.0s
 ⠿ Volume "maintenance_mysql-data"         Created                                                                                                 0.0s
 ⠿ Volume "maintenance_grafana-data"       Created                                                                                                 0.0s
 ⠿ Volume "maintenance_alertmanager-data"  Created                                                                                                 0.0s
 ⠿ Volume "maintenance_prometheus-data"    Created                                                                                                 0.0s
 ⠿ Container maintenance-node-exporter-1   Started                                                                                                 0.7s
 ⠿ Container maintenance-db-1              Started                                                                                                 0.7s
 ⠿ Container maintenance-alertmanager-1    Started                                                                                                 0.8s
 ⠿ Container maintenance-prometheus-1      Started                                                                                                 1.0s
 ⠿ Container maintenance-grafana-1         Started                                                                                                 1.3s
​
Setup grafana dashboard
Access http://localhost:3000 via a browser.
Login
The screenshot above shows the login page. The default login account and password have been set to admin/ admin. You can set the password after initial authentication.
After setting you can log in with admin/{set password}.
Add data sources
Untitled
Untitled
This is the page you will see once you log in. Click on the settings icon on the left sidebar to access “Add data source”.
Untitled
First, add Prometheus. Click to proceed.
Untitled
Untitled
Fill in the URL section with “http://prometheus:9090” and click on “Save & test” at the bottom of the page to save.
Untitled
Return to “Add data sources” to find Prometheus AlertManager which should be located at the very bottom.
Untitled
Fill in the URL section with “http://alertmanager:9093” and click on “Save & test” at the bottom of the page to save.
Import dashboard
Untitled
You will see “Import” if you hover over the plus icon on the left sidebar. Click to proceed.
Untitled
Enter “17725” right below “Import via grafana.com”, and clock on the “Load” button on the right.
Untitled
Select the data source of Prometheus AlertManager and Prometheus and it will show the data source added during the previous step. Click on the “Import” button below after selecting.
Monitoring
Untitled
The dashboard has been successfully imported if you see this screen.
Untitled
You can view the collected metrics by expanding ‘Server health’, ‘Chain’, ‘Relayer - Health’, ‘Relayer - Protocol’.
Some metrics are not shown on the default dashboard. You may change the dashboard setting to edit the visible metrics.
Testnet syncing
Nominators
Last modified 1mo ago