The Metrika Agent is a configurable software agent that regularly collects metrics and events from the host system as well as the blockchain node running on it. This data is then exported to one or more external sources via an Exporter API. By default the Metrika Agent sends data to the Metrika Platform for Blockchain Operational Intelligence. Find out more and create an account for free.
This is the list of all currently supported blockchains (several coming soon and we welcome contributions!):
The Metrika Agent can be installed either as a systemd service or as a Docker container.
To install and run the agent as a systemd service you can download and run the Installation Script using this one-liner:
MA_BLOCKCHAIN=flow MA_API_KEY={api_key} bash -c "$(curl -L https://raw.githubusercontent.com/Metrika-Inc/agent/main/install.sh)"The script serves as installer by default, but can also be used with flags --upgrade or --uninstall. In its default mode it does the following:
- Determines the latest version published on Github and downloads its binary for
{blockchain}. - Creates a the
metrikaduser and group on the system, - Sets up Metrika Agent as a
systemdservice. - Creates and populates
/etc/metrikad: - Creates and populates
/opt/metrikadwith the Metrika Agent binary. - Starts Metrika Agent service.
The agent can run as a standalone binary, as long as configuration files are set up correctly.
These instructions require running Caddy server as Docker proxy. Before proceeding, make sure a proxy is listening on a DOCKER_HOST address and forwards requests to the host Docker daemon non-networked socket. You can find instructions about a recommended setup here.
To install and run the agent as a systemd service without adding metrikad user to docker group, use the following one-liner:
MA_PLATFORM={platform_endpoint} DOCKER_HOST=tcp://127.0.0.1:2379 DOCKER_API_VERSION=1.41 MA_BLOCKCHAIN=flow MA_API_KEY={api_key} bash -c "$(curl -sL https://raw.githubusercontent.com/Metrika-Inc/agent/main/install.sh)" -- --no-docker-grpBy default the agent will add metrikad user to systemd-journal group to enable access to journald logs. Similarly to Docker, you can explicitly disable this behavior with --no-systemd-journal-grp flag, for example:
MA_PLATFORM={platform_endpoint} MA_BLOCKCHAIN=flow MA_API_KEY={api_key} bash -c "$(curl -sL https://raw.githubusercontent.com/Metrika-Inc/agent/main/install.sh)" -- --no-systemd-journal-grpNote: On startup, if metrikad is in systemd-journal group, the agent will force set discovery.systemd.deactivated=true since journal logs will not be available.
To run the agent in a Docker container using the latest image use this one-liner:
docker run --rm \
-d \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc:/host/proc:ro \
-v /sys:/host/sys:ro \
-e MA_PLATFORM={platform_endpoint} \
-e MA_API_KEY={api_key} \
--name metrikad-flow \
--network host \
ghcr.io/metrika-inc/agent:latest-flow ./metrikad-flow -procfs /host/proc -sysfs /host/sysSimilarly to the corresponding systemd section, these instructions require setting up a docker proxy. You can find more about a recommended setup here.
To run the agent in a Docker container with a user that has no elevated permissions use this one-liner:
docker run --rm \
-d \
-v /proc:/host/proc:ro \
-v /sys:/host/sys:ro \
-e MA_PLATFORM={platform_endpoint} \
-e MA_API_KEY={api_key} \
-e DOCKER_HOST=$DOCKER_HOST \
-e DOCKER_API_VERSION=1.41 \
--name metrikad-flow \
--network host \
--user metrikad \
ghcr.io/metrika-inc/agent:latest-flow ./metrikad-flow -procfs /host/proc -sysfs /host/sysTo avoid adding the metrikad user to the docker group, you can use a reverse proxy. The reverse proxy filters traffic before it reaches the non-networked Docker socket, preventing the agent process from speaking directly to it. The reverse proxy still needs a user that belongs to the docker group to run it so it can proxy requests to the UNIX socket.
The following process assumes a Debian based host and describes the steps required to install the agent, using Caddy as a reverse proxy to the Docker daemon. The Caddyfile used is the least required configuration needed by the agent to perform its container discovery operations.
-
Install Caddy on your system by following these instructions.
-
Add user
caddytodockergroup:
usermod --append --groups docker caddy- Download the recommended Caddyfile and replace
<ma_container>with the name of the blockchain node container. Finally moveCaddyfileto/etc/caddy/Caddyfile:
curl -sL https://raw.githubusercontent.com/Metrika-Inc/agent/main/caddy/Caddyfile | sed 's/<ma_container>/example-name/g' | tee /etc/caddy/CaddyfileThis Caddyfile filters requests to the Daemon based on the API paths needed by the agents.
- Restart Caddy:
systemctl restart caddy- Use the following one-liner to install the Metrika Agent bypassing adding
metrikaduser todockergroup and configuringDOCKER_HOSTenvironment variable.
DOCKER_HOST=tcp://127.0.0.1:2379 DOCKER_API_VERSION=1.41 MA_BLOCKCHAIN={protocol} MA_API_KEY={api_key} bash -c "$(curl -sL https://raw.githubusercontent.com/Metrika-Inc/agent/main/install.sh) --no-docker-grp"- Test the proxy forwards requests to the Docker daemon for
metrikaduser:
sudo -u metrikad docker -H tcp://127.0.0.1:2379 ps- Test
metrikaduser cannot access the non-networked Docker Socket:
sudo -u metrikad docker psThe agent loads its configuration by looking at the following sources in order:
- Load configuration
agent.ymlfrom./configsor current directory. The first to be found takes effect. If no files exist, proceed to step 2. - Load configuration from environment variables and overload any parameter set (except exporters).
- Ensure sane defaults are set for configuration parameters that has not been set by a configuration file or an environment variable.
- Ensure loaded configuration has all required parameters set.
Customization is possible by modifying the agent.yml found in /etc/metrikad/configs/agent.yml after the installation, or before installation with environment variables passed to the installation script. All configuration parameters (except runtime.exporters) can be overloaded by environment variables prefixed by MA_. For example, to overload sampling frequency set MA_RUNTIME_SAMPLING_INTERVAL=30s.
By default, the agent uses sensible defaults to detect the blockchain node by probing all supported daemons or service managers. Currently the agent can detect nodes that are run as:
- Docker container. Requires write access to daemon's UNIX socket.
- Systemd service. Requires access to journald via Dbus connection.
discovery.systemd and discover.docker are required and used to hint the node discovery process executed by the agent.
A watcher is responsible for collecting metrics or events from a single source at regular intervals. Watchers are composable - a watcher can collect data from another watcher to do additional transformations on data.
By default, Metrika Agent starts watchers for system metrics and a subset of blockchain-specific metrics and events.
Be advised that the Exporter API is work in progress and may change without notice.
All the data points collected by the enabled watchers are passed to one or more exporters, as defined by the Exporter interface in exporter_registry.go.
By default, the only enabled exporter is Metrika Platform exporter, which encodes the data as protocol buffers (proto definition) and transmits them to Metrika Platform.
To disable Metrika Platform exporter, set the value of platform.enabled configuration parameter to false.
To enable any other exporter, specify its configuration under runtime.exporters. Example:
runtime:
# ...
exporters:
file_stream_exporter:
output_path: "/var/metrikad/agent.log"More on exporter implementations can be found in CONTRIBUTING.md
The agent exposes an HTTP endpoint under <listen_addr>/write that serves both as a reverse proxy to an upstream InfluxDB server and as a Prometheus exporter. It uses influxdb_exporter and exposes <listen_addr>/metrics which returns any metrics sent to /write endpoint in Prometheus Exposition Format. By default, the agent will:
- Proxy requests to
127.0.0.1:8086/writeto Solana's upstream InfluxDB server (i.e. https://metrics.solana.com:8086). - Transform to Influx metrics to Prometheus Exposition Format (PEF) and expose it under
127.0.0.1:8086/metrics.
To install and run the agent as a systemd service you can download and run the Installation Script using this one-liner:
MA_BLOCKCHAIN=solana bash -c "$(curl -L https://raw.githubusercontent.com/Metrika-Inc/agent/main/install.sh)" -- --prereleaseThe script serves as installer by default, but can also be used with flags --upgrade or --uninstall.
To run the agent in a Docker container using the latest image use this one-liner:
docker run --rm \
-d \
--name metrikad-solana \
--network host \
-p 8086:8086 \
ghcr.io/metrika-inc/agent:latest-solana ./metrikad-solanaInfluxDB default configuration can be overridden by configuring the corresponding watcher under runtime.watchers section. For example:
- type: influx
# exporter_activated: bool, activates influx watcher.
exporter_activated: true
# listen_addr: string, address to listen for InfluxDB /write requests. For example: 127.0.0.1:8086.
listen_addr: 127.0.0.1:8086
# upstream_url: string, URL of upstream InfluDB server to proxy /write requests. For example: https://127.0.0.1:8087. (optional). To deactivate the reverse proxy to forward write requests to an upstream InfluxDB server, `upstream_url` must be explcitly set to an empty value.
upstream_url: https://metrics.solana.com:8086Node discovery is not supported and the relevant functionality is deactivated by default for Solana binaries.
Docker images are signed by Metrika using Github's sigstore integration. Images can be verified with cosign following the steps below:
- Install cosign by following these instructions.
- Verify the image using Metrika's cosign public key:
cat <<EOT >> ma-cosign.pub
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEP6OfMlstCuLkvrKVahxxvnaDW+Tw
wUJusBmsWSOMQVnYDb/jVfL/e/rgQxvFbulNiHGIRF3IXWzE3Jr8wTjdrg==
-----END PUBLIC KEY-----
EOT
cosign verify --key ma-cosign.pub ghcr.io/metrika-inc/agent:<tag>You can read more details on verifying containers with cosign here.
Agent issues can be troubleshot with the help of logs. By default agent logs can be accessed via:
journalctl -u metrikad-{blockchain}.serviceRequires: runtime.http_addr.
You can modify the logging level without restarting the agent by sending an HTTP PUT request to the /loglvl endpoint:
curl -X PUT 127.0.0.1:9999/loglvl -d level=debugRequires: runtime.http_addr and runtime.metrics_enabled.
The agent listens for HTTP requests on the address specified by runtime.metrics_addr. You can scrape Prometheus metrics by hitting the /metrics endpoint, for example:
curl 127.0.0.1:9999/metrics # when runtime.http_addr=127.0.0.1:9999When runtime.http_addr is set, by default the agent will validate the Host header of incoming HTTP requests against a list of allowed hosts configured by runtime.allowed_hosts. In this case, a request without an allowed Host header will be rejected by the agent with HTTP 400.
Host header validation can be disabled by setting runtime.host_header_validation_enabled to false, in which case runtime.allowed_hosts has no effect.
The Metrika Agent attempts to discover a supported running blockchain node in the system. In case the blockchain node is containerized, Agent attempts to find the container by matching container name or image to a list of regular expressions specified in discovery configuration key. If the container name is not in the list of common names Metrika Agent is aware of, it can be added to the list of discovery.docker.regex.
Please note that for containerized blockchain nodes, the agent needs to either:
- Be added to the
dockergroup OR - a suitable docker proxy needs to run on the host enabling partial access to the Docker API, and the
DOCKER_HOSTenvironment variable needs to be correctly set to point to said proxy, in order to allow the Metrika Agent to retrieve blockchain node log and configuration files.
For issues pertaining to the agent itself, feel free to open up an Issue here on Github and we will try and help you reach a resolution. If you are experiencing issues with the Metrika Platform please use this form.
See CONTRIBUTING.md
Reach out to us via Discord!
Metrika Agent is licensed under the terms of Apache 2.0 License.