We already learned [[how to add prometheus alerts at Grafana|Adding-alerts-in-Grafana]]. However, we had only tested for Prometheus data sources, and we want them also for logs.
Currently there’s only 1 grafana panel that supports loki datasource, “Logs”, and it looks like we couldn’t create a Graph Panel with some statistical processing. However:
After mailing grafana.com , they told us how to do it:
Unfortunately, it is still not possible to use these functions for Loki datasource. This function should be available in Grafana in future releases, when we add the metrics from logs feature. We do not yet have ETA for that though.
However, there is a sort of a workaround for this, where you can add Loki as a Prometheus datasource. Once that is added, you can use PromQL on logs and get metrics from logs that way.
Some information about that is available in this video:
https://www.youtube.com/watch?v=J_nAt0XX0qw
At grafana cloud dashboard, generate a new key.
[[img/prom-loki-api-key.png]]
/loki to the url[[img/prom-loki-config-data-source.png]]
Alarms are created asociated to a query of a panel inside a dashboard. Therefore, first you need a dashboard.
Inside Coopdevs grafana cloud, you can start from this demo dashboard.
Create or use a Graph panel that points to the new Prom-Loki datasource. You can set up a query like:
sum(rate({filename="/var/log/opencell.d/server.log", loglevel=~".+", host="$host_loki"} [10s])) by (loglevel)
With keywords:
sum()rate()filenameloglevelhost$host_lokiYou may see something like this
[[img/prom-loki-histogram.png]]
At this point, you can already follow the tutorial for Prometheus, [[Adding alerts in Grafana]]