prometheus_textfile#
Take data from salt and “return” it into a file formatted for Prometheus using the Text Exposition Format which rolls up state success and failure data.
New in version 3005.
The intended use case for this module is to have distributed success/failure reporting from minions for unattended state or highstate runs.
Add the following to the minion or master configuration file to configure the output location which Prometheus will monitor via Node Exporter.
prometheus_textfile.filename: <path_to_output_file>
Default is /var/cache/salt/minion/prometheus_textfile/salt.prom using the
cachedir minion configuration.
The salt_procs metric will look for salt-minion processes by name. If
you have a custom installation of Salt, you might want to change the psutil
process name to be matched or switch to matching the “exe” attribute of the
Process.info dictionary.
prometheus_textfile.proc_name: custom-minion
prometheus_textfile.match_exe: True
prometheus_textfile.exe: /opt/salt/bin/python3
The default operation of sending state metrics to a single file works well for the use case of running distributed highstate or a single state run on minions. However, there may be a use case for tracking multiple scheduled states in separate files. To enable this behavior, set the following option in the configuration file:
prometheus_textfile.add_state_name: True
This option will add the state name which was run as a filename suffix and also
inside the file as a metric parameter. Highstate runs will receive highstate
as the state name, while running specific states will pass the first argument to
state.apply or state.sls as the state name.
# Filename is "salt-highstate.prom" and metrics get '{state="highstate"}'
salt-call state.highstate
# Filename is "salt-highstate.prom" and metrics get '{state="highstate"}'
salt-call state.apply
# Filename is "salt-test.prom" and metrics get '{state="test"}'
salt-call state.apply test
# Filename is "salt-test.prom" and metrics get '{state="test"}'
salt-call state.sls test
Additionally, the inferred state name can be overridden on the command line by
passing the prom_textfile_state keyword argument to the state function.
# Filename is "salt-hello.prom" and metrics get '{state="hello"}'
salt-call state.highstate prom_textfile_state=hello
# Filename is "salt-hello.prom" and metrics get '{state="hello"}'
salt-call state.apply test prom_textfile_state=hello
Output file user, group, and mode can optionally be set through configuration options:
prometheus_textfile.uid: 0
prometheus_textfile.gid: 0
prometheus_textfile.mode: "0644"
The metrics can include the names of failed states if necessary. This is sometimes beneficial for monitoring purposes so engineers can see what in particular caused a failure condition on a host and whether it is critical.
prometheus_textfile.show_failed_states: true
An additional way of viewing failure conditions is through the
abort_state_ids option. If this option is used, a state ID or list of state
IDs can be provided to indicate an “abort” condition. This allows the user to
see that a failure was related to a “circuit breaker” condition which prevented
the state run to complete.
prometheus_textfile.abort_state_ids:
- circuit_breaker_state
- my_other_circuit_breaker
The state IDs in this configuration option can also be presented as a string in the following manner:
# comma-separated states
prometheus_textfile.abort_state_ids: circuit_breaker_state,my_other_circuit_breaker
# single state
prometheus_textfile.abort_state_ids: circuit_breaker_state
If you have systems running Salt versions which have a non-standard version scheme
(such as running from patched versions), the entire version string can be shown
by utilizing the raw_version parameter. Otherwise, only the portion of the
string before a plus sign (+) will be shown. So, a version string such as
“3004+12.g557e6cc0fc” will be shown as “3004” by default unless raw_version
is enabled.
prometheus_textfile.raw_version: true