Introduction
You can accomplish so much with Ansible, and it's so simple to get started with it. The more you use it, the more it becomes a part of you. Prometheus, grafana, and node-exporter are all good options for a monitoring stack. Because you can use a small or large scale to practise a variety of standard modules.
I will guide you how you can use Ansible script to install Grafana and Prometheus in your Host/Main monitor server and install node_exporter in other server that you would like to monitor.
Use-Case
Step 1: Set with static inventory file and Pinging the Target Nodes
- As a best practice, I prefer to create a new folder for each project and create a config file in it.
Note: Remember that Ansible will process the below list and use the first file found, all others are ignored.
ANSIBLE_CONFIG (environment variable if set)
ansible.cfg (in the current directory)
~/.ansible.cfg (in the home directory)
/etc/ansible/ansible.cfg
- So, make a directory named grafana-prometheus under the home directory and switch into it.
$ mkdir dynamic-inventory
$ cd dynamic-inventory
Create a file named inventory.txt
$ sudo vi inventory.txt
- Paste the content below into the inventory.txt file.
[monitorserver]
db_server ansible_host=<YOUR-DB-SERVER-IP> ansible_user=ec2-user ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>
[nodeservers]
server1 ansible_host=<YOUR-WEB-SERVER-IP> ansible_user=ec2-user ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>
server2 ansible_host=<YOUR-WEB-SERVER-IP> ansible_user=ec2-user ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>
Note: Don’t forget the change the IP addresses of the target nodes and the path of your pem key.
- Create file named
ansible.cfgunder the thedynamic-inventorydirectory. - Paste the content below into ansible.cfg file.
[defaults]
host_key_checking = False
inventory=inventory.txt
interpreter_python=auto_silent
localhost_warning=false
Validate and check the inventory.
ansible-inventory --graph
- Check the connectivity to the target nodes.
$ ansible all -m ping
- So, that's great! We have connected to the target nodes with a static inventory.
Step 2: Create a role to install node-exporter
To install a binary or a tarball with an IaC or configuration manager such as ansible, you must first master a basic case. A good example is the node exporter. First and foremost, the ansible-galaxy command aids in the construction of a role's skeleton:
ansible-galaxy init roles/node-exporter
Then we can set some defaults variables in the default directory :
node_exporter_version: "1.1.2"
node_exporter_bin: /usr/local/bin/node_exporter
node_exporter_user: node-exporter
node_exporter_group: "{{ node_exporter_user }}"
node_exporter_dir_conf: /etc/node_exporter
And now the main file of tasks directory :
- name: check if node exporter exist
stat:
path: "{{ node_exporter_bin }}"
register: __check_node_exporter_present
- name: create node exporter user
user:
name: "{{ node_exporter_user }}"
append: true
shell: /usr/sbin/nologin
system: true
create_home: false
- name: create node exporter config dir
file:
path: "{{ node_exporter_dir_conf }}"
state: directory
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
- name: if node exporter exist get version
shell: "cat /etc/systemd/system/node_exporter.service | grep Version | sed s/'.*Version '//g"
when: __check_node_exporter_present.stat.exists == true
changed_when: false
register: __get_node_exporter_version
- name: download and unzip node exporter if not exist
unarchive:
src: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
dest: /tmp/
remote_src: yes
validate_certs: no
- name: move the binary to the final destination
copy:
src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
dest: "{{ node_exporter_bin }}"
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
mode: 0755
remote_src: yes
when: __check_node_exporter_present.stat.exists == false or not __get_node_exporter_version.stdout == node_exporter_version
- name: clean
file:
path: /tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/
state: absent
- name: install service
template:
src: node_exporter.service.j2
dest: /etc/systemd/system/node_exporter.service
owner: root
group: root
mode: 0755
notify: reload_daemon_and_restart_node_exporter
- meta: flush_handlers
- name: service always started
systemd:
name: node_exporter
state: started
enabled: yes
We need to create node_exorter.service.j2 file in teplates directory :
[Unit]
Description=Node Exporter Version {{ node_exporter_version }}
After=network-online.target
[Service]
User={{ node_exporter_user }}
Group={{ node_exporter_user }}
Type=simple
ExecStart={{ node_exporter_bin }}
[Install]
WantedBy=multi-user.target
Finally the handler file in the handler directory :
- name: reload_daemon_and_restart_node_exporter
systemd:
name: node_exporter
state: restarted
daemon_reload: yes
enabled: yes
Step 3: Create a role for prometheus and its configuration
We can initialize a prometheus role :
ansible-galaxy init roles/prometheus
We can set some defaults variables in the default directory :
prometheus_dir_configuration: "/etc/prometheus"
prometheus_retention_time: "365d"
prometheus_scrape_interval: "30s"
prometheus_node_exporter: true
prometheus_node_exporter_group: "all"
prometheus_env: "production"
prometheus_var_config:
global:
scrape_interval: "{{ prometheus_scrape_interval }}"
evaluation_interval: 5s
external_labels:
env: '{{ prometheus_env }}'
scrape_configs:
- job_name: prometheus
scrape_interval: 5m
static_configs:
- targets: ['{{ inventory_hostname }}:9090']
Since we define a part of the prometheus configuration, the header exactly with prometheus_var_config.Now I create tasks in the main.yml file of tasks directory :
- name: update and install prometheus
apt:
name: prometheus
state: latest
update_cache: yes
cache_valid_time: 3600
- name: prometheus args
template:
src: prometheus.j2
dest: /etc/default/prometheus
mode: 0644
owner: root
group: root
notify: restart_prometheus
- name: prometheus configuration file
template:
src: prometheus.yml.j2
dest: "{{ prometheus_dir_configuration }}/prometheus.yml"
mode: 0755
owner: prometheus
group: prometheus
notify: reload_prometheus
- name: start prometheus
systemd:
name: prometheus
state: started
enabled: yes
Then, we I create the prometheus.yaml.j2 file:
#jinja2: lstrip_blocks: "True"
{{ prometheus_var_config | to_nice_yaml(indent=2) }}
{% if prometheus_node_exporter_group %}
- job_name: node
scrape_interval: 15s
metrics_path: /metrics
static_configs:
- targets:
{% for server in groups[prometheus_node_exporter_group] %}
- '{{ server }}:9100'
{% endfor %}
{% endif %}
And the prometheus.j2 file for the prometheus CLI :
ARGS="--web.enable-lifecycle --storage.tsdb.retention.time={{ prometheus_retention_time }} --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries
Finally handlers of this prometheus role;We have two handlers :
- Restart with the systemd service
- Reload with a curl on the prometheus API.
- name: restart_prometheus
systemd:
name: prometheus
state: restarted
enabled: yes
daemon_reload: yes
- name: reload_prometheus
uri:
url: http://localhost:9090/-/reload
method: POST
status_code: 200
Step 4: Create a role for Grafana
Now we can create a last role to install grafana-server package and start it. Just edit the main.yml file in the tasks directory :
- name: install gpg
apt:
name: gnupg,software-properties-common
state: present
update_cache: yes
cache_valid_time: 3600
- name: add gpg hey
apt_key:
url: "https://packages.grafana.com/gpg.key"
validate_certs: no
- name: add repository
apt_repository:
repo: "deb https://packages.grafana.com/oss/deb stable main"
state: present
validate_certs: no
- name: install grafana
apt:
name: grafana
state: latest
update_cache: yes
cache_valid_time: 3600
- name: start service grafana-server
systemd:
name: grafana-server
state: started
enabled: yes
- name: wait for service up
uri:
url: "http://127.0.0.1:3000"
status_code: 200
register: __result
until: __result.status == 200
retries: 120
delay: 1
- name: change admin password for grafana gui
shell : "grafana-cli admin reset-admin-password {{ grafana_admin_password }}"
register: __command_admin
changed_when: __command_admin.rc !=0
Don't forget to set your Admin password for start and you can set it in default directory:
grafana_admin_password: "abc1234"
Step 5: Create Ansible Playbook
Create an Ansible playbook file like below :
- name: install monitoring stack
hosts: monitorserver
become: yes
roles:
- prometheus
- grafana
- name: install node-exporter
hosts: nodeservers
become: yes
roles:
- node-exporter
Step 6: Testing
You can run this playbook file via below command:
ansible-playbook -i inventory.txt playbook.yml
devops4me@ % ansible-playbook -i inventory.txt playbook.yml
/usr/local/Cellar/ansible/5.7.1/libexec/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
"class": algorithms.Blowfish,
PLAY [install monitoring stack] ****************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : update and install prometheus] **********************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : prometheus args] ************************************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : prometheus configuration file] **********************************************************************************************************
ok: [18.136.207.64]
TASK [prometheus : start prometheus] ***********************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : install gpg] *******************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : add gpg hey] *******************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : add repository] ****************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : install grafana] ***************************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : start service grafana-server] **************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : wait for service up] ***********************************************************************************************************************
ok: [18.136.207.64]
TASK [grafana : change admin password for grafana gui] *****************************************************************************************************
ok: [18.136.207.64]
PLAY [install node-exporter] *******************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]
TASK [node-exporter : check if node exporter exist] ********************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]
TASK [node-exporter : create node exporter user] ***********************************************************************************************************
[WARNING]: 'append' is set, but no 'groups' are specified. Use 'groups' for appending new groups.This will change to an error in Ansible 2.14.
ok: [54.251.28.26]
ok: [13.215.254.70]
TASK [node-exporter : create node exporter config dir] *****************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]
TASK [node-exporter : if node exporter exist get version] **************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]
TASK [node-exporter : download and unzip node exporter if not exist] ***************************************************************************************
changed: [13.215.254.70]
changed: [54.251.28.26]
TASK [node-exporter : move the binary to the final destination] ********************************************************************************************
skipping: [54.251.28.26]
skipping: [13.215.254.70]
TASK [node-exporter : clean] *******************************************************************************************************************************
changed: [13.215.254.70]
changed: [54.251.28.26]
TASK [node-exporter : install service] *********************************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]
TASK [node-exporter : meta] ********************************************************************************************************************************
TASK [node-exporter : service always started] **************************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]
PLAY RECAP *************************************************************************************************************************************************
13.215.254.70 : ok=9 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
18.136.207.64 : ok=12 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
54.251.28.26 : ok=9 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
Improvement
We can improve this soltuion by apply following steps:
1.Security:
1.1 Encrypt The Credentials/Password/Private key
Ansible Vault encrypts variables and files so you can protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in playbooks or roles
2.Agility:
2.1 Continuous Integration and Continuous Delivery:
Integrate our Ansible script with CI/CD and apply it to other projects or needs.
2.2 Disable fact gathering :
When a playbook executes, each play runs a hidden task, called gathering facts, using the setup module. This gathers information about the remote node you're automating, and the details are available under the variable ansible_facts. But if you're not using these details in your playbook anywhere, then this is a waste of time. You can disable this operation by setting gather_facts: False in the play.
Conclusion
Finally, with all above steps we have learned :
- How to create Dynamic Ansible Inventory file.
- How to provision and installed Grafana & Prometheus using Ansible script
- How to automatically installed Node Exporter via Ansible
Additional:
You may find the full code for this post from here-> DevOps4Me Global Code
