Zabbix Agent Over an SSH Tunnel

Today I set up Zabbix monitoring of a bunch of boxes. A couple have public IP addresses – the load-balancers – so they were pretty standard. However, most of them are sitting behind a NAT, so are a little trickier. I played around with Zabbix proxy for a while – whose purpose is to solve this exact problem. In the end though, just to be different/awkward, I opted to set up ssh tunnels and just pass the Zabbix traffic through the load-balancer boxes. The Zabbix agent config was fiddly and took a lot of trial and error to get right, so, maybe this will help someone else.
First of all, the ssh tunnels. This is the command I used, but you’ll probably want to set up ssh keys and use autossh or some such thing. This is run on the load-balancer box, as it’s acting as a bridge between the public network and the NAT’ed network.

ssh -f user@10.0.0.25 -L0.0.0.0:14050:10.0.0.25:10050 -N

Breaking this down:

-f : Background the ssh process

user@10.0.0.25 : these details are for the NAT’ed machine – the one we want to monitor

-L0.0.0.0:14050 : the IP/port that we should listen to on the load-balancer side

10.0.0.25:10050 : the IP/port that we should tunnel to on the client side – 10050 is the default Zabbix port

-N : Tell ssh to not run any commands, just set up a tunnel

So, we’re tunnelling from 14050 on the load-balancer to 10050 on the client.
Now the tricky bit – the Zabbix config. First of all the client. Open up the config file – /etc/zabbix/zabbix_agentd.conf and add the config bits below:

Server=10.0.0.25
Hostname=my-awesome-client-1
DisableActive=1

First of all, we’re setting ‘Server’ to the IP of the actual client itself. This one took me ages to get right. As we’re tunnelling through, the Zabbix requests appear to come from the client itself. If we don’t set this right, the Zabbix agent will just send back a ‘not authorised’ message to the server.
Next, you need to set ‘Hostname’ to the hostname of the box itself which should also be the same as the hostname you’ll give it in the Zabbix server-side config.
Lastly, with this setup, we don’t seem to be able to use active checks. I.e., checks initiated by the client and sent to the server. With this setup, the client is passive and just sits there waiting for the server to ask it to execute checks.
Now for the Zabbix server-side config. All of the ‘Items’ for your client have to be of the type ‘Zabbix agent’, *not* ‘Zabbix agent (active)’. As I was just saying, they all need to be server-initiated. To do this, I cloned an existing template and used ‘Mass update’ to set the ‘Type’ to ‘Zabbix agent’.
Next, create a host; Configuration > Hosts > Create Host. Link your template created above and set name and group as you normally would. Then set the following:

DNS name : same as Hostname from agent config – not sure if this is crucial, but it’s how I have it set.

IP Address : Set this to the IP address of the load-balancer (the box where you set up the tunnel).

Connect To : ‘IP address’

Zabbix Agent Port : The listening side of the tunnel on the load-balancer box, 14050 in my case.

And, in theory, that’s it. If you look at Monitoring > Overview, you should start seeing data trickle in.