For those unfamiliar with it Milestone XProtect is one of the most widely used video management software (VMS) on large scale environment.
Quoting from the company’s website:
The XProtect VMS products are video management software designed for installations of all shapes and sizes. Whether you want to protect your store from vandalism or you want to manage a multi-site, high security installation, XProtect makes it possible. The solutions offer centralized management of all devices, servers, and users, and provide an extremely flexible rule system driven by schedules and events.
It is therefore natural that, given its prevalence, sooner or later it might happen to have one active on environments managed by us as well. And in fact that is exactly what happened to me!
Now, as far as I have seen over the years, overseeing the operation of a VMS infrastructure is not strictly a matter that is handled on the IT side.
However, having some background with this product, even culminated with some certification in the past, I was hoping to find a fairly quick solution…
NOPE!
Don’t get me wrong, a native tool exists and is very good, it is
MilestonePSTools, a PowerShell module that wraps many functions of the
Milestone SDK.
Lately, however, I’ve been working with Zabbix a lot, natural that after searching in vain for a ready-made template, I said to myself, why not create one?
Zabbix and Milestone XProtect
The idea is to leverage the
Milestone SDK, which thank goodness is HTTP-based (btw thanks for the REST part, not so much for the SOAP part), to monitor the status of all components of the VMS infrastructure: individual servers, storage, cameras, services.
To do this we will use a Milestone user and all the power of Zabbix Discovery in order to make the configuration as streamlined as possible.
XProtect configuration
So let’s start by setting up a Role and a User on the Management Client that has the following permissions:
- Info ->
Allow Web Client Login
- Overall Security -> Management Server ->
Connect
- Overall Security -> Management Server ->
Status API
- Overall Security -> Cameras ->
Read
Zabbix configuration
Define, if not already present, a Host pointing to the Milestone Management Server IP or DNS and set three simple MACROs:
{$MILESTONE.USER}
- The user previously configured on XProtect{$MILESTONE.PASSWORD}
- The XProtect user password{$MILESTONE.CONN}
- The Milestone API connection protocol,https
(default) orhttp
Once this is done, associate this Host with the
Milestone XProtect Template and wait for the Discovery process to generate all the Items that we are going to monitor.
Let’s look at them in detail.
Milestone XProtect VMS - Zabbix monitoring
At first the Milestone XProtect Template Items can be counted on the fingers of one hand, there are literally 5 of them. Among these the one to be considered as key is one:
Milestone: XProtect Discovery
- The real kicker of this integration. It is this Script in fact that is responsible for enumerating all the configured VMS components and maintaining a valid authentication token to pass to the other Script Items created in Discovery.
If all goes as it should, when discovery is complete we will get numerous Items identified by different [tags]
depending on which VMS component they belong to:
- [MGM] - Management Server
- [REC] - Recording Server
- [CAM] - Camera
- [STR] - Storage
- [ARC] - Archive storage
- [MOB] - Mobile Server
- [SVC] - Service registered (e.g. Event Server, Report Server)
- [UPD] - Hotfixes available on Milestone website
TLS certificate verification
Security of data in transit is a must-have, which is why encryption via TLS certificates should be enabled on all Milestone components, and these certificates are monitored by this Zabbix template. How?
To date, the only native Zabbix solution is to use a Zabbix Agent 2 function, so the most obvious options are:
- Have an active Zabbix Agent 2 on the Milestone Management Server.
- Use Zabbix Agent 2 on the same Zabbix server.
Only in the second case a little customization needs to be done, proceed as follows:
- On the Host defined on Zabbix add a second Interface of type Agent with IP
127.0.0.1
- On Host’s Discovery Rules set this Host Interface on all Items with Key
web.certificate.get[{#SSL_HOSTNAME},{#SSL_PORT},{#SSL_IP}]
As shown in the figure we will thus constantly keep track of the expiration of the certificate and its thumbprint.
Monitoring of Storage and Archive Storage
Storage and its availability is one of the essential components of a VMS platform, so we do monitor the available space and status of each Storage or Archive resource configured on the Recording Servers. Especially the status which, mainly in the case of iSCSI or SMB, could become Unavailable or Unmounted.
Cameras status
We monitor the status of the cameras by requesting it directly from the Recording Server, so we get high-level detail that covers both a trivial Offline camera problem and more specific issues such as Not licensed or Media overflow.
Mobile Server
On the Milestone Mobile Server side, the existing APIs will provide a fair amount of detail on connected users, bandwidth usage, and resources consumed.
If, as it should be, the Mobile Server is configured for HTTPS there is nothing more to know, otherwise I strongly recommend customizing the Macro {$MILESTONE.MOBILE.DH.PRIME}
used for initial encryption of user and password.
{$MILESTONE.MOBILE.DH.PRIME}
- Mobile Server expects the username and password to be encrypted with a Diffie-Hellman-Merkle shared secret. This is the prime number used (length 1024 or 2048 bit) in hexadecimal format
Monitoring Event Server and other services
Many other services usually run on XProtect, whether they are native like the Event Server or developed by third parties, if they are registered correctly they are monitored by this template. The monitoring is limited to checking that the corresponding URI HTTP/S is reachable which usually means that the service is running.
Milestone Hotfix Notifications
The availability of updates, especially if they concern security patches, must always be kept under control. This is exactly what the latest Discovery available in the Milestone XProtect template takes care of: it periodically checks for patches published on the Milestone website and notifies via Trigger the presence of new Hotfixes.
For quick reference, the descriptive detail of the latest fixes is also provided as visible in the image.
Wrapping up
The Milestone XProtect Template for Zabbix, while still open to improvement, provides out-of-the-box 360° monitoring of XProtect installations.
It should be compatible with all XProtect versions that implement ApiGateway and IDP, if you encounter problems or have a chance to improve/test it feel free to open Issues and PR on GitHub in full FOSS spirit!