Monitoring HP Storage environments with System Center Operations Manager 2012 R2 and the HP Storage Management Pack for System Center v 4.2.1 – Experience from the field
In this blog post, I want to share my personal experience with the HP Storage Management Pack for System Center and give you some hints, which could save you precious time in case you come across the same problems.
The HP Storage Management Pack for System Center in its current version (v 4.2.1, at the time of writing) is provided as part of the “HP OneView for Microsoft System Center v8.0” image download. It supports the monitoring of variety of different HP storages, like HP 3PAR StoreServ Storage, HP MSA Storage, HP StoreVirtual Storage, HP XP 9000 Storage, HP StoreEver Tape Library and HP StoreOnce Backup (for a full list of the supported storage environments, please refer to the HP Storage Management Pack for System Center User Guide v4.2.1).
Planing your design
During the planning phase, you have to decide how the design of the solution will look like. The discovery of the HP storages (excluding the HP StoreOnce) is done via the HP User Configuration Tool, which is installed during the installation of the management pack itself. It enables a nice Diagram View, on its turn provides a nice overview of your storage environment. In addition, if you want to gather all the events from your storages you will have to go for the so-called SNMP collection point.
The SNMP collection point could be also your SCOM Management Server, but only in cases, where the Management Server is NOT part of a Network Monitoring Resource Pool. The reason for this is that the SCOM Management Servers, which are part of a network device monitoring resource pool do not leverage the Windows SNMP stack, but use the Health Service (MonitoringHost.exe ) instead to listen for and process SNMP traps from network devices. Of course if you want the SCOM HealthService to process the SNMP traps from a device, this device has to be discovered as a network device via the discovery wizard so that it gets instantiated as a System.NetworkManagement$Node in the SCOM Operational Database.
This is also confirmed by Kevin Holman in his article:
SNMP Trap monitoring with SCOM 2012 R2
Using a SCOM Management Server (not participating in network monitoring) as a SNMP collection point would be possible too, but in my case, I have offloaded this to a dedicated SCOM managed server. The difference in this case is that you have to choose the option “Collection Point Server” when preparing your SNMP collection point:
I used a setup with System Center Operations Manager 2012 R2 UR6, running on Windows Server 2012 R2 Standard Edition in a restrictive Active Directory environment.
During the installation I have selected the option “OpsMgr Server” (opposite to the installation of the SNMP collection point) and selected the necessary management packs:
The installation wizard imported the desired management packs and, went smooth and I got a confirmation about a “successful installation” at the end of the wizard:
“The InstallShield Wizard has successfully installed HP Storage Management Pack for System Center. Click Finish to exit the wizard.”
Next, I had to use the HP User Configuration Tool in order to be able to discover my storage environment. As a first step, a SCOM management had to be specified, which than appears in the “Management Server” pane on the left side of the tool:
The surprise came after I decided to use the “Add HP Storage System” option of the User Configuration Tool to add the first storage device:
After selecting the correct storage device and entering its IP address and connection, credentials the HP User Configuration Tool experienced an unhandled exception and crashed with the following error message:
“Service SCOMMPServer was not found on computer’ <IP address of the SCOM Management Server>’.”
Some of the stack trace entries revealed details, like:
The first thing was to check the presence of a service with this specific name, but unfortunately, it did not exist. So at this point I remembered seeing the reference to the Windows Installer log at the end of the installation wizard. The information found in the log indicated that an attempt has been made to install a service with this specific name:
MSI (s) [15:47:56:218]: Executing op: FileCopy(SourceName=SCOMMP~1.EXE|SCOMMPServerw.exe,SourceCabKey=scommpserverw.exe,DestName=SCOMMPServerw.exe,Attributes=16384,FileSize=104448,PerTick=65536,,VerifyMedia=1,,,,,CheckCRC=0,Version=188.8.131.52,Language=1033,InstallMode=58982400,,,,,,,)
MSI (s) [15:47:56:218]: File: D:\Program Files (x86)\HP\HP Storage Management Pack for System Center\apache-tomcat-7.0.42\bin\SCOMMPServerw.exe; To be installed; Won’t patch; No existing file
MSI (s) [15:47:56:218]: Source for file ‘scommpserverw.exe’ is compressed
Unfortunately, I was not able to find any source of information, describing this behavior and the related error message, neither in the community, nor in the guide, provided with the management pack itself.
At this stage, I decided to have a closer look on the content of the folder, referenced by the installation log entry (…Program Files (x86)\HP\HP Storage Management Pack for System Center\apache-tomcat-7.0.42\bin\SCOMMPServerw.exe):
The service.bat file seemed exactly what I needed so I examined its content and was pleased to find, that it actually contained the code for the installation of the service “SCOMMPServer”. I used the simple command line to trigger the installation:
“Service.bat install SCOMMPServer”
After completing these steps, I was able to successfully add the storage devices in the HP User configuration tool:
Therefore, now that the MP has been installed, I started wondering on what the cause could be. My account was surely member of the local “Administrators” group on the server and I was sure that I have elevated the installation in order to avoid potential permission issues. My next guess was that something is preventing the InstallShield from installing the service and this could only be a Group Policy Setting, being applied on the Management Server.
Indeed, after blocking the inheritance of all GPOs, ensuring that there are no enforced GPOs and cleaning the GPO settings from the registry I was able to confirm a clean installation of the SCOMMPService as a part of the Management Pack installation process.
Unfortunately, I had no possibility to check exactly which GPO setting was causing the issue, but this information should be sufficient for you to know where to look for the cause, in case you encounter this same problem.
Management pack workflow redundancy
Along with the HP Storage Management Pack for System Center, I was able to test also the HPE OneView SCOM Integration Kit and I must say that it was a pleasant surprise to see that they have used the concept of SCOM resource pools to distribute the management pack workflows and provide redundancy on application level.
Unfortunately, this is not the case with HP Storage Management Pack (v 4.2.1.), where you can add only one Management Server in the HP User Configuration Tool and it is the only instance doing your discoveries and handling the workflows.
By examining the files in the installation directory, I was able to tell that the Configuration Tool uses xml-based configuration file, named “ConfigFile.xml”, which defines the SCOM Management Server:
All my attempts to make alter it in order to achieve some kind of redundancy were left with no result. The same applies to the SNMP collection point, which also cannot be made redundant. Hopefully this will change in the upcoming releases.
Supported SNMP trap versions
As already mentioned the HP Storage Management Pack offers the possibility to deliver storage events in form of SNMP traps. In my setup, I have configured the SNMP collection point and configured all three storage devices (HP 3PAR StoreServ, HP StoreVirtual and HP StoreOnce) to forward the events to it.
The “HP Storage Management Pack for System Center User Guide v4.2.1” explicitly mentions that all HP 3PAR StoreServ storage alerts and alert state changes are converted into SNMPv2 traps and forwarded to the registered SNMP collection server. The specific I came across concerns the delivery of the events from the HP StoreVirtual and HP StoreOnce storage devices. Shortly after configuring the SNMP target (the collection server) on those particular devices, I noticed that the events are either delivered in an intermittent way or in the most cases – not delivered at all. Doing a network capture revealed that the SNMP traps are actually reaching the network interface on the collection point, but the respective workflows do not handle them properly. After hours of troubleshooting, I got my hands on statement from the development team, which says that SNMPv1 is currently the only supported way of delivering SNMP traps for HP StoreOnce and HP StoreVirtual. So if you encounter the same issue, please be sure to configure your storage devices (HP StoreOnce and HP StoreVirtual) to send only SNMPv1 traps. This way you will ensure steady delivery of storage events to your SCOM infrastructure.
If you need to monitor your HP storage devices in SCOM, using the HP Storage Management Pack for System Center please ensure that the following topics have been covered:
- You have configured a SNMP collection point, which either a dedicated server or SCOM Management server that is not part of network monitoring resource pool.
- You have ensured that there are no GPO settings, which might prevent the installation of the SCOMMPServer service on your SCOM Management Server.
- Port 8080 (TCP) is opened on both your SCOM management Server and your SNMP collection point server. Filtering this port will lead to issues with the web service communication and as a result the discovery will not work and your storage diagram view will never get updated. There is a certain danger in this, as it will always stay “green” giving you a false feeling that everything with your storage is fine.
- You have ensured that your HP StoreVirtual and HP StoreOnce storage devices are configured to send SNMPv1 traps to your SNMP collection point. Otherwise, the delivery of the events from those devices cannot be guaranteed.