Optimizing Cisco UCS Management Pack for larger(any) environments

Some months ago Cisco released a brand new management pack for the UCS hardware platform, this update was more or less a rewrite of the entire solution, including switching all alarms from monitor based to rule based. This was done in order to minimize the number of discoveries needed and therefor the objects created by the management pack, this could have quite a performance impact since every little memory module and fan was discovered as objects, and this could be many fans and memory blocks in a large UCS environment.

Since it is all rule based now we would miss the auto-resolve feature usually used by monitors, but to solve this Cisco created a rule using a simple schedule to close and update alerts as needed based on the event entries on the server running the Cisco UCS monitoring service. But with the first release (4.0.1) there was a problems, it was only checking for alerts with resolution state New (0) and i guess in most production environments many are using resolution states to route alerts to correct teams or set the progress of the alert, therefor when you changed resolution state on your alerts they would never update or close. This problem is now solved in version 4.1.1 but now a new problem appears! Cisco choose to use “Where-Object {$_.ResolutionState -ne 255” to find all the non-closed alerts, using this method in our large environment it takes 10-12 seconds to find all the closed alerts and to make it even worse this command is running for every event collected by the rule “Cisco.Ucs.Watcher.UCSFault.Event.Collection.Rule” for the selected interval meaning that the script would never complete in our case and end on timeout.

To resolve all this I found all the elements needed for the rule “Cisco.Ucs.Library.UpdateAndCloseAlert.Rule” in the original Cisco Management pack and created my own fix for this problem.

Instead of

$scomActiveAlerts = Get-SCOMAlert | Where-Object {$_.ResolutionState -ne 255}

I changed it to

$ActiveAlerts = Get-SCOMAlert -ResolutionState (0..254)

This simple change gives the same result but in 0,5 seconds instead of 10-12. And instead of running the command for each event collected I changed it to run one time for the script (just before the ForEach) and save the result in a variable $ActiveAlerts and use that in the ForEach instead.

This just shows how easy you can improve (and decrease) the performance of your PowerShell scripts.

I uploaded my fix management pack to GitHub, if you import this one remember to disable the original rule.

GitHub Download

 

 

MPTool: Automate Windows Service Monitoring

Monitoring a Windows Service running state is a common request at my work, and therefor a obvious thing to automate and win back some free time. Thanks to the MPTool PowerShell module this can easily done with four lines of PowerShell

First you need a management pack to place it in, either use a existing or create a new. In this example we will place all resources, class, discovery and monitor, in the same management pack, for more complex management packs you will usually split into separate management packs.

New-MPToolManagementPack -ManagementServerFQDN scomms01.cloudmechanic.net `
-ManagementPackName "CM.MyApp.ServiceMonitoring" `
-ManagementPackDisplayName "CM MyApp Service Monitoring" `
-ManagementPackDescription "Contains monitor for MyApp Windows service Monitor" 

Once you have a management pack to place the monitor in, you need a class for the specific Windows service, you don’t need to add any properties for just basic service monitoring since the name of the Windows service cannot exist more than once on a Windows computer.

New-MPToolLocalApplicationClass -ManagementServerFQDN scomms01.cloudmechanic.net `
-ManagementPackName "CM.MyApp.ServiceMonitoring" `
-ClassName "CM.MyApp.WindowsService" `
-ClassDisplayName "CM MyApp Windows Service" `
-ClassDescription "CM MyApp Windows Service - Created with MPTool" 

We now have a class for our Windows service and we can then go ahead and create the discovery to find where the service is existing. All Windows Services has a key under HKLM:SYSTEM\CurrentControlSet\Services\[NameOfTheService] so this can be used to discover the service. Note that this should be the service name and not the display name.

New-MPToolFilteredRegistryDiscovery -ManagementServerFQDN scomms01.cloudmechanic.net `
-ManagementPackName "CM.MyApp.ServiceMonitoring" `
-DiscoveryName "CM.MyApp.WindowsService.Discovery" `
-TargetClassName "Microsoft.Windows.Computer" `
-RegistryPath "SYSTEM\CurrentControlSet\Services\MyApp\" `
-DiscoveryClassName "CM.MyApp.WindowsService" `
-IntervalSeconds 300 

Now the final step is to create a monitor for the discovered instances of the Windows service. Here you simply just need to target the newly created class, select the state of the alert if it triggered, Error or Warning, and last again defined the name of the Windows service to monitor the state of.

New-MPToolWindowsServiceMonitor -ManagementServerFQDN scomms01.cloudmechanic.net `
-ManagementPackName "CM.MyApp.ServiceMonitoring" `
-TargetClassName "CM.MyApp.WindowsService" `
-UnhealthyState "Error" `
-ServiceName "MyApp" 

Of course now we want to put it all together in a script that we can use for self-service scenarios! I have created one example and placed  my GitHub for MPTool.

Happy automating!

 

MPTool, Automate and Simplify Management Pack Development

This is something I have been looking forward to for a long time! The first public release of the PowerShell module MPTool, it is a module for automating and simplify OpsMgr management pack development I have been working on for almost a year together with Martin Dyrlund Arnoldi

Back in the end of 2014 we started introducing automation and self-service into our company and spend the most of 2015 automating common tasks around our daily server provisioning and de-provisioning work. In the end of 2015 we looked into automation for SCOM as we couldn’t keep up with the work the way we did it at the current point. We needed a way to automate our management pack creation to be able to create them faster and more standardized, and so we came by the awesome PowerShell module OpsMgrExtended from Tao Yang. We started to play around with Tao’s module and quickly figured out that the concept of module was exactly what we needed, but also that we required the ability to create more advanced monitoring solutions, so we decided to build our own PowerShell module, and the result is this, MPTool.

MPTool is created with the goal that if you know PowerShell you should pretty easily be able to create management packs. It contains a lot of build-in logic in each function, it automatically adds management pack references if needed, for a PowerShell discovery you only need to create a PowerShell array with the discovery data and the function it self will create the SCOM specific code needed, and so on.

We have now with great success used it internally for more than 6 months and is now ready to share it with the community, I hope you will all find it as useful as us!

Over the next weeks I will post some more articles on how you can use this module in different examples

Please report if you hit any Bugs or troubles so we can improve where it is needed, also if you have any idea for future releases we will take it in and evaluate.

Detailed documentation is available at GitHub

Download Link

GitHub Repository

16 CmdLets is available now, and a few more is already in testing and will be added soon.

New-MPToolManagementPackAlias
Get-MPToolManagementPackReferenceAlias
New-MPToolManagementPack
New-MPToolOverrideManagementPack
Add-MPToolManagementPackReference
New-MPToolApplicationComponentClass
New-MPToolComputerRoleClass
New-MPToolLocalApplicationClass
New-MPToolClass
New-MPToolWindowsEventAlertRule
New-MPToolFilteredRegistryDiscovery
New-MPToolPSDiscovery
New-MPToolPSStateMonitor
New-MPToolWindowsServiceMonitor
New-MPToolDependencyMonitor
New-MPToolHostingRelationship

 

Cleaning up unused management packs.

I recently did some house keeping in our OpsMgr environment, cleaning up old and unused Management Packs, and to easily identify potential unused Management Packs I of course turned to PowerShell!

This script can walk through the management packs and get the number of instances for each class. If there is no instances of a class it is very likely that it is not used. After running this script you will end up with a list of potential unused classes, you can now check each management pack for it’s content and see if there is other classes in it that is in use.

################################################################
#   Author: Andreas Sobczyk, CloudMechanic.net
#
#   Get Empty classes from Management Packs
################################################################

## Get all management packs starting with Contoso.
$MPs = Get-SCOMManagementPack -name Contoso*

## For each MP, For each Class, count classinstances.
$Overview = @()
Foreach($MP in $MPs){
$Classes = $mp.GetClasses()

foreach($Class in $Classes){
$ClassInstance = $null
$ClassInstance = Get-SCOMClassInstance -Class $class

## If you want you can change the number of class instances to sort on.
if($ClassInstance.Count -le 0){
$Overview += $Class
}
}
}

$Overview | Format-Table Name,Displayname,ManagementpackName

GitHub Download

Maintenance Mode Desktop Shortcut

In my daily work it is a very common problem that people often forget to start maintenance mode in OpsMgr for a server when they are working on it. This results in too many “false” alarms for our Operation Services Team that handle the alarms. The second problem is that we have many people managing servers and not all of them has permissions access to OpsMgr, but only to access the servers they manage. Therefor they cannot put the servers in maintenance mode them selves.

To solve these two issues i came up with this maintenance mode tool. The solution consists of two components, a small .Net application to place on the servers, and a management pack for OpsMgr. It requires no permissions for the end user in OpsMgr, but it does requires .Net 3.0 or higher to be installed on the servers for the application to work.

The application opens a little form with two fields, first is the end time for the maintenance mode (default it sets +30 min from current datetime), and second is a comment field that will be added to the maintenance mode comment field in OpsMgr.
snip_20161011214716
When you click OK what will then happen is that a event will be created in the event log with EventID 19999 in the OperationsManager log with the information entered.

The management pack then contains a event collection rule, CloudMechanic Collect Maintenance Mode Events, that will pick up these events.
It also contains a Event View for these events so you can easily get a overview of them.
snip_20161011220956

A second rule, CloudMechanic MaintenanceMode Tool Trigger, runs on a schedule every 240 second and will execute a PowerShell script that will set the maintenance mode for the computer based on the comment entered in the application and the username which did it.
snip_20161011221538

The rule CloudMechanic MaintenanceMode Tool Trigger contains some overrideable parameters so you can change the timers to fit your need.

  • IntervalSeconds – The interval for how often the script runs.
  • EventQueryIntervalInSeconds – The interval for how long back in the the script looks for events, this must always be higher that the IntervalSeconds parameter.
  • TimeoutSeconds – Timeout for the script.
  • Logging –  Enables logging for the script for debugging.
    snip_20161011223036

I also added a PowerShell script for installing the application on the servers and create a desktop shortcut. You can create a package in SCCM and use this script to deploy the application to your servers.snip_20161012140626

Download Link

GitHub Link