There was an interesting discussion about putting all groups for overrides in a sealed MP. At first, you might think it’s a great idea because you can reuse the same groups in different MPs and to be honest, when I started with SCOM 2007 several years ago, I was also tempted to design overrides and groups in such a way.

The Basics

The previously mentioned discussion also links to a blog post about where to put those groups in. This post perfectly describes the behavior of objects in unsealed MPs and the behavior of referencing objects in sealed MPs:

When you create a group, you save it to an unsealed management pack. However, an element in an unsealed management pack, such as a group, cannot reference an element in a different unsealed management pack, such as an override or a view. If you are going to use a group to target an override or scope a view, you must either save the group to the same unsealed management pack as the override or view, or you must seal the management pack that contains the group.
 
If you save the group to the same unsealed management pack as the override or view, you can only use that group for overrides and views that are also contained in that unsealed management pack.
 
If you seal the management pack that contains the group, you can reference that group from other unsealed management packs. However, you cannot easily change any group settings in the sealed management pack or add new groups to the sealed management pack.
 
Before you create any groups in Operations Manager, you should plan what groups you will need, the purpose each group will serve, and how to save the group in a manner that will allow it to be used for its intended purpose.

The above is valid for SCOM 2007, SCOM 2007 SP1, SCOM 2007 R2 and upcoming SCOM 2012 and it just hits the spot. When you read this you might still consider using all your groups for overrides from a sealed MP. I’m not saying that this is wrong. There are situations where you need to do that but in my experience these situations are very rare and very special. Anyway, to find out how to seal an MP, take a look at David Allen’s blog post: http://wmug.co.uk/blogs/aquilaweb/archive/2009/07/21/sealing-a-management-pack-in-r2.aspx

The Danger of having one single sealed MP for Overrides

Let me outline what implications this has, especially when you need to maintain those groups:

• Once you’ve sealed an MP containing groups, you cannot edit those groups in the operations console. You cannot change explicit membership, dynamic membership rules, etc.
• You cannot easily remove the sealed MP with your groups from your management group. To remove it you also need to remove all override MPs which are using those groups
• You need to be careful modifying the sealed group MP. You cannot change a group ID. This will break update compatibility. You also cannot easily remove a group from your MP. Adding new groups is no problem though.
• When you want to remove a group, you first need to export and edit all your override MPs which are referencing the group and remove all overrides which are using this group. Only after re-import of the modified override MPs, an update of the sealed MP without the group will work.
• You need to keep your unsealed MP around and when you want to modify your sealed MP you always edit the unsealed MP. Since you cannot have both installed in one management group, you most likely need to modify your MP in the authoring console. Creating and modifying groups in the authoring console is by far not that easy as in the operations console.
• Always increase the version number before you seal your modified MP and update your MP in the management group. If you do not do that, you might end up with groups not being updated, etc.
• As stated above, you cannot easily remove the sealed group MP, so you just update your existing one in the management group.

As you can see, sealing group management packs has its challenges.

Best Practice for Overrides

In SCOM you need to plan carefully and think about structuring your MPs, naming conventions and organization. As a best practice you should always create a dedicated override management pack for each sealed custom or 3rd party management pack.

For example, if you use the Active Directory management pack, create a management pack called “Active Directory Overrides” (or “Overrides – Active Directory” depending what naming convention you prefer) to store all the overrides in that management pack. It’s important to keep a consisting naming convention for the MP naming as it will help you later to find your stuff. Create those override MPs for each sealed MP you import – best would be right after the import! Microsoft is even considering to ship their MPs with empty override MPs from the start which may help users to follow this pattern right away.

Most sealed MPs have already their own groups (based on application discovery) which can be used for your overrides. If you need custom groups for Active Directory overrides, you need to create them in the same override MP as the overrides (see “The Basics” above). At first this might let you think that this is counter productive. The group you create in this MP will not be available in other unsealed MPs. And you’re right if you think “what if I need the same group in another override MP because these are the same servers?”. Yes. you need to create the same group again in the other override MP - but this is a good thing! Consider, you want to remove an MP permanently because you do not have the application anymore. Having the application MPs and overrides MPs stringed together allows you to easily remove or update an application MP in your management group – without affecting anything else. It’s a self-containing set of objects, completely independent from the other MPs.

This makes maintenance and lifecycle management of applications and management packs much more easier!

And one more thing: DO NOT EVER USE THE DEFAULT MANAGEMENT PACK! (I think this is already a well known fact…)

cheers,
Stefan
http://www.code4ward.net

This forum post http://www.systemcentercentral.com/Default.aspx?tabid=177&IndexId=90237 inspired me to write this blog post. Sometimes, especially when you create script based rules or monitors, you need to ensure the script runs under a specific account. In SCOM there’s something called “Run As” Accounts and “Run As” Profiles. Some management packs require you to setup run as accounts and assign them to run as profiles in order to successfully monitor the application. If you do not set run as accounts, all the workflows (monitors, rules, etc.) run with the default action account.

So you might wonder, if they can do that, can I do that too? Short answer is yes. Longer answer is: yes, but you can’t do it with the Operations Console. You either hack XML or you use the Authoring Console.

You can create so called “Secure References” to run a rule or monitor with a different user.

I’ve created a simple monitor in the Operations Console which will run with the default action account. If you explore the properties of any monitor in the Operations Console, there’s no way to change the action account. There’s also no way to override the action account. As mentioned before, the Operations Console is quite limited.

We need the Authoring Console!

You can download and install the authoring console here: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=18222

Before we continue, we will need to export the management pack where our monitor was created. Use the Administration space, select Management Packs, locate the management pack and click on Export Management Pack:

Start the authoring console and open the just exported management pack (where your monitor is located). In case you have troubles locating the System.Center.Library management pack, look at this blog post from Chris Harris on how to deal with that: http://blogs.technet.com/b/momteam/archive/2011/08/05/authoring-console-looking-for-microsoft-systemcenter-library-mp-6-1-7221-61-or-later.aspx

Create a new Secure Reference

After you’ve opened the management pack, go to the Health Model space and click on Secure References. In Secure References you can define “Run As” profiles that can be used to assign credentials to your monitors and rules. Right-click and select New Secure Reference from the context menu. Provide a new identifier for your secure reference (note the ID must be unique, it’s recommended to use the the management pack name as a prefix):

Provide a meaningful name for your secure reference. The name you choose here will appear in the Operations Console in the Administration space under Run As Configuration\Profiles. It’s also recommended to provide a description to ensure that everyone knows what this profile is used for (for example: is needed because the monitor XYZ needs special permissions, etc.).

As Context you could just select “System.Entity” which will allow you to use the secure reference for any target class of your workflows.

The following link provides more information about the secure reference element: http://msdn.microsoft.com/en-us/library/ee533832.aspx

Click OK to close the dialog.

Assign the Secure Reference to our Monitor

Change to Monitors (or if you want to assign your secure reference to a rule, go to Rules) and locate your monitor you want to change the Run as profile. Note that objects (such as monitors and rules) created in the Operations Console have an ID like UIGeneratedMonitor_and_some_GUID. If you have multiple monitors in this view, you may to open them all up until you find the one you are looking for.

In the configuration tab, select your Run as profile (the one we just created) and click on Apply:

Save and Re-Import

Before you save the management pack, I strongly recommend to increase the version number using File –> Management Pack Properties:

I’ve seen all sorts of weird behavior if you import a changed management pack with the same version, so don’t forget to always increase the version number.

Save the management pack using File –> Save and switch back to the Operations Console to import the management pack. After the import finished, ensure that the new version is shown correctly:

Assign a User Account to your new Run As Account

Go to the Run As Configuration\Accounts and create a new Run as account for your monitor:

Configure the windows account:

Choose the distribution model, depending on your requirements and your security policy and finish the wizard.

Now go to the Run As Configuration\Profiles node in the Administration space and look for your new Run as Profile:

Click on Properties and assign the created Run as account to your new profile as you do it for other management packs:

One easy way to verify if your new action account is being used, is to check the task manager processes. Look for a MonitoringHost.exe running with your action account. SCOM creates at least one MonitoringHost.exe for each action account.

cheers,
Stefan
http://www.code4ward.net

In this blog post I will go through the installation process of my “lab” machine with all relevant SCOM roles installed on a single box. I’ve prepared a virtual machine running Windows Server 2008 R2 fully patched and SQL Server 2008 R2 fully patched (incl. SP1). If you know SCOM 2007, you should know that 2012 now requires you to install DW components because some dashboard widgets depend on DW data. You also need to stick with the default collation in order to run SCOM (SQL_Latin1_General_CP1_CI_AS) and last but not least, you need Full Text Search enabled. Because I’m also going to install the web consoles, I’ve added the IIS role to my Windows Server 2008 R2.

To download SCOM 2012 RC, go to: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=27974

A word of caution though: “Upgrades from RC to RTM in *any* scenario will not be supported by Microsoft support unless you are part of the TAP program for the specific product that they are trying to upgrade.  Only TAP customers are licensed to run pre-RTM software in production.”

Installation

After we took care of all the required stuff, let’s jump in and start the setup:

As mentioned before, I will install all components for a typical single-server lab configuration.

In my case, the prerequisites checker failed with the following errors:

This happens if you install IIS role after you’ve installed the .NET 4 Framework. It’s easy to fix. Just go to the .net 4 framework directory and execute:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319>aspnet_regiis.exe -i

After you’ve registered .NET 4 in IIS, go to the ISAPI and CGI Restrictions and enable both, the 32 bit and 64 bit ISAPI extension:

Since I’m installing a fresh, new management group for my lab environment, I select to create a new management group and give it a meaningful name

Oddly enough (maybe this is an RC thing), the next page shows the license terms. Usually it’s one of the first pages in the setup wizard. Anyway, agree to the terms and continue.

I think this is pretty much self explanatory. I did increase the initial database size because 1 gig of operational data is reached quickly. Let’s head over to the next page:

Here’s an interesting part. In previous versions 2007 and 2007 R2, data warehouse was an optional component. In 2012 it’s mandatory. At least having the DW database is mandatory. Having reporting is still optional (as you could see in the very first setup page).

SQL reporting services (SRS) are also used exclusively in 2012 as it was in the 2007 and 2007R2 versions. You need to have a dedicated SRS instance because after setup is done with SRS (mainly putting a custom security model on top of it) it’s “SCOM-ified” and unusable for other stuff!

The next two pages are dedicated to the web console:

If you want to publish the web console to the internet, I strongly recommend to enable SSL.

Since I’m using this internally in my lab only, I leave the default settings.

The new consolidated account configuration page makes it much more easier to configure all the action accounts.

The next page is dedicated to error reporting and customer experience improvement. Enable those settings if you can!

Verify the summary page and click next to start the installation process:

That’s it. After the installation finished you can close the installer and start the Operations Console:

cheers,
Stefan
http://www.code4ward.net

My previous blog posts about the brand new System Center Orchestrator 2012 Release Candidate (SCO) were about the installation experience and the basics of Integration Pack (IP) deployment. In this blog post I want to explore the Orchestrator Integration Toolkit.

Installation

I was very impressed with the setup experience of SCO 2012 RC, also the IP deployment wasn’t that hard. The Integration Toolkit setup is a different story. When you want to start the setup on a fully patched machine, you’ll see a message like this:

There’s nothing wrong with a message like this but you would expect having a browser window open after you click OK, redirecting you to the web page mentioned in the message box. Not going to happen! So you have to manually open http://wix.codeplex.com and install WIX 3.5 on the machine before you can start the setup. Install the WIX 3.5 and restart the Setup. Once you can proceed with the setup, it’s really simple and doesn’t deserve any screenshot here.

After the setup finished you’ll see a new folder with two new items in the start menu:

Command-Line Activity Wizard

This wizard simply lets you create an activity based on a simple command-line (including arguments).  In this post I will demonstrate how this wizard works based on a very simple example: creating an event log entry.

You may wonder why I want to create an event log entry when SCO already has a “Send Event Log Message” activity. I just want to demonstrate to concept how simple it is to integrate existing tools in runbooks. You may also wonder why I’m just not using the “Run Program” activity. Well, sometimes you want or need to wrap existing tools into something you can distinguish visually in your runbook or limit access to parameters for the runbook designer.

Anyway, let’s start the fun. Before we can create our own IP we need to generate a .NET assembly containing our activities. The "Command Line Activity Wizard” helps us to do that. When you start the wizard you’ll see a button “Load existing assembly” which you can ignore for now. Click on Next:

This wizard page asks for basic assembly information: name, location in filesystem (where the assembly will be saved), clicking on the “Assembly Information” button also allows you to add additional meta data, like version, company, etc. After providing the assembly details, click on next.

This screen allows you to add one or more command-lines (each can be an activity in the Runbook Designer) to your assembly. Click on “Add…” to add a new command:

For this example we use a very simple command line: eventcreate /L APPLICATION /T INFORMATION /SO MySample /D "This is a test event." /ID 200

To make it even more fun, we allow the event ID to be injected from the workflow. Switch over to the Arguments tab:

Note that we still haven’t added the /ID parameter to the command line. First we need to add a new parameter for that. Click on the “Add…” button:

After we created the parameter you can use the “Insert” button to put it in the command-line:

Type /ID and then use the Insert button to insert the variable $(EVENTID)

The Published Data tab allows you to put the output of the command to the data bus of the runbook. In this case, eventcreate will not create any valuable output for us, so we skip this.

Let’s close the Add/Edit Command dialog with OK and proceed with the wizard. The wizard will then create a .NET assembly wrapper for the activities and writes it to the path you’ve selected previously in the Assembly Details wizard page. The last wizard page includes a button “Build Integration Pack”, allowing you to make an IP from your assembly.

Before we proceed to the next wizard: remember that button “Load existing assembly” from the first page of the Command-Line Activity Wizard? You can use it to load .NET assembly wrapper from this wizard and add/edit/remove activities. I strongly recommend that you use the “Assembly Information” page and apply proper versioning on your assemblies to keep track which assemblies/IPs are in use.

Integration Pack Wizard

This wizard allows you to package a complete, ready to ship integration pack. The first page of the integration pack wizard has a button “Import Integration Pack” which we will ignore for now. Click on next:

This page allows you to control how the IP will present itself to the user.

Click on “Add” to add one or more activities from libraries like those we just created:

Browse to the .NET assembly wrapper we created with the previous wizard, select the class, you may adjust the name, description and the icon for the activity. Click OK and proceed with the wizard. The next wizard page allows you to bundle other files or dependent applications in your IP. When your IP is deployed all the bundled files are also deployed. In our case we do not need to bundle anything. Click on next to specify the .oip file we want to save our activities in:

After the wizard is finished, your .oip file is ready to deploy. Once again, if you want to change your IP (add, remove or edit activities) you can use the “Import Integration Pack” button on the very first page of the wizard. Note that this wizard automatically increases the version of the IP when you edit an existing one!

Test the Activity

Now let’s test our IP/activity in a runbook. First we need to register and deploy the IP we just created. Take a look at my previous blog post on how to install and deploy IPs. Once you’ve deployed your IP, you’ll see the activities in your runbook (you need to restart the designer if it was running while you deployed the IP):

Just integrate your activity in a runbook, check it in and launch the runbook using the Orchestrator Console.

cheers,
Stefan
http://www.code4ward.net

Yesterday I’ve blogged about installing Orchestrator (SCO) – which was very easy. Once you are done with the installation you can open up “Runbook Designer” and start creating so called “Runbooks” right away.

Histrory

System Center Orchestrator 2012 (SCO) is the first release from Microsoft with the new brand and product name. Before Microsoft acquired the product, it was called “Opalis”. The terminology was a bit different there. A “Runbook” was called “Policy” in Opalis. Microsoft changed a couple of terms. Anyway, both terms are basically describing a “Workflow”. I don’t want to use the term Workflow further because it might be confusing. As a beginner who just jumped into the SCO fun you might have a better understanding if you think of a workflow. You may wonder why Microsoft isn’t just using the term workflow? I’m not sure about that but other System Center products (like SCOM) already have something called workflow and they are very different. So I guess they just wanted to introduce a new term to clearly distinguish those.

What is a Runbook?

A runbook defines a number of “activities” to be executed in a certain order or under certain conditions. You can also think of it as a workflow or a graphically composed batch or script file. It’s basically the same just much more powerful (and simpler). The beauty and power of SCO is the ability to execute those activities on different systems, share and process their output at any time within the runbook execution, create complex and even nested runbooks, extend those activities using “Integration Packs” and even create your own activities using the SDK.

The above screenshot shows a (very) simple runbook containing two activities: one stops the spooler service on a certain machine and two sends an email about it to a defined recipient. I will dedicate a separate blog post about runbooks later, I just wanted to quickly explain the concept of a runbook. This blog post will focus more on the activities and integration packs.

What is an Integration Pack?

An integration pack (IP) is like an “add-in”. It allows you to extend SCO with additional activities. As you can see in the above screenshot there are a lot of activities shipped with SCO but there are more. Microsoft offers additional IPs for the System Center product line (see my previous blog post) but there are and will be 3rd party IPs. Once an IP is installed, you will see additional activities in the runbook designer.

How to install IPs

Installation of IPs isn’t rocket science but it’s not as simple as installing a Management Pack. After you have downloaded and extracted the IPs (like the System Center IPs) you will see .oip files:

Open the “Deployment Manager”. If you have UAC enabled, be sure you start it as Admin! Right-click on the “Integration Packs” node in the tree and select "Register IP with the Orchestrator Management Server…”:

Use the wizard to install one or more IPs (in my case I will install the SCOM, VMM and SCSM IPs):

After you’ve finished the wizard you may see one or more license agreements you have to accept, then all the installed IPs should show up in the Integration Packs folder. Also check the log entries for issues.

The IPs are now registered but in order to use them you need to deploy them. Right-click on an IPs and click on “Deploy to a runbook server or runbook designer”:

A list of all registered IPs appears. Select all IPs you want to deploy:

Enter the target computers where you want to deploy the IPs. Note: pressing Enter while typing a computer name causes the wizard to move on to the next page (which is not possible as long as no computer is in the list). Be sure to click on “Add” first to get the computer into the list!

After that you can set a scheduled installation time and choose whether or not to stop runbooks while deployment is done.

Be sure to keep an eye on the log entries. If you have issues with the deployment the log entries should help you find the cause of the issues. After the deployment is done, you can right-click on an IP and select Properties. The Integration Pack tab allows you to look up where an IP is deployed (click on Resolve):

Now, all you need to do is to restart your Runbook Designer and verify that all activities from the deployed IPs are available:

cheers,
Stefan
http://www.code4ward.net

The System Center 2012 product line is nearing its final release date. Beta and release candidate versions are publicly available and can be tested. You can download the bits from: http://www.microsoft.com/download/en/details.aspx?id=27842

There are three separate downloads available:

1. System_Center_2012_Orchestrator_RC.exe
   this is the actual product and is around 60 MB. Pretty slick!
2. System_Center_2012_Orchestrator_Integration_Packs_RC.exe
   this are the integration packs you need to use when you want to create runbooks involving SCOM 2007, SCSM 2010, SCVMM 2008, DPM 2010 and SCCM 2007
3. System_Center_2012_Orchestrator_Integration_Toolkit_RC.exe
   this one is needed if you want to use the Orchestrator SDK to create custom integration packs

System Requirements

I think it’s interesting and notable that Orchestrator needs you to have Windows Server 2008 R2, SQL Server 2008 R2 and .NET 4.0 (when you install the web features) installed. If you want to use the integration toolkit, you need Windows 7 (or Windows Server 2008 R2) with .NET 3.5 installed. If memory server, MS supported older OS/SQL/Frameworks in the past so it seems there was a strategy shift. In my opinion this makes total sense. It’s easier to support, maintain and test a product in an environment like this. Hopefully we see improvements in quality as well.

Installation

Before I start with the installation I always ensure that my OS and components are up-to-date. SQL Server 2008 R2 is also installed and up-to-date. Before you proceed you might also check out the deployment guide and release notes: http://technet.microsoft.com/en-us/library/hh237242.aspx

After you’ve started setuporchestrator.exe the setup bootstrapper appears and we click on “Install” to start the installation:

One thing I forgot before I installed Orchestrator, was installing the IIS role. After I selected “Install” from the bootstrapper, skipped the license key page and agreed to the license terms, you’ll see a feature picker where you can choose what components should be installed on the machine:

As you can see, I’ve selected all components to be installed and the web service component even states that it needs IIS. Clicking on Next…

Here you see that the Prerequisites check complains about the memory but still nothing about IIS. This is only a warning and you can skip it by just clicking Next…

The next screen mentions that the setup is missing the IIS role and the nice thing is: setup will install the role for you. Very convenient! After clicking Next, setup will install IIS role:

After a while setup is happy and allows you to continue:

The rest of the installation is pretty painless. The following wizard pages provide good instructions on what is needed and where to read more about it. To make this blog post complete I will attach the other screenshots as well.

  

  

  

  

After the summary page, installation takes off:

  

Installing a single server containing all components of SCO is very easy and I’m really impressed with the simple setup experience. However, there are a couple of things you have to keep in mind:

• An SCO deployment can only have one Management server. In the previous screenshot you’ve seen that you cannot uncheck this component. So there’s no redundancy or failover system for this component. So if your management server dies you can still execute runbooks using the Orchestration console but you cannot connect to your SCO deployment using the Runbook Designer!
• To install additional runbook servers or web service installations you need to start the appropriate installations from the bootstrapper in the “Standalone Installations” section:

I was asked recently how to implement a notification subscription for service owners of distributed applications. Here’s a little tutorial on how I did it.

For this example I created a very simple distributed application, just to illustrate the solution.

Override for the Distributed Application

In order to get a notification we need to create an alert whenever the distributed application changes it state from healthy to non-healthy. Notification subscriptions are based on alerts. So in order to get notifications we need alerts first. By default, distributed applications itself do not raise an alert when the state changes but it is very simple to enable alerting:

Go to the Monitoring space and select the node “Distributed Applications” in the state view. Select the distributed application you want to enable alerting and open the health explorer:

Once the health explorer is loaded, DO NOT select the top level node (Entity Health), expand the Availability node and select the “All Contained Objects” node of your distributed application and create an override:

If you override the top level node, you will get an alert for each component group. I’m not sure if this behavior is by design or not but it’s clearly not what we want here. The catch is, that if you want to get notified for Performance, Configuration or Security issues as well, you need to repeat the steps above for each section.

In the Override Properties dialog, setup alerting:

To enable alerting you need to setup the following:

• Override “Generate Alert” with the value “True”.
• Override “Alert On State” with “The monitor is in a critical or warning state”. You might want to change this if you only want to be notified for critical states.
• Override “Auto-Resolve Alert” with the value “True”. This way you can also send out notifications when the distributed application recovered and is healthy again.
• Override “Alert Priority” with the value “Low”. This depends on your requirements. I chose Low because I want to keep down the noise. Usually you get alerts from each component, so it makes sense to keep it on Low.
• Override “Alert severity” with the value “Information”. Again this serves noise reduction and you could change it to any other value or matching the monitor value if you want or need to.

Click on OK to apply the override and close the properties dialog.

Ensure that the Alert is raised

As you can see, as soon as a component turns red, you get an informational alert for your distributed application:

Create a Subscription

Now we only need to create a notification subscription. Go to the Administration space and select Notifications/Subscriptions. Create a new subscription:

It is important that you choose the “All Contained Objects” role up monitor AND the instance of your distributed application (in my example “Test Application”).

If you want to react on multiple distributed applications in one subscription, you can create a group and put all distributed applications you want to notify on in this group. To do so, change to the “Authoring“ space and select Groups. Create a new group and add instances of “Distributed Applications” in the group using explicit members or with dynamic members if you want to follow a naming scheme. For this example I’m using explicit members:

Then change the subscription to include instances from a specific group:

After setting up the subscription, your recipients (the app owners) can now get emails whenever a component degrades an application.

If you have any feedback, questions or suggestions, leave a comment or contact me directly stefan.koell (-at-) code4ward.net

cheers,
Stefan
http://www.code4ward.net
(Also check out Royal TS or LogSmith to slice and dice Windows and SCOM events – free download!)

The following article applies to SCOM 2012 BETA and may or may not apply to RC or RTM release. I’ll try to repro the issue in the upcoming releases to see if the behavior changed and provide updates if necessary.

I guess everyone is testing SCOM 2012 Beta right now and a lot of people are already blogging about their experience. I thought it’s time to do the same and share some experience I had with network discovery.

Content of this post:

1. Introduction
2. Discovery
3. Troubleshooting
     3.1 Discovery configuration
     3.2 Events
     3.3 Diagnostic Tracing
     3.4 Network Tracing
4. Solution
5. Conclusion

1. Introduction

My small lab environment doesn’t really have lots of high-end network devices (in fact the number of high-end network devices in my lab is ZERO!) but I thought why not bring in my low-end, consumer grade devices for testing. First you should know, that SCOM treats certain devices different as others. Supported Cisco routers for example are monitored much more detailed than a consumer grade Zyxel Zywall-2. SCOM doesn’t really know much about those devices and are therefore handled as a “generic” device with a very limited set of monitoring rules (mostly availability and response times but no monitoring of ports, CPU, bandwidth etc.). By the time of RTM, Microsoft should release a complete list of supported network devices… (see Update 2 below)

As soon as I started the network device discovery I realized that my device couldn’t be discovered. Running the discovery manually showed that the task completed successfully:

The task window shows that the discovery job was successfully kicked off but this doesn’t really mean that the discovery is already done. The actual discovery process runs asynchronously but more on that later in the Discovery section. After a while the “Network Devices Pending Management” view shows my device:

No response ping?! Needless to say that my management server is able to ping the device and responds just nice to the ping requests.

2. Discovery

Let’s have a look how discovery is set up and how it’s working: When you create a network discovery rule you go through a wizard providing a lot of information about discovery methods, accounts, devices, schedule, etc. I will not go through all the options, the wizard provides. Only the parts necessary for this case will be discussed.

Because I only want to discover one single device, I chose “Explicit discovery” as discovery method. My device configuration looks like this:

Note that I selected “ICMP and SNMP” as Access mode. To tell you upfront, selecting only SNMP will successfully add my device but in this case I want to know why ICMP and SNMP is not working – especially since my discovery management server can successfully ping the machine. Besides, I learned a lot about network discovery which might help in other scenarios…

The discovery schedule was set to manually. After you finished the wizard, start the discovery:

3. Troubleshooting

3.1 Configuration

After you finished the wizard, an unsealed management pack with the name “Microsoft.SystemCenter.NetworkDiscovery.Internal” will be modified with your configuration. Once you export and open this management pack in a text editor, you’ll see the module configuration for the discovery:

First thing worth checking is to look into that MP and see if the configuration you’ve done with the wizard is reflected in the MP. Once you are sure the MP was configured correctly and was successfully deployed to the managing management server, examine the discover events…

3.2 Events

Thankfully, Microsoft created a view (and a rule) to pick-up and show all the relevant events for the network discovery progress right in the console:

The view can be found in the Monitoring space under “Operations Manager\Network Discovery\Network Discovery Progress Events”. I highlighted the events in the view for one discovery run. Usually the event 12024 indicates that the discovery is done and the discovery data was created but event 12008 is an interesting one: It shows a summery of the discovery. As you can see one device (my only one) is in pending list. Anyway, the events weren’t very helpful in finding a solution in my particular case…

3.3 Diagnostic Tracing

If you are familiar with tracing in SCOM 2007 (R2), you might not find anything different in SCOM 2012. In fact, I believe that the (hopefully) well known KB is still valid for SCOM 2012: http://support.microsoft.com/kb/942864

However, I learned that the TraceConfig tool now offers a lot more trace providers to do more granular tracing compared to SCOM 2007:

Btw, you still need the FormatTracing command to make the trace log readable.

Before we start with the tracing, we need to edit a config file to enable Network Discovery debug logging:

• Navigate to C:\Program Files\System Center Operations Manager 2012\Server\NetworkMonitoring\conf\discovery location on management server which is used for network discovery.
• Edit discovery.conf at this location as below
  • remove "#" before "DebugEnabled = TRUE"
  • Add line "LogDiscoveryProgress = TRUE" (without quotes) below the DebugEnabled = TRUE
• Restart the Health service on management server.
• Execute the network discovery. Wait for the discovery to complete. Verify the status is "idle" in administration ->network management->discovery rules or look at the events as shown above.
• Take a look at the <ManagementServer-FQDN.log> from location c:\temp on the Management Server.

Unfortunately tracing didn’t show any indication of what is wrong with the discovery. Next stop, network tracing…

3.4 Network Tracing

To download and install the Microsoft Network Monitor (3.4) visit: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=4865

Once the network trace was captured, I realized that the ICMP requests and replies went out and came in as expected:

4. Solution

Thanks again to Yuvraj Attarde from Microsoft, who actually found the solution and helped me through the process. The issue was caused by THE FIREWALL !

Disabling the windows firewall on the management server solved the issue but it’s not really a solution. You might want to keep the firewall on in production. To successfully discover using ICMP and SNMP, you need to allow ping.exe (from %WINDIR%\system32) and monitoringhost.exe (from the SCOM install directory) to communicate without restrictions.

But be careful, there are two ways to configure this:

1. Using the Windows Firewall\Allowed Programs control panel applet:

2. Firewall MMC Snap-In:

You can use either of those options but if you use the MMC snap-in, be sure to create an Outbound and an Inbound rule!

After that, discovery works just fine:

5. Conclusion

When troubleshooting network related issues, always try to disable the firewall (temporarily) to see if the issue originates from that area I know, I could have done that sooner but then again, I may not have learned those lessons above.

I’m still not sure if this behavior is “as expected” and as Microsoft often nicely states “by design” or just simply a bug. What puzzles me is that SNMP works out of the box without any firewall exception and ICMP does not. It’s also confusing that a ping from command prompt works but in conjunction with the monitoringhost.exe it doesn’t. If you cannot count on ping as a diagnostic tool anymore, what’s the point of having it then?

I hope that the above is useful information for some of you in the need of troubleshooting network discovery issues. If you have any comments or feedback, let me know: stefan.koell@code4ward.net

Special thanks to Yuvraj for teaching me!

Cheers,

Stefan Koell
Microsoft MVP for System Center Operations Manager
http://www.code4ward.net

UPDATE:

Got response from MS about the issue. They plan to create the exception for monitoringhost.exe during install time by RTM. That’s great news and means that the issue I was having doesn’t affect anyone else. Although I guess the troubleshooting guide I’m offering here might help in other cases as well.

UPDATE 2:

MS is already offering a list of all network devices with extended monitoring capabilities: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=26831

“Congratulations! We are pleased to present you with the 2011 Microsoft® MVP Award! …”

Thanks to Microsoft and especially to my MVP lead for the great support and for recognizing my work in the SCOM community. I’m happy and proud to be a member of this extraordinary group of people.

This year will be very exciting because Microsoft will release major new versions of the System Center products – including Operations Manager. I’m looking forward to get my hands dirty with SCOM 2012 and SCSM 2012. Stay tuned…

Download: Link

The System Center Operation Manager 2007 Administration Resource Kit provides the following features to aid in management group administration:

Scheduled Maintenance Mode – Ability to schedule and manage maintenance mode in the management group.
Clean Mom – Helps remove all installed R2 components.
MP Event Analyzer – MP Event Analyzer tool is designed to help a user with functional and exploratory testing and debugging of event based management pack workflows like rules and monitors.

Feature Summary:

Ability to schedule Maintenance mode for “run once”, daily, weekly, and monthly schedules
Ability to removal all OpsMgr 2007 R2 files from a system
Ability to debug event based management pack workflows