Dual Home SCOM UNIX/LINUX Agents and Migrate – Part 2

Continuing from Part 1 where I explained how to dual home SCOM UNIX/LINUX agents (the method is still NOT supported by Microsoft), it is time to blog about how to migrate the dual homed SCOM UNIX/LINUX agents to the New Management Group.

Pre-Requisites:

You should be in SCOM 2012 R2 UR12 or higher if you are migrating to SCOM 2016 UR2+ or SCOM 2019. Because SCOM 2016 UR2+ and SCOM 2019 only supports SHA256 SCX certificate. I explained this in Part 1.

My setup:

Management Group 1 :
– 1 SCOM 2012 R2 UR14 Management Server in X-Plat Resource Pool 1.
– 1 SCOM 2012 R2 RTM Management Server in X-Plat Resource Pool 2 (I will explain you why I have made this setup for this demo only).
Management Group 2:
SCOM 2019 RTM / 1 Management Server in the X-Plat Resource Pool.
Agent version – 1.5.1-256.

** Management Group name does not matter in this process. So I am omitting those.
** I will refer the SCOM 2012 R2 MG as Old MG and SCOM 2019 MG as New MG.

Steps:

At this point, your agents are dual homed and healthy on both the MG. In order to migrate the agents to the new MG, firstly, you have to re-sign the SCX certificate on the X-Plat agent with one of the new MS/GW from the new MG. This will remove the dependency of the old MS/GW in the old MG which you will decommission at some point of time.
But before you sign the certificate from the new MS/GW in the new MG, you have to make sure the certificates on the X-Plat agents are SHA256. Because from SCOM 2012 R2 UR12 onwards, SHA1 certificates are deprecated. All the details about this change are documented here.

There are different ways to find the signing algorithm of the SCX certificate on the X-Plat agents. But before you jump into any of the options, you might have to modify your SUDOERS file with the below line if you are using the minimum privilege. I ran into permission issue which doing this in my LAB. If you need help to configure your sudoers read this.

monuser ALL=(root) NOPASSWD: /bin/sh -c openssl x509 -noout -text -in /etc/opt/microsoft/scx/ssl/scx.pem | grep 'Signature Algorithm'

Option 1: From the SCOM console
Navigate to Monitoring -> Unix/Linux Computers -> Select an agent -> Run the task “UNIX/Linux Verify Certificate Signature Task”.
**You might not see the task if you are not having the last UNIX/LINUX library MP.

Option 2: From the X-Plat agent
Run the below command on the X-Plat agent

openssl x509 -noout -text -in /etc/opt/microsoft/scx/ssl/scx.pem | grep 'Signature Algorithm'

Option 3: Using the below PowerShell script for bulk check

But before you can use the script, create a xml file in one of the MS and paste these contents.

** This is the reason I left one MS in the RTM version to create a certificate with SHA1.

<p:ExecuteShellCommand_INPUT xmlns:p="http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem">
<p:Command>openssl x509 -noout -text -in /etc/opt/microsoft/scx/ssl/scx.pem | grep 'Signature Algorithm'</p:Command>
<p:timeout>10</p:timeout>
</p:ExecuteShellCommand_INPUT>

Now run the below script on the MS.
** Replace the Username and Password of the LINUX user.
** Replace the location where the xml file is saved. In my case it is C:\temp\SCXSignature.xml

#Import SCOM Module
$InstallDir=(Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup').InstallDirectory
$OMPowerShellDir=$InstallDir.Replace('Server','PowerShell')
$Modulepath=$OMPowerShellDir + "OperationsManager\OperationsManager.psd1"
Import-Module $Modulepath


#Get all X-Plat agents
$SCXAgents=Get-SCXAgent

foreach($SCXAgent in $SCXAgents)
{ 
    $output=winrm invoke ExecuteShellCommand http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem?__cimnamespace=root/scx -remote:https://$($SCXAgent.Name):1270 -auth:basic -username:<replaceusername> -password:<replacepassword> -skipCAcheck -skipCNcheck -skiprevocationcheck -encoding:utf-8 -file:<replace the xml location>

            if($Output -cmatch "sha256WithRSAEncryption")
            {
                $taskresultOutput="SHA256"
            }
            elseif($Output -cmatch "sha1WithRSAEncryption")
            {
                $taskresultOutput="SHA1"
            }
            else
            {
                $taskresultOutput="No SHA256 or SHA1"
            }
            [PSCustomObject]@{
                Agent    = $SCXAgent.Name                
                Output   = $taskresultOutput
            }
}

Sample results:

Agent Output
—– ——
ubuntu1704.nfs.lab SHA1
SLES12SP3.nfs.lab SHA256

For the X-Plat agents that already have SHA256 certificate we are good. For the ones which are still using SHA1 you have to upgrade to SHA256. Steps are available in this blog.

Now it is time to sign the certificate on the dual homed agents from the new MS/GW in the new MG. Use this PowerShell script to verify the signing MS for each SCX Agent.
But before you can use the script, create a xml file in one of the MS and paste these contents.

<p:ExecuteShellCommand_INPUT xmlns:p="http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem">
<p:Command>openssl x509 -noout -text -in /etc/opt/microsoft/scx/ssl/scx.pem | grep Issuer</p:Command>
<p:timeout>10</p:timeout>
</p:ExecuteShellCommand_INPUT>

Now run the below script on the MS.
** Replace the Username and Password of the LINUX user.
** Replace the location where the xml file is saved. In my case it is C:\temp\SCXCertsigningMS.xml

#Import SCOM Module
$InstallDir=(Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup').InstallDirectory
$OMPowerShellDir=$InstallDir.Replace('Server','PowerShell')
$Modulepath=$OMPowerShellDir + "OperationsManager\OperationsManager.psd1"
Import-Module $Modulepath


#Get all X-Plat agents
$SCXAgents=Get-SCXAgent

foreach($SCXAgent in $SCXAgents)
{ 
    $output=winrm invoke ExecuteShellCommand http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem?__cimnamespace=root/scx -remote:https://$($SCXAgent.Name):1270 -auth:basic -username:<replace username> -password:<replace password> -skipCAcheck -skipCNcheck -skiprevocationcheck -encoding:utf-8 -file:<replace the xml file path>
    $issuer=$output[3]
    $issuer=$issuer.split(',')[1]
    $issuer=$issuer.Split('=')[1]
    
     [PSCustomObject]@{
                Agent    = $SCXAgent.Name                
                Issuer   = $issuer
            }
}

Sample results:

Agent Issuer
—– ——
rhel7-8.nfs.lab SCOM2019MS1
SLES12SP3.nfs.lab SCOM2012R2MS

As we can see the dual homed agent SLES12SP3 agent is signed by the old MS. Let us sign it with the new MS.

In order to do this, Navigate to Monitoring -> Unix/Linux Computer -> Select an agent -> Run the task “UNIX/LINUX Update Certificate Task”

To do this in bulk a script is provided in this blog. Once the task completes successfully, run the script again to verify all the SCX Agents have the certificate signed from the new MS/GW in the new MG.

Finally it is time to upgrade the dual homed agents. On SCOM console connected to your new MG, navigate to Administration -> Device Management -> UNIX/LINUX Computers -> Select the Agent -> Upgrade Agent.
** Again let me repeat, your SUDOERS has to be configured properly for the upgrade to work.As you can see I have an upgrade available. It picks up the package from the SCOM installation directory in your MS in X-Plat Resource Pool. In my case it is C:\Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKitsMy Run As Account and Profiles are configured already. So I am using the below option. In case you have doubt, you can refer to this blog.If everything goes well, the agent should upgrade successfully.
Yeaahh!!!!! After all the hard work, my agent is upgraded and healthy in the new MG and also in the old MG.
Now delete the entry of this agent from the old MG.
Note** Do not uninstall. It will remove the package.