Sunday, September 3, 2017

Recover from failed attempt to rebuild Windows Server performance counters without backup

Scenario:

For any reason you try to recover/rebuild Windows Server performance counters using KB2554336 or other "smart" blog. After procedure you have some counters are missing and some 3rd party services cannot start at all. In my case it was one of Skype for Business Edge servers from Edge pool with Application Log errors like:
Log Name:      Application
Source:        usbperf
Date:          9/3/2017 11:23:41 AM
Event ID:      2001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
Unable to read the "First Counter" value under the usbperf\Performance Key. Status codes returned in data. 
Log Name:      Application
Source:        Microsoft-Windows-IIS-W3SVC-PerfCounters
Date:          9/3/2017 11:23:41 AM
Event ID:      2002
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
Setting up Web Service counters failed, please make sure your Web Service counters are registered correctly. 
Log Name:      Application
Source:        Microsoft-Windows-Perflib
Date:          9/3/2017 11:20:53 AM
Event ID:      1008
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
The Open Procedure for service "aspnet_state" in DLL "C:\Windows\System32\aspnet_counters.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.
Log Name:      Application
Source:        Application Error
Date:          9/3/2017 12:14:05 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
Faulting application name: ReplicaReplicatorAgent.exe, version: 6.0.9319.0, time stamp: 0x55276ea8
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00007ff7ae9db0f1
Faulting process id: 0xf28
Faulting application start time: 0x01d32494fba1def3
Faulting application path: D:\Skype for Business Server 2015\Server\Replica Replicator Agent\ReplicaReplicatorAgent.exe
Faulting module path: unknown
Report Id: 3b46686b-9088-11e7-80e6-00155d083802
Solution:

Option 1. Recover server from backup/snapshot/etc.

Option 2. In my case I didn't have backup. We'll focus on this recovery scenario.

Step 1. Make performance counters backup from similar neighbor "healthy" server.

"Healthy" server should be configured with exactly the same Windows Server and Application components (ex. server from a server pool, a node from failover or nlb cluster, etc.)

from cmd (Run As Administrator)
lodctr /S:C:\temp\counters_healthy-server.ini
Step 2. Copy counters_healthy-server.ini file from "healthy" to "unhealthy" server with missing counters.

Step 3. Make performance counters backup on "unhealthy" server (just for case)

from cmd (Run As Administrator):
lodctr /S:C:\temp\counters_unhealthy-server.ini
Step 4. Rebuild performance counters from counters_healthy-server.ini backup file on "unhealthy" server:

from cmd (Run As Administrator):
lodctr /R:C:\temp\counters_healthy-server.ini
Step 5. Resync the counters with Windows Management Instrumentation (WMI):

from cmd (Run As Administrator):
WINMGMT.EXE /RESYNCPERF
Step 6. Restart "unhealthy" server. Feel happy.

Moral: always make backup prior to any change even if it looks like very simple change.

References: