Sunday, September 3, 2017

Recover from failed attempt to rebuild Windows Server performance counters


For any reason you try to recover/rebuild Windows Server performance counters using KB2554336 or other "smart" blog. After procedure you have some counters are missing and some 3rd party services cannot start at all. In my case it was one of Skype for Business Edge servers from Edge pool with Application Log errors like:
Log Name:      Application
Source:        usbperf
Date:          9/3/2017 11:23:41 AM
Event ID:      2001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Unable to read the "First Counter" value under the usbperf\Performance Key. Status codes returned in data. 
Log Name:      Application
Source:        Microsoft-Windows-IIS-W3SVC-PerfCounters
Date:          9/3/2017 11:23:41 AM
Event ID:      2002
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Setting up Web Service counters failed, please make sure your Web Service counters are registered correctly. 
Log Name:      Application
Source:        Microsoft-Windows-Perflib
Date:          9/3/2017 11:20:53 AM
Event ID:      1008
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
The Open Procedure for service "aspnet_state" in DLL "C:\Windows\System32\aspnet_counters.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.
Log Name:      Application
Source:        Application Error
Date:          9/3/2017 12:14:05 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Faulting application name: ReplicaReplicatorAgent.exe, version: 6.0.9319.0, time stamp: 0x55276ea8
Faulting module name: unknown, version:, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00007ff7ae9db0f1
Faulting process id: 0xf28
Faulting application start time: 0x01d32494fba1def3
Faulting application path: D:\Skype for Business Server 2015\Server\Replica Replicator Agent\ReplicaReplicatorAgent.exe
Faulting module path: unknown
Report Id: 3b46686b-9088-11e7-80e6-00155d083802

Option 1. Recover server from backup/snapshot/etc.

Option 2. In my case I didn't have backup. We'll focus on this recovery scenario.

Step 1. Make performance counters backup from similar neighbor "healthy" server.

"Healthy" server should be configured with exactly the same Windows Server and Application components (ex. server from a server pool, a node from failover or nlb cluster, etc.)

from cmd (Run As Administrator)
lodctr /S:C:\temp\counters_healthy-server.ini
Step 2. Copy counters_healthy-server.ini file from "healthy" to "unhealthy" server with missing counters.

Step 3. Make performance counters backup on "unhealthy" server (just for case)

from cmd (Run As Administrator):
lodctr /S:C:\temp\counters_unhealthy-server.ini
Step 4. Rebuild performance counters from counters_healthy-server.ini backup file on "unhealthy" server:

from cmd (Run As Administrator):
lodctr /R:C:\temp\counters_healthy-server.ini
Step 5. Resync the counters with Windows Management Instrumentation (WMI):

from cmd (Run As Administrator):
Step 6. Restart "unhealthy" server. Feel happy.

Moral: always make backup prior to any change even if it looks like very simple change.


Tuesday, August 29, 2017

Validation errors for users/contacts/groups in the Office 365


You get something like:

Get-MsolUser -UserPrincipalName | fl *status*,errors
OverallProvisioningStatus : Success
ValidationStatus          : Error
Errors                    : {Microsoft.Online.Administration.ValidationError}

How to get more details about the "ValidationStatus" Error?


for users:

$errors = (Get-MsolUser -UserPrincipalName "").Errors

for contacts:

$errors = (Get-MsolContact -ObjectID <Object_ID>).Errors

for groups:

$errors = (Get-MsolGroup -ObjectID <Object_ID>).Errors

and then expand $errors variable:

$errors | % {"`nService: "+ $_.ErrorDetail.Name.split("/")[0]; "Error Message: "+ $_.ErrorDetail.ObjectErrors.ErrorRecord.ErrorDescription}  


$errors = (Get-MsolUser -UserPrincipalName

$errors | % {“`nService: ” + $_.ErrorDetail.Name.split("/")[0]; “Error Message: ” + $_.ErrorDetail.ObjectErrors.ErrorRecord.ErrorDescription}

Service: exchange
Error Message: Exchange can't disable the mailbox " Exchange Hosted Organizations/ Deleted Objects/UserNameHere" because it is on In-Place Hold.


  • You see validation errors for users in the Office 365 portal or in the Azure Active Directory Module for Windows PowerShell
  • Monday, August 7, 2017

    Install Windows Feature with DISM (without PowerShell)


    Imagine that you can't Install-WindowsFeature (Add-WindowsFeature for 2008 R2) on Windows Server using PowerShell for some reason.

    Recently I had a problem with broken WMI where I couldn't manage Windows Server features of affected server. My wish was to make Full Server backup prior to any changes with broken server using built-in "Windows Server Backup" and I couldn't install backup feature using PowerShell.


    DISM commands running from an Elevated Command Prompt (Admin) saved my life:
    Dism /online /enable-feature /featurename:WindowsServerBackup
    Deployment Image Servicing and Management tool
    Version: 6.3.9600.17031
    Image Version: 6.3.9600.17031
    Enabling feature(s)
    The operation completed successfully.
    Dism /online /enable-feature /featurename:WindowsServerBackupSnapin
    Deployment Image Servicing and Management tool
    Version: 6.3.9600.17031
    Image Version: 6.3.9600.17031
    Enabling feature(s)
    The operation completed successfully.
    You can install any Windows feature in similar way. Get list of all available features and their names on server:
    Dism /online /get-features /format:table | more

    WMI is broken. "Configuration refresh failed with the following error: Invalid class"


    When we start Server Manager on Windows Server 2012 R2 the flag lights up "Red" almost immediately with the error:
    "Configuration refresh failed with the following error: Invalid class"

    Get-WindowsFeature could end with the same "Invalid class" error.


    Backup WMI:

    • Launch the WMI MMC snapin by Start -> Run -> then enter WMIMGMT.MSC
    • Right click WMI Control (Local) and click Properties
    • Tab Backup/Restore, press Back Up Now...

    Recompile .mof & .mlf files by running the following commands from an Elevated Command Prompt (Admin):
    CD C:\Windows\System32\WBEM
    dir /b *.mof *.mfl | findstr /v /i uninstall > moflist.txt & for /F %s in (moflist.txt) do mofcomp %s
    Refresh "Server Manager" or re-open it.


    Thursday, June 1, 2017

    Cannot sign in to Skype for Business because your computer clock is not set correctly


    User cannot sign-in to Skype for Business via client and has error:

    "Cannot sign in to Skype for Business because your computer clock is not set correctly. To check your computer clock settings, open Date and Time in the Control Panel."

    Per KB2581291 Microsoft suggests to make sure that the computer's clock and time zone settings are set correctly. And it's correct article except the fact that the registry key "ClockSkew" is located under another registry path:


    instead of documented:


    In my particular case users messed with time and had unchecked "Daylight Saving Time" setting.

    The Time Zone, the "Daylight Saving Time" settings were corrected, time synchronization were completed, registry key "ClockSkew" was deleted and Skype for Business client could sign-in.


    Sunday, March 19, 2017

    Reset Skype for Business Statistics Manager (StatsMan) history


    You have a brand new or existing deployment of Skype for Business Statistics Manager. Your graph shows some weird old statistics (few days/weeks ago) as well as recent statistics (by default: 24 hours old) in a single chart.

    Don't know yet. Could be related to Open Source of Redis in-memory caching system.


    Reset statistics completely (--age:0) or to 24 hours old.
    1. Run as Administrator "cmd" on server where SfB Statistics Manager Listener is installed:
    2. Run either:

    >"C:\Program Files\Skype for Business Server StatsMan Listener\PerfAgentStorageManager.exe" --redis:localhost --action:cleanup --age:0


    >"C:\Program Files\Skype for Business Server StatsMan Listener\PerfAgentStorageManager.exe" --redis:localhost --action:cleanup --age:24

    Using age: 1.00:00:00
    FillNameCacheAsync of 6,889 counters took 180ms
    GetLastWriteTimesAsync of 6,889 counters took 56ms
    Scanned through 7,389 redis keys in 79ms and found 36 that should be deleted.
    36 counters, 0 hosts, 36 storage keys, haven't been written to in 6.00:00:00
    Do you want to remove the above hosts,36 counters and 36 storage keys? Y|[N]:y
    Removed: 36 keys in 29ms
    BulkDereferenceCounters: Attempting to dereference 36 storage keys
    BulkDereferenceCounters: Deleted 36 last write time entries in 10ms
    BulkDereferenceCounters: Removed 36 storage key entries from set in 1ms

    Deleted 0 server infos.

    I got normal graph after clearing history.

    More information 
    about PerfAgentStorageManager.exe /?


        updateServerInfo : Need --redis : Must provide --file or (--hostname and one or more --population options), --file should be in the format of getServerInfo output
                         : optional --noPrompt (will automatically do it without asking)
                         : optional --disableUpdate (will do adds only)
                         : optional --debug will show extra output
                         : The following shard scheme docs are only for multi-azure storage account setups:
                         : optional --shardScheme=none (to disable updating shard assignement for new servers)
                         : optional --shardScheme=<scheme>
                         :          <scheme> is one or more population matches separated by comma.
                         :          The default is Pool_,Site_ and it is recommended to not change this.
                         :          This will look at each new server's populations and try to match (start with) with a
                         :          scheme component then it will assign to that populations shard if it exists.
                         :          So for the default scheme it will look for the servers Pool shard, if that doesn't exist
                         :          then it will look for the servers Site shard, if that doesn't exist it will just stay as

        getServerInfo : Need --redis : optional --uri for the location to upload to, Default is redishash:///_ServerInfo
                      : optional --file will output the xml for edit
                      : optional --mode=command will output the information with the command line options for PerfAgentStorageManager
                      : optional one or more --population will get only the servers in one of those populations

        getRedirectCandidates : Need --redis, default is to find populations with a single server where the name of the server is the last part of the population
                      : optional --mode=<current>, current will retrieve the current list of configured redirected populations
                      : optional --file=filename.csv will output a file that can be used with --action=updateRedirectCandidates
                      : example: Pool_mySEserverFE01 server=mySEServerFE01

        updateRedirectCandidates : Need --redis --file is a csv with Population,Server

        getShardInfo : Need --redis.
                    : Optional: One or more modes
                    : --mode=verbose to output every assignment
                    : --mode=password to see the configured password (careful!)
                    : --mode=history to see the history of all shard assignments (assumes --mode=verbose)

        updateShardAssignments : Need --redis and one or more --population or a single --hostname and --mode=specific or --mode=population and --value
                    : --value=# the number of the shard to assign them to
                    : mode=specific means assign that population or host to the shard
                    : mode=population means assign the population(s) to the shard and all of the servers in all populations given
                    : mode=correction (be careful with this) means update any shard assignement that matches -value=shardid and -match=tableID (see -a=getshardinfo -verbose)
                    : this is used when a mistake in timing was made and the shard assignement hasn't been used yet and this will update to the next
                    :if you get this wrong and use it on something that is in use, the data will be missing
    and not easily recovered
                    : The assignment won't switch until the next 12 hour period. Be careful that you assign far enough from daily 12 hour boundaries
                    : Optional: if in --mode=population you can also add a --match=<population match> to also add any population to the shard for servers
                    : in the given population. For example: --mode=population --population=Site_Blah --match=Pool_ this will add any Pool in has servers in that
                    : site. It will ask you for each one unless you have --noPrompt

        updateAzureShard : Need --redis and --value=<shard ID> --azure=<accountName> --azurePassword=<azure key> --thumbprint=<password decryption certificate thumbprint>
                     : Optional --noPrompt to not prompt, you can add a --new to fail if expect it to be a new account (it will fail if the account or shard ID exists)

        validateSharding : Need --redis and --azure=<accountName> --azurePassword=<azure key> (for the default account)
                     : Optional --noPrompt will not prompt for keys to check but you'll need to provide --value=<hostname or population name or all>

        getRedirectedCounters : Need --redis

        deleteServerInfo : Need --redis and --file or one or more --hostname entries : File contains names of servers (one p
    er line). optional --uri for the key to upload to, Default is redishash:///_ServerInfo (only redishash uri supported)

        checkServerInfo : Need --redis and --file or one or more --hostname entries : File contains names of servers (one pe
    r line).

        cleanup : Need --redis, this will cleanup all hosts, populations and counters that haven't been written to in the given age time
                : Optional --age=<hours> (default 24)
                : Optional --match=<regex> to only match certain hosts/populations
                : Optional --notmatch=<regex> to exclude certain hosts/populations

        listKeys : Need --redis optional --file and --match and --key (key will make this a hash key list) --includeValues will show the value of the hash keys as well

        deleteKeys : Need --redis and --match  optional --notMatch, optional: --count=<batchSize> default=1000

        listHostIds : Need --redis and --match  optional --notMatch

        backupIdentifiers : Need --redis and --file, optional: --mode=overwrite

        subscribe : Need --redis and --counter

        listCounterStorageNames : Need --redis optional --file and --match and --notMatch

        listBuckets : Need --redis optional --match and --notMatch, optional --mode=<counters>, counters mode will try to distill down to the list of perf counters polled

        readListValues : Need --redis and --key optional --count

        deleteHashKeys : Need --redis and --key optional --match and --count (this will decide batch size), you will be prompted

        lookupStorageName: Need --redis and one or more --counter and/or --bucket this will lookup the counter or bucket storage names, optional: --mode=redirected

        findBadServers: Need --redis and optional --bucket (it will by default use \\*\Memory\% Memory Free_Maximum\@ bucket)

        getCounterValues : Need --redis and --counter : Optional --file=<csv output file path>

        getCounterValuesRange : Need --redis and --azure. Optionally need --counter (default is memory max) --startTime and --endTime (both in local time), defaults 24 hours ago and now

        counterLastWriteTimes: Need --redis optional --age=<seconds> will only print counters older than this, optional --count will limit the output, optional --mode=host will output the hosts and their youngest value age

        getBucketValues : Need --redis and optional --bucket (default will be \\*\Memory\% Memory Free_Maximum\@), optional --count will limit the output

        getBucketDueTimes: Need --redis optional --key default is _BucketDueTime

        getStats: Shows info about number of counters/buckets/etc. Need --redis

        listAggLocks: Need --redis. Shows which server owns which agg lock for buckets.

        setting: Need --redis and --setting=SettingName and --value=[true|false|int32#]. Updates/gets.

        all: retrieves all of the current values
        aggregation: optional --value=[true|false] (if not given it gets the current value)
        redirectedAggregations: optional --value=[true|false] (if not given it gets the current value)
        persistentStore: optional --value=[true|false] (if not given it gets the current value)
        rediscounterStore: optional --value=[true|false] (if not given it gets the current value)
        bucketManagerMaxConcurrency: optional --value=int32# (if not given it gets the current value)

      --value<:value>  Some actions require a 'value' parameter.
      --redis<:redis>  Set the redis server configuration
      --redisDatabase<:redisDatabase>  Set the redis database number. Default: 0
      --redisPassword<:redisPassword>  Optional redis password. Otherwise it will be looked up from the config.
      --azure<:azure>  Set the azure storage account name
      --azurePassword<:azurePassword>  Optional azure storage password. Otherwise it will be looked up from the config.
      --startTime<:startTime>  Some actions take a startTime arg.
      --endTime<:endTime>  Some actions take a endTime arg.
      --file<:file>  Some actions take a file arg.
      --hostname<:hostname>  Some actions take a hostname arg.
      --population<:population>  Some actions take a population arg.
      --match<:match>  Some actions uses a pattern or match.
      --notMatch<:notMatch>  Some actions uses a pattern or match to exclude.
      --new  Some actions are being tested with a 'new' mode.
      --disableUpdate  Some actions need a way to disable updates (and do adds only).
      --counter<:counter>  Some actions take a counter arg.
      --bucket<:bucket>  Some actions take a bucket arg.
      --shardScheme<:shardScheme>  Some actions take a shard scheme.
      --age<:age>  Some actions take an age arg. In seconds.
      --noPrompt  Some actions prompt to continue, this will answer affirmative to any prompt.
      --count<:count>  Some actions can take a count
      --includeValues  Some actions have an option to show the values.
      --key<:key>  Some actions need a key arg.
      --mode<:mode>  Some actions take a mode arg (often optional).
      --thumbprint<:thumbprint>  Some actions need a certificate thumbprint arg.
      --debug[:debug]  This will enable debugging output.
      --uri<:uri>  Some actions need a uri arg.
      --help  Print this help and exit.

    Sunday, March 12, 2017

    Configure static IP address in Windows PE environment


    You've made backup image of your Windows OS and put it on the network share. 
    At some point you need this backup image for system recovery purposes. You boot from Windows PE disk to recover system image backup in non-DHCP enabled network. You can't access to your backup image file via network. You must setup static IP on the network interface prior to network discovery and image recovery.

    Quick notes

    Enter to command prompt:

    netsh interface ip show addresses
    netcfg -winpe
    wpeutil InitializeNetwork

    or simply


    NOTE: The default connection names are "Local Area Connection" or "Ethernet" for wired adapters and "Wireless Network Connection" for Wi-Fi adapters. You can check network connection name by entering "ipconfig" command.

    The IP address order: client IP, subnet mask, and gateway IP.

    netsh interface ipv4 set address "Local Area Connection" static
    netsh interface ipv4 add dns "Local Area Connection" index=1
    netsh interface ipv4 add dns "Local Area Connection" index=2

    The service has not been started