Sunday, September 3, 2017

Recover from failed attempt to rebuild Windows Server performance counters without backup

Scenario:

For any reason you try to recover/rebuild Windows Server performance counters using KB2554336 or other "smart" blog. After procedure you have some counters are missing and some 3rd party services cannot start at all. In my case it was one of Skype for Business Edge servers from Edge pool with Application Log errors like:
Log Name:      Application
Source:        usbperf
Date:          9/3/2017 11:23:41 AM
Event ID:      2001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
Unable to read the "First Counter" value under the usbperf\Performance Key. Status codes returned in data. 
Log Name:      Application
Source:        Microsoft-Windows-IIS-W3SVC-PerfCounters
Date:          9/3/2017 11:23:41 AM
Event ID:      2002
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
Setting up Web Service counters failed, please make sure your Web Service counters are registered correctly. 
Log Name:      Application
Source:        Microsoft-Windows-Perflib
Date:          9/3/2017 11:20:53 AM
Event ID:      1008
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
The Open Procedure for service "aspnet_state" in DLL "C:\Windows\System32\aspnet_counters.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.
Log Name:      Application
Source:        Application Error
Date:          9/3/2017 12:14:05 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      server1.contoso.com
Description:
Faulting application name: ReplicaReplicatorAgent.exe, version: 6.0.9319.0, time stamp: 0x55276ea8
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x00007ff7ae9db0f1
Faulting process id: 0xf28
Faulting application start time: 0x01d32494fba1def3
Faulting application path: D:\Skype for Business Server 2015\Server\Replica Replicator Agent\ReplicaReplicatorAgent.exe
Faulting module path: unknown
Report Id: 3b46686b-9088-11e7-80e6-00155d083802
Solution:

Option 1. Recover server from backup/snapshot/etc.

Option 2. In my case I didn't have backup. We'll focus on this recovery scenario.

Step 1. Make performance counters backup from similar neighbor "healthy" server.

"Healthy" server should be configured with exactly the same Windows Server and Application components (ex. server from a server pool, a node from failover or nlb cluster, etc.)

from cmd (Run As Administrator)
lodctr /S:C:\temp\counters_healthy-server.ini
Step 2. Copy counters_healthy-server.ini file from "healthy" to "unhealthy" server with missing counters.

Step 3. Make performance counters backup on "unhealthy" server (just for case)

from cmd (Run As Administrator):
lodctr /S:C:\temp\counters_unhealthy-server.ini
Step 4. Rebuild performance counters from counters_healthy-server.ini backup file on "unhealthy" server:

from cmd (Run As Administrator):
lodctr /R:C:\temp\counters_healthy-server.ini
Step 5. Resync the counters with Windows Management Instrumentation (WMI):

from cmd (Run As Administrator):
WINMGMT.EXE /RESYNCPERF
Step 6. Restart "unhealthy" server. Feel happy.

Moral: always make backup prior to any change even if it looks like very simple change.

References:

Tuesday, August 29, 2017

Validation errors for users/contacts/groups in the Office 365 portal

Scenario:

Let's say you get something like:

Get-MsolUser -UserPrincipalName user@domain.com | fl *status*,errors
OverallProvisioningStatus : Success
ValidationStatus          : Error
Errors                    : {Microsoft.Online.Administration.ValidationError}

How to get more details about the "ValidationStatus" Error?

Solution:

for users:

$errors = (Get-MsolUser -UserPrincipalName "user@domain.com").Errors

for contacts:

$errors = (Get-MsolContact -ObjectID <Object_ID>).Errors

for groups:

$errors = (Get-MsolGroup -ObjectID <Object_ID>).Errors

and then expand $errors variable:

$errors | % {"`nService: "+ $_.ErrorDetail.Name.split("/")[0]; "Error Message: "+ $_.ErrorDetail.ObjectErrors.ErrorRecord.ErrorDescription}  

Example:

$errors = (Get-MsolUser -UserPrincipalName user@domain.com).Errors

$errors | % {“`nService: ” + $_.ErrorDetail.Name.split("/")[0]; “Error Message: ” + $_.ErrorDetail.ObjectErrors.ErrorRecord.ErrorDescription}

Service: exchange
Error Message: Exchange can't disable the mailbox "NAMPR07A002.prod.outlook.com/Microsoft Exchange Hosted Organizations/xxxxx.onmicrosoft.com/Soft Deleted Objects/UserNameHere" because it is on In-Place Hold.

Reference:


  • You see validation errors for users in the Office 365 portal or in the Azure Active Directory Module for Windows PowerShell
  • Saturday, August 12, 2017

    Exchange Cutover Migration Endpoints. PowerShell configuration samples

    # Exchange Server migration account credentials
    $cred = Get-Credential

    # If migration account is not hidden from GAL and you are going to use Autodiscover
    $TSMA = Test-MigrationServerAvailability -ExchangeOutlookAnywhere -Autodiscover -EmailAddress administrator@contoso.com -Credentials $cred

    # If migration account is hidden from GAL and you are going to use Autodiscover
    $TSMA = Test-MigrationServerAvailability -ExchangeOutlookAnywhere -Autodiscover -EmailAddress administrator@contoso.com -Credentials $cred -SourceMailboxLegacyDN "/o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=3fd752ef466144ff83d418e6060da2
    26-Stani"

    # If migration account is not hidden from GAL and you are not going to use Autodiscover
    $TSMA = Test-MigrationServerAvailability -ExchangeOutlookAnywhere -EmailAddress administrator@contoso.com -Credentials $cred -RPCProxyServer "us.exg7.exghost.com" -ExchangeServer "ac2b1533-71c9-48e3-9a58-8f7ce004cf01@contoso.com"

    # If migration account is hidden from GAL and you are not going to use Autodiscover
    $TSMA = Test-MigrationServerAvailability -ExchangeOutlookAnywhere -EmailAddress administrator@contoso.com -Credentials $cred -RPCProxyServer "us.exg7.exghost.com" -ExchangeServer "ac2b1533-71c9-48e3-9a58-8f7ce004cf01@contoso.com" -SourceMailboxLegacyDN "/o=First Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=3fd752ef466144ff83d418e6060da2
    26-Stani"

    New-MigrationEndpoint -ExchangeOutlookAnywhere -Name CutoverEndpoint -ConnectionSettings $TSMA.ConnectionSettings

    where:
    • the -RPCProxyServer parameter specifies the FQDN of the RPC proxy server for the on-premises Exchange server (ExternalHostname); 
    • the -ExchangeServer parameter specifies the FQDN of the on-premises Exchange server (InternalHostname). 
    Get-OutlookAnywhere | fl ExternalHostname,InternalHostname

    PowerShell command above gives you right values (assuming that Outlook Anywhere is configured correctly)

    Question: Why will you choose not to use Autodiscover for migration endpoint?

    Answer: After DNS MX and A/CNAME (autodiscover.contoso.com) records switch-over to Exchange Online you can continue mailbox items sync from source Exchange platform. If you use "autodiscover" method for Cutover endpoint then:

    • Per Microsoft "Autodiscover service will be used to connect to each user mailbox in the migration batch"
    • you will be pointed (after DNS switch-over autodiscover.contoso.com to autodiscover.outlook.com) to destination Exchange Online platform instead of source Exchange Server and mailbox sync could be broken.
    References:

    Keep installed drivers after running SYSPREP

    Scenario:

    I need to keep installed drivers on an OS even after running SYSPREP, how do I do this?

    Solution:

    There may be scenarios where its critical drivers are not removed from an OS instance when SYSPREP is executed. To do this run the following prior to SYSPREP
    • Run regedit and move to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\Sysprep\Settings\sppnp 
    • Select Edit - New DWORD value 
    • Enter a name of PersistAllDeviceInstalls 
    • Set to a value of 1 and click OK
    References:

    SCOM Management Pack is dependent on Microsoft.SystemCenter.SecureReferenceOverride

    Scenario:

    You try to delete SCOM 2012+ Management Pack. You get error:
    "Following Management Packs depend of the management pack you are trying to delete. Please delete the following management packs manually first.
    Microsoft.SystemCenter.SecureReferenceOverride"
    Solution:

    No manual XML file editing.
    • Open a PowerShell session with the Operations Manager module/snappin loaded.
    • Type: $MP = Get-SCOMManagementpack -Name Microsoft.SystemCenter.SecureReferenceOverride
    • Now we can view the referenced management packs by typing $MP.References
    • From the list of items in the Key column, note down the alias of your MP you wish to delete. If you are having trouble finding it, the Value column will list the full ID of the MP.
    • Now that we know the MP alias, we can remove it from the Secure Reference MP by typing $MP.References.Remove(“yourMPAliasGoesHere“)
    • Now we can verify the MP is valid by entering $MP.Verify() to ensure there are no orphaned overrides, etc.
    • Finally, we can save our changes by typing: $MP.AcceptChanges()

    Once this is done, give the SCOM management group a chance to catch up (you might have to wait a minute and then refresh your console). When you now check the Dependencies tab of the MP you want to delete, you’ll see that the SecureReferenceOverride MP is no longer listed and you’ll be able to remove your MP.

    Reference:

    Monday, August 7, 2017

    Can't Install-WindowsFeature on Windows Server using PowerShell

    Scenario:

    You can't Install-WindowsFeature (Add-WindowsFeature for 2008 R2) on Windows Server using PowerShell for various reasons.
    Recently I had a problem with broken WMI where I couldn't manage Windows Server features of affected server. My wish was to make Full Server backup prior to any changes using built-in "Windows Server Backup" and I couldn't install this feature using PowerShell.

    Solution:

    Following DISM commands running from an Elevated Command Prompt (Admin) saved my ass:
    Dism /online /enable-feature /featurename:WindowsServerBackup
     
    Deployment Image Servicing and Management tool
    Version: 6.3.9600.17031
    
    Image Version: 6.3.9600.17031
     
    Enabling feature(s)
    [==========================100.0%==========================]
    The operation completed successfully.
    Dism /online /enable-feature /featurename:WindowsServerBackupSnapin
     
    Deployment Image Servicing and Management tool
    Version: 6.3.9600.17031
     
    Image Version: 6.3.9600.17031
     
    Enabling feature(s)
    [==========================100.0%==========================]
    The operation completed successfully.
    You can install any Windows feature in similar way. Get list of all available features and their names on server:
    Dism /online /get-features /format:table | more
    References:

    WMI is broken. "Configuration refresh failed with the following error: Invalid class"

    Scenario:

    When we start Server Manager on Windows Server 2012 R2 the flag lights up "Red" almost immediately with the error:
    "Configuration refresh failed with the following error: Invalid class"


    Get-WindowsFeature ends with the same "Invalid class" error.

    Solution:

    Backup WMI:

    • Launch the WMI MMC snapin by Start -> Run -> then enter WMIMGMT.MSC
    • Right click WMI Control (Local) and click Properties
    • Tab Backup/Restore, press Back Up Now...

    Recompile .mof & .mlf files by running the following commands from an Elevated Command Prompt (Admin):
    CD C:\Windows\System32\WBEM
    dir /b *.mof *.mfl | findstr /v /i uninstall > moflist.txt & for /F %s in (moflist.txt) do mofcomp %s
    Refresh "Server Manager" or re-open it.

    References:

    Thursday, June 1, 2017

    Cannot sign in to Skype for Business because your computer clock is not set correctly

    Scenario

    User cannot sign-in to Skype for Business via client and has error:


    "Cannot sign in to Skype for Business because your computer clock is not set correctly. To check your computer clock settings, open Date and Time in the Control Panel."
    Solution

    Per KB2581291 Microsoft suggests to make sure that the computer's clock and time zone settings are set correctly. And it's correct article except the fact that the registry key "ClockSkew" is located under another registry path:

    HKEY_CURRENT_USER\Software\Microsoft\MSOIdentityCRL

    instead of documented:

    HKEY_USERS\.DEFAULT\Software\Microsoft\MSOIdentityCRL

    In my case user messed with time and had unchecked "Daylight Saving Time" setting.


    The Time Zone, the "Daylight Saving Time" settings were corrected, time synchronization were completed, registry key "ClockSkew" was deleted and Skype for Business client could sign-in.

    References

    Sunday, March 19, 2017

    Reset Skype for Business Statistics Manager (StatsMan) history

    Scenario

    You have a brand new or existing deployment of Skype for Business Statistics Manager. Your graph shows some very old statistics (few days/weeks ago) as well as recent statistics (by default: 24 hours old) in a single chart.



    Cause
    Don't know yet. Maybe related to Open Source of Redis in-memory caching system :-P

    Solution

    Reset statistics completely (--age:0) or to 24 hours old.
    1. Run as Administrator "cmd" on server where SfB Statistics Manager Listener is installed:
    2. Run either:

    >"C:\Program Files\Skype for Business Server StatsMan Listener\PerfAgentStorageManager.exe" --redis:localhost --action:cleanup --age:0

    or

    >"C:\Program Files\Skype for Business Server StatsMan Listener\PerfAgentStorageManager.exe" --redis:localhost --action:cleanup --age:24

    Using age: 1.00:00:00
    FillNameCacheAsync of 6,889 counters took 180ms
    GetLastWriteTimesAsync of 6,889 counters took 56ms
    Scanned through 7,389 redis keys in 79ms and found 36 that should be deleted.
    36 counters, 0 hosts, 36 storage keys, haven't been written to in 6.00:00:00
    Do you want to remove the above hosts,36 counters and 36 storage keys? Y|[N]:y
    Removed: 36 keys in 29ms
    BulkDereferenceCounters: Attempting to dereference 36 storage keys
    BulkDereferenceCounters: Deleted 36 last write time entries in 10ms
    BulkDereferenceCounters: Removed 36 storage key entries from set in 1ms

    Deleted 0 server infos.

    I got normal graph after clearing history.



    More information 
    about PerfAgentStorageManager.exe /?

      --action<:action>

        updateServerInfo : Need --redis : Must provide --file or (--hostname and one or more --population options), --file should be in the format of getServerInfo output
                         : optional --noPrompt (will automatically do it without asking)
                         : optional --disableUpdate (will do adds only)
                         : optional --debug will show extra output
                         : The following shard scheme docs are only for multi-azure storage account setups:
                         : optional --shardScheme=none (to disable updating shard assignement for new servers)
                         : optional --shardScheme=<scheme>
                         :          <scheme> is one or more population matches separated by comma.
                         :          The default is Pool_,Site_ and it is recommended to not change this.
                         :          This will look at each new server's populations and try to match (start with) with a
                         :          scheme component then it will assign to that populations shard if it exists.
                         :          So for the default scheme it will look for the servers Pool shard, if that doesn't exist
                         :          then it will look for the servers Site shard, if that doesn't exist it will just stay as
     default

        getServerInfo : Need --redis : optional --uri for the location to upload to, Default is redishash:///_ServerInfo
                      : optional --file will output the xml for edit
                      : optional --mode=command will output the information with the command line options for PerfAgentStorageManager
                      : optional one or more --population will get only the servers in one of those populations

        getRedirectCandidates : Need --redis, default is to find populations with a single server where the name of the server is the last part of the population
                      : optional --mode=<current>, current will retrieve the current list of configured redirected populations
                      : optional --file=filename.csv will output a file that can be used with --action=updateRedirectCandidates
                      : example: Pool_mySEserverFE01 server=mySEServerFE01

        updateRedirectCandidates : Need --redis --file is a csv with Population,Server

        getShardInfo : Need --redis.
                    : Optional: One or more modes
                    : --mode=verbose to output every assignment
                    : --mode=password to see the configured password (careful!)
                    : --mode=history to see the history of all shard assignments (assumes --mode=verbose)

        updateShardAssignments : Need --redis and one or more --population or a single --hostname and --mode=specific or --mode=population and --value
                    : --value=# the number of the shard to assign them to
                    : mode=specific means assign that population or host to the shard
                    : mode=population means assign the population(s) to the shard and all of the servers in all populations given
                    : mode=correction (be careful with this) means update any shard assignement that matches -value=shardid and -match=tableID (see -a=getshardinfo -verbose)
                    : this is used when a mistake in timing was made and the shard assignement hasn't been used yet and this will update to the next
                    :if you get this wrong and use it on something that is in use, the data will be missing
    and not easily recovered
                    : The assignment won't switch until the next 12 hour period. Be careful that you assign far enough from daily 12 hour boundaries
                    : Optional: if in --mode=population you can also add a --match=<population match> to also add any population to the shard for servers
                    : in the given population. For example: --mode=population --population=Site_Blah --match=Pool_ this will add any Pool in has servers in that
                    : site. It will ask you for each one unless you have --noPrompt

        updateAzureShard : Need --redis and --value=<shard ID> --azure=<accountName> --azurePassword=<azure key> --thumbprint=<password decryption certificate thumbprint>
                     : Optional --noPrompt to not prompt, you can add a --new to fail if expect it to be a new account (it will fail if the account or shard ID exists)

        validateSharding : Need --redis and --azure=<accountName> --azurePassword=<azure key> (for the default account)
                     : Optional --noPrompt will not prompt for keys to check but you'll need to provide --value=<hostname or population name or all>

        getRedirectedCounters : Need --redis

        deleteServerInfo : Need --redis and --file or one or more --hostname entries : File contains names of servers (one p
    er line). optional --uri for the key to upload to, Default is redishash:///_ServerInfo (only redishash uri supported)

        checkServerInfo : Need --redis and --file or one or more --hostname entries : File contains names of servers (one pe
    r line).

        cleanup : Need --redis, this will cleanup all hosts, populations and counters that haven't been written to in the given age time
                : Optional --age=<hours> (default 24)
                : Optional --match=<regex> to only match certain hosts/populations
                : Optional --notmatch=<regex> to exclude certain hosts/populations

        listKeys : Need --redis optional --file and --match and --key (key will make this a hash key list) --includeValues will show the value of the hash keys as well

        deleteKeys : Need --redis and --match  optional --notMatch, optional: --count=<batchSize> default=1000

        listHostIds : Need --redis and --match  optional --notMatch

        backupIdentifiers : Need --redis and --file, optional: --mode=overwrite

        subscribe : Need --redis and --counter

        listCounterStorageNames : Need --redis optional --file and --match and --notMatch

        listBuckets : Need --redis optional --match and --notMatch, optional --mode=<counters>, counters mode will try to distill down to the list of perf counters polled

        readListValues : Need --redis and --key optional --count

        deleteHashKeys : Need --redis and --key optional --match and --count (this will decide batch size), you will be prompted

        lookupStorageName: Need --redis and one or more --counter and/or --bucket this will lookup the counter or bucket storage names, optional: --mode=redirected

        findBadServers: Need --redis and optional --bucket (it will by default use \\*\Memory\% Memory Free_Maximum\@ bucket)

        getCounterValues : Need --redis and --counter : Optional --file=<csv output file path>

        getCounterValuesRange : Need --redis and --azure. Optionally need --counter (default is memory max) --startTime and --endTime (both in local time), defaults 24 hours ago and now

        counterLastWriteTimes: Need --redis optional --age=<seconds> will only print counters older than this, optional --count will limit the output, optional --mode=host will output the hosts and their youngest value age

        getBucketValues : Need --redis and optional --bucket (default will be \\*\Memory\% Memory Free_Maximum\@), optional --count will limit the output

        getBucketDueTimes: Need --redis optional --key default is _BucketDueTime

        getStats: Shows info about number of counters/buckets/etc. Need --redis

        listAggLocks: Need --redis. Shows which server owns which agg lock for buckets.

        setting: Need --redis and --setting=SettingName and --value=[true|false|int32#]. Updates/gets.

      --setting<:setting>
        all: retrieves all of the current values
        aggregation: optional --value=[true|false] (if not given it gets the current value)
        redirectedAggregations: optional --value=[true|false] (if not given it gets the current value)
        persistentStore: optional --value=[true|false] (if not given it gets the current value)
        rediscounterStore: optional --value=[true|false] (if not given it gets the current value)
        bucketManagerMaxConcurrency: optional --value=int32# (if not given it gets the current value)

      --value<:value>  Some actions require a 'value' parameter.
      --redis<:redis>  Set the redis server configuration
      --redisDatabase<:redisDatabase>  Set the redis database number. Default: 0
      --redisPassword<:redisPassword>  Optional redis password. Otherwise it will be looked up from the config.
      --azure<:azure>  Set the azure storage account name
      --azurePassword<:azurePassword>  Optional azure storage password. Otherwise it will be looked up from the config.
      --startTime<:startTime>  Some actions take a startTime arg.
      --endTime<:endTime>  Some actions take a endTime arg.
      --file<:file>  Some actions take a file arg.
      --hostname<:hostname>  Some actions take a hostname arg.
      --population<:population>  Some actions take a population arg.
      --match<:match>  Some actions uses a pattern or match.
      --notMatch<:notMatch>  Some actions uses a pattern or match to exclude.
      --new  Some actions are being tested with a 'new' mode.
      --disableUpdate  Some actions need a way to disable updates (and do adds only).
      --counter<:counter>  Some actions take a counter arg.
      --bucket<:bucket>  Some actions take a bucket arg.
      --shardScheme<:shardScheme>  Some actions take a shard scheme.
      --age<:age>  Some actions take an age arg. In seconds.
      --noPrompt  Some actions prompt to continue, this will answer affirmative to any prompt.
      --count<:count>  Some actions can take a count
      --includeValues  Some actions have an option to show the values.
      --key<:key>  Some actions need a key arg.
      --mode<:mode>  Some actions take a mode arg (often optional).
      --thumbprint<:thumbprint>  Some actions need a certificate thumbprint arg.
      --debug[:debug]  This will enable debugging output.
      --uri<:uri>  Some actions need a uri arg.
      --help  Print this help and exit.

    Sunday, March 12, 2017

    Configure static IP address in Windows PE environment

    Scenario

    You've made backup image of your Windows OS and put it on the network share. 
    At some point you need this backup image for system recovery purposes. You boot from Windows PE disk to recover system image backup in non-DHCP enabled network. You can't access to your backup image file via network. You must setup static IP on the network interface prior to network discovery and image recovery.

    Quick notes

    Enter to command prompt:

    netsh interface ip show addresses
    netcfg -winpe
    wpeutil InitializeNetwork

    or simply

    wpeinit

    NOTE: The default connection names are "Local Area Connection" or "Ethernet" for wired adapters and "Wireless Network Connection" for Wi-Fi adapters. You can check network connection name by entering "ipconfig" command.

    The IP address order: client IP, subnet mask, and gateway IP.

    netsh interface ipv4 set address "Local Area Connection" static 192.168.0.101 255.255.255.0 192.168.0.1
    netsh interface ipv4 add dns "Local Area Connection" 192.168.0.10 index=1
    netsh interface ipv4 add dns "Local Area Connection" 192.168.0.11 index=2

    The service has not been started

    Friday, March 10, 2017

    Cannot grant AccessRights for RTCUniversalServerAdmins/RTCComponentUniversalServices of SfB to read Exchange UM AD objects using ExchUCUtil.ps1

    Scenario


    Per Microsoft guide you try to integrate Exchange Server (2013/2016) with Skype for Business (or Lync Server 2013) for UM. You run ExchUCUtil.ps1 integration script with required permissions (Exchange Organization administrator, Exchange Recipient administrator)

    C:\Program Files\Microsoft\Exchange Server\V15\Scripts> .\ExchUCUtil.ps1

    at the end you get something like:


    Grants Skype for Business Server 2015 permission to read Exchange UM Active Directory Domain Services objects

    Configuring UM IP Gateway objects...
    Pool: pool.domain.com
    A UMIPGateway already exists in Active Directory for the Lync Server pool. A new UM IP gateway wasn't created for the po
    ol.
    IsBranchRegistrar: False
    MessageWaitingIndicatorAllowed: True
    OutcallsAllowed: True
    WARNING: The command completed successfully but no settings of 'pool' have been modified.
    Dial plans: Contoso Dial Plan

    Permissions for group domain.com\RTCUniversalServerAdmins

    ObjectName                              AccessRights                            Configured
    ----------                              ------------                            ----------
    First Organization                      ListChildren                            True
    UM DialPlan Container                   ListChildren, ReadProperty              True
    UM AutoAttendant Container              ListChildren, ReadProperty              True
    Administrative Groups                   ListChildren, ReadProperty              False

    Permissions for group domain.com\RTCComponentUniversalServices

    ObjectName                              AccessRights                            Configured
    ----------                              ------------                            ----------
    First Organization                      ListChildren                            True
    UM DialPlan Container                   ListChildren, ReadProperty              True
    UM AutoAttendant Container              ListChildren, ReadProperty              True
    Administrative Groups                   ListChildren, ReadProperty              False

    PoolFqdn                                UMIPGateway                             DialPlans
    --------                                -----------                             ---------
    pool.domain.com                         pool                                    {Contoso Dial Plan}

    You may notice that some ObjectNames have "Configured" status as "False".
    In the event log "MSExchange Management" (Applications and Service Logs) following event maybe registered after running this script:

    Log Name:      MSExchange Management
    Source:        MSExchange CmdletLogs
    Date:          10/03/2017 10:38:07
    Event ID:      6
    Task Category: (1)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      computer1.domain.com


    The following information was included with the event:


    Add-ADPermission -InheritanceType "All" -AccessRights ("ListChildren, ReadProperty") -Identity "Administrative Groups" -Debug "True" -User "domain.com\RTCUniversalServerAdmins" -ErrorVariable "ErrorList"
    domain.com/UMAdmin
    Local-ConsoleHost-Unknown
    12164 powershell.exe
    00:00:53.5386187
    View Entire Forest: 'True',
    Microsoft.Exchange.Configuration.Tasks.ManagementObjectAmbiguousException: There are multiple objects matching the identity "Administrative Groups". Please specify a unique value.

    The same error occurs if to run script commands manually (obviously):

    Add-ADPermission -InheritanceType "All" -AccessRights ("ListChildren, ReadProperty") -Identity "Administrative Groups" -User "domain.com\RTCUniversalServerAdmins"  

    Add-ADPermission -InheritanceType "All" -AccessRights ("ListChildren, ReadProperty") -Identity "Administrative Groups" -User "domain.com\RTCComponentUniversalServices"  

    Cause


    In my case my domain.com forest had Organization Unit (OU) named like "Administrative Groups" and created manually by helpdesk group.  As you probably know "Administrative Groups" AD container with the same exact name is used for Exchange AD objects by Microsoft design. But I'm sure the same issue occurs if you have OU named like: "First Organization", "UM DialPlan Container", "UM AutoAttendant Container". Don't name OUs with such system names and you'll never get this problem.

    Solution


    Rename OU "Administrative Groups" created by helpdesk to something else (ex. "Admin Groups") and run ExchUCUtil.ps1 script again.
    How to determine where this "smart" OU is located?

    Get-ADPermission -Identity "Administrative Groups" | Select-Object Identity | Sort-Object Identity | Get-Unique -AsString

    I my case I got something like:

    Identity
    --------
    Administrative Groups
    contoso.com/HelpDesk Groups/Administrative groups

    where "contoso.com/HelpDesk Groups/" is a path to the OU with a source of problems