Saturday, January 16, 2016

RemoteApp: Authentication Error has Occurred (Code : 0x607)

Issue: 
User receives error message 0x607 - An authentication error has occurred,
when attempting to launch a published RemoteApp or Desktop, check the logs on the client PC.


Cause:
In general there is problem with digital certificates you are using in RDS configuration.

Case 1:
This is likely due to the client PC not trusting your certificate. Either procure a certificate from a trusted third-party certificate authority, or the user will need to install and trust the root certificate authority and any intermediate certificate authorities in the certificate chain.

Case 2:
If the user is unable to contact the certificate revocation list that is listed on the certificate (third-party or internal CA) to verify the revocation status of the certificate, they will receive the 0x607 error. So for example, if the URL of your certificate vendor is being blocked by a corporate web filter, you will have receive this error when launching a RemoteApp.

Case 3:
Certificate name mismatch during external access to RemoteApp or Session Host Desktop via Remote Desktop Gateway and 3rd-party digital certificate:
  • rdgw.extdomain.com (Public name of RDS Gateway) configured with *.extdomain.com certificate
  • rdcb.extdomain.com (Public/Internal name of RDS Connection Broker) configured *.extdomain.com SSO certificate
  • rdsh1.intdomain.com (Internal name of RDS Session Host) protected with *.extdomain.com certificate. Here is where error occurs.
  • rdsh2.intdomain.com (Internal name of RDS Session Host) protected with *.extdomain.com certificate. Here is where error occurs.

Use Ryan's script to change *.extdomain.com to another *.intdomain.com trusted certificate (internal or third-party CA) on Remote Desktop Session Hosts. Another solution is to keep split-brain DNS scenario when design Active Directory/DNS domain name so you can use the same wildcard digital certificate internally and externally.

Office 365 Import Service: Get-MailboxImportRequest and Get-MailboxImportRequestStatistics

Office 365 Import Service (currently in Preview and free) can provide PST files import via network copy and it is a two-step process, from copying the files to Azure and then importing the PST into Exchange online. Recently I've described how to throttle traffic during PST upload to Azure Storage account. Now I'd like to document issues we've got during Import process (Mailbox Import Request):


There are two PowerShell cmdlets you can use to control import process: Get-MailboxImportRequest and Get-MailboxImportRequestStatistics:

PS > Get-MailboxImportRequest

Name                                           Mailbox                  Status
----                                           -------                  ------
54db24b4-b5bb-4750-9455-a11c21c698ac           sarag                    Completed
9c70991e-4998-4667-9e22-ff567c982971           btitus                   Completed
c0f7dbd0-e415-495f-b283-01e16e2d4ddb           ruth                     Completed
4153202d-2eb3-406f-8e43-6895dc70e3ac           larry                    Completed
b1dc5874-ae7b-4d70-8987-6171b788521f           ckepes                   Completed
0e00d89c-199f-4fb9-8f5a-c86a97ca433b           btitus                   Completed
5e9397b2-6773-4bfa-989d-c98b9df0b8c0           sarag                    Completed
6cce4b99-dc13-4b3e-9646-232c3c979e7e           pete                     Completed
e3915ef6-c57b-4450-a629-96d823f758fb           roughmill                Completed
14d4a6fa-5efc-4fc9-a0bc-3752767934b7           btitus                   Completed
0ec3c64c-71be-455e-8e53-29ad6226276d           ruth                     InProgress
1efae770-5acb-47a4-94c7-aa9df7197407           ckepes                   Failed
ffd410bc-5767-4f7b-8c34-5119b0e96c7d           adam                     Completed
6312d55b-dfe3-4fba-8193-a408861e10b0           adam                     Completed

PS > Get-MailboxImportRequest -status failed | fl

RunspaceId     : 35130605-b66f-412c-b680-636bf10513eb
FilePath       : AzureImport
Mailbox        : ckepes
Name           : 1efae770-5acb-47a4-94c7-aa9df7197407
RequestGuid    : b132acbb-1272-4959-bdc5-ebe1e0f72b76
RequestQueue   : NAMPR10DG030-db079
Flags          : IntraOrg, Pull, Suspend
BatchName      :
Status         : Failed
Protect        : False
Suspend        : True
Direction      : Pull
RequestStyle   : IntraOrg
OrganizationId : XXXXXXXXX.PROD.OUTLOOK.COM/Microsoft Exchange Hosted Organizations/contoso.onmicrosoft.com - XXXXXXXXX.PROD.OUTLOOK.COM/ConfigurationUnits/contoso.onmicrosoft.com/Configuration
WhenChanged    : 12/23/2015 10:45:41 PM
WhenCreated    : 12/23/2015 7:27:18 PM
WhenChangedUTC : 12/23/2015 8:45:41 PM
WhenCreatedUTC : 12/23/2015 5:27:18 PM
Identity       : ckepes\1efae770-5acb-47a4-94c7-aa9df7197407
IsValid        : True
ObjectState    : New

If you will try Get-MailboxImportRequest | Get-MailboxImportRequestStatistics you will end with:

PS > Get-MailboxImportRequest | Get-MailboxImportRequestStatistics
Couldn't find a request that matches the information provided. Reason: No such request exists in the specified index.
   + CategoryInfo : NotSpecified: (:) [Get-MailboxImportRequestStatistics], ManagementObjectNotFoundException
   + FullyQualifiedErrorId : [Server=XXXXXXXXXXX,RequestId=7ebf175c-0e3c-41dd-a9a8-92d7d07d3a2b,TimeStamp=12/24/201 5 1:24:29 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 928718B3,Microsoft.Exchange.Management.Migration.MailboxReplication.MailboxImportRequest.GetMailboxImportRequestStatistics
   + PSComputerName : outlook.office365.com

If you have some job failed you can try to fix PST file with scanPST tool and upload it again to Azure. Just like an Exchange migration, PST Imports are limited to the number of failed or corrupt messages that can be skipped in the process. The number is over 200 for an Azure PST Import and appears to be around 218 items. Downloaded Error Report information would be something like:
12/23/2015 8:45:30 PM A missing item was encountered: Missing Item () Subject:"", Folder:"Calendar"
12/23/2015 8:45:30 PM A missing item was encountered: Missing Item () Subject:"", Folder:"Calendar"
12/23/2015 8:45:30 PM A missing item was encountered: Missing Item () Subject:"", Folder:"Calendar"
12/23/2015 8:45:31 PM Fatal error TooManyMissingItemsPermanentException has occurred. 
After another try to increase limit of corrupted items we've ended with another frustrating error:
PS > Get-MailboxImportRequest -Status Failed | Set-MailboxImportRequest -BadItemLimit 500
WARNING: When an item can't be read from the source database or it can't be written to the destination database, it will be considered corrupted. By specifying a non-zero BadItemLimit, you are requesting Exchange not copy such items to the destination mailbox. At move completion, these corrupted items will not be available at the destination mailbox.
Couldn't find a request that matches the information provided. Reason: No such request exists in the specified index.
    + CategoryInfo          : NotSpecified: (:) [Set-MailboxImportRequest], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=xxxxxxxxxxxxxxx,RequestId=37c78693-1607-458c-be3f-7378e3507efa,TimeStamp=12/24/2015 1:00:47 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 3D3E249B,Microsoft.Exchange.Management.Migration.MailboxReplication.MailboxImportRequest.SetMailboxImportRequest
    + PSComputerName        : outlook.office365.com

PS > Get-MailboxImportRequest -Status Failed | Resume-MailboxImportRequest
Couldn't find a request that matches the information provided. Reason: No such request exists in the specified index.
    + CategoryInfo          : NotSpecified: (:) [Resume-MailboxImportRequest], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=xxxxxxxxxxxxxx,RequestId=a3616eca-8348-4e22-9cc8-05bea70d5114,TimeStamp=12/24/201
   5 1:03:27 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 3D3E249B,Microsoft.Exchange.Management.Migration.MailboxReplication.MailboxImportRequest.ResumeMailboxImportRequest
    + PSComputerName        : outlook.office365.com

So what's wrong here? If you are going to use Set-MailboxImportRequest or Get-MailboxImportRequestStatistics or similar import cmdlets make sure you are using -Identity switch and RequestGuid value from Get-MailboxImportRequest.


PS > Get-MailboxImportRequestStatistics -Identity b132acbb-1272-4959-bdc5-ebe1e0f72b76

Name                                   StatusDetail            TargetAlias  PercentComplete
----                                   ------------            -----------  ---------------
1efae770-5acb-47a4-94c7-aa9df7197407   CreatingFolderHierarchy      ckepes              5
Next PowerShell commands will work for single ImportRequest:
Set-MailboxImportRequest -Identity b132acbb-1272-4959-bdc5-ebe1e0f72b76 -BadItemLimit 500
Resume-MailboxImportRequest -Identity b132acbb-1272-4959-bdc5-ebe1e0f72b76
Next PowerShell commands will work for the few ImportRequests:
Get-MailboxImportRequest | % {Get-MailboxImportRequestStatistics $_.RequestGuid}
Get-MailboxImportRequest -Status Failed | % {Set-MailboxImportRequest  $_.RequestGuid -BadItemLimit 500 | Resume-MailboxImportRequest}

GA Manual released:
https://support.office.com/en-us/article/Use-network-upload-to-import-PST-files-to-Office-365-103f940c-0468-4e1a-b527-cc8ad13a5ea6

Friday, January 15, 2016

NVGRE Tenant VM is unable to communicate with the Internet as DNS cannot resolve

Issue:

HNV Multitenant Gateway VM is providing NAT/VPN capabilities for VMs in virtual networks so they could communicate with the Internet and through VPN tunnels.
HNV Gateway VM is able to communicate to the Internet and it is able to resolve any DNS names.
However the Tenant VM itself is unable to communicate to the Internet as it is unable to resolve any DNS names and to bind to DNS server with nslookup even though it can ping and "telnet dns-server-IP 53" (ex. 8.8.8.8). In my case it is not KB2918813

Hardware environment: 10G NIC cards with Broadcom (now it is called QLogic because of recent acquisition) chipset (BCM57xxx) and Encapsulated Task Offload  enabled (which should provide CPU offload related to NVGRE operations):
Driver: 7.12.32.0 or older
Firmware: 7.12.17 or older



Resolution:

The Broadcom (QLogic now) Network adapters have a feature called Encapsulated Task Offload which is enabled by default. If you disable Encapsulated Task Offload on Hyper-V hosts with Tenant VMs everything works fine. You can disable it by using the following PowerShell cmdlet.

Set-NetAdapterEncapsulatedPacketTaskOffload -EncapsulatedPacketTaskOffloadEnabled $false -Name "NICNAME"

where "NICNAME" in my case "SLOT 2 Port 1" and "SLOT 2 Port 2":
PS > Get-NetAdapterAdvancedProperty | where {$_.Displayname -like "encaps*"} | ft Name,DisplayName,DisplayValue,InterfaceDescription

Name                          DisplayName                   DisplayValue                  InterfaceDescription
----                          -----------                   ------------                  --------------------
SLOT 2 Port 1                 Encapsulated Task Offload     Disabled                      QLogic BCM57810 10 Gigabit...
SLOT 2 Port 2                 Encapsulated Task Offload     Disabled                      QLogic BCM57810 10 Gigabit...
Converged                     Encapsulated Task Offload     Disabled                      Microsoft Network Adapter ...

If you have other issues related to NVGRE I'd recommend to look at recommended hotfixes, updates, and known solutions for Windows Server 2012 R2 Hyper-V Network Virtualization (HNV) environments.