Exchange Hybrid Configuration Wizard

Introduction

Essentially, configuring an Exchange Hybrid is quite simple, as we will use a wizard that sets everything up based on the provided parameters in both the On-Premises and cloud environments. However, we must meet certain prerequisites beforehand. We also need to plan how we will operate the mail and potentially make some adjustments. It is important (almost necessary) to familiarize yourself with the workings of Exchange Hybrid - mail flow, connectors, domains before starting. I also recommend reading the information in the following article Exchange Hybrid - mailboxes and their locations, recipients, attributes and bug fixes.

Here we will also address a specific situation where we have users in Azure AD who already have an Exchange Online license assigned. In discussions, I have additionally found that if we have not yet set up Exchange Hybrid and do not want to use the cloud mailbox, we should not assign the license to the users (it is apparently meant that we should not assign the Exchange Online service from the Office 365/Microsoft 365 license). At the moment the license is assigned, a mailbox is created in the cloud (if Exchange Hybrid is not configured and the data is not properly synchronized).

I first tried everything in a lab environment, where the environment was gradually built up with a basic configuration. During this process, I encountered various issues, which I will mention. Most of them did not manifest in the production environment.

We start from the state where Exchange Server 2016 (CU18) or Exchange Server 2019 (CU7) with the latest Cumulative Updates (CU) is installed locally. And our Azure AD Connect is functioning correctly, including the options for Hybrid Exchange.

Documentation

• Exchange Server hybrid deployments
• Create a hybrid deployment with the Hybrid Configuration wizard
• Hybrid Configuration wizard
• Hybrid deployment prerequisites
• Exchange/Office 365 Hybrid Configuration Wizard - step by step guide
• How to Migrate Exchange to Office 365: Step by Step - Part 1
• Office 365 Hybrid Configuration Wizard Step by Step

Hybrid Configuration Wizard (HCW)

The Microsoft Office 365 Hybrid Configuration Wizard can be launched from various locations. It is recommended to check that it is the latest version (currently 17.0.4544.0, during the tests a version 17.0.5378.0 appeared). It should be available at https://aka.ms/HybridWizard. Otherwise, it can be launched directly from the Exchange Admin Center (EAC) - Hybrid - Configure button. This is available in both the On-Premises EAC and the cloud EAC (where the latest version should always be available).

The first time, the installation takes place, and next time it can be launched from the Start menu. The wizard can be run repeatedly when we want to make changes or when a new version is released. For example, when transitioning from 16.x to 17.x, the original version needs to be uninstalled and the new one installed. The old version displays a warning Hybrid Configuration Service may be limited.

Launching the wizard and issues

On the first run, the application is installed, and then the installed version is launched, but it always seems to access the internet. It calls (downloads) the file Microsoft.Online.CSE.Hybrid.Client.application from Microsoft. When there are any problems (the wizard doesn't start), we can look in the log %appdata%\Microsoft\Exchange Hybrid Configuration. This log also contains detailed information about the Exchange Hybrid configuration process. It's useful to check and perhaps back up after the wizard completes. In the future, you'll also find interesting information here about what was set up internally and in the cloud.

If the wizard fails to start, a solution may be to add the addresses https://outlook.office365.com and https://aka.ms, and possibly also https://shcwreleaseprod.blob.core.windows.net, to the Trusted sites in IE. I also repeatedly encountered the problem of restarting the wizard. When launching from the web or the start menu, something would just flash and nothing would start. This was temporarily resolved by right-clicking the downloaded file Microsoft.Online.CSE.Hybrid.Client.application, selecting Open with and ClickOnce Application Deployment Support Library. This did return an error that the application is installed from a different location. But now it could be launched from the start menu or EAC.

Information about Exchange organizations

The wizard detects the optimal On-Premises Exchange server (Client Access Server accessible from the internet) and the hosting of Exchange Online, or we can manually select them. Server checks are performed (for example, it must be licensed). The On-Premises Exchange Account (Exchange Administrator) and Office 365 Exchange Online Account (Tenant Administrator) are used (we set them up). The configuration is downloaded from both On-Premises and Online.

Hybrid configuration

Hybrid Configuration Wizard options

Next, we select several critical options, based on which the configuration is performed.

Hybrid Features

• Minimal Hybrid Configuration - only basic features
• Full Hybrid Configuration - full features

Hybrid Topology

• Exchange Classic Hybrid Topology - classic topology, requires a trusted certificate, full communication to and from the internet
• Exchange Modern Hybrid Topology - uses the Microsoft Hybrid Agent

Options for Teams integration

If we want to use MS Teams integrated with On-Premises Exchange, we must use the Exchange full Classic Hybrid deployment (Full Hybrid Configuration, Exchange Classic Hybrid Topology). Described in How Exchange and Microsoft Teams interact.

EWS (Exchange Web Services)

In the following steps, we select the Web Services Virtual Directory (EWS) accessible from the internet (it must have the external URL set). In the EWS settings, it is recommended (and required for certain operations) to enable MRS Proxy. The wizard should turn this on, but it is often mentioned that it is better to set it up beforehand.

Mail transfer and routing

Secure Mail Transport

TLS-encrypted and authenticated message transfer between the On-Premises and Online Exchange organizations is set up. We choose how we want to configure the mail transfer (standard Client Access and Mailbox servers). Which certificate to use, on which servers the Receive and Send Connector should be created.

Centralized Mail Transport

We can enable this if we want emails to the internet from mailboxes in the Exchange Online organization to be sent from the On-Premises servers. By default, they are delivered directly from the online environment.

Organization FQDN

We enter the public domain name used by the On-Premises Exchange. It is used for connectors for secure message transfer. Often, mail.company.com is used. The address is used in the Outbound connector in Exchange Online as the Smart Host for our domains (we can change it later).

Completion and configuration information

After making the configuration changes, the wizard will inform us whether everything was configured correctly, if a minor error occurred, or if the configuration could not be performed.

We can also use the command in the Exchange Management Shell on the On-Premises organization, which displays information about the hybrid configuration (it returns no data before the hybrid setup). Sample partial output.

[PS] C:\>Get-HybridConfiguration

RunspaceId                : 0751c621-c940-4eae-88a1-b8c37a0c50e3
ClientAccessServers       : {}
EdgeTransportServers      : {}
ReceivingTransportServers : {AZURETEST}
SendingTransportServers   : {AZURETEST}
OnPremisesSmartHost       : mailtest.company.com
Domains                   : {company.com}
Features                  : {FreeBusy, MoveMailbox, Mailtips, MessageTracking, OwaRedirection, OnlineArchive, SecureMail,
                            Photos}
ExternalIPAddresses       : {}
TlsCertificateName        : <I>CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US<S>CN= mailtest.company.com
ServiceInstance           : 0
AdminDisplayName          :
ExchangeVersion           : 0.20 (15.0.0.0)
Name                      : Hybrid Configuration

Changes made by the wizard

We can also take a look (and check) at some of the visible changes made by the wizard (we can save detailed outputs before running the wizard and then compare). Specifically, the following changes were made:

• a new accepted domain company.mail.onmicrosoft.com was set, Get-AcceptedDomain
• Email Address Policies were modified, Get-EmailAddressPolicy
• new Remote Domains were created, Get-RemoteDomain
• the organizational relationship was set, Get-OrganizationRelationship, Get-IntraOrganizationConnector
• a new Inbound and Outbound Connector were created in Exchange Online, Get-OutboundConnector, Get-InboundConnector
• a new Send Connector was created in the On-Premises Exchange, Get-SendConnector
• the Default Frontend Receive Connector in the On-Premises Exchange was modified, Get-ReceiveConnector
• either federation was set up, Get-FederationTrust, a new Exchange Delegation Federation certificate was created, Get-ExchangeCertificate, or (in the new HCW version primarily) the OAuth authentication relationship was set up (Exchange federated sharing) Get-AuthServer

Default Frontend Receive Connector

The wizard makes several modifications to the Default Frontend Receive Connector. I didn't find any information about this in the official documentation, but the changes are quite significant, so for example, sending mail from application servers may stop working. We can find out what changes the wizard made in the Hybrid Configuration Wizard log %appdata%\Microsoft\Exchange Hybrid Configuration\<date>.log. There is also a table of Receive Connectors with the current and expected state.

One change is that the Exchange Users group's permissions are removed from the connector if we had it set up (I described this in Exchange Server 2016 Mail Flow - Mail Routing and Connectors). We can easily restore the group using the Exchange Admin Center (EAC). Or we can create a special connector for application servers, but we would have to list them (or use a different port).

The problem will manifest in practice later. If a user from an application server tries to send mail, they will get an authentication error.

535 5.7.3 Authentication unsuccessful

In the FrontEnd SmtpReceive log on the Exchange server, we'll find the error.

Inbound authentication failed because the client FIRMA\name doesn't have submit permission.

When listing the permissions on the connector, we see the following.

PS C:\>Get-ReceiveConnector -Identity "MAIL1\Default Frontend MAIL1" | Get-ADPermission |
 Where-Object {$_.IsInherited -eq $false -and $_.User -ilike "NT*"} | Sort-Object User |
 FT User, ExtendedRights

User                              ExtendedRights
----                              --------------
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-SMTP-Accept-Authoritative-Domain-Sender}
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-SMTP-Accept-Any-Sender}
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-SMTP-Submit}
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-Accept-Headers-Routing}

After restoring the Exchange users, the original state is restored.

User                              ExtendedRights                                   
----                              --------------                                   
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-SMTP-Accept-Authoritative-Domain-Sender}
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-SMTP-Accept-Any-Sender}
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-SMTP-Submit}
NT AUTHORITY\ANONYMOUS LOGON      {ms-Exch-Accept-Headers-Routing}
NT AUTHORITY\Authenticated Users  {ms-Exch-Bypass-Anti-Spam}
NT AUTHORITY\Authenticated Users  {ms-Exch-Accept-Headers-Routing}
NT AUTHORITY\Authenticated Users  {ms-Exch-SMTP-Submit}
NT AUTHORITY\Authenticated Users  {ms-Exch-SMTP-Accept-Any-Recipient}

The second change is that a certificate selected in the wizard is set on the connector. If Exchange Online (using the Outbound connector) connects to this connector, it requires this certificate (authentication is performed), so we should leave it here. If we want to undo this setting for some reason, the following command will help.

Set-ReceiveConnector -Identity "MAIL1\Default Frontend MAIL1" -TlsCertificateName $none

Issues after completing the wizard

Non-functioning EAC link to Office 365

After completing the Hybrid Configuration Wizard, when logging into the Exchange Admin Center (EAC), there are two tabs at the top - Enterprise and Office 365. The second one is supposed to lead to the Exchange Online settings after logging in. Instead, it may open the pricing comparison https://go.microsoft.com/fwlink/p/?LinkId=258351 (the state before using HCW). Microsoft has a quite nonsensical article for this issue Office 365 link in On-Premises EAC goes to product comparison, not Exchange admin center.

In my test environment, everything worked, but not in production. I found a tip to perform an IIS restart on the Exchange server. When I then opened the EAC page on this server, the link worked. However, it's strange that when calling it from another computer, it didn't work. I tried again after a few days (I don't know if the server had been restarted in the meantime) and it was working from the network as well.

Checking federation and organizational sharing

O365 Hybrid - Exchange Federation Trust, How to address Federation Trust issues in Hybrid Configuration Wizard (HCW)

In older articles, for example, when experiencing issues with Teams accessing the On-Premises calendar, it was mentioned that it's necessary to check the functionality of the federation. Eventually, I read that the new version of HCW 17.0 in most cases does not set up the federation (it's used for some old versions of Exchange), but instead, OAuth is used, which is crucial for Teams. In the official documentation Hybrid configuration options, there is a mention:

The wizard checks to see if there is an existing OAuth authentication relationship or a federation trust with the Azure Active Directory authentication system for the on-premises organization. If not present, the wizard configures OAuth authentication or creates a federation trust for the on-premises organization with the Azure AD authentication system, depending on the type of on-premises Exchange configuration.

In any case, organizational sharing (Organizational Relationships/Sharing) is set up for both organizations. This is important for many features, such as sharing Free/Busy, Outlook on the web redirection, or MailTips.

We can look in the Exchange Admin Center (EAC) - Organization - Sharing. If we look in Exchange Online, we'll see information about the configured Organizational Sharing with our internal organization. When we look at the On-Premises server, we'll probably see the following.

If we click on Enable, information about the Organizational Sharing will be displayed.

We can easily check everything in PowerShell and verify that it's set up.

[PS on-premises] C:\>Get-FederationTrust | select TokenIssuerUri | ft -AutoSize

TokenIssuerUri
--------------
urn:federation:MicrosoftOnline

[PS Online] C:\> Get-FederationTrust | select TokenIssuerUri | ft -AutoSize

TokenIssuerUri
--------------
uri:WindowsLiveID
urn:federation:MicrosoftOnline

[PS on-premises] C:\>Get-OrganizationRelationship | select Domainnames

DomainNames
-----------
{company.mail.onmicrosoft.com}

[PS Online] C:\> Get-OrganizationRelationship | select Domainnames

DomainNames
-----------
{company.com}

OAuth authentication

Configure OAuth authentication between Exchange and Exchange Online organizations

For some features, it is necessary to have OAuth authentication set up and working. The Hybrid Configuration Wizard (HCW) should set everything up. In my test, I ran into an issue with Exchange Server 2019 CU 6 and the latest version of HCW 17.0.5378.0. The wizard displayed a warning at the end:

HCW8064 The HCW has completed, but was not able to perform the OAuth portion of your Hybrid configuration. If you need
 features that rely on OAuth, you can try running the HCW again or manually configure OAuth using these manual steps.

In the log %appdata%\Microsoft\Exchange Hybrid Configuration\<date>.log, there was an error:

*ERROR* 10242 [Client=UX, Page=Configuring, fn=RunWorkflow, Workflow=Hybrid, Task=IntraOrganization, Phase=Configure, Thread
=20] 
Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-AuthServer': A parameter
 cannot be found that matches parameter name 'DomainName'. ---> System.Management.Automation.RemoteException: A parameter
 cannot be found that matches parameter name 'DomainName'.

This is mentioned in the article Can't view Online user free/busy status in Exchange Server 2019 and 2016. You need to install the latest Cumulative Update, which includes the new DomainName parameter.

Issues with Teams access to the On-Premises Exchange calendar

How Exchange and Microsoft Teams interact

For some, the main motivation for configuring Exchange Hybrid may be to allow MS Teams to access the calendars of users who are on the internal Exchange servers. A variety of problems can arise here.

There is a lot of information in the Czech article Exchange Hybrid a Microsoft Teams, another interesting one is Configuring Teams calendar access for Exchange on-premises mailboxes.

From the Teams application, we can save logs using the key combination CTRL+ALT+SHIFT+1 and examine the file MSTeams Diagnostics Log mm_dd_yyyy__hh_mm_ss_XXXX.txt. If we find the following line, it means the mailbox access failed.

UserAppsStore: Skipped calendar app with isFirstParty as true. isMailboxDiscoverable: false, isFreemiumTenant: false,
 enableFreemiumCalendar: true

Autodiscover test

There are many things we should check. We can try an Autodiscover access to the internal mailbox through Online.

https://outlook.office365.com/autodiscover/autodiscover.json?Email=bouska@company.com&Protocol=EWS&RedirectCount=5

There should be a redirect to the on-premises Autodiscover and the EWS address should be returned in the body.

https://autodiscover.company.com/autodiscover/autodiscover.json?Email=bouska%40company.com&Protocol=EWS&RedirectCount=6
{"Protocol":"EWS","Url":"https://mail.company.com/EWS/Exchange.asmx"}

We can also verify the availability of the REST API, which should return the URL.

https://mail.company.com/autodiscover/autodiscover.json/v1.0/bouska@company.com?Protocol=Rest
{"Protocol":"Rest","Url":"https://mail.company.com/api"}

Service Principal Name (SPN) in Azure AD

Another tip is to verify that the SPN is set up properly in Azure AD. There should be records for the internal Autodiscover and EWS, it's about https://namespace, not 00000002-0000-0ff1-ce00-000000000000/namespace.

Install-Module -Name MSOnline
Connect-MsolService
Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 | select -ExpandProperty ServicePrincipalNames