Exchange Hybrid - mail flow, connectors, domains

Note: I often use the term local or internal instead of On-Premises. Generally, for Microsoft cloud services (Azure AD, Office 365, Microsoft 365, Exchange Online) either the term cloud or online is used. On-Premises Exchange is simply referred to as Exchange, and the cloud version as Exchange Online or the abbreviation EXO. For the Hybrid Configuration Wizard, the abbreviation is HCW. Microsoft itself uses the terms Microsoft 365, Office 365, Exchange Online (often in the context of an organization / tenant) in various ways. Another term related to EXO is Exchange Online Protection (EOP), which usually refers to the service that receives mail into EXO and performs protection. In various addresses, I use the term tenant, which in practice is replaced by the name of the given Tenant.

Receiving and Sending Messages

Transport options in Exchange hybrid deployments, Transport routing in Exchange hybrid deployments

The basic thing is receiving and sending mail for our domain firma.cz. When we suddenly have two different Exchange organizations that share the same domain name. Incoming messages are primarily delivered according to the DNS MX record. Depending on whether the record points to internal servers or the cloud, all internet messages will be received either by the On-Premises or Exchange Online organization. If the recipient's mailbox is located in the other organization, the message is then forwarded using the created connectors.

Note: If we decide to set Microsoft 365 or Office 365 as the primary, and route the MX records to Exchange Online, we need to make several changes that the HCW does not perform.

When sending messages, the message is normally sent by the organization where the mailbox is located. In the Hybrid Configuration Wizard, we have the option to enable Centralized Mail Transport. Then, messages sent to the internet from mailboxes in Exchange Online are forwarded to the On-Premises servers for sending (all outgoing mail is sent by the internal servers). If we want to send everything via the cloud servers, we can modify (or create) a connector on the On-Premises side (AddressSpaces * and send using Smart Host tenant.mail.protection.outlook.com). Set up your email server to relay mail to the Internet via Microsoft 365 or Office 365.

Microsoft emphasizes that there must be no component between the On-Premises and Online Exchange servers that processes/modifies the SMTP traffic. Apparently, not even an AntiSpam gateway (I made a small test and the messages passed correctly even through the gateway). Microsoft adds certain of its own headers to these messages that identify the message as internal. If this information is removed, the message is considered external/anonymous and is handled differently.

We can solve this by having two public addresses for receiving mail. The standard mail reception (SMTP) uses the address specified in the domain's MX record. This is routed to the mail gateway. In the Outbound connector in our Exchange Online organization, we set another address as the Smart Host. This routes directly (through the firewall) to the Exchange server (it needs to use a trusted certificate). We can restrict reception through this address to IP addresses from EXO. The list of these can be found in the Exchange Online IP addresses and URLs.

Accepted Domains - Authoritative vs. InternalRelay

Manage accepted domains in Exchange Online

Another important thing concerns the information about the recipients. By default, for message delivery to work between the On-Premises and Exchange Online organizations (which in many cases also means delivering messages from the internet), both organizations must have information about all recipients and know whether the mailbox is located locally or externally (type Mail User and User Mailbox, external mailboxes are Contact or RemoteMailbox). If a message arrives for a recipient within an authoritative accepted domain, and the receiving Exchange organization does not have the user listed with an email (Recipient), the message will be rejected.

We can also set a different behavior, where the message for an unknown recipient is forwarded to the other organization. This is determined by setting the accepted domain type to InternalRelay. For example, in a situation where we still want to primarily use the On-Premises Exchange and only use certain services in the cloud. Then we may not want to synchronize all mail groups and accounts to the cloud. To ensure that mail from the cloud also reaches addresses that are not synchronized here, we need to change the domain type.

Adding domains is done in the Microsoft 365 admin center. The domain is automatically set as an Authoritative Accepted Domain for Exchange Online (so it can send and receive mail). The type can be changed in the Exchange admin center (EAC) - Mail flow - Accepted domains. Two types can be set:

Authoritative - only receives messages for valid recipients in this organization, other recipients are rejected, all recipients must exist in the online organization (even if their mailbox is On-Premises), Directory Based Edge Blocking (DBEB) is performed, where non-existent recipients are rejected already at the perimeter
InternalRelay - the message is delivered for known recipients, for unknown recipients it is forwarded to the On-Premises server, the recipient can therefore only exist on the internal servers and does not need to be in the cloud, we must have defined connectors (these are created by the Hybrid Configuration Wizard), MS does not recommend this option if we have all recipients in the cloud

Problem with Delivering Messages to Other Organizations in EXO

Right after configuring the Exchange Hybrid, I ran into problems with delivering mail to certain domains. Apparently, many companies do not encounter this problem and do not care how the email messages are routed. But I find it wrong that Microsoft does not write about this behavior at all in the official documentation. I only later found a few articles that describe this behavior.

I was planning to keep all mailboxes on the On-Premises Exchange servers for now. So based on the information, I expected that all mail would be sent from the internal environment (just like it would come in according to the MX record). I expected the connectors created by the wizard to be used only for transferring messages within our domain. If some mailboxes are internal and some in the cloud. I found that this is not the case. Let's describe this further.

Shortly after completing the Exchange Hybrid configuration, several of our partner companies with whom we exchange a lot of messages reported that they stopped receiving emails from us. I quickly found that all of these companies have their MX record pointing to Exchange Online. That is, mail hosted at Microsoft. In the outgoing mail log, the last event was that the messages were correctly received by the Microsoft server.

I only learned more after one company did an analysis on their side (in EXO) and sent me a lot of information. The first information would have been still misleading. All our messages (from the HCW configuration) ended up in quarantine, had a Spam Confidence Level of 9. This would suggest that Microsoft suddenly started blocking us for SPAM. Fortunately, they also sent me the header of one such message. Here, the real reason was visible, the SPF failed. It stated that the message was not received from our server (IP addresses), but from the address 40.107.21.80. This is a Microsoft address that belongs to outbound.protection.outlook.com, the sending address of Exchange Online.

Sender Policy Framework (SPF) and Exchange Hybrid

The solution is simple, add to the SPF record (DNS record type TXT)

include:spf.protection.outlook.com

The entire record may look like this.

"v=spf1 ip4:10.20.30.40 include:spf.protection.outlook.com -all"

This leads to the lesson that if we use SPF (description Email verification using SPF - Sender Policy Framework or at MS How Microsoft 365 uses Sender Policy Framework (SPF) to prevent spoofing), we always have to allow sending from Microsoft servers for Exchange Hybrid. Even if we don't plan to have any mailboxes in the cloud yet.

Mail Flow through the Corporate Tenant in Exchange Online

When I looked at Message trace (New Exchange admin center) in Exchange Online, I found all those undelivered messages (and many more that I didn't know about). The simplified report showed that the message was received from our IP address and sent to some Microsoft address inbound.protection.outlook.com (e.g. 104.47.9.36).

At first glance, it looks like all messages for other organizations in Exchange Online are captured by our Inbound Connector. The message is therefore received by our Tenant, it knows that it belongs to another recipient, so it sends it using the MX record. It is sent as if it was sent from Exchange Online. On the second reception, it no longer falls into our connector (the certificate or IP does not match), but is delivered to the target Tenant.

Exchange Hybrid schéma posílání mailu v rámci EXO

I have conducted a number of experiments. Since the most information about how the message traveled is visible in the header of the delivered message, I have written to various hosted domains and received the delivered emails back. The Message Header Analyzer tool can help with header analysis. It can be seen that the message passes through mail.protection.outlook.com to some prod.outlook.com and is then sent again from outbound.protection.outlook.com. mail.protection.outlook.com receives it again until it reaches the target prod.outlook.com.

I was quite confused when I found that some messages did not go through our Exchange Online, but were delivered directly to the target Tenant. But it makes sense, our connector is located on certain MS servers, if the target tenant is on the same servers (IP addresses), the connector is used. I couldn't figure out whether it's about geolocation (MS in the admin center only mentions European Union) or data center or forest or what exactly, so that organizations have the same addresses.

To identify whether two Tenants (domains) are on the same servers, we can do it simply. We look at the MX record and what IP addresses the given DNS name returns. Our organization, and most others, have (in practice, the word Tenant is replaced by the name of the given organization)

tenant1.mail.protection.outlook.com - IP addresses 104.47.8.36, 104.47.9.36, 104.47.10.36

In a few cases, the addresses were different and the messages were delivered directly.

tenant2.mail.protection.outlook.com - IP addresses 104.47.0.36, 104.47.1.36, 104.47.2.36

Connectors between On-Premises and Online

Everything is caused by the Inbound Connector on Exchange Online. The Hybrid Configuration Wizard standardly creates and sets up several connectors. We can look at their details (here with Centralized Mail Transport disabled, which is the default).

On On-Premises Exchange Servers

Default Frontend Receive Connector - modifies the Default Frontend SERVER connector, mainly sets the certificate (TLSCertificateName) and removes the ExchangeUsers group
Send Connector - creates the Outbound to Office 365 - TenantOrganizationGUID connector, sets it for addresses from the domain (tenant = our tenant) tenant.mail.onmicrosoft.com (AddressSpaces), delivery according to the MX record, enables TLS (RequireTLS, TLSCertificateName) and domain verification mail.protection.outlook.com (TLSAuthLevel, TLSDomain), enables CloudServicesMailEnabled (determines the use of internal headers for hybrid mail flow)

In Exchange Online

Standard Inbound Connector - creates the Inbound from OrganizationGUID connector, receives all domains (SenderDomains smtp:*;1), connector type (ConnectorType) OnPremises (handles domains used in the internal organization), requires TLS (RequireTLS) and our certificate (TLSSenderCertificateName, RestrictDomainsToCertificate serves to identify the sending server, the other option is IP address), enables CloudServicesMailEnabled
Standard Outbound Connector - creates the Outbound to OrganizationGUID connector, for addresses from our domain firma.cz (RecipientDomains), routes to the specified internal server address (SmartHosts), connector type OnPremises, TLS verification of our certificate (TLSSettings, TLSDomain), enables CloudServicesMailEnabled, does not enable (according to the settings in the wizard) Centralized Mail Transport (RouteAllMessagesViaOnPremises)

Application of Connectors

From this, it follows that everything could work even without connectors on the internal Exchange servers. Mainly, TLS and identification using a certificate are addressed so that the correct two parties can connect. For sending, internal items are added to the message header. Microsoft states that if we didn't use connectors and exchanged a large number of messages between our mail servers and the Microsoft 365 or Office 365 organization, Graylisting (traffic restriction as protection against SPAM) would be applied.

Sending from our cloud Tenant is set for recipient addresses from our domain and is sent via the specified server (it doesn't have to be the same as in the MX record, because it should route directly to the Exchange server, and not through an AntiSpam gateway, for example).

Receiving into our cloud Tenant is identified using a TLS connection with a certificate in our name, receiving messages from any sending domain. The recipient's address is not handled. We can modify various settings, but we cannot determine that the connector should only work for recipients in our domain. I also tried setting a specific domain in SenderDomains, and emails from other domains still passed through the connector.

Documentation on Mail Flow and What We Find Here

The description of Exchange Server hybrid deployments in this area provides only basic and very limited information (Transport options and routing). We can find a little more information in the Exchange Online documentation (Mail flow best practices). For example, there are articles such as

Various Mail Flow Scenarios

Here, various scenarios and required configurations (of connectors) are described. But it only talks about our mailboxes in the cloud and on internal servers and sending and receiving mail from the internet or partner organizations. It describes variants as to whether all incoming mail goes to the cloud (what changes we need to make) or to the On-Premises servers, where the check is performed and where it is sent from. It does not mention the situation where the recipient is another organization on the same EXO servers, and that the behavior is then different than one would expect. It also seems a pity to me that for different scenarios, the individual configuration steps are described, but a detailed description of the connector configuration is missing (it is always stated to go through the wizard).

Connector to the Cloud and Delivery

Microsoft states (How do connectors route mail between Microsoft 365 or Office 365 and my own email server?) that the connector from our corporate servers to EXO receives all messages from our mail servers and sends them on our behalf (in our name) to the recipients. The recipient can be a mailbox in our EXO organization or an internet recipient. We need to create a connector on the internal servers that sends messages directly to EXO. If it sends all messages to EXO, all mail will be sent from the cloud (and we can use its security features).

Options for Sending from the Cloud via Connector

When creating an outbound connector, we have several configuration options. We can create it for all our accepted domains, for listed domains, or use a Transport Rule (Scenario: Conditional mail routing in Exchange Online). We can also set up so that all outgoing messages are sent to a special service (server) where some modifications/checks are made and then sent back to EXO from where they will be sent to the recipients as usual (Scenario: Integrate Microsoft 365 or Office 365 with an email add-on service). Another special thing, only for specific scenarios, is Enhanced Filtering for Connectors.

Where the Mailboxes Are

In many scenarios, Microsoft assumes that all mailboxes are in the cloud (this is what they try to recommend, because it is advantageous for them). The situation where some mailboxes are local and some in the cloud is described in Manage mail flow with mailboxes in multiple locations (Exchange Online and On-Premises). How to send emails from devices or applications, generally how to use SMTP Relay, is described in How to set up a multifunction device or application to send email using Microsoft 365 or Office 365.

When Our Tenant Processes the Outgoing Message

After the examination that showed me the behavior described in the previous chapters, and studying the official documentation, I managed to formulate a query in Google. It returned the following articles that describe exactly this behavior. When a message for another Microsoft 365 Tenant passes through our Tenant.

Assigning a Message to a Tenant and Incoming vs. Originating Type

They explain that Microsoft 365 is a very complex multi-tenant environment, where the entire infrastructure (IP addresses, certificates, transport servers) can be shared with other Tenants. Therefore, it is very important to assign messages to the organization (Tenant). We may want to send all messages through EXO so that the required filtering, checks, etc. are performed. So when we send a message from our On-Premises server (and domain) to another organization (domain) in the cloud, we must distinguish whether it comes (Incoming) to the target domain or originates (Originating) from our domain.

If the incoming message matches the inbound connector of a Tenant and is of type OnPremises (i.e., it comes from the On-Premises server of that Tenant) or comes from a mailbox in Exchange Online, it is considered Originating. In all other cases, and if the recipient's domain is an accepted domain of an Office 365 organization, it is considered Incoming.

Internal vs. Anonymous Messages

Another article describes how important it is to distinguish messages as internal - Internal (authenticated in some way, like sent by a logged-in user) and anonymous - Anonymous (received from an anonymous source).

One of these variants is found in the email header:

X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthAs: Anonymous

Internal messages are processed differently, bypass anti-spam checks, can be delivered to distribution groups that require authenticated senders, and are processed by the Resource Booking Attendant. The exact decision-making on whether a message will be considered internal is quite complex, depending on certain headers and connectors. Therefore, it is necessary to have everything set up correctly and not modify the headers during transfer between internal servers and the cloud.

Using Multiple Domains On-Premises

I have been dealing a lot with the fact that messages from internal servers pass through our Tenant before being delivered to recipients in another organization on the same EXO servers. The primary domain is not important to me, but if we have additional domains on the On-Premises Exchange servers for which we send and receive mail.

If we use multiple public domains for email services and want to use them also in the cloud on Exchange Online, it should be simple, and most of the official documentation scenarios mention this. We need to add all domains as Accepted Domains, either Authoritative or InternalRelay. And the Hybrid Configuration Wizard should set up the connectors and everything else correctly.

A different situation is if we want to use the additional domains only on the internal servers and not include them in the cloud. If we send messages from the same internal servers (more precisely, with the same/similar certificate) and the message arrives at the Exchange Online servers, where our Tenant is also located, our Inbound Connector will catch it. The sender's domain doesn't matter. Tests show that the Relay works, and the messages are forwarded from our Tenant to the recipient. And this even without the domain being set as an Accepted Domain.

Details on the On-Premises Inbound Connector

The crucial thing is how the Inbound Connector that receives messages from internal servers behaves exactly, and how it can be configured if necessary. From the various documentation (mentioned above), I have put together the following. The description often seems confusing to me.

How Our Servers (Connector) Are Identified

How does Microsoft 365 identify our mail server and assign messages to the incoming connector? That is, how we can configure the Inbound Connector. It is described in various places Identifying email from your email server, Configure a certificate-based connector to relay email messages through Office 365, Configure your Microsoft 365 or Office 365 environment, but it doesn't seem entirely clear to me (the information seems to not fully correspond).

certificate used for TLS on the mail server - (recommended) it is a trusted certificate that the sending mail server will use to authenticate with Exchange Online, it is recommended that the Subject Common Name or Subject Alternative Name of the certificate match the organization's primary SMTP domain
outgoing IP address of the mail server - from which address, addresses or range of addresses our mail is sent to the cloud, EXO identifies these as our organization's mail servers based on the addresses, they must be our own IP addresses, you cannot use a shared service

For Relay to Work

It is further stated that if we want Exchange Online or Exchange Online Protection (EOP) to forward (relay) emails from our organization to the internet, one of the following must be met:

a certificate is used where the Subject Name matches the accepted domain in our Tenant
all sending domains and subdomains are configured as accepted domains in our Tenant

TlsSenderCertificateName Parameter and Certificate Subject

In several places, it is stated that when configuring the Subject Name of the certificate for the Inbound Connector, a * (wildcard) can be used, e.g. *.firma.cz. This way, we can use the server's certificate where its domain name, e.g. mail.firma.cz, and we don't have to register this name as an accepted domain (where we have registered firma.cz). The wildcard notation then matches both the name in the certificate and the registered domain.

Forwarding Messages for Other Domains

From this, I conclude that if I use a certificate for sending mail to EXO where the name matches the accepted domain, and set this name (domain) on the incoming connector, then I can send mail for various domains through this connector, without having to set them up in the cloud. These messages can also be forwarded (relayed) to external recipients.

One article mentions that a working Spoofing (the ability to send from other addresses/domains) is normal for SMTP. But Exchange Online is not an Open Relay, it only allows sending messages to its identified customers. Legitimate reasons are also given for when a server needs to send a message on behalf of another domain (e.g., forwarding). Therefore, it is recommended to use a certificate-based connector.

If I create a connector in the EAC, quite a bit of this information is displayed as a hint. For example:

Office 365 will only accept messages through this connector if the sender's domain or TLS certificate domain is configured
 as an accepted domain for your Office 365 organization.

Exchange Online New Inbound Connector z On-Premises

Do I Have to Set Up All Domains as Accepted in the Cloud?

Based on some articles, I would understand that all sender domains must be set up as accepted (Configuring the sender domain as an accepted domain). But maybe these are old articles and it changed on 5. 6. 2017, as described in Configure a certificate-based connector to relay email messages through Office 365.

But there is also a Non-accepted domain report in the Office 365 Security & Compliance - Mail flow Dashboard, which shows information about messages from the On-Premises organization where the sender's domain is not set as an accepted domain in Microsoft 365 Non-accepted domain report in the Security & Compliance Center. The article states that such messages may be restricted.

As a conclusion, it seems to me that I don't have to set up the additional domains that I use in the internal organization in the cloud. And in those special cases where they pass through our Tenant, they will be correctly forwarded. But if there were a large number of such messages, I should set them up.

How It Is Determined Which Tenant a Message Belongs To

The description How Office 365 determines to which tenant a specific message belongs is interesting. Let me try to rewrite this process (and simplify it a bit).

The certificate used by the sending server is taken
The name in the certificate (or SAN) is used to search for an Inbound Connector of type OnPremises, based on TlsSenderCertificateName, among all Tenants in the given EOP forest
From the found connectors, the one Tenant that has the TlsSenderCertificateName from the connector set as an accepted domain is searched for (the most specific result wins) - if found, the message is assigned as Originating from this Tenant
If not found, the Connecting IP (the address of the sending server) is used to search among all Tenants in the given EOP forest for an Inbound OnPremises connector that matches the IP address
From the found connectors, the Tenant that has the MailFrom domain set as an accepted domain is searched for - if found, the message is assigned as Originating from this Tenant
If not found, the domain in RcptTo is used to search for which Tenant has it as an accepted domain - if found, the message is assigned as Incoming to this Tenant and it is further checked whether the message matches any Inbound Partner connector in this Tenant, if so, it is assigned to it, otherwise it comes through the Default Inbound Connector
If not found, the message is rejected

Several interesting conclusions can be drawn from this process, which were mentioned more or less in the documentation.

To have our EXO Tenant forward a message from a sender domain that we don't have set as accepted, we need to use a certificate. If we have an incoming connector based on IP address, the sender's domain must be set as accepted.
If we use a certificate, the incoming connector is searched for only based on the name in the certificate, the sending domain is not checked (unless this step was just omitted from the process description).

How to Ensure Our Outgoing Messages from On-Premises Do Not Pass Through Our EXO Tenant

I found only one description FAQ #6(b) - I don't want outbound email originating from my on-premises gateway going through my Office 365 tenant. How do I fix this? I would say there are two options.

different certificates - on the Send Connector we have for sending messages to Exchange Online, we use a different certificate than the one we have on the standard outgoing connector (to the internet), on the Inbound Connector we then set the name of this certificate (probably without the wildcard, if both certificates were on the same domain, and we have to set the name as an accepted domain)
different IP addresses - ensure that the mail leaving our Send Connector to Exchange Online goes out from a different public IP address than the rest of the mail to the internet, set the Inbound Connector according to this IP address (even though it is recommended everywhere to use a certificate)

Limiting the Inbound Connector

Neither of these is entirely easy to implement. I would like to solve the problem so that no emails from other sending domains end up in our cloud Tenant. The best thing to me would be to limit the Sender Domains on the Inbound Connector. When I look at the cmdlet Set-InboundConnector, there are parameters SenderDomains and AssociatedAcceptedDomains. But I've experimented with both parameters, and it seems they have no effect.

OnPremises vs. Partner Connector Type

When I create a connector in the cloud EAC that receives messages from our organization into our Office 365, I choose how the messages are identified, either by the name in the certificate or by the IP address of the sending server. I can't set anything else further.

When I create a connector from a partner organization into our Office 365, I choose whether the messages are identified by the sender's domain or the IP address of the sending server. I can then set restrictions on the connector. Requiring TLS and a certain name in the certificate. Receiving only from selected sender server IP addresses. Set up connectors for secure mail flow with a partner organization

When I create both types of connectors with similar settings and then view them using PowerShell, their parameters are almost identical. Of course, the ConnectorType value OnPremises vs. Partner differs, the first has CloudServicesMailEnabled allowed and by default SenderDomains is * for the first, and specified as *.firma.cz for the second.

Connector Parameters

I couldn't find any documentation on configuring connector parameters. In New-InboundConnector, there is nothing about some parameters only being usable for a certain connector type, but it seems that is the case. I'm quite lost in the description of the parameters RestrictDomainsToCertificate, RestrictDomainsToIPAddresses, SenderDomains, SenderIPAddresses, TlsSenderCertificateName, mainly in their mutual relationship. Maybe the only mention I found in the note Manage mail flow using a third-party cloud service with Exchange Online explains it a bit.

If you already have an OnPremises inbound connector for the same certificate or sender IP addresses, you still need to
 create the Partner inbound connector (the RestrictDomainsToCertificate and RestrictDomainsToIPAddresses parameters are
 only applied to Partner connectors). The two connectors can coexist without problems.

It is clearly stated here that the RestrictDomainsToCertificate and RestrictDomainsToIPAddresses parameters are only used for the partner connector. So the same may apply to SenderDomains and AssociatedAcceptedDomains.