FortiGate two Internet connections (Dual WAN)

Note: The description in the article is based on FortiGate FG-300E with FortiOS version 6.2.3. Which is configured as an FGCP cluster and uses VDOM.

Note: The description is relatively brief. I only tested some things briefly and didn't study all the possibilities. The documentation is again fragmented and contradicts itself in some areas. This may be due to significant changes between versions. Therefore, I don't guarantee the accuracy of the information provided in the article.

In this article, we'll look at various ways to use two internet connections on one FortiGate (for all methods, there can be more). Each internet connection is brought to one interface of the FortiGate unit. It has a public IP address assigned from a defined range. It may have different speed and other parameters. Internet is available behind each line.

Basic Setup

We'll perform the basic setup the same way as if we had one internet line. Identically for both connections. For various configurations, we'll later adjust the static routing settings.

• (Global/VDOM) > Network > Interfaces

We'll set up the interfaces where individual internet lines are connected, including the public IP address.

• (VDOM) > Network > Static Routes

For communication to the internet, we typically set up a static Default Route (for destination 0.0.0.0/0), where we enter the gateway address (ISP router) from the range where we have the public IP address (i.e., Next Hop). For static routing, the default value for Administrative Distance is 10 and priority is 0.

When we add a second internet line, we'll set it up identically at this point (of course, we choose a different Interface and gateway address).

Communication from the Internet

The situation is simpler when we publish some services on a public IP address to the internet. We usually have public IP addresses assigned by the ISP. So we have different addresses on each line. We don't solve high availability; if one line is unavailable, its IP addresses won't work either. But we can publish services on FortiGate using IP addresses from both ISPs and thus use both lines.

We can create, for example, a standard Virtual IP (Destination NAT) and use the public IP address of any connection. If we've done the configuration from the previous chapter, everything will work. FortiGate will receive the communication, process it, and respond from the same IP address and through the interface where the request came from. Apparently, at this point, it searches the routing table taking into account the interface.

Communication to the Internet

Documentation Dual Internet connections, Routing concepts

For user communication to the internet via two WAN lines, we have several configuration options. We can address load balancing as well as high availability (High Availability or Fault Tolerance using Redundancy and Failover).

• static routing with primary and backup path
• Equal Cost Multi-Path (ECMP) Load Balancing and Failover
• Policy Routing
• Software-Defined Wide Area Networks (SD-WAN)

In the basic setup, we created multiple static routing records (routes) for multiple internet connections. So we have multiple paths for the same destination that use a different interface and a different gateway. FortiGate must decide which Next Hop to use when routing.

When creating the routing table, and there are multiple records for the same destination, it compares the Administrative Distance (AD) value and selects the value with the lowest distance to insert into the table. If they have the same AD, both records are inserted into the table. When deciding on routing, FortiGate then looks at the priority value (a proprietary data used by Fortinet) and uses the path with the lower value. Only records whose interface is up are installed in the routing table.

If there are multiple paths for the same destination and they have the same distance and priority, we're talking about Equal Cost Multi Path (ECMP) routes. In that case, FortiGate automatically uses ECMP Load Balancing.

If we want to use two paths to the internet, we must also have two Firewall Policies (or use Multiple Interface Policies and assign both interfaces), because that's where we define the outgoing interface. In any situation where load balancing on two lines or switching in case of failure is used, a policy must be found that allows communication.

Primary and Backup Path

Documentation Technical Tip: Redundant Internet connection without load-balance

• (VDOM) > Network > Static Routes

If we have two internet connections, we can actively use one and have the other as a backup. We can achieve this simply by configuring default routes with different Administrative Distance or priority (which Fortinet recommends). Then the path with the lower value is preferred, and only if it's unavailable, the second one is used.

For FortiGate to recognize that a path is unavailable, we can use the Link Health Monitor.

ECMP Load Balancing and Failover

Documentation Technical Tip: ECMP - Load balancing algorithms for IPv4 and IPv6, Technical Tip: Configure Equal Cost MultiPath (ECMP), Advanced static routing example: ECMP failover and load balancing, Technical Note: Configuring link redundancy - Traffic load-balancing / load-sharing - ECMP (Equal Cost Multiple Path) - Dual Internet or WAN scenario

Equal Cost Multi-Path (ECMP) Load Balancing and Failover are methods that extend static routing. They are used to distribute traffic to the same destination across multiple paths. On FortiGate, load balancing happens automatically, and the default behavior is load balancing based on the source IP address, from which a hash is calculated and a path is assigned.

This ensures routing of traffic to the internet via different paths for different clients. The same source IP address always uses the same path. If we also want Failover to work, and when one path is unavailable, it stops being used (removed from the routing table). So we must use the Link Health Monitor.

We can change the load balancing method using CLI.

config system settings 
    set v4-ecmp-mode source-ip-based
end

Link Health Monitor (Interface Status Detection)

Documentation CLI Reference - Configure Link Health Monitor, Link health monitor, Technical Tip: Bring other interfaces down when link monitor fails

If any interface through which a static route is created is down, this record is removed from the routing table. If any of the internet connections is unavailable, we want this path to be removed from the routing table. Another available path will then be used automatically.

In practice, the problem is usually not in the connection on the FortiGate interface, but further along the way, where the gateway at the ISP is not unavailable. Link Health Monitor can check the availability of the gateway (or some server on the internet) using the ping function at regular intervals. If the target is not available (defined number of tests), it considers the line unavailable and removes the path from the routing table. If the path becomes available again, it returns the record. In case of unavailability, it can also turn off another interface.

It is configured using CLI

config system link-monitor
    edit "test"     
        set srcintf "WAN1"
        set server "8.8.8.8"
        set update-cascade-interface disable
        set update-static-route enable
end

Policy Routing

Documentation Policy routing, Technical Note : Configuration example of Policy Based Routing and VIP for SMTP services in Dual Wan scenario, YouTube - How to Configure Policy Base Routing on Fortigate, Technical Tip: How to create 'Stop Policy Route'

• (VDOM) > Network > Policy Routes

If we have two internet lines and want to determine which traffic should use which line, Policy Routing can help us. Classic routing looks for a path only based on the destination address. Using Policy Routing, we can decide based on the source interface, source address, destination address, protocol, or service. And to that, determine the gateway address (Next Hop) and possibly the interface. We can thus create rules for one internal network to use one internet line and the rest to use the other. Or for a certain type of traffic (communication to Office 365 or SMTP) to use a dedicated line.

The documentation states that for common settings, there must be a record in the routing table (FIB) corresponding to the defined Policy Route. During routing, Policy Route is traversed first (list of policies from top to bottom), if none matches, static routes are traversed. This means it applies before all Static Routes. If we create a Policy Route for all destination addresses (Default Route), it will be used and more specific static routes or routes from directly connected interfaces will be ignored.

If we want to deliberately distribute traffic (without automatic load balancing) to different internet lines, a possible solution is as follows. For the line we consider primary (default), we create a Default Route with standard distance (AD) 10 and priority 0. For the second line, we create a Default Route also with distance 10, but higher priority 10. For traffic that should use the second line, we define a Policy Route identifying this traffic (perhaps by incoming interface) and specify the outgoing interface and gateway address of the second connection.

If the interface for which the Policy Route is defined is unavailable, this policy will not be used. Because there is no record in the FIB and the Policy Route is skipped. We can thus also use the Link Health Monitor. This way, it will normally fail over to the second line.

In Policy Route, we can also use the Stop Policy Routing action. This causes the Policy Route traversal to end and transition to the standard routing table (more precisely to the Forwarding Information Base).

Software-Defined Wide Area Networks (SD-WAN)

Documentation SD-WAN (6.2.3), SD-WAN (6.0.0), CLI Reference - Configure redundant internet connections using SD-WAN, Technical Tip: Multiple default routes where sdwan rules are not preferred

A complex solution for multiple internet connections (Multi-path WAN) is called SD-WAN. It can be said that it replaces everything we've described above.

SD-WAN is a virtual interface that consists of a group of member interfaces (minimum one, maximum 255), which can be connected to different types of lines. Configuration is simplified because we set up one group of routes and FW policies. Where we use the SD-WAN interface, it automatically applies to all interfaces that are members of SD-WAN. We can use various load balancing algorithms for routing traffic to individual lines. For example, according to bandwidth usage or number of sessions.

Within a VDOM, we can create only one SD-WAN interface. IPv6 is configured using CLI. We can create a maximum of 4000 SD-WAN rules and link health monitors. Interfaces we want to include in SD-WAN must not be used in most configurations (otherwise they can't be included). Such as membership in a zone, firewall policy, default route (but can be used in Virtual IP or Address).

If we enable SD-WAN, a virtual interface is created with the fixed name SD-WAN, to which it's not possible to add an Alias. We can use this interface only in some places, primarily in Static Route and Firewall/Security Policy. It can't be included in a Zone, nor used in Address or Virtual IP as an interface.

Creating SD-WAN Interface

• (VDOM) > Network > SD-WAN

We enable SD-WAN, which creates a virtual interface that we see under Network > Interfaces. As Interface Members, we include interfaces where we have internet lines. We set the gateway address (which we would otherwise enter in the Default Route) and we can set the cost, which only makes sense if we'll be using the Lowest Cost (SLA) load balancing strategy.

Adding Static Route

• (VDOM) > Network > Static Routes

We create a Default Route (i.e., destination 0.0.0.0/0.0.0.0), we choose SD-WAN as the interface (gateway address is not entered). Whatever distance (AD) we enter, it changes to 1.

Setting Default Load Balancing Algorithm

• (VDOM) > Network > SD-WAN Rules

Among the SD-WAN rules, there is an implicit (last) rule sd-wan, which is used if no other routing rule matches. In it, we set the algorithm that will be used for load balancing traffic to individual SD-WAN members. The default is Source IP, which is the same as for ECMP Load Balancing.

Setting Up Link Monitoring

• (VDOM) > Network > Performance SLA

Link Health Monitoring is set up using Performance SLA, which has many options. The basis is monitoring line quality (according to latency, jitter and packet loss) and detecting line failures for SD-WAN member interfaces. We can monitor one or two targets/servers (one is monitored, if it's not available, the second is tested, if both are not, the check fails). In the GUI, we can set up the use of PING and HTTP GET. Using CLI, we can also use TCP-Echo, UDP-Echo and Two-Way Active Measurement Protocol (TWAMP). We set the interval of checks, when it's considered inactive and after how many successful attempts it's considered active again.

The basic feature is modifying the routing table (Update static route), where unavailable lines are removed, so they are not used.

Creating SD-WAN Rules

• (VDOM) > Network > SD-WAN Rules

We can create rules, which is similar to Policy Route. In these rules, we can identify traffic and specify the outgoing interface for it. Rules are traversed from top to bottom and the first matching one is used (First Fit). If no rule matches, the implicit sd-wan is used. If we specify multiple outgoing interfaces in a rule, the chosen interface selection strategy is used. We can choose manual assignment of one interface.

Similar to Policy Route, there must be a record in the Forwarding Information Base (FIB) corresponding to the SD-WAN Rule. If it doesn't exist, the rule is skipped. This behavior can be changed for a specific rule with the commands

config system virtual-wan-link
    config service
        edit 2
            set gateway enable
            set default enable
        next
    end
end

If we don't want to load balance traffic across multiple lines, we can create a rule for all destination addresses (thus covering all traffic) and manually assign an interface. So it never gets to the implicit rule.

If we want to create a rule based on incoming interface, it's not possible in the GUI, but we must use CLI.

config system virtual-wan-link
    config service
        edit 2
            set input-device "DMZ"
        next
    end
end

Creating Security Policies

• (VDOM) > Policy & Objects > IPv4 Policy

For any traffic to go out to the internet, we must create a corresponding Firewall / Security Policy. In these, we select sd-wan as the Outgoing Interface, which covers all member interfaces.

Monitoring SD-WAN

• (VDOM) > Network > SD-WAN
• (VDOM) > Monitor > SD-WAN Monitor
• (VDOM) > Network > Performance SLA
• (VDOM) > FortiView > All Sessions

In two places in the GUI, we can monitor the usage of individual lines of SD-WAN. In Performance SLA, we see current data (and whether they meet SLA targets) and short history in graphs. In the session list, we can display Destination Interface and check which interface which traffic is sent to.

Note: Perhaps only in version FortiOS 6.2.3 there is a bug (something like this is described in the Release Notes and should be fixed in 6.2.4). If we run an FGCP cluster with VDOM, we only see the values on the FortiGate unit where the given VDOM is Master. In some cases, we must manually connect to the other unit, otherwise the link is displayed as Error.

• (VDOM) > Monitor > Routing Monitor

Here we see the routing table and on the right we can switch from Static & Dynamic to Policy, where Policy Route and SD-WAN Rules are displayed.

We can use the Route Lookup button (in case of an FGCP cluster with VDOM, we must be connected to the FortiGate unit where the given VDOM is Master). Here we enter data (at least the destination IP address) and the record that will be used for routing is highlighted.

Commands for Monitoring and Diagnostics

Note: In case of an FGCP cluster with VDOM, we must use most commands on the FortiGate unit where the given VDOM is Master.

Display all active records in the routing table

get router info routing-table all

Display of all active and inactive records in the routing table

get router info routing-table database

Listing of Policy Route (these are also SD-WAN Rules)

diagnose firewall proute list

Displays Forwarding Information Base (FIB)

get router info kernel

Listing of Route Cache

diagnose ip rtcache list

Listing of SD-WAN members

diagnose sys virtual-wan-link member

Listing of all IP addresses assigned to FortiGate interfaces

diagnose ip address list

Routing Lookup - finding a routing record

Generally, routing is done according to the routing table (Routing Table), where a record for the destination IP address of the packet is sought. The table has records sorted by mask length from longest (smallest subnet or single address /32) to shortest (which may contain all addresses /0). It is traversed sequentially until a matching record is found or the destination network is unreachable. If we have defined a Default Route (equivalent to Default Gateway for a server/station), a record is always found (Default Route contains all addresses and is at the end of the list).

FortiGate uses a Route Cache to increase performance, where it first looks for a record. It creates a Forwarding Information Base (FIB) from the Routing Table, which it refers to as the Kernel Routing Table. It searches for routing records in the FIB, which is optimized for faster destination address lookup (FIB Lookup).

When we start using Policy Routing or SD-WAN or both, the situation becomes more complicated. I haven't found documentation that clearly describes everything. So I don't guarantee the accuracy of the following. The diagram in the article Technical Tip: Multiple default routes where sdwan rules are not preferred provides the most hints.

• first, Policy Routes are traversed, if a policy is found
  • the action is Forward Traffic, then a record for the interface is searched in the FIB, if not found, the policy is skipped, otherwise the traffic is routed according to the data (Forward Traffic)
  • the action is Stop Policy Routing, then it goes to FIB (SD-WAN is skipped)
• SD-WAN Rules are traversed, if a rule is found
  • the rule setting is set default enable, set gateway enable, then traffic is routed according to the data
  • the rule setting is set default disable, set gateway disable, then FIB Lookup is performed according to the destination IP address
    • if a record is found and the interface is not a member of SD-WAN, then traffic is routed according to the record
    • if a record is found and the interface is a member of SD-WAN, then traffic is routed according to the SD-WAN rule
    • if no record is found, it is skipped
• Forwarding Information Base (FIB) is traversed according to the destination IP address, first it looks in the Route Cache and then FIB
  • if a record is found, traffic is routed according to the data
• end of Route Lookup - no path exists