FortiGate IPsec VPN, debug and issues

Note: The description in the article is based on FortiGate FG-300E with FortiOS version 6.2.7. It is configured as a FGCP cluster and uses VDOM Partitioning (Virtual clustering). And it was the VDOM Partitioning that eventually turned out to be the reason for the problem with IPsec Rekey.

Note: I received several additions and information from a certified Fortinet expert, so I have incorporated them into the article.

Basic Concepts and Principles of IPsec VPN

Documentation

Precise information can be found in the RFCs for the individual areas, of which there are many. Some descriptions on the internet:

IPsec VPN

VPN - Virtual Private Network - main types:

Site to Site VPN - two gateways (Peers) establish a static tunnel
Remote Access VPN - a client establishes a tunnel on-demand to the gateway

IPsec - Internet Protocol Security

a standardized suite of protocols for securing communication between two end systems in IP networks
includes bi-directional authentication and negotiation of cryptographic methods and keys (authenticates and encrypts packets)
we can use a variety of standard protocols and algorithms

Main protocols used in IPsec

IKE - Internet Key Exchange - the protocol used to set up SA when negotiating IPsec (here we're mainly talking about IKEv2), uses ISAKMP (Internet Security Association and Key Management Protocol) and the Oakley protocol, communicates using UDP port 500
ESP - Encapsulated Security Payload - the encrypted communication itself, uses IP protocol number 50

Simplified, IKE negotiates the SA and from its parameters establishes the encrypted tunnel using ESP.

IPsec Encapsulation Modes

Tunnel mode - encrypts the entire packet (including the header) and adds a new header, standard for Site to Site VPN
Transport mode - encrypts only the data, the IP header is left as is and only an IPsec header is added, standard for Remote Access VPN

Security Attributes and Algorithms

SA - Security Association

shared security attributes, such as encryption algorithms, keys and parameters (Packet Sequence Number)
created by negotiation using IKE (each phase has its own, so IKE SA and IPsec SA are used)
each SA has a Security Parameter Index (SPI) identifier, an index into the Security Association DataBase (SADB), which is passed to the other party
SA is used for one-way communication, for bi-directional communication we need two

Authentication Methods

PSK - PreShared Key - Shared Secret - text string
RSA certificate - public and private key
EAP- Extensible Authentication Protocol

Algorithm Types and Examples

Encryption - DES, 3DES, AES128, AES256, AES256GCM, CHACHA20/POLY1205
Key exchange - Diffie-Hellman method (creates shared session secret from which encryption keys are derived), various DH groups are used
Authentication / Integrity - uses hash functions MD5, SHA1, SHA256, SHA512 (sometimes the terms authentication and integrity are used interchangeably, HMAC is also mentioned, e.g. HMAC-SHA256)
Pseudo-Random Function (PRF) - similar to the integrity algorithm, not used for message verification, but to ensure randomness, used in Phase 1, utilizes hash functions SHA1, SHA256, SHA512
Perfect Forward Secrecy (PFS) - optional security enhancement for Phase 2, normally Phase 2 keys are derived from Phase 1 keys, PFS performs a new key exchange using Diffie-Hellman, we choose DH group

If an authenticated encryption algorithm like AES256GCM is used in IKE, then an authentication algorithm is not set. For IKEv2 Phase 1, PRF is used instead of an authentication algorithm.

Note: In the debug, I encountered that although PFS was set on both sides for Phase 2, the PFS is disabled was shown in the Proposal. I found information that says the keys for the CHILD_SA, which is created during IKE_AUTH, are always derived from the IKE keys (Phase 1). Only during Rekey (CREATE_CHILD_SA) is PFS used and visible in the debug. Some systems therefore immediately perform a new negotiation of the IPsec SA (CREATE_CHILD_SA) after the initial establishment (IKE_AUTH) to include PFS.

IKE Phase 1 and Phase 2

Proposal

used in IKE Phase 1 and Phase 2, also referred to as IKE Proposal and IPsec or Child SA Proposal
may contain one or more items (supported variants)
includes specification of the protocol, standard for Phase 1 is IKEv1 or IKEv2, for Phase 2 commonly ESP, Encapsulation for Phase 2 Tunnel or Transport
also includes encryption algorithm, integrity / authentication algorithm or PRF (Phase 1), Diffie-Hellman group or PFS (Phase 2), in Phase 2 also whether ESN (Extended Sequence Number) should be used

Key Lifetime

lifetime of the SA keys
IKEv1 negotiates a common Lifetime, IKEv2 each Peer uses its own set Lifetime
Phase 1 Lifetime must be greater than Phase 2 Lifetime
FortiOS allows a range of 120 to 172800 seconds (for Phase 2 the number of bytes of traffic can be set)
the set Lifetime value is called Hard SA Lifetime, there is also a Soft SA Lifetime, which is a slightly lower value and at which a new SA negotiation can occur

Rekey and Reauth

if the SA Lifetime (Hard) expires, the SA is deleted, then it must be re-established for communication to continue
when the Soft SA Lifetime expires, a rekey or reauthentication may occur
Rekey will attempt to create a new set of keys (CREATE_CHILD_SA), which can be exchanged during runtime without interruption, supported in IKEv2
Reauth performs a complete new creation including authentication (IKE_SA_INIT and IKE_AUTH), may interrupt traffic, in IKEv2 it is possible to do without interruption
Phase 2 SA is negotiated only if there is traffic, also Rekey occurs only if there is traffic, otherwise the tunnel goes down, Fortinet has solutions to make both happen without existing traffic, Auto-negotiate and Autokey Keep Alive

The IPsec VPN tunnel is established in two phases:

Phase 1 - IKE Policy
- IKE SA is negotiated
- establishes a secure connection for exchanging authentication information
- authentication is performed
- we enter our and the remote gateway's IP addresses, IKE version, authentication credentials, Phase 1 Proposal, Key Lifetime
- we can use NAT Traversal, Dead Peer Detection (DPD)
Phase 2 - IPsec Policy
- IPsec SA is negotiated (protected by the existing IKE SA)
- VPN tunnel is created
- we define one or more Phase 2 Selectors and their Phase 2 Proposal

Phase 2 Selector

combination of source and destination subnet
we can define one or more address combinations, thereby limiting the communication that can pass through the tunnel
or we can create a Selector for all addresses 0.0.0.0/0
performs SA negotiation (there is a separate SA for each Selector)

Note: IKEv1 uses Main Mode or Aggressive Mode in Phase 1 and Quick Mode in Phase 2. Therefore, the term Quick Mode Selectors is sometimes used.

NAT-T - Network Address Translation - Traversal

IPsec packets are protected against modification (hash)
NAT modifies the IP header (IP addresses), so the hash no longer matches
NAT-T encapsulates in UDP, adds a header that can be modified, which is removed by the recipient
during IKE negotiation, both sides can agree to use NAT-T
communication typically uses UDP port 4500 (referred to as IPsec over UDP or IPsec over NAT-T)

Dead Peer Detection (DPD)

regular checks to see if the IKE Peer on the other side of the tunnel is still alive
if the check fails, the tunnel is terminated and a new negotiation is attempted
uses the IKE Phase 1 notification (R-U-THERE)

Initiator vs. Responder

Initiator - the party that initiates the IPsec tunnel, triggers the IKE negotiation
Responder - the party that accepts the first IKE messages
the tunnel establishment can be automatic (standard when traffic arrives within the VPN) or manual
Responder Only - we can set that we only respond to tunnel establishment from the other side
Initiator Only - in some places it is possible to set that we only establish the tunnel

Internet Key Exchange version 2 (IKEv2) messages

IKEv2 (compared to IKEv1) does not completely use Phase 1 and Phase 2. Some documentation does not talk about phases, but about IKE SA and CHILD SA. In the packet exchange, the authentication is combined with the negotiation of the first IPsec SA (CHILD SA) into a single message. So part of Phase 1 and Phase 2 are combined (so the question is whether to classify the IKE_AUTH below rather into phase 1 or 2). It is always an exchange of Initiator Request - Responder Response messages.

IKEv2 message flow

Phase 1 - IKE_SA - successful result is an encrypted connection, but the Remote Peer is not authenticated
- pair of messages IKE_SA_INIT - negotiates encryption algorithms, performs Diffie-Hellman exchange
Phase 2 - CHILD_SA (IPsec SA)
- pair of messages IKE_AUTH - performs authentication and creates the first CHILD_SA according to the Phase 2 Selector (depending on the type of authentication, multiple IKE_AUTH messages may be exchanged)
- pair of messages CREATE_CHILD_SA - if we have multiple Phase 2 Selectors (local and remote subnets), it is used to establish another CHILD_SA, it is also used for CHILD_SA rekey
- pair of messages INFORMATIONAL - used to exchange control messages related to errors or event notifications

Route-based vs Policy-based VPN

Documentation IPsec VPN overview - Types of VPNs, IPsec VPN - Defining security policies

I learned that IPsec VPN can be configured in two different ways (IPsec Operation Modes). The newer, more transparent, and more feature-rich approach is Route-based. The only advantage of Policy-based is that it is supported if we have a FortiGate in Transparent mode. It should also work if one side of the IPsec tunnel uses Route-based and the other Policy-based. However, it always depends on the devices used (manufacturer).

Fortinet does not write much about it in the documentation (for example, in the FortiOS 6.2.3 Cookbook IPsec VPN configuration examples, we find only a small mention). Fortinet prefers Route-based VPN and by default in the GUI we can only configure this. If we want to set up a Policy-based IPsec VPN, we must first enable it in Feature Visibility.

Description from Fortinet documentation:

Route-based VPN or also Interface-based VPN
- creates a virtual IPsec network interface (VTI - Virtual Tunnel Interface), which applies encryption or decryption to all transmitted traffic
- we set up static routing (static route) for the VPN interface and the destination subnets
- for bi-directional communication, we must create two policies so that the communication can pass through the tunnel
- supports dynamic routing protocols
Policy-based VPN or also Tunnel-mode VPN
- we define a special policy (Security Policy, action IPsec instead of Allow), which applies the encryption
- one policy allows communication in both directions, the action is IPsec and we assign a VPN tunnel

On FortiGate, Route-based VPN is configured using the commands

config vpn ipsec phase1-interface
config vpn ipsec phase2-interface

While for Policy-based VPN, the following is used

config vpn ipsec phase1
config vpn ipsec phase2

For Route-based VPN, I came across different information on two questions. In the case of FortiGate, it doesn't matter, but if there is a different device on the other side, one specific option may be required.

whether in the Phase 2 Selector the subnets should be precisely defined (and thus limit the possible communication, further solved by the policy) or use all addresses 0.0.0.0/0
whether on the virtual tunnel interface (VTI), which is created for the VPN, addresses must be used and a certain connecting subnet (example IP 192.168.0.1, Remote IP 192.168.0.2/30), or left empty

I found examples of configurations where they mention that for Route-based VPN, the subnet should not be defined. We determine what goes into the VPN using static routes. Fortinet's documentation contains both variants, with a defined subnet and with all addresses.

Similarly, it is possible that a VPN tunnel between FortiGate and Cisco ASA requires setting addresses on the VTI interface. But for a regular VPN between FortiGate and FortiGate, we don't need to set it up. There are special situations where addressing is required (e.g., when the interface address is to be used as the outgoing address or for dynamic routing). There are also methods for automatic addressing IKE Mode Config and DHCP over IPsec.

Configuration on FortiGate

Documentation FortiOS 6.2.7 Cookbook - IPsec VPNs, FortiOS 6.2.7 CLI Reference - config vpn ipsec phase1-interface, FortiOS 6.0.0 Handbook - IPsec VPN from the GUI

Basic configuration of IPsec VPN is not complicated on FortiGate. Most of the things can be set up using the GUI, only a few special settings need to be done in the CLI. We can use the wizard (IPsec Wizard), but many guides recommend creating the VPN as a Custom type.

Example of creating a Site to Site IPsec VPN using the GUI

(VDOM) > VPN > IPsec Tunnels

The following images are from FortiOS 6.2.7. A Route-based VPN is created (the Policy-based IPsec VPN option is not enabled in Feature Visibility).

In the wizard, we enter the VPN tunnel name, which will also be used for the Virtual Tunnel Interface. We choose the Custom type and then go directly to setting the parameters, which are divided into sections.

FortiGate IPsec VPN Phase 1 Network a Authentication

We enter the IP address of the remote gateway and choose the local interface through which the remote Peer will connect. The IP address of the local VPN gateway is defaulted to the IP address of the chosen interface. If we want to set a different address, we choose Local Gateway. We can select the Secondary IP address set on the interface or enter any address (in which case the address must be activated somehow so that FortiGate listens on it).

Technical Tip: Forward Error Correction for IPsec VPN, Adding IPsec aggregate members in the GUI

A special setting is Mode Config for automatic configuration of Dialup clients Supporting IKE Mode Config clients.

In the Phase 1 Proposal section, we can set one or more combinations of encryption and authentication (integrity). If we choose the AES128GCM, AES256GCM or CHACHA20/POLY1305 encryption algorithm, we set the PRF instead of authentication (it's still a hash function).

FortiGate IPsec VPN Phase 1 Proposal PRF

In addition, we select the allowed Diffie-Hellman Group. We set the Key Lifetime for phase 1 (IKE SA).

FortiGate IPsec VPN Phase2 Selector / Proposal

As the Phase 2 Selector, we enter a named pair of local and remote addresses. To that, we define one or more Phase 2 Proposal. For AES128GCM, AES256GCM or CHACHA20/POLY1305, authentication is not entered. We can also enable PFS and specify the Diffie-Hellman Group. Enable Replay Detection against Replay Attacks.

The Auto-negotiate feature ensures the initialization of the Phase 2 SA negotiation without network traffic. The Autokey Keep Alive feature ensures the Rekey is performed when the Phase 2 SA is about to expire, without network traffic.

FortiOS 6.2.7 Cookbook - Phase 2 configuration, Using the IPSec auto-negotiate and keepalive options

Note: On FortiGate, we do not set the IPsec protocol, because it only supports ESP. By default, it also uses Tunnel Mode encapsulation.

Multiple subnets - Phase 2 Selectors

Some time ago, I needed to create an IPsec VPN where there were two subnets on the other side of the tunnel. In the Fortinet documentation, I found that it is necessary to create an Address Group, where the individual subnets are placed. In the VPN configuration, the Named Address and the given group are selected. Monitoring showed that the VPN had two Phase 2 Selectors, so it should have worked, but only one was established. According to the information, this does not work when there is a Cisco on the other side.

The functional solution was to create two separate Phase 2 under the same Phase 1 using the CLI. This is also mentioned in several places in the Fortinet documentation, Technical Tip: IPSec VPN between a FortiGate and a Cisco ASA with multiple subnets.

It seemed to me that it is not possible to create two selectors in the GUI. After creating them in the CLI, they are displayed here and can be edited. Adding another selector (Add) is only available when the group is not open.

Static Routing - Static Route

(VDOM) > Network > Static Routes

The next step is to create a static route for the destination network (subnet) that should be accessible through the VPN. We enter the Subnet and choose the Virtual Tunnel Interface (VTI) for our VPN as the interface.

For some cases, we don't need to set up the routing manually, but we can use a dynamic routing protocol (BGP or OSPF). Or we can set FortiGate to create a record based on the received Phase 2 Selector.

config vpn ipsec phase1-interface
   edit VPN-name
       set add-route enable
   next
end

Policies - Firewall Policy

(VDOM) > Policy & Objects > IPv4 Policy

To allow communication through the VPN, we need to create corresponding policies. From the network interface of the internal network to the VTI IPsec VPN and vice versa. Typically without NAT. If there is no policy, the VPN cannot be established.

ike 2:VPNtest: ignoring request to establish IPsec SA, no policy configured

Network Interface

(Global/VDOM) > Network > Interfaces

If necessary in the given configuration, we set the IP address of our and the remote gateway on the Virtual Tunnel Interface. The VTI is located under the interface on which the VPN is created.

Configuring Responder Only

Documentation Technical Tip: IPsec VPN response only in phase-1

If we want FortiGate to only wait for the IKE negotiation of the remote party, we can set it up using the CLI.

config vpn ipsec phase1-interface
    edit VPN-name
       set auto-negotiate disable
    next
end

HA (FGCP) cluster and IPsec VPN

Documentation FortiOS 6.2.7 Cookbook - IPsec VPN in an HA environment, FortiOS 6.0.0 Handbook - IPsec VPN SA sync

If we operate an FGCP cluster, we can set up Session Failover (Pickup), where session synchronization of certain protocols, including IPsec SA synchronization, is performed between cluster members. If an HA Failover (switch) occurs, the IPsec VPN traffic should not be interrupted.

Based on the description, I would understand that when Session Failover (Pickup) is disabled, IPsec SA will not be synchronized either. But if we enable debug IKE, we see many messages about synchronization and there are also several errors.

config system ha
    set session-pickup disable
end

For Session Failover to work, synchronization of ESP Sequence Numbers also needs to be enabled (the default setting is enabled).

config vpn ipsec phase1-interface
    edit JMENO
        set ha-sync-esp-seqno enable
    next
end

If we have debugging enabled and restart IKE, it will be logged.

ike 0: HA syncing enabled
ike 1: HA state change detected -> master
ike 1: HA syncing enabled

There are many messages in debug which I guess are related to connection synchronization and SA. Even if we filter the output for a specific VPN tunnel, these HA messages are displayed for all tunnels. Below is an example of HA logs during correct functioning of Active-Passive cluster (clear tunnel and new connection took place).

Master Node

ike 2:VPNtest:9665: HA send IKE SA del d239ff6252ee86df/45755640c64a91f4
ike 2:VPNtest: HA send IKE connection add X.X.X.X->Y.Y.Y.Y
ike 2:VPNtest:9666: HA send IKE SA add da32bee3295fc2c7/49815d13129c5bf8
ike 2:VPNtest: HA send IKEv2 message ID update send/recv=0/2
ike 2:VPNtest: HA IPsec send ESP seqno 1

Slave Node

ike 2:VPNtest: HA del IKE SA d239ff6252ee86df/45755640c64a91f4
ike 2:VPNtest: deleting IPsec SA with SPI 678def9b
ike 2:VPNtest:VPNtest: deleted IPsec SA with SPI 678def9b, SA count: 0
ike 2: HA add conn, unsupported type 144
ike 2: HA add conn, unsupported type 145
ike 2:VPNtest: HA IKE add conn

ike 2:VPNtest: deleting
ike 2:VPNtest: flushing 
ike 2:VPNtest: flushed 
ike 2:VPNtest: deleted
ike 2:VPNtest: set oper down
ike 2:VPNtest: HA add new connection X.X.X.X->Y.Y.Y.Y
ike 2:VPNtest: HA add IKE SA da32bee3295fc2c7/49815d13129c5bf8
ike 2:VPNtest:9666: HA add IKE SA done
ike 2:VPNtest: set oper up
ike 2:VPNtest:VPNtest: HA add IPsec SA 154ced4e/9ef46b80 seq 1 1
ike 2:VPNtest:VPNtest: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 2:VPNtest:VPNtest: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 2: HA receive seqno update send/recv=0/2
ike 2:VPNtest: carrier up
ike 2:VPNtest:: HA del IPsec SA 9ef46b80
ike 2:VPNtest: deleting IPsec SA with SPI 9ef46b80
ike 2:VPNtest:VPNtest: deleted IPsec SA with SPI 9ef46b80, SA count: 0
ike 2:VPNtest: HA IPsec receive ESP seqno 1, updating to 47868c01

When VDOM Partitioning was turned on (malfunctioning), the messages below appeared on the Master Node.

ike 2:VPNtest: HA send IKE connection add X.X.X.X->Y.Y.Y.Y
ike 2:VPNtest:2014: HA send IKE SA add 07d1d5afe37a6977/e88c7b2c40f3e162
ike 2:VPNtest:2014: deleting HA IKEv2 SA b88da824d5851609/359afb17f1288b7e
ike 2:VPNtest:2006: HA send IKE SA del b88da824d5851609/359afb17f1288b7e
ike 2:VPNtest: HA send IKEv2 message ID update send/recv=2/0
ike 2: HA add conn, unsupported type 144
ike 2: HA add conn, unsupported type 145
ike 2:VPNtest: HA IKE add conn
ike 2:VPNtest: HA add IKE SA 07d1d5afe37a6977/e88c7b2c40f3e162
ike 2:VPNtest:2014: HA add IKE SA done
ike 2:VPNtest: HA del IKE SA b88da824d5851609/359afb17f1288b7e
ike 2:VPNtest:: HA del IPsec SA f7738b0b
ike 2:VPNtest:VPNtest: HA add IPsec SA 154cd2cc/a6e192b3 seq 1 1
ike 2: HA receive seqno update send/recv=2/0
ike 2:VPNtest: HA IPsec send ESP seqno 47868c01
ike 2:VPNtest: HA IPsec master ignored unexpected ESP seqno update
ike 2:VPNtest:2006: ignoring message on HA synced IKE_SA
ike 2:VPNfg:1974: have HA IKE SA, attempting message ID sync

Configuration in CLI

Just a sample of the test configuration parts.

Parametry Phase 1

config vpn ipsec phase1-interface
    edit "VPNtoTest"
        set interface "Internet1"
        set ike-version 2
        set local-gw X.X.X.X
        set peertype any
        set net-device disable
        set proposal aes256-sha512
        set dhgrp 21
        set nattraversal disable
        set remote-gw Y.Y.Y.Y
        set psksecret ENC zzzzzz
    next
end

Parametry Phase 2

config vpn ipsec phase2-interface
    edit "VPNtoTest"
        set phase1name "VPNtoTest"
        set proposal aes256-sha512
        set dhgrp 21
        set replay disable
        set keepalive enable
        set keylifeseconds 600
    next
end

Firewall Policy

config firewall policy
    edit 56
        set name "To Test VPN"
        set uuid zzzzzz
        set srcintf "VPNtestLocalZone"
        set dstintf "VPNToTest"
        set srcaddr "VPNtestLocalNetwork"
        set dstaddr "VPNtestRemoteNetwork"
        set action accept
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
    edit 57
        set name "From Test VPN"
        set uuid zzzzzz
        set srcintf "VPNToTest"
        set dstintf "VPNtestLocalZone"
        set srcaddr "VPNtestRemoteNetwork"
        set dstaddr "VPNtestLocalNetwork"
        set action accept
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
end

Static Route

config router static
    edit 8
        set dst 192.168.0.0 255.255.255.0
        set device "VPNtoTest"
    next
end

Default values are not displayed in the listing, so it is convenient to display the complete listing with:

show full-configuration

List of Phase 1 and Phase 2

config vpn ipsec phase1-interface
    edit "VPNtoTest"
        set type static
        set interface "Internet1"
        set ip-version 4
        set ike-version 2
        set local-gw X.X.X.X
        set keylife 86400
        set authmethod psk
        unset authmethod-remote
        set peertype any
        set net-device disable
        set passive-mode disable
        set exchange-interface-ip disable
        set aggregate-member disable
        set mode-cfg disable
        set proposal aes256-sha512
        set localid ''
        set localid-type auto
        set auto-negotiate enabled
        set negotiate-timeout 30
        set fragmentation enable
        set ip-fragmentation post-encapsulation
        set dpd on-demand
        set forticlient-enforcement disable
        set comments ''
        set npu-offload enable
        set dhgrp 21
        set suite-b disable
        set eap disable
        set ppk disable
        set wizard-type custom
        set reauth disable
        set idle-timeout disable
        set ha-sync-esp-seqno enable
        set auto-discovery-sender disable
        set auto-discovery-receiver disable
        set auto-discovery-forwarder disable
        set encapsulation none
        set nattraversal disable
        set fragmentation-mtu 1200
        set childless-ike disable
        set rekey enable
        set network-overlay disable
        set remote-gw Y.Y.Y.Y
        set monitor ''
        set tunnel-search selectors
        set add-gw-route disable
        set psksecret ENC ...
        set dpd-retrycount 3
        set dpd-retryinterval 20
    next
end
config vpn ipsec phase2-interface
    edit "VPNtoTest"
        set phase1name "VPNtoTest"
        set proposal aes256-sha512
        set pfs enable
        set ipv4-df disable
        set dhgrp 21
        set replay disable
        set keepalive enable
        set auto-negotiate disable
        set auto-discovery-sender phase1
        set auto-discovery-forwarder phase1
        set keylife-type seconds
        set encapsulation tunnel-mode
        set comments ''
        set protocol 0
        set src-addr-type subnet
        set src-port 0
        set dst-addr-type subnet
        set dst-port 0
        set keylifeseconds 600
        set src-subnet 0.0.0.0 0.0.0.0
        set dst-subnet 0.0.0.0 0.0.0.0
    next
end

Monitoring, Troubleshooting and Debugging IPsec VPN

Documentation Technical Tip: Troubleshooting IPsec VPNs, FortiOS 6.0.0 Handbook - Troubleshooting, FortiOS 6.2.7 Cookbook - VPN IPsec troubleshooting

Note: If we operate an HA (FGCP) cluster, it is a good idea (and sometimes necessary) to work on the active FortiGate (VDOM Master Node where the given VDOM is active)

Monitoring

(VDOM) > Monitor > IPsec Monitor

Here we can monitor the status and basic information about the VPN. We can also bring down or bring up the VPN. If we right-click on the table header, we can display additional columns.

Event Logs

(VDOM) > Log & Report > Events

In the top right, we switch the display to VPN Events. Here are basic events such as negotiate IPsec phase 1, IPsec phase 2 status change - phase2-down, phase2-up, IPsec connection status change - tunnel-up. But there are no details, for example, why Phase 1 failed to establish. For more details, we need to use debug.

Note: I don't really understand why, but IPsec VPN communication is not visible in the Traffic log.

Obtaining Information about the VPN Tunnel

Note: Examples of the output of individual commands are provided at the end of the article.

List of IPsec Tunnels

get vpn ipsec tunnel summary

Tunnel Details

Displays main information about the IPsec tunnel, such as IP addresses, type (Route-based), mode (IKEv2), transferred data, DPD, Phase 2 Selector, its SA including SPI, Lifetime and time to Rekey, algorithms.

get vpn ipsec tunnel name JMENO

Displays the tunnel status, authentication method, IP addresses, brief Phase 2 details.

diagnose vpn ike config list summary

Information about Phase 1 of the given tunnel (two commands with almost the same result)

Commonly used. Displays information about Phase 1 - addresses, creation time, number of IKE SA and IPsec SA, details of IKE SA including SPI, direction of establishment (whether it's a Responder or Initiator), status, used Proposal (algorithms), Lifetime and Rekey (Soft Lifetime), DPD.

diagnose vpn ike gateway list name JMENO
get vpn ike gateway JMENO

Information about Phase 2 of the given tunnel

Commonly used. Displays basic information about the tunnel (Phase 1) and then details about Phase 2 - Selector, its SA including SPI, Lifetime (Hard/Soft), time to expiration, algorithms and keys, NPU (hardware acceleration).

diagnose vpn tunnel list name JMENO

Tunnel Operations

Restart IKE (all tunnels will be terminated)

diagnose vpn ike restart

Clear (terminate) IPsec Tunnel (either all tunnels or a specified one, instead of clear we can use flush the same way)

diagnose vpn ike gateway clear
diagnose vpn ike gateway clear name JMENO

Delete Tunnel SA (likely Phase 2 IPsec SA)

diagnose vpn tunnel flush JMENO

Bring Down (disable) Tunnel (two same options, we can specify only Phase2)

execute vpn ipsec tunnel down Phase2-name Phase1-name
diagnose vpn tunnel down Phase2-name Phase1-name

Bring Up Tunnel (two same options, we can specify only Phase2)

execute vpn ipsec tunnel up Phase2-name Phase1-name
diagnose vpn tunnel up Phase2-name Phase1-name

Ping from a Specific Address

For VPN testing (or tunnel activation), it can be helpful to trigger a ping from an IP address of the interface that is in our VPN range.

execute ping-options source 172.19.14.1
execute ping 192.168.0.1

Debug

Note: Examples of the debug output are provided at the end of the article.

When troubleshooting any issue, the most information is displayed when we enable the debug of the given protocol. The downside, however, is that Fortinet has no documentation for the messages (there is not even a proper description of enabling the debug) that appear in the debug. Most of them cannot be found anywhere, not even on Google. So we can only guess based on the meaning of the keywords, but for errors we don't know what's causing them.

We can set IKE debugging log filtering. The first command displays the set filters, the second clears the settings. There are several options for filtering. Many discussions indicate that filtering by Phase 1 name doesn't work, and I can confirm that. Filtering by destination IP address is usable. Even though we have filtering enabled for a specific destination IP address and thus a VPN tunnel (which works), messages related to HA for all tunnels are still displayed.

diagnose vpn ike log filter list
diagnose vpn ike log filter clear
diagnose vpn ike log-filter name JMENO
diagnose vpn ike log-filter dst-addr4 X.X.X.X

We can enable timestamp display for individual log entries.

diagnose debug console timestamp enable

Enable IKE protocol debugging (for 30 minutes) at the highest level (-1). The article FortiOS 5.6.0 Cookbook - Results advises using the value 63, which removes the encryption hash from the output, making it more readable.

diagnose debug application ike -1
diagnose debug enable

To disable debug mode (and reset the settings), we can use

diagnose debug disable
diagnose debug reset

Overall debug enablement may look like this

diagnose vpn ike log-filter dst-addr4 X.X.X.X
diagnose debug console timestamp enable
diagnose debug application ike 63
diagnose debug enable

Disabling and resetting the settings

diagnose vpn ike log filter clear
diagnose debug disable
diagnose debug reset

IPsec VPN Issues

Certificate Authentication and SHA-2

I was working on a Site to Site VPN between FortiGate and Cisco Firepower. Certificate authentication and a Phase 1 Proposal with AES-256-GCM with PRF SHA512 (aes256gcm-prfsha512) were used. Configuring the VPN was not a problem, but the tunnel would not establish. The reason was only displayed when debugging was enabled. Part of the events:

2020-07-01 08:29:07.977165 ike 1:JMENO:2924: certificate validation pending
2020-07-01 08:29:07.977755 ike 1:JMENO:2924: certificate validation complete
2020-07-01 08:29:07.977763 ike 1:JMENO:2924: certificate validation succeeded
2020-07-01 08:29:07.977899 ike 1:JMENO:2924: signature verification failed

For this error, I only found some old discussions saying that with a certificate, SHA-1 should be used. I made some attempts and the following worked:

change the authentication from certificate to Pre-shared Key (PSK)
in Phase 1, use any encryption, but authentication or PRF SHA1

Eventually, the technician on the other side came up with the fact that Cisco Firepower has a special command that allows SHA-512 for Phase 1, but uses SHA-1 for hashing the certificate response. The tunnel then established with the required settings.

Non-functioning IKEv2 IPsec VPN Rekey

The IPsec tunnel mentioned above had been working quite well for several months. Then it started being used more intensively, and a small configuration change was made, only the authentication certificate was replaced. And suddenly, when Rekey Phase 2 (IPsec SA) was supposed to occur, the communication through the tunnel stopped. FortiGate still showed the tunnel as established, but on the Cisco side it was reported as down.

We made an adjustment to the configuration to a proper Route-based IPsec VPN and things improved a lot. But still, whenever Rekey was supposed to happen, the communication was interrupted. The VPN was very important, so we couldn't continue experimenting. It was temporarily resolved in an alternative way. But we created a test VPN between our FortiGate cluster and Cisco Firepower in the lab.

I also opened a ticket with Fortinet Support. More than 14 days have passed and the support response is the same as I described in FortiGate 6.2.3 bugs, debug and support.

From the beginning, I started studying the operation of IPsec and wrote this article. And we are doing experiments and debugging on the test VPN. It seems that the problem is on the FortiGate side. Cisco TAC, who analyzed the VPN connection on the other side, also came to this conclusion.

It seems to me that the problem is in the operation of the FortiGate HA cluster (with Virtual clustering) and IPsec SA synchronization. When the Phase 2 SA Lifetime is about to expire, Cisco Firepower sends a Rekey request (CREATE_CHILD_SA). This is visible on the Cisco side. But also in the FortiGate debug, where the incoming message is visible, which is repeated several times in a row. FortiGate always logs that the message is ignored.

ike 2: comes Y.Y:Y.Y:500->X.X.X.X:500,ifindex=59....
ike 2: IKEv2 exchange=CREATE_CHILD id=b88da824d5851609/359afb17f1288b7e:00000002 len=528
ike 2:VPNtest:2006: ignoring message on HA synced IKE_SA

Because Cisco doesn't get a response, it brings down (Bring Down) Phase 1 (IKE SA). And it starts a complete tunnel negotiation (IKE_SA_INIT), which succeeds. But this causes some tunnel unavailability. If there is a ping running through the VPN, 1 to 4 messages will be lost.

If the situation is reversed and FortiGate is to perform a Rekey, the log shows that it is preparing for it, but finds that it is missing the IKE SA. Even though, just before that, the Phase 1 details looked fine. Just like in the first case, FortiGate performs a complete new tunnel negotiation.

ike 2:VPNtest:VPNtest: IPsec SA f7738b0b/154cd2c6 rekey 59 X.X.X.X->Y.Y:Y.Y:0
ike 2:VPNtest:VPNtest: using existing connection
ike 2:VPNtest:VPNtest: config found
ike 2:VPNtest:VPNtest: IPsec SA connect 59 X.X.X.X->Y.Y:Y.Y:500 negotiating
ike 2:VPNtest: no suitable IKE_SA, queuing CHILD_SA request and initiating IKE_SA negotiation
ike 2:VPNtest:2014: sent IKE msg (SA_INIT): X.X.X.X:500-> Y.Y:Y.Y:500, len=260, id=07d1d5afe37a6977/0000000000000000

I sent all this information to Support, including a number of clear questions. I did not receive an answer to any of the questions, and their responses were off-base. I will update the article if the situation changes.

Updated on 4/19/2021

After three weeks, I received a response from Fortinet Support:

We suspect the reason why the proposal is rejected is somehow affected by the fact that the fortigate is part of an HA cluster. Would it be possible to fail over the units, swap the roles and confirm if the problem remains?

So, what I had been writing to them for some time. I had considered trying to turn off VDOM Partitioning (Virtual clustering) since I discovered the HA issues, but I was concerned about how much it would impact operations. I was hoping that the support would come up with a better solution and recognize the exact problem from the debug log. In the end, I tried it and described the disabling in FortiGate High Availability cluster and Virtual Domains (VDOM).

I can confirm that after disabling VDOM Partitioning, when the FGCP cluster started working as a classic Active-Passive, all IPsec VPN tunnels started working correctly. Rekey of IPsec SA (Phase 2) is performed correctly, IKE SA is not re-established. The debug log does not contain any messages about ignoring or missing IKE SA. The ping in the tunnel has no interruption.

This means that VDOM Partitioning (Virtual clustering), which seems to me like a nice feature, does not work well. It's a shame that Fortinet has a number of features (which have been here for years) that it's better not to use. Personally, disabling VDOM Partitioning doesn't seem to me like a solution to the problem, but like a Workaround. I'll see how support will react.

Updated on 4/26/2021

The support response was ridiculous again. They wanted me to try turning VDOM Partitioning back on to see if the problem returns. I still don't understand why Fortinet is not able to test the problem in their own lab. I performed a few tests.

I turned on VDOM Partitioning, but set the priorities of both clusters the same, so all VDOMs remained on the same FortiGate (the Master was the same for both clusters). I changed the priority for the second virtual cluster, so the VDOM with the IPsec VPN tunnels moved to the second node. Immediately, the described IPsec VPN problem manifested.

I restored the priority for the second virtual cluster, the VDOM moved back (all were on the same unit). At that moment, the problems disappeared and the IPsec VPN worked correctly.

Updated on 4/29/2021

I refused to be a free tester for Fortinet and agreed to close the ticket without a resolution.

Real Examples of Command Output

FW2 (FWEXT) # get vpn ipsec tunnel summary

'VPNtest' Y.Y.Y.Y:0 selectors(total,up): 1/1 rx(pkt,err): 1/0 tx(pkt,err): 10/0

FW2 (FWEXT) # get vpn ipsec tunnel name VPNtest

gateway
  name: 'VPNtest'
  type: route-based
  local-gateway: X.X.X.X:0 (static)
  remote-gateway: Y.Y.Y.Y:0 (static)
  mode: ike-v2
  interface: 'Internet1' (59)
  rx  packets: 1  bytes: 172  errors: 0
  tx  packets: 10  bytes: 840  errors: 0
  dpd: disabled
  selectors
    name: 'VPNtest'
    auto-negotiate: disable
    mode: tunnel
    src: 0:0.0.0.0/0.0.0.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA
      lifetime/rekey: 600/150   
      mtu: 1422
      tx-esp-seq: 1
      replay: disabled
      qat: 0
      inbound
        spi: 154ccf99
        enc:  aes-cb  53a58d31cf4cd87514472d525089af1be2e3df8a661d29976baf748035becae6
        auth: sha512 df7d3307ff818668ee31a6f9a49d394b1c869d0f6c106f3af1231b97ba7e1a1afd0307ded0ae243c90ba16195d7ff393fb4a3450849374cb30ec703f8ee8823a
      outbound
        spi: bf67d15d
        enc:  aes-cb  061c2415262bbe26fce6ac8eb69f57563a49957931fc6d639d4217c2ae572b42
        auth: sha512  3babf1d32e39324d26ac4b9a3a1490706f6052151ac8a9de6b44488b63a5f53188056943f172b8b94e2d8dc548575504ed0b28a9493db5200d7c98685862dc49
      NPU acceleration: none

FW2 (FWEXT) # diagnose vpn ike config list summary

vd: FWEXT/2
name: VPNtest
serial: 3
version: 2
status.admin: up
status.operational: up
type: static
local: X.X.X.X
remote: Y.Y.Y.Y
mode: main
dpd: disabled
auth: psk
dhgrp:  21
xauth: none
interface: Internet1
virtual-interface-addr: 192.168.199.2 -> 192.168.199.1
auto-discovery-sender: disable
auto-discovery-receiver: disable
phase2s:
  VPNtest proto 0 src 0.0.0.0/0.0.0.0:0 dst 0.0.0.0/0.0.0.0:0  dhgrp 14  keep-alive  route-new
policy: yes

FW2 (FWEXT) # get vpn ike gateway VPNtest

vd: FWEXT/2
name: VPNtest
version: 2
interface: Internet1 59
addr: X.X.X.X:500 -> Y.Y.Y.Y:500
created: 489s ago
IKE SA  created: 1/1  established: 1/1  time: 0/0/0 ms
IPsec SA  created: 1/1  established: 1/1  time: 0/0/0 ms

  id/spi: 1169 e870277e2268318c/acb0001b24748673
  direction: responder
  status: established 489-489s ago = 0ms
  proposal: aes-256-sha512
  SK_ei: 3a83d24fab8f492e-ad914b1c46678cf5-3888fc9960303321-9feebc6df28fa2e6
  SK_er: a401942671d47623-6a4f692609b2ca87-59f2c1461cc6ed07-4390453c2697171b
  SK_ai: 14912c8eec3b0331-a9ee44ed7a3de18d-139206b22aa983ac-5f9067063500fa2a-0fce49e5baea07bf-3127112da547f587-35539d4d71c960a2-7ab822ca0963d51d
  SK_ar: d06391ab26ef3653-a3d5cda87eb44fb9-e738204c986dcabc-efa75f6e5d9cd38f-a44c45b7ead4989d-410dc247d496fed7-91ee40d7757dc7dc-ab1310a7ad744f8b
  lifetime/rekey: 86400/85640
  DPD sent/recv: 00000000/00000000

FW2 (FWEXT) # diagnose vpn ike gateway list name VPNtest

vd: FWEXT/2
name: VPNtest
version: 2
interface: Internet1 59
addr: X.X.X.X:500 -> Y.Y.Y.Y:500
virtual-interface-addr: 192.168.199.2 -> 192.168.199.1
created: 535s ago
PPK: no
IKE SA: created 1/1  established 1/1  time 0/0/0 ms
IPsec SA: created 1/1  established 1/1  time 0/0/0 ms

  id/spi: 1169 e870277e2268318c/acb0001b24748673
  direction: responder
  status: established 535-535s ago = 0ms
  proposal: aes256-sha512
  child: no
  SK_ei: 3a83d24fab8f492e-ad914b1c46678cf5-3888fc9960303321-9feebc6df28fa2e6
  SK_er: a401942671d47623-6a4f692609b2ca87-59f2c1461cc6ed07-4390453c2697171b
  SK_ai: 14912c8eec3b0331-a9ee44ed7a3de18d-139206b22aa983ac-5f9067063500fa2a-0fce49e5baea07bf-3127112da547f587-35539d4d71c960a2-7ab822ca0963d51d
  SK_ar: d06391ab26ef3653-a3d5cda87eb44fb9-e738204c986dcabc-efa75f6e5d9cd38f-a44c45b7ead4989d-410dc247d496fed7-91ee40d7757dc7dc-ab1310a7ad744f8b
  PPK: no
  message-id sent/recv: 0/2
  lifetime/rekey: 86400/85594
  DPD sent/recv: 00000000/00000000

FW2 (FWEXT) # diagnose vpn tunnel list name VPNtest

list ipsec tunnel by names in vd 2
------------------------------------------------------
name=VPNtest ver=2 serial=3 X.X.X.X:0->Y.Y.Y.Y:0 dst_mtu=1500
bound_if=59 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/520 options[0208]=npu frag-rfc  run_state=0 accept_traffic=1 overlay_id=0
proxyid_num=1 child_num=0 refcnt=13 ilast=191 olast=191 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=off on=0 idle=20000ms retry=3 count=0 seqno=0
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=VPNtest proto=0 sa=1 ref=2 serial=14
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA:  ref=3 options=25 type=00 soft=0 mtu=1422 expire=26/0B replaywin=0
       seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0 hash_search_len=1
  life: type=01 bytes=0/0 timeout=580/600
  dec: spi=154ccf99 esp=aes key=32 53a58d31cf4cd87514472d525089af1be2e3df8a661d29976baf748035becae6
       ah=sha512 key=64 df7d3307ff818668ee31a6f9a49d394b1c869d0f6c106f3af1231b97ba7e1a1afd0307ded0ae243c90ba16195d7ff393fb4a3450849374cb30ec703f8ee8823a
  enc: spi=bf67d15d esp=aes key=32 061c2415262bbe26fce6ac8eb69f57563a49957931fc6d639d4217c2ae572b42
       ah=sha512 key=64 3babf1d32e39324d26ac4b9a3a1490706f6052151ac8a9de6b44488b63a5f53188056943f172b8b94e2d8dc548575504ed0b28a9493db5200d7c98685862dc49
  dec:pkts/bytes=0/0, enc:pkts/bytes=0/0
  npu_flag=00 npu_rgwy=Y.Y.Y.Y npu_lgwy=X.X.X.X npu_selid=20 dec_npuid=0 enc_npuid=0
  run_tally=1

Real Examples of IPsec VPN Tunnel Debug Logs

I have created a test Site to Site IPsec VPN between a FortiGate cluster with FortiOS 6.2.7 (IP X.X.X.X), with the VPN name VPNfg, and an old FortiGate with FortiOS 5.6.8 (IP Y.Y.Y.Y), with the VPN name VPNfg2.

Direction from FortiGate cluster to standalone FortiGate

When the VPN is established from the FortiGate cluster (Initiator) to the old FortiGate (Responder), everything seems to work well. Here are the debug logs.

Tunnel establishment (SA_INIT) - Initiator

ike 2:VPNfg:VPNfg: chosen to populate IKE_SA traffic-selectors
ike 2:VPNfg: no suitable IKE_SA, queuing CHILD_SA request and initiating IKE_SA negotiation
ike 2:VPNfg:4039: sent IKE msg (SA_INIT): X.X.X.X:500->Y.Y.Y.Y:500, len=260, id=45a1740c06c3cfc5/0000000000000000
ike 2:VPNfg: HA del IKE SA 9f5b9daecb715a9f/0a9e39dfd071718c
ike 2: comes Y.Y.Y.Y:500->X.X.X.X:500,ifindex=59....
ike 2: IKEv2 exchange=SA_INIT_RESPONSE id=45a1740c06c3cfc5/1680a67638fca00e len=260
ike 2:VPNfg:4039: initiator received SA_INIT response
ike 2:VPNfg:4039: processing notify type FRAGMENTATION_SUPPORTED
ike 2:VPNfg:4039: incoming proposal:
ike 2:VPNfg:4039: proposal id = 1:
ike 2:VPNfg:4039:   protocol = IKEv2:
ike 2:VPNfg:4039:      encapsulation = IKEv2/none
ike 2:VPNfg:4039:         type=ENCR, val=AES_CBC (key_len = 256)
ike 2:VPNfg:4039:         type=INTEGR, val=AUTH_HMAC_SHA2_512_256
ike 2:VPNfg:4039:         type=PRF, val=PRF_HMAC_SHA2_512
ike 2:VPNfg:4039:         type=DH_GROUP, val=ECP521.
ike 2:VPNfg:4039: matched proposal id 1
ike 2:VPNfg:4039: proposal id = 1:
ike 2:VPNfg:4039:   protocol = IKEv2:
ike 2:VPNfg:4039:      encapsulation = IKEv2/none
ike 2:VPNfg:4039:         type=ENCR, val=AES_CBC (key_len = 256)
ike 2:VPNfg:4039:         type=INTEGR, val=AUTH_HMAC_SHA2_512_256
ike 2:VPNfg:4039:         type=PRF, val=PRF_HMAC_SHA2_512
ike 2:VPNfg:4039:         type=DH_GROUP, val=ECP521.
ike 2:VPNfg:4039: lifetime=86400
ike 2:VPNfg:4039: initiator preparing AUTH msg
ike 2:VPNfg:4039: sending INITIAL-CONTACT
ike 2:VPNfg:4039: sent IKE msg (AUTH): X.X.X.X:500->Y.Y.Y.Y:500, len=288, id=45a1740c06c3cfc5/1680a67638fca00e:00000001
ike 2: comes Y.Y.Y.Y:500->X.X.X.X:500,ifindex=59....
ike 2: IKEv2 exchange=AUTH_RESPONSE id=45a1740c06c3cfc5/1680a67638fca00e:00000001 len=272
ike 2:VPNfg:4039: initiator received AUTH msg
ike 2:VPNfg:4039: peer identifier IPV4_ADDR Y.Y.Y.Y
ike 2:VPNfg:4039: auth verify done
ike 2:VPNfg:4039: initiator AUTH continuation
ike 2:VPNfg:4039: authentication succeeded
ike 2:VPNfg:4039: processing notify type MESSAGE_ID_SYNC_SUPPORTED
ike 2:VPNfg:4039: established IKE SA 45a1740c06c3cfc5/1680a67638fca00e
ike 2:VPNfg: HA send IKE connection add X.X.X.X->Y.Y.Y.Y
ike 2:VPNfg:4039: HA send IKE SA add 45a1740c06c3cfc5/1680a67638fca00e
ike 2:VPNfg: set oper up
ike 2:VPNfg: schedule auto-negotiate
ike 2:VPNfg:4039:12090: peer proposal:
ike 2:VPNfg:4039:12090: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:12090: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12090: comparing selectors
ike 2:VPNfg:4039:VPNfg:12090: matched by rfc-rule-2
ike 2:VPNfg:4039:VPNfg:12090: phase2 matched by subset
ike 2:VPNfg:4039:VPNfg:12090: accepted proposal:
ike 2:VPNfg:4039:VPNfg:12090: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12090: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12090: autokey
ike 2:VPNfg:4039:VPNfg:12090: incoming child SA proposal:
ike 2:VPNfg:4039:VPNfg:12090: proposal id = 1:
ike 2:VPNfg:4039:VPNfg:12090:   protocol = ESP:
ike 2:VPNfg:4039:VPNfg:12090:      encapsulation = TUNNEL
ike 2:VPNfg:4039:VPNfg:12090:         type=ENCR, val=AES_CBC (key_len = 256)
ike 2:VPNfg:4039:VPNfg:12090:         type=INTEGR, val=SHA512
ike 2:VPNfg:4039:VPNfg:12090:         type=ESN, val=NO
ike 2:VPNfg:4039:VPNfg:12090:         PFS is disabled
ike 2:VPNfg:4039:VPNfg:12090: matched proposal id 1
ike 2:VPNfg:4039:VPNfg:12090: proposal id = 1:
ike 2:VPNfg:4039:VPNfg:12090:   protocol = ESP:
ike 2:VPNfg:4039:VPNfg:12090:      encapsulation = TUNNEL
ike 2:VPNfg:4039:VPNfg:12090:         type=ENCR, val=AES_CBC (key_len = 256)
ike 2:VPNfg:4039:VPNfg:12090:         type=INTEGR, val=SHA512
ike 2:VPNfg:4039:VPNfg:12090:         type=ESN, val=NO
ike 2:VPNfg:4039:VPNfg:12090:         PFS is disabled
ike 2:VPNfg:4039:VPNfg:12090: lifetime=600
ike 2:VPNfg:4039:VPNfg:12090: set sa life soft seconds=569.
ike 2:VPNfg:4039:VPNfg:12090: set sa life hard seconds=600.
ike 2:VPNfg:4039:VPNfg:12090: IPsec SA selectors #src=1 #dst=1
ike 2:VPNfg:4039:VPNfg:12090: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12090: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12090: add IPsec SA: SPIs=154cda88/a2664922
ike 2:VPNfg:4039:VPNfg:12090: added IPsec SA: SPIs=154cda88/a2664922
ike 2:VPNfg: HA send IKEv2 message ID update send/recv=2/0
ike 2:VPNfg:4039:VPNfg:12090: sending SNMP tunnel UP trap
ike 2: HA add conn, unsupported type 144
ike 2: HA add conn, unsupported type 145
ike 2:VPNfg: HA IKE add conn
ike 2:VPNfg: HA add IKE SA 45a1740c06c3cfc5/1680a67638fca00e
ike 2:VPNfg:4039: HA add IKE SA ignored, already exists
ike 2:VPNfg:VPNfg: HA add IPsec SA 154cda88/a2664922 seq 47868c01 1
ike 2:VPNfg:VPNfg: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:VPNfg: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 2: HA receive seqno update send/recv=2/0

Tunnel Establishment (SA_INIT) - Responder

ike 0: comes X.X.X.X:500->Y.Y.Y.Y:500,ifindex=8....
ike 0: IKEv2 exchange=SA_INIT id=45a1740c06c3cfc5/0000000000000000 len=260
ike 0:45a1740c06c3cfc5/0000000000000000:175: responder received SA_INIT msg
ike 0:45a1740c06c3cfc5/0000000000000000:175: received notify type FRAGMENTATION_SUPPORTED
ike 0:45a1740c06c3cfc5/0000000000000000:175: incoming proposal:
ike 0:45a1740c06c3cfc5/0000000000000000:175: proposal id = 1:
ike 0:45a1740c06c3cfc5/0000000000000000:175:   protocol = IKEv2:
ike 0:45a1740c06c3cfc5/0000000000000000:175:      encapsulation = IKEv2/none
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=ENCR, val=AES_CBC (key_len = 256)
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=INTEGR, val=AUTH_HMAC_SHA2_512_256
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=PRF, val=PRF_HMAC_SHA2_512
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=DH_GROUP, val=ECP521.
ike 0:45a1740c06c3cfc5/0000000000000000:175: matched proposal id 1
ike 0:45a1740c06c3cfc5/0000000000000000:175: proposal id = 1:
ike 0:45a1740c06c3cfc5/0000000000000000:175:   protocol = IKEv2:
ike 0:45a1740c06c3cfc5/0000000000000000:175:      encapsulation = IKEv2/none
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=ENCR, val=AES_CBC (key_len = 256)
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=INTEGR, val=AUTH_HMAC_SHA2_512_256
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=PRF, val=PRF_HMAC_SHA2_512
ike 0:45a1740c06c3cfc5/0000000000000000:175:         type=DH_GROUP, val=ECP521.
ike 0:45a1740c06c3cfc5/0000000000000000:175: lifetime=86400
ike 0:45a1740c06c3cfc5/0000000000000000:175: SA proposal chosen, matched gateway VPNfg2
ike 0: found VPNfg2 Y.Y.Y.Y 8 -> X.X.X.X:500
ike 0:VPNfg2:175: processing notify type FRAGMENTATION_SUPPORTED
ike 0:VPNfg2:175: responder preparing SA_INIT msg
ike 0:VPNfg2:175: sent IKE msg (SA_INIT_RESPONSE): Y.Y.Y.Y:500->X.X.X.X:500, len=260, id=45a1740c06c3cfc5/1680a67638fca00e
ike 0: comes X.X.X.X:500->Y.Y.Y.Y:500,ifindex=8....
ike 0: IKEv2 exchange=AUTH id=45a1740c06c3cfc5/1680a67638fca00e:00000001 len=288
ike 0:VPNfg2:175: responder received AUTH msg
ike 0:VPNfg2:175: processing notify type INITIAL_CONTACT
ike 0:VPNfg2:175: processing notify type MESSAGE_ID_SYNC_SUPPORTED
ike 0:VPNfg2:175: peer identifier IPV4_ADDR X.X.X.X
ike 0:VPNfg2:175: auth verify done
ike 0:VPNfg2:175: responder AUTH continuation
ike 0:VPNfg2:175: authentication succeeded
ike 0:VPNfg2:175: responder creating new child
ike 0:VPNfg2:175:895: peer proposal:
ike 0:VPNfg2:175:895: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:895: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:895: comparing selectors
ike 0:VPNfg2:175:VPNfg2:895: matched by rfc-rule-2
ike 0:VPNfg2:175:VPNfg2:895: phase2 matched by subset
ike 0:VPNfg2:175:VPNfg2:895: accepted proposal:
ike 0:VPNfg2:175:VPNfg2:895: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:895: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:895: autokey
ike 0:VPNfg2:175:VPNfg2:895: incoming child SA proposal:
ike 0:VPNfg2:175:VPNfg2:895: proposal id = 1:
ike 0:VPNfg2:175:VPNfg2:895:   protocol = ESP:
ike 0:VPNfg2:175:VPNfg2:895:      encapsulation = TUNNEL
ike 0:VPNfg2:175:VPNfg2:895:         type=ENCR, val=AES_CBC (key_len = 256)
ike 0:VPNfg2:175:VPNfg2:895:         type=INTEGR, val=SHA512
ike 0:VPNfg2:175:VPNfg2:895:         type=ESN, val=NO
ike 0:VPNfg2:175:VPNfg2:895:         PFS is disabled
ike 0:VPNfg2:175:VPNfg2:895: matched proposal id 1
ike 0:VPNfg2:175:VPNfg2:895: proposal id = 1:
ike 0:VPNfg2:175:VPNfg2:895:   protocol = ESP:
ike 0:VPNfg2:175:VPNfg2:895:      encapsulation = TUNNEL
ike 0:VPNfg2:175:VPNfg2:895:         type=ENCR, val=AES_CBC (key_len = 256)
ike 0:VPNfg2:175:VPNfg2:895:         type=INTEGR, val=SHA512
ike 0:VPNfg2:175:VPNfg2:895:         type=ESN, val=NO
ike 0:VPNfg2:175:VPNfg2:895:         PFS is disabled
ike 0:VPNfg2:175:VPNfg2:895: lifetime=600
ike 0:VPNfg2:175: responder preparing AUTH msg
ike 0:VPNfg2:175: established IKE SA 45a1740c06c3cfc5/1680a67638fca00e
ike 0:VPNfg2:175: processing INITIAL-CONTACT
ike 0:VPNfg2:175: processed INITIAL-CONTACT
ike 0:VPNfg2: schedule auto-negotiate
ike 0:VPNfg2:174: IKE SA is a duplicate of 45a1740c06c3cfc5/1680a67638fca00e
ike 0:VPNfg2:174: schedule delete of IKE SA 9f5b9daecb715a9f/0a9e39dfd071718c
ike 0:VPNfg2:175:VPNfg2:895: set sa life soft seconds=580.
ike 0:VPNfg2:175:VPNfg2:895: set sa life hard seconds=600.
ike 0:VPNfg2:175:VPNfg2:895: IPsec SA selectors #src=1 #dst=1
ike 0:VPNfg2:175:VPNfg2:895: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:895: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:895: add IPsec SA: SPIs=a2664922/154cda88
ike 0:VPNfg2:175:VPNfg2:895: added IPsec SA: SPIs=a2664922/154cda88
ike 0:VPNfg2:175:VPNfg2:895: sending SNMP tunnel UP trap
ike 0:VPNfg2:175: sent IKE msg (AUTH_RESPONSE): Y.Y.Y.Y:500->X.X.X.X:500, len=272, id=45a1740c06c3cfc5/1680a67638fca00e:00000001

Messages in progress (INFORMATIONAL) - Initiator

ike 2:VPNfg: HA IPsec send ESP seqno 47868c01
ike 2:VPNfg: HA IPsec master ignored unexpected ESP seqno update
ike 2:VPNfg:4039: have HA IKE SA, attempting message ID sync
ike 2:VPNfg:12096: sending IKEV2_MESSAGE_ID_SYNC NOTIFY msg
ike 2:VPNfg:4039:12096: send informational
ike 2:VPNfg:4039: sent IKE msg (INFORMATIONAL): X.X.X.X:500->Y.Y.Y.Y:500, len=112, id=45a1740c06c3cfc5/1680a67638fca00e
ike 2:VPNfg:4039: IKEV2_MESSAGE_ID_SYNC nonce=b645f272 expected send/recv msg IDs=6/1
ike 2: comes Y.Y.Y.Y:500->X.X.X.X:500,ifindex=59....
ike 2: IKEv2 exchange=INFORMATIONAL_RESPONSE id=45a1740c06c3cfc5/1680a67638fca00e len=112
ike 2:VPNfg:4039: received informational response
ike 2:VPNfg:12096: received NOTIFY acknowledgement
ike 2:VPNfg:4039:12096: processing informational acknowledgement
ike 2:VPNfg:4039: processing notify type MESSAGE_ID_SYNC
ike 2:VPNfg:4039: message ID sync response received nonce=b645f272 expected send/recv msg IDs=1/6
ike 2:VPNfg:4039: synced message IDs: send/recv=6/1
ike 2:VPNfg: HA send IKEv2 message ID update send/recv=6/1
ike 2: HA receive seqno update send/recv=6/1

Messages in progress (INFORMATIONAL) - Responder

ike 0: comes X.X.X.X:500->Y.Y.Y.Y:500,ifindex=8....
ike 0: IKEv2 exchange=INFORMATIONAL id=45a1740c06c3cfc5/1680a67638fca00e len=112
ike 0:VPNfg2:175: received informational request
ike 0:VPNfg2:175: processing notify type MESSAGE_ID_SYNC
ike 0:VPNfg2:175: message ID sync request received nonce=b645f272 expected send/recv msg IDs=6/1
ike 0:VPNfg2:175: current send/recv msg IDs=0/2 updated to new send/recv msg IDs=1/6
ike 0:VPNfg2:175: sent IKE msg (INFORMATIONAL_RESPONSE): Y.Y.Y.Y:500->X.X.X.X:500, len=112, id=45a1740c06c3cfc5/1680a67638fca00e

Performing Rekey (CREATE_CHILD) - Initiator

ike 2:VPNfg:VPNfg: IPsec SA a2664922/154cda88 rekey 59 X.X.X.X->Y.Y.Y.Y:0
ike 2:VPNfg:VPNfg: using existing connection
ike 2:VPNfg:VPNfg: config found
ike 2:VPNfg:VPNfg: IPsec SA connect 59 X.X.X.X->Y.Y.Y.Y:500 negotiating
ike 2:VPNfg:4039: have HA IKE SA, attempting message ID sync
ike 2:VPNfg:12103: sending IKEV2_MESSAGE_ID_SYNC NOTIFY msg
ike 2:VPNfg:4039:12103: send informational
ike 2:VPNfg:4039: sent IKE msg (INFORMATIONAL): X.X.X.X:500->Y.Y.Y.Y:500, len=112, id=45a1740c06c3cfc5/1680a67638fca00e
ike 2:VPNfg:4039: IKEV2_MESSAGE_ID_SYNC nonce=1a2b0a55 expected send/recv msg IDs=14/1
ike 2:VPNfg:4039:VPNfg:12102: IKEv2 queuing create-child request
ike 2: comes Y.Y.Y.Y:500->X.X.X.X:500,ifindex=59....
ike 2: IKEv2 exchange=INFORMATIONAL_RESPONSE id=45a1740c06c3cfc5/1680a67638fca00e len=112
ike 2:VPNfg:4039: received informational response
ike 2:VPNfg:12103: received NOTIFY acknowledgement
ike 2:VPNfg:4039:12103: processing informational acknowledgement
ike 2:VPNfg:4039: processing notify type MESSAGE_ID_SYNC
ike 2:VPNfg:4039: message ID sync response received nonce=1a2b0a55 expected send/recv msg IDs=1/14
ike 2:VPNfg:4039: synced message IDs: send/recv=14/1
ike 2:VPNfg: HA send IKEv2 message ID update send/recv=14/1
ike 2:VPNfg:4039:12102 initiating CREATE_CHILD exchange
ike 2:VPNfg:4039:VPNfg:12102: PFS enabled
ike 2:VPNfg:4039:12102 rekey in progress for SPI 154cda88
ike 2:VPNfg:4039: sent IKE msg (CREATE_CHILD): X.X.X.X:500->Y.Y.Y.Y:500, len=384, id=45a1740c06c3cfc5/1680a67638fca00e:0000000e
ike 2: comes Y.Y.Y.Y:500->X.X.X.X:500,ifindex=59....
ike 2: IKEv2 exchange=CREATE_CHILD_RESPONSE id=45a1740c06c3cfc5/1680a67638fca00e:0000000e len=368
ike 2:VPNfg:4039: received create-child response
ike 2:VPNfg:4039: initiator received CREATE_CHILD msg
ike 2:VPNfg:4039:VPNfg:12102: found child SA SPI 154cda8e state=3
ike 2:VPNfg:4039:VPNfg:12102: PFS enabled, group=21
ike 2:VPNfg:4039:12102: peer proposal:
ike 2:VPNfg:4039:12102: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:12102: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12102: comparing selectors
ike 2:VPNfg:4039:VPNfg:12102: matched by rfc-rule-2
ike 2:VPNfg:4039:VPNfg:12102: phase2 matched by subset
ike 2:VPNfg:4039:VPNfg:12102: accepted proposal:
ike 2:VPNfg:4039:VPNfg:12102: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12102: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12102: autokey
ike 2:VPNfg:4039:VPNfg:12102: incoming child SA proposal:
ike 2:VPNfg:4039:VPNfg:12102: proposal id = 1:
ike 2:VPNfg:4039:VPNfg:12102:   protocol = ESP:
ike 2:VPNfg:4039:VPNfg:12102:      encapsulation = TUNNEL
ike 2:VPNfg:4039:VPNfg:12102:         type=ENCR, val=AES_CBC (key_len = 256)
ike 2:VPNfg:4039:VPNfg:12102:         type=INTEGR, val=SHA512
ike 2:VPNfg:4039:VPNfg:12102:         type=DH_GROUP, val=ECP521
ike 2:VPNfg:4039:VPNfg:12102:         type=ESN, val=NO
ike 2:VPNfg:4039:VPNfg:12102: matched proposal id 1
ike 2:VPNfg:4039:VPNfg:12102: proposal id = 1:
ike 2:VPNfg:4039:VPNfg:12102:   protocol = ESP:
ike 2:VPNfg:4039:VPNfg:12102:      encapsulation = TUNNEL
ike 2:VPNfg:4039:VPNfg:12102:         type=ENCR, val=AES_CBC (key_len = 256)
ike 2:VPNfg:4039:VPNfg:12102:         type=INTEGR, val=SHA512
ike 2:VPNfg:4039:VPNfg:12102:         type=DH_GROUP, val=ECP521
ike 2:VPNfg:4039:VPNfg:12102:         type=ESN, val=NO
ike 2:VPNfg:4039:VPNfg:12102: lifetime=600
ike 2:VPNfg: schedule auto-negotiate
ike 2:VPNfg:4039:VPNfg:12102: set sa life soft seconds=572.
ike 2:VPNfg:4039:VPNfg:12102: set sa life hard seconds=600.
ike 2:VPNfg:4039:VPNfg:12102: IPsec SA selectors #src=1 #dst=1
ike 2:VPNfg:4039:VPNfg:12102: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12102: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:4039:VPNfg:12102: add IPsec SA: SPIs=154cda8e/a2664923
ike 2:VPNfg:4039:VPNfg:12102: added IPsec SA: SPIs=154cda8e/a2664923
ike 2:VPNfg: HA send IKEv2 message ID update send/recv=15/1
ike 2:VPNfg:4039:VPNfg:12102: scheduling rekeyed SPI 154cda88 for deletion
ike 2:VPNfg:4039:VPNfg:12102: rekey in progress, old SPI 154cda88
ike 2: HA receive seqno update send/recv=14/1
ike 2:VPNfg:VPNfg: HA add IPsec SA 154cda8e/a2664923 seq 47868c01 1
ike 2:VPNfg:VPNfg: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 2:VPNfg:VPNfg: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 2: HA receive seqno update send/recv=15/1

Performing Rekey (CREATE_CHILD) - Responder

ike 0: comes X.X.X.X:500->Y.Y.Y.Y:500,ifindex=8....
ike 0: IKEv2 exchange=INFORMATIONAL id=45a1740c06c3cfc5/1680a67638fca00e len=112
ike 0:VPNfg2:175: received informational request
ike 0:VPNfg2:175: processing notify type MESSAGE_ID_SYNC
ike 0:VPNfg2:175: message ID sync request received nonce=1a2b0a55 expected send/recv msg IDs=14/1
ike 0:VPNfg2:175: current send/recv msg IDs=1/6 updated to new send/recv msg IDs=1/14
ike 0:VPNfg2:175: sent IKE msg (INFORMATIONAL_RESPONSE): Y.Y.Y.Y:500->X.X.X.X:500, len=112, id=45a1740c06c3cfc5/1680a67638fca00e
ike 0: comes X.X.X.X:500->Y.Y.Y.Y:500,ifindex=8....
ike 0: IKEv2 exchange=CREATE_CHILD id=45a1740c06c3cfc5/1680a67638fca00e:0000000e len=384
ike 0:VPNfg2:175: received create-child request
ike 0:VPNfg2:175: responder received CREATE_CHILD exchange
ike 0:VPNfg2:175: processing notify type REKEY_SA
ike 0:VPNfg2:175:VPNfg2:895: rekeying child SA SPI 154cda88
ike 0:VPNfg2:175: responder creating new child
ike 0:VPNfg2:175:898: peer proposal:
ike 0:VPNfg2:175:898: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:898: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:898: comparing selectors
ike 0:VPNfg2:175:VPNfg2:898: matched by rfc-rule-2
ike 0:VPNfg2:175:VPNfg2:898: phase2 matched by subset
ike 0:VPNfg2:175:VPNfg2:898: accepted proposal:
ike 0:VPNfg2:175:VPNfg2:898: TSi_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:898: TSr_0 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:898: autokey
ike 0:VPNfg2:175:VPNfg2:898: incoming child SA proposal:
ike 0:VPNfg2:175:VPNfg2:898: proposal id = 1:
ike 0:VPNfg2:175:VPNfg2:898:   protocol = ESP:
ike 0:VPNfg2:175:VPNfg2:898:      encapsulation = TUNNEL
ike 0:VPNfg2:175:VPNfg2:898:         type=ENCR, val=AES_CBC (key_len = 256)
ike 0:VPNfg2:175:VPNfg2:898:         type=INTEGR, val=SHA512
ike 0:VPNfg2:175:VPNfg2:898:         type=DH_GROUP, val=ECP521
ike 0:VPNfg2:175:VPNfg2:898:         type=ESN, val=NO
ike 0:VPNfg2:175:VPNfg2:898: matched proposal id 1
ike 0:VPNfg2:175:VPNfg2:898: proposal id = 1:
ike 0:VPNfg2:175:VPNfg2:898:   protocol = ESP:
ike 0:VPNfg2:175:VPNfg2:898:      encapsulation = TUNNEL
ike 0:VPNfg2:175:VPNfg2:898:         type=ENCR, val=AES_CBC (key_len = 256)
ike 0:VPNfg2:175:VPNfg2:898:         type=INTEGR, val=SHA512
ike 0:VPNfg2:175:VPNfg2:898:         type=DH_GROUP, val=ECP521
ike 0:VPNfg2:175:VPNfg2:898:         type=ESN, val=NO
ike 0:VPNfg2:175:VPNfg2:898: lifetime=600
ike 0:VPNfg2:175:VPNfg2:898: PFS enabled, group=21
ike 0:VPNfg2: schedule auto-negotiate
ike 0:VPNfg2:175:VPNfg2:898: set sa life soft seconds=580.
ike 0:VPNfg2:175:VPNfg2:898: set sa life hard seconds=600.
ike 0:VPNfg2:175:VPNfg2:898: IPsec SA selectors #src=1 #dst=1
ike 0:VPNfg2:175:VPNfg2:898: src 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:898: dst 0 7 0:0.0.0.0-255.255.255.255:0
ike 0:VPNfg2:175:VPNfg2:898: add IPsec SA: SPIs=a2664923/154cda8e
ike 0:VPNfg2:175:VPNfg2:898: added IPsec SA: SPIs=a2664923/154cda8e
ike 0:VPNfg2:175:VPNfg2:898: responder preparing CREATE_CHILD message
ike 0:VPNfg2:175: sent IKE msg (CREATE_CHILD_RESPONSE): Y.Y.Y.Y:500->X.X.X.X:500, len=368, id=45a1740c06c3cfc5/1680a67638fca00e:0000000e

Delete old IPsec SA after Rekey (CREATE_CHILD) - Initiator

ike 2:VPNfg:4039:VPNfg:12090: sending delete for IPsec SA SPI 154cda88
ike 2:VPNfg:4039:12104: send informational
ike 2:VPNfg:4039: sent IKE msg (INFORMATIONAL): X.X.X.X:500->Y.Y.Y.Y:500, len=96, id=45a1740c06c3cfc5/1680a67638fca00e:0000000f
ike 2: comes Y.Y.Y.Y:500->X.X.X.X:500,ifindex=59....
ike 2: IKEv2 exchange=INFORMATIONAL_RESPONSE id=45a1740c06c3cfc5/1680a67638fca00e:0000000f len=96
ike 2:VPNfg:4039: have HA IKE SA, attempting message ID sync
ike 2:VPNfg:4039: removing old active request before message ID sync

Delete old IPsec SA after Rekey (CREATE_CHILD) - Responder

ike 0: comes X.X.X.X:500->Y.Y.Y.Y:500,ifindex=8....
ike 0: IKEv2 exchange=INFORMATIONAL id=45a1740c06c3cfc5/1680a67638fca00e:0000000f len=96
ike 0:VPNfg2:175: received informational request
ike 0:VPNfg2:175: processing delete request (proto 3)
ike 0:VPNfg2:VPNfg2:898: send SA_DONE SPI 0x154cda8e
ike 0:VPNfg2: deleting IPsec SA with SPI 154cda88
ike 0:VPNfg2:VPNfg2: deleted IPsec SA with SPI 154cda88, SA count: 1
ike 0:VPNfg2:175: sending delete ack
ike 0:VPNfg2:175: sent IKE msg (INFORMATIONAL_RESPONSE): Y.Y.Y.Y:500->X.X.X.X:500, len=96, id=45a1740c06c3cfc5/1680a67638fca00e:0000000f

Direction from standalone FortiGate to FortiGate cluster

However, when the VPN tunnel is established to the cluster (as the Responder), there is a similar problem with the Rekey, as between the FortiGate and the Cisco Firepower. While the debug messages look different, the Rekey does not occur, but a new tunnel establishment instead.

Note: This was also resolved by disabling the VDOM Partitioning.