FortiGate 6.2.3 bugs, debug and support

I am now seriously upset with Fortinet. Fortinet definitely does not care about customers. It seems very bad to me that the Czech branch does not solve anything. I have experiences, for example, with Cisco, where the technicians are very helpful and escalate problems even abroad. I do not have so much doubt about the intelligence of the support (even though it is supposed to be a senior engineer) as a clear opinion, it is extremely low (unless it is an order to behave that way to deter customers). Allegedly, the Research and development team has been dealing with the problem for some time, where they must also be very limited. Then it is clear why FortiGate has more bugs than functionality. I sent them a clear explanation and evidence of the problem, but they are not able to understand it. What I absolutely do not understand is why they do not try the situation in their own environment!!! That is what every company does when solving a problem.

Note: On September 8, 2020, I added how I had previously solved the IPsec VPN.

Note: On September 11, 2020, I added the situation around the problem being solved.

Note: On October 7, 2020, I added information about the bug fix in FortiOS 6.2.6. And mentioned a new problem with HTTPS access in FortiOS 6.2.5.

Note: On December 2, 2020, I added information about the upgrade to FortiOS 6.2.6, which did not solve the problem, but worsened it (it no longer works even with Certificate Inspection).

Note: On February 1, 2021, I added information about the upgrade to FortiOS 6.2.7, which did not solve the problem.

Note: On February 12, 2021, I added information about how the resolution with Fortinet is continuing catastrophically and the initial opinion.

Note: On March 11, 2021, I added another problem. Non-functional HTTPS communication from Dynamics 365 to Exchange Web Services in Proxy Mode with a certificate.

Note: On April 15, 2021, I solved another problem with Site to Site IPsec VPN, I described everything in a new article. I have an open ticket on Support. The use of VDOM Partitioning (Virtual clustering) turned out to be the problem.

Note: On May 2, 2021, I added information about the upgrade to FortiOS 6.2.8, which finally solved the problem.

Virtual clustering with VDOM partitioning

A relatively cosmetic problem, but one that a person must not forget. If we operate an FGCP cluster and have divided VDOM into different nodes, then a number of information is not transferred to the second node and we have to connect to the one where the given VDOM Master is. This is the case with SD-WAN (statistics and monitoring), Site to Site IPsec VPN (often shown as Inactive on the second node, which is quite unpleasant), routing table display, debugging using CLI (and many CLI display commands).

SSL VPN and poor LDAP group

After upgrading to FortiOS 6.2.4, users in SSL VPN are often (but not always) incorrectly assigned to the LDAP group (FortiGate reads the correct one from LDAP, but passes a different one to VPN), so they can connect to the VPN, but cannot communicate there. So I immediately made a downgrade. It's a known bug, said to be fixed in FortiOS 6.2.5. Technical Tip: Incorrect portal mapping because of SSL VPN LDAP user matching incorrect group

Failure to connect to SSL VPN - FortiClient gets stuck at 98%

According to the discussions, a very well-known problem that has been going on since the beginning of Fortinet SSL VPN. Connecting to SSL VPN from FortiClient (6.0.x, 6.2.x, 6.4.x) gets stuck at 98% (either ends after a longer time or remains stuck) and the user does not connect. Sometimes it works on the x-th attempt or after restarting the application (interestingly, it's better to close and start the application, not restart the computer). Some people can't connect at all. It seems that anyone who has an internet connection through IPv6 won't connect (if IPv6 is disabled on the machine, it will work). Even a functional connection takes much longer compared to Cisco AnyConnect. In general, FortiClient error messages don't say anything (just like debug).

After a long time, it turned out that the problem is caused by the attempt to connect using DTLS. More information in FortiGate problems connecting to SSL VPN via FortiClient.

Session timeouts - Poll Active Directory Server

An absolutely crucial problem that appeared in about half of the people (perhaps even more, but they didn't notice it). After switching the FortiGate to production, many people started experiencing that after a certain time (sometimes a few seconds, sometimes up to tens of minutes), any session would drop. For example, with RDP, the connection is automatically reconnected (the screen flashes black, or there is information about reconnecting), but SSH is terminated. Similarly, connecting to Exchange will re-establish every time the connection drops, and the user may not notice the problem (but everything is slower and sometimes information is not displayed correctly). Work using SSH or DB connection is unusable because it is constantly being terminated and has to be re-established.

In the FortiGate Traffic log, these sessions showed Firewall Action: timeout. I tried adjusting the policies (disabling checks, hardware acceleration, etc.), increasing timeouts, looking for problems in routing and NAT. Then I debugged everything possible, output of session with a filter, Packet Sniffer and Debug Flow (the message no session matched and RST ACK from the destination server often appeared). Otherwise, there was nothing crucial to see.

Until I got the advice to disable authentication on the rule, I turned to a Fortinet specialist for advice, who saw the auth_server=Local FSSO Agent in the session output. I don't use authentication anywhere, but I had created an Active Directory Fabric Connector without an agent. Which I understood to be the method for identifying users in the logs. The identification worked very nicely, but also caused the interruption of many sessions. At the moment when I disabled the connectors, everything started working. I think that even if there was any problem in obtaining user information, it should not have caused the disruption of communication.

I plan to try Fortinet Single Sign-On (FSSO) with the help of agents (agent-based), to see if the problem was not in the remote querying of DC logs.

DNS error and IP connection error

When looking at the Traffic log, a DNS communication error often appears. The record has Action: DNS error and Firewall Action: dns. Occasionally there is Action: IP connection error and Firewall Action: ip-conn. This is always immediately followed by another record where the DNS communication passes correctly. It happens for the UDP protocol (occasional DNS queries over TCP always pass).

In the discussions, this situation is quite addressed, but there is probably no solution and it is often stated that it is not necessary to worry about this error. Mentioned is to delete the system session-helper 14 for dns-udp. This had no effect on me.

IPsec Site to Site VPN - certificate authentication and SHA-2

Note: IPsec VPN I worked on again, tried to study and wrote an article FortiGate IPsec VPN, debug and issues, which also describes a lot of debugging. There is also a problem where Phase 2 (IPsec) Rekey does not occur (I am solving it with Fortinet support). It was resolved by turning off VDOM Partitioning (Virtual clustering).

We needed to build a VPN tunnel with one customer who had strict security requirements. He specified all the parameters, which were:

• Phase 1 - Exchange mode IKEv2, authentication certificate, Encryption AES-256-GCM, Integrity HMAC-SHA2, Pseudo Random Function (PRF) SHA512, DH Group 21, SA Lifetime
• Phase 2 - Protocol ESP, Encryption AES-256-GCM, Integrity HMAC-SHA2, DH Group 21, Lifetime
• two subnets on the remote side, one on our side

I am not very familiar with this area, so I spent a little time solving the terminology as well. FortiGate uses Encryption and Authentication, which together form a Proposal. For AES-GCM-256, it states that the Encryption algorithm contains authentication. But for Phase 1, it is mandatory to use PRF. So for Phase 1 it was set to aes256gcm-prfsha512 and for Phase 2 aes256gcm.

Everything could be easily set up on the FortiGate GUI. But the tunnel did not establish. We can look at the Log & Report > Events - VPN Events, but these are only rough events. So we have to use the CLI. An important thing is, if we use VDOM and cluster, always connect to the Master Node (even though the tunnel subsequently worked, I almost always saw it down, but another tunnel was displayed correctly). Output of IPsec VPN information. Technical Tip: Troubleshooting IPsec VPNs

get vpn ipsec tunnel summary
diagnose vpn ike gateway list name NAME

And most importantly, enable IPsec VPN debugging for the target IP.

diagnose debug application ike -1
diagnose vpn ike log-filter dst-addr4 X.X.X.X
diagnose debug console timestamp enable

diagnose debug enable

Part of the debug with the error

2020-07-01 08:29:07.977165 ike 1:NAME:2924: certificate validation pending
2020-07-01 08:29:07.977755 ike 1:NAME:2924: certificate validation complete
2020-07-01 08:29:07.977763 ike 1:NAME:2924: certificate validation succeeded
2020-07-01 08:29:07.977899 ike 1:NAME:2924: signature verification failed

For this error, I only found some old discussions that the certificate needs to use SHA-1. I did some experiments and the following was functional:

• change the authentication from certificate to Pre-shared Key (PSK)
• in Phase 1, use any encryption, but authentication or PRF SHA1

Eventually, the technician on the other side came up with the fact that Cisco Firepower has a special command that allows SHA-512 for Phase 1, but uses SHA-1 for hashing the certificate response. The tunnel then connected with the required settings.

I also solved one thing that is more my lack of knowledge and I quickly got advice from a FortiGate specialist. On the other side there are two subnets, I was looking for how to specify them in Phase 2. Somewhere in the documentation I found that you need to create an Address Group where the individual subnets are placed. In the VPN configuration, the Named Address and the given group are selected. In the monitoring, I then saw both Phase 2 Selectors, but only one connection was established.

The solution was to configure it using the CLI and create two separate Phase 2 under the same Phase 1. Indicated in Technical Tip: IPSec VPN between a FortiGate and a Cisco ASA with multiple subnets

Non-functional HTTPS connection to certain websites in FortiOS 6.2.5

I upgraded from FortiOS 6.2.3 to FortiOS 6.2.5 and immediately encountered new problems. Access to some websites that previously worked stopped working. Firefox displays the error

Secure Connection Failed 
Error code: PR_END_OF_FILE_ERROR

It happens when using a policy in Proxy-based Inspection Mode, where only SSL Certificate Inspection is used together with any Security Profile (AV, Web Filter, Application Control,IPS). When the policy is switched to Flow-based Inspection Mode, the access works (but there is a problem with some other addresses).

It's similar to the known bug Proxy web filter with SSL inspection may fail for websites that allow TLS versions below 1.3 after upgrading to FortiOS 6.2.5, but it's not just tied to Web Filtering and I don't know exactly what they mean by SSL inspection.

Non-functional HTTPS communication from Dynamics 365 to Exchange Web Services in Proxy Mode

I solved a strange problem for a very long time, where communication from the cloud service Microsoft Dynamics 365 (CRM) to the internal Exchange 2016 Server did not work. More precisely, its Exchange Web Services (EWS) service, which is used to access user mailboxes using a service account with the Application Impersonation role. The access returned the error

Error : System.Net.WebException: The remote server returned an error: (401) Unauthorized.

The same problem was also returned with direct user authentication (without Impersonation). But if the Connection Test was run from Dynamics 365, it went through fine (including EWS access to the mailbox). The Test and allow mailbox did not work.

Another thing is that all tests (including Impersonation) from Microsoft Remote Connectivity Analyzer passed without error. In addition, other services also use EWS and do not have any problems. So for a long time we assumed the problem was in Dynamics 365. Later, we tried connecting to the test Exchange and that went through without errors.

I examined the Application Impersonation setting, which is often discussed in the discussions. I analyzed the logs on the FortiGate and tested disabling various checks. I detailed the Exchange logs for EWS and IIS. Many connections passed through the FortiGate during the test. A number of records were also visible in the IIS log, always with the 401 Unauthorized code. However, in the EWS log there was always only one record. It had the error

ErrorImpersonateUserDenied - The account does not have permission to impersonate the requested user

For the tests that worked, there was a similar record, the same authenticated user (but written in a different format) and the Impersonated user. Of course, without error. For example, in the IIS log, you can search well using UserAgent = CRM/9.0.0.0/Live.

FortiGate - Proxy Mode and SSL Inspection

Eventually I found that the problem is in a certain FortiGate setting. I tested that this behavior occurs in both FortiOS 6.2.7 and FortiOS 6.4.5. If the policy is in Proxy Mode and at the same time a certificate on the FortiGate is used, let's say SSL Inspection, the connection does not work. As soon as one of these conditions is not met, the connection is established and everything works without errors.

More detailed description of the non-functional setting. If we have multiple Exchange servers and want to use Server Load Balancing HTTPS connection on the FortiGate using a Virtual Server. Then we have to use a certificate in the Virtual Server and the policy must be in the Proxy-based inspection mode. There is probably no way to configure Load Balancing on multiple servers so that the connection from Dynamics 365 works.

The solution is to create a new publication of only one Exchange server (maybe only for Dynamics 365, although I also found that the Internet Service Database on the FortiGate contains the Microsoft-Dynamics item with many IP addresses, but none of them are the one that Dynamics 365 connects to our Exchange from). We create Destination NAT using a Virtual IP, and if we don't use SSL inspection, the policy can be in Proxy Mode or Flow Mode. But we will probably be bothered that the Firewall does not check the traffic. So we can use SSL Inspection of the Protecting SSL Server type, where we specify the server certificate. The policy must then be in Flow Mode for the communication to work.

Or information on configuring FortiGate Firewall policies, NAT, Load Balancing, Debug.

I don't have the energy to look for what differs in the communication that works and what doesn't, with different FortiGate settings. And certainly not to try to solve it with Fortinet Support, that would probably be a waste of time. But unfortunately another important thing where FortiGate has failed, and it cost me a lot of time.

Non-functional HTTPS connection from Java version 11 and higher with SSL inspection in Proxy Mode

So far the biggest problem that has not yet been resolved and is greatly limiting us. I even opened a ticket with Fortinet Support, although I don't have very good experiences with the support of large companies, so I try to solve everything myself. But here I am convinced that it is a FortiOS bug. How bad the Fortinet support is, I will describe further.

Problem description and TLS 1.3

I have an Internet Access Firewall Policy that is in Proxy-based Inspection Mode and uses SSL Deep Inspection along with AV, Web Filter, Application Control and IPS. Access to websites works well from all web browsers (of course after solving the CA certificate and exceptions for certain websites). But we also use many Java applications, whether our own or various development tools, etc. If Java 8 is used, everything is fine. But today a lot of people are switching to Java 11 and from there you can't connect to most HTTPS sites. The connection ends with a handshake failure.

At first, I tried various experiments and debugged the connection from Java (the developers wrote a small application for me to do that). I tried different versions of OpenJDK 11 and 14 (Adopt Open JDK, Amazon Corretto) as well as Oracle Java. The same behavior everywhere.

The debug log showed that the problem occurs when TLS 1.3 is tried to be established. Java 8 does not support TLS 1.3 and tries TLS 1.2, which goes through without any problems. Similarly, if TLS 1.2 is forced to be used in Java 11 at startup, with the parameter -Dhttps.protocols="TLSv1.2", or TLS 1.3 is disabled in the configuration, the connection over TLS 1.2 goes through.

Disabling the TLS 1.3 protocol is done in the configuration file java.security by adding to the item (here is only the beginning)

jdk.tls.disabledAlgorithms=TLSv1.3, SSLv3, RC4

Interestingly, browsers connect using TLS 1.3. So I don't think the problem is in this protocol or the TLS 1.3 support on the FortiGate.

Attempts to configure on FortiGate

I tried various settings on the FortiGate and the results are below.

• when SSL Deep inspection is switched to Certificate Inspection, the connection works
• if I leave SSL Deep inspection, but disable all other Security Profiles, the connection works, as soon as I enable any other profile, it doesn't work (apparently the Deep inspection is not performed)
• if I switch the policy from Proxy-based to Flow-based, the connection starts to work, I thought I would solve it this way, but another serious problem appeared (I'll describe it further)
• if an exception from SSL inspection is set for an address, the connection works

Tested addresses

At first I had the feeling that access to no website was working, but I found that some were. Examples of functional websites and the protocol and cipher chosen by the browser.

• https://www.google.com/ TLS 1.3 TLS_AES_128_GCM_SHA256
• https://www.vinodoc.cz TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
• https://apache.org/ TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

There are many more that do not work. For example:

• https://www.seznam.cz/ TLS 1.3 TLS_AES_256_GCM_SHA384
• https://repo1.maven.org TLS 1.2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
• https://repo.maven.apache.org/ TLS 1.2 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
• https://www.microsoft.com TLS 1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
• https://www.samuraj-cz.com/ TLS 1.3 TLS_AES_256_GCM_SHA384
• https://www.apple.com/ TLS 1.3 TLS_AES_256_GCM_SHA384

Debugging on the web server

Comparing functional and non-functional communication (it can be seen that on some servers the same parameters work and on others they don't) led me to one piece of information that I verified on the test Linux web server. If OpenSSL 1.0.x (which does not support TLS 1.3) is used, the connection works. If OpenSSL 1.1.1 is used, it doesn't work, even if TLS 1.3 is not allowed in the server configuration.

When debugging on the server was enabled, more informative information was also displayed (I tried this after the debugging on the FortiGate described later).

fd[000e] OpenSSL error[0x14201076] tls_choose_sigalg: no suitable signature algorithm

I tried to perform various debugging on the FortiGate. The individual things are described approximately in the following chapters.

FortiGate (FortiAnalyzer) logs

In the FortiGate Traffic log there is Action: Accept and Firewall Action: server-rst. In the SSL inspection log nothing. In the Event log (All Types), an event WAD SSL Fatal Alert received, handshake failure can be found for the destination IP.

Session output

It doesn't show anything interesting.

diagnose sys session filter src 10.0.100.10 
diagnose sys session filter dst 192.168.50.45 
diagnose sys session list
session info: proto=6 proto_state=56 duration=2 expire=2 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0
 use=5
...

Packet Sniffer

diagnose sniffer packet Local 'host 10.0.100.10 and host 192.168.50.45' 4 100

interfaces=[Local]
filters=[host 10.0.100.10 and host 192.168.50.45]
24.710746 Local -- 10.0.100.10.51369 -> 192.168.50.45.443: syn 2544371997 
24.713290 Local -- 192.168.50.45.443 -> 10.0.100.10.51369: syn 3361056785 ack 2544371998 
24.713921 Local -- 10.0.100.10.51369 -> 192.168.50.45.443: ack 3361056786 
24.771173 Local -- 10.0.100.10.51369 -> 192.168.50.45.443: psh 2544371998 ack 3361056786 
24.771303 Local -- 192.168.50.45.443 -> 10.0.100.10.51369: ack 2544372422 
24.773966 Local -- 192.168.50.45.443 -> 10.0.100.10.51369: fin 3361056786 ack 2544372422 
24.774212 Local -- 10.0.100.10.51369 -> 192.168.50.45.443: ack 3361056787 
24.777604 Local -- 10.0.100.10.51369 -> 192.168.50.45.443: psh 2544372422 ack 3361056787 
24.777648 Local -- 192.168.50.45.443 -> 10.0.100.10.51369: rst 3361056787

The Packet Sniffer doesn't help much either. The person from support ran it extended (they said they can convert it to Wireshark).

diagnose sniffer packet any 'host 192.168.50.45' 6 0 1

Debug Flow

diagnose debug flow filter proto 6
diagnose debug flow filter saddr 10.0.100.10
diagnose debug flow filter daddr 192.168.50.45
diagnose debug flow show function-name enable
diagnose debug console timestamp enable
diagnose debug flow trace start 100
diagnose debug enable

Debug WAD

The WAD daemon is used for SSL inspection in Proxy Mode. For Flow Mode, it's the IPS engine. For debugging, the verbose level needs to be enabled (which I didn't do at first). Then some information is displayed.

diagnose wad filter dst 192.168.50.45
diagnose wad debug enable category ssl
diag wad debug enable level verbose
diagnose debug enable

Just a few lines from the output.

...
SESSION-ID: d4f4e378b0c9b6e00d1ebf5c0c9106d4ae2021021e6c719f80d354667f41aa75
wad_ssl_proxy_find_session(2667): can't find the session!
__wad_ssl_proxy_filter_ciphers(7560): unsupported num = 12, supported num = 32, intersection num = 32
wad_ssl_proxy_filter_ciphers(7609): wsp(0x7f49d461a880/6) cipher filter: ns=12, is=32
wad_ssl_proxy_srv_continue_fwd_client_hello(9236): sp=0x7f49d461a880/6 start handshake with server. changed_ch=1
wad_ssl_proxy_clt_caps_find_session(12611): sp=0x7f49d461a510/7 find session from (nil)
...
wad_ssl_port_caps_on_alert_recv(12368): fts recv alert level=2 desc=handshake failure
...

Packet Capture

Capturing the communication ultimately showed the most and likely identifies the problem. First, I captured the communication with Wireshark on the client station. But I verified that using Network > Packet Capture in the FortiGate GUI, for the interface to the network where the client is, the result is the same and the capture is convenient. Subsequently, I analyzed the traffic using Wireshark. I must admit that it was only when the person from support guided me to capture the communication on the input (internal network) and output (internet) interface. He kept trying to set the interface to any, when he couldn't do it in the GUI, he used the Packet Sniffer in the CLI.

The following image shows the communication from the client to the FortiGate.

The next one shows how it looks from the FortiGate to the server.

Communication analysis

It can be seen that the target server receives a Client Hello to negotiate the TLS communication and immediately responds with a Handshake Failure. So I compared how the Client Hello sent by Java looks, and how the one that goes out from the FortiGate looks. It establishes communication to the target server on behalf of the client and apparently based on the allowed ciphers in its configuration. I thought the FortiGate would always create a Client Hello the same way (according to the configuration), but based on the tests, it is seen that it is strongly reflected in the parameters that come from the client.

I'm not very familiar with the TLS protocol and a few things surprise me. For example, the mention of the TLS version, where Wireshark sometimes marks the Client Hello packet as TLS 1.2 Record Layer and elsewhere (in my opinion, the same parameters) TLS 1.3 Record Layer. Then there are several times Version: TLS 1.2 (and for example from the browser TLS 1.0) and finally TLS 1.3 is established (of course, the list of offered protocols is mentioned in supported_versions).

The comparison showed that Java offers 45 Cipher Suites, the FortiGate removes several of them and offers 33 Cipher Suites. But all the main and supported ciphers on the server are here, including the one that is used without SSL inspection. Some extensions (extension) also differ a bit. According to the error displayed during debugging on the server, the main two are:

• signature_algorithms
• signature_algorithms_cert

The signature_algorithms extension contains the same 16 Signature Hash Algorithms from Java and from the FortiGate. The signature_algorithms_cert extension from Java contains the same 16 algorithms, but from FortiGate there are only 9 of them. (it doesn't offer the RSASSA-PKCS1-v1_5 algorithms and Legacy algorithms). The comparison is in the following images.

I looked at the communication in various cases where the connection is established. I focused on the signature_algorithms_cert extension.

• the only page with TLS 1.3 that works for me from Java 11 is google.com, interestingly, in this single case the FortiGate does not use this extension
• when TLS 1.3 is disabled in Java 11, Java still inserts this extension, but it's not in the packet from the FortiGate
• from browsers the connection always works, it seems they don't insert this extension
• to some pages with TLS 1.2 you can connect from Java 11 (e.g. where OpenSSL 1.0.x is), in the packet from the FortiGate the extension is present, but according to information on the internet some servers don't use it (they only work with signature_algorithms)
• when SSL inspection is disabled or the policy is in Flow Mode, it looks like the original Client Hello created by Java (and the extension has all 16 algorithms and the connection is established)

I looked for some documentation, interesting for example The Evolution of Signatures in TLS and Signature algorithms. It seems that the signature_algorithms extension was added in TLS 1.2. And signature_algorithms_cert not until TLS 1.3 (older versions of TLS 1.3 don't even mention it), although some TLS 1.2 implementations seem to use it as well.

I couldn't find how to define allowed algorithms in Java 11 or on the web server, unlike determining the allowed Cipher Suites. Nor to find out which of the offered algorithms was used on the server.

Conclusion

From the above, I am convinced that there is a bug in FortiOS. It handles the TLS message extension signature_algorithms_cert incorrectly. If it allows the given algorithms for signature_algorithms, it should also allow them for signature_algorithms_cert. I'm very surprised that no other customer, especially large companies, has encountered this problem yet. I keep assuming what the representatives of all firewall manufacturers convinced me of, that SSL inspection is really widely used today. Then Java applications are absolutely common and the transition to version 11 is recommended today.

Continuing the resolution and testing

I contacted the Czech Fortinet, where a technician immediately responded to me and willingly helped me, even though he said he had no influence on the support. Thanks to him, I got an Eval license for FortiGate-VM. So I did tests on a freshly installed FortiOS 6.2.3 with a minimal configuration. The problem was there. I tried gradual upgrades and finally a clean installation of FortiOS 6.4.2 as well. The described problem occurs everywhere.

Note: Incidentally. I don't understand why in FortiOS 6.4 FortiView and Monitor were moved to Dashboard. Moreover, the default items in my Dashboard look different than in all the documentation images.

During that time, I received a message from the support that the development team started to address the problem. After a few days, they said they found that the behavior corresponds to a known issue with ID #658654. The developers are currently working on fixing this issue. I'm supposed to get information about the progress.

FortiOS 6.2.6 and the unresolved bug

According to the Fortinet Support, the bug should have been fixed in FortiOS 6.2.6. Also the FortiOS 6.2.6 Release Notes lists the problem with ID 658654 (Cannot access the specific website using proxy-based UTM with certification inspection.) as resolved, although from the description I wouldn't know that this is my problem.

I upgraded the cluster to FortiOS 6.2.6 and quick tests immediately showed that the problem still persists. I performed a number of experiments and the behavior is completely identical to before. What's worse, previously when an SSL Inspection exception was set (or the policy had only Certificate Inspection instead of SSL Deep inspection), the communication worked. Now the same problem occurs. Everything only works in Flow-based mode or when no Security Profile is used.

I opened a new ticket with Fortinet Support.

Access to websites with SSL inspection in Flow Mode

While testing the problems with Java 11, I found that if the policy is in Flow-based inspection mode, the connection negotiation works. So I switched the policy for the entire company and after a few hours, many people told me they were having problems with HTTPS communication to the internet. In the end, it was a few of the same addresses. When I tested them from the browser, at first everything worked for me, but I found that at a certain time the page really didn't load. But if I try to reload it in the browser, it usually displays. Then the given address works for a while. Where we ran into the problem:

• https://www.linkedin.com/ - I accidentally found that LinkedIn doesn't load for me, the page times out the first time
• https://epodani.cssz.cz/ - a very important address for us, which is called from an application (so you can't do a new reload), the first time it fails to negotiate SSL, returning SSL_ERROR_BAD_HANDSHAKE_HASH_VALUE
• our Java application - works as a client (currently with Java 8) server, during the work it started to repeatedly pop up the Read time out error (SocketTimeoutException)

Since the errors occur occasionally, they are very difficult to debug. I let Flow Mode enabled for a group of users (who, however, do not use the two problematic applications mentioned). They don't report any problems to me. But already when I was looking for information on the problem with Java 11, I found many discussions where people were solving the problem with access to certain web addresses. Sometimes even just from a certain browser. The solutions they mention are completely different, often a return to some previous version of FortiOS. Various switching to Proxy mode or Flow mode (obviously something different works for everyone). Someone supposedly TAC advised to block TLS 1.3 on the FortiGate.

My experience with Fortinet Support

As I wrote, I don't have very good experiences with the support of large companies. In the past, I solved something with Symantec, Microsoft, Cisco. I prefer to try to solve the problem myself. But here I think it's a system bug. Fortinet's support is even worse than what I've experienced so far.

Right away when creating the ticket, I described everything, when the problem occurs and when it doesn't, I attached the debugs, logs, and traffic capture. I didn't want to give our configuration, but of course they immediately requested it in the first message. So I sent it. They had everything to try this problem themselves. I think it's a completely general problem, so they wouldn't even have to try our configuration, just try the connection from Java 11. That hasn't happened yet.

For the first 14 days, nothing happened. Then a foreign engineer contacted me and a remote connection was arranged. I'm convinced he didn't read my problem description and didn't even look at the materials I sent. He asked about everything, the debugs I sent, he created them again himself. Then every time he said he had to go through it, or wanted to send something else, and another connection was arranged. And during that connection, the same again. Only then did he start looking at the materials I had sent. He wasn't prepared for anything in advance.

At that point, I was already convinced that the problem is in the Client Hello and the TLS extension signature_algorithms_cert. I described it all to him in detail, just like above. But it seems to me that he ignores it and doesn't listen to me. He tries, for example, to attribute the problem to our application (he doesn't understand the argument that you can't connect to a Microsoft page from general Java). He keeps asking me to send the captured communication again. And he tries to look for something in the CLI (it seems to me that he doesn't know it very well).

And the simplest thing, to try the problem somewhere in the lab, he probably can't do that. After a month, it started being addressed with the development team and everything moved forward.

Note: I found out quite late that emails are not coming in properly and you need to open them on the web. Not only are the line endings missing, but sometimes a piece of the message is missing (apparently some character in their system causes the message to be terminated and sent).

Timeline

• August 4, 2020 ticket creation
• August 4, 2020 very soon a person with a Czech name responded to me, requested the configuration and network diagram, and advised to upgrade to 6.2.4
• August 4, 2020 I sent him everything, wrote that 6.2.4 is unusable (VPN doesn't work) and asked a question, he didn't respond again
• August 10, 2020 automatic message that the ticket was escalated to a senior engineer
• August 16, 2020 he planned a remote session with me
• August 18, 2020 first session took place
• August 25, 2020 second session took place
• August 31, 2020 third session took place
• September 8, 2020 I received a request from the development team to capture communication on the client, server and FortiGate at the same time
• September 10, 2020 a message came that the R&D team found that the behavior corresponds to a known issue with ID #658654 and they are working on a fix
• September 27, 2020 I upgraded to FortiOS 6.2.5
• October 6, 2020 a message came that the developer team fixed the bug and it will be part of FortiOS 6.2.6 planned for November 3, 2020
• November 30, 2020 I upgraded to FortiOS 6.2.6 (released sometime in mid-November), the problem still persists, it even appears now when we set an exception from SSL Deep Inspection
• January 31, 2021 I upgraded to FortiOS 6.2.7 based on a request from support (from January 25, 2021), who wrote that although they are still working on the problem, since a new version is available, they require testing. The problem is still the same.
• February 9, 2021 a message came that really upset me. After 5 months of solving, when they had already acknowledged the bug twice and once claimed to have fixed it, they went back to the very beginning. They sent questions that we had solved at the beginning, so that they would even deal with it. They keep claiming again that the problem is on our server. They don't understand at all what I keep telling them, that the problem is with large servers like Microsoft, Apple or Apache. Even though I proved to them that the problem is with the signature_algorithms_cert extension, they are looking for the problem in the supported Cipher Suites. So I described the behavior in different situations and captured the communication before the FortiGate and how it modifies it. It is clearly visible there that in Proxy Mode (under certain conditions) the FortiGate modifies the Client Hello in such a way that the servers reject it. And from FortiOS 6.2.6 it happens even with Certificate Inspection. So I'm curious what the continuation will be.
• April 27, 2021 a message came that FortiOS 6.2.8 was released that day, where the bug is fixed, and that I have 5 days to test it, then the ticket will be closed (funny). On May 1, I deployed the new version, and it seems they really fixed the bug. BTW I started watching the RSS feed https://pub.kb.fortinet.com/rss/firmware.xml, where information about new versions is posted.