In this article, we want to point out some indicators used to determine if your FortiGate is under attack.
Indicator 1: Session counter
Session Counter “Active Sessions”
The session counter is a tool that tracks the number of active sessions on the FortiGate. It can be an indicator of potential attacks in several ways:
- Unusually High Number of Sessions: If the firewall detects an abnormally high number of sessions over a short period, it could indicate a Denial-of-Service (DoS) or Distributed Denial-of-Service (DDoS) attack, where attackers attempt to overwhelm the network by flooding it with traffic.
- Unexpected Session Patterns: Firewalls can also analyze the pattern of sessions. A sudden surge in sessions from a particular source or to a specific destination might suggest a targeted attack. Also, an atypical amount of inbound local sessions may be a hint for an ongoing attack.
- Session Duration and Frequency: Short-lived sessions occurring at a high frequency could be a sign of a scanning attack, where an attacker probes the network to find vulnerabilities.
- Geographical Irregularities: Sessions originating from geographical locations that are not typical for the network’s user base might indicate a compromised account or an external attacker.
- Protocol Anomalies: A spike in sessions using unusual protocols or ports could suggest an attacker is trying to exploit vulnerabilities or bypass security controls.
fgt01 # get system performance status
$CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 3806668k total, 1643628k used (43.2%), 1859712k free (48.9%), 303328k freeable (7.9%)
Average network usage: 688 / 202 kbps in 1 minute, 882 / 405 kbps in 10 minutes, 773 / 296 kbps in 30 minutes
Maximal network usage: 956 / 398 kbps in 1 minute, 23293 / 22778 kbps in 10 minutes, 23293 / 22778 kbps in 30 minutes
Average sessions: 1496 sessions in 1 minute, 1549 sessions in 10 minutes, 1518 sessions in 30 minutes
Maximal sessions: 1523 sessions in 1 minute, 1862 sessions in 10 minutes, 1862 sessions in 30 minutes
Average session setup rate: 1 sessions per second in last 1 minute, 1 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Maximal session setup rate: 6 sessions per second in last 1 minute, 41 sessions per second in last 10 minutes, 41 sessions per second in last 30 minutes
Average NPU sessions: 143 sessions in last 1 minute, 99 sessions in last 10 minutes, 84 sessions in last 30 minutes
Maximal NPU sessions: 146 sessions in last 1 minute, 157 sessions in last 10 minutes, 190 sessions in last 30 minutes
Average nTurbo sessions: 141 sessions in last 1 minute, 81 sessions in last 10 minutes, 71 sessions in last 30 minutes
Maximal nTurbo sessions: 144 sessions in last 1 minute, 155 sessions in last 10 minutes, 155 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 12 hours, 0 minutes
fgt01 # get sys session status
The total number of IPv4 sessions for the current VDOM: 1502
fgt01 # diag sys session stat
misc info: session_count=1651 setup_rate=4 exp_count=1 clash=2
memory_tension_drop=0 ephemeral=0/245760 removeable=0 extreme_low_mem=0
npu_session_count=112
nturbo_session_count=57
delete=42, flush=205, dev_down=18/227 ses_walkers=0
TCP sessions:
57 in ESTABLISHED state
1 in TIME_WAIT state
1 in CLOSE state
4 in CLOSE_WAIT state
firewall error stat:
error1=00000000
error2=00000000
error3=00000000
error4=00000000
tt=00000000
cont=0002ba75
ips_recv=00348791
policy_deny=001cf58b
av_recv=0004b4c6
fqdn_count=00000009
fqdn6_count=00000000
global: ses_limit=0 ses6_limit=0 rt_limit=0 rt6_limit=0
Session Counter “Ephemeral Sessions”
Ephemeral sessions on a FortiGate firewall refer to temporary sessions that are typically in an early stage of establishment. They are characterized by:
- TCP sessions that are in the process of being established, but not yet fully established. (incomplete handshake)
- UDP sessions that have only received a single packet.
These sessions are considered ephemeral because they are not fully established and may not last long. The FortiGate firewall manages these sessions carefully to prevent potential Denial-of-Service (DoS) attacks, which often involve creating a large number of open sessions to exhaust system resources. To mitigate such attacks, FortiGate sets limits on the number of ephemeral sessions that can exist at once, ensuring that memory is not depleted.
Session Counter “Session Clashes”
A session clash on a FortiGate firewall occurs when a new session is created but a conflicting similar session already exists. This can happen when two sessions have the same source and destination IPs and ports. When a session clash is detected, the FortiGate firewall will close the old session and replace it with the new one. The primary consequence of a session clash is that it may cause some retransmissions of network packets.
Session clashes are logged by the firewall, and administrators can monitor these events. The logs will provide details about the sessions involved in the clash, including their states, tuple numbers, policy IDs, and the actions taken by the FortiGate.
Indicator 2: Conserve Mode
If the memory usage on a FortiGate is very high, the FortiGate goes into the so called “conserve mode”. The conserve mode protects memory ressources with different measures to prevent daemons (services) from crashing and the system from becoming instable.
We have already written a CPU and Memory Troubleshooting guide where the conserve mode is described in detail. Also, the troubleshooting steps are descibed in this post.
Indicator 3: Memory Tension Drops
What most administrators are not aware about is, that the FortiGate also has another mechanism to prevent instability caused by high memory load: Memory tension drops. This mechanism has nothing to do with the conserve mode. As soon as the kernel is not able to allocate anymore memory pages, it removes the oldest sessions in the session table.
There is a counter telling us, if any sessions have been dropped by the “memory tension mechanism”:
fgt01 # diag sys session stat
misc info: session_count=75 setup_rate=3 exp_count=0 clash=0
memory_tension_drop=0 ephemeral=0/126976 removeable=0 extreme_low_mem=0
npu_session_count=21
nturbo_session_count=21
delete=10, flush=13, dev_down=274/41 ses_walkers=0
TCP sessions:
26 in ESTABLISHED state
1 in TIME_WAIT state
firewall error stat:
error1=00000000
error2=00000000
error3=00000000
error4=00000000
tt=00000000
cont=00000000
ips_recv=000947dd
policy_deny=001703c4
av_recv=00000000
fqdn_count=00000012
fqdn6_count=00000000
global: ses_limit=0 ses6_limit=0 rt_limit=0 rt6_limit=0
If this counter is not zero, it’s recommended that you find out why.
Indicator 4: Interface Drop Counter
The interface drop counter is an incremental number that rises everytime when a frame is being discarded. On the FortiGate, the most often reason for dropped frames are traffic shaper policies. Those can come into action, when an attacker causes huge amount of traffic over the FortiGate.
fgt01 # fnsysctl ifconfig
fortilink Link encap:Ethernet HWaddr 00:11:22:33:44:55
inet addr:10.10.10.1 Bcast:10.10.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:422251437 errors:0 dropped:0 overruns:0 frame:0
TX packets:189665747 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:248481066274 (231.4 GB) TX bytes:112066092902 (104.4 GB)
loopback0 Link encap:Unknown
inet addr:127.0.0.1 Bcast:0.0.0.0 Mask:255.0.0.0
inet addr6: 1:2:3:::1250 prefixlen 128
UP BROADCAST LOOPBACK RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0 Bytes) TX bytes:304 (304 Bytes)
wan2 Link encap:Ethernet HWaddr 00:11:22:33:44:54
inet addr:192.0.2.1 Bcast:192.0.2.255 Mask:255.255.255.0
link-local6: 1:2:3:::2115 prefixlen 64
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:92739287 errors:0 dropped:0 overruns:0 frame:0
TX packets:70430473 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:74709379581 (69.6 GB) TX bytes:15983161340 (14.9 GB)
fgt01 # diagnose netlink device list
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
fortilink: 248481221743 422251683 0 0 0 0 0 11490437 112066117391 189665835 0 0 0 0 0 0
loopback0: 0 0 0 0 0 0 0 0 304 4 0 0 0 0 0 0
wan2: 74709382279 92739326 0 0 0 0 0 28154 15983176891 70430527 0 0 0 0 0 0
Reasons for rising drop counters can be:
- Overload on the network interface. Packets that exceed the interface speed are dropped. This is also affecting some FortiGate models where the Internal Switch Fabric (ISF) buffer size may be larger than the buffer size of an NP6 XAUI port that receives traffic from the ISF. If this happens, burst traffic from the ISF may exceed the capacity of an XAUI port and egress or EHP sessions (EHP sessions on a FortiGate refer to Expectation Helper Protocol sessions. These are pseudo-sessions created by session helpers for ports negotiated by upper-layer protocols.) may be dropped during traffic bursts. You can find more information to this topic under this link.
- Traffic Shaping is dropping packets that exceed a limited bandwith. This behaviour is documented in this KB article from Fortinet.
- Access Control Lists (ACLs) use NP6 or NP6XLite offloading to drop IPv4 or IPv6 packets at the physical network interface before the packets are analyzed by the CPU. On a busy appliance this can really help the performance. This feature is available on FortiGates with NP6 or NP6XLite processors and is not supported by FortiGates with NP6Lite processors.
- Link Aggregation Groups (LAGs): In some cases, a FortiGate with NP6 processors may experience dropped egress or EHP (EHP sessions on a FortiGate refer to Expectation Helper Protocol sessions. These are pseudo-sessions created by session helpers for ports negotiated by upper-layer protocols.) packets on LAG interfaces. The dropped packets may be caused by the default algorithm used to select the egress path for packets on LAG interfaces. In some cases, this algorithm can cause fast path congestion.
Indicator 5: Malformed or Invalid Packets
Malformed or invalid packets on a FortiGate can indicate an attack because these packets often do not conform to standard network protocols and may be designed to exploit vulnerabilities in a system. FortiGate devices use an Intrusion Prevention System (IPS) to analyze network traffic. The IPS engine uses signature databases to detect known attacks and protocol decoders to identify network errors and protocol anomalies.
When a packet is malformed or invalid, it might be an intentional attempt to:
- Trigger a denial-of-service (DoS) condition by overwhelming the system with traffic that it cannot properly process.
- Exploit specific vulnerabilities in applications or services, such as the MySQL authentication protocol, where improper handling of malformed packets can lead to a server crash.
- Bypass security measures by using packets that are crafted in a way that is unexpected by normal security systems.
The IPS engine on FortiGate is designed to detect such anomalies and take appropriate actions to protect the network. It’s part of the broader Fortinet Security Fabric, which aims to provide comprehensive cybersecurity protection across all network edges. Malformed packets are a common technique used in various types of cyber attacks, and their detection is crucial for maintaining network security.
You can enable logging of invalid packets on the FortiGate over the CLI:
config log setting
set log-invalid-packet enable
end
There is a documentation with details about this log setting under: https://docs.fortinet.com/document/fortigate/6.0.0/handbook/442340/log-invalid-packet
It’s also possible to enable “strict header checking” on the FortiGate. More details can be found at: https://docs.fortinet.com/document/fortigate/7.4.3/hardware-acceleration/39956/strict-protocol-header-checking-disables-hardware-acceleration
Indicator 6: Portscans
An admin should block port scan attempts for several important reasons:
Security: Port scans are often the first step in an attack. They allow attackers to discover open ports and identify potential vulnerabilities to exploit. Blocking port scans proactively prevents attackers from gaining the information they need to plan more sophisticated attacks.
Privacy: By detecting services and their versions, attackers can gather sensitive information about the network infrastructure.
Resource Protection: Frequent port scans can consume network and system resources, potentially leading to performance issues.
Compliance: Many security standards and regulations require that measures be taken to detect and prevent port scanning activities.
By blocking port scans, admins can maintain the integrity and security of their networks, ensuring that potential attackers cannot easily map out the network’s structure and plan their next move.
Portscans can be detected and blocked on a FortiGate firewall through an anomaly detection policy. Here’s how it can be done: Use the DoS policy to detect port scans by configuring it to monitor the number of sessions (both complete and incomplete) from a single source IP address. If the number of sessions exceeds a predefined threshold, the sensor will trigger an action that the admin can configure.
By implementing DoS policies, a FortiGate firewall can effectively detect and block port scanning attempts, thereby protecting the network from potential vulnerabilities that could be exploited by attackers.
Indicator 7: IPS matches
Rate-Based signatures
When an IPS sensor match traffic patterns, a known attack was found. Fortunately this attack was already known and an IPS signature is in place so the FortiGate is able to block or at least monitor (log) it.
Some of the signatures are rate-based signatures. Rate based signatures count a match and are able to take action only after a specified number of matches. An example herefore would be a failed FTP login attempt. If a FTP login fails once or twice and is successful afterwards, it is not harmful or intrusive in most cases. But if a FTP login fail happens for 10 times in 10 seconds, this may be a login guessing attack.
Attack Patterns matching
When testing a system for vulnerabilities, an attacker may try to exploit one vulnerability after another. IPS patterns can identify these attacks and prevent any harm on the target system. This behaviour can reveal an attacker if it’s noticed and identified as such.
Indicator 8: High Bandwith
High bandwidth throughput on a FortiGate device can indicate an attack because it may suggest that there is a significant amount of traffic being processed, which could be abnormal. Here are some reasons why high bandwidth throughput might signal an attack:
- Denial of Service (DoS) or Distributed Denial of Service (DDoS) Attacks: These attacks flood the network with excessive traffic, which can lead to high bandwidth consumption as the FortiGate attempts to process the incoming data.
- Traffic Flooding: Similar to DDoS, other forms of traffic flooding can cause a surge in bandwidth usage as the FortiGate processes more data than usual.
- Malware Activity: Certain types of malware can generate a lot of network traffic, either from the infected host or to the infected host, leading to unusual spikes in bandwidth usage.
- Data Exfiltration: An attacker might be attempting to extract large volumes of data from within the network, which would show up as a high bandwidth throughput on the FortiGate device.
It’s important to monitor and analyze the traffic patterns using tools like FortiAnalyzer to determine the cause of high bandwidth usage. If the traffic is unexpected and cannot be attributed to normal network operations, it could be indicative of malicious activity, and appropriate security measures should be taken to investigate and mitigate the potential attack.
Indicator 9: High System Load
A high system load on a FortiGate device can indicate an attack because it may be a sign that the device is processing an unusually large volume of traffic or operations. This could be due to a Denial of Service (DoS) attack, where an attacker attempts to overwhelm the network or system with traffic, causing legitimate requests to be denied or delayed.
Here are some reasons why high system load might indicate an attack:
- Intensive Scanning: If the FortiGate is performing intensive scanning of all traffic, it could be due to a large amount of malicious traffic that requires inspection.
- High-Level Encryption: Utilizing VPN high-level encryption can be CPU intensive. An attacker might exploit this by initiating numerous VPN connections to drive up CPU usage.
- Logging Traffic: If the FortiGate is logging all traffic and packets, a sudden increase in traffic could cause a spike in CPU usage as it tries to log more information than usual.
- Dashboard Widgets: Widgets that frequently perform data updates can consume more CPU resources if they are processing a higher volume of data, possibly due to an attack.
It’s important to monitor the system load and investigate any anomalies that could indicate malicious activity. Regularly checking the CPU usage levels and understanding the normal operational baseline can help in quickly identifying potential attacks. If you suspect an attack, it’s crucial to conduct a thorough investigation and implement appropriate security measures to mitigate the threat.
Indicator 10: Failed Login Attempts
Failed login attempts on a FortiGate device can indicate an attack, particularly a brute force attack, where an attacker systematically tries numerous username and password combinations to gain unauthorized access. Here’s why these failed attempts are a red flag:
- Brute Force Attacks: Multiple failed login attempts could be a sign that someone is trying to guess the user’s or administrator’s password.
- Automated Tools: Attackers often use automated tools that scan for vulnerable targets and attempt to exploit known vulnerabilities or perform password brute force.
- A large amount of failed login attempts can cause a high system load on the FortiGate.
To mitigate such attacks, it’s recommended to:
- Set trusted hosts to allow connections only from trusted and allowed IP addresses.
- Change default SSH and HTTPS ports to non-standard ones.
- Increase the lockout duration to deter attackers.
- Use long and complex usernames and passwords and enforce strong password policies.
- Remove or rename the default ‘admin’ account to reduce the risk of targeted attacks.
- Configure local-in policies to block access from unauthorized IPs.
Monitoring for failed login attempts and implementing these security measures can help protect your network from unauthorized access and potential breaches.
We have a blog post about failed SSL VPN login tries which gives some detailed information about how to harden the SSL VPN part of the FortiGate:
Indicator 11: Configuration Changes
If an attacker was successfull and has entered your system, they often try to gain more access and fetch as much information as possible. Often they change some configurations on the systems so they are able to control traffic and also listen to the it’s contents. Therefore, it is very useful to have an eye on configuration changes on your FortiGates. As example an automation task can inform you about any configuration change made. This way it is very easy to find configuration changes that are not done by yourself.
Indicator 12: Process Crashes
Process crashes can be an indicator for attempted exploitation of a FortiGate service.
If a program does not properly check the length of the input, an attacker can input more data than the program expects. This can overwrite adjacent memory locations, including the return address of a function. By carefully crafting the input, an attacker can change the return address to point to malicious code. This is called a Buffer Overflow.
Attackers might attempt to access restricted areas of memory, which can cause segmentation faults. If they find a way to bypass the system’s memory protection mechanisms, they could potentially gain unauthorized access to sensitive information or system controls. This is called Invalid Memory Access.
fgt01 # diag debug crashlog read
2024-01-01 01:01:01 <17678> firmware FortiGate-60F v7.2.4,build533b533 (GA.M) (Release)
2024-01-01 01:01:01 <17678> application wad
2024-01-01 01:01:01 <17678> *** signal 11 (Segmentation fault) received ***
2024-01-01 01:01:02 <17678> AVDB 91.03222(01/01/0024 00:01)
2024-01-01 01:01:02 <17678> ETDB 91.03222(01/01/0024 00:01)
2024-01-01 01:01:02 <17678> AVSO 01000000AVES0070006011130
2024-01-01 01:01:02 <17678> Register dump:
2024-01-01 01:01:02 <17678> RAX: 0000000000000000 RBX: 0000000000000000
2024-01-01 01:01:02 <17678> RCX: 0000000000000000 RDX: 0000000000000000
2024-01-01 01:01:02 <17678> R01: 0000000000000000 R09: 0000000000000000
2024-01-01 01:01:02 <17678> R10: 0000000000000000 R11: 0000000000000000
2024-01-01 01:01:02 <17678> R12: 0000000000000000 R13: 0000000000000000
2024-01-01 01:01:02 <17678> R14: 0000000000000000 R15: 0000000000000000
2024-01-01 01:01:02 <17678> RSI: 0000000000000000 RDI: 0000000000000000
2024-01-01 01:01:02 <17678> RBP: 0000000000000000 RSP: 0000000000000000
2024-01-01 01:01:02 <17678> RIP: 0000000000000000 EFLAGS: 0000000000000000
2024-01-01 01:01:02 <17678> CS: 0033 FS: 0000 GS: 0000
2024-01-01 01:01:02 <17678> Trap: 0000000000000000 Error: 0000000000000000
2024-01-01 01:01:02 <17678> OldMask: 0000000000000000
2024-01-01 01:01:02 <17678> CR2: 0000000000000000
2024-01-01 01:01:02 <17678> stack: 0x000000000000 - 0x000000000000
2024-01-01 01:01:02 <17678> Backtrace:
2024-01-01 01:01:02 <17678> [0x00000000] => /bin/wad
2024-01-01 01:01:02 <17678> [0x00000000] => /bin/wad
2024-01-01 01:01:02 <17678> [0x00000000] => /bin/wad
2024-01-01 01:01:02 <17678> [0x000000000000] => /usr/lib/x86_64-linux-gnu/libc.so.6
2024-01-01 01:01:02 (__libc_start_main+0x000000eb) liboffset 00000aeb
2024-01-01 01:01:02 <17678> [0x00000000] => /bin/wad
2024-01-01 01:01:02 <17678> process=wad type=2 idx=5 av-scanning=no total=7978 free=3090
2024-01-01 01:01:02 [AV Engine <17678>] AV Engine version: 3.4.45
2024-01-01 01:01:02 [AV Engine <17678>] Last file info:
2024-01-01 01:01:02 [AV Engine <17678>] filename: , filesize: 0, filebuffer: (nil)
2024-01-01 01:01:02 [AV Engine <17678>] Native script imagebase: 0x00000000
2024-01-01 01:01:02 [AV Engine <17678>] Native script imagesize: 0x00000000
2024-01-01 01:01:02 [AV Engine <17678>] AV Engine imagebase: 0x00000000
2024-01-01 01:01:03 wad crashed 1 times. The latest crash was at 2024-01-01 01:01:01
What to do when your FortiGate got Hacked?
We have made a blog post about the remediation steps to follow if your FortiGate got hacked:
If you have any valuable additions or your own experiences, please feel free to share your thoughts in our comments section. We look forward to reading your feedback.
