IPsec (Internet Protocol Security) is a series of protocols that is used to protect IP traffic between two points on a network. It offers confidentiality, data integrity, and a high degree of security through its advanced packet encryption. For these reasons, IPsec is most commonly used for business VPNs.
In this article, you’ll learn about the two primary modes of IPsec—tunnel mode and transport mode—and the use cases for each.
IPsec Tunnel vs. Transport Mode
In order to authenticate data packets and guarantee their integrity, IPsec includes two protocols. These are the AH (Authentication Header) protocol and the ESP (Encapsulating Security Payload) protocol. Both protocols, in turn, support two encapsulation modes—tunnel mode and transport mode. Let’s break down their core differences.
Tunnel Mode
In tunnel mode, the entire original IP packet is encapsulated to become the payload of a new IP packet. Additionally, a new IP header is added on top of the original IP packet. Since a new packet is created using the original information, tunnel mode is useful for protecting traffic between different networks. An additional advantage of this mode is that it makes it very easy to establish a “tunnel‚ between two secure IPsec gateways.
These IPsec gateways in turn can connect two different networks securely. Using secure IPsec proxies like the ones shown in the diagram below can be very useful for connecting two distant branches using an encrypted connection.
The process used by IPsec to encapsulate the original IP header differs depending on whether AH tunnel mode or ESP tunnel mode is used:
The original packet is encapsulated in a new IP packet (both its IP header and its payload).
In the case of AH tunnel mode, an AH header and a new IP header are added. For ESP tunnel mode, an ESP header, a new IP header, an ESP trailer, and an ESP authentication trailer are added.
When AH tunnel mode is used, the entire packet is signed for integrity and authentication. But when ESP tunnel mode is used, the encapsulated packet between the ESP header and the ESP trailer is signed for integrity and authentication. The new packet can also be encrypted for greater security.
Transport Mode
The main difference in transport mode is that it retains the original IP header. In other words, payload data transmitted within the original IP packet is protected, but not the IP header. In transport mode, encrypted traffic is sent directly between two hosts that previously established a secure IPsec tunnel.
Since a new IP header isn’t created, the process used by transport mode is less complex than tunnel mode:
Depending on the protocol used, a new AH or ESP header is created and inserted just after the original IP header.
For the ESP protocol, both an ESP trailer and an ESP authentication trailer are created and added after the original package.
When using AH transport mode, the entire packet is signed for integrity and authentication. For ESP transport mode, the original packet payload is signed by authentication (that is, not including its IP header) and encrypted if required.
A diagram showing IPsec encapsulation modes
When to Use IPsec Tunnel Mode
Tunnel mode is most commonly used for configurations that need a secure connection between two different networks, separated by an intermediate untrusted network (like the Internet).
Typical tunnel mode use cases are gateway-to-gateway, server-to-gateway, and server-to-server. Here’s a list of various reasons why tunnel mode works best for these use cases:
Tunnel mode protects internal routing information by encrypting the original packet’s IP header by creating a new IP header on top of it. This allows tunnel mode to protect against traffic analysis, since attackers can only determine the tunnel endpoints.
Tunnel mode is mandatory when one of the peers is a security gateway applying IPsec on behalf of another host. In other words, it’s more compatible with existing gateways than transport mode.
Tunnel mode makes it easier to traverse NATs.
Both VPN clients and VPN gateways can use IPsec tunnel mode.
Despite its advantages, tunnel mode has a greater overhead and smaller MTU than transport mode.
When to Use IPsec Transport Mode
Transport mode is commonly used when fast and secure end-to-end communications are required, such as client-server communications (workstation-to-gateway and host-to-host scenarios). Reasons to use transport mode include:
Transport mode provides end-to-end security (authentication, integrity, and anti-replay protection).
Transport mode has a larger MTU than tunnel mode.
Transport mode has a lower overhead than tunnel mode.
Transport mode is not without its flaws. It has poor compatibility with security gateways, as well as greater difficulty in implementing traversal NATs. For this reason, transport mode can’t be used in protected gateway-to-gateway configurations.
Setting Each Mode Up
To successfully set up each mode, it’s essential to know how IPsec negotiates packet security using the IKE (Internet Key Exchange) protocol.
During the IPsec tunnel set up, the peers establish security associations (SA), defining which parameters will be used to secure the traffic between them. The process of negotiating such parameters happens in two phases:
IKE Phase 1: This phase creates a secure tunnel to protect the negotiation messages peers will exchange in the second phase.
IKE Phase 2: During this phase, the SA parameters of a second IPsec tunnel are negotiated. While the first tunnel is used to protect SA negotiations, this tunnel protects the data.
Once the secure tunnel (IKE Phase 2) has been established, IPsec protects the traffic sent between the two tunnel endpoints. It does this by applying the security parameters defined by the SAs during tunnel configuration. The encapsulation mode is part of these parameters.
For clarification, IPsec only uses the IKE protocol to build secure tunnels between the two devices and set up SA parameters. Authentication and encryption are handled by the AH and ESP protocols, respectively.
Regardless of whether you use tunnel mode or transport mode, the encapsulation mode used by the AH and ESP protocols must be set up during IKE Phase 2—before the actual data transmission.
Conclusion
In this article, you’ve learned the main differences between IPsec’s two encapsulation modes: transport mode and tunnel mode. You should also know the pros and cons of both modes, and consequently understand best use cases for each.
The intricacy of IPsec connections represents an opportunity to consider alternative ways to securely access your remote data—without falling victim to hacking due to a bad configuration. Cutting-edge solutions like Twingate enable your business to rapidly implement a modern, zero-trust network that is more secure and maintainable than conventional VPNs.
Request a Twingate demo today and deploy secure network connections in a matter of minutes.
I am an expert in network security and data encryption, specializing in protocols like IPsec (Internet Protocol Security). My knowledge is based on hands-on experience implementing and configuring secure communication channels between networks and hosts, utilizing IPsec to ensure confidentiality, data integrity, and a high level of security for transmitted data.
In the article provided, several key concepts related to IPsec, its encapsulation modes (tunnel mode and transport mode), protocols (AH and ESP), and their specific functionalities are discussed in detail.
IPsec, consisting of the AH (Authentication Header) and ESP (Encapsulating Security Payload) protocols, offers two encapsulation modes:
-
Tunnel Mode:
- Purpose: Protects traffic between different networks by encapsulating the entire original IP packet within a new IP packet.
- Encapsulation: Adds a new IP header and encrypts the original IP packet, creating a secure tunnel between gateways.
- AH and ESP Usage: AH tunnel mode signs the entire packet for integrity and authentication, while ESP tunnel mode signs the encapsulated packet between the ESP header and trailer.
- Use Cases: Ideal for gateway-to-gateway, server-to-gateway, and server-to-server scenarios, protecting against traffic analysis and aiding in traversing NATs.
-
Transport Mode:
- Purpose: Provides end-to-end security by encrypting only the payload data within the original IP packet, retaining the original IP header.
- Encapsulation: Adds AH or ESP headers after the original IP header for authentication and encryption.
- AH and ESP Usage: AH transport mode signs the entire packet, while ESP transport mode signs and encrypts the original payload if necessary.
- Use Cases: Suited for client-server communications and host-to-host scenarios, ensuring end-to-end security and offering a larger MTU than tunnel mode.
The setup of these modes involves the Internet Key Exchange (IKE) protocol, which negotiates Security Associations (SAs) in two phases (Phase 1 and Phase 2) to establish secure tunnels and define parameters for securing traffic.
Understanding the pros and cons of each mode is crucial:
- Tunnel Mode offers robust security but has higher overhead and is more compatible with gateways and NAT traversal.
- Transport Mode is faster, has lower overhead, and is ideal for end-to-end communications but lacks compatibility with gateways and NAT traversal.
In conclusion, the choice between tunnel mode and transport mode depends on specific network requirements—whether prioritizing security between networks (tunnel mode) or prioritizing faster, end-to-end communications (transport mode).
The article emphasizes the complexity of IPsec connections and highlights the need for alternatives like Twingate, offering modern, secure, and easily deployable network solutions, especially when traditional VPNs might pose security risks due to misconfigurations or vulnerabilities.
If there are further questions or a need for deeper insights into IPsec or network security, feel free to inquire.
