IPsec: The Complete Guide to How It Works and How to Use It | Twingate (2024)

IPsec is a suite of protocols widely used to secure connections over the internet. The three main protocols comprising IPsec are: Authentication Header (AH), Encapsulating Security Payload (ESP), and Internet Key Exchange (IKE). This architectural framework for network data security specifies how to select security protocols, determine security algorithms, and exchange keys between peer layers, in addition to providing services such as access control, data source authentication, and data encryption.

In this article, you’ll learn more about IPsec’s development, features, capabilities, and drawbacks, along with some newer technologies that address these drawbacks.

IPsec’s network-layer security architecture applies its security protections to each IP packet, effectively securing them with specific forms of safeguarding including data source authentication, integrity verification of connectionless data, confidentiality protection of data content, and more. IPsec VPN solutions are one of the most popular approaches to safeguarding remote access and site-to-site connections since most apps will work with them.

In the early days of the internet, theft of confidential data and attacks on enterprise networks happened often because of security deficiencies in the design of IP protocol. Though the IETF has now researched and developed a set of security protocols to protect IP communications, IPsec was developed to provide IP-based network layer security, which serves all IP-based network communications and is completely transparent to upper-layer protocol applications.

The AH and ESP protocols used by IPsec protect IP datagrams and upper-layer protocols (such as UDP and TCP) using the two operating modes, tunnel mode and transport mode. These protocols verify the data source, guarantee data integrity, and prevent successive replays of identical packets. In addition to these services, ESP has the additional feature of guaranteeing data confidentiality and providing limited confidentiality to the data stream.

Depending on how it is deployed and configured, IPsec can ensure confidentiality, integrity, and authentication of IP communications.

Confidentiality

IPsec protects data from being accessed by unauthorized people by encrypting and decrypting data with a cryptographic method and a secret key—a value that is known only by the two parties exchanging data; only someone with the secret key may decrypt the information. While using IPsec without encryption is conceivable, it is not advised.

Integrity

IPsec also checks whether data has been altered (intentionally or unintentionally) while in transit. The integrity of data can be ensured by generating a message authentication code (MAC) value, which is a cryptographic checksum (hash) of the data generated with a secret key that has been agreed upon (different from the encryption secret key). The MAC’s verification will fail if the data is tampered with.

Authentication

Finally, each IPsec endpoint verifies the identity of the other endpoint it desires to communicate with, ensuring that network traffic and data are only sent to the intended and permitted endpoint.

IPsec Drawbacks

Despite its great utility, IPsec has a few issues worth mentioning. First, direct end-to-end communication (i.e., transmission method) is not always available. Consider the following scenario:

H1 and H2 are two hosts connected by a direct tunnel, and H1 employs the FW1 firewall. The adoption of various regional security regulations in large-scale distributed systems or inter-domain settings may pose severe issues for end-to-end communication. In this example, assume that FW1 needs to inspect traffic content to detect intrusions and that a policy is set at FW1 to deny all encrypted traffic so as to enforce its content inspection requirements. However, H1 and H2 construct direct tunnels without knowledge of the firewall and its policy rules. As a result, all traffic will be dropped by FW1. Thus, meeting each policy’s corresponding requirements may lead to conflicts.

Additionally, one of the biggest disadvantages of IPsec is its complexity. Although IPsec’s flexibility makes it popular, it can also be confusing. Security experts point out that IPsec contains too many options and too much flexibility. Most of the flexibility and complexity of IPsec may be attributed to the fact that IPsec was developed through a committee process. Due to the political nature of the committee, additional functions, options, and flexibility were added to the standard to satisfy the various factions of the standardization agency. Complexity can lead to incorrectly implementing or configuring IPsec, leading to unintended security consequences.

IPsec is commonly used when implementing VPNs as it offers a high level of protection and allows numerous private networks to connect securely over the internet. IPsec protects all data transferred between terminal sites at the network layer, independent of the kind of network application. Users who use VPNs to remotely access a private business network are placed on the network itself, giving them the same rights and operational capabilities as a user who is connecting from within that network.

An IPsec-based VPN may be created in a variety of ways, depending on the needs of the user. In most cases, IPsec is used by a mix of clients, servers, firewalls, and routers. Because these components may originate from various suppliers, interoperability is a must. IPsec VPNs enable smooth access to enterprise network resources, and users do not necessarily need to use web access (access can be non-web); it is therefore a solution for applications that need to automate communication in both ways.

IPsec provides a robust, long-lasting foundation for delivering network layer security. Its framework can support today’s cryptographic algorithms as well as more powerful algorithms as they become available in the future. IPsec is a mandatory component of Internet Protocol Version 6 (IPv6), which companies are actively deploying within their networks, and is strongly recommended for Internet Protocol Version 4 (IPv4) implementations.

IPsec uses two modes to send data—tunnel mode and transport mode:

In tunnel mode, IPsec uses two dedicated routers, each acting as one end of a virtual “tunnel‚ over a public network. In addition to protecting the packet content, the original IP header containing the packet’s final destination is also encrypted in this mode. IPsec introduces a new IP header to notify intermediary routers where to forward traffic.

Pros

• More compatible with existing VPN gateways

• No need to implement IPsec on the IPS entity

• Easier to traverse NAT

Cons

• More overhead

• Smaller maximum transmission unit (MTU)

  See Also

  Select a Mobile VPN Type

In transport mode, each packet’s payload is encrypted, but not the IP header. Unless a separate tunnelling protocol such as GRE is employed, intermediary routers are able to see the final destination of each packet.

Pros

• Provides end-to-end encryption

• Minimal overhead compared to tunnel mode

• Larger MTU

Cons

• Requires IPsec to be implemented on the Intrusion Prevention System (IPS) entities

• There is greater difficulty with NAT traversal (TCP checksum invalidation)

IPsec passthrough is a technique for allowing IPsec packets to pass through a NAT router. VPN passthrough is a broader term that refers to a technique for allowing various VPN tunnelling protocols (including IPsec, PPTP and L2TP) to successfully traverse NAT; it is essentially a way to support routing of older VPN tunnelling protocols that were not built with that ability.

IPsec defines a standard set of protocols for securing internet connections, providing for the authentication, confidentiality, and integrity of communications. It provides a transparent end-to-end secure channel for upper-layer protocols, and implementations do not require modifications to those protocols or to applications. While possessing some drawbacks related to its complexity, it is a mature protocol suite that supports a range of encryption and hashing algorithms and is highly scalable and interoperable.

The pandemic has changed the way we work and collaborate. Even post-pandemic, remote working will remain a prominent feature of corporate life. While IPsec VPNs are a common and widespread way of enabling workforces to gain access to corporate IT resources, as this article has covered, the technology is complicated to understand, deploy, and maintain. Instead of dealing with this complexity, consider adopting the next generation of technology for secure remote access: Zero Trust Network Access (ZTNA). ZTNA is a modern approach that fits how organizations operate today while offering stronger security than a VPN. Like VPNs, there are many ways a Zero Trust model can be implemented, but solutions like Twingate make the process significantly simpler than having to wrangle an IPsec VPN. Contact Twingate today to learn more.