How IPsec works, it’s components and purpose (2024)

IPsec was one of the internet's first security protocols, and it still powers most VPNs.

Credit: Thinkstock

What is IPsec?

IPsec is a suite of protocols that are used to secure internet communications—in fact, the name itself is an abbreviation for Internet Protocol Security.

IPsec was first codified in the ’90s, spurred on by the dawning realization that internet traffic needed to be protected: the early internet mostly connected secured government and university buildings, and the internet protocol (IP) that defined how communications online worked sent information whizzing around unsecured and unencrypted.

IPsec was designed to create a universal standard for internet security and enabled some of the first truly secure internet connections. IPsec isn’t the most common internet security protocol you’ll use today, but it still has a vital role to play in securing internet communications.

What is IPsec used for?

If you’re using IPsec today, it’s probably in the context of a virtual private network, or VPN. As its name implies, a VPN creates a network connection between two machines over the public internet that’s as secure (or almost as secure) as a connection within a private internal network: probably a VPN’s most well-known use case is to allow remote employees to access secured files behind a corporate firewall as if they were working in the office.

The protocols in the IPsec suite are the technologies that secure one of the main kinds of VPN—called an IPsec VPN, naturally. For most of this article, when we say VPN, we mean an IPsec VPN, and over the next several sections, we’ll explain how they work.

A note on IPsec ports: If you’re looking to set up your firewall to allow an IPsec VPN connection, be sure to open UDP port 500 and IP ports 50 and 51.

IPsec layer

Before we get into the nitty gritty of how IPsec VPNs work, we need to understand what gives them their special place in the world of networking, and to that end we need to talk about the OSI networking model. The OSI model defines seven layers—basically, levels of increasing abstraction—on which networked communication takes place. At the top of the stack is layer 7, the application layer, where your web browser lives; at the bottom is layer 1, the physical layer, where pulses of electricity travel through wires.

At the heart of the model are the transport layer (layer 4) and the network layer (layer 3). Code written to manage the transport layer runs on individual computers and deals with the coordination of data transfer between end systems and hosts: how much data to send, at what rate, and where it goes. Once this has all been set, the transport layer hands off the data to the network layer, which is mostly controlled by code running on the routers and other components that make up a network. These routers decide on the route individual network packets take to their destination, but the transport layer code at either end of the communication chain doesn’t need to know those details.

The TCP/IP protocol suite that’s at the heart of the internet straddles these two layers—TCP (or transport control protocol) is for transport, and IP is for networking. On its own, IP doesn’t have any built-in security, which, as we noted, is why IPsec was developed. But IPsec was followed closely by SSL/TLS—TLS stands for transport layer security, and it involves encrypting communication at that layer.

Today, TLS is built into virtually all browsers and other internet-connected applications, and is more than enough protection for everyday internet use. But it’s not perfect, and if an attacker were able to crack or otherwise sidestep your TLS encryption, they’d have access to the data in the individual network packets you’re sending out over the internet. That’s why an IPsec VPN can add another layer of protection: it involves securing the packets themselves.

How IPsec works

An IPsec VPN connection starts with establishment of a Security Association (SA) between two communicating computers, or hosts. In general, this involves the exchange of cryptographic keys that will allow the parties to encrypt and decrypt their communication. (For more on how cryptography works in general, check out CSO’s cryptography explainer.) The exact type of encryption used is negotiated between the two hosts automatically and will depend on their security goals within the CIA triad; for instance, you could encrypt messages to ensure message integrity (i.e., to ensure that data hasn’t been altered) but not confidentiality. But in most cases you will try to keep data confidential as well.

The information about the SA is passed to the IPsec module running on each of the communicating hosts, and each host’s IPsec module uses that information to modify every IP packet sent to the other host, and to process similarly modified packets received in return. These modifications can affect both the packet’s header—metadata at the beginning of the packet explaining where the packet is going, where it came from, its length, and other information—and its payload, which is the actual data being sent.

For a full technical explanation of IPsec works, we recommend the excellent breakdown on NetworkLessons.

IPsec components

There are three main IPsec protocols that determine how IPsec modifies IP packets:

• Internet Key Exchange (IKE) establishes the SA between the communicating hosts, negotiating the cryptographic keys and algorithms that will be used in the course of the session.
• Authentication Header (AH) adds a header field to the packet being sent that includes a cryptographic hash of the packet’s contents. The host that receives the packet can use this hash to ensure that the payload hasn’t been modified in transit.
• Encapsulating Security Payload (ESP) encrypts the payload. It also adds a sequence number to the packet header so that the receiving host can be sure it isn’t getting duplicate packets.

Note that it’s possible to use both AH and ESP simultaneously, although newer versions of the ESP protocol incorporate much of AH’s functionality. At any rate, both protocols are built into IP implementations.

IPsec encryption

The encryption established by IKE and ESP does much of the work we expect out of an IPsec VPN. You’ll notice that we’ve been a little vague about how the encryption works here; that’s because IKE and IPsec permit a wide range of encryption suites and technologies to be used, which is why IPsec has managed to survive over more than two decades of advances in this area. It’s fairly common for IPsec VPNs to use Public Key Infrastructure (PKI) for their cryptographic purposes, but that’s by no means required and there are other options available.

IPsec modes: IPsec Tunnel vs. IPsec Transport

There are two different ways in which IPsec can operate, referred to as modes: Tunnel Mode and Transport Mode. The difference between the two pertains to how IPsec treats packet headers. In Transport Mode, IPsec encrypts (or authenticates, if only AH is being used) only the payload of the packet, but leaves the existing packet header data more or less as is. In Tunnel Mode, IPsec creates an entirely new packet with a new header, encrypts (or authenticates) the entire original packet including its header, and uses the modified original packet as the payload for the new packet.

When would you use the different modes? If a network packet has been sent from or is destined for a host on a private network, that packet’s header includes routing data about those networks—and hackers can analyze that information and use it for nefarious purposes. Tunnel Mode, which protects that information, is generally used for connections between the gateways that sit at the outer edges of private corporate networks. A packet is encrypted as it leaves one network, and put inside a new packet whose destination is the gateway for the target network. Once it arrives at the gateway, it’s decrypted and removed from the encapsulating packet, and sent along its way to the target host on the internal network. The header data about the topography of the private networks is thus never exposed while the packet traverses the public internet.

Transport mode, on the other hand, is generally used for workstation-to-gateway and direct host-to-host connections. A service tech using Windows Remote Desktop to help diagnose a problem on a user’s computer would be using a Transport Mode connection, for instance.

IPsec VPNs vs. SSL VPNs

As noted above, IPsec VPNs are not the only game in town. There are also SSL VPNs, which as the name implies are secured by the TLS protocol rather than IPsec. SSL VPNs operate via a web browser and are generally used for accessing firewalled intranet websites. Because they’re built into the browser software that everyone’s familiar with, SSL VPNs are much simpler to use; IPsec VPNs often require the installation and configuration of specialized software. SSL VPNs can also provide more granularly limited access to the private network.

On the other hand, because it uses TLS, an SSL VPN is secured at the transport layer, not the network layer, so that may affect your view of how much it enhances the security of your connection.

IPsec tutorial

Where to learn more:

• NetworkLessons has a great in-depth look at IPsec that walks you through the technical details we’ve skimmed over here.
• IBM also has a good series of quick guides that explain how to configure the topologies of various IPsec connections.
• Netgate walks you through an example of a site-to-site IPsec VPN connection that should be illuminating