IPsec VPNs: What They Are and How to Set Them Up | Twingate (2024)

If you’ve ever had to work remotely, you might have used a VPN to securely connect to resources on your company’s private network. Some VPNs use the IPsec protocol to establish secure connections. But what exactly is a VPN and what’s an IPsec VPN?

In a business context, a VPN establishes a protected connection to a secure network environment that would otherwise be inaccessible by an external user. Importantly, VPNs allow users to establish these connections over a public network, such as the internet, in a manner that protects the security and privacy of those connections.

Imagine you are meeting a client at a coffee shop and you need to access sensitive data hosted on your company’s private servers. Accessing private servers from insecure public networks such as a Starbucks coffee shop Wi-Fi can put your company at risk.

VPNs allow users to access a private network and share sensitive data remotely via public networks such as the internet. A secure VPN benefits both small and large companies by protecting their confidential data.

Following are some of the benefits of using a VPN:

• a VPN protects your data from snooping on untrustworthy Wi-Fi hotspots.

• a VPN can hide private information so bad actors don’t steal your information and try to impersonate you.

• a VPN can also block some sites and services from certain sanctioned countries.

Traditional perimeter-based network architecture with VPN

As illustrated above, a traditional network architecture consists of some combination of allowlisting static IP addresses, subnet mapping for remote users using group policy, firewall, route management and a gateway on the public internet.

IPsec stands for Internet Protocol Security. IPsec is a secure network protocol that encrypts and authenticates packets of data to provide secure communication between two devices.

In this article, you will learn about IPsec VPNs in detail, including how to configure an IPsec VPN. Let’s get started!

What Are IPsec VPNs?

An IPsec VPN is a VPN that uses IPsec as its protocol to establish secure connections between devices. Other protocols used by VPNs include SSL/TLS and L2TP. Let’s explore these different protocols:

• Transport Layer Security (TLS), the successor of the Secure Sockets Layer (SSL), is a cryptographic protocol that provides communication security over a computer network. TLS is well known for providing security over HTTPS connections.

• Internet Protocol Security (IPsec) is a secure network protocol that is used in VPNs to authenticate and encrypt the packets of data to provide secure communication.

• Layer Tunneling Protocol (L2TP) is a tunneling protocol used to support VPNs. L2TP does not provide strong authentication by itself. IPSec is often used to secure L2TP packets. The combination of these two protocols is generally referred to as L2TP/IPsec.

For the purposes of this article, our main focus will be on IPsec VPNs.

The IPsec suite of protocols secures data transmitted over the internet or any public network using encryption between two endpoints. IPsec is often used to establish mutual authentication between computers at the beginning of a session and to negotiate cryptographic keys during a session. It can protect data flows between a pair of hosts, networks, or between a network and a host.

IPsec can operate in the following two modes:

• Tunneling mode—encrypts the entire data packet.

• Transport mode—only encrypts the data packet message.

Using an IPsec VPN provides multiple layers of security—it provides authentication to secure any modification to data packets and it encrypts the payloads within each packet. This ensures the security and integrity of data that is being transmitted through the encrypted tunnel. This allows the integrity of data in transition without being intercepted or modified.

There are two IPsec VPN modes:

• IPsec Tunnel Mode VPN—this mode encrypts the entire outgoing packet. This is typically implemented on a secure gateway using a firewall or a router port. For example, employees from an enterprise branch can securely connect with systems in the main office via secure gateways. The IPsec tunnel is established between two gateway hosts.

• IPsec Transport Mode VPN—this mode encrypts only IP payload and ESP trailer being sent between two endpoints. This is primarily used during end-to-end communication and does not alter the IP header of the outgoing packet. For example, this type of mode enables a remote IT support staff to log in to a remote server to do maintenance work. Transport mode is used where one host needs to interact with another host.

How to Set Up an IPsec VPN Server

Here, I will show you how to quickly and automatically set up your own IPsec VPN server in CentOS/RHEL, Ubuntu, and Debian Linux distributions.

To begin, you’ll need a dedicated server or virtual private server (VPS) installed with one of the following OS:

• Ubuntu 20.04 (Focal) or 18.04 (Bionic)

• Debian 11 (Bullseye), 10 (Buster) or 9 (Stretch)

• CentOS 8 or 7

• Rocky Linux 8

  See Also

  IKEv2 VPN Protocol Explained: What It Is and How It WorksIPsec: The Complete Guide to How It Works and How to Use It | TwingateHow to Configure a Site-to-Site IPsec IKEv1 VPN TunnelComparing IPsec vs. SSL VPNs - ONLC

• AlmaLinux OS 8

• Red Hat Enterprise Linux (RHEL) 8 or 7

• Amazon Linux 2

First, log in to the VPS via SSH and then run the commands for your Linux distribution to set up the VPN server. By default, the script will generate random VPN credentials (pre-shared key, VPN username and password) for you and display them at the end of the installation.

However, to use your credentials, you will first need to generate a strong password and PSK, respectively, with the following commands:

> openssl rand -base64 106xWSdx0q7hrUAQ==> openssl rand -base64 16bcM90acDBKB6qdmsZM63Vg==

The output of your first command is your password, and the output from your second command is your PSK.

Next, set these generated values as described in the following command. All the values must be placed inside single quotes as shown.

VPN_IPSEC_PSK: 'Your IPsec pre-shared key'VPN_USER: 'Your VPN user name'VPN_PASSWORD: 'Your VPN password'

On CentOS/RHEL or Ubuntu/Debian

wget https://git.io/vpnsetup -O vpnsetup.shVPN_IPSEC_PSK='KvLjedUkNzo5gBH72SqkOA==' VPN_USER='bobalice' VPN_PASSWORD='8DbDiPpGbcr4wQ==' sudo sh vpnsetup.sh

The main packages to be installed are:

• bind-utils

• net-tools

• bison

• flex

• gcc

• libcap-ng-devel

• libcurl-devel

• libselinux-devel

• nspr-devel

• nss-devel

• pam-devel xl2tpd

• iptables-services

• systemd-devel

• fipscheck-devel

• libevent-devel

• fail2ban

Download, compile, and install Libreswan from the source code, then enable and start the necessary services.Once the installation is complete, you will see the displayed VPN details in the following format:

Server IP: xxx.xxx.xxx.xxxIPsec PSK: VPN_IPSEC_PSKUsername : VPN_USERPassword : VPN_PASSWORD

With this output, your IPsec VPN server is now ready for use. Now, let’s explore how to set up an IPsec VPN Client on Windows, Linux, and OS X machines.

How to Set Up an IPsec VPN Client

Combining L2TP with IPsec protocols will create a highly secure VPN client. L2TP generates a tunnel since it isn’t capable of encryption. IPsec protocol handles encryption, channel security, and data integrity to ensure all packets are secure and not compromised.

Windows 10 and 8.x

After setting up your IPsec VPN server, follow the steps below to set up an IPsec VPN client on Windows 8.x and 10. IPsec VPN is supported natively on Windows—there is no additional software to install.

• Right-click on the wireless/network icon in your system tray.

  See Also

  IPsec vs. SSL VPN: Comparing speed, security risks and technology | TechTarget

• Select Open Network and Sharing Center. Or, if using Windows 10 version 1709 or newer, select Open Network & Internet settings, then click Network and Sharing Center.

• Click Set up a new connection or network.

• Select Connect to a workplace and click Next.

• Click Use my Internet connection (VPN).

• Enter Your VPN Server IP in the Internet address field.

• Enter anything you like in the Destination name field, and then click Create.

• Return to Network and Sharing Center. On the left, click Change adapter settings.

• Right-click on the new VPN entry and choose Properties.

• Click the Security tab. Select Layer 2 Tunneling Protocol with IPsec (L2TP/IPSec) for the type of VPN.

• Click Allow these protocols. Check the Challenge Handshake Authentication Protocol (CHAP) and Microsoft CHAP Version 2 (MS-CHAP v2) checkboxes.

• Click Advanced settings.

• Select Use preshared key for authentication and enter Your VPN IPsec PSK for the key.

• Click OK to close the Advanced settings.

• Click OK to save the VPN connection details.

OS X (macOS)

After setting up your VPN server, follow the steps below to set up an IPsec VPN on OS X. IPsec VPN is natively supported on OS X. There is no additional software that needs to be installed.

• Open System Preferences and go to the Network section.

• Click the + button in the lower-left corner of the window.

• Select VPN from the Interface dropdown menu.

• Select L2TP over IPSec from the VPN Type dropdown menu.

• Enter anything you like for the service name.

• Click Create.

• Enter Your VPN Server IP for the server address.

• Enter Your VPN Username for the Account Name.

• Click the Authentication Settings button.

• In the User Authentication section, select the Password radio button and enter Your VPN Password.

• In the Machine Authentication section, select Shared Secret and enter Your VPN IPsec PSK.

• Click OK.

• Check the Show VPN status in menu bar checkbox.

• Click Advanced and make sure the Send all traffic over VPN connection checkbox is checked.

• Click the TCP/IP tab and make sure Link-local only is selected in the Configure IPv6 section.

• Click OK, and then click Apply to save the VPN connection information.

Ubuntu Linux

Ubuntu (18.04 and newer) users can install the network-manager-l2tp-gnome packaging using apt, then configure the IPsec/L2TP VPN client using the GUI. Follow these steps:

• Go to Settings > Network > VPN. Click the + button.

• Select Layer 2 Tunneling Protocol (L2TP).

• Enter anything you like in the Name field.

• Enter Your VPN Server IP for the Gateway.

• Enter Your VPN Username for the User name.

• Right-click the ? in the Password field and select Store the password only for this user.

• Enter Your VPN Password for the Password.

• Leave the NT Domain field blank.

• Click the IPsec Settings… button.

• Check the Enable IPsec tunnel to L2TP host checkbox.

• Leave the Gateway ID field blank.

• Enter Your VPN IPsec PSK for the Pre-shared key.

• Expand the Advanced section.

• Enter aes128-sha1-modp2048 for the Phase1 Algorithms.

• Enter aes128-sha1 for the Phase2 Algorithms.

• Click OK, then click Add to save the VPN connection information.

• Turn the VPN switch ON.

Conclusion

A virtual private network (VPN) helps shield your infrastructure from hackers. VPNs are commonly used in business to enable employees to access a corporate network remotely.

With an IPsec VPN, IP packets are protected as they travel to and from the IPsec gateway. IPsec protocols implement a secure network that protects VPN data from external bad actors.

However, a VPN can create a false sense of security. For example, your third-party vendors or hackers can exploit vulnerable information thanks to the single entrypoint to your private network. VPNs also aren’t the cheapest solution. They require high support costs—a sudden shift to remote work while maintaining the network infrastructure can put strain on the IT team.

Check out Twingate, a modern, zero-trust alternative to VPNs for securing your network. Unlike traditional VPNs, Twingate doesn’t require you to expose a public gateway, and it reduces lateral attack vulnerabilities.

Request a demo if you’d like to learn more.