What is IPSec, a Protocol for VPN with Better Security and How it Works

A VPN is the acronym for “Virtual Private Network“, or also known as virtual private network, and is a network technology that allows us to extend LAN communications over the Internet network, and all this in a totally secure way thanks to use of cryptography. A VPN will allow a computer to send and receive data on shared or public networks, but logically being in the private network with all its functionalities, permissions, security, management policies, etc.

Some very typical uses of VPNs are as follows:

• Possibility of connecting two or more offices of a company with each other, using Internet connectivity.
• Allow members of the technical support team to connect from their homes to the company.
• That a user can access his home computer from a remote site, such as a hotel.

All these uses will always be through an infrastructure that we all know well: the Internet.

Within the VPNs, we have mainly two VPN architectures, the remote access VPNs (Roadwarrior VPN or Mobile Clients) and the Site-to-Site VPNs (Site-to-site VPN). Depending on our needs, we must configure one architecture or another.

• Remote Access VPN (Roadwarrior or Mobile Client) : This VPN architecture is designed so that one or more users connect to a VPN server, and can access all the shared resources of your home or company, in addition, it allows it to be done a redirection of traffic, in this way, we will go to the Internet through the VPN server (and with the public IP of the VPN server). This type of VPN is the most typical that we can configure on NAS servers, routers, and other devices whose objective is to provide us with safe browsing through the Internet. These enterprise-level VPNs would also serve to isolate zones and internal network services that require additional authentication, in addition, it could also be a good idea whenever we use WiFi connectivity, both at home and business, to add another layer of encryption .
• Site-to-Site VPN (VPN Site-to-Site) : this VPN architecture is designed to interconnect different sites, if, for example, we have a company with different sites, we can interconnect them via VPN and access all resources. The establishment of the connection is not carried out in the end client, as is the case with remote access VPNs, but is done by routers or firewalls, in this way, the entire network will be seen as “one”, although the traffic travel through multiple VPN tunnels.

In the following image we can see a VPN architecture with both models, both site-to-site VPN (left) and remote access VPN (right):

What must a VPN guarantee to be secure?

To ensure that a connection is secure through a virtual private network (VPN), certain functions must be guaranteed, otherwise, we could be facing an unreliable VPN. We already anticipate that the IPsec protocol complies with all of them, since it is a secure VPN protocol, which is widely used in companies.

Authentication

Authentication is one of the most important processes in a VPN, this feature allows a user to be shown that they really are who they say they are. The way to prove it is by entering a passcode, making use of a digital certificate, or a combination of both forms of authentication. When the host receives an IPsec datagram from a source, the host is sure that the source IP address of the datagram is the actual source of the datagram, because it has previously been successfully authenticated.

Confidentiality

Confidentiality is another of the fundamental characteristics of VPNs, confidentiality means that the information is required to be accessible only to authorized entities, that is, all communications are encrypted point-to-point, and only who has previously authenticated in the system, it will be able to decrypt all the information exchanged. If someone is able to get in the middle of the communication and capture it, they will not be able to decrypt it because they will be using cryptography, either symmetric or asymmetric key cryptography.

Integrity

Authentication and confidentiality are just as important as integrity. Integrity means that it can be ensured that the information has not been modified between the origin of the communication and the destination. All communications in a VPN include error detection codes and that the information is not modified. In case of being modified, the packet is automatically discarded, and it could even cause a VPN tunnel crash for security reasons. The IPsec protocol allows the receiving host to verify that the datagram header fields and the encrypted payload have not been modified while the datagram was en route to the destination.

Let’s imagine we have authentication and confidentiality in a VPN, but we don’t have integrity. If a user in the middle of the communication modifies some values, instead of sending a money transfer of € 10, he could convert it to € 1,000. Thanks to the integrity feature, as soon as a bit is modified the packet is discarded and will wait for it to be sent again.

I do not repudiate

This characteristic of cryptography means that you cannot say that you have not sent any information, because it is signed by your digital certificate, or by your username / password pair. In this way, we can know for sure that that user has sent specific information. Non-repudiation can only be “circumvented” if someone is able to steal the username / password pair or the digital certificates.

Access control (authorization)

It is about ensuring that authenticated participants have access only to the data to which they are authorized. The identity of users must be verified and their access restricted to those who are authorized. In a business environment this is very important, a user should have the same access level and the same permissions as if they were physically, or fewer permissions, but never more permissions than those they had physically.

Activity Register

It is about ensuring proper operation and resilience. The VPN protocol should record all the established connections, with the source IP address, who has authenticated, and even what they are doing in the system based on the virtual IP address that has been provided.

Quality of service

It is about ensuring good performance, that there is no unacceptable degradation in transmission speed. We must bear in mind that when we establish a VPN connection, we will always have less real speed because all traffic is encrypted point-to-point, and depending on the strength of the VPN server and the clients, we can achieve a higher or lower speed. Before starting to deploy a VPN, we should look at the hardware of the equipment and the maximum bandwidth that we can have.

Introduction to IPsec

The IPsec protocol is one of the most important security protocols, and it is widely used in companies, and also in home users. In recent times, manufacturers such as ASUS, AVM and even D-Link, are integrating VPN in their home routers based on the IPsec protocol. This protocol provides security services to the IP layer and to all the higher protocols, such as TCP and UDP (Internet transport layer). Thanks to IPsec, we can communicate between different points of the Internet in a secure way, such as two or more companies with each other, or a user with his home, IPsec is perfectly suited to the VPN needs of both “worlds”.

A very important feature of IPsec is that it works at layer 3 of OSI (network layer), other VPN protocols such as OpenVPN or WireGuard work at layer 4 (transport layer), since the latter two base their security on TLS and DTLS respectively. IPsec in IPv4 networks is just above the IP header, however, in IPv6 networks it is integrated (ESP) in the header itself in the “Extensions” section.

IPsec provides all services necessary for communication to be secure, as we have explained previously, these services are authentication, confidentiality, integrity and non-repudiation . Thanks to these services, the security of communications is guaranteed. Of course, we also have access control, quality of service and activity log.

Another very important feature of IPsec is that it allows both VPN architectures , both remote access VPN and site-to-site VPN. Regarding the negotiation of cryptography, IPsec integrates a negotiation system so that the end teams negotiate the best possible encryption they support, agree on the exchange keys, and choose the encryption algorithms they have in common. Depending on the IPsec header used (AH or ESP), we can only check the authenticity of the packet, or encrypt the payload of the entire IP packet and also check its authenticity.

When two hosts have established an IPsec session, the TCP segments and UDP datagrams are sent between them encrypted and authenticated, in addition, the integrity is also checked to prevent someone from modifying it. Therefore, IPsec guarantees the security of communications.

Some advantages of IPsec are that it is supported by all IETF standards, and provides a VPN “standard” so all devices should be compatible. IPSec is receiving very important support from all communications equipment, as it is the “standard” for VPN, much more widely used than OpenVPN or WireGuard. All versions of PC operating systems such as Windows or Linux, MacOS for Apple computers, and also Android and iOS support the IPsec protocol. In addition, another very important characteristic is that, being a standard, there is interoperability between manufacturers, which constitutes a guarantee for users. Another remarkable feature of IPSec is its nature as an open standard, and it is perfectly complemented by PKI (Public Key Infrastructure) technology.

IPsec combines public key technologies (RSA or Elliptical Curves), symmetric encryption algorithms (mainly AES, although it also supports others such as Blowfish or 3DES), and hashing algorithms (SHA256, SHA512 etc), as well as digital certificates based on X509v3.

IPsec headers

The IPsec protocol has an architecture with several headers, depending on what we are interested in “ensuring”, we can choose one header or another, we cannot choose both headers simultaneously in the same IPsec tunnel. The headers that we have in this protocol are the following:

• Authentication Header (AH)
• Encapsulated Security Payload (ESP)

Next, we are going to explain in detail how both headers work.

Authentication header (AH)

This header provides authentication and integrity to the transmitted IP packets, to provide this IPsec feature, it makes use of HMAC fingerprints. The protocol itself will be in charge of calculating a hash function to the content of the IP packet, some of the hash functions used by this protocol are MD5 or SHA-1 which are not secure, but it also supports SHA256 or SHA512 which are secure.

This header provides the receiver of the IP packets with a method to authenticate the origin of the data, and verify that said data has not been altered in the communication. A very important detail is that this header does not provide confidentiality because it does not encrypt the data of the IP packet, therefore, the information exchanged can be seen by third parties unless they use protocols such as HTTPS or FTPES with TLS security.

AH is an authentication header that is inserted between the standard IP header (both in IPv4 and IPv6 networks) and the transported data. This transported data can be a TCP, UDP or ICMP message, and even a complete IP datagram. Within the AH header is where the upper layer data is indicated, in addition, AH ensures the integrity and authenticity of the IP header itself, except for variable changes such as TOS, TTL, flags, offset and checksum.

The operation of the AH protocol is as follows:

1. The sender calculates the hash function from the message to be transmitted. It will be copied to the AH header in the “Authentication data” field.
2. The data is transmitted via the Internet.
3. When the packet reaches the receiver, it will apply the hash function and compare it with the one it already had (they both have the same shared secret key).

If the fingerprints match, it means that the datagram has not been modified, otherwise, we can claim that the information has been tampered with.

Encapsulated Security Payload (ESP)

The Encapsulated Security Payload, or also known as ESP, offers authentication, integrity and confidentiality of data transmitted over IPsec. That is to say, in this case we will be encrypting the entire data field so that all communications are confidential, unlike AH that does not encrypt the transmitted message. To achieve these security features, a public key exchange is made using Diffie-Hellmann to ensure communication between both hosts.

The main function of the ESP protocol integrated in IPsec, is to provide confidentiality to the data, in order to do this, ESP defines the encryption and the way in which the data will be located in a new IP datagram. To provide authentication and integrity, ESP uses mechanisms similar to AH. Because ESP provides more functions than AH, the header format is more complex: this format consists of a header and a tail (which is placed at the end of the packet), therefore ESP “surrounds” the transported data. Regarding data, ESP allows you to use any IP protocol, for example, TCP, UDP, ICMP and even a complete IP packet.

The structure of an ESP packet is as follows:

ESP belongs to the network level within TCP / IP. The data area is fully encrypted , the datagram itself could also be authenticated to provide greater security. The encryption of the data is done using symmetric key algorithms , usually block ciphers are used (such as AES), the data encryption is done using multiples of the block size , for this reason we have the “Padding”, a field of filling.

To encrypt the data, the sender first encrypts the original message using a key and inserts it into a new IP datagram (which is protected by the ESP header). In the hypothetical case that someone intercepts the message (Man In The Middle), they will only get meaningless data since they do not have the secret key to decrypt the message. When the message reaches the destination, it will apply the secret key on the data and decrypt the packet.

The most widely used algorithm is AES in all its versions (128 and 256 bits) and in its different encryption modes such as AES-CBC, AES-CFB and AES-OFB . However, it is advisable to make use of AES-GCM that AEAD will provide us and it is much more secure than the others. Therefore, it is essential to use a good encryption algorithm to protect all data , the distribution of the keys in a secure way will be very important. A delicate issue is that both sides of the communication agree on the algorithms and authentication, this will be done by the IKE protocol.

IKE: what it is and what it is for

This IKE (Internet Key Exchange) protocol is used to generate and manage the keys necessary to establish the AH (Authentication Header) and ESP (Encapsulated Security Payload) connections . The two or more participants of the IPsec connection must agree in some way, the types of encryption and the authentication algorithms to be able to establish the connection in a secure way. This configuration can be done manually at both ends of the channel, or through a protocol (the IKE protocol ) to take care of the automatic negotiation of the participants (SA = Security Association).

The IKE protocol is not only responsible for the management and administration of the keys, but also for establishing the connection between the corresponding participants. IKE is not only in IPsec, but it can be used in different routing algorithms such as OSPF or RIP.

Phases of IKE negotiation

The establishment of the secure channel will be done using a key exchange algorithm such as Diffie-Hellman to encrypt the IKE communication. This negotiation is done through a single bidirectional SA. Authentication can be through PSK (shared key) or with other methods such as RSA certificates. Using the secure channel that has been created, the IPsec (or other services) security association will be negotiated.

Some features of IKE

IKE supports NAT traversal , even if one or both participants are behind a NAT, the connection can be made without many problems, although we will have to open ports on the VPN server if it is behind the NAT. Sequence numbers and ACK’s are used to provide reliability, it also includes an error processing system. IKE is resistant to denial-of-service attacks, and IKE takes no action until it determines if the requesting endpoint really exists, thus protecting against attacks from fake IP addresses.

Currently IKEv2 is widely implemented in all professional firewalls and routers, however, it is not yet fully extended in the Android or iOS world, only Samsung smartphone users support IPsec with IKEv2. Windows, Linux and macOS operating systems do support this protocol.

IKEv2: what has changed

IKEv2 is the second version of this popular Internet Key Exchange protocol, it incorporates improvements for NAT traversal, which makes it easier to communicate and pass through firewalls in general. It also supports a new mobility standard, which allows the reconnection of communication very quickly, in addition, it also allows multihoming (multi-origin), ideal for users of smartphones, tablets or laptops. IKEv2 allows to use SCTP that is used in VoIP, it also has a simpler message exchange, it has fewer cryptographic mechanisms allowing only the most secure ones. What’s the point of using a VPN with insecure ciphers? IKEv2 only allows to use the most secure

Operating modes: transport or tunnel

IPsec provides us with two very different modes of operation, both for the authentication header (AH) and for the encapsulated security payload (ESP). These modes of operation differ in the way the packets are addressed. Next, we explain in more depth the differences between the two.

Transport mode

The AH or ESP header is inserted between the data area and the IP header, in such a way that the original IP addresses are kept. The content encapsulated in an AH or ESP datagram comes directly from the transport layer. Therefore, the IPsec header will be inserted after the IP header and just before the data provided by the transport layer. In this way, only the payload is encrypted and authenticated. The scheme of the datagram would be the following:

The transport mode ensures end-to-end communication, but the ends must be aware of the existence of the IPsec protocol in order to understand each other.

Tunnel mode

In tunnel mode, the entire IP packet (header + data) is encrypted and authenticated if ESP is used. This packet will be encapsulated in a new IP packet, therefore, the IP address will change to that of the last IP packet. Therefore, an AH or ESP header is added to the original packet, and then the IP header is added that will serve to route the packet through the network.

Tunnel mode is normally used to communicate networks with networks, but it can also be used (and is in fact used), to communicate computers with networks and computers with computers. This operating mode makes it easier for nodes to hide their identity from other nodes that are communicating. By using tunnel mode, we will be able to have a subnet specifically dedicated for VPN clients. Tunnel mode is used mainly by IPSec gateways, in order to identify the network that they protect under the same IP address, and thus centralize the processing of IPSec traffic on a computer.

In the following image you can see a comparison between both operating schemes, always using ESP:

Once we have seen the two modes of operation, we are going to see what authentication methods we have in IPsec.

Authentication methods

In the IPsec protocol we have a total of four authentication methods: shared key, RSA digital signatures, X.509 digital certificates and authentication through a group of XAuth users. Depending on the scenario where we want to implement IPsec, one authentication method or another will be used, each of these methods have advantages and disadvantages that will be mentioned. Next, we explain these four methods in detail.

Shared key

The shared key is a key made up of a string of characters (that of a lifetime) that only the two ends of the communication will know to establish the IPsec connection. Through the use of authentication algorithms (HASH), it will be verified that the keys are correct without the need for said keys to be revealed. For this method to be secure, there must be a key for each pair of participants in the communication. This type of authentication is not feasible for many participants, since there will be a very large number of keys.

Although hashes are used for the exchange of keys in the connection, before this connection, the keys must be at both ends of the communication, for this reason, we cannot know for sure if that key has been captured when it was sent. We can only ensure that it is safe if we deliver it by hand. This technique is useful for small networks, but for medium and large networks it is totally unfeasible.

RSA Digital Signatures

IPsec works with the IKE protocol for automatic key management and security, uses RSA digital signatures for the secure exchange of keys through the public and private key pair. These keys have the same problem as the shared key, somehow we have to send the keys to “the other side”, but we can safely modify the keys using the IKE protocol.

Therefore, to secure the network, it is advisable to change these keys on a regular basis. These RSA signatures provide the network with authentication and confidentiality.

X.509 certificates

One of the most secure forms of authentication in IPsec is to work with digital certificates, creating a public key infrastructure (PKI) with its corresponding CA (Certification Authority), the server’s digital certificate and the clients’ digital certificates. Thanks to these digital certificates, we can establish a very robust authentication, in addition, we can also work with digital certificates, these certificates contain the public key of the owner and their identification. The owner also has a pair of public and private keys to operate with at the time of validation.

The use of these certificates makes the PKI protocol appear on the scene to authenticate the nodes involved in IPsec communication. The use of this PKI helps in the task of creating new certificates and removing others. The validity of the digital certificate is granted by the PKI, this PKI integrates the CA that contains the public key and the identity of the owner. The endpoints involved in the IPsec connection will recognize the CA as valid, since they own a copy of that CA (the CA’s public key).

Certificate validation is done using the certificate revocation list (CRL) that is stored in the PKI. All participants will have a copy of this CRL that is constantly updated.

XAuth User Group Authentication

This method adds a user and a password to the digital certificates previously seen (X.509), in such a way that, apart from validating the certificate, it will also validate the user and password. To validate these users and passwords, we can use a Radius server, or directly a small database with the list of users and passwords.

Establishing the connection

The negotiation of an IPsec tunnel is carried out through the IKE protocol that will provide us with an encrypted and authenticated connection between the two ends of the communication. In the connection procedure, the keys and the security used to establish the IPsec connection will be agreed. The connection procedure is carried out in two distinct parts. We explain these two parts below.

1. Provide authentication and security to the connection

To secure the connection, a symmetric encryption algorithm and an HMAC signature will be used. Keys are exchanged using a key exchange algorithm, such as Diffie-Hellman. This method does not guarantee that the participants are who they say they are, therefore, we will use a pre-shared key or digital certificates.

The first part of the communication ends when the security parameters have been agreed, and the communication channel has been secured.

2. Provide confidentiality of data.

The IKE secure channel that we have established is used to negotiate specific IPsec security parameters (AH or ESP header, authentication algorithms, etc.). These specific parameters can include new Diffie-Hellman keys, to provide greater security, as long as we have configured the PFS (Perfect Direct Confidentiality), which is highly recommended to make the VPN even more robust.

Security services offered by IPsec

Confidentiality

The confidentiality service is obtained through the encryption function included in the ESP protocol. In this case, it is advisable to activate the authentication option, since, if the integrity of the data is not guaranteed, the encryption is useless. This is due to the fact that although the data could not be interpreted by anyone in transit, it could be altered by sending meaningless traffic to the recipient of the message that would be accepted as valid traffic.

In addition to offering traffic encryption, the ESP protocol also has tools to hide the type of communication that is being carried out; for this, it allows filling characters to be entered in the content of the packet data, so that the true length of the packet is hidden. This is a useful protection against traffic analysis techniques, which allow an attacker to deduce useful information from studying the characteristics of encrypted traffic.

Integrity and authentication of the data origin

The AH protocol is the most suitable if no encryption is required. The ESP protocol authentication option offers similar functionality, although this protection, unlike AH, does not include the IP header. As previously mentioned, this option is of great importance for those applications in which it is important to guarantee the invariability of the content of the IP packets.

Repeat detection

Authentication protects against IP spoofing, however an attacker could still capture valid packets and forward them to the destination. To avoid this attack, both ESP and AH incorporate a procedure to detect repeated packets. Said procedure is based on a sequence number included in the ESP or AH header, the sender increments said number for each datagram it sends and the receiver checks it, so that repeated packets will be ignored.

This sequence cannot be modified by the attacker, since it is protected by means of the integrity option for either of the two protocols (AH and ESP) and any modification in this number would cause an error in the integrity check of the package.

Access control: Authentication and authorization

Since the use of ESP and AH requires knowledge of keys, and these keys are distributed in a secure way through an IKE session in which both nodes authenticate each other, there is a guarantee that only the desired computers participate in the communication.

It is worth clarifying that valid authentication does not imply full access to resources, since IPSec also provides authorization functions. During the IKE negotiation, the flow of IP traffic that will circulate through the IPSec connection is specified. This specification is similar to a packet filter, considering the protocol, the IP addresses of the source and destination ports, the “TOS” byte and other fields. For example, IPSec can be used to allow access from a branch office to the local network of the center.

I do not repudiate

Non-repudiation service is possible using IKE with digital certificate authentication. In this case, the authentication procedure is based on the digital signature of a message that contains the identity of the participant. This signature, thanks to the link between the public key and the identity that the digital certificate guarantees, is unequivocal proof that an IPSec connection has been established with a specific computer, so that it cannot deny it. In practice, however, this test is more complex, as it would require storing the IKE negotiation messages.

L2TP / IPsec – what is this?

L2TP (Layer 2 Tunneling Protocol) is a protocol used for VPN that was designed by an IETF working group, as the heir to PPTP, and was created to correct the shortcomings of this protocol and establish itself as a standard. L2TP uses PPP to provide dial-up access, which can be tunnelled over the Internet to a specified point. L2TP includes the authentication mechanisms of PPP, PAP and CHAP, in addition, similar to PPTP, it supports the use of these authentication protocols, such as RADIUS.

Although L2TP offers multi-protocol supported access and access to remote local area networks, it does not have particularly robust cryptographic characteristics. The authentication operation is only performed between the endpoints of the tunnel, but not for each of the packets that travel through it. This can lead to spoofing somewhere inside the tunnel. Without checking the integrity of each packet, it would be possible to perform a denial-of-service attack using bogus control messages that terminate the underlying L2TP tunnel or PPP connection.

L2TP does not robustly encrypt user data traffic, thus causing problems when it is important to keep data confidential. Despite the fact that the information contained in the packets can be encrypted, this protocol does not have mechanisms for automatic key generation, or automatic key refresh. This can give someone who listens on the network and discovers a single key to have access to all the transmitted data.

Taking into account all these weaknesses of L2TP, the IETF made the decision to use the protocols of the IPsec protocol itself, to protect the data that travels through the L2TP tunnel. For this reason, they are always written “L2TP / IPsec”, because both protocols are used simultaneously, and this joint protocol is widely used. It could be said that L2TP is the protocol at the “link” layer level, and that it has no security, however, IPSec provides the security at the network layer so that the use of this protocol is secure.

For this reason, we will always find the L2TP / IPSec nomenclature together, because both protocols are used to have a secure VPN connection.

Conclusions

IPSec is an extraordinarily powerful and flexible security standard. Its importance lies in the fact that it addresses a traditional lack in the IP protocol: security. Thanks to IPSec, it is now possible to use IP networks for critical applications, such as business transactions between companies. At the same time, it is the ideal solution for those scenarios in which security is required, regardless of the application, so it is an essential piece in the security of IP networks. The IPSec protocol is already one of the basic components of security in IP networks today.