What Is IPsec? Everything You Need to Know About IPsec

IPsec has been mentioned a lot lately. It is the security protocol used together with L2TP to set up VPN connections. In this article, let's Mona Media take a closer look at what IPsec is.

What is IPsec?

IPsec is short for Internet Protocol Security, a standard suite of protocols defined by the IETF (the Internet Engineering Task Force).

IPsec provides authentication, integrity and confidentiality for network connections IP between two communicating points. It also defines the packets for encryption, decryption and authentication, and the protocols needed for secure key exchange and key management.

Uses of IPsec

IPsec can be used for tasks such as:

• Encrypting application-layer data.
• Providing security for routers that send routing data across the public internet.
• Providing authentication without encryption, such as verifying that data comes from a known sender.
• Protecting network data by setting up circuits that use IPsec tunnels, in which all data sent between two endpoints is encrypted, as with a Virtual Private Network (VPN).

What components does IPsec include?

1. Encapsulating Security Payload (ESP)

ESP provides integrity, encryption, authentication and anti-replay by encrypting the IP headers and payload of each packet. In Transport mode, however, only the payload is encrypted.

2. Authentication Header (AH)

AH provides data integrity, authentication and anti-replay just like ESP, but it does not provide encryption. Anti-replay protects against unauthorized retransmission of packets by numbering them in sequence, which blocks hackers from forging packets. Note that anti-replay does not protect the confidentiality of the data.

3. Internet Key Exchange (IKE)

IKE, or Internet Key Exchange, is a network security protocol designed to exchange encryption keys through a Security Association (SA) between two devices. A Security Association sets up the security attributes shared between two network entities to support secure communication.

The key management protocol (ISAKMP) and Internet Security Association provide a framework for authentication and key exchange. ISAKMP specifies how to establish a Security Association (SA) and how to connect directly between two hosts using IPsec.

IKE protects packet contents and is also an open framework for implementing standard algorithms such as SHA and MD5. The IPsec algorithm builds a unique identifier for each packet, which lets a device determine whether the packet is genuine. Unauthorized packets are dropped and not delivered to the recipient.

How IPsec works

IPsec works in four phases:

Phase 1: Identify interesting traffic

After a network device receives a packet, it matches the packet's 5-tuple against the configured IPsec policy to decide whether the packet needs to go through an IPsec tunnel. Traffic that must pass through an IPsec tunnel is called interesting traffic.

Phase 2: Negotiate the Security Association (SA) and key exchange

An SA defines the elements for transmitting data securely between the communicating parties. These include the security protocols, the data encapsulation mode, the encryption and authentication algorithms, and the keys used to transmit data.

After identifying interesting traffic, the local network device starts negotiating an SA with its peer device. In this phase, the parties use the Internet Key Exchange (IKE) protocol to set up an IKE SA to authenticate identities and exchange key information, and then set up an IPsec SA to transmit data securely based on the IKE SA.

Phase 3: Data transfer

Once the IPsec SA is established between the parties, they can send data through the IPsec tunnel.

To keep data confidential in transit, either Authentication Header (AH) or Encapsulating Security Payload (ESP) is used to encrypt and authenticate the data. Encryption keeps the data confidential and prevents it from being intercepted in transit. Authentication ensures the integrity and reliability of the data and prevents it from being forged or tampered with in transit.

The IPsec sender uses an encryption algorithm and an encryption key to encrypt an IP packet, which means it encapsulates the original data. The sender and receiver then use the same authentication algorithm and authentication key to process the encrypted packets and obtain an integrity check value (ICV). If the ICVs at both ends match, the packet was not tampered with in transit and the receiver decrypts it. If the ICVs differ, the receiver drops the packet.

Phase 4: Termination

This is the final step and involves ending the IPsec secure channel. Termination happens when the data exchange is complete or the session times out. The cryptographic keys are also discarded. To save system resources, the tunnel between the two parties is torn down automatically once it reaches its idle timeout.

How IPsec operates

Operating modes

• Tunnel mode: In Tunnel mode, the entire packet is protected, including the IP header and payload. IPsec wraps the data inside a new packet, encrypts it and adds a new IP header. It is commonly used to set up site-to-site VPNs.
• Transport mode: In Transport mode, the original IP header stays in place and is not encrypted. Only the payload and the ESP trailer are encrypted. Transport mode is commonly used to set up client-to-site VPNs.

How IPsec operates

IPsec works in 5 steps:

• Step 1: The host checks whether an outgoing packet should be sent over IPsec. The sending system applies a suitable encryption method to these packets. The host also checks incoming packets to see whether they were encrypted properly.
• Step 2: IKE phase 1 begins, in which the two hosts (using IPsec) authenticate each other to start a secure channel. The channel has two modes: Main mode, which provides stronger security, and Aggressive mode, which lets the hosts set up the IPsec circuit faster. This channel is then used to securely exchange how the IP circuit will encrypt all the data.
• Step 3: IKE phase 2 runs over the secure channel. The two hosts negotiate which encryption algorithm will be used in the session and exchange the secret keys to be used with those algorithms.
• Step 4: Data is exchanged through the newly created IPsec encrypted tunnel. These packets are encrypted and decrypted by the hosts using the IPsec SA.
• Step 5: When communication between the hosts is complete or the session expires, the IPsec tunnel is ended by discarding the keys on both machines.

Differences between IPsec and SSL

Both are common security protocols used for VPN connections, but IPsec and SSL certificate are completely different, and it is important to tell the two apart.

We hope this article has given readers a better understanding of the protocol suite what IPsec is and how it is used. We will keep sharing more useful information about information technology in general and computer networks in particular every day.