IPSec Fundamentals - Part I

IPSec (IP Security) is a set of features used to protect traffic on IP networks generally over an Internet connection.

Since IPSec connections are built over an existing IP infrastructure, you can think of it as a common transport protocol (Layer 4) such as TCP or UDP, but with much more security features of course. Anyway the point here is that the IPSec headers are always transported by Layer 3 connections through the network (if you think about it you will easily understand that if you want to make a secure connection between two hosts you have to be able to establish an IP connection first)


IPSec we said is actually a set of features and not a protocol, so let's take a look to the offered features:

• Data integrity (data has not been modified)


• Data origin authentication (sender and receiver are verified)


• Anti-replay (data cannot be replicated over the network)


• Data confidentiality (data is encrypted)

Each of these features can be implemented using open standard protocols to assure interoperability between different vendors. These protocols are:

• IKE (Internet Key Exchange)


• AH (Authentication Header)


• ESP (Encapsulating Security Payload)

IKE is used to negotiate parameters and to establish the connection between two hosts, while AH and ESP are used over that connection to provide some or all of the features listed above

Now we have an overview of IPSec, we know that is similar to a Layer4 transport protocol and is used to establish secure connections between two hosts over an unprotected network, it uses different protocol to fit security requirements, each protocol has its own function and limitations

IPSec connection can be made in two different modes:

• Transport Mode


• Tunnel Mode

Before discussing the differences between them, remember that IPSec headers are always encapsulated into Layer 3 headers, so anyone who would see our packets travelling over the unsecured network will always be able to see the content of this Layer 3 header (and it's required otherwise no IP connectivity would be possible).

To better explain the difference think of two hosts with public IP addresses connected to the Internet by two routers/firewalls. The hosts can communicate between them but we need to make these communication secure, so IPSec is the answer. Please consider that in this post I will not take in consideration the IPSec services available on servers like Windows Server 2003, 2008 or Unix-like. Each of these operating systems can support the creation of IPSec connections directly but it's not our scenario. We are going to protect traffic using router's capabilities only.

In Transport Mode, we tell the routers to protect all of IP traffic between the two hosts with IPSec starting from Layer 4, so all of the IP packets to or from these destinations would be modified as follows:

The Layer 3 IP Header is not modified and it contains the target host's IP


The Layer 4 (TCP or UDP) Header and the data are processed by the chosen protocol (AH or ESP)


IPSec encapsulates into a new AH or ESP header the processed data


The outgoing packet will contain the original Layer 3 header, the AH or ESP Header, the original Layer 4 header and the data


In Tunnel Mode, we tell the routers to protect all of IP traffic between the two hosts with IPSec starting from Layer 3, so IP packets between them will be modified as follows:


The Layer 3 Header containing the target host's IP address, the Layer 4 (TCP or UDP) Header and the data are processed by the chosen protocol (AH or ESP) IPSec encapsulates into a new AH or ESP header the processed data IPSec creates a new Layer 3 Header containing the remote router's IP address


The outgoing packet will contain the new Layer 3 header, the AH or ESP header, the original Layer 3 header, the original Layer 4 header and the data




As you can easily see Tunnel Mode not only provides protection for Layer 4 communications, but it protects Layer 3 headers too with the main advantage to hide actual IP addresses inside the protected traffic leaving routers' IP addresses exposed to potential unauthorized intruders

Still consider that not all of the IPSec protected traffic is encrypted, encryption is just one of the features of IPSec (even if important). Since any of the capabilities offered by IPSec require overhead and router's workload, encrypting all of the traffic is usually not a good idea.