TheTAZZone - Internet Chaos

Tutorial-Using IPSec to Secure Computers and Network Traffic


Do not use, republish, in whole or in part, without the consent of the Author. TheTAZZone policy is that Authors retain the rights to the work they submit and/or post…we do not sell, publish, transmit, or have the right to give permission for such…TheTAZZone merely retains the right to use, retain, and publish submitted work within it’s Network

Code: Select all
Tiger Shark has kindly given his permission for his tutorial to be hosted at The Taz.Enjoy


One of the most overlooked security features in Windows 2000 and Windows XP is IP Security or IPSec for short. Between Active Directory, (AD), Domain members setting up encryption of network traffic between servers and clients, server to server or client to client is actually rather easy. Even setting up encrypted connections between non-AD Domain computers is quite simple. IPSec can also be used to create secure tunnels but I will not be covering that in this tutorial. IPSec can also be used to lock down a computer in much the same way as a personal firewall and in many cases it can be equally effective and be less obtrusive subsequently. Windows IPSec driver resides just before, or just after for incoming traffic, the transport layer so it is transparent to both the applications that end up using it and therefore the users. IPSec policies can be controlled at the domain level in an AD environment, Under Start – Programs – Administrative Tools – Local Security Policy or Start – Run – mmc.exe – Console – Add/Remove Snap-ins – Add – IP Security Policy Management.

IP Security concepts:

IPSec has five basic concepts that are required to be understood in order to effectively and efficiently manage an IPSec environment.

1. Policies: Policies are a single entity that encapsulates the entire IPSec ruleset. Only one policy can be assgned at a time but a policy can contain numerous Filter Lists and Filter Actions.

2. Filter Lists: Filter lists are where we specify the traffic that will be covered by the policy. In a filter list you could specify “All traffic to this IP address from that IP address on port 80”. As you can see, the list says nothing about what to do with the traffic, it merely specifies the traffic to be acted upon by a Filter Action.

3. Filter Actions: Filter Actions are fairly limited. They allow you to Permit, Block or Negotiate Security. In addition you are allowed to tell the system to Require Security or to allow fallback to unsecured communication.

4. Mirroring: This automatically creates an opposite filter. Without this you would need to create two filters, one allowing the outbound and the other allowing the inbound traffic. You need to be careful with this though because in the Wizards it is checked by default. This could lead to a situation where you want to allow all outbound traffic from your machine to any IP address but block all inbound. If you left the “Mirror” checked it would also allow all connections from any IP address to your IP address… Not good….

5. Security Rules: Security rules are the combination of a Filter and a Filter Action. Thus if I create a filter called “HTTP Traffic” with a Filter Action “Permit” the combination of those two become a Security Rule. You can manage Filter Lists on their own and you can manage Filter Actions on their own but it is the combination of the two within a Policy that lay down the rules governed by that policy.

Creating Basic Encrypted Connections:

IPSec can use three methods to negotiate the security and authenticate each other. Kerberos, (the default in an AD domain and the easiest to use), a Certificate Authority, (CA), or a Pre-Shared Key, (PSK, most often used between non-domain members). It is my understanding of kerberos that a third, trusted by both computers, computer is required to use kerberos so this is not likely to work between two standalone computers so the easiest to use is the PSK which can be any word or phrase or combination of characters you chose just like a password). It is important to note that settings must be created on both ends of the connection. In this example I will show you how to set up secured traffic between a private Web server on a LAN and a client in an AD domain, (both are domain members) and we will be creating our policy from scratch rather than adjusting the three built in policies.

The Server Policy:

In the IPSec management console right click and select Create IP Security Policy to start the Wizard. Select Next and name your policy “Tiger’s Custom Policy” and give it a meaningful description so that when the next admin comes along he has a clue as to why this is there. Click Next and uncheck the Activate the default response rule and click next again and Finish. This will create the policy and automatically begin editing it. You will notice that the “Dynamic/Default Response” Filter List is there but it is unchecked and it can’t be removed.

The Server Security Rule:

Click Add to start the Security Rule Wizard and click Next. Since we are not creating a tunnel leave this page as “This rule does not specify a tunnel” and click next. Now you have a choice of “All Network Connections”, “LAN connections” or Remote Access. All connections refers to all connections you have set up in the network properties and includes NIC’s, Modems etc. LAN connections refers to all NIC’s you have set up and Remote Access refers to all Dial-up/Modem connections you have set up. For this example we have no modem so you can leave it as either All connections or LAN Connections and click Next. This brings you to the Authentication Method Panel where your three options are Kerberos, CA or PSK. Since this is an AD domain we can leave it at Kerberos and click Next.

The IP Filter List:

This brings you to the IP Filter List where you tell the Rule what is to be filtered. The defaults built in are “All ICMP, (Ping), traffic” or “All IP Traffic” and both are unselected. You can only select one filter list per Security Rule and in this case neither is suitable for the traffic we want to secure so we will have to create our own. Click Add to bring up the IP Filter List Wizard where we will tell the Rule what traffic is involved in the rule. Name it “HTTP Traffic” and give it a meaningful description and click Add to bring up the IP Filter Wizard and click Next. A drop down box with five apparently self explanatory choices appears for the Source Address. When I say apparently it’s because there is a little trap here that one could easily fall into and cause yourself some grief for a while. On a multi-homed NIC, (A NIC with several IP addresses), or on a server with multiple network cards the selection “My IP Address” refers to _all_ IP addresses assigned to this machine thus this filter will be applied to all the IP’s. If you have a public web site on one address on a multihomed NIC and a private web site for internal use on another IP on the same multihomed NIC you will bring down your public web site the moment you assign the policy… beware. Use the “Specific IP Address” choice for multihomed or multi NICed servers. In this case we have a single IP on a single NIC so we will leave it as “My IP Address” and click Next. The next screen is the destination address and it has the same five choices the source address had. Select the “Any IP Address” choice, (the default), and click next. Select TCP for the Protocol Type and click Next. Select “To this Port” and enter 80, (HTTP), in the box and click Next then Finish. I like to leave the “Edit Properties” box checked here so that I can add a description which is handy when you have larger Filter Lists. When you have added your description you will need to determine whether you want the mirrored checked. In this case we do because otherwise the incoming requests will not be able to pass so the web site would appear down. When you click Close on the IP Filter Wizard you will see your new Filter List named “HTTP Traffic”. check it and click Next.

The Filter Action:

This takes you to the Filter Action, (what to do when traffic of this nature is encountered). The three default options are there, (Permit, Request Security and Require Security). Since this is the first time we are using the Policy Manager I like to click Add and create a new Filter Action called “Block” with the Action “Block” so all my possibilities are available in the future. In this case where we want to require security we will check the Require Security button and click Next.The difference between request and require security should be clear but it is important to remember that this server will not communicate with any other computer if the setting is “Require” unless they successfully negotiate the secure connection whereas “Request” asks for the secure connection but communucations will be established with clients that are unable to negotiate but the communications will be in plain text and therefore easily sniffable and read. Clicking Close drops you back to the Management Console with “Tiger’s Custom Policy” added to the list of Policys available to you.

The final step to enforcing any policy is to right click the policy and click Assign, (to un-enforce a Policy do the same but the “Assign” will read “Un-Assign” for an already assigned policy. If you have an assigned Policy and you try to assign a different one you will notice that the one that was assigned automatically becomes unassigned. However, assigning a policy is also a matter of timing. Since we haven’t set up the IPSec Policies on the clients yet none of them will be able to negotiate security with this server when you assign and therefore your private web site will go down until they are aware of the policy. In our AD environment this is quite easy. We would carry out the steps to create the policy on the client in the AD Group Policy and assign it there. Once you have assigned it you will need to wait for the amount of time you have set in AD for the clients to check for changes in policy, (30 minutes default IIRC), and then assign the policy at the server level. In this case we will set the client individually because there are differences.

The Client Policy:

On the client create the new policy called “Tiger’s Client Policy”. Start the Security Rule Wizard and follow the instructions as you did on the server. When you get to the IP Filter List is where things change. Since this is one single server that we want to force a secure connection with but still allow normal unsecured communication with the rest of the web we have to specify the server as the one to negotiate with. So the source address will be “My IP Address”, (on a single NIC or non multi-homed PC), but the destination address should be specified as the IP address of the Private Web Site. Continue on through the wizards assigning the same things you did for the server and complete the process by right clicking the Policy and “Assigning” it. You should then find that you can reach any other web site except your Private Web Site because we haven’t assigned the policy yet at the server end. Go back to the server and assign the policy and upon your return to the client you will find that you can now communicate with it.

Confirmation that the Traffic is Secure:

On either or both machines you can confirm the IPSec connection by selecting Start – Run – ipsecmon. This brings up the IP Security Monitor that shows you all the Security Associations, (SA’s), that your computer currently has. You should see the connection between the server and the client on both machines. Another way to conform the communication is secure is to run a packet sniffer such as Ethereal and capture the packets. If the communication is secure you will see lots of “Other” protocol packets being captured and upon analysys you will see that they are of the ESP, (Encapsulated Security Protocol), type indicating that your IPSec is, indeed, functioning. Out of interests sake when you look at an ESP packet you will notice that it is structured just like any other packet of the same original protocol, (IP, TCP etc.), insofar as you can see the MAC address, source and destination addresses, flags etc but when you look at the payload portion of the packet it is encrypted, (that’s what the “encapsulated” means. It’s an encrypted payload encapsulated within a normal packet.

From this simple example you can see that the process is very akin to creating firewall rules. You have a source address/domain name/subnet, a destination address/domain name. You can specify protocols and source and destination ports and you can specify the actions. Once you have been through the wizards a couple of times you will find that it is really rather intuitive to begin setting up quite complicated rule sets within a policy.

A Small Issue you should Note:

Word of warning before we start looking at how to make more complicated rulesets to lock down a computer. IPSec can lose packets in non-reliable protocols such as UDP or ICMP, (protocols that do not require confirmation of the connection or receipt of the data). This can occur on initial transmissions and when the SA “dies” when not in use and has to be renegotiated. The reason is that a computer that uses the “Client (Respond Only)” policy that is already provided in the Management Console attempts a normal communication and is then challenged by a server that requires security. Those first few packets from the client are dropped while the security negotiation takes place. This is ok when using a reliable protocol such as TCP that will retry the three way handshake after the SA has been established but in unreliable protocols the receiving server dropped the packets because no SA was in force with the client at the time of the receipt of the packets. This can most easily be demonstrated by setting up the server to require security for all ICMP traffic and assigning “Client (Respond Only)” on the client machine. Then try to ping the server from the client. Instead of the usual four ping replies you will receive four messages saying “Negotiating IP Security”. Notice, your pings were dropped. The same would occur if you were using Syslog, (via UDP), to send log messages securely to a server. Your first message, (or messages if they followed in quick succession), would not reach the logs because the SA has yet to be established. You can avoid this situation by using a “Require Security” Filter Action for those unreliable protocols at the client and server end. When the require security setting is set at the client the data packet is held at the client until the SA has been negotiated and is passed subsequent to the SA being set up so no data is lost. Murphy’s Law clearly states that those two or three log items you lose would be the ones you really needed…

Securing a Computer:

Ok, with the warning out of the way and the understanding that we can refer to the Security Rules in a similar way as we can for most firewalls let’s take a look a a more likely scenario. Let’s assume that we can’t afford a firewall so the server will have to be secured by IPSec alone. But let’s also assume that this is a gateway server with two NIC’s, one to the public network with an IP address of, (for this example), and the other with an address of connected to the private network It is the email server for the company and also provides a public Web site and DNS for the internal workstations. We also want the email within the network to pass securely, (encrypted), so Bill can’t sniff Sue’s email straight off the wire.

Remembering that IPSec applies less specific rules after it applies the specific rules we create a policy and set up the following Security Rules:-

1. Allow incoming and outgoing mail from the public network, (SMTP on port 25), to pass. So we set up a rule stating TCP on port 25 is allowed, (Permit), from to Any IP address and it is mirrored, (allowing the incoming mail).
2. Allow mail, (SMTP), from the private network to pass from the network to the server secured. So we set up a rule stating TCP on port 25 is secured, (Requires Security), from with a subnet mask of to and it is mirrored.
3. Allow POP3, (Post Office Protocol 3 on port 110 so people can collect their mail), from the private network to pass securely. So we set up a rule stating TCP on port 110 is secured from, (subnet mask of, to and it is mirrored.
4. Allow DNS requests, (port 53 on both TCP and UDP), to be made from this server to the public network. This requires two rules. The first states that TCP on port 53 is allowed, (Permit), from to Any IP address and it is mirrored, (We aren’t providing DNS so the port is closed but we have to allow the responses back in). The second rule states UDP on port 53 is allowed from to Any IP address and is mirrored.
5. We need to allow the web server to provide the web content, (HTTP on port 80) to the public network so we create a rule stating that TCP on port 80 is allowed, (Permit), from Any IP address to and it is mirrored.
6. To prevent all other communication with the public network we need a final rule. This rule states that All protocols from Any IP address to are prevented, (Blocked), and it is mirrored, (we don’t need the server to talk outbound to the public network for any other reason than the ones we have previously stated).

These rules allow the following traffic on the public network:-

1. Incoming and outgoing mail, (SMTP).
2. Incoming and outgoing DNS, (but there is no DNS server available so the port appears closed).
3. Incoming and outgoing Web access.
4. Nothing else

On the private network they allow for:-

1. Secure transmission of outgoing mail, (SMTP), from the clients to the server
2. Secure reception of mail, (POP3), to the clients from the server
3. Normal communication with the server on all other protocols and ports.

The final step in this process when using IPSec to secure a computer is to bring good old NMap out of the toolbox and let it do it’s thing. It’s easy to make a small mistake, (especially with the mirror setting), that will leave your computer wide open to the world. If things aren’t quite right it’s easy enough to determine what needs to be tweaked to close a hole or sixty five and a half thousand…..

Note for the “Real World”:

This is obviously a pretty simple example and it really wouldn’t be a very secure set up. However it should be realized that IPSec can provide very granular control over exactly what a computer can and can’t do within a networked environment. In a real world situation where you want a server to be thoroughly locked down one would start with a traffic analysis on all the interfaces the machine has. A traffic analysis uses some kind of protocol analyzer or packet sniffer, (TCPDump, WinDump or Ethereal), where all the traffic to and from each interface is captured for as much as a 24 hour period, (remember that on a busy network these captures can be huge – in some cases it might be as well to run the capture for a few minutes and make a note of the really common traffic and filter it out of the capture. When more traffic is subsequently captured do the same again and keep repeating till no more traffic is captured). Then, when you have all the data you can determine which is required and which are undesirable forms of traffic. From that point you can build your Security Rule Set and apply it knowing that only the appropriate traffic for the role of the machine is being allowed in or out of it.

Leave a Reply

Your email address will not be published. Required fields are marked *


If you'd like to advertise on The Mutt ( aka ) feel free to contact us at: administration[at]

TheTAZZone is a non-commercial entity. We do not sell any products or services ourselves. Our revenue comes from advertising and donations only.

We appreciate your support! Your advertising revenue ( or donations ) helps us to continue to upgrade, improve, and offset the costs of maintaining this site.

Donations can be made through the page ' Donate '.