Manage Learn to apply best practices and optimize your operations.

Best practices for implementing IPSec policies

There are several tricks to configuring IPSec effectively, depending on what Windows servers you're using. Find out what they are in this tip by site contributor Brien Posey.

The following is the second part of a two-part series. Click for part one.

As I explained in my previous article, the IPSec protocol helps you encrypt data flowing across your network, but it is not universally supported. In this article, I will outline some best practices for implementing IPSec to help you gain the maximum-security benefits without compromising compatibility.

How IPSec works

Before I show you the best way to implement IPSec policies, there are two things you must understand. First, IPSec is implemented at the group policy level. Group policies are hierarchical, and multiple group policy elements can be combined to create the overall system policy. This means you can create multiple IPSec policies for multiple situations rather than have one blanket policy for the entire system.

Second, IPSec is not universally compatible. Microsoft first introduced IPSec in Windows 2000, so systems running Windows 2000, XP and 2003 can use IPSec. However, systems running Windows ME, 98 or 95 can not. Furthermore, any systems running non-Microsoft operating systems, such as Linux, Unix or NetWare, do not support IPSec.

That in mind, the trick to configuring IPSec effectively is to set up a series of group policy elements. The goal is to have Windows use IPSec where supported, and not require IPSec where it is not supported. This is not as hard to do as it sounds because Windows has three IPSec policies built in.

Windows built-in IPSec policies

The built-in IPSec policies are Server, Client and Secure Server, but the naming scheme is a bit misleading. For example, a Server policy does not have to be used on a server; it can be used on clients as well. What is important is what the policies contain.

Server policy: This is a request-security policy. Any system that this policy is applied to will request IPSec encryption for any communications session between that machine and another computer. If the other computer doesn't support IPSec encryption, the session is allowed to remain unencrypted.

Client policy: This is a response-only policy. It will never request IPSec encryption. However, if another computer requests IPSec encryption, a system with the Client policy will respond by allowing the session to be encrypted.

Secure Server policy: This policy requires IPSec encryption. It does not support any non-encrypted sessions.

Implementing IPSec policies

Now that you know about the various types of IPSec policies available to you, let's look at how to put everything together. The proper method depends on what type of systems make up your network. For the purpose of this article, I will assume that you have all Windows 2000 and Windows 2003 servers with workstations running a wide variety of operating systems.

For a setup like this, you could technically apply the Server IPSec policy to every machine in the organization. That way every machine capable of running IPSec would request IPSec encryption for every session. Remember, though, the Server policy offers no guarantees. If a computer refuses to use IPSec encryption then the session will be unencrypted. It would be nice to have some guarantees.

I recommend setting up a group policy so all servers require IPSec encryption for communications with each other. That way you know inter-server communications will be secure. You can then use optional encryption for everyone else.

The procedure for creating such policies is fairly straight forward. The Active Directory already has container objects for servers, domain controllers and computers. Simply create two different group policy objects, one based on the Secure Server policy and one based on the Server policy. The group policy object based on the Server policy can be applied to the computer container without modification.

However, the policy object based on the Secure Server policy can not be applied as is. If you applied the policy in its current form, all server communications would be encrypted. Instead, you must use the policy object's filter capability to build a filter that requires IPSec encryption for communication between any of the servers. To do this, base the filter on an IP address or a network segment. You also need to design the policy to request security for anyone not on the filter list. Then encryption is guaranteed for communications between two servers, and encryption is requested (not required) for communications between a server and a workstation.

For more information regarding IPSec best practices, check out the information at Microsoft's Web site.

Dos and don'ts for working with new IPSec options in Windows Server 2003

  • Do not use preshared keys
  • Do not use Diffie-Hellman Group 1 (low-key strength)
  • Do use Triple Des (3DES) for stronger encryption
  • Do consider applying IPSec policies locally just in case domain policies are ever unavailable
  • Do not use Kerberos as an authentication method if a computer is directly connected to the Internet

Brien M. Posey is a regular contributor on

Dig Deeper on Microsoft Hyper-V management