General Troubleshooting Guidelines
Regardless of the type of problem you are experiencing, you should first make sure that the necessary services are started and set to automatic on both IPSec peers. On computers running Windows Server 2003, the IPSec Services service must be started. On computers running Windows 2000, the IPSec Policy Agent service must be started. Sometimes, especially after making significant changes, you might be able to resolve a problem by restarting IPSec services. This completely clears the IKE negotiation state. You can restart IPSec services from a command prompt by running the following commands:
net stop policyagent
net start policyagent
This is simply a quick way to restart IPSec without restarting the computer. After restarting the IPSec services on both computers, attempt to establish a secure connection. If the problem persists, restart the operating systems on both IPSec peers and try again.
Kerberos Authentication Problems :
Kerberos authentication is the default IPSec authentication method. You can quickly identify whether IPSec connectivity problems are caused by authentication by temporarily changing the IPSec authentication method on both IPSec peers to Preshared Key. If IPSec communications succeed with Preshared Key, Kerberos authentication is probably the source of the problem.
For Kerberos authentication to be successful, both IPSec peers must have valid computer accounts in trusted domains, and they must be able to authenticate the remote computers. Each IPSec peer must be able to communicate with domain controllers without having the authentication requests filtered. In earlier versions of Windows, IPSec automatically allowed Kerberos traffic. However, the Kerberos protocol is no longer a default exemption in Windows Server 2003.
Certificate Authentication Problems :
Certificates are a common method for authenticating computers that are not in a trusted domain environment. If you are experiencing problems with IPSec and want to verify that the problem is related to authentication, temporarily change the IPSec authentication method on both IPSec peers to Preshared Key. If IPSec communications succeed with Preshared Key but fail with certificates, the problem is almost certainly related to certificates.
If you have multiple rules in a policy, double-check that those rules will use the same authentication method consistently for any single remote computer. It is acceptable to have a policy that configures Kerberos authentication for hosts on an internal network and uses certificates for hosts on an external network. However, you cannot create one rule that uses Kerberos to authenticate just Transmission Control Protocol (TCP) data and a second rule that authenticates User Datagram Protocol (UDP) traffic by using certificates, for example. The IP Security Policy Management snap-in will not prevent you from creating these rules, but they will not work properly. All rules that apply to a single remote host must use a single authentication method.
Troubleshooting Firewalls, Routers, and Packet Filtering :
Packet filtering at firewalls is a common source of IPSec problems because IPSec cannot be permitted or blocked by applying the techniques used for most applications. First, your firewall must allow two-way traffic with a UDP destination port of 500. If the firewall is also a NAT server and you will be using Network Address Translation Traversal (NAT-T), you must also allow UDP traffic with a destination port of 4500. Second, the firewall must allow traffic with an IP protocol ID of 50, which is used by ESP. If you are using AH instead of ESP, you must allow IP protocol 51.
Network Address Translation Problems :
Network Address Translation (NAT) is a common technique for connecting a privately numbered internal network to a public network such as the Internet. As Chapter 8 discussed, earlier implementations of IPSec were not compatible with NAT. This makes sense, because NAT’s purpose is to modify the source or destination IP address in a packet without the client or server being aware, and part of IPSec’s purpose is to discard packets that have been modified in transit.
Lesson 3: Troubleshooting IPSec
