Method and apparatus for preventing a denial of service attack during key negotiation

The invention provides a method for preventing a denial-of-service attack on a responder during a security protocol key negotiation. The responder receives key negotiation requests designating a source port and source IP address. The responder only maintains state when a key negotiation request is received from an initiating computer with a valid, non-spoofed, source IP address. The responder further limits the number of in-process key negotiations for which the responder maintains state. If a key negotiation request is received from a valid source IP address and the responder has at least one established security association for that source IP address, the responder limits the number of ongoing key negotiations to a maximum number on a per port address basis for that source IP address. If an established security association does not exist for that source IP address, the responder limits the number of ongoing key negotiations to a maximum number based on the source IP address regardless of the source port address.