[vpn-devel] SHA2 Inteoperability issues?

[Paul Wouters]

On Mon, 20 May 2013, Mark Larwill wrote:

> I'm a developer at WatchGuard and I'm working on adding SHA2 support for SHA2-256, SHA2-384, and SHA2-512 to our firewall. I have
> noticed that when testing with NCP's mobile VPN client, everything works. When I test with Shrew's client it does not work. Here's
> what I know:
> 1. Phase1 and Phase2 both complete successfully
> 2. If I send ping traffic, Shrew does generate ESP packets
> 3. ESP packets are dropped at the firewall and a log message is generated saying that "Integrity check failed"
> 
> May 20 20:45:30 2013 Firebox local0.warn kernel: [ 1553.257472] WG: Packet rejected. Gateways (src 4.4.4.1 dst 8.8.8.1) Selector (src
> 77.0.0.1 dst 0.0.0.0) spi 0xb5d723ce proto 50 reason "Integrity check failed"
> 
> I understand that the problem could be on either side, so at this point it is not clear if the Shrew client is doing something wrong,
> or if the firewall is. However if I had to make an educated guess, I would speculate that maybe the shrew client is sending the full
> SHA2 hash, instead of the truncated value (per RFC4868). This could cause the integrity check to fail. For example if using SHA2-256
> and sending a 100 byte packet, with a full 32 bytes of hash, for a total of 132 bytes, the firewall might assume it's a 116 byte
> packet with 16 bytes of truncated hash.

The Linux kernel implemented sha2_256 wrong. It used a draft instead of
the final RFC document. They then fixed it using a different xfrm_algo
struct in the kernel. So if your software uses the old method, it will
fail with RFC compliant devices. If your software uses the new method,
it will fail with older Linux kernels. Note that SHA2-384 and SHA2-512
were correctly implemented, so you can test with that see if this is
indeed your problem (assuming you use shrew on linux)

For libreswan/openswan, we introduced the sha2_truncbug= option to be
able to do both:

        sha2_truncbug
            The default hash truncation for sha2_256 is 128 bits. Linux
            implemented the draft version which stated 96 bits. This option
            enables using the bad 96 bits version to interop with older linux
            kernels (unpatched version 2.6.33 and older) and libreswan versions
            before 2.6.38. Currently the accepted values are no, (the default)
            signifying default IETF truncation of 128 bits, or yes, signifying
            96 bits broken Linux kernel style truncation.

You can find the code in openswan/libreswan in programs/pluto/kernel_netlink.c

Paul