[vpn-help] Require help with troubleshooting samba shares over ipsec

[Matthew Grooms]

On 8/29/2010 7:08 PM, Brett Morrison wrote:
> Hi Matthew
> Have given this another go from home on the weekend, appears that MTU
> may be the issue.  The 1500 byte ping does not work, dropping it below
> 1000 works though, so I am assuming the MTU is set too high on the
> server?
> I'll bounce this off of our app server hosting guys.

Hi Brett,

MTU problems are sometimes difficult to track down when using IPsec 
connections. The problem is that ESP or UDP/ESP will add headers to 
packets during encryption/encapsulation. This causes packets to be 
fragmented if the host uses a full 1500 byte MTU which is the default.

For TCP sessions, this isn't much of a problem as the MSS ( Maximum 
Segment Size ) is negotiated between the two connection endpoints. The 
VPN client sets its virtual adatper MTU to 1380 by default so that the 
MSS value used is lower. This allows final packets to fit in a standard 
Ethernet frame even with the additional IP headers. Protocols that use 
UDP ( such as CIFS/SMB ) can be more troublesome as there is no analog 
to a TCP MSS. Instead, the adapter MTU is used and you end up with full 
1500 byte packets being sent from a server back to the VPN client. A VPN 
gateway is typically smart enough to handle this without too much 
headache. It just fragments the packet ( either before or after IPsec 
processing ) and then forwards them to the peer. If you rolled your own 
gateway ( using Linux/BSD and an open source IKE daemon ), special care 
needs to be taken to ensure the outbound packet fragments will traverse 
the tunnel back to the client. If a DF bit is set in a packet that 
should be encapsulated, the gateway can still cope. It will either ...

A) Ignore the DF bit and fragment the packet anyway.
B) Send an ICMP 'Can't Fragment' message back to the server ( PMTU ).

If the gateway is also a firewall, it needs to make sure that its not 
filtering ICMP messages back to the server so that it will know to lower 
its packet size for (B). Otherwise, they will never reach the client. 
Some firewalls also need special rule modifiers so that it knows to 
match fragments as a whole, or they will be dropped. If you don't have 
control over the VPN gateway, you can try to lower the MTU on the server 
itself ( at least as a test case ).

Hope this helps,

-Matthew