[vpn-help] Apple OSX >= 10.4 DNS resolver, very disappointing ...

[Matthew Grooms]

All,

As most of you are probably aware, I have been working hard to improve 
the OSX port lately and have run into some very interesting challenges. 
If your not particularly interested in OSX, you should skip this email 
as its mostly a rather long rant about how Apple screwed the pooch WRT 
DNS resolution and why OSX will be 2nd class citizen for remote access 
products until they fix some things under the hood. I usually don't go 
out of my way to write about this kind of topic, but I thought it would 
be a good exercise for myself and perhaps a decent read for others that 
have some interest in the topic. Now, on to the brain dump ...

Back in the OSX Tiger days, Apple introduced a new method to configure 
adapter settings ( like DNS ) which leverages information stored in the 
configd dynamic store. The store is a nexus for several components that 
are interested in the same configuration data. For example: There are 
GUI config components that publish IP/DNS settings to the dynamic store 
and there are service components, like the DNS resolver, that read 
settings from the dynamic store. On the command line, the configuration 
information in the dynamic store can be modified or queried using the 
scutil program.

So what does this have to do with the VPN Client? Well, on *nix like 
systems, the Shrew Soft client assigns IP address info using a tried and 
true unix staple, ioctl(). DNS settings are written out to a file named 
/etc/resolv.conf. Route table manipulation is handled by sending and 
receiving messages via a special socket and protocol family named 
PF_ROUTE on BSD and PF_NETLINK on Linux. Since OSX had grafted a fair 
amount of code from the BSD family onto the Mach kernel, configuration 
used to be pretty similar to a BSD system. But with the introduction of 
Tiger, all this changed to look a lot more like a Windows system. You 
write a bunch of information into the registry ( or configd on OSX ) and 
the system does stuff with it. For example, in Windows you have a 
registry section that correlates to an adapters IPv4 configuration. It 
holds information like the IP address, Netmask, DNS settings, etc. You 
plug the information in, enable the adapter, and it's configured. With 
OSX it's pretty much the same thing, but unfortunately the system DNS 
resolver doesn't honor one of the fundamental aspects of TCP/IP, the 
route table.

To configure an adapter via the command line on OSX, you specify some 
information related to an adapter IP config. For example, in a shell you 
could do something like this ...

 >scutil
  open
  d.init
  d.add DomainName shrew.prv
  d.add ServerAddresses * x.x.x.x
  set State:/Network/Service/tap0/DNS
  d.init
  d.add InterfaceName tap0
  d.add Addresses * y.y.y.y
  d.add SubnetMasks * 255.255.255.0
  d.add Router z.z.z.z
  d.add OverridePrimary # 1
  set State:/Network/Service/tap0/IPv4
  close

Awesome, the adapter is configured and we have DNS settings. But how do 
I make the DNS settings active? Well, you make the adapter override the 
primary adapter by including the "OverridePrimary # 1" in the second 
section of the configuration. The catch is, it won't work unless you 
also specify a Router x.x.x.x which becomes the new default gateway for 
the system. So to get a new DNS server, you have to change the way in 
which all IP packets ( that don't follow a more specific route ) leave 
the host.

Great. No problem, we can just change the DNS settings associated with 
the Primary adapter ( like your Ethernet or Wifi ) and then restore the 
old settings when we shut down the VPN tunnel. Nope. Even if there is an 
entry in the route table that says the DNS server should be reached via 
another adapter, the packet is always sent out the Primary adapter. 
Apparently at Apple, DNS over UDP is not part of the TCP/IP protocol 
suite, it's a layer 2 protocol that just happens to contain UDP and IP 
headers. Interesting.

Lets try another approach. Maybe we can just change the priority of the 
DNS servers used by the system resolver. Not possible. If your adapter 
isn't primary, your DNS server simply won't get used except under one 
special condition. The behavior is achieved by specifying a list of DNS 
suffixes that should be serviced by your adapters name server. For 
example ...

 >scutil
  open
  d.init
  d.add DomainName shrew.prv
  d.add SupplementalMatchDomains shrew.prv
  d.add ServerAddresses * x.x.x.x
  set State:/Network/Service/tap0/DNS
  d.init
  d.add InterfaceName tap0
  d.add Addresses * y.y.y.y
  d.add SubnetMasks * 255.255.255.0
  set State:/Network/Service/tap0/IPv4
  close

Now, any request that matches *.shrew.prv will be forwarded to the DNS 
server associated with your adapter. Great for forward DNS lookups as 
long as you have an exact list of all domains that would be significant 
to your VPN connection, but what about reverse lookups? Nah, your out of 
luck.

My conclusion: The system DNS resolver in OSX is flawed by design. All 
you have to do is google 'OSX 10.4 DNS' and you will find thousands of 
posts by people with multi-homed hosts trying to figure out how to work 
around the shortcomings of Apples OS. Lots of folks who need to use a 
private DNS server for internal use or development resort to installing 
a local instance of bind ( the defacto DNS server on *nix ) and forcing 
all queries to localhost. Way to go apple.

In any case, the Shrew Soft client is more or less ported to OSX. How 
useful it will be to users is another story. I tried to get help from 
the Apple network developers mailing list, but there was no response ...

http://lists.apple.com/archives/macnetworkprog/2012/Dec/msg00006.html

All the information in this email was learned from reading the scarce 
documentation and by trial and error. I would love it if someone would 
show me light and disclose how this should really be done, but I'm not 
going to hold my breath.

-Matthew