What software are you all using for node failover?
Right now we are looking at using rsync and Understudy as a means of mirroring our primary server to a slave at a remote location. This seems an acceptable solution. However, Understudy depends on BIND. And freebsd has pointed out that TinyDNS is a better DNS server over BIND. Understudy carries a price tag with it, while TinyDNS is free.
freebsd,
I know you prefer TinyDNS to BIND. TinyDNS supports failover, but it needs an external program to edit its files. Any recommendations?
How do I direct traffic away from a dead server? If one of my servers dies, I want tinydns to stop providing that server's IP address, so clients won't waste time trying that server.
Answer: The following answer is for versions 1.04 and above.
tinydns is designed to work with external programs that monitor the health of your servers. Specify each address as an alias with a 5-second TTL:
An external program can remove an alias by simply changing + to - on the relevant line, then running make. Later, when that server has recovered, the program can change - back to +.
Note that standard client behavior is to try each of the addresses provided by tinydns, so a server outage will merely produce delays, not failures. To minimize the delay, smart clients will try each address with a two-second timeout before retrying each address with a long timeout.
>> but it needs an external program to edit its files. Any recommendations?
1) By hand
2) Write a script ( Parsing the zone data format of Tinydns is relatively simple. So it'd take someone with 1 month of programming knowledge to write such a script)
3) Go to http://www.djbdns.org and search for twa web-based tool. Note, you need to install python binary (not mod_python).
Man freebsd you are fast! Thanks. I'll check out that link.
I also found an example script while searching on this same topic:
Use http://cr.yp.to/djbdns.html for your DNS server; has built in
DNS load balancing/failover stuff, sophisticated enough that all you should need to write a simple script like the following:
#!/bin/sh
cd /service/tinydns
while true ; do
ping -c1 theotherwebserver 2>&1 > /dev/null
if [ "$?" != "0" ] ; then
cp -f data.down data
make > /dev/null
else
cp -f data.up data
make > /dev/null
fi
sleep 15s
done
You'll have the files data.up and data.down created before hand,
with the .down file having the second server removed, and the .up file having both webservers listed.
You can run the script at any interval you want (every 15 seconds
as written above), and when one box can't reach the other for any reason, it removes it from it's DNS tables. As soon as it comes back up, it re-adds it.
Now, PLEASE note, that my script above is most likely NOT what you want to write (although it should work), it's just an example showing the logic. More likely you'll want to use something like "wget -q http://theotherwebserver/index.html" to verify that the other end is actually serving webpages, and not just responding to ping's, or maybe get even more elegant and use a small C or Perl program to do this.
That logic could do the trick nicely in the event of any failure on the slave. If you think twice, the slave DNS server shouldn't be down that often, especially when it's supervised. It'd be a serious problem if supervise can't be restarted automatically. As a precaution, though that's necessary.