Serverless and mailless
Oops.
Yesterday (Sunday, 31/08/09) I far from any computer-like object for most of the day. When I got back home, of course, I promptly opened my laptop to check my mail — who knows what destiny might have for me in a 24 hour period? Maybe I won yet another fortune I have to cash in Nigeria? Maybe there is (GASP!) a new RC bug on one of my packages?
But no, my mail server didn’t feel like answering to my ssh queries. The connection was established, but shut down before even sending the protocolary SSH-2.0-OpenSSH_5.1p1 string. Fearing an overload (after all, the little bugger is just a Mac Mini running in another room in my house), I tried to check (via Web) its Munin status — Apache didn’t want to listen either. It answered, but got only access denied. Things started worrying me… But (silly me) not enough — The machine runs headless[fn]for those not used to computerspeak: without a connected monitor[/fn], so I just danced the boring raising elephants song[fn]For those following at home who don’t understand how Raising Elephants Is So Utterly Boring, this is a (silly, stupid but useful) mnemotecnic for as-properly-as-possible restarting hung Linux systems: Hold AltGr (right-Alt) + SysRq (in reduced/notebook keyboards, Fn+SysRq), and type R E I S U B (leaving a couple of seconds between key, ideally waiting for all disk activity to settle between each). That means:
- unRaw
- Take control of keyboard back from X,
- tErminate
- Send SIGTERM to all processes, allowing them to terminate gracefully,
- kIll
- send SIGKILL to all processes, forcing them to terminate immediately,
- Sync
- Flush data to disk),
- Unmount
- Remount all filesystems read-only,
- reBoot.
- Well... Reboot the system
[/fn].
Allowed for a couple of minutes for everything to settle, and tried to connect. Horror, now even pings didn’t work!
So I ran to fetch my old, bulky and trusty monitor. Went back to the machine, plugged it in, switched it off and back on. Everything worked fine this time — At least appearingly. I opened up mutt and started happily reading mails, while trying to understand on another console what happened at 07:06 that didn’t get logged anywhere and had the machine dead for basically all the day. And then, BRRRT-BRRRT-BRRRT, I started hearing the HDD seeking.
I was able to send a couple of mails, but decided to let the machine rest and… Will reduce its disk usage to an absolute minimum. Fortunately, I have already the machine meant to replace it — A much nicer, beefier iMac G5, waiting to be vacated from its data, task which has suddenly become prioritary.
So, in short: If you need to get in touch with me in the next day or two, don’t count on my usual @gwolf.org mail, as it is down. I hope to be able to get the data out of the poor little bugger painlessly after it rests a bit. And I hope not to drown in a sea of mails after I get the replacement back online :-/
Comments
Anonymous 2009-08-31 11:24:00
I hope you have mx backup in
I hope you have mx backup in place..
:~$ dig @208.67.222.222 gwolf.org mx
; «» DiG 9.5.1-P3 «» @208.67.222.222 gwolf.org mx ; (1 server found) ;; global options: printcmd ;; Got answer: ;; -»HEADER«- opcode: QUERY, status: NOERROR, id: 8924 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;gwolf.org. IN MX
;; ANSWER SECTION: gwolf.org. 7195 IN MX 0 cajita.gwolf.org.
;; Query time: 30 msec ;; SERVER: 208.67.222.222#53(208.67.222.222) ;; WHEN: Mon Aug 31 20:21:42 2009 ;; MSG SIZE rcvd: 50
:~$
gwolf 2009-08-31 14:26:55
Nope, I don’t
…I thought about getting down to business and setting it up today, but frankly, my Real-Life work is quite busy today. I expect to set up the server replacement in a day or two. Even if the data is not recovered in time, I prefer to invest energy into that new server setup. As long as it does not take me five days from the first failure, I should not worry (too) much about losing mail… Maybe I will lose some valuable spam, though.