O'Reilly Hacks
oreilly.comO'Reilly NetworkSafari BookshelfConferences Sign In/My Account | View Cart   
Book List Learning Lab PDFs O'Reilly Gear Newsletters Press Room Jobs  



Recovering unlinked, but still open files.
Unlinked files that still have open file descriptors can be recovered with relative ease.

Contributed by:
Albert Tobey
[06/12/03 | Discuss (0) | Link to this hack]

If you lose access to a file that is still open in a process (check with lsof(8)), you can recover it without using the "crash the server and let fsck sort 'em out" technique (this is all I could find while googling about).

1. figure out which pid is holding the file descriptor (fd).
2. cd /proc/[pid number]/fd
3. cat [fd number] >new_filename

Here's the real-world example that I needed this for - my syslog script rotated and unlinked /var/log/syslog/syslog.log on my centralized loghost. The script that bounces syslog-ng failed, so syslog-ng continued to log to the now unlinked file. My log rotation script actually moves the file before bouncing syslog-ng, so the filename was syslog.log--rotating at the time of unlinking (with rm). When somebody realized that there was no syslog.log on the system, the first thing I did was the following:
root> lsof |grep syslog
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
....
syslog-ng 23210 root 6u unix 0xdbb230e0 151727656 /dev/log
syslog-ng 23210 root 7w REG 58,2 1006948 372 /var/log/syslog/syslog.log--rotating (deleted)
....

You can see that FD 7w is the logfile, which is 1006948 bytes in size and is held open by pid 23210. Of course, this is a reenactment of the event, so the file I dealt with was a bit bigger than this. Here goes the rest:

root> cd /proc/23210/fd
root> ls -l
total 0
....
l-wx------ 1 root root 64 Jun 12 14:58 7 -> /var/log/syslog/syslog.log--rotating (deleted)
....
root> file 7
7: broken symbolic link to
/var/log/syslog/syslog.log--rotating (deleted)

Looks like a dead end, but it's not. All that's left to do is retrieve our data.

root> cat 7 >/var/log/recovered-syslog.log
root> head -4 /var/log/recovered-syslog.log
Jun 10 16:17:39 hostname syslog-ng: syslog-ng version 1.6.0rc3 starting
Jun 10 16:17:39 hostname syslog-ng: syslog-ng startup succeeded
Jun 10 16:17:39 hostname syslog-ng: klogd startup succeeded
root> tail -1 /var/log/recovered-syslog.log
Jun 12 14:17:18 hostname sendmail[xxxx]: ...

That's it, now I can archive the log file and know that I haven't lost any of today's logs.

Notes:
This was done on a modified RedHat 7.3 system running the 2.4.19aa1 (Andrea Arcangeli) kernel.

lsof(8) is not strictly necessary, but makes things a lot easier.

If I remember correctly, this will also work on 2.2 kernels. A quick check on my workstation running 2.5.70 shows that this hack will work there, too. This means it should still apply for the 2.6 series once it arrives.


O'Reilly Home | Privacy Policy

© 2007 O'Reilly Media, Inc.
Website: | Customer Service: | Book issues:

All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners.