Best of the ISP-Lists

General

Don't Destroy What You Want to Save

Members of the ISP-Tech list discuss the many pitfalls of data backup. Tight budgets are conspiring to make a vital business process more difficult that it should be.

[March 7, 2003]

On the ISP-Tech list in February, KW asked:

"We have recently installed some equipment that relies on a Sun E250 machine. It came with only one 9 GB drive. It takes SCSI drives with an SCA connector. I'd like to drop in another drive to do RAID 1, but that is going to be in the future as our budget is completely frozen in a huge block of ice and I can't even get funds to purchase a simple IDE drive right now.

So I've been wanting to do a backup. UFSDump says I have to bring the file system to single user, but I've been doing dumps on our BSD systems for years without going single user both over the network and on local machines. Does Solaris require single user for a simple UFSDump?

I just need one file system backed up to tape. If something does crash, I can reinstall Solaris easily enough as it is a very static system. It's only required to run LDAP and a few other programs that are all installed into one file system and I can restore that one file system from tape.

I cannot bring this box down to single user as it is required for operation 24/7 and right now they won't spring for overtime, otherwise I'd come in at 3 AM and do it. It's about 1.5 GB onto a DDS3 tape drive. Any caveats?"

[BS warned] "If your budget won't allow you to externally backup your data, obviously the cost of data loss is not understood. I'm not suggesting a bazillion dollar scheme, but at least a separate machine with a disk/tape drive would be a good idea."

[KW replied] "I'll be able to do that soon, but right now I can't. At least I can backup to tape. I've got a DDS3 tape drive in the box; I just needed to know if Solaris really needs single user, like it says in the manual pages, to really backup one file system."

[PH cautioned] "You can do this, but you really ought not to, as it may produce a backup that is unusable and unrecoverable. The fact that a backup takes some time is a critical issue here.

If you were to simply pull the power plug on a live machine, then bring it back up in single user mode and make the backup, you would have a backup of an unchecked file system. You can recover from that by checking the file system after restore.

But doing a backup during live file system activity is worse than the above scenario. Suppose the backup takes 2 hours to complete (which is about the time for 9 GB to DDS3). What you have is a "slant in time" view of the file system. Basically it's like pulling the plug on the machine at different times for different drive sectors over that 2 hour period.

I cannot describe the impact you may see without knowing what you are running. But if it is a database, the impact can be unrecoverable corruption, even if you were to simply copy the files to tape, or use the tar program. OTOH, if your data is file oriented, dumping just the files with a command like tar or cpio can be quite safe.

As to backing up to tape, if everything is shutdown so the state of data and file system is in sync, then a safe backup can be made. Even copying to another disk drive can be a problem in certain cases, such as with databases. You need to use RAID and/or LVM to manage the data between the drives so it can be kept in logical synchronization."

[DA advised] "Strictly speaking, you should be single user because you want to quiesce the file system. However, unless there is a lot of activity (such as a news partition), it will not much matter, since ufs backup makes a copy of all the inodes in the system. Directories are just inodes. The files may not be consistent among themselves, but the restored file system and the backup will be consistent and usable after you run fsck.

Files that change frequently often don't need to be backed up. Of course, with a database, this isn't the case. You cannot backup a an Oracle database this way. You must use Oracle's backup to a backup file, and then ufs backup those files to tape. Likewise, to restore Oracle (or any other db), you must recover the db backup files, and then recover the db from the backup files. (This oversimplifies a lot regarding Oracle.) Note also that file systems that do copy-on-write snapshots also cannot automatically backup databases, since it does not know what is in the database engine memory.

Note that any "live" file system may not be consistent, and so every restored file system will need to be fsck'd before use. ufs backup reads the disk through the raw block device, so the changes to the file system in memory will not be seen. I think Linux/EXT2 has a different buffer cache implementation, and that the block device on Linux does see the file system buffer cache.

To have a perfect ufs backup, the file system should be unmounted."

[KW replied] "It's just 1 GB, and with a DDS3 drive, I'd expect the backup to take less than say 15 minutes or less.

I've been going over this with Nortel (the box runs some Nortel gear) and I can do a live backup of the system in question as all I want is the file system that has all the Nortel data on it. I don't really care about the rest of the Solaris machine. I would rather reinstall Solaris, and then recover the one file system off the tape.

They would rather I shut down their processes first then do the backup, but it can be done live if needed. Guess I'll plan for some maintenance time here PDQ."

After this explanation, there was general relief.

[SS said] "That's a little different. I thought you were talking about backing up the entire file system, and judging from other replies here, I bet others were too."

[DA added] "Ahh. Well, in that case, the gtar command is probably your friend. If you don't have gtar, there are several sites that offer Solaris binaries. Google for it."

—End