RAID 5 drive failure

[gchq]

We have an old W2K Server box with three drives in RAID 5 configuration using a MegaRAID475-64..

It has run without major issues since 2001, but recently fell over. Tried starting it and the RAID controller reported a mismatch between the disks and NVRAM - reset it to the disk configuration (one physical drive failed, two OK), started the server and ordered a new drive..

The same thing happened again and this time CHKDSK ran when the server started. Now a few files are showing as 0 (zero) bytes..

This is not end of the world stuff as critical data is now on RAID 10 (7 drive  - 6 and 1 redundant) Server 2008 R2 boxes, but I really could do with recovering one NOTES file in particular where the data is quite current and the backup is out of date.

What would be the best method to attempt recover of this file?

It does seem files that were being written to and/or locked at the time it fell over are the ones causing the problem


Thanks

[Tom]

I would think your chances of success here are close to zero. Active databases, writing at the time of crash, chkdsk... All that you can try is a run with iRecover, see what comes up.
If the RAID is still intact you can run iRecover directly on the RAID. If you have the disks as separate disks now (minus one disk) you can still use iRecover, in RAID recovery mode. Look at the help for details.

[gchq]

Hi Tom

Thanks for your response..

The server fires up and runs on the two drives for about 30 mins or so and then falls over again - same issue with a mismatch between the disks (two online, one offline) and NVRAM (all three offline) so I can't help but think the controller is up the wall as well!

Should iRecover be run from the OS or from a CD? (If used on a 'working' RAID system?

[Joep]

Hello,

The current version needs to be installed and does not run from CD.

If the array is sort of running then this is basically a file system issue (because chkdsk ran), you need to run the 'data recovery for linux and windows' option. Given the chkdsk damage, make sure you check the 'recover both current and deleted files' option. It's always guessing what chkdsk does but the original files may still exist as 'not in use'/deleted files. Speaking of which: It may even be worth a shot to run a simple undeleter instead (for example iUndelete). iRecover treats the disk as a total disaster area and will scan the entire disk where an undeleter may pick up deleted files within a minute or so.

If the array is not intact then iRecover needs to be able to access the individual array members (as separate disks). It is ok to run without the failed disk (so one disk missing is ok). The process will take several passes: iRecover needs to work out array parameters, and then it needs to analyze the file system on the (virtual/reconstructed) array.

[gchq]

Hi Joep

Thanks for the additional info... I will try running undelete first, then have a go with iRecover..

I do have a 'retired' W2K workstation with SCSI and, somewhere in my box of tricks, an adapter for removable to standard drive.. I assume that I can then scan each disk (including the failed one) this way?

What are the chances the entire file would be available on one drive? The file is about 3GB.. Is there a way to 'stitch' it back together if it was spread over the array?

Have you any idea why the mismatch between NVRAM and the disks? I do have another identical server (retired) with the same RAID controller - would it be OK so swap them, or will that cause another unwanted problem?

Thanks again for your prompt response...   Last time this sort of thing happened was on a Netware box back in 1999 and I've slept since then.

========== UPDATE ================= 18:19 UTC ==============================

I had a licenced copy of Disk Internals Uneraser so I ran that - it found a deleted file from 2007 (I was upgrading Notes server, copied everything in that directory to another directory and a few months later deleted it) but nothing of the new one.

Ran iRecover and it found 45,000 damaged files (are they really damaged or just not recognised??) - Checked the directory where the file I want should be, but it wasn't there (not even the 0 bytes shadow). Should I be looking anywhere else?

There are, by the way, a whole raft of *.chk directories and files but nothing the size of the file I am looking for..

[Joep]

Hello,

I do have a 'retired' W2K workstation with SCSI and, somewhere in my box of tricks, an adapter for removable to standard drive.. I assume that I can then scan each disk (including the failed one) this way?

Meaning one by one? If you can only attach disks one at the time then you need to image those disks. All image files can then be treated as disks by iRecover. It needs access to them all at the same time to work out array parameters.

Have you any idea why the mismatch between NVRAM and the disks? I do have another identical server (retired) with the same RAID controller - would it be OK so swap them, or will that cause another unwanted problem?

I have no idea about these things. To recover data from RAID 0 and 5 arrays all we need to understand is the basic principles, how redundancy is created, how data is layed out on disks. I have little knowledge of individual RAID controllers.

Ran iRecover and it found 45,000 damaged files (are they really damaged or just not recognised??)

If iRecover finds allmost all files damaged then it can mean a few things:

- Data is really bad
- iRecover did not find the correct parameters (like file system cluster size) and as a result it's looking at the wrong places for the files.

Ran iRecover and it found 45,000 damaged files (are they really damaged or just not recognised??) - Checked the directory where the file I want should be, but it wasn't there (not even the 0 bytes shadow). Should I be looking anywhere else?

There are, by the way, a whole raft of *.chk directories and files but nothing the size of the file I am looking for..

Sounds like we're wasting time. I wasn't very hopefull from the beginning (chkdsk is NOT data recovery, it's quite destructive in achieving it's primary goal: file system consistency) and from what you have told me I estimate chances for recovery are poor.

[gchq]

Meaning one by one? If you can only attach disks one at the time then you need to image those disks. All image files can then be treated as disks by iRecover. It needs access to them all at the same time to work out array parameters.

Do they have to be imaged whilst connected to the controller? Or can I image them one at a time from as standard SCSI controller and save them to the hard drive of that machine - then select all two/three images using iRecover?

[Joep]

Hello,

The latter:

Or can I image them one at a time from as standard SCSI controller and save them to the hard drive of that machine - then select all two/three images using iRecover?

[gchq]

Using an 80 pin caddy attached to the workstation 68 pin bus I am now in the process of imaging one drive at a time... Hard drive space wise this is going to be very close indeed, more so because all three drives contain duplicated data. Is the image file any smaller than the original? If so is there a rough rule of thumb that would give me an idea of the space required?

So far the faulty drive is not showing any bad sectors, just a lot of data fragments! If this drive was offline the last time the server was booted (and CHKDSK decided to lend a helping hand) will iRecover still be able to reconstruct the original RAID configuration?

Thanks

[Joep]

Hello,

The image files are raw files. Each and every sector is imaged so a 100GB disk will result in a 100GB image file. The image file may actually take less space to store if written to a compressed NTFS folder.

So far the faulty drive is not showing any bad sectors, just a lot of data fragments! If this drive was offline the last time the server was booted (and CHKDSK decided to lend a helping hand) will iRecover still be able to reconstruct the original RAID configuration?

I'm not sure I understand. If any one disk was altered by the RAID controller, or the remaining two were while the other being offline then it is best to run analysis on two disks (the consistent disks as far as RAID-ing is concerned). Pick the two you expect to be ok, leave out the possibly 'damaged' one. The damaged one will only confuse iRecover, and with RAID 5 an array can be (virually) reconstructed due to the availability of parity data.

But apart from the RAID thing: As chkdsk was run there will be damage done to the file system as it was before chkdsk was run. Chkdsk alters the file system and may render data unrecoverable. So even if iRecover can successfully reconctruct the array it remains to be seen if intact data can be recovered.

[gchq]

I know that there is nothing on the two good disks as I was able to run iRecover on them as a unit.

I can run iRecover on the 'bad' disk and it shows files and directories, but not all of them (for obvious reasons)

What I was hoping to do was see if it can possibly reconstruct some of the directories/files using all three drives  as there might be something on the 'bad' drive that points to the file that our friend CHKDSK so helpfully messed up!

As 'Create Image File' is greyed out I assume that is not available on the trial version. I did seem to remember seeing something about the licence for iRecover professional was for one PC - the only way I would be able to do this would be to image the files using an old Workstation, copy them to a network drive and then copy them to my Workstation (that uses SAS RAID so can't image them on this) and then run iRecover on the images so the operation would be spread across two machines.

Having said all that the chances are so remote of recovering that one file...............

[Joep]

Hello.

Creating image file is greyed as long as no disk/partition is selected.

I know that there is nothing on the two good disks as I was able to run iRecover on them as a unit.

I don't get that. As a unit?

[gchq]

I don't get that. As a unit?

Whilst they were still attached to the RAID controller as an array

[Tom]

Sorry, but I'm completely lost here. The controller does work then? The 2 disks are presented as one volume, and the third disk has been rebuilt? I'll tell you what I consider to be the best approach here, given the situation:
All of the former RAID disks should be connected to a PC, each of them as an individual disk. If one of them is/was faulty, leave that one out. Then let iRecover loose on these disks, performing a RAID recovery. If you don't have the equipment to connect all the disks, imaging all of them and then performing the RAID recovery on the images is also fine.
If that doesn't yield the desired results it's pretty much over, especially given the factors involved (large database, chkdsk etc.).

[gchq]

Sorry, but I'm completely lost here. The controller does work then? The 2 disks are presented as one volume, and the third disk has been rebuilt? I'll tell you what I consider to be the best approach here, given the situation:
All of the former RAID disks should be connected to a PC, each of them as an individual disk. If one of them is/was faulty, leave that one out. Then let iRecover loose on these disks, performing a RAID recovery. If you don't have the equipment to connect all the disks, imaging all of them and then performing the RAID recovery on the images is also fine.
If that doesn't yield the desired results it's pretty much over, especially given the factors involved (large database, chkdsk etc

.).

Here is what happened

1. Three drive RAID 5 system stopped. When I restarted it I got a mismatch error between the disks and NVRAM (disks said one drive failed and two were online, NVRAM said all three were offline). Reset it to disks and rebooted.

2. Removed the faulty drive and ordered another

3. About an hour later the server fell over again with the same mismatch error. Went through the same procedure and rebooted. At this point CHKDSK kicked in.

4. Server ran for about an hour or so and fell over again, with the same error..

5. Corrected the mismatch again and rebooted. Ran undelete and iRecover. The file I wanted was not found.

6. Using a caddy I was able to run all three drives on an old workstation.

7. Attempted to run RAID recovery on all three drives - part way through the caddy alarm goes off - I could not connect the fan so I guess they overheated.

8. Now attempting to create an image file of each disk so that I can return the disks to the server and use the images to work with, but...

Open iRecover 5.2
Select RAID recovery
Check the relevant disk
Click 'More functions'
The only options are 'Load Image File' and 'Define RAID Manually' - 'Create image file' is greyed out!

The logic behind running all three drives (including the 'bad' one) is that there might be some flags left on the bad one as CHKDSK didn't do anything to that drive...  As yet everything suggests that the bad drive is still OK..

This particular server was going to be decommissioned in a couple of months in any event..


==== UPDATE ============ 15:00 UTC=======================

Sussed out the Create Image File issue - not only does the drive have to be checked, but the disk name has to be highlighted as well! Running Image File Creation now...

====================================================