Degraded then failed RAID5 on Intel Matrix Controller

[berabi]

Hi all,

So I'll be as thorough as I can in my steps so far so that I can give you a clear picture of what's happened and hopefully, get from you what the best course of action will be. I had a RAID 5 array degrade at the beginning of the week and the Intel Matrix Console showed me which drive had failed, but the RAID array was still accessible as normal at this stage and simply running in a 'degraded' state. Details of this array are as follows:

3 x ST32000542AS (2TB Seagate Barracuda Green)
RAID Level: RAID 5 (striping with parity)
Strip Size: 64 KB
Size: 3726 GB
Physical Sector Size: 512 Bytes
Logical Sector Size: 512 Bytes

I switched off my server, took the drive out and diagnosed it against SeaTools for Windows on another machine where it told me that the drive was failing SMART test. I started the same machine up from the SeaTools for DOS as the screen said that it may be able to repair the issue (I doubted it, but thought it might be worth validating the result). To my surprise, SeaTools for DOS reckoned that the drive was fine; no SMART issues or anything wrong with it. On this basis, I went back in to Windows, and ran another utility against the drive (test only) and it also said that the SMART information was fine. Very confused at this point, I put the drive back into the server, switched it on, and to my horror when it got in to Windows it had decided that the RAID array had failed!

I have since then been trying a multitude of things to get it working again; starting off with the use of RAID Reconstructor, to try and get the data backed up to an image as this seems to be the things everyone recommends to do. RAID recontructor has managed to successfully diagnose the disks (all 3 together as it's incapable of working with only 2 disks) and has discovered that the RAID is set up as the following:

• 128 sectors (64KB)
• Start sector 8
• Backward (dyn. disk)

I did however, get sod all success with trying to get RAID Reconstructor to read the data as an image to another spanned volume (2 x 2TB - no RAID) as it constantly failed at around 1TB (approx 28%) complaining about some access violation.

It was at this point that I decided to take the plunge with your software instead on the basis that you guys seem to have had some pretty good successes with people in the past and I'm confident that a decent tool able to read 2 drives of a RAID 5 array should be able to recover the data. I've done the following so far though, and not had very good success at all:

• Scanned with all 3 drives
• Got to the file recovery tree following leaving it overnight
• Tried to recover some files, but all are unreadable

I've now gone back to the beginning on the basis that I think that from my research, as soon as a RAID member is marked as 'failed', the controller stops using it immediately and continues with just the other two drives. On this basis, I think that it's highly likely that using all three disks is probably going to be unsuccessful as it's very possible that data changed on the drives during the period that the array was marked as 'degraded' (I run Hyper-V on the server and had virtual disks sitting on the array, plus a number of downloads that were progressing the night before I noticed the degradation). So as I've now provided the background, I have the following questions:

• Is my understanding of the degraded state and behaviour of the RAID controller during this period correct?
• If it is, am I right in thinking that only the combination of the two known 'good' drives will provide valid data?
• I'm working on this basis currently and therefore trying to recover the first of two partitions (a 1TB partition), after selecting just the two good drives, to an .img file. Is this the best course of action?
• If I'm not doing the right thing, can someone give me the best suggested course of action so that I don't waste time?
• Ultimately, I bought the software because it seems that you guys are pretty confident of being able to recover RAID arrays, but also because the refund policy was pretty clear to say that if you couldn't, a refund can be requested if support had been involved and still no joy had been had. Here's me requesting the support; hoping that someone can provide me some reassurance that there's a chance of me actually recovering the data from the two good drives.

Any help and experience greatly appreciated as like anyone in this forum, I'm not a pro with data recovery, but I am an IT professional and not a computer noddy just learning the concepts of RAID 5.

Thanks in advance,

Barry

[Joep]

Hello,

So you have not yet used iRecover, correct? Based on your story I would indeed try with 2 disks, and that would be the 2 you haven't 'messed' with yet. To see if iRecover can recover the data, the only way is trying.

•Is my understanding of the degraded state and behaviour of the RAID controller during this period correct?

Well I can't answer that. Although the principle of all RAID 5 controllers is the same, how a controller will behave when a disk fails etc. is upto the controller manufacturer. If indeed the controller stopped 'updating' the bad disk, then scanning all 3 disks may confuse iRecover, and even if does not confuse iRecover, one disk will contain out-of-sync data. So by the time you start copying files from the disk, files that span all three disks may be invalid.

All in all you have a quite complex situation there. Am I correct in understanding that those virtual arrays 'use' virtual disks (in the form of disk images)?

[berabi]

Hi Joep,

I'm using iRecover now and decided that I would try to see what the scan results directly against the array would be before trying to image it so I stopped the image creation to the spanned 2 x 2TB volume I have provisioned for the recovery and decided to do a direct scan. It finished sometime yesterday but it didn't find the directory structure of the old file system, something I'm really surprised about, and from the files that I did try to recover, they were all corrupted or unreadable. I'm now creating the whole array as an image to do some further scanning work offline, but I'm a little surprised how long just the preallocation process takes to be honest. It's been running for around 18 hours and it's still got another 30-35% to go before it even finishes the preallocation phase; it then has the forward and backward pass phases to go; something I'm not sure how long it takes to do. Anyway, I hope that my understanding of the terminology is wrong because if the program will have spent the best part of 24 hours allocating space for an .img file, surely there's a better way?

On your advice though, I did give iRecover instruction to do the direct scan and .img file creation against the two known good drives given that my own theory about this, your confirmation and my father in law's confirmation of this all seem to be consistent.

I didn't think, based on the site's information about iRecover, that my situation wasn't particularly complex to be honest. It ought to be a simple case of reconstructing the array from 2 member drives and then recovering the data; something you guys claim iRecover is able to do. I will be interested to see whether you think that me creating an image is the best thing to do or if I've already pushed the program by doing the scan yesterday and that my data is in fact lost for good. The thing I'm pretty sure about is that it does still seem to be there as some photos did open, but they were heavily corrupted with strips of the picture offset at different points and missing accurate colour attributes. If the file structure information is missing, what do you think the chances are? The last data scan showed almost all the data in the LostDirs and LostData folders and even then, there wasn't a great amount of consistency between which files were next to other files in their folders. For example, I had Windows system files sat next to pictures from a birthday party; created at totally different times.

With regards to your question about virtual arrays; I don't use virtual arrays - only virtual hard disks. They would have been in one of two formats; either .VHD (Microsoft's VirtualPC/Hyper-V images) or .VMDK (VMware's proprietory format). If you need more information about these, Wikipedia has a pretty good article: http://en.wikipedia.org/wiki/VHD_(file_format)

Ultimately, although I've lost the best part of 2TB of data here, I'm mostly interested in the 4 or 5GB of photos that I may have lost because thanks to Seagate and an unfortunate series of events, my laptop drive broke just 2 weeks ago where these pictures were backed up courtesy of Windows Offline Files. What I neglected to do though, was to re-sync the files when I reimaged the laptop to another drive and unfortunately, Windows Backup doesn't capture Offline Files and Folders by design to save image space. The other stuff, it's annoying but I can get it back somehow. However, pictures in a crap state are as good as not recovering them at all so all options welcome.

Thanks,

Barry

[Joep]

Hello,

With regards to your question about virtual arrays; I don't use virtual arrays - only virtual hard disks. They would have been in one of two formats; either .VHD (Microsoft's VirtualPC/Hyper-V images) or .VMDK (VMware's proprietory format). If you need more information about these, Wikipedia has a pretty good article: http://en.wikipedia.org/wiki/VHD_(file_format)

Yes and it's those virtual volumes that add to the complexity. Those virtual disks are complete file systems in themselves and it is difficult to determine if any given file system structure we encounter is the actual underlying file system or a 'virtual file system'.

I didn't think, based on the site's information about iRecover, that my situation wasn't particularly complex to be honest. It ought to be a simple case of reconstructing the array from 2 member drives and then recovering the data; something you guys claim iRecover is able to do.

iRecover has reconstructed thousands of arrays, it is not just a claim. But after figuring out array parameters the next hurdle is file system reconstruction. It is never simple and VHD files add to the complexity.

Regarding image creation: There is no added benifit unless you're dealing with bad disks. With regards to array and file system reconstruction iRecover treats an image file no different than a physical disk. When creating image files, write the image files to a compressed folder, it will speed up the process.

[berabi]

So the allocation phase finished and failed courtesy of Seagate making the two new Barracuda Green drives I have 1MB smaller each than the 3 from the RAID 5 array - thanks Seagate for maintaining consistency!

I'm currently trying a scan of partitions but again, is it pointless? It's 38% through and found the best part of 150 partitions so far and it doesn't seem to be stopping. Many of the 'partitions' though seem to be 99MB partitions that I can't remember creating and aren't consistent with the VHDs I have on the drives. However, as one would expect, some of the partitions found are consistent with the VHD sizes so I understand why these are being pulled up.

I take the point around the VHDs adding complexity, but is there no way for me to force iRecover to us the original partition tables from the two partitions on the drive? I think that the partition scan has already found them both (one offset by 137MB @ 1023GB and another offset by 1024GB @ 2701GB) but I'm worried about stopping the scan until you've pointed me in the direction of the best way to recover the data here. These partition 'discoveries' are consistent with what I would expect, but should I leave the scan to continue or stop it now and proceed to the next step with the bigger of the two (largely because it has a bigger offset and will take longer to scan for again)? I know you said that the image creation has no benefit, so I won't bother with doing that as it's time consuming as it is but what are your suggested next steps from here?

Ultimately, I'm not rubbishing your tool in anyway because if I didn't believe it could work, I wouldn't have bought it. But I could use a nudge in the right direction at this point because I'm starting to lose hope after over a week of trial, error, failure and still not one contiugous file recovered. 

[Joep]

Hello,

You are scanning for partitions, why? If the array was re-constructed by iRecover and the partition tables or LDM (for dynamic disks) are present it should show you volumes as they were. Did volumes pop up when you scanned all 3 disks?

If no volumes are shown reconsruction may have failed or the LDM is unreliable/not present. If this information is there then iRecover wil show it once array reconstruction completes. Once array reconstructtion completes steps are the same as:
http://www.diydatarecovery.nl/forum/index.php/topic,368.0.html (where Step 1. would be different of course, this is where you'd pick RAID recovery instead).

What happens then depends on how the disk/array was partitioned. If there was only one big volume, right click reconstructed array, select define partitions manually and then pick 'there was only one volume, scan entire disk'.

When we scan for partitions we look for sectors that look like a partition boot sector. A boot sector is the first partition of a partition/volume. If many of those show up then these sectors meet the criteria for a boot sector.

Normally RAID recovery is pretty straight forward. Array is reconstructed > volumes on array are shown > scan volume and select data to be recovered. And again normally, there is no problem if one disk is missing as iRecover can use parity information to reconstruct the missing disk on the fly. This thing not running as smoothly as normal may indicate that there is more to this than just a failed array.

RAID recontructor has managed to successfully diagnose the disks (all 3 together as it's incapable of working with only 2 disks)

I was suprised by that statement and per the information on runtimes website the statement is incorrect (http://runtime.org/raid-reconstructor-faq.htm#drive5).

[berabi]

Hi Joep,

I'm scanning for partitions because the result following the RAID reconstruction with two disks simply showed 'Virtual RAID #0' in the left pane and then a description of the RAID in the right panel. I've stopped the partition discovery as you seem to be indicating that the RAID reconstruction hasn't completed properly. Here's a screenshot of what I see following the partition discovery anyway (attached).

It's strange because I've just tried it again with RAID reconstructor since you said that, having only two drives, and now it seems to work - classic - it definitely did compain about missing one drive before though hence the comment; but stop helping the competition! Also in the attachment, I've strapped the two drives we believe are OK together an taken what RAID Reconstructor says alongside iRecover to compare the settings discovered. Are they the same? The only setting I think differs is the start sector, but I don't know if your settings of 'Parity Start/rotation' is the same or not? On the basis it looks pretty likely that the RAID hasn't reconstructed correctly automatically, what are my options?

[attachment deleted by admin]

[Tom]

It looks like there isn't much more to try. Has at any point a RAID rebuild been initiated (after you put the bad disk back in)? If so, things are likely to be very messed up. And the VHD files are not making this easier. Any chance you can post an iRecover log of the analysis phase? Look for it in the program's folder, attach it as a ZIP, if possible.

If there's stuff on the disks that you really really need back, you should probably contact a specialist RAID recovery outfit near you.

[berabi]

Log file attached. It's not expensive data, but it's frustrating to lose as it's just pictures I don't have elsewhere. If you can't find anything following a review of the log file, can you put me in touch with someone who can facilitate the refund process please?

Thanks for your help Joep,

Barry

[attachment deleted by admin]

[Joep]

Hello,

I'm looking into this but array reconstruction doesn't look good. It may very well be impossible.

[Joep]

Hello,

Well, array seems ok, at least what iRecover makes of it. So then file system reconstruction becomes the most interesting part. I can't tell from the logfile if you tried what I suggested before:

Once array portion is done > right click Virtual RAID#0 > define volume manually > scan entire disk there was only one volume > select the new 'unknonw' entry. If you'd try again, then please first delete the existing iRecover logfile so we get a new one.

[berabi]

Hi gents,

Tried to get this going again last night with the virtual disk filtering set to precise but it's crashed twice somewhere when it got to the end because when I log back in to the server, iRecover isn't running and there's no crash report screen. It's as if it closes itself, but I don't know if it's something to do with this setting. On that basis, I've decided to sort out another scan sessiona again following Joep's steps but without this setting. Once it finishes, I'll let you know the results and upload the log file.

Barry

[berabi]

Log file attached guys.

Thanks,

Barry

[attachment deleted by admin]

[Joep]

Hello,

Yeah, but, did it finish? Or not? Virtual disk filtering will have huge impact on resources. Try running without that. It's just another variable we'd rather do without right now.

[berabi]

I don't know if it finished or not because the program wasn't running anymore when I RDP'd back on to the server in the morning. I've since disabled this setting and came back to us this morning to find that the recovery tree was presented as I have had success with it before but having recovered every *.jpg file, I can tell you that only the very small ones (<10KB) actually seemed to survive. Anything I really need, didn't recover properly and has resulted in corrupt files. The server is currently only running this task so I'm not worried about resources to be honest and therefore didn't mind trying out the virtual disk setting but I'm happy to leave it off if you would prefer to keep it simple for now.

Is it possible that the RAID recovery is somehow not getting the strips in the right order? I don't really know what to do in terms of the settings to be honest. I tried using RAID reconstructor again and set the number of sectors it scanned to be a factor of 10 larger than is set by default to try and get the RAID layout from a larger sample of the disk. The results still came up the same as before with the only differing setting being the Entropy was 0.05 rather than 0.00. I don't know enough about exactly what this means to know what the implications are but is there anything I can do with iRecover to force it to try and build the RAID based on a different sample of the array's data?