A quick reminder on what to do when a disc fails in your 3ware RAID5 array...
There's a reason I went for a proper 3ware RAID card... with terabytes of data, some of it worth quite a bit from a personal perspective (home video, photos and so forth) it's not practical to back this sort of volume up - and the moment you lose it you just know you're going to be seriously peeved. And hard discs do not last forever, boy do I know that. I've seriously lost count of how many have failed on me over the years - they are consumables, really.
So a disk goes AWOL and your RAID5 array is degraded; what do you do? It's maybe not massively clear so let's summarise for you.
First, launch the command-line admin tool, tw_cli, as root... thus...sudo tw_cli
Then issue a show command to list the controllers installed and detected...//server1> showCtl Model (V)Ports Drives Units NotOpt RRate VRate BBU------------------------------------------------------------------------c6 9590SE-8ML 8 5 1 0 3 1 OK//server1>
Issue a focus command, with the controller ID displayed above, to set the focus of the tool to that specific controller. This saves you specifying "c4" in front of every command. Then do a show, to list all of the units and ports (devices - or discs) on the array...//server1> focus c6//server1/c6> showUnit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy------------------------------------------------------------------------------u0 RAID-5 DEGRADED - - 256K 3725.25 ON OFFPort Status Unit Size Blocks Serial---------------------------------------------------------------p0 OK u0 931.51 GB 1953525168 WD-WCAU45216260p1 OK u0 931.51 GB 1953525168 WD-WCAU45190409p2 OK u0 931.51 GB 1953525168 WD-WCAU45228622p3 OK u0 931.51 GB 1953525168 WD-WCAU45226112p4 OK - 931.51 GB 1953525168 WD-WMC0T0354125p5 DEVICE-ERROR u0 931.51 GB 1953525168 WD-WCAU45195008p6 NOT-PRESENT - - - -p7 NOT-PRESENT - - - -Name OnlineState BBUReady Status Volt Temp Hours LastCapTest---------------------------------------------------------------------------bbu On Yes OK OK OK 255 23-Oct-2011
Now the above shows a 6-disc setup, where port 5 is showing a drive error (and port 4 is sitting waiting and unused)... so what we're going to do here is put p4 into the array, take p5 out, and start a rebuild. Your array (unit 0) is still useable, but with one drive down it's on the brink of disaster and cannot afford another failure. So it's time to swap out the failed drive with a fresh one, and start rebuilding the array.
First, take the drive offline by issuing a 'remove' command. This removes the failed drive from the array (unit).//server1/c4> /c6/p5 remove quiet
You can now pull the drive physically out (you've got them all in hotswap caddies, haven't you?!) and fit in it's place a nice new one. Issue a rescan command, and the controller will scan the ports and detect the newly-installed drive...//server1/c6> rescan
Next we are ready to start the array rebuilding with the new drive. If we did a show now, we'd see something like :-//server1/c6> showUnit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy------------------------------------------------------------------------------u0 RAID-5 DEGRADED - - 256K 3725.25 ON OFFPort Status Unit Size Blocks Serial---------------------------------------------------------------p0 OK u0 931.51 GB 1953525168 WD-WCAU45216260p1 OK u0 931.51 GB 1953525168 WD-WCAU45190409p2 OK u0 931.51 GB 1953525168 WD-WCAU45228622p3 OK u0 931.51 GB 1953525168 WD-WCAU45226112p4 OK - 931.51 GB 1953525168 WD-WMC0T0354125p5 OK - 931.51 GB 1953525168 WD-WCAU45195008p6 NOT-PRESENT - - - -p7 NOT-PRESENT - - - -
Interestingly, you'll see above that the failed drive immediately shows as OK! Must have got it's knickers in a twist somehow, but I'm going to bring port 4 online anyway and investigate the p5 drive for problems. So issue a rebuild command and the start the array on it's long, merry path to data integrity...//server1> /c6/u0 start rebuild disk=4Sending rebuild start request to /c6/u0 on 1 disk(s) [4] ... Done.//server1/c6> showUnit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy------------------------------------------------------------------------------u0 RAID-5 REBUILDING 0% - 256K 3725.25 ON OFF...
If you've got a spare disc in the enclosure that you'd like to use as a hot standby, you can do this by adding it into a new unit with the type of array specified as 'spare'. In the example below, this adds port 5's disc as a hot standby. If a subsequent disk fails in unit u0, this disk will automatically get assigned to the array and an autorebuild started (assuming you've got auto-rebuild enabled of course).//server1/c6> /c6 add type=spare disk=5Creating new unit on controller /c6 ... Done. The new unit is /c6/u1.WARNING: This Spare unit may replace failed drive of same interface type only.//server1/c6> showUnit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy------------------------------------------------------------------------------u0 RAID-5 REBUILDING 36% - 256K 3725.25 ON OFFu1 SPARE OK - - - 931.505 - OFFPort Status Unit Size Blocks Serial---------------------------------------------------------------p0 OK u0 931.51 GB 1953525168 WD-WCAU45216260p1 OK u0 931.51 GB 1953525168 WD-WCAU45190409p2 OK u0 931.51 GB 1953525168 WD-WCAU45228622p3 OK u0 931.51 GB 1953525168 WD-WCAU45226112p4 DEGRADED u0 931.51 GB 1953525168 WD-WMC0T0354125p5 OK u1 931.51 GB 1953525168 WD-WCAU45195008p6 NOT-PRESENT - - - -p7 NOT-PRESENT - - - -
u0 is a main RAID5 array unit, and it's rebuilding after incorporating port p4 back into it. Our original failed disk, p5, is now replaced and allocated into a hot spare unit (u1) and is ready to go, should the same happen again. We're 36% through rebuilding too - the one downside to RAID5 :( takes forever and a day to rebuild the array's redundancy checksums.//server1/c4> showUnit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy------------------------------------------------------------------------------u0 RAID-5 DEGRADED - - 256K 3725.25 ON OFFPort Status Unit Size Blocks Serial---------------------------------------------------------------p0 OK u0 931.51 GB 1953525168 WD-WCAU45216260p1 FAILED u0 931.51 GB 1953525168 WD-WCAU45190409p2 OK u0 931.51 GB 1953525168 WD-WCAU45228622p3 OK u0 931.51 GB 1953525168 WD-WCAU45226112p4 OK u0 931.51 GB 1953525168 WD-WCAU45195008p5 NOT-PRESENT - - - -p6 NOT-PRESENT - - - -p7 NOT-PRESENT - - - -