Posted by: Anthony Drewery | May 16, 2006

Battlefield lessons

We have a total of 4 Storage Area Networks attached to our primary Exchange clusters. We have 2 HP MSA 1000’s , an EMC CLARiiON CX300 and an EMC CLARiiON CX500. When designing our storage solutions I try to ensure a maximum database size of 50GB and leave twice that amount of free space so there is room for DB maintenance if necessary. I would normally put all databases for a storage group on a single RAID 1+0 array. This way there’s a little give and take with the DB sizes. For example if I planned to have 4 mailbox stores in a storage group I would make the partition at least 300GB (4 x 50Gb for stores + 2 x 50GB free space for maintenance).

Unfortunately I haven’t been as strict with our older hardware and have learnt an important "battlefield lesson" (thanks go to Gary for the post title). We have a server (an old HP ML570 ) that we have been using as a transitional area for mailboxes that we’re manually archiving. As it doesn’t normally contain much production data we’ve been a little lax with our housekeeping and have allowed the 2 stores to grow in excess of 90GB in size (although with plenty of white space) and allowed the free space to drop to around 15GB. We had a failed drive on the box at the weekend and the RAID controller didn’t do too good a job of keeping things going resulting in corrupted stores/logs. We mounted empty stores to allow the affected users to continue working but now have to copy almost 200GB of data across the network to attempt a repair or restore (I believe strongly in keeping an original intact copy of the corrupt DBs/logs when trying any repair procedure). We are running a restore using a recovery storage group on an alternative server but until we have a safe copy of the old DBs on another box we cannot risk deleting the originals to make room for merging the data back.

We’re out of empty drive bays and the server doesn’t support USB so for now we’re just waiting for the copy to complete. 24 hours and counting…

[tags] Exchange, SAN, storage[/tags]


Responses

  1. Your name is Anthony Drewery and this is the longest 24hrs of your life….Bleep Bleep, Bleep Bleep

  2. Heh heh… I came over to make the same kind of wisecrack 😉


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: