Archive for the ‘storage’ Category

QNAP Data Recovery Hell…….

Wednesday, May 11th, 2011

It’s never goes easily does it ?!

Somehow, I had managed to almost fill my 1TB drives in my x2 bay QNAP NAS. Owing to my being paranoid about my data, I choose to mirror a pair of x1TB drives for 1TB total storage, rather the create a 2TB spanned volume using both of them.

I was down to a couple of hundred MB of free space, time to take action. I ordered a couple of shiny new Seagate 2TB drives online with the intention of going from a pair of mirrored 1TB drive to a pair of mirrored 2TB drives.

My plan was to removed one drive from the mirror set, and one of the new 2TB drives, wait for it to rebuild, and then replace the second drive and allow a subsequent rebuild to take place. Did not go according to plan :o(

I removed one of the 1TB disks, and then to ensure the 2TB disk would stand a greater chance of working, I upgraded the firmware (dumb dumb dumb !). I then installed one of the new 2TB disks.

The NAS did indeed rebuild/re-mirror with the new drive, but it created a 1TB volume on the 2TB disk. Not entirely unexpected, but not quite what I wanted :o(

Now I needed to get all the data off of the existing NAS 1TB disk onto something else, install both the new 2TB disks, create an empty 2TB volume, mirror it, and re-copy the data back onto the new larger volume. I attempted to connect one of the original 1TB disks to a Windows desktop machine with a SATA converter cabe/kit, and to mount the file system to copy the data off. And here’s where I hit real problems.

I could see the top level folder structure on the disk, but nothing below it ! :o( The QNAP version of EXT4 is a custom patched version that it seems can only be read by their chassis running their firmware. Fair enough, intellectual property and all that, but this was making my life a mite difficult now.

I tried putting the original drive back into the chassis and booting it to copy the files off over the network, but the new firmware update seemed not to like this, it booted ok, but the actual file server part did not kick in, I couldn’t see the device on the LAN to map a drive to.

In the end I had to ssh to the NAS device itself and use the Samba services to run a CIFS mount to my Windows desktop. I then spent several glorious days moving the files off folder by folder till I had them all. Then I put both new 2TB disks in, did a factory reset (including down grading the firmware as it seemed a little….buggy).

Once the device booted up and presented itself as a shiny new empty 2TB NAS, I began the unenviable task of copying all my crap back over…….several more lost days. Moral of the story for me would be:

a) don’t upgrade drivers/firmware for the sake of it
b) when dealing with a *lot* of data, *copy* and then delete, do not use move. even if a 20 stone psycho has a knife pressed to your throat

Ironically as I write this, my 2TB is being eaten into at a rate of knots. Am probably gonna have to buy a bigger NAS. Have my eye on one of these full to the brim with 3TB disks.

possibly all the storage I could need.......?

Tags:
Posted in IT, NAS, networking, storage, unix | 3 Comments »

VMWare ESX Thin Provisioned Disks And Credit Cards…….

Friday, March 11th, 2011

In my youth, I was offered credit cards. I thought they were great, allowing me lots of instant retail gratification, I was on cloud 9 on the high street…….then I had to repay them. My cloud evaporated and I came back down to earth with a bump right on my wallet !

The principal behind thinly provisioned disks on VMWare ESX storage LUNs would appear to be along the same lines as that of credit cards. Utilising more than you actually have at your disposal.

With credit cards the impact is that later down the line you have to repay the money you spent that you don’t actually have (ouch !). With thinly provisioned disks the pain is that of not being able to reclaim free disk space without a lot of work.

With normal *fat* disk allocation when you create a volume all the space is allocated at creation time. So if you have a 100GB storage LUN and you create a 50GB volume on it, that 50GB is immediately deducted from the 100GB, leaving you with only 50GB free space on the storage LUN.

But with thin provisioned disks, if you allocate a 50GB *thin* volume, the space is only deducted from the storage LUN as it gets used. So if you only write 20GB of files to the 50GB thin volume, the storage LUN will report 80GB of free space out of the total 100GB.

But here’s the kicker, if you then delete 10GB of those files, the free space remains at only 80GB, the space does not get returned back to the storage LUN. This is because Windows/Linux does not actually delete files, it simply marks those blocks as being avilable for use in the file table. As the blocks do not actually get cleared, VMWare does not pick up on this and remains ignorant to the free’d up filespace.

In order to shrink the disk and reclaim the space, you have to actually clear the physical blocks so that they are actually empty. You can do this using the free tool Sdelete from Sysinternals. You use the -c switch (which tells it to zero out the blocks). This is a rather I/O demanding task best done out of peak useage hours.

Once sdelete has completed you will need to migrate the VM files to another datastore for VMWare to re-read the free blocks and give back the cleared up space. You may then have to re-migrate the VM files back to where you actually want them to run from if your fussy about the location of your VM’s on your datastores.

I’m not saying thin disk provisioning has no place, it’s great for R&D, labs and proof of concept type setups where you will be setting up and ripping down and don’t really care too much about long term storage levels. But for production systems, the administration overhead is just too great for my liking.

disk squish

Tags: , ,
Posted in esxi, iscsi, NAS, SAN, storage, vmware | 2 Comments »

Rather Impressive Storage…….

Saturday, November 6th, 2010

And so it came to pass, the awesome Qnap TS209 Pro II that I purchased and lovingly installed last year, is almost full :o/

Admittedly, it only has 1TB of storage (x2 1TB drives, but I mirror them coz’ I’m paranoid and IT hates my flatmate). Now that they are becoming a little more common place on the market, I decided the most cost effective solution was going to be to replace the 1TB drive with 2TB drives.

However, the replacement process is not straightforward. Removing x1 1TB drive and replacing it with x1 2TB drive has resulted in the device rebuilding the data on the 2TB drive to re-establish the mirror, but rendering the remaining 1TB of space inaccessible :o(

So, I am going to have to transfer almost 1TB of data from the mirror set to an external drive, replace the other drive so the device contains x2 2TB drives, erase all data/partitions on them. Configure them as a RAID mirror, and copy all the data back onto them.

Time to complete……..well I started the data copy 2 days ago and it’s still going……I am hopeful that it may complete by the weekend ??! :os

I wish I had known about these guys at the time of purchase. From what I can gather, they are using a proprietary algorithm for data protection (beyondRAID) that seems to be rather dynamic ! It protects and shifts data between drives to accommodate a failure of any given drive. It also allows dynamic drive addition using mis-matched drive capacities, vendors, firmware etc.

I’m fairly certain my desire to purchase one is not being influenced in any way whatsoever by the demo being given buy the fit woman in the tight sweater. I need a backend network storage device for a project at work….wonder if I get them to buy a really big expensive one if the lady in the sweater comes to install it for you ? hmmmmmm?

:op

Posted in NAS, RAID, SAN, storage | No Comments »

Home NAS To The Rescue

Saturday, May 23rd, 2009

My 500gb lacie external mac mini storage unit decided it was no longer going to work :o(

The fault seemed to be with the bus interface of the casing, rather than the drive itself. The drive is a bulk standard ide drive which gets converted to a usb port connection on the outside of the case via an internal cross connector.

First off I tried connecting the device to a PC instead of the Mac to see if anything could be seen/salvaged at all (the Mac mini’s role is an mp3 file repository for the Sonos wireless hi-fi units around the home……no Mac mini = no music !! gaaaaaaahhhhhhh :o(

I opened the case and extracted the physical drive and connected it to a PC using a standard ide cable. The PC recognised the drive no problem, but clearly Windows does not understand the Mac OSX file system without some help.

Luckily, macdrive to the rescue. Once installed I could read all files on the mac volume and was able to copy them across to a more reliable drive.

Next step was to provide new storage. Almost losing 10 years worth of music collecting was quite a panic, so I wanted the new solution to provide some piece of mind. enter the Qnap TS209 Pro II nas server.

ts-209 pro ii

The device costs just under £230 from scan.co.uk without drives. Offically Qnap only support the device with 1tb drives fitted, but I have seen reports of people running them with 1.5TB drives (probably 2TB drives by now !). A pair of 1tb seagate drives cost £63 each so the whole thing came in for under £400.

The device is small enough to fit into the little cupboard next to my media unit and runs almost silent, even during write operations. The device is powered by a marvell cpu running at 500Mhz and configuration is managed by a well designed web interface.

Embedded applications include ftp server, web server, mysql database server and more, although I don’t need any of these as I just need somewhere to dump files. Having x2 drives the configuration options are either RAID 0 striping across the x2, which will improve read/write performance, but provides no resiliency, or RAID 1 mirroring which halves the total storage capacity but provides piece of mind.

The initial setup requires a direct ethernet connection from a PC/Mac directly into the units ethernet port. Once you have assigned the device a tcp/ip address you can connect it to the network and finish the setup.

You have the option to select either NTFS, FAT and EXT3 filesystems so you can use it with both PC and Mac/Linux/Unix systems. My only gripe with it would be the pre-configured shares defined on the system. There are shares such as Qdownload, Qweb, Qusb etc. etc. (you get the idea) that you cannot remove. You can however mark the shares as hidden so they don’t show up on the network so this is not a major show stopper.

Performance seems ok. The device has been handling simultaneous read/write requests from me, my flatmate, a visiting friend, a x3 sonos wi-fi players without any blips or stutters. All in all, money well spent and I sleep much better at night knowing my data is duplicated.

Tags: ,
Posted in storage | 2 Comments »