I have been testing the linux server setup for our storage solution needs and minimal downtime in Amazon EC2 which uses XEN virtualization. I have an guest machine running virtualized with 2 EBS volumes(acts as harddisks) in a Raid1 setup so that even if one EBS starts giving issues the data is still accessible on the other one. I created a raid1 array using mdadm and also created a filesystem on it and then mounted it as a drive.
Now I am trying to simulate the drive failure by forcefully detaching it from the guest machine (Detaching it from the Amazon EC2 web console). It detaches the volume in the console but it still appears in the /dev directory inside the guest instance and any read write operation to the mounted directory just hangs with process status as "D" i.e "uninteruptible sleep usually IO". mdadm command gets stuck when I issue a command to fail that device. fdisk -l gets stuck while checking the details for the detached device. Even a reboot hangs, only a shutdown works. I do not want to reboot the server for some reason.
What i want to know is, is there a way through which we can notify the guest instance that the drive has been knocked off and remove it from its device lists?
Any suggestions/advise would be greatly appreciated.