Dont remove a LUN cleanly!!! (LVM)

Recently I've added some FC LUNs and created LVM partitions on top of that in my RHEL 5 Box. Later on we had to delete the LUNs in our SAN and recreate them. I deleted the LUNs from our SAN before clean them up in the system! later on when I wanted to remove the volume groups:

UPDATE: if you haven't removed the LUNs from SAN and just deleted the LUNS
with echo 1>/sys/block//device/delete without deleteing the related
volume group, vgs command will hangs. To fix that you need to delete the
mpaths with dmsetup remove /dev/mpath/mpath and rescan the luns with
echo "- - -" > /sys/class/scsi_host/host#n/scan. It brings up sanity to
the system! (at least for me)


# vgremove VG
/dev/VG/otestapps: read failed after 0 of 4096 at 4096: Input/output error
/dev/VG/odevlapps: read failed after 0 of 4096 at 106300375040: Input/output error
/dev/VG/odevlapps: read failed after 0 of 4096 at 106300432384: Input/output error
/dev/VG/odevlapps: read failed after 0 of 4096 at 0: Input/output error
..
Volume group "VG" not found

So, dmsetup comes to rescue (after you probably lost some pounds and pulling some hair!)

# dmsetup remove --force /dev/mpath/mpath41
# for i in `ls /dev/VG/LV-0*`;do dmsetup remove $i;done

# ls -l /dev/mapper/VG* (should be gone)
# ls -l /dev/VG/* (soft link to /dev/mapper/VG*)

I also had to remove the LUNs with this commands:

echo 1 > /sys/class/scsi_device/0\:0\:0\:0/device/delete
echo 1 > /sys/class/scsi_device/0\:0\:0\:1/device/delete
echo 1 > /sys/class/scsi_device/0\:0\:1\:0/device/delete
echo 1 > /sys/class/scsi_device/0\:0\:1\:1/device/delete

multipath -F


To prevent this issue, let's see what is the best practice recemmended by Redhat:

Procedure 1. Ensuring a Clean Device Removal
1. Close all users of the device and backup device data as needed.

2. Unmount any file systems that mounted the device.

3. Remove the device from any md and LVM volume using it. If the device is a member of an LVM Volume group, then it may be necessary to move data off the device using the pvmove command, then use the vgreduce command to remove the physical volume, and (optionally) pvremove to remove the LVM metadata from the disk.

4. If the device uses multipathing, run multipath -l and note all the paths to the device. Afterwards, remove the multipathed device using multipath -F device.

5. Run blockdev –flushbufs device to flush any outstanding I/O to all paths to the device. This is particularly important for raw devices, where there is no umount or vgreduce operation to cause an I/O flush.

6.Remove any reference to the device's path-based name, like /dev/sd, /dev/disk/by-path or the major:minor number, in applications, scripts, or utilities on the system. This is important in ensuring that different devices added in the future will not be mistaken for the current device.

7.Finally, remove each path to the device from the SCSI subsystem. To do so, use the command echo 1 > /sys/block/device-name/device/delete where device-name may be sde, for example.
Another variation of this operation is echo 1 > /sys/class/scsi_device/h:c:t:l/device/delete, where h is the HBA number, c is the channel on the HBA, t is the SCSI target ID, and l is the LUN


Ref: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/removing_devices.html