Twisted IT paired

copy/paste from: http://grub.enbug.org/MirroringRAID

Many people use mirrored RAID (also known as 'RAID 1') to protect themselves against data loss caused by hard disk failure. Sometimes, you even want GRUB to boot from the secondary hard disk in case the primary fails to keep the system up and running. This is however not as easy as one might think...

GRUB keeps track of the hard disks currently available on your system, on most distributions you can find this information in /boot/grub/device.map. You might have a file like this:

hopper:~# cat /boot/grub/device.map
(hd0) /dev/sda
(hd1) /dev/sdb

Of course you can install GRUB to /dev/sdb (which is hd1), but obviously GRUB will be confused if /dev/sda fails and hd1 becomes hd0. Most likely, it will complain about a failing hard disk at boot time:

GRUB Hard Disk Error

In this case, you want to install GRUB to /dev/sdb and have sdb also mapped to hd0:

hopper:~# cat /boot/grub/device.map
(hd0) /dev/sda
(hd0) /dev/sdb
hopper:~# grub-install /dev/sdb
The drive (hd0) is defined multiple times in the device map /boot/grub/device.map

GRUB doesn't accept this duplicate definition (which is indeed incorrect), so you need to configure things by hand:

hopper:~# grub
grub> device (hd0) /dev/sdb
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 15 sectors are embedded.
succeeded
Running "install /boot/grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded
Done.

grub> quit

Now, /dev/sda and /dev/sdb are configured as hd0 and the system remains bootable if /dev/sda fails.

Assumptions about partitions

The above information only works if your boot filesystem can be found on both /dev/sda1 and /dev/sdb1. If you have /boot on e.g. /dev/sda5 and /dev/sdb5, you'll have to replace root (hd0,0) with something more applicable for your specific configuration.

Allright.
I've got a server (actually my desktop testing machine) with two brand new installed 2x1T disks.
I'm going to setup the disks like this:
3 partitions:
1 swap (we really don't need abstractions for just keeping swap)
2 boot partition in md raid1 (grub2 really sux, so no boot from lvm support in the old one...)
3 all othe space for md raid1 and lvm over it.
It's a good idea to use lvm because you can always add another disk and also can make snapshots... and in short - have more fun with space allocating.


1. partition the two disks identically:
#> fdisk -l /dev/sdb
Device Boot Start End Blocks Id System
/dev/sdb1 1 974 7823623+ 82 Linux swap / Solaris
/dev/sdb2 975 1461 3911827+ fd Linux raid autodetect
/dev/sdb3 1462 121601 965024550 fd Linux raid autodetect

(yes, I know the boot partition is quite big, but there is a lot of space and I prefer to have more space than to wonder wtf I've done... happened few timesof course )


2. create raids.
#> mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb2 /dev/sdc2
and activate
#> mdadm --readwrite /dev/md0
be sure it's sync-ing
#> cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc2[1] sdb2[0]
3911744 blocks [2/2] [UU]
[=>...................] resync = 5.2% (206848/3911744) finish=0.5min speed=103424K/sec

do the same for lvm raid partitions....


3. format boot partition (I'll use etx3) and copy boot files in there.
#> mkfs.ext3 /dev/md0
#> mount /dev/md0 /mnt/
#> cd /mnt/
#> cp -a /boot/ .
#> cd grub
WARNING: sda2 is supposed to be your current partition.
#> sed -i'' -e 's/\/boot\//\//g' -e 's/sda2/mapper\/1tb-root/g' menu.lst
#> cd /;unmount /mnt


4. create physical volume on md1.
#> pvcreate /dev/md1
Physical volume "/dev/md1" successfully created


5. create volume groups - my vg name 1tb
#> vgcreate -A y 1tb /dev/md1
Volume group "1tb" successfully created


6. then add root volume group. mkfx.ext3.mount and copy the currently running root system in new root partition...
#> lvcreate -A y -L 30G -Z y -n root 1tb
#> mkfs.ext3 /dev/1tb/root
#> mount /dev/1tb/root /mnt
#> cd /mnt
#> cp -a {/bin,/cdrom,/emul,/etc,/home,/initrd*,/lib,/lib32,/lib64,/media,/opt,/root,/sbin,/selinux,/srv,/tmp,/usr,/var,/vmlinuz*} .
#> mkdir dev proc sys mnt misc boot
#> cd etc
#> sed -i'' -e 's/sda2/mapper\/1tb-root/g' fstab
WARNING: There is a nasty bug with initramfs tools described here: http://www.mail-archive.com/debian-kernel@lists.debian.org/msg32272.html
You MUST set root to /dev/mapper/VGNAME-LVNAME otherwise you won't get lvm support in your kernel.
#> echo "/dev/md0 /boot ext3 defaults 0 0" >> fstab
#> mount -o bind /dev /mnt/dev
#> mount -o bind /prov /mnt/proc
#> cp -a /boot/ /mnt/boot/
#> chroot /proc
#> update-initramfs -u -t -k `uname -r`
#> exit
reboot the machine, edit the grub menu by hand to boot from (hd0,0) as boot and load /dev/mapper/1tb-root as root.
login as root and make:
#> cd /boot; mkdir oldrootfs; mv oldrootfs; mv -fr oldrootfs/boot/* .;
edit grub/menu.lst to have / instead ot /boot/ dirs.
run:
#> grub-install --root-directory=/ "(hd0)" (described in /usr/share/doc/grub/README.Debian.gz
(root-directory is where the boor partition is on hd0, if you put /boot/ then in live system you'll have /boot(md0)/boot/)...
reboot again and it must be ok now.

7. Install grub on both disks as described here...

8. reboot and disable hdd from bios. the system should boot normally and you should see that you are using the lvm root partition.

I'm migrating a xen machine (my gallery - g.pechurka.com ) from one machine to another.
I wanted to research from quite a long time what should I use in Linux.
I've got a nice working setup but when you research and test you always makes things better.

So, this night I've researched on if I should do MD raid1 or LVM2 mirroring.
Short answer: MD0 raid1 and Physical Volume over it (as used to now). Sometimes LVM is faster but when using single file. You can have serious problems if you have power failure or disk failure/disconnection when using LVM mirroring. It relies on underneath layer to write caches. For further reading open the link below.
Long answer here: lvm2-mirrors-vs-md-raid-1

Just a simple test for speed of ext2, ext3, ext4dev, reiserfs and xfs.
I've created a 5G lvm and did mkfs.* to it with all these systems.
lvm> lvcreate -L 5G -n speedtest tvg
mkfs (all default opts)
mount (no opts)
Then make dd if=/dev/zero of=a bs=G count=3
Then make dd if=a of=b


Here are the results:

FS type	dd if=/dev/zero of=a speed	dd if=a of=b speed
ext2	3221225472 bytes (3.2 GB) copied, 225.283 s, 14.3 MB/s	2047094784 bytes (2.0 GB) copied, 38.4914 s, 53.2 MB/s
ext3	3221225472 bytes (3.2 GB) copied, 248.433 s, 13.0 MB/s	1912479744 bytes (1.9 GB) copied, 52.1087 s, 36.7 MB/s
ext4	3221225472 bytes (3.2 GB) copied, 281.699 s, 11.4 MB/s	1918500864 bytes (1.9 GB) copied, 49.1343 s, 39.0 MB/s
reiserfs	3221225472 bytes (3.2 GB) copied, 248.827 s, 12.9 MB/s	2108379136 bytes (2.1 GB) copied, 62.5061 s, 33.7 MB/s
xfs	3221225472 bytes (3.2 GB) copied, 426.823 s, 7.5 MB/s	2132619264 bytes (2.1 GB) copied, 55.3356 s, 38.5 MB/s

It seems like ext3 is working best with big continious files currently.
Of course this is not a real life situation.
Maybe xfs and reiser will be better for a lot of small files? (reiser has it's glory about that).

Please let me know what you think!

Have you ever been in the middle of deployment and noticed that the owner and permissions on dev package you have and production one are different?
Very oftern when deploing some sites I have two identical dirs with different owners and permissions - one from a dev web server and another from the production (where everything is tuned and working).
Here is a quick and durty way to change all ownership and permissions from prodution to development dir.
BE VERY CAREFULL. These permissions are not all masks. These are the one I needed!
Don't blind apply this script and run chm.sh after that!!!
FIRST check that you don't have some permissions rules in the chmod!!!
If you have 'chmod rw-r---- FILENAME' you will finish with -------- (0000) permissions!!!
You were warned!
mysite.production is a copy of the production dir somewhere else in the tree (your home for example).


$> find mysite.production/ ! -type l -ls | awk '{print "chmod "$3" "$11 " && chown "$5":"$6" "$11}' | sed -e 's/\.production//g' -e 's/-r--r--r--/044/' -e 's/-rw-rw-rw/0666/' -e 's/-rwxrwxrwx/0777/' -e 's/-rwx------/0700/' -e 's/-rwxr-xr-x/0755/' -e 's/-rwxr-x---/0750/' -e 's/-rw-r--r--/0644/' -e 's/-r-x------/0500/' -e 's/drwxr-xr-x/755/' -e 's/drwxrwxrwx/777/' > chm.sh
$>sh chm.sh


and try to compare some files that have specific perms.

that's all.
I'm SURE there is an easier way, and I'll be glad to share!!!

Short story:
If you have a machine that you want to have XEN+3D use ATI with radeonhd driver (check wich cards are supported).
Take a look here before try more than basic configuration of the driver: http://wiki.debian.org/XStrikeForce/HowToRandR12.

Long story:
For more than 5 months I've had a machine at home that was always off.
But not until tonight.
It is a Core 2 Quad @ 2.5 GHZ with 4 gigs ram - a killer for desktop, very descent for server.
You'll almost never need that kind of power (expect you are a mad scientist doing some kind of calculations or a crazy gamer - not me!).
You'll definitely don't need it if you only watch movies, play some stupid doom like games, browse, read pdfs and do all kind of daily stuff on it.
When I got it I told to myself that I'll never use it until I can both have it as my test deployment server and as my primary desktop (including a dual headed video with two monitors connected and using two different GNOMEs on each).

You'll tell to yourself: 'uh! easy stuff.' (at least that's what I thought)
Well here I am five months later, at last I was able to complete my setup.

I've started by installing Debian Lenny (current unstable - now stable) and started digging....
The easiest part was configuring the dual head card to use the two separate monitors and to have two totally separated GNOMEs running.
It took me about two days to figure out that. It turned out that it is impossible to do this with only two X servers.
That is because they can't get attached to the same hardware card (if you have two different hardware cards - no problems).
The trick was to start a X server with wide screen splited on the two monitors (so called Xinerama or dual-view-mode - you know when you can move a window from one monitor to another). Upon this server you get another two 'virtual servers' started each running a GDM.
I'll find the link describing this and put it here.

So far, so good - I've got the two separate X working perfectly and I was almost complete.
Last step was just to change the kernel with XEN enabled one and I'm done.
When I rebooted in the XEN kernel I was 'WOW---F**KING' ... the XEN kernel was not able to work with Nvidia proprietary drivers.
At that time the machine was with NVIDIA GForce 7600 card.
I dug few weeks just to find nothing new - there were two ways to use NVIDIA (drivers):

• stop using XEN


• patch your kernel and hope you get it running and hopefully you'll have no problems

I didn't wanted to change my vanilla debian kernel because this is going to be my test deployment server and it should be as close as it can to my production machines.
Until then NVIDIA is dropped from my favorite video cards list - sad but true, it was my leader and most recommended card to be used it with linux...
What can you think of after NVIDIA?
ATI of course!
I have one - Asus X1600 (ATI chipset: RV530 X1600).
I've plugged it in. At that time I knew that only proprietary drivers were available for newer ATI cards, otherwise you get VERY bad performance and no 3d. That's why I've installed fglrx driver simply to find out that they also can't work with XEN.
DAMN.
The situation seemed hopeless. What other card can you use? (matrox? they've reports to have troubles with xen too!) and have descent 3d in linux? (please If you are aware of any card, post into the comments!)
I've left the project for few months but from few days while talking with a friend of mine, I've started looking for alternative of NVIDIA and ATI.

Tonight I was very surprised when I saw someone mentioned radeonhd driver. I've never heard of it. I've started digging and in about an hour I've got my ATI running with radeonhd driver and showing about 1500 FPS in glxgearx (WOW!!!) with XEN ENABLED KERNEL!
What was my suprise to see this. My problem was solved!
Just lost another hour to figure out how to start my two monitors split ed out in two with one on the left of the other... Xinerama won't work with this driver.
To fix things as you want use xrandr command ( from irc.freenode.org #radeonhd hint)
You can read more here: http://wiki.debian.org/XStrikeForce/HowToRandR12
Be careful with the device names.
These commands did the job for me:

xrandr --output DVI-I_1/digital --off
xrandr --output DVI-I_1/digital --mode 1680x1050 --rate 75
xrandr --output VGA_1 --left-of DVI-I_1/digital

Now I'm happy opensource driver user with most debian vanilla system I can imagine.