Difference between revisions of "Install Bubba 2 on RAID-1 volumes"

From BubbaWiki
Jump to navigation Jump to search
m
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Introduction  =
<h1> Introduction  </h1>
 
<p>As you may know, Bubba devices support RAID for home storage. However the install procedure does not offer the possibility to have your system partitions (root and swap) on raid. Besides the technical challenge, there are a few good reasons to raidify the system partitions. First it protects your fine-and-many-hours-spent tuned configuration of your device from hard drive failure. And second it gives your bubba a much better fault-tolerant design.  
As you may know, Bubba devices support RAID for home storage. However the install procedure does not offer the possibility to have your system partitions (root and swap) on raid. Besides the technical challenge, there are a few good reasons to raidify the system partitions. First it protects your fine-and-many-hours-spent tuned configuration of your device from hard drive failure. And second it gives your bubba a much better fault-tolerant design.  
</p><p>The goal of this how-to is to perform a plain standard software system installation on your bubba. You will have however the possibility to copy the data from your home partition. So with a backup of your settings from the web interface, you should get everything back.  
 
</p>
The goal of this how-to is to perform a plain standard software system installation on your bubba. You will have however the possibility to copy the data from your home partition. So with a backup of your settings from the web interface, you should get everything back.  
<h1> <span style="color:#FF0000">Big red warning</span>  </h1>
 
<p>This procedures involves commands changing u-boot variables. U-boot is the bootloader of the bubba device, located on a flash chip of the board. Although everything explained here has been carefully tested (at least by myself), you might mess things up and not be able to boot your bubba without a serial console cable (USB rescue rescue system won't fix it). Don't worry though, you will have plenty of verifications before doing the sensible stuff, and the risk is quite low. Nevertheless, you've been warned&nbsp;!!  
= <span style="color:#FF0000">Big red warning</span>  =
</p><p>In addition, <span style="color:#FF0000">these modifications prevent you from using the usb rescue disk</span>, because your system won't boot after a rescued installed software. The u-boot utils are not available in the rescue system, so you can't restore the modifications you've made to boot the raid array. I am currently working on a workaround, but as of today <u>it doesn't work</u>.  
 
</p>
This procedures involves commands changing u-boot variables. U-boot is the bootloader of the bubba device, located on a flash chip of the board. Although everything explained here has been carefully tested (at least by myself), you might mess things up and not be able to boot your bubba without a serial console cable (USB rescue rescue system won't fix it). Don't worry though, you will have plenty of verifications before doing the sensible stuff, and the risk is quite low. Nevertheless, you've been warned&nbsp;!!  
<h1> Prerequisites  </h1>
 
<p>This how-to has been tested only on <b>Bubba Two</b>. It would probably work on a bubba 3, with a proper configuration of u-boot stuff. If anyone owns a B3 and have validated that part, you're welcome to add it to this page.  
In addition, <span style="color:#FF0000">these modifications prevent you from using the usb rescue disk</span>, because your system won't boot after a rescued installed software. The u-boot utils are not available in the rescue system, so you can't restore the modifications you've made to boot the raid array. I am currently working on a workaround, but as of today <u>it doesn't work</u>.  
</p><p>This how-to applies to the bubba release 2.4RC1 for the bubba 2. I think it will probably work with future releases too. You will need&nbsp;:  
 
</p>
= Prerequisites  =
<ul><li>A working bubba two with 2.4RC1 software  
 
</li><li>An install/rescue USB stick with 2.4RC1 software on it  
This how-to has been tested only on '''Bubba Two'''. It would probably work on a bubba 3, with a proper configuration of u-boot stuff. I anyone owns a B3 and have validated that part, you're welcome to add it to this page.  
</li><li>An external esata hard drive, ideally the same model as your system disk (or at least the same size)  
 
</li><li>Screwdrivers to open the bubba and the external disk box (we will need to swap them).
This how-to applies to the bubba release 2.4RC1 for the bubba 2. I think it will probably work with future releases too. You will need&nbsp;:  
</li></ul>
 
<h1> Principles  </h1>
*A working bubba two with 2.4RC1 software  
<p>The procedure is divided in several parts&nbsp;:  
*An install/rescue USB stick with 2.4RC1 software on it  
</p>
*An external esata hard drive, ideally the same model as your system disk (or at least the same size)  
<ul><li>Preparing the new disk to receive the raid volumes, create them and copy the software and data on it.
*Screwdrivers to open the bubba and the external disk box (we will need to swap them).
</li></ul>
 
<p>In this step, from the running bubba system, we will partition the new hard drive with 4 partitions (/boot (50Mo), / (10Go), swap (1Go) and /home with the rest). We need to make a separate and small /boot raid partition because for an unknown reason u-boot refuses to boot on a 10Go root raid partition. Then we will create the four RAID-1 arrays with the newly created partitions&nbsp;; these arrays will be in a degraded state because we will add the other disk later. We won't change the kernel or the initrd, so the arrays will need to be configured to allow the kernel to detect them directly during its boot sequence.  
= Principles  =
</p><p>After creating the LVM volume for the data, we will format the arrays and extract the bubba filesystem from the install USB key on them. All these steps are greatly inspired by the standard install script from excito (apart from the raid part).  
 
</p><p>Then we will need to adjust <tt>/etc/fstab</tt> to reflect the disk configuration  
The procedure is divided in several parts&nbsp;:  
</p>
 
<ul><li>Modifying the boot behavior of the device
*Preparing the new disk to receive the raid volumes, create them and copy the software and data on it.
</li></ul>
 
<p>The boot modification is quite simple&nbsp;: we just need to change the kernel boot parameter <tt>root=/dev/sda1</tt> to <tt>root=/dev/md0</tt>. It is stored in the u-boot environnment as the <tt>diskdev</tt> parameter.  
In this step, from the running bubba system, we will partition the new hard drive with 4 partitions (/boot (50Mo), / (10Go), swap (1Go) and /home with the rest). We need to make a separate and small /boot raid partition because for an unknown reason u-boot refuses to boot on a 10Go root raid partition. Then we will create the four RAID-1 arrays with the newly created partitions&nbsp;; these arrays will be in a degraded state because we will add the other disk later. We won't change the kernel or the initrd, so the arrays will need to be configured to allow the kernel to detect them directly during its boot sequence.  
</p>
 
<ul><li>Powering off, physically swapping the disks and reboot
After creating the LVM volume for the data, we will format the arrays and extract the bubba filesystem from the install USB key on them. All these steps are greatly inspired by the standard install script from excito (apart from the raid part).  
</li></ul>
 
<p>I didn't manage to get u-boot to boot from a differend disk than the one plugged inside the bubba. So we need to place our newly configured disk inside the device. The old disk will go inside the external disk box. After everyhing is closed and plugged, we restart the bubba which should boot on the newly created RAID arrays.  
Then we will need to adjust <tt>/etc/fstab</tt> to reflect the disk configuration  
</p>
 
<ul><li>Reconfiguring the old system disk and add it to the RAID arrays.
*Modifying the boot behavior of the device
</li></ul>
 
<p>The final step is to repartition the old disk and add the partitions to the RAID arrays. The system will then automatically sync the drives, which concludes the setup.  
The boot modification is quite simple&nbsp;: we just need to change the kernel boot parameter <tt>root=/dev/sda1</tt> to <tt>root=/dev/md0</tt>. It is stored in the u-boot environnment as the <tt>diskdev</tt> parameter.  
</p>
 
<h1> Tasks  </h1>
*Powering off, physically swapping the disks and reboot
<p>All the commands below need to be run in a ssh session as root.  
 
</p>
I didn't manage to get u-boot to boot from a differend disk than the one plugged inside the bubba. So we need to place our newly configured disk inside the device. The old disk will go inside the external disk box. After everyhing is closed and plugged, we restart the bubba which should boot on the newly created RAID arrays.  
<h2> Preparing the new disk  </h2>
 
<h3> Partitions and raid arrays  </h3>
*Reconfiguring the old system disk and add it to the RAID arrays.
<ul><li>Plug/turn on the external disk. We assume from now that it has been detected as <tt>/dev/sdb</tt>.  
 
</li><li>Create the partitions with the following command&nbsp;:
The final step is to repartition the old disk and add the partitions to the RAID arrays. The system will then automatically sync the drives, which concludes the setup.  
</li></ul>
 
<pre class="_fck_mw_lspace">root@b2:~# sfdisk -uM /dev/sdb &lt;&lt; EOF
= Tasks  =
,50,fd
 
,10240,fd
All the commands below need to be run in a ssh session as root.  
,1024,fd
 
,,fd
== Preparing the new disk  ==
EOF
 
</pre>
=== Partitions and raid arrays  ===
<ul><li>Create the four arrays with the previously created partition&nbsp;:
 
</li></ul>
*Plug/turn on the external disk. We assume from now that it has been detected as <tt>/dev/sdb</tt>.  
<pre class="_fck_mw_lspace">root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md0 /dev/sdb1 missing
*Create the partitions with the following command&nbsp;:
mdadm: array /dev/md0 started.
 
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md1 /dev/sdb2 missing
root@b2:~# sfdisk -uM /dev/sdb &lt;&lt; EOF
mdadm: array /dev/md1 started.
,50,fd
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md2 /dev/sdb3 missing
,10240,fd
mdadm: array /dev/md2 started.
,1024,fd
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md3 /dev/sdb4 missing
,,fd
mdadm: array /dev/md3 started.
EOF
</pre>
 
<ul><li>Create the LVM data volume&nbsp;:
*Create the four arrays with the previously created partition&nbsp;:
</li></ul>
 
<pre class="_fck_mw_lspace">root@b2:~# pvcreate /dev/md3
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md0 /dev/sdb1 missing
Physical volume "/dev/md3" successfully created
mdadm: array /dev/md0 started.
root@b2:~# vgcreate bubba2 /dev/md3
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md1 /dev/sdb2 missing
Volume group "bubba2" successfully created
mdadm: array /dev/md1 started.
root@b2:~# lvcreate -l 100%FREE --name storage bubba2
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md2 /dev/sdb3 missing
Logical volume "storage" created
mdadm: array /dev/md2 started.
</pre>
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md3 /dev/sdb4 missing
<h3> Filesystem and data  </h3>
mdadm: array /dev/md3 started.
<ul><li>Format the swap array&nbsp;:
 
</li></ul>
*Create the LVM data volume&nbsp;:
<pre class="_fck_mw_lspace">root@b2:~# mkswap /dev/md2
 
</pre>
root@b2:~# pvcreate /dev/md3
<ul><li>Format and mount the system array:
Physical volume "/dev/md3" successfully created
</li></ul>
root@b2:~# vgcreate bubba2 /dev/md3
<pre class="_fck_mw_lspace">root@b2:~# mkfs.ext3 -q -L "Bubba root" /dev/md1
Volume group "bubba2" successfully created
root@b2:~# tune2fs -c0 -i0 /dev/md1
root@b2:~# lvcreate -l 100%FREE --name storage bubba2
root@b2:~# mkdir /mnt/bubba
Logical volume "storage" created
root@b2:~# mount /dev/md1 /mnt/bubba
 
</pre>
=== Filesystem and data  ===
<ul><li>Format and mount the /boot array:
 
</li></ul>
*Format the swap array&nbsp;:
<pre class="_fck_mw_lspace">root@b2:~# mkfs.ext2 -q -L "Bubba boot" /dev/md0
 
root@b2:~# tune2fs -c0 -i0 /dev/md0
root@b2:~# mkswap /dev/md2
root@b2:~# mkdir /mnt/bubba/boot
 
root@b2:~# mount /dev/md0 /mnt/bubba/boot
*Format and mount the system array:
</pre>
 
<ul><li>Format and mount the data array:
root@b2:~# mkfs.ext3 -q -L "Bubba root" /dev/md1
</li></ul>
root@b2:~# tune2fs -c0 -i0 /dev/md1
<pre class="_fck_mw_lspace">root@b2:~# mkfs.ext3 -q -L "Bubba home" /dev/mapper/bubba2-storage
root@b2:~# mkdir /mnt/bubba
root@b2:~# tune2fs -c0 -i0 /dev/mapper/bubba2-storage  
root@b2:~# mount /dev/md1 /mnt/bubba
root@b2:~# mkdir /mnt/bubba/home
 
root@b2:~# mount /dev/mapper/bubba2-storage /mnt/bubba/home
*Format and mount the /boot array:
</pre>
 
<ul><li>Plug in the USB install/rescue key. Assuming it is <tt>/dev/sdc</tt>, mount it and extract the OS&nbsp;:
root@b2:~# mkfs.ext2 -q -L "Bubba boot" /dev/md0
</li></ul>
root@b2:~# tune2fs -c0 -i0 /dev/md0
<pre class="_fck_mw_lspace">root@b2:~# mkdir /mnt/usb
root@b2:~# mkdir /mnt/bubba
root@b2:~# mount /dev/sdc1 /mnt/usb
root@b2:~# mount /dev/md0 /mnt/bubba/boot
root@b2:~# tar zxf /mnt/usb/install/payload/*.tar.gz -C /mnt/bubba
 
</pre>
*Format and mount the data array:
<ul><li>[Optional]&nbsp;: copy your data from home to the new disk&nbsp;:
 
</li></ul>
root@b2:~# mkfs.ext3 -q -L "Bubba home" /dev/mapper/bubba2-storage
<pre class="_fck_mw_lspace">root@b2:~# cp -a /home /mnt/bubba
root@b2:~# tune2fs -c0 -i0 /dev/mapper/bubba2-storage  
</pre>
root@b2:~# mkdir /mnt/bubba/home
<h3> Adjusting configuration  </h3>
root@b2:~# mount /dev/mapper/bubba2-storage /mnt/bubba/home
<ul><li>Edit <tt>/mnt/bubba/etc/fstab</tt> with your favorite editor so it contains&nbsp;:
 
</li></ul>
*Plug in the USB install/rescue key. Assuming it is <tt>/dev/sdc</tt>, mount it and extract the OS&nbsp;:
<pre class="_fck_mw_lspace">/dev/md1        /      ext3    noatime,defaults        0      1
 
/dev/md0        /boot  ext2    noatime,defaults        0      2
root@b2:~# mkdir /mnt/usb
/dev/mapper/bubba2-storage      /home  ext3    defaults                0      2
root@b2:~# mount /dev/sdc1 /mnt/usb
/dev/md2        none    swap    sw                      0      0
root@b2:~# tar zxf /mnt/usb/install/payload/*.tar.gz -C /mnt/bubba
usbfs          /proc/bus/usb  usbfs  defaults        0      0
 
/proc          /proc  proc    defaults                0      0
*[Optional]&nbsp;: copy your data from home to the new disk&nbsp;:
</pre>
 
<ul><li>Create a soft link in /boot&nbsp;:
root@b2:~# cp -a /home /mnt/bubba
</li></ul>
 
<pre class="_fck_mw_lspace">ln -s . /mnt/bubba/boot/boot
=== Adjusting configuration  ===
</pre>
 
<h2> Changing u-boot variables  </h2>
*Edit <tt>/mnt/bubba/etc/fstab</tt> with your favorite editor so it contains&nbsp;:
<ul><li>Install the uboot-envtools package&nbsp;:
 
</li></ul>
/dev/md1        /      ext3    noatime,defaults        0      1
<pre class="_fck_mw_lspace">root@b2:~# aptitude install uboot-envtools
/dev/md0        /boot  ext2    noatime,defaults        0      2
</pre>
/dev/mapper/bubba2-storage      /home  ext3    defaults                0      2
<ul><li>Create the configuration file for the uboot env tools&nbsp;:
/dev/md2        none    swap    sw                      0      0
</li></ul>
usbfs          /proc/bus/usb  usbfs  defaults        0      0
<pre class="_fck_mw_lspace">root@b2:~# cat &gt; /etc/fw_env.config &lt;&lt; EOF
/proc          /proc  proc    defaults                0      0
# MTD definition for Bubba|2
 
# MTD device name      Device offset  Env. size      Flash sector size
*Create a soft link in /boot&nbsp;:
/dev/mtd0              0x050000        0x002000        0x010000
 
/dev/mtd0              0x060000        0x002000        0x010000
ln -s . /mnt/bubba/boot/boot
EOF
 
</pre>
== Changing u-boot variables  ==
<ul><li>Test the uboot configuration&nbsp;; The command should output something like&nbsp;:
 
</li></ul>
*Install the uboot-envtools package&nbsp;:
<pre class="_fck_mw_lspace">root@b2:~# fw_printenv
 
baudrate=115200
root@b2:~# aptitude install uboot-envtools
loads_echo=1
 
......
*Create the configuration file for the uboot env tools&nbsp;:
</pre>
 
<p><span style="color:#FF0000"><b>WARNING&nbsp;: </b></span> You must not see the following output&nbsp;:  
root@b2:~# cat &gt; /etc/fw_env.config &lt;&lt; EOF
</p>
# MTD definition for Bubba|2
<pre class="_fck_mw_lspace">root@b2:~# fw_printenv  
# MTD device name      Device offset  Env. size      Flash sector size
Warning: Bad CRC, using default environment
/dev/mtd0              0x050000        0x002000        0x010000
bootcmd=bootp; setenv bootargs root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}::off; bootm
/dev/mtd0              0x060000        0x002000        0x010000
bootdelay=5
EOF
baudrate=115200
 
</pre>
*Test the uboot configuration&nbsp;; The command should output something like&nbsp;:
<p>If you do go check <tt>/etc/fw_env.config</tt>, with the content above. If it looks OK, then your system is different for a non obvious reason, and you cannot continue before fixing this issue.  
 
</p>
root@b2:~# fw_printenv
<ul><li>Modify the <tt>diskdev</tt> variable&nbsp;:
baudrate=115200
</li></ul>
loads_echo=1
<pre class="_fck_mw_lspace">root@b2:~# fw_setenv diskdev /dev/md1
......
</pre>
 
<p>If the above command fails, read carefully the error. Do not turn off or reboot your device before being sure that your bootloader will still work (if <tt>/etc/fw_env.config</tt> is correct with the above , there are no reasons that <tt>fw_setenv</tt> would break anything. Worst case scenario the environment will be reset on the next reboot). If in doubt ask on the forum&nbsp;!  
<span style="color:#FF0000">'''WARNING&nbsp;: '''</span> You must not see the following output&nbsp;:  
</p>
 
<ul><li>Check that the environment is correct&nbsp;:
root@b2:~# fw_printenv  
</li></ul>
Warning: Bad CRC, using default environment
<pre class="_fck_mw_lspace">root@b2:~# fw_printenv diskdev
bootcmd=bootp; setenv bootargs root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}::off; bootm
diskdev=/dev/md1
bootdelay=5
</pre>
baudrate=115200
<ul><li>If everything looks OK, turn off the bubba&nbsp;:
 
</li></ul>
If you do go check <tt>/etc/fw_env.config</tt>, with the content above. If it looks OK, then your system is different for a non obvious reason, and you cannot continue before fixing this issue.  
<pre class="_fck_mw_lspace">root@b2:~# /usr/lib/web-admin/backend.pl power_off
 
</pre>
*Modify the <tt>diskdev</tt> variable&nbsp;:
<ul><li>Once the bubba is off, you can turn off the external drive.
 
</li></ul>
root@b2:~# fw_setenv diskdev /dev/md1
<h2> Swapping disks and first boot  </h2>
 
<ul><li>Open the bubba and the external drive box, and swap the disks.  
If the above command fails, read carefully the error. Do not turn off or reboot your device before being sure that your bootloader will still work (if <tt>/etc/fw_env.config</tt> is correct with the above , there are no reasons that <tt>fw_setenv</tt> would break anything. Worst case scenario the environment will be reset on the next reboot). If in doubt ask on the forum&nbsp;!  
</li><li>Turn on the bubba first  
 
</li><li>If everything works, you should be able to configure your bubba as usual after a reinstall
*Check that the environment is correct&nbsp;:
</li></ul>
 
<p>You'll notice that the LED flashes quickly after the boot&nbsp;: that's because of the degraded RAID arrays.  
root@b2:~# fw_printenv diskdev
</p>
diskdev=/dev/md1
<h2> Reconfiguring old system disk  </h2>
 
<ul><li>Turn on the external drive&nbsp;; we'll assume it to be on <tt>/dev/sdc</tt>.  
*If everything looks OK, turn off the bubba&nbsp;:
</li><li>Partition the hard drive:  
 
<ul><li>If the disks models differs&nbsp;:
root@b2:~# /usr/lib/web-admin/backend.pl power_off
</li></ul>
 
</li></ul>
*Once the bubba is off, you can turn off the external drive.
<pre class="_fck_mw_lspace">root@b2:~# sfdisk -uM /dev/sdb &lt;&lt; EOF
 
,50,fd
== Swapping disks and first boot  ==
,10240,fd
 
,1024,fd
*Open the bubba and the external drive box, and swap the disks.  
,,fd
*Turn on the bubba first  
EOF
*If everything works, you should be able to configure your bubba as usual after a reinstall
</pre>
 
<ul><li><ul><li>Or, if the models are identical
You'll notice that the LED flashes quickly after the boot&nbsp;: that's because of the degraded RAID arrays.  
</li></ul>
 
</li></ul>
== Reconfiguring old system disk  ==
<pre class="_fck_mw_lspace">root@b2:~# sfdisk -d /dev/sda | sfdisk /dev/sdb
 
</pre>
*Turn on the external drive&nbsp;; we'll assume it to be on <tt>/dev/sdc</tt>.  
<ul><li>Add the newly created partitions to the raid arrays&nbsp;:
*Partition the hard drive:  
</li></ul>
**If the disks models differs&nbsp;:
<pre class="_fck_mw_lspace">root@b2:~# mdadm -a /dev/md0 /dev/sdb1
 
root@b2:~# mdadm -a /dev/md1 /dev/sdb2
root@b2:~# sfdisk -uM /dev/sdb &lt;&lt; EOF
root@b2:~# mdadm -a /dev/md2 /dev/sdb3
,50,fd
root@b2:~# mdadm -a /dev/md3 /dev/sdb4
,10240,fd
</pre>
,1024,fd
<p>The arrays will begin to sync right away&nbsp;; you can watch th progress in <tt>/proc/mdstat</tt>&nbsp;:  
,,fd
</p>
EOF
<pre class="_fck_mw_lspace">root@b2:~# cat /proc/mdstat
 
Personalities&nbsp;: [raid0] [raid1]  
**Or, if the models are identical
md1&nbsp;: active raid1 sdb2[2] sda2[0]
 
  10490368 blocks [2/1] [U_]
root@b2:~# sfdisk -d /dev/sda | sfdisk /dev/sdb
  [====&gt;................]  recovery = 24.0% (2518400/10490368) finish=3.3min speed=40208K/sec
 
 
*Add the newly created partitions to the raid arrays&nbsp;:
md2&nbsp;: active raid1 sdb3[2] sda3[0]
 
  1052160 blocks [2/1] [U_]
root@b2:~# mdadm -a /dev/md0 /dev/sdb1
  resync=DELAYED
root@b2:~# mdadm -a /dev/md1 /dev/sdb2
 
root@b2:~# mdadm -a /dev/md2 /dev/sdb3
md3&nbsp;: active raid1 sdb4[2] sda4[0]
root@b2:~# mdadm -a /dev/md3 /dev/sdb4
  476785024 blocks [2/1] [U_]
 
  resync=DELAYED
The arrays will begin to sync right away&nbsp;; you can watch th progress in <tt>/proc/mdstat</tt>&nbsp;:  
 
 
md0&nbsp;: active raid1 sdb1[1] sda1[0]
root@b2:~# cat /proc/mdstat
  56128 blocks [2/2] [UU]
Personalities&nbsp;: [raid0] [raid1]  
 
md1&nbsp;: active raid1 sdb2[2] sda2[0]
unused devices: &lt;none&gt;
    10490368 blocks [2/1] [U_]
</pre>
    [====&gt;................]  recovery = 24.0% (2518400/10490368) finish=3.3min speed=40208K/sec
<h1> Conclusion and what's left to do  </h1>
   
<p>If you've got that far, you owns a complete RAID-1 protected bubba, which is great (at least that's what I think)&nbsp;!  
md2&nbsp;: active raid1 sdb3[2] sda3[0]
</p><p>There are still some work to do&nbsp;:  
    1052160 blocks [2/1] [U_]
</p>
    resync=DELAYED
<ul><li>Understand why u-boot needs a small ext2 boot partition insteadof the default big root ext3 partition<br />
   
</li><li>Make uboot-envtools available from the USB rescue system to allow to restore u-boot environment to classic behavior<br />
md3&nbsp;: active raid1 sdb4[2] sda4[0]
</li></ul>
    476785024 blocks [2/1] [U_]
    resync=DELAYED
   
md0&nbsp;: active raid1 sdb1[1] sda1[0]
    56128 blocks [2/2] [UU]
   
unused devices: &lt;none&gt;
 
= Conclusion and what's left to do  =
 
If you've got that far, you owns a complete RAID-1 protected bubba, which is great (at least that's what I think)&nbsp;!  
 
There are still some work to do&nbsp;:  
 
*Understand why u-boot needs a small ext2 boot partition insteadof the default big root ext3 partition<br>
*Make uboot-envtools available from the USB rescue system to allow to restore u-boot environment to classic behavior<br>

Latest revision as of 18:14, 27 June 2013

Introduction

As you may know, Bubba devices support RAID for home storage. However the install procedure does not offer the possibility to have your system partitions (root and swap) on raid. Besides the technical challenge, there are a few good reasons to raidify the system partitions. First it protects your fine-and-many-hours-spent tuned configuration of your device from hard drive failure. And second it gives your bubba a much better fault-tolerant design.

The goal of this how-to is to perform a plain standard software system installation on your bubba. You will have however the possibility to copy the data from your home partition. So with a backup of your settings from the web interface, you should get everything back.

Big red warning

This procedures involves commands changing u-boot variables. U-boot is the bootloader of the bubba device, located on a flash chip of the board. Although everything explained here has been carefully tested (at least by myself), you might mess things up and not be able to boot your bubba without a serial console cable (USB rescue rescue system won't fix it). Don't worry though, you will have plenty of verifications before doing the sensible stuff, and the risk is quite low. Nevertheless, you've been warned !!

In addition, these modifications prevent you from using the usb rescue disk, because your system won't boot after a rescued installed software. The u-boot utils are not available in the rescue system, so you can't restore the modifications you've made to boot the raid array. I am currently working on a workaround, but as of today it doesn't work.

Prerequisites

This how-to has been tested only on Bubba Two. It would probably work on a bubba 3, with a proper configuration of u-boot stuff. If anyone owns a B3 and have validated that part, you're welcome to add it to this page.

This how-to applies to the bubba release 2.4RC1 for the bubba 2. I think it will probably work with future releases too. You will need :

  • A working bubba two with 2.4RC1 software
  • An install/rescue USB stick with 2.4RC1 software on it
  • An external esata hard drive, ideally the same model as your system disk (or at least the same size)
  • Screwdrivers to open the bubba and the external disk box (we will need to swap them).

Principles

The procedure is divided in several parts :

  • Preparing the new disk to receive the raid volumes, create them and copy the software and data on it.

In this step, from the running bubba system, we will partition the new hard drive with 4 partitions (/boot (50Mo), / (10Go), swap (1Go) and /home with the rest). We need to make a separate and small /boot raid partition because for an unknown reason u-boot refuses to boot on a 10Go root raid partition. Then we will create the four RAID-1 arrays with the newly created partitions ; these arrays will be in a degraded state because we will add the other disk later. We won't change the kernel or the initrd, so the arrays will need to be configured to allow the kernel to detect them directly during its boot sequence.

After creating the LVM volume for the data, we will format the arrays and extract the bubba filesystem from the install USB key on them. All these steps are greatly inspired by the standard install script from excito (apart from the raid part).

Then we will need to adjust /etc/fstab to reflect the disk configuration

  • Modifying the boot behavior of the device

The boot modification is quite simple : we just need to change the kernel boot parameter root=/dev/sda1 to root=/dev/md0. It is stored in the u-boot environnment as the diskdev parameter.

  • Powering off, physically swapping the disks and reboot

I didn't manage to get u-boot to boot from a differend disk than the one plugged inside the bubba. So we need to place our newly configured disk inside the device. The old disk will go inside the external disk box. After everyhing is closed and plugged, we restart the bubba which should boot on the newly created RAID arrays.

  • Reconfiguring the old system disk and add it to the RAID arrays.

The final step is to repartition the old disk and add the partitions to the RAID arrays. The system will then automatically sync the drives, which concludes the setup.

Tasks

All the commands below need to be run in a ssh session as root.

Preparing the new disk

Partitions and raid arrays

  • Plug/turn on the external disk. We assume from now that it has been detected as /dev/sdb.
  • Create the partitions with the following command :
root@b2:~# sfdisk -uM /dev/sdb << EOF
,50,fd
,10240,fd
,1024,fd
,,fd
EOF
  • Create the four arrays with the previously created partition :
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md0 /dev/sdb1 missing
mdadm: array /dev/md0 started.
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md1 /dev/sdb2 missing
mdadm: array /dev/md1 started.
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md2 /dev/sdb3 missing
mdadm: array /dev/md2 started.
root@b2:~# mdadm --create --metadata=0.9 --level=1 --raid-devices=2 /dev/md3 /dev/sdb4 missing
mdadm: array /dev/md3 started.
  • Create the LVM data volume :
root@b2:~# pvcreate /dev/md3
Physical volume "/dev/md3" successfully created
root@b2:~# vgcreate bubba2 /dev/md3
Volume group "bubba2" successfully created
root@b2:~# lvcreate -l 100%FREE --name storage bubba2
Logical volume "storage" created

Filesystem and data

  • Format the swap array :
root@b2:~# mkswap /dev/md2
  • Format and mount the system array:
root@b2:~# mkfs.ext3 -q -L "Bubba root" /dev/md1
root@b2:~# tune2fs -c0 -i0 /dev/md1
root@b2:~# mkdir /mnt/bubba
root@b2:~# mount /dev/md1 /mnt/bubba
  • Format and mount the /boot array:
root@b2:~# mkfs.ext2 -q -L "Bubba boot" /dev/md0
root@b2:~# tune2fs -c0 -i0 /dev/md0
root@b2:~# mkdir /mnt/bubba/boot
root@b2:~# mount /dev/md0 /mnt/bubba/boot
  • Format and mount the data array:
root@b2:~# mkfs.ext3 -q -L "Bubba home" /dev/mapper/bubba2-storage
root@b2:~# tune2fs -c0 -i0 /dev/mapper/bubba2-storage 
root@b2:~# mkdir /mnt/bubba/home
root@b2:~# mount /dev/mapper/bubba2-storage /mnt/bubba/home
  • Plug in the USB install/rescue key. Assuming it is /dev/sdc, mount it and extract the OS :
root@b2:~# mkdir /mnt/usb
root@b2:~# mount /dev/sdc1 /mnt/usb
root@b2:~# tar zxf /mnt/usb/install/payload/*.tar.gz -C /mnt/bubba
  • [Optional] : copy your data from home to the new disk :
root@b2:~# cp -a /home /mnt/bubba

Adjusting configuration

  • Edit /mnt/bubba/etc/fstab with your favorite editor so it contains :
/dev/md1        /       ext3    noatime,defaults        0       1
/dev/md0        /boot   ext2    noatime,defaults        0       2
/dev/mapper/bubba2-storage      /home   ext3    defaults                0       2
/dev/md2        none    swap    sw                      0       0
usbfs           /proc/bus/usb   usbfs   defaults        0       0
/proc           /proc   proc    defaults                0       0
  • Create a soft link in /boot :
ln -s . /mnt/bubba/boot/boot

Changing u-boot variables

  • Install the uboot-envtools package :
root@b2:~# aptitude install uboot-envtools
  • Create the configuration file for the uboot env tools :
root@b2:~# cat > /etc/fw_env.config << EOF
# MTD definition for Bubba|2
# MTD device name       Device offset   Env. size       Flash sector size
/dev/mtd0               0x050000        0x002000        0x010000
/dev/mtd0               0x060000        0x002000        0x010000
EOF
  • Test the uboot configuration ; The command should output something like :
root@b2:~# fw_printenv
baudrate=115200
loads_echo=1
......

WARNING : You must not see the following output :

root@b2:~# fw_printenv 
Warning: Bad CRC, using default environment
bootcmd=bootp; setenv bootargs root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}::off; bootm
bootdelay=5
baudrate=115200

If you do go check /etc/fw_env.config, with the content above. If it looks OK, then your system is different for a non obvious reason, and you cannot continue before fixing this issue.

  • Modify the diskdev variable :
root@b2:~# fw_setenv diskdev /dev/md1

If the above command fails, read carefully the error. Do not turn off or reboot your device before being sure that your bootloader will still work (if /etc/fw_env.config is correct with the above , there are no reasons that fw_setenv would break anything. Worst case scenario the environment will be reset on the next reboot). If in doubt ask on the forum !

  • Check that the environment is correct :
root@b2:~# fw_printenv diskdev
diskdev=/dev/md1
  • If everything looks OK, turn off the bubba :
root@b2:~# /usr/lib/web-admin/backend.pl power_off
  • Once the bubba is off, you can turn off the external drive.

Swapping disks and first boot

  • Open the bubba and the external drive box, and swap the disks.
  • Turn on the bubba first
  • If everything works, you should be able to configure your bubba as usual after a reinstall

You'll notice that the LED flashes quickly after the boot : that's because of the degraded RAID arrays.

Reconfiguring old system disk

  • Turn on the external drive ; we'll assume it to be on /dev/sdc.
  • Partition the hard drive:
    • If the disks models differs :
root@b2:~# sfdisk -uM /dev/sdb << EOF
,50,fd
,10240,fd
,1024,fd
,,fd
EOF
    • Or, if the models are identical
root@b2:~# sfdisk -d /dev/sda | sfdisk /dev/sdb
  • Add the newly created partitions to the raid arrays :
root@b2:~# mdadm -a /dev/md0 /dev/sdb1
root@b2:~# mdadm -a /dev/md1 /dev/sdb2
root@b2:~# mdadm -a /dev/md2 /dev/sdb3
root@b2:~# mdadm -a /dev/md3 /dev/sdb4

The arrays will begin to sync right away ; you can watch th progress in /proc/mdstat :

root@b2:~# cat /proc/mdstat
Personalities : [raid0] [raid1] 
md1 : active raid1 sdb2[2] sda2[0]
   10490368 blocks [2/1] [U_]
   [====>................]  recovery = 24.0% (2518400/10490368) finish=3.3min speed=40208K/sec
   
md2 : active raid1 sdb3[2] sda3[0]
   1052160 blocks [2/1] [U_]
   	resync=DELAYED
   
md3 : active raid1 sdb4[2] sda4[0]
   476785024 blocks [2/1] [U_]
   	resync=DELAYED
   
md0 : active raid1 sdb1[1] sda1[0]
   56128 blocks [2/2] [UU]
   
unused devices: <none>

Conclusion and what's left to do

If you've got that far, you owns a complete RAID-1 protected bubba, which is great (at least that's what I think) !

There are still some work to do :

  • Understand why u-boot needs a small ext2 boot partition insteadof the default big root ext3 partition
  • Make uboot-envtools available from the USB rescue system to allow to restore u-boot environment to classic behavior