Save your SD cards: Raspberry Pi on a network file system

silex (25)in #raspberrypi • 6 years ago

If you work a lot with your Raspberry Pi, you probably have burned one or another SD card. While SD flash cards are great for storing photos or music, they suck at being written to very often. This is exactly the case if you run a whole operating system on them. Hundreds of thousands of read and write operations wear your SD cards out and if you do not have a solid backup concept you have probably lost some data, too.In my home, a couple of systems are running on the cheap, embedded hardware by the Raspberry Pi Foundation because they are both small and power-saving at a very affordable price point. And so are the accessories like cases and add-on-boards. At the time of writing, these are the jobs that are done by my Raspberries:

SSH server (as gateway and tunnel from the internet into the LAN)
Monitoring
OpenHAB smart home controller
TV Server
Backup
Ambilight

When living alone, having all this run on SD card-based Raspberries is fine and you know what to do when something breaks. As soon as you have a partner to share your home with, you probably need to agree on an SLA to ensure a high WAF:SLA (Service Level Agreement):

If you want to have a smart home controlled by a computer, you have a very short acceptable "TTR" if something breaks. Our Time-To-Recover is usually as short as 1 day. Honestly, it is more like same-day. Do not go to sleep until it is fixed.

WAF (Wife Acceptance Factor):

If you stick to the SLA, you are free to do all this crazy stuff; your partner accepts it and maybe even likes it.

And that is basically the reason I am writing this. It is completely OK if the ambilight stops working, it is also acceptable if the backups do not run for some days. But if the smart home controller fails, ... well, you better have a backup plan. You do not want to deal with someone who spent two hours trying to turn the lights up or the television on and ultimately failed because it simply does not work anymore without the central controlling instance.So we should reduce these outages to a minimum:

The Objective

Increase device uptime / reduce device outages
save data on a central storage for easy backups or snapshots

There are so many other things that can fail. The processor may overheat and hang, forcing a reboot by the watchdog, the power consumption could be too high and disable the USB bus (including the Ethernet Port), the cheap phone charger you use for powering may fail as well, but the single biggest problem is - without a doubt - the SD card.
So today I am showing you how simple it is to boot and run your Pi completely from the network.

The Requirements

always-on network storage with NFS server software
stable network connection to the storage (wired whenever possible)
16+ MB SD card for the Raspberry Pi bootloader and config file that points to your network storage
(Note: Starting with the Raspberry Pi 3 there are ways to modify the internal bootloader to boot from network without an SD card. I am not going to cover this here; this method will work with all the Raspberry Pi versions)
3 beers of time

Everything I do is based on a Linux computer (I use Arch Linux and Manjaro) but you can also do this on OS X and maybe on Windows with a lot of tools. I really recommend you to use a Linux computer or a linux VM.

The Network Storage

You need a network storage capable of running an nfs-server. You can even use a Raspberry Pi as sever but that would not really solve the problem here, would it? ;-)[caption id="attachment_105" align="alignleft" width="300"] Ensure the NFS service is globally enabled[/caption]Most commercial NAS systems support NFS and so does my Seagate BlackArmor NAS 4000. I highly recommend to create a seperate share for the filesystems of your Raspberries to be able to add stricter permissions later.You may need to enable the NFS service first, because a lot of people do not need it and use CIFS only.[caption id="attachment_116" align="alignnone" width="300"] Create a share where you will put the filesystems for your Raspis[/caption]The resulting config, that my NAS generates, looks like this:

~ $ grep rootfs /etc/exports

/nfs/rootfs *(rw,async,insecure,no_root_squash,no_subtree_check)

Please note that the options rw, no_root_squash and no_subtree_check are obligatory for what we want to achieve. For a detailed description of all possible options, please consult the man page on your system or here.

The Extraction Of The SD Card Image Content

You have your NFS Server up and running and configured.I already stumbled across some distributions of Raspberry Pi SD card images, that depend on and wait for partitions on the local sd card - even if you boot them from NFS. If you see your Raspberry Pi failing at boot, because it waited for a local partition or device for too long, simply flash the sd card image onto that sd card like you would normally. Then, change the settings to have it boot over NFS. Your local SD card still will not be touched, but the boot scripts and dependencies will work now.This topic can be split into two subtopics: Using a freshly downloaded image to start from scratch, or converting a normal SD-card-based Raspberry Pi to a NFS-based one. Skip to Convert SD-Card-Based Raspberry Pi To NFS if you already have an SD card image, which you want to convert.

Create A RootFS From Scratch (New Download)

Now you can download any SD card image like you would do normally. But instead of flashing it onto an SD card, you will mount it and copy the content to your NFS share.In this example I will download the official Raspberry Pi Foundation Raspbian Stretch Lite Image over the Torrent protocol on my Manjaro Linux machine.

[silex@lindaddy Downloads]$ transmission-cli https://downloads.raspberrypi.org/raspbian_lite_latest.torrent

2017-09-07-raspbian-stretch-lite.zip: State changed from "Incomplete" to "Complete"

Be nice and give something back to the torrent community by leaving this open seeding (uploading) for the other downloaders. Meanwhile in another terminal window:Check the checksum to check if the data was downloaded successfully and was not modified:

[silex@lindaddy Downloads]$ sha256sum 2017-09-07-raspbian-stretch-lite.zip 

bd2c04b94154c9804cc1f3069d15e984c927b750056dd86b9d86a0ad4be97f12 2017-09-07-raspbian-stretch-lite.zip

The checksum equals the checksum from the website. Great! Let's unpack it and have a look inside the partition table. It is a complete device image, so it contains everything from (or for) the SD card, including partition table.

[silex@lindaddy Downloads]$ unzip 2017-09-07-raspbian-stretch-lite.zip 

Archive: 2017-09-07-raspbian-stretch-lite.zip

 inflating: 2017-09-07-raspbian-stretch-lite.img

[silex@lindaddy Downloads]$ fdisk -l 2017-09-07-raspbian-stretch-lite.img 

Disk 2017-09-07-raspbian-stretch-lite.img: 1.7 GiB, 1854590976 bytes, 3622248 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

                                ^^^-----<<< write this down! <<<-------

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0x11eccc69



Device                           Boot Start  End      Sectors Size Id Type

2017-09-07-raspbian-stretch-lite.img1 8192   93813    85622   41.8M c W95 FAT32 (LBA)

2017-09-07-raspbian-stretch-lite.img2 94208  3622247  3528040 1.7G 83 Linux

                                      ^^^^^-----<<< and this! <<<-------

In the output of fdisk -l you see some important information. As we are not interested in putting the boot partition onto our NFS server (but the rootfs partition, the second partition) instead, we need to know the offset where the interesting partition starts. To get this offset you simply multiply the sector size (usually always 512 bytes) with the start sector of the second partition (both are marked above).

[silex@lindaddy Downloads]$ echo $(( 512 * 94208 ))

48234496

Let's create a loopback device with this information so we can mount it and steal the data from it.

[silex@lindaddy Downloads]$ sudo losetup -f --show -o $((512*94208)) 2017-09-07-raspbian-stretch-lite.img 

/dev/loop1

You need to be root to create loopback devices. -f found us a free loop device, -o took the offset we calculated and --show made sure we get to know which loop device losetup used. Now we can go and mount it like any other partition.

[silex@lindaddy Downloads]$ mkdir /tmp/fresh-pi-rootfs

[silex@lindaddy Downloads]$ sudo mount -v /dev/loop1 /tmp/fresh-pi-rootfs

mount: /dev/loop1 mounted on /tmp/fresh-pi-rootfs.

[silex@lindaddy Downloads]$ cd /tmp/fresh-pi-rootfs

[silex@lindaddy fresh-pi-rootfs]$ ls

bin boot dev etc home lib lost+found media mnt opt proc root run sbin srv sys tmp usr var

Now let's copy that data to your NFS share. Refer to the code snippet in Convert SD-Card-Based Raspberry Pi To NFS, but copy from /tmp/fresh-pi-rootfs instead of /.Afterwards, burn the SD card image to your SD card like you would normally. This ensures that you have the proper boot partition (the first partition in the partition table) on your card. If you feel fancy or have a tiny SD card, you can also extract the partition data like we did above and put it onto a fresh FAT partition on the SD card with the same start sector.

Convert SD-Card-Based Raspberry Pi To NFS

If you already have a running Raspberry Pi, you probably do not want to rebuild everything from scratch but move your existing data to the network share. All you need to do is to log into your Pi, install rsync (or any other application that reliably copies filesystem attributes like scp), and copy all your stuff over. Do not forget to stop your applications to have them in a defined (stopped) state.

root@raspberrypi ~ $ mkdir /tmp/rootfs

root@raspberrypi ~ $ mount nas:/nfs/rootfs /tmp/rootfs

root@raspberrypi ~ $ rsync -Phax --numeric-ids / /tmp/rootfs/openhab

There are about 31.000 files to copy, so this might take a while. Go brew some coffee or grab a beer.

The Configuration

All your data are belong to the NFS share. The last step is to tell the Raspberry Pi to boot from the share instead of the local SD card. One widespread method that works on almost every linux device is to modify the kernel command line. This is usually done in the bootloader (Syslinux, Grub, U-Boot). The Raspberry Pi got a configurable U-Boot bootloader starting with version 3. In the versions before you have to have a partition on your SD-Card, containing the boot files and a file called "cmdline.txt" that contains the kernel command line. This also works with Version 3 and newer, if you do not want to bother with U-Boot and you are fine with using an SD card for that.Modify the "cmdline.txt" file on the boot partition. You can do this directly on your Raspberry Pi, or by plugging the SD card into another computer. When using another computer, you'll find the file on the first Partition of the card, usually called "boot" partition. When editing the file directly on the Pi, it is located at "/boot/cmdline.txt".This is an example cmdline.txt I use in production:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.21.1:/nfs/rootfs/openhab ip=dhcp elevator=deadline

This is basically the default cmdline.txt with additions of the root, nfsroot and ip options. If you want to define a static IP address, use something like

ip=192.168.21.50::192.168.21.1:255.255.255.0:raspberrypi:eth0:off



(ip=<client-IP-number>::<gateway-IP-number>:<netmask>:<client-hostname>:eth0:off)

Please keep in mind, that there must only be exactly one line in the cmdline.txt file. No comments, nothing else. Just one line.

The First Boot

Save it, unmount the boot partition (Apple and Microsoft call this "eject securely" or similar), put the SD card into your Pi and fire it up. If you have a screen attached, you will see the communication with the DHCP server and the handover to the NFS filesystem. This usually takes only a second. My boot time increases by 50% which is most likely the case because of my very old and slow NAS. Results may vary.Congratulations, you just separated hardware from software resources and made your IT a lot more agile. Now, if your Pi explodes, you can just swap it with a new one from you Raspberry-Pi-drawer and be up again in no time. I am not responsible for exploding Pis, though.

The Remaining Questions

People asked me about this a lot so maybe this is interesting for you as well:Q: Wait! What happens, if my NAS crashes, or is offline in some other way?
A: As soon as the NFS server doesn't answer anymore, your devices will just stop completely. The root filesystem is essential so your Raspberry Pi will just wait until it is reachable again. It will not crash and it won't reboot or panic. It just waits until your server is back. I think this is quite nice. If you have any other questions, feel free to ask in the comments section below!