Contents

  1. Persistent live systems
  2. mkusb / dus
  3. The classic dus-persistent method is described here
    1. Method developed from 'grub-n-iso'
      1. Cloning the head end of the drive - using 'grub.img'
    2. Using data from the source iso file
    3. Compressed image file with a persistent live system
      1. Download links
    4. Small 9w systems with guidus alias mkusb-dus and gparted installed
    5. Xubuntu Core 20.04 LTS with mkusb 12.4.5 (mkusb-dus and mkusb-plug)
    6. mkusb 12.5.0 makes a nice grub menu
    7. Link to mkusb page about Knoppix
    8. Partitions
    9. Comparison with 'live only' systems
    10. Advantages
    11. Disadvantages
  4. Alternatives
  5. 'Download and install automatically' can break persistent live media
    1. Discussion
    2. Select 'Display immediately', if your flavour and version is affected
    3. Solution for iso file versions that are affected
      1. Change the default settings by mkusb
      2. Change the default settings in the Ubuntu flavours
  6. Backup and restore of persistent overlay data
    1. Update and upgrade
      1. Common experience
      2. Test with update and full-upgrade
      3. home directory survives
    2. Backup & restore of the whole casper-rw partition
      1. Backup example
      2. Restore example
      3. Automatic tarball names
      4. Copy the tarball to another drive
      5. Select another directory/file
      6. Screenshots
    3. Backup & restore of the home directory in a casper-rw partition
      1. Upgrade (or repair) a persistent live system like this
      2. Tests
      3. Notes about backup & restore of the home directory
      4. Screenshots
  7. Installing - using the Install icon on the desktop
    1. The installer Ubiquity in most Ubuntu flavours
      1. Boot persistent live to RAM
    2. The installer Calamares in Lubuntu 20.04

Return to the main mkusb page


Persistent live systems

Most of the time it is enough with a live only USB pendrive and only a waste of effort to create a persistent live system. The cloning method of mkusb is very reliable and works with most linux distros. It needs hybrid iso files. But sometimes it is better to have a persistent live system, and beginning with version 10 mkusb can create such systems.

mkusb / dus

mkusb version 10-11, old:

../pictures/persist-01-persistence-selected-small.png

mkusb version 12, dus:

../pictures/persist-01-select-persistence-1202-small.png

Select dus-iso2usb or dus-persistent:

select_method-tool_persist_lubuntux.png


The classic dus-persistent method is described here

... and dus-iso2usb is described at the following link: ../dus-iso2usb.


../pictures/dus-24-persistent-live-system_quick-start-manual.png

Method developed from 'grub-n-iso'

This persistent live system is developed from booting via grub and the ISO file 'grub-n-iso' by Andre Rodovalho, described in

One pendrive for all PC (Intel/AMD) computers - single-boot dual-boot multi-boot and

How to Create a EFI/UEFI GRUB2 Multiboot USB drive to boot ISO images

The same pendrive can boot in UEFI mode and BIOS mode with a 32-bit operating system. Today 64-bit operating systems can boot almost all except the oldest computers. But this persistent live system cannot boot with secure boot. Data from usb-pack_efi.tar.gz are used to boot.

Cloning the head end of the drive - using 'grub.img'

The program package grub-pc cannot be used in installed systems running in UEFI mode. This means that the BIOS bootloader cannot be installed with the grub-install program. Instead, a compressed image file was made from a system created where is was possible, grub.img.xz. This image contains the first mibibyte of the drive plus the bios_grub and the usbboot partitions. The usbboot partition is also an efi partition for booting in UEFI mode.

This feature is introduced in mkusb version 12.2.7 and some details are described at ../12#Improvements_and_new_features

Using data from the source iso file

The grub boot system can be copied from an amd64 iso file. It can be modified slightly (to make a menuentry for persistence and to point to another partition). This works with the amd64 (64-bit) systems based on Ubuntu also with secure boot.

Persistent live systems made with this method boot in UEFI mode and can be made to boot in BIOS mode, if the program grub-install for BIOS is available. And it is, except in installed systems running in UEFI mode, where the package grub-pc cannot be installed without conflict with existing EFI grub packages. So this method provides a tool the make persistent live systems in UEFI mode and for UEFI mode (for amd64 systems).

In a live or persistent live system it is 'always' possible to install grub-pc, so it is possible to create systems that boot also in BIOS mode from a persistent live system running in UEFI mode.

Compressed image file with a persistent live system

Use a cloning tool, for example Disks or mkusb in linux, or 7-zip and Win32 Disk Imager in Windows to expand and {copy/clone/flash/burn/restore/install} this kind of files into a USB pendrive, a flash memory card or a hard disk drive or SSD with at least 8 GB. You can expand one or more partitions in order to use the whole drive, if it is bigger than 8 GB. This is described in the following link: growit.pdf; alternate link: ../../Installation/UEFI-and-BIOS/GrowIt.pdf

Short explanation: The original reason to create such images with mkusb was that, if the computer runs an installed Ubuntu system in UEFI mode, it will not be possible to install and use the package 'grub-pc' to make the target persistent live drive bootable in BIOS mode (because there is a conflict with 'grub-efi'). But in a persistent live Ubuntu, both 'grub-pc' and 'grub-efi' can be installed alongside each other. So it is possible to create systems that boot also in BIOS mode from a persistent live system running in UEFI mode. The current version of mkusb can work around that problem by cloning a small image of the head end of a drive, that boot in UEFI and BIOS mode.

But there can still be a reason to use these compressed images: They can be used also from Windows and be installed into a USB pendrive (by expansion and cloning).

There is a question "Why are img files provided for mkusb persistent but only iso file are allowed?" and an answer at AskUbuntu: with a more detailed explanation.

Download one of the following compressed image files from one of these links,

drive.google.com/open?id=0BzX-18u3W1sQUXI2YV95dHJxVlE

phillw.net/isos/linux-tools/uefi-n-bios

If one link does not work, please try the other one.

Small 9w systems with guidus alias mkusb-dus and gparted installed

If the explicit link (to http://phillw.net/isos/linux-tools/9w) fails, please try drive.google.com/open?id=0BzX-18u3W1sQUXI2YV95dHJxVlE

There are small files, only 230 MiB, as iso files, compressed image files and self-extracting exe files, that can provide systems with mkusb version 12 alias dus, guidus (June 2 2017). These systems can be cloned from their files to USB pendrives or memory cards in Windows as well as linux. These are 32-bit systems, that normally only boot in BIOS mode. One 9w image can boot in UEFI mode (but not with secure boot). It is a persistent live system using usb-pack-efi to allow booting also in UEFI mode.

See the following links

9W/Small 9w iso file with guidus and gparted installed

http://phillw.net/isos/linux-tools/9w/

Win32DiskImager/compressed-image_2_USB-or-SD

9W-DUS_i686.exe

These 9w systems can be used to create live-only as well as persistent live drives with Ubuntu and Debian Jessie.

../pictures/niow-12_9w-dus12.1.9-screenshot_800x600.png

../pictures/niow-13_9w-dus12.1.9-screenshot_work-done.png

Xubuntu Core 20.04 LTS with mkusb 12.4.5 (mkusb-dus and mkusb-plug)

New Xubuntu Core and new mkusb

  • This compressed image file contains a semi-cloned persistent live system made with mkusb-plug.
  • The following program packages are added: mkusb, usb-pack-efi, firefox, gedit.
  • Download link

    • md5sum=5705d687d4b75c4657a3d1392da5c13d

    • The size 1.14 GiB is smaller than the standard 20.04 LTS iso files (Xubuntu: 1.55 GiB, Lubuntu: 1.63 GiB).
  • It is straightforward to extract and clone from the compressed image file to a USB pendrive or memory card. The size of the drive should be at least 8 GB.

  • When you boot into this persistent live system, you will notice that some files are modified from the original Xubuntu versions,
    • boologo, txt.cfg, loopback.cfg, grub.cfg. This is expected with this systsem.

xubuntu-core_w_mkusb.png

mkusb 12.5.0 makes a nice grub menu

In mkusb 12.5.0 dus-persistent makes nice grub menus. The text size is controlled and there is a wallpaper. The menu looks the same in BIOS mode and UEFI mode. In high resolution screens the menu text was too small in UEFI mode, and it was difficult to read. Please notice that this feature works for Ubuntu and the Ubuntu family flavours with the default settings, but it does not work with usb-pack-efi. See this screenshot.

35-mkusb-dus-persistent-with-wallpaper1.png

The following link describes how to make a Knoppix system with mkusb, ../install-to-debian#Knoppix

Partitions

When you select persistent mkusb creates a partition for persistence in the target drive and the target system can use it automatically. It creates totally five partitions automatically for you.

In a separate menu window you can choose between a GUID partition table, GPT, and an old style MSDOS partition table. Normally GPT is recommended (and it works with huge drives (greater than 2 TB), but many HP computers need an MSDOS partition table to boot directly from USB.

The first partition, labeled 'usbdata', is actually located at the end of the drive, but it has number one because several Windows versions will only read the first partition (of the partition table) on USB drives. This partition has an NTFS file system, and can be used both when connected to a computer running linux and Windows, so it is suitable for transfer of files as well as a common storage.

The second partition is used to store program code for the grub BIOS bootloader if a GUID partition table, GPT, and as an extended partition, a container for logical partitions if an MSDOS partition table.

The persistent live drive boots from the third partition with a FAT32 alias vfat file system.

The operating system is cloned from the iso file to the fourth partition.

The fifth partition with an ext4 file system stores the overlay data for persistence.

  • partition: (NTFS) usbdata for storage and transfer of files

  • partition: GPT: bios_grub flag for booting in BIOS mode; MSDOS: extended partition

  • partition: (FAT32) boot partition

  • partition: (ISO 9660) cloned iso file

  • partition: (ext4) casper-rw or live-rw or persistence

../pictures/persist-05-mkusb-console-only-partitions-3.png

Comparison with 'live only' systems

Otherwise, when you select live only mkusb clones the iso file and there will be no persistence (except with some openSUSE iso files, that make persistent live systems as cloned). This kind of cloned system with an Ubuntu 64-bit operating system can boot also in secure mode.

Advantages

  • works with all current Ubuntu flavour desktop files (Ubuntu, Kubuntu, Lubuntu, ... Xubuntu) and with Linux Mint, LXLE, ToriOS, several other distros/re-spins based on Ubuntu and with Debian Jessie
  • very safe (minimal risk to overwrite the wrong drive by mistake)
  • easy to use
  • the target drive with the persistent live system works in [almost] all PC (Intel/AMD) computers

Disadvantages

  • works but less flexible in installed systems running in UEFI mode
  • cannot make a system that works in 32-bit systems as well as with secure boot with Ubuntu family iso files, but it works with LXLE and Bodhi Linux.
  • does not work with linux distros that are not based on Ubuntu or Debian (maybe you can tweak the grub.cfg file and make it work)
  • does not work with non-desktop iso files for example the Ubuntu mini.iso or the Ubuntu Server
  • in some computers (for example middle-aged HP desktop and laptop computers), ToriOS can boot only with a cloned 'live-only' pendrive, and with the system of method 3 from the compressed image file

Alternatives

Unetbootin and the 'Ubuntu Startup Disk Creator' alias usb-creator 0.2.x can also be used, when you want persistence in the target drive. They create and use a casper-rw file with the maximum size 4 GB (due to the file system FAT32). (But usb-creator version 0.3.2 in Xenial Xerus is cloning like mkusb makes live-only drives.)

Persistence works with an ext2, ext3 or ext4 partition with the label casper-rw, and that partition can be 'anywhere', in another pendrive or in an internal drive (which makes it faster). Persistence with an ext partition 'anywhere' works when booting from a DVD as well as from a cloned iso file (from a pendrive made by mkusb). Just add the boot option persistent when you want persistence. Then the live system will search for a casper-rw file or casper-rw partition.

Debian Jessie uses the partition label or file name persistence and ToriOS uses live-rw (instead of casper-rw, and this is managed by mkusb).

You can also create a home-rw partition to store the content of your home directory and it will survive even if the system modifications are damaged in the casper-rw partition.

'Download and install automatically' can break persistent live media

Look at the persistent live Ubuntu 16.04 LTS system - the 'Software & Updates' screen / 'Update' - and notice, that 'Automatic updates' is set to Download and install automatically alias 'unattended upgrades'. This can create severe problems in a persistent live system.

After leaving the persistent live Ubuntu 16.04 LTS system running overnight, it had performed an automatic upgrade:

df revealed that the content in casper-rw had increased to 1.6 GiB (in a fresh system with an almost empty casper-rw partition).

Discussion

If not enough space (and inodes) in the casper-rw partition or file, the upgrade operation will be incomplete and fail. There might be big problems even if there is space left in casper-rw. Finally, it is wasting the available space, if you have a small casper-rw partition or casper-rw file.

This is caused by a change of the default action, when there are security updates. A survey indicates that this default setting is different between the flavours and versions of Ubuntu. Lubuntu keeps the setting 'Display immediately', while the other tested flavours change it from 14.04 LTS to 16.04 LTS.

Select 'Display immediately', if your flavour and version is affected

Until this is resolved, it is a good idea to select 'Display immediately' via the graphical interface, but above all, to take regular backups, when you use a persistent live system. If you want more details, see the following link with screenshots from different Ubuntu flavours and versions:

Re: Unattended-upgrades can break persistent live media

Solution for iso file versions that are affected

Change the default settings by mkusb

A method to change from 'Download and install automatically' to 'Display immediately' was tested in mkusb version 11.0.2, and is available in the current versions of mkusb. See ../v11.

Change the default settings in the Ubuntu flavours

The bug report Bug #1619188: Unattended upgrades can break persistent live media was created at Launchpad in order to change the default settings for security updates for persistent live and live-only systems. The current setting 'Download and install automatically' is good for installed systems. The bug affects Ubuntu 16.04 and 16.04.1. It is squashed in Ubuntu 16.04.2 LTS.

Backup and restore of persistent overlay data

The partition that stores persistent overlay data is easily damaged, particularly if you remove the pendrive before it is unmounted.

Update and upgrade

It is a good idea to use persistence to add functionality by installing packages and tweak the system to what you like.

Common experience

But it is bad idea to update/upgrade a persistent live system continuously. Then you will soon get a borked system according to the experience of several people.

Test with update and full-upgrade

I have tested (and am still testing) a persistent live Lubuntu 16.04.1 LTS system in a 60 GB SSD. Compared to a typical USB pendrive the SSD is fast, big and reliable. It was possible to update and upgrade it to be up to date in June 2018, which is almost 2 years after the iso file was released, and the system works after several reboots and installation of new program packages.

sudo apt update
sudo apt full-upgrade
  • 'apt full-upgrade' throws errors caused by cryptsetup (and affecting the installer ubiquity too), but these are not affecting the way I use the persistent live drive. For example,
    • I can still install new program packages into the persistent live system,
    • I tested today, and I can install (Lubuntu 16.04.1) from the persistent live system, so ubiquity is still working,
    • It is always possible to select 'Try Lubuntu' or 'Install Lubuntu' when booting and run live-only, in order to install (with or without encrypted disk alias LVM with LUKS encryption).
  • 'df -h' reports 1.9 GiB used space in the 'casper-rw' partition, and there are no problems so far. Maybe this system will fail soon, maybe it will continue working for a long time. Anyway, particularly if you have a rather small USB pendrive, you don't want to waste drive space on system upgrades. So it is a good idea upgrade by switching to a system made from an iso file, that is up to date.

  • There are reasons to use a current kernel and its hardware drivers in the kernel series or skip to the next kernel series of 16.04.2 - 16.04.5 and 18.04 LTS. A persistent live system will only use the kernel from the iso file, so start from a current iso file for this purpose too.

home directory survives

If/when the persistence no longer works you may need to start all over again by removing what has been stored. Often you can keep the [overlay copy of the] home directory (and delete all other subdirectories of your overlay system in the casper-rw partition). Delete while running it live-only! This will save many settings, tweaks and personal files, but you must reinstall the program packages, that have been added.

Backup & restore of the whole casper-rw partition

Therefore it is important to have a convenient way to backup and restore the persistent data. This was introduced in mkusb 10.3.4 with shell-scripts to make it easy to backup and restore the overlayed persistent data of the persistent live system created by mkusb. These shell-scripts are written to

  • /dev/sdx1 mounted at /media/xubuntu/usbdata and

  • /dev/sdx3 which can be mounted at /mnt.

These scripts work in linux operating systems based on Ubuntu. In some versions, they are not executable, but can be run as parameter to bash. Boot into the live-only mode and run the scripts according to the following examples. (The scripts will complain if you try to use them in persistent mode.)

Backup example

cd /media/xubuntu/usbdata
ls -l
bash backup

Restore example

cd /media/xubuntu/usbdata
ls -l
bash restore

Automatic tarball names

The name of the tarball file (containing the backup) is updated automatically to create versions. Remove old versions manually!

1-casper-rw.tar.gz
2-casper-rw.tar.gz
3-casper-rw.tar.gz
...

Copy the tarball to another drive

Finally, to be sure the backup survives, you should copy the tarball to another drive, that you keep somewhere else, so that both backup copies are not lost. If you use the pendrive regularly, you should make backup copies of the casper-rw partition regularly too, maybe once a week or twice a month.

Select another directory/file

It is also possible to select another directory where to write/read the tarball. In that case, you add a parameter to the command line, for example

bash backup /media/xubuntu/backup  # target directory in another drive, a 'backup' drive

bash restore /media/xubuntu/backup/1-casper-rw.tar.gz  # source tarball in another drive, 'backup'

The commands under the hood of backup and restore are described in this post in the Ubuntu Forums.

Screenshots

See the whole screenshots at /LXLE#Backup_and_Restore

../pictures/persist-26-LXLE_16_04_02_32_persistent_live_backup-and-restore_after-backup_truncated.png

../pictures/persist-26-LXLE_16_04_02_32_persistent_live_backup-and-restore_after-restore_truncated.png

Backup & restore of the home directory in a casper-rw partition

New in version mkusb 12.3.2

The whole casper-rw partition can be restored to a persistent live drive made from the same iso file. But if you want to upgrade the persistent live system from a new daily iso file of an LTS system or to another version, it will not work. The overlay is not compatible with the system from from iso file.

However, if you backup the home directory in a casper-rw partition. you can restore it to a persistent live system from a new daily iso file of an LTS system or to another version (newer or older). So you can upgrade (and downgrade) a persistent live system including the data files, settings and tweaks in your home directory. But you must reinstall programs that you installed manually, and re-do global tweaks, if any.

This was introduced in mkusb 12.3.2 with shell-scripts to make it easy to backup and restore the overlayed persistent data of the persistent live system created by mkusb. These shell-scripts are merged into mkusb-dus. There is a detailed description at the following link to the Ubuntu Forums Tutorial "One pendrive for all PC (Intel/AMD) computers - single-boot dual-boot multi-boot"

Post #149 - #156: Backup and restore the /home directory in casper-rw partitions of mkusb persistent drives

There are two modes, to

  • backup the /home directory to a tarball
  • restore the /home directory from a tarball

The purpose of this tool is to make it easy to keep the persistent live drive up to date. It is usually possible to keep the home directory including personal files and the settings (usually in hidden configuration files in the user's home directory).

Upgrade (or repair) a persistent live system like this

  • Backup the /home directory to a tarball.
  • Create a fresh persistent live system of standard Ubuntu or the same flavour (Kubuntu, Lubuntu ... Xubuntu) as before using mkusb. Use the daily iso files of the LTS versions to get a system that is up to date.
  • Restore the /home directory from the tarball to the fresh persistent live system.

Tests

This method works with standard Ubuntu desktop and the Ubuntu community flavours. It is tested with

  • Ubuntu, Lubuntu and Xubuntu,
    • when upgraded within the same version
    • when upgraded from 16.04.1 LTS to 18.04 LTS
    • when downgraded from 16.04.1 LTS to 14.04.1 LTS
  • Ubuntu and Lubuntu,
    • when upgraded directly from 12.04.5 LTS to 14.04 LTS
    • when upgraded directly from 12.04.5 LTS to 16.04 LTS
    • when upgraded directly from 12.04.5 LTS to 18.04 LTS
    • when downgraded directly from 18.04 LTS to 16.04 LTS
    • when downgraded directly from 18.04 LTS to 14.04 LTS

Notes about backup & restore of the home directory

  • Developed and tested only for Ubuntu and Ubuntu community flavours.
  • Needs a graphical desktop environment and zenity.

  • Works in Wayland without extra tweak (the tasks with elevated permissions are done in text mode).
  • After some tweaks it is possible to use a tarball of the home directory of an installed system. See Using /home from an installed system in a persistent live system

  • You can backup a running persistent live drive. In that case you should not write the tarball into your home directory, but into another directory, for example an internal drive, another external drive or the drive's own 'usbdata' partition, if it is big enough. Mount the partition if necessary.

Screenshots

Select backup or restore persistent live home from the starter menu "Install / Backup / Restore / Wipe"

29-install-backup-restore-wipe_Backup-persistent-live-home.png

Select persistent live drive, if more than one is connected

08-mkusb-backup-n-restore-home.png

Link to more screenshots ../pictures#Backup-n-restore-home

Installing - using the Install icon on the desktop

Normally you would create a live-only USB drive, if you 'only' want to install your linux system, but the Install icon on the desktop is available also in a persistent live system.

The installer Ubiquity in most Ubuntu flavours

The installer icon starts the ubiquity installer, and there can be problems. The installer might be confused by the partition structure of the persistent live system created by mkusb.

Boot persistent live to RAM

This method works with standard Ubuntu, most Ubuntu flavours and older versions of persistent live Lubuntu:

- Boot a persistent live system with to RAM

- If you want to install with encrypted home, you should swapoff everything (including zRAM) before starting the installer

sudo swapoff -a

- Start the installer

- Let the installer unmount all partitions that it wants to unmount (including on the boot drive; this is possible when using to RAM)

The installer Calamares in Lubuntu 20.04

The installer icon starts the installer Calamares, and there can be problems (different from those of Ubiquity).

- Calamares works from live (live-only) as well as from persistent live systems made by mkusb.

- Calameres does not work from live (live-only) systems booted to RAM using the boot option 'toram'. See this bug report.


Return to the main mkusb page

mkusb/persistent (last edited 2022-09-01 13:31:22 by nio-wiklund)