.. contents::

Overview
--------

The upgrader works by reading the command file created by
``system-image-cli(1)``. This command file specifies, declaratively, how
to apply one ore more system-images to the system. The main command in
the command file is ``update`` which is handled by the
``Upgrader._cmd_update()`` method and requests a new system image be applied
to the running system.

System Update
-------------

Considerations
==============

Simplistically, all the ``Upgrader._cmd_update()`` method needs to do is
unpack each system image file in turn and apply it to the system.

However, since the upgrade happens on a live system, extra care must
be taken to satisfy the following requirements:

- Existing services are not unecessarily disrupted.

- The latest file versions from the newly applied system image(s)
  are being used.

Environment
===========

The root filesystem is "double-mounted" having a read-only bind mount
*on top* of the real read-only rootfs mount. This setup is required do
avoid running processes from being aware of the filesystem changes
applied by the upgrader.

Further, the upgrader is run in a new mount namespace where the
read-only root is made writable for the duration of the upgrade.

Steps
=====

An upgrade performs the following steps within the new, writable
rootfs mount namespace:

1. The root filesystem is made private.

   This is required since systemd makes root shared (the default is
   private), and the upgrader must ensure that the filesystem events
   that follow are not propagated to the rest of the system (in the
   main mount namespace).

2. All bind-mounted created for ``writable-paths(5)`` are unmounted.

   This ensures that the core updates do not touch the writable user
   partition.

3. A new bind mount to the rootfs is created in preparation for the
   unpack.

4. Each system image file is inspected. For every file that is going
   to be changed (or deleted), a hard link is created below
   ``/lost+found/`` with a path that mirrors the original.

5. If the system image specifies that any files should be deleted,
   they are removed.

6. The main system image is then unpacked directly over the bind mount
   created above.

7. At this point, there are two copies of all the files that were
   affected by the upgrade:

   a) The original file, referenced by the hard-link created below
      ``/lost+found/``.

   b) The new version of the file (with a different inode), which has
      just been unpacked onto the root filesystem.

8. A list of processes that are still using the *original* versions
   of the files is created.

9. The hard-link tree below ``/lost+found/`` is then walked - if no
   processes are referencing the original inode, it is unlinked
   (leaving only the new version of the file on disk).

10. A lookup is performed on each processes referencing one or more files
    we have replaced is performed to map back to that processes
    service. If possible, the service is restarted to:

    a) Ensure it is using the latest version of the file on disk.

    b) Allow the saved inodes it was referencing below ``/lost+found/``
       to be deleted.

11. If any processes could not be either mapped back to services, or
    those services failed to restart or if the system image requested it,
    a reboot is performed.

12. On next boot, any remaining links below ``/lost+found/`` will be
    deleted, since they are no longer in use by the services that could
    not be restarted.

13. If no reboot is required, the root filesystem in the upgraders
    mount namespace is made read-only once more and then the mount
    namespace created to perform the upgrade is destroyed.

    The final read-only switch may seem redundant as the mount context
    is about to be destroyed. However, it provides a further check on
    the upgrade operation since the kernel will disallow the remount
    if the filesystem is not representable (which would happen if files
    were outright deleted that were still in use by processes by example).

Overall Effect
==============

The overall effect of the upgrade process is that the main read-only
root filesystem mount context remains read-only for the duration of
the upgrade and post-upgrade, all processes are guaranteed to be using
the latest versions of the files from the applied system-images.

Design Rationale
----------------

The hard-link strategy is used to ensure the upgrade process is
race-free: ideally, the upgrader would only create hard-links for
files that it has calculated are in use by processes. But the
operation to determine that list of pids is inherently racy since the
system is dynamic and processes are being created and destroyed all
the time.

Hence, the upgrader creates hard-links initially for all the affected
files and then later removes the links it can after the new file
versions were atomically created. This guarantees that regardless of
when new processes are spawned, it is always possible to determine
post-unpack which processes are using the old files since they are
referencing the old inodes - any new processes that start can only
possibly reference the new inodes as the old ones have been removed
from their original locations.

Failure Strategy
----------------

The upgrader is reliant on the integrity of the system images. As long
as the unpack was successful, the integrity of the system is
guaranteed with the potential need for a single reboot to clear any
locks on inodes held by unrestartable services.

Security
--------

Note that the upgrader does *NOT* re-check file signatures since:

1) The signatures will have been checked by system-image-cli(1) on
   download.

2) The upgrader is run by system-image-cli(1) once it is has
   determined it is safe to proceed.

