apertis-update-manager-ota-rollback manual

low

Image Types:
fixedfunction-armhf / fixedfunction-arm64 / fixedfunction-amd64
Image Deployment:
OSTree
Type:
functional

Description

Test the automatic rollback and blacklist mechanism of apertis-update-manager with network updates.

Resources

  • The DUT u-boot environment must be clean: in u-boot, run: `env default -a` followed by `saveenv`
  • A PC must be connected to DUT serial port
  • The DUT must be connected to network

Pre Conditions

  1. This test requires a properly configured time source: when testing devices not carrying a battery-backed real time clock in a network which prevents connections to NTP servers (and only in that case) manually ensure that the time on the device is set appropriately and that it is propagated to the hardware clock so it is stay set on the next reboot if the power is not plugged off.
    2. $ sudo timedatectl --adjust-system-clock set-time "2019-08-21 18:49:03"
  2. If direct access to repository with updates for DUT is restricted and proxy server should be used, then need to add the address of this proxy for OSTree on DUT by command:
    3. $ sudo ostree config set 'remote "origin"'.proxy "http://10.168.128.45:3128"

Execution Steps

  1. Check the initial deployment
    2. $ sudo ostree admin status
  2. Prepare the copy of commit and deploy to allow the upgrade to the same version
  3. Command below shows you an initial commit ID, for instance
    4. $ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
  4. Get the Collection ID and ref
    5. $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
    $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
  5. Create the commit with changed timestamp to allow upgrade with recent update file
    6. $ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago"); echo "New commit: $NEWID"
  6. Deploy the prepared commit
    7. $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
  7. Wait until the system is booted again and check the deployment
    8. $ sudo ostree admin status
  8. The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'
  9. Remove the initial deployment
    10. $ sudo ostree admin undeploy 1
  10. Reboot the system
  11. Check the current deployment
    12. $ sudo ostree admin status
  12. Remove blacklist file if it exists
    13. $ sudo rm -f /var/aum_blacklist.conf
  13. Start the user interface agent with mode preventing automatic system reboot after update
    14. $ sudo updatectl --register-upgrade-handler &
  14. Check if network update is available
    15. $ sudo updatectl --check-network-updates --dry-run

    AUM-Message: 09:38:03.883: Network connected: Yes

     AUM-Message: 09:38:03.890: Upgrade status: Checking

     AUM-Message: 09:38:03.890: Upgrade status: Checking

     AUM-Message: 09:38:05.967: Upgrade status: Available

  15. Enable network updates with CLI tool
    16. $ sudo updatectl --check-network-updates
  16. Check that the user interface agent reports the pending update

    17. AUM-Message: 09:41:03.263: Network connected: Yes

    AUM-Message: 09:41:03.268: Upgrade status: Checking

    AUM-Message: 09:41:03.269: Upgrade status: Checking

    AUM-Message: 09:41:04.503: Upgrade status: Available

    AUM-Message: 09:41:04.505: Upgrade status: Downloading

    AUM-Message: 09:41:16.943: Upgrade status: Deploying

    AUM-Message: 09:41:35.660: An upgrade is pending

  17. After the update, the device does *not* reboot automatically
  18. Check if there is pending deployment and reboot the DUT
    19. $ sudo ostree admin status
    $ sudo reboot
  19. In `U-Boot` console check the status of upgrade
    20. $ printenv bootcount bootlimit

    bootcount=1

    bootlimit=3

  20. Restart the device by pressing the restart button before the boot finishes.
  21. Restart the device a second time by pressing the restart button before the boot finishes.
  22. Restart the device a third time by pressing the restart button before the boot finishes.
  23. U-Boot should be able to detect the rollback mode and boot the system in rollback mode

    24. Warning: Bootlimit (3) exceeded. Using altbootcmd.

    Hit any key to stop autoboot:  0 

    switch to partitions #0, OK

    mmc0 is current device

    Scanning mmc 0:1...

    Found /extlinux/extlinux-rollback.conf

    Retrieving file: /extlinux/extlinux-rollback.conf

  24. Wait for system boot
  25. Wait a few seconds after the boot to allow ostree to undeploy the deployment. Check the update has been rolled back and that only single deployment exists.
    26. $ sudo ostree admin status
  26. Check if the file with blacklisted commit exists
    27. $ cat /var/aum_blacklist.conf

    [blacklist]

    7dfbc519eea384ed357920f733b051e1b06175834cbfdc1d6ef034bf7a5500ee=true

  27. Start the user interface agent
    28. $ sudo updatectl &
  28. Enable network updates with CLI tool with setting the update delay
    29. $ sudo updatectl --check-network-updates 15
  29. Check that the user interface agent reports the system is up to update

    30. AUM-Message: 09:47:19.339: Network connected: Yes

    AUM-Message: 09:47:19.343: Upgrade status: Checking

    AUM-Message: 09:47:19.344: Upgrade status: Checking

    AUM-Message: 09:47:21.100: Upgrade status: Available

    AUM-Message: 09:47:21.101: System is up to date

  30. Check the journal log should mention that the update ID has been blacklisted
    31. $ sudo journalctl -ef --unit apertis-update-manager --no-pager

     Apr 29 09:47:19 apertis apertis-update-[485]: Auto update status: active

    Apr 29 09:47:19 apertis apertis-update-[485]: Ostree upgrade poll starting

    Apr 29 09:47:21 apertis apertis-update-managerd[485]: libostree pull from 'origin' for apertis/v2023/armhf-uboot/fixedfunction complete

                                                          security: GPG: disabled 

                                                          security: SIGN: commit http: TLS

                                                          non-delta: meta: 1 content: 0

                                                          transfer: secs: 1 size: 95 bytes

    Apr 29 09:47:21 apertis apertis-update-managerd[485]: 1 metadata, 0 content objects fetched; 95 B transferred in 1 seconds; 0 bytes content written

    Apr 29 09:47:21 apertis apertis-update-[485]: Network upgrade is available

    Apr 29 09:47:21 apertis apertis-update-[485]: Revision '7dfbc519eea384ed357920f733b051e1b06175834cbfdc1d6ef034bf7a5500ee' is marked as blacklisted; skipping

    Apr 29 09:47:21 apertis apertis-update-[485]: Ostree already up to date

  31. Wait for 20 seconds, you should to see the update is triggered again and the output is similar to steps above.

Expected

U-Boot is able to detect rollback situation

U-Boot is able to use rollback configuration for bootloader

The "failed" update is rolled back

"Failed" update is marked as blacklisted

Apertis-update-manager is able to detect blacklisted update and refuse to update the system with it