apertis-update-manager-ota-rollback manual

low

Image Types:
fixedfunction-armhf / fixedfunction-arm64 / fixedfunction-amd64
Image Deployment:
OSTree
Type:
functional

Description

Test the automatic rollback and blacklist mechanism of apertis-update-manager with network updates.


Resources

  • The DUT u-boot environment must be clean: in u-boot, run: `env default -a` followed by `saveenv`
  • A PC must be connected to DUT serial port
  • The DUT must be connected to network

Pre Conditions

  1. This test requires a properly configured time source: when testing devices not carrying a battery-backed real time clock in a network which prevents connections to NTP servers (and only in that case) manually ensure that the time on the device is set appropriately and that it is propagated to the hardware clock so it is stay set on the next reboot if the power is not plugged off.
  2. $ sudo timedatectl --adjust-system-clock set-time "2019-08-21 18:49:03"
  3. If direct access to repository with updates for DUT is restricted and proxy server should be used, then need to add the address of this proxy for OSTree on DUT by command:
  4. $ sudo ostree config set 'remote "origin"'.proxy "http://10.168.128.45:3128"

Execution Steps

  1. Check the initial deployment
  2. $ sudo ostree admin status
  3. Prepare the copy of commit and deploy to allow the upgrade to the same version
  4. Command below shows you an initial commit ID, for instance
  5. $ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
  6. Get the Collection ID and ref
  7. $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
    $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
  8. Create the commit with changed timestamp to allow upgrade with recent update file
  9. $ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago"); echo "New commit: $NEWID"
  10. Deploy the prepared commit
  11. $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
  12. Wait until the system is booted again and check the deployment
  13. $ sudo ostree admin status
  14. The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'
  15. Remove the initial deployment
  16. $ sudo ostree admin undeploy 1
  17. Reboot the system
  18. Check the current deployment
  19. $ sudo ostree admin status
  20. Remove blacklist file if it exists
  21. $ sudo rm -f /var/aum_blacklist.conf
  22. Start the user interface agent with mode preventing automatic system reboot after update
  23. $ sudo updatectl --register-upgrade-handler &
  24. Check if network update is available
  25. $ sudo updatectl --check-network-updates --dry-run

    AUM-Message: 09:38:03.883: Network connected: Yes

     AUM-Message: 09:38:03.890: Upgrade status: Checking

     AUM-Message: 09:38:03.890: Upgrade status: Checking

     AUM-Message: 09:38:05.967: Upgrade status: Available

  26. Enable network updates with CLI tool
  27. $ sudo updatectl --check-network-updates
  28. Check that the user interface agent reports the pending update
  29. AUM-Message: 09:41:03.263: Network connected: Yes

    AUM-Message: 09:41:03.268: Upgrade status: Checking

    AUM-Message: 09:41:03.269: Upgrade status: Checking

    AUM-Message: 09:41:04.503: Upgrade status: Available

    AUM-Message: 09:41:04.505: Upgrade status: Downloading

    AUM-Message: 09:41:16.943: Upgrade status: Deploying

    AUM-Message: 09:41:35.660: An upgrade is pending

  30. After the update, the device does *not* reboot automatically
  31. Check if there is pending deployment and reboot the DUT
  32. $ sudo ostree admin status
    $ sudo reboot
  33. In `U-Boot` console check the status of upgrade
  34. $ printenv bootcount bootlimit

    bootcount=1

    bootlimit=3

  35. Restart the device by pressing the restart button before the boot finishes.
  36. Restart the device a second time by pressing the restart button before the boot finishes.
  37. Restart the device a third time by pressing the restart button before the boot finishes.
  38. U-Boot should be able to detect the rollback mode and boot the system in rollback mode
  39. Warning: Bootlimit (3) exceeded. Using altbootcmd.

    Hit any key to stop autoboot:  0 

    switch to partitions #0, OK

    mmc0 is current device

    Scanning mmc 0:1...

    Found /extlinux/extlinux-rollback.conf

    Retrieving file: /extlinux/extlinux-rollback.conf

  40. Wait for system boot
  41. Wait a few seconds after the boot to allow ostree to undeploy the deployment. Check the update has been rolled back and that only single deployment exists.
  42. $ sudo ostree admin status
  43. Check if the file with blacklisted commit exists
  44. $ cat /var/aum_blacklist.conf

    [blacklist]

    7dfbc519eea384ed357920f733b051e1b06175834cbfdc1d6ef034bf7a5500ee=true

  45. Start the user interface agent
  46. $ sudo updatectl &
  47. Enable network updates with CLI tool with setting the update delay
  48. $ sudo updatectl --check-network-updates 15
  49. Check that the user interface agent reports the system is up to update
  50. AUM-Message: 09:47:19.339: Network connected: Yes

    AUM-Message: 09:47:19.343: Upgrade status: Checking

    AUM-Message: 09:47:19.344: Upgrade status: Checking

    AUM-Message: 09:47:21.100: Upgrade status: Available

    AUM-Message: 09:47:21.101: System is up to date

  51. Check the journal log should mention that the update ID has been blacklisted
  52. $ sudo journalctl -ef --unit apertis-update-manager --no-pager

     Apr 29 09:47:19 apertis apertis-update-[485]: Auto update status: active

    Apr 29 09:47:19 apertis apertis-update-[485]: Ostree upgrade poll starting

    Apr 29 09:47:21 apertis apertis-update-managerd[485]: libostree pull from 'origin' for apertis/v2022/armhf-uboot/fixedfunction complete

                                                          security: GPG: disabled 

                                                          security: SIGN: commit http: TLS

                                                          non-delta: meta: 1 content: 0

                                                          transfer: secs: 1 size: 95 bytes

    Apr 29 09:47:21 apertis apertis-update-managerd[485]: 1 metadata, 0 content objects fetched; 95 B transferred in 1 seconds; 0 bytes content written

    Apr 29 09:47:21 apertis apertis-update-[485]: Network upgrade is available

    Apr 29 09:47:21 apertis apertis-update-[485]: Revision '7dfbc519eea384ed357920f733b051e1b06175834cbfdc1d6ef034bf7a5500ee' is marked as blacklisted; skipping

    Apr 29 09:47:21 apertis apertis-update-[485]: Ostree already up to date

  53. Wait for 20 seconds, you should to see the update is triggered again and the output is similar to steps above.

Expected

U-Boot is able to detect rollback situation

U-Boot is able to use rollback configuration for bootloader

The "failed" update is rolled back

"Failed" update is marked as blacklisted

Apertis-update-manager is able to detect blacklisted update and refuse to update the system with it