apertis-update-manager-ota-diskfull manual

medium

Image Types:
minimal-armhf
Image Deployment:
OSTree
Type:
functional

Description

Ensure that failures due to disk full errors during network updates have no effect on the system

Resources

  • The DUT u-boot environment must be clean: in u-boot, run: `env default -a` followed by `saveenv`
  • A PC must be connected to DUT serial port
  • The DUT must be connected to network

Pre Conditions

  1. This test requires a properly configured time source: when testing devices not carrying a battery-backed real time clock in a network which prevents connections to NTP servers (and only in that case) manually ensure that the time on the device is set appropriately and that it is propagated to the hardware clock so it is stay set on the next reboot if the power is not plugged off.
    2. $ sudo timedatectl --adjust-system-clock set-time "2019-08-21 18:49:03"
  2. If direct access to repository with updates for DUT is restricted and proxy server should be used, then need to add the address of this proxy for OSTree on DUT by command:
    3. $ sudo ostree config set 'remote "origin"'.proxy "http://10.168.128.45:3128"

Execution Steps

  1. Check the initial deployment
    2. $ sudo ostree admin status
  2. Prepare the copy of commit and deploy to allow the upgrade to the same version
  3. Command below shows you an initial commit ID, for instance
    4. $ export BOOTID=$(sudo ostree admin status | sed -n -e 's/^\* apertis \([0-9a-f]*\)\.[0-9]$/\1/p'); echo $BOOTID
  4. Get the Collection ID and ref
    5. $ export CID=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 1 -d ' '); echo COLLECTION_ID=$CID
    $ export REF=$(sudo ostree refs -c | head -n 1 | tr -d '(),' | cut -f 2 -d ' '); echo REF=$REF
  5. Create the list of files to skip and enshure there are some files in these directories
    6. $ ls -1d /usr/share/locale /usr/share/man /usr/share/zoneinfo > /tmp/skip
    $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
  6. Create the commit with changed timestamp and skipped list from above to allow upgrade with recent update file
    7. $ export NEWID=$(sudo ostree commit --orphan --tree=ref=$BOOTID --add-metadata-string=ostree.collection-binding=$CID --bind-ref=$REF --timestamp="1 year ago" --skip-list=/tmp/skip); echo "New commit: $NEWID"
  7. Deploy the prepared commit
    8. $ sudo ostree admin upgrade --allow-downgrade --deploy-only --override-commit=$NEWID --reboot
  8. Wait until the system is booted again and check the deployment
    9. $ sudo ostree admin status
  9. The booted commit (started with '*') must have ID which we prepare and the initial commit ID should be marked as '(rollback)'
  10. Check booted deployment have no file objects which we skip
    11. $ du -sh /usr/share/locale /usr/share/man /usr/share/zoneinfo
  11. Remove the initial deployment
    12. $ sudo ostree admin undeploy 1
  12. Reboot the system
  13. Check the current deployment
    14. $ sudo ostree admin status
  14. Need to monitor the journal log
    15. $ sudo journalctl -ef --unit apertis-update-manager &
  15. Check the free space on rootfs, calculate the size of file to leave insufficient free space for upgrade to happen (this prevents the unexpected reboot during the test caused by other services). For instance, in the case below, NUM value would be 1451. It is the result of the calculation 1823-348-24, there 24 is a magic constant, so that almost, but not all free space is used. The free space can be calculated with command line `sudo df -B 1M / | grep / | awk '{print $2-$3-24}'`
    16. $ sudo df -B 1M /

    Filesystem           1M-blocks      Used Available Use% Mounted on

    /dev/mmcblk0p2            1823       348      1365  21% /

    $ sudo dd if=/dev/zero of=/var/bigfile bs=1M count=<NUM> ; echo "dd completed!"
  16. Wait until 'dd' completes (it should take time), you will see the output similar to

    17. 1521483776 bytes (1.5 GB) copied, 82.1661 s, 18.5 MB/s

    dd completed!

  17. Enable network updates with CLI tool
    18. $ sudo updatectl --check-network-updates
  18. The update should fail with error message

    19. ** Message: 20:59:57.249: Network connected: Yes

    ** Message: 20:59:57.251: Auto update status: active

    ** Message: 20:59:57.251: Ostree upgrade poll starting

    ** Message: 20:59:58.697: Ostree upgrade failed: min-free-space-percent '3%' would be exceeded

  19. Set reserved space for ostree to '0'
    20. $ sudo ostree config set core.min-free-space-percent "0"
  20. Restart the Apertis Update Manager
    21. $ sudo systemctl restart apertis-update-manager
  21. Enable network updates with CLI tool
    22. $ sudo updatectl --check-network-updates
  22. The update should fail with some error message related to disk space, for instance:

    23. ** Message: 21:02:31.972: Network connected: Yes

    ** Message: 21:02:31.974: Auto update status: active

    ** Message: 21:02:31.974: Ostree upgrade poll starting

    ** Message: 21:02:33.800: Ostree upgrade progress: 2 metadata, 0 content objects fetched; 284 B transferred in 1 seconds

    ** (process:532): WARNING **: 21:02:33.801: Cannot check the ID in black list: No such file or directory

    ** Message: 21:02:41.054: Ostree upgrade failed: Error writing to file descriptor: No space left on device

  23. Remove temporary file
    24. $ sudo rm -f /var/bigfile
  24. Reboot the system
    25. $ sudo reboot
  25. Check that there is only one deployment
    26. $ sudo ostree admin status

Expected

The update wasn't applied

System boots using the initial deployment