Test Suite: v2020 - Test Cases


ade-commands automated

Test ade command and sub-command with default and custom options and arguments.

critical
apertis-update-manager-api manual

Test the apertis-update-manager automatic update. The automated version of this test: https://qa.apertis.org/aum-api.html

low
apertis-update-manager-automount manual

Test the apertis-update-manager automatic update via mass storage device.

medium
apertis-update-manager-diskfull manual

Ensure that failures due to disk full errors during updates have no effect on the system The automated version of this test: https://qa.apertis.org/aum-out-of-space.html

low
apertis-update-manager-ota-api manual

Test the apertis-update-manager API for updates over the air.

medium
apertis-update-manager-ota-auto manual

Test the apertis-update-manager automatic updates over the air.

medium
apertis-update-manager-ota-diskfull manual

Ensure that failures due to disk full errors during network updates have no effect on the system

medium
apertis-update-manager-ota-rollback manual

Test the automatic rollback and blacklist mechanism of apertis-update-manager with network updates.

medium
apertis-update-manager-ota-signed manual

Test the apertis-update-manager uses only signed updates over the air.

medium
apertis-update-manager-powercut manual

Ensure that failures due to power losses during updates have no effect on the system The automated version of this test: https://qa.apertis.org/aum-power-cut.html

low
apertis-update-manager-rollback manual

Test the automatic rollback and blacklist mechanism of apertis-update-manager. The automated version of this test: https://qa.apertis.org/aum-rollback-blacklist.html

low
apertis-update-manager-usb-unplug manual

Test the apertis-update-manager automatic update interrupted by mass storage removing.

critical
apertis-update-manager manual

Test the manual update with apertis-update-manager. The automated version of this test: https://qa.apertis.org/aum-offline-upgrade.html

low
apparmor-basic-profiles automated

Test basic apparmor profiles. Only parsing and syntax of profiles are tested for now. Security infrastructure: parsing and syntax of basic Apparmor profiles is part of strategic application and services confinement.

critical
apparmor-bluez-avrcp-volume manual

Test AppArmor profile for Bluez AVRCP volume up and down commands.

medium
apparmor-bluez-setup manual

Test AppArmor profiles for setting up of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices.

high
apparmor-bluez manual

Test AppArmor profile for support for SPP profile (bluez-spp) and core functionalities (bluez-core).

high
apparmor-chaiwala-system automated

Test that AppArmor's system wide rules works according to Apertis security goals (Chaiwala was an older codename for Apertis).

high
apparmor-dbus automated

Security infrastructure: AppArmor D-Bus mediation is part of strategic application and service confinement.

critical
apparmor-functional-demo automated

Show some main AppArmor functionalities, like transitions, net access or resource permissions, under the form of tests.

medium
apparmor-geoclue automated

Test that geoclue apparmor profiles are working properly.

medium
apparmor-gstreamer1-0 automated

Test that AppArmor profiles provided in GStreamer 1.0 works without Auditd complaining.

medium
apparmor-ofono manual

Test that AppArmor profiles provided in ofono package work without unexpected auditd log entries.

medium
apparmor-pulseaudio automated

Test that AppArmor profiles provided in PulseAudio work without apparmor complaining.

medium
apparmor-session-lockdown-no-deny manual

Test that the session lockdown profile is not blocking more than it should.

medium
apparmor-tumbler automated

Tests that the tumbler AppArmor profile doesn't cause false negatives and that it does not allow arbitrary reading of files in the home directory.

medium
apparmor-utils automated

Test apparmor_parser and other tools in apparmor package can be executed

medium
apparmor automated

Security infrastructure: checking that AppArmor is available in the kernel and active is part of strategic application and services confinement.

critical
aum-api automated

Apertis update manager: API test This test ensures that the update manager API behaves as expected.

critical
aum-offline-upgrade-collection_id automated

Apertis update manager: test of update failure with invalid update file. This test ensures that the update manager refuses to install an update with improper Collection ID, refs or timestamps.

critical
aum-offline-upgrade-signed automated

Apertis update manager: check signed offline upgrade test This test ensures that the update manager uses static bundle with signed commit only.

critical
aum-offline-upgrade automated

Apertis update manager: Offline upgrade test This test ensures that the update manager is able to install an update provided offline.

critical
aum-ota-api automated

Apertis update manager: OTA API test This test ensures that the update manager API behaves as expected.

critical
aum-ota-auto automated

Apertis update manager: automatic OTA updates test This test ensures that the update manager is able to work in unattended mode.

critical
aum-ota-out-of-space automated

Apertis update manager: Out of space update test during OTA upgrade This test ensures that the update manager behaves appropriately in low disk space conditions.

critical
aum-ota-rollback-blacklist automated

Apertis update manager: blacklist and rollback for OTA updates. This test ensures that the update manager is able to rollback and blacklist an invalid update.

critical
aum-ota-signed automated

Apertis update manager: check signed OTA upgrade test This test ensures that the update manager uses commit signed with known key during OTA upgrades.

critical
aum-out-of-space automated

Apertis update manager: Out of space update test This test ensures that the update manager behaves appropriately in low disk space conditions.

critical
aum-power-cut automated

Apertis update manager: Power cut resiliency test This test ensures that the update manager resists a power cut.

critical
aum-rollback-blacklist automated

Apertis update manager: Blacklist and rollback This test ensures that the update manager is able to rollback and blacklist an invalid update.

critical
bluez-avrcp-volume manual

Test the AVRCP volume up and down commands.

medium
bluez-hfp manual

Test Hands Free Profile (HFP) BlueZ feature.

medium
bluez-phone manual

Test all BlueZ features needing interaction with a phone (Pairing, OBEX, PBAP, AVRCP, A2DP, HSP, HFP, PAN). Strategic component: Bluetooth phone calls is part of smartphone interaction functionality.

critical
bluez-setup manual

Test setup of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices. Strategic component: Bluetooth setup and basic functionality is part of smartphone interaction functionality.

critical
boot-no-crashes automated

Check that nothing crashed when booting.

high
boot-performance automated

Test boot performance. This test checks boot performance using systemd-analyze statistics.

high
canterbury automated

Test for Canterbury library.

critical
cgroups-resource-control automated

Run cgroup resource control tests

medium
connman-new-supplicant automated

Test the wpa_supplicant interface.

medium
connman-pan-network-access manual

Test ConnMan support for bluetooth PAN Network Access Profile. This test will test internet access from Apertis to a bluetooth device sharing its internet connection. The underlying connection makes use of Bluetooth Network Access Profile with connman. First the two devices will be paired, then connection will be established. It will then be possible to verify that internet is usable from Apertis.

medium
connman-pan-tethering manual

Test ConnMan support for PAN connectivity.

medium
connman-services manual

Tests Network connection management: ensuring that ConnMan can connect to actual networks is part of strategic network management functionality.

critical
connman-usb-tethering manual

Test ConnMan support for USB tethering (connection side).

medium
connman automated

Runs some automated tests using the ConnMan D-Bus API. It guarantees most of the basic features are working. Network connection management: ensuring that ConnMan's APIs are accessible is part of strategic network management functionality.

critical
dbus-dos-reply-time automated

Check the reply time of a few D-Bus method calls while dbus-daemon is loaded by D-Bus traffic. Different patterns of D-Bus loads are tested: 1) A connection calling a D-Bus method repetitively. 2) A process calling a D-Bus method repetitively using a different connection each time to generate traffic related to connections joining and leaving the bus (NameOwnerChanged signals). 3) The same with 100 threads generating traffic. 4) Tens of threads registering hundreds of D-Bus match rules DoS resistance is part of strategic basic services functionality.

high
dbus-installed-tests automated

Regression tests for D-Bus.

high
didcot automated

Test the content hand-over feature by launching an application. Didcot service is responsible for the content hand-over by launching an application which supports the mime type or uri schemes . The script launches a browser application which supports http uri schemes

medium
disk-rootfs-fsck automated

Ensure that the root file system check is enabled on boot

high
eclipse-plugins-install-target manual

Test copying files to the remote device from within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device.

medium
eclipse-plugins-remote-debugging manual

Test remote debugging within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device and debug it remotely using Eclipse.

medium
eclipse-sysroot-management manual

Test sysroot management and update within Eclipse. It will go through configuring an existing sysroot in Eclipse, and then updating it from the web with the latest version.

medium
evolution-sync-bluetooth manual

Synchronizing Bluetooth-paired phone contacts to Evolution.

medium
folks-ofono-backend manual

Reading contacts from 3G modem SIM card.

medium
frome automated

Frome library test.

low
gettext-i18n automated

Check that gettext internationalization works.

medium
glib-gio-fs automated

Run the GLib gio tests that do I/O on the local filesystem.

medium
grilo automated

Check grilo metadata notifications, and filesystem browsing.

medium
gstreamer1-0-decode automated

Test video and audio decoders and demuxers on a list of media. Audio/video decoding through GStreamer is important but not required by any of the currently strategic functionalities.

high
gupnp-services automated

Run a few tests related to service browsing, introspection, etc.

medium
image-bootable manual

Test whether the image is bootable. Basic functionality: booting up.

critical
image-gui-start manual

Test whether the image starts a graphical environment. Basic functionality: ensuring that the GUI boots up.

critical
iptables-basic automated

Network connection management: ensuring that iptables is available and usable is part of strategic networking management functionality.

critical
iptables-nmap manual

Test the firewall using nmap from another computer.

high
newport automated

Run the Newport test suite

medium
nfsroot-simple-boot automated

This test checks that an image that was flashed using the nfsroot properly booted.

high
ofono-sms-receive manual

Test receiving SMS messages with oFono.

medium
ofono-sms-send manual

Test sending of SMS with oFono.

medium
ostree-collection-id automated

OSTree: Collection ID test This test ensures that the system is booted with proper Collection ID.

critical
polkit-parsing automated

Parse polkit policy files

high
rfkill-toggle manual

Test that rfkill is functional.

medium
rhosydd automated

Rhosydd module test.

medium
ribchester automated

Ribchester library test.

critical
sanity-check-manual manual

This test checks that the system bus is up, that a user session is created and that a graphical interface is successfully running in the system, as well as other similar basic services so that a stable and reliable environment can be confirmed before futher tests are executed. Basic functionality: services.

critical
sanity-check automated

This test checks that the system bus is up, that a user session is created and that a graphical interface is successfully running in the system, as well as other similar basic services so that a stable and reliable environment can be confirmed before futher tests are executed. Basic functionality: services.

critical
sdk-ade-remote-debugging manual

Test remote debugging with ADE. Cross-build an example HelloWorld agent, deploy it to the target device and debug it with proper backtraces.

high
sdk-code-analysis-tools-gcov automated

Checks basic gcov functionality by running a sample program a few different ways and ensuring that a sufficient portion of its code paths are executed.

medium
sdk-code-analysis-tools-splint automated

Checks basic splint functionality by checking its output when run on a simple source file.

medium
sdk-cross-compilation automated

Attempt building bash package in the devroot SDK images ship.

medium
sdk-dbus-tools-bustle manual

Test that bustle-dbus-monitor shows services on the session bus.

medium
sdk-dbus-tools-d-feet manual

Test that D-Feet both shows services on the session bus and that new services appear automatically.

medium
sdk-debos-image-building automated

Ensure that the sample image recipes can be built on the SDK and that the commands in the documentation work.

critical
sdk-debug-tools-gdb automated

Checks basic gdb functionality by stopping a program at a set point and checking its backtrace matches expectations.

medium
sdk-debug-tools-strace automated

Checks basic strace functionality by checking its output when run on a simple command line program.

medium
sdk-debug-tools-valgrind automated

Checks basic valgrind functionality by running a program with and without memory leaks, ensuring expected results from valgrind.

medium
sdk-ide-build-run-program manual

Creates a simple test project and confirms that compiling and executing work.

medium
sdk-performance-tools-gprof automated

Checks basic gprof functionality by ensuring a CPU-intensive program shows up in its log as consuming the vast majority of the CPU cycles.

medium
sdk-performance-tools-sysprof automated

Checks basic sysprof functionality by ensuring a CPU-intensive program shows up in its log as consuming the vast majority of the CPU cycles.

medium
sdk-persistent-disk-etc manual

Tests that persistent disk can be used when upgrading the SDK.

critical
sdk-persistent-disk-home-user manual

Tests that persistent disk can be used when upgrading the SDK.

critical
sdk-persistent-disk manual

Tests that persistent disk can be used when upgrading the SDK.

critical
sdk-vb-fullscreen manual

Tests that the SDK VM can be fullscreened in VirtualBox. Development environment: ensuring that the VirtualBox SDK provides a seamless experience. Check that when fullscreened the output is not stretched, even for accelerated contents.

critical
tiny-container-system-aa-enforcement automated

Test that the AppArmor profile for dbus-daemon is loaded in system-wide privileged container

medium
tiny-container-system-basic automated

Test of Tiny system-wide privileged container using shared networking with the host

medium
tiny-container-system-connectivity-profile automated

Test of AppArmor profile for Tiny system-wide privileged container

medium
tiny-container-system-device-sharing automated

Test of device sharing between host and Tiny system-wide privileged container

medium
tiny-container-system-folder-sharing automated

Test of folder sharing between host and Tiny system-wide privileged container started as user

medium
tiny-container-system-seccomp automated

Test that the custom seccomp profile could be applied for system-wide privileged container

medium
tiny-container-user-aa-enforcement automated

Test that the AppArmor profile for dbus-daemon is loaded in unprivileged container started as user

medium
tiny-container-user-basic automated

Test of Tiny unprivileged container without any networking started as user

medium
tiny-container-user-connectivity-profile automated

Test of AppArmor profile for Tiny unprivileged container started as user

medium
tiny-container-user-device-sharing automated

Test of device sharing between host and Tiny unprivileged container started as user

medium
tiny-container-user-folder-sharing automated

Test of folder sharing between host and Tiny unprivileged container started as user

medium
tiny-container-user-seccomp automated

Test that the custom seccomp profile could be applied for unprivileged container started as user

medium
traprain automated

Traprain library test.

medium
tumbler-thumbnailing automated

Check that all thumbnailing features of tumbler are working properly (large, small, videos, documents, images).

medium
video-animation-on-boot manual

Test whether Plymouth is able to display a graphical theme.

high
webkit2gtk-ac-3d-rendering manual

Test that 3D transforms are applied correctly.

low
webkit2gtk-ac-animations manual

Test that compositing layers are created for animating properly.

medium
webkit2gtk-ac-scrolling manual

Test that actors created for representing layers scroll properly.

medium
webkit2gtk-drag-and-drop manual

Test HTML5 drag-and-drop support in WebKit2GTK.

low
webkit2gtk-event-handling-redesign manual

Test WebKit2Gtk ability to scroll by touch with inertial behaviour after the touch ended.

medium
webkit2gtk-gstreamer1.0 manual

Tests html5 video playback with GStreamer 1.0 in WebKit2GTK. This test opens an URL to stream video and audio using MiniBrowser.

medium
webkit2gtk-mt-touch-events manual

Test implementation of W3C touch events API in WebKit2GTK.

low
x-hw-accelerated manual

Test whether the correct GLES2 render is used.

medium