Commit Graph

138 Commits

Author SHA1 Message Date
Martin Jansa
43f1cf2d3d image-artifact-names: introduce new bbclass and move some variables into it
* similar to kernel-artifact-names for other recipes/bbclasses which
  need to use some deployed artifacts

* bitbake.conf: move IMAGE_BASENAME, IMAGE_VERSION_SUFFIX, IMAGE_NAME,
  IMAGE_LINK_NAME variables

* image_types.bbclass: move IMAGE_NAME_SUFFIX variable

* currently IMAGE_NAME_SUFFIX is used only by image.bbclass,
  image_types.bbclass and meta/recipes-core/images/build-appliance-image_15.0.0.bb
  but if it's needed by some recipe which isn't itself an image, then
  it's useful in bitbake.conf, e.g. we have a recipe for creating
  VirtualBox appliances which combines .wic.vmdk with .ovf file to
  create .zip with appliance, but for that we need the filename of
  .wic.vmdk which now contains IMAGE_NAME_SUFFIX
  4980ce52a4/meta-luneos/recipes-core/images/luneos-emulator-appliance.inc (L24)

* we were hardcoding .rootfs suffix where needed, but for quite long
  time it's configurable with IMAGE_NAME_SUFFIX since:

  commit 380ee36811939d947024bf78de907e3c071b834f
  Author: Patrick Ohly <patrick.ohly@intel.com>
  Date:   Mon Mar 7 18:07:52 2016 +0100

    image creation: allow overriding .rootfs suffix

  and might not match with hardcoded .rootfs, so make it easier to
  use IMAGE_NAME_SUFFIX where needed even without inheritting whole
  image_types.bbclass

[YOCTO #12937]

(From OE-Core rev: 456b700d51a5052a285a8477304f902c335223be)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-17 09:47:45 +01:00
Alexander Kanavin
c7ce37d3dd testimage: add an overall timeout setting
This is useful when tests misbehave and get stuck, or when
a significant increase in testing time is undesirable and
needs to be caught automatically.

(From OE-Core rev: d77546e910ad9048f0057f4465716d417b810065)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Alexander Kanavin
4ae71d667d testimage.bbclass: correctly process SIGTERM
Python's unittest will not propagate exceptions upside
of itself, but rather will just catch and print them.

The working way to make it stop is to send a SIGINT
(e.g. simulate a ctrl-c press), which will make it exit
with a KeyboardInterrupt exception.

This also makes pressing ctrl-c twice from bitbake work
again (previously hanging instances of bitbake and qemu were
left around, and bitbake would no longer start until they
were killed manually).

(From OE-Core rev: 06568a06ca22ee279d1829d26d4c38738233c06a)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-26 14:00:50 +01:00
Konrad Weihmann
da0ab97f61 testimage: enable ovmf support
Add support for running wic images with EFI as testimage.
Introduces a variable called QEMU_USE_OVMF for configuration.

(From OE-Core rev: 3af8aaff68ed332d812ea7dc184d392700ad7882)

Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24 14:10:08 +01:00
Robert P. J. Day
e1a2a5a206 fix various, innocuous typos
Corrections:

  - environment
  - accommodate
  - conversion
  - compatible

(From OE-Core rev: 9797d3b45b4e1b9d77f0f2ee299c17b48d8d3cf6)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-18 23:53:54 +00:00
Alexander Kanavin
ae6ec0b809 weston: add a basic runtime test
The test is checking that weston is able to start.

(From OE-Core rev: 57700767f90eea8f2b78187c42581aca42d50bbf)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-18 23:53:54 +00:00
Alejandro Hernandez Samaniego
a3416a5933 testimage: Extend runtime testing infrastructure to allow unconventional booting processes to be tested
The current runtime infrastructure contains hardcoded values which Ill refer to
as patterns, these patterns are either searched through or sent via the serial
terminal to communicate between HOST and TARGET.

These patterns are required since they allow us to check when a device has
finished booting, to log in, and to check whether a command sent from our tests
has returned, this way we are able to check both the status of the commands that
were sent along with its output.

The testing process goes somewhat as follows:
1. Launch QEMU and start booting.
2. Check when the device has booted by looking for the pattern login:.
3. Log in as the root user (default for our images).
4. Check that we were able to log in succesfully.
5. Start running the runtime test cases defined by TEST_SUITES.
6. One of such test cases could send a command to the QEMU target.
7. Check whether that command returned.
8. Check its output and status, return whether the test case passed or failed.

This patch allows this set of patterns to be defined instead of being hardcoded,
but it also automatically sets the defaults that we have been using in the past
if they have not been manually defined, for this reason, the patch is less
invasive and should not affect in any way how tests are currently being run.

Cases that can be enabled with this patch:
- A customized image that does not use the root user (or maybe we want to check
what happens if we dont use the root user).
- An image where the PS1 env variable has been modified, and the prompt pattern
wouldnt match the default.
- Baremetal applications, which do not follow the conventional way of booting
Linux and would probably not show a prompt for a user to log in, same applies
for testing bootloaders.
- poky-tiny: Using DISTRO=poky-tiny and an image such as the core-image-tiny
from meta-intel, which boots directly to RAM, and does not show a log in prompt
since it does not contain a conventional init process.

The code itself contains comments that should be self explanatory but here is an
example on how these patterns can be defined in a hypothetical case where we
want to run test cases as the webserver user instead:

TESTIMAGE_BOOT_PATTERNS = "send_login_user search_login_succeeded"
TESTIMAGE_BOOT_PATTERNS[send_login_user] = "webserver\n"
TESTIMAGE_BOOT_PATTERNS[search_login_succeeded] = "webserver@[a-zA-Z0-9\-]+:~#"

The variable TESTIMAGE_BOOT_PATTERNS defines which patterns to override when
used to communicate with the target when booting, anyone familiar with the
PACKAGECONFIG syntax should have no trouble setting these.

Other patterns would still be set up as default, e.g.
search_reached_prompt would still be login:

The accepted flags for TESTIMAGE_BOOT_PATTERNS are the following:
search_reached_prompt, send_login_user, search_login_succeeded,
search_cmd_finished.

They are prefixed with either search/send, to differentiate if the pattern is
meant to be sent or searched to/from the target terminal.

A working example of this code that falls under the baremetal case mentioned
above along with a test case is present on the meta-freertos layer, which tests
an RTOS image built with OpenEmbedded and automatically runs a test case on it
after booting such image:

As usual, INHERIT += "testimage" needs to be present on local.conf
$ bitbake freertos-demo -c testimage

RESULTS:
RESULTS - freertos_echo.FreeRTOSTest.test_freertos_echo: PASSED (2.00s)
SUMMARY:
freertos-demo () - Ran 1 test in 2.006s
freertos-demo - OK - All required tests passed (successes=1, skipped=0,
failures=0, errors=0)

(From OE-Core rev: 3ab2cbfeff371e8791b031a2852eeef80101a831)

Signed-off-by: Alejandro Hernandez Samaniego <aehs29@gmail.com>
Signed-off-by: Alejandro Hernandez Samaniego <alejandro@enedino.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-13 12:19:14 +00:00
Alejandro Hernandez Samaniego
e597809307 testimage: Allow testing on QEMU machines with a single serial port
commmit 6cde67d0a8 enables the use of qemurunner on machines that
only have a single serial port, but still sets the default value
as serial_ports=2 if not provided.

The testimage class does not call qemurunner with a serial_ports
argument, hence always defaulting to two.

Pass the serial_ports argument from the testimage class to allow
tests to run on QEMU machines with a single serial port.

(From OE-Core rev: a50d0163770f0b405a8de8a8a9cccd48c1de4112)

Signed-off-by: Alejandro Hernandez Samaniego <alejandro@enedino.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-13 12:19:14 +00:00
André Draszik
8073884454 testimage.bbclass: enable ssh agent forwarding
Some targets might use ssh to do their power- or serial-
control. In that case, ssh might need access to the
ssh agent, or otherwise won't work.

So export it into the environment.

Note that the (old) oeqa/controllers/masterimage.py
tries to do that as well by exporting all of BB_ORIGENV
into the test environment. Here in testimage.bbclass we
are a bit more strict and only pass the ssh related
environment variables.

(From OE-Core rev: d99617b8f2a9354b7357524fcd2eee16af0677b7)

Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-19 23:18:33 +01:00
André Draszik
c25a3336e2 testimage.bbclass: support hardware-controlled targets
Since the introduction of the new runtime framework for target
testing in commit 2aa5a4954d76
("testimage.bbclass: Migrate class to use new runtime framework")
commit 3857e5c91d in poky.git, target controllers have no
access to the global datastore 'd' anymore.

This makes it impossible for a specific OEQA (hardware)
controller to access documented properties like
TEST_POWERCONTROL_CMD, TEST_SERIALCONTROL_CMD, etc,
meaning it's impossible for those controllers to actually
control the hardware.

To solve this, simply add those documented variables into
the target_kwargs[].

(From OE-Core rev: 0ab7e3b573a58cc3a869ef33fd8737ca7fe04550)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-19 23:18:33 +01:00
Adrian Freihofer
13aa3bea40 testimage: consider QB_DEFAULT_FSTYPE
testimage.bbclass starts qemu with the first image type found in
the IMAGE_FSTYPES list. It's weird: this ['wic', 'tar'] works but
this ['tar'. 'wic'] does not. If QB_DEFAULT_FSTYPE is defined,
this fstype is booted.

(From OE-Core rev: aedb6bf9b6ccf37f69372642bc4c5dbbca92d0d9)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-08 16:01:41 +01:00
Mardegan, Alberto
e03d103e10 oeqa/core/runner: dump stdout and stderr of each test case
Some CI pipelines might perform further processing of the test output
(for instance, to plot some metrics into a chart). However, Since `thud`
we switched away from the XML-based jUnit reporting, and at the same
time we lost the ability of collecting the stdout and stderr of the
various tests.

We now restore this functionality by adding `stdout` and `stderr` keys
to the JSON reports. This behavior is off by default; in order to enable
it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake
configuration.

(From OE-Core rev: fd0048630ece5b21efb3a79e97046be0ab2a1514)

Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-26 10:09:08 +01:00
Alexander Kanavin
f7718fa8ae testimage.bbclass: add support for passing runqemu params
This is particularly useful when setting up GL tests.

(From OE-Core rev: 167a46775059b782c6f82ce8c5a47b27262e95d4)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-26 13:39:37 +00:00
Robert Yang
c856ed29df oeqa: make it work for multiple users
There are failures when multiple users run oe-selftest on the same
host:

PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved-
tests/201812250324_qemu'

This is because /tmp/oe-saved-tests was created by user A, while user B tries
to write data in it, then the error will happen. This patch can fix the
problem.

Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than
/tmp/oe-saved-tests/ to fix the problem.

(From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-11 10:39:09 +00:00
Robert Yang
dda8084d7a oeqa: Fix for QEMU_USE_KVM
Fixed:
MACHINE = "qemux86"
QEMU_USE_KVM = "qemux86"
IMAGE_CLASSES += "testimage"

$ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs

[snip]
  File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean
    raise ValueError("Invalid boolean value '%s'" % value)
ValueError: Invalid boolean value 'qemux86'

Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm
will be enabled if target_arch == build_arch or both of them are x86 archs.

(From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08 11:16:44 +00:00
Aníbal Limón
fed984dc8e meta/classes/testimage.bbclass: Only validate IMAGE_FSTYPES when is QEMU
When use simpleremote target the flash/boot process is executed
manually, the IMAGE_FSTYPES validation is only needed when execute
testimage against qemu.

The supported_fstypes comes from oeqa.core.target.qemu module.

(From OE-Core rev: e7dc5963adbacc091fe8943119262166977623ad)

Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-03 12:35:54 +00:00
Kai Kang
a32c80e6f0 testimage.bbclass: remove boot parameter systemd.log_target
Boot parameter systemd.log_target=console affects command journalctl's
output and causes oe selftest case test_systemd_boot_time fail to pass.

| Error at obtaining the boot time from journalctl
| RESULTS:
| RESULTS - systemd.SystemdJournalTests.test_systemd_boot_time - Testcase -1: SKIPPED (0.74s)

systemd.log_target=console was introduced by oe-core commit a0bb649 and
work with parameter systemd.log_level to enable systemd debug.
systemd.log_level has been removed already, so remove systemd.log_target
too to make case test_systemd_boot_time pass.

(From OE-Core rev: caa776bdcf8ea34c857f45970370bf771075f4bc)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-27 22:52:58 +00:00
Erik Botö
217035f098 testimage: Add possibility to pass parmeters to qemu
Add a variable called TEST_QEMUPARAMS in testimage.bbclass to make it
possible to pass parameters to qemu. This can be useful for e.g.
increasing the amount of RAM available during testimage runs.

(From OE-Core rev: 1a9163f5779d233c884c8fd50e0812eabab4fdf3)

Signed-off-by: Erik Botö <erik.boto@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-15 17:10:51 +00:00
Yeoh Ee Peng
fea124ae44 testimage: Add support for slirp
Enable testimage to support qemu slirp. Configure "QEMU_USE_SLIRP"
& "TEST_SERVER_IP" variables to enable slirp.

[YOCTO#10713]

(From OE-Core rev: 3df9ee85ce7fe52f0893fd33aea3bf1fcc6ead0a)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-23 23:35:18 +00:00
Richard Purdie
fabffda42e testimage: Enable autorunning of the package manager testsuites
Now that the hangs in httpservice are fixed we can let these tests
auto skip as appropriate.

(From OE-Core rev: 42a0d70291d551578e21f590fcb85ca72a78ccb5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-16 11:46:07 +00:00
Richard Purdie
99f0e68973 testimage/testsdk/selftest: Avoid platform.distro_identifier deprecation warnings
Use our own lsb function instead as used elsewhere by the codebase.

(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-16 11:46:07 +00:00
Richard Purdie
8291cd25d8 testimage: Further cleanup DEFAULT_TEST_SUITES
Now the test markup of the development tools tests is complete, this
can be further tweaked to auto run the correct tests.

(From OE-Core rev: bd4f8d12fe1f9f2643ee9e68fa2bb981134294fb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-14 11:14:40 +00:00
Richard Purdie
8c4ba1afdd testimage: Simplfy DEFAULT_TEST_SUITES logic
Now that the tests have correct markup to automatically determine which images
they can run against, clean up the default test suites logic to be simpler
and not image specific.

Some cleanup of the compiler tests still needs to be completed but this
is a good first step.

The only downside to this is more noise during testing as we now see
many skipped messages for simple images like core-image-minimal.

The auto type is being removed since it currently breaks badly due to the
socat mandatory inclusion from the meta-selftest layer which is a problem
which needs to be addressed seperately.

(From OE-Core rev: 4966bc33845752eb0aeae54b72e8ba0146a7ed52)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-14 11:14:40 +00:00
Richard Purdie
7ed0af48df testimage: Remove duplicate dependencies
(From OE-Core rev: 16d4368e7b85d5179374adb0a2d1ac215793e104)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-14 11:14:39 +00:00
Richard Purdie
8daaae4206 testimage: Improvements to the json logging
Tweak the preceeding commit to:

* Add STARTTIME to the identifier to make it unique
* Log DISTRO
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes

(From OE-Core rev: dda94cfa708cf3c6e5408d6dbcece360b4e8cd27)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29 17:26:47 +00:00
Yeoh Ee Peng
e73bcdd059 testimage.bbclass: write testresult to json files
As part of the solution to replace Testopia to store testresult,
OEQA testimage need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

By default, json testresult file will be written to "oeqa"
directory under the "WORKDIR" directory.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.

(From OE-Core rev: 5d135d4769b6bb60d575eb6ed196367f9e077cc4)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29 17:26:47 +00:00
Armin Kuster
3be5331986 testimage.bbclass: define ptest suite variable
Ptest has dependencies on other tests so contain them in a
variable. This allows layers to not worry about maintaining the depends
themselves.

(From OE-Core rev: c5c5ad9030ec7ddaa6387e2ba52e9e37b9ea76a2)

Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-25 14:53:44 +01:00
Richard Purdie
d91a282947 testimage: Ensure opkg/apt test suites are run
Currently only the dnf package manager tests are automatically added to the
list of tests to run. Improve the code to handle automatic addition of
the apt and opkg tests too.

(From OE-Core rev: c9169e4d1fa67289d5beb0e1f8fb5c54153c4242)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-12 16:57:21 +01:00
Richard Purdie
64ca57fe52 testimage: Ensure package-index is triggered for opkg/apt tests
If the opkg/apt tests are run without a package-index they will fail.
Trigger this here for now as a dependency until the code can be improved.

(From OE-Core rev: 7805eec0f522da2d2b6a08c8a07a9243e26c014a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11 23:06:35 +01:00
Richard Purdie
3ad7ccc21d testimage: Ensure TESTIMAGEDEPENDS is appended to for qemuall override
The current qemuall override overwrites other dependencies unintentionally.
Tweak the code to avoid this by appending to the variable.

(From OE-Core rev: fc8df45e6b27dd7a26b7bf88a06ffa831fcf075c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-11 23:06:35 +01:00
Emmanuel Roullit
a8948584c1 testimage.bbclass: fix qemu_use_kvm handling
QEMU_USE_KVM can either be a boolean or a whitespace separated list
of kvm supported machines.
For the 'intel-corei7-64' machine, defined in meta-intel, kvm could not be
used as the 'x86' substring is not part of its machine name.

By changing the order of this 'or' statement and setting
the 'QEMU_USE_KVM' variable to 'intel-corei7-64', it is possible to run the
'testimage' task with kvm support successfully.

(From OE-Core rev: a22789253aa653dc50fb159b40910248c2f98dd4)

Signed-off-by: Emmanuel Roullit <emmanuel.roullit@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-13 07:42:27 +01:00
Richard Purdie
1bbcc5931e bitbake.conf/testimage: Add in compatiblity hacks
The testimage changes break the autobuilder. The plan is to revert these hacks
but due to various changes happening with the autobuilder, we need these for now
to keep things working until we can move to the new autobuilder codebase.

(From OE-Core rev: 309a02931779f32d1139cc1169a039cbe4638706)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21 09:34:40 +01:00
Richard Purdie
fae0a9a439 image/testimage: Rework auto image test execution
The TEST_IMAGE interface has never particularly worked and image testing currently
gets configured manually.

This reworks the interface to better serve the needs of its users, replacing it
with TESTIMAGE_AUTO.

This does away with the need for the separate class due to better bitbake APIs for
changing tasks.

TESTIMAGE_AUTO will automatically boot under qemu any image that is built. It
also adds in dependencies so that any SDK for which testing is requested will
automatically be built first.

The code in bitbake.conf was error prone (e.g. testsdk wasn't considered), this
improves it to standardise on IMAGE_CLASSES as the standard configuration mechanism.

(From OE-Core rev: b34d44f3dfea8254826a46701a4fe3769a900434)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21 09:34:40 +01:00
Chen Qi
272ef77429 testimage.bbclass: move codes into testimage_main
testimage-auto is expected to run testimage task's codes automatically.
But in fact, it's currently missing some codes, including testimage_sanity
and create_rpm_index.

This leads to the problem of unexpected runtime failure of test_dnf_makecache.
The error message is as below.

  RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

This error is caused by the fact that create_rpm_index is not executed
before running the tests.

There's no reason why such codes should not be in testimage_main, so
move them into it.

(From OE-Core rev: fa7ba486ded13907f63f9300f66350ba2835a3f7)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04 15:15:00 +01:00
Chen Qi
d478a26e5f testimage.bbclass: also check 'auto' to create rpm index
Having 'auto' in TEST_SUITES will also run the 'dnf' test cases,
so also check it to determine whether to create rpm index or not.

This is to fix the following error when TEST_SUITES = "auto".

  RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

(From OE-Core rev: c1f4177848d25a9121f2a85da655ee414cd424b1)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04 15:15:00 +01:00
Chen Qi
4afe2567b7 testimage.bbclass: fix behavior of empty TEST_SUITES
The current behaviour of TEST_SUITES is very confusing.

setting: TEST_SUITES = ""
result:  Execute all test cases.

setting: TEST_SUITES = "some_case_not_exist"
result:  Error out with 'Empty test suite' message.

The expected behaviour of TEST_SUITES should be:
1. when 'auto' is in it, execute as many test cases as possible
2. when no valid test case is specified, error out and give user message

The first one is implemented by a previous patch.
The second one is fixed in this patch.

Also add debug message to show test cases to be executed. This is
for easier debugging.

(From OE-Core rev: 909568821fbad8a6a7034b10a2dc349a210fdfc6)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-04 15:15:00 +01:00
Jose Perez Carranza
cb68e9a2fe runtime/dnf: Add new dnf test cases
Add test cases to test “exclude” and “installroot“ options, also modify
the logic of filtering packages on the feed to have all the packages
needed by the tests.

[YOCTO #10744]

(From OE-Core rev: 1121806603c6f621d084b692216f3f616a0768dc)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-15 10:56:48 +01:00
Ross Burton
fb9e636538 testimage: run systemtap test on SDK images
(From OE-Core rev: f7f26bd0fe7085515bdebf23107ed8647a0c98fe)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-04-18 18:57:06 +01:00
Ross Burton
0c537554fc testimage: enable gi test suite
(From OE-Core rev: d8243c4588d4f1bb057fd917bfea130c4907e24c)

(From OE-Core rev: 1c896b34d2d9fdf941941bd29257a4714795da5a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-29 10:31:58 +00:00
Alexander Kanavin
9b32b8516a testimage.bbclass: add ptest to the list of runtime tests whenever possible
If no ptest packages are installed in the image, the test does nothing;
if ptest packages are installed in the image, then they should be
run without user having to enable that manually.

(From OE-Core rev: f57feab2727dca916744deb64825f3beaf07961d)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-05 11:55:35 +00:00
Robert Yang
60ceabb922 targetcontrol.py: use oe.types.boolean for QEMU_USE_KVM
So that both QEMU_USE_KVM = "True" and "1" will work.

[YOCTO #12343]

(From OE-Core rev: f28890ae2eb8a9cb2da39588e696a4b0c501bdf0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:56 +00:00
Richard Purdie
702f4692a2 testimage: Ensure full logs are shown for failures
Currently, the fact an error message is shown means the rest of the
task logs are suppressed. In this case we don't want that as it hides
the real errors and useful information. Therefore override this behaviour.

(From OE-Core rev: 7058ac23c5ce32fbdc4be6c05f85550e1fd9fe77)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-10 14:44:41 +00:00
Richard Purdie
7608f33f0f testimage: Pass the logger into OERuntimeTestContextExecutor.getTarget()
I have no idea why we didn't do this but it means the code has nowhere
to log to unless we do this. This means we can then use the logger
to log data to the task logs.

(From OE-Core rev: e241fb128840889acff79dbf3fae7bfaafcb5357)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-09 12:24:24 +00:00
Robert Yang
8a5d3bf302 testimage.bbclass: update comments
It's very important to add IMAGE_CLASSES += "testimage" in local.conf firstly,
otherwise the var like TEST_LOG_DIR (defined in testimage.bbclass) will not be
in testdata.json.

[YOCTO #11547]

(From OE-Core rev: 5663ed989f0af5b1c61c74288ec421cbca2261e7)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 14:16:01 +01:00
Khem Raj
3df1c95ba7 testimage: Use the renamed buildlzip
buildiptables has been replaced with buildlzip

(From OE-Core rev: 57cd682e05f82b63090b0cbb4a1d8feb3bfbff23)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:34:46 +01:00
Khem Raj
7acb3ae1cd testimage.bbclass: Correct the comment to state right dir for test cases
(From OE-Core rev: b464e338b30fe038e081e0dcae9f50c120d2f81c)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:08:31 +01:00
Aníbal Limón
e9a6fd9e97 oeqa: Change the order to logDetails and logSummary
Is better to log the summary at end to see in an easy way
the actual result of the test run.

[YOCTO #11622]

(From OE-Core rev: 4e3ab36e8c90abc740cce1ba31faf6595116e1e2)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-09 17:12:15 +01:00
Aníbal Limón
ead62f9a40 classes/test{export,images}: Change modules to list
With the new filters by <module>.[class].[testname] the modules
arg is expected to be a list so use split into TEST_SUITES variable.

(From OE-Core rev: b588c5c65e55a31d61c86553158eef409cd71086)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:14 +01:00
Aníbal Limón
b4b9e22c40 oeqa/core: Move OETestContext.log{Summary, Details} into OETestResult
Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.

[YOCTO #11450]

(From OE-Core rev: 33a783f59ed4e232f41f8b09dfa7955f2ddc2f80)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:13 +01:00
brian avery
36db8f2b78 testimage.bbclass: add additional dependency
qemu-native-helper has an additional task that needs to be run in order
for testimage to work. This task is usually run by default in a full
build but there are use cases where it might be skipped. This commit
adds the dependency explicitly.

Also, this commit adds a try/catch error message to make it clearer what
you need to do if you try to run testimage before you have built or
downloaded the image artifacts.

[YOCTO #11375]

(From OE-Core rev: 6e019537b9eb3af482e474a8cb248fe7312f4b58)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-21 08:22:09 +01:00