From 2497f29afd1ff66964b84dc79a7cd1126bbc94fa Mon Sep 17 00:00:00 2001 From: Antonin Godard Date: Thu, 26 Dec 2024 16:20:16 +0100 Subject: [PATCH] ref-manual/packages: move ptest section to the test-manual [ YOCTO #15106 ] It makes more sense to document ptests in the test-manual. Since ptests are still related to packages, keep a link to ptests from packages.rst to the test-manual. Reported-by: Yoann Congal (From yocto-docs rev: 8b6ada020d595d86c7bbe78a27b7a6301715b039) Signed-off-by: Antonin Godard (cherry picked from commit b389c06b709e4791e1cce5e8a5b58f6b0cd03a14) Signed-off-by: Antonin Godard Signed-off-by: Steve Sakoman --- documentation/dev-manual/packages.rst | 112 +---------------- .../migration-guides/migration-1.6.rst | 2 +- documentation/ref-manual/classes.rst | 4 +- documentation/ref-manual/features.rst | 2 +- documentation/ref-manual/qa-checks.rst | 1 - documentation/ref-manual/release-process.rst | 2 +- documentation/ref-manual/variables.rst | 2 +- documentation/test-manual/index.rst | 1 + documentation/test-manual/intro.rst | 2 +- documentation/test-manual/ptest.rst | 114 ++++++++++++++++++ 10 files changed, 125 insertions(+), 117 deletions(-) create mode 100644 documentation/test-manual/ptest.rst diff --git a/documentation/dev-manual/packages.rst b/documentation/dev-manual/packages.rst index e5028fffdc..0c948cdaef 100644 --- a/documentation/dev-manual/packages.rst +++ b/documentation/dev-manual/packages.rst @@ -16,7 +16,7 @@ This section describes a few tasks that involve packages: - :ref:`dev-manual/packages:generating and using signed packages` - :ref:`Setting up and running package test - (ptest) ` + (ptest) ` - :ref:`dev-manual/packages:creating node package manager (npm) packages` @@ -887,114 +887,8 @@ related to signed package feeds are available: Testing Packages With ptest =========================== -A Package Test (ptest) runs tests against packages built by the -OpenEmbedded build system on the target machine. A ptest contains at -least two items: the actual test, and a shell script (``run-ptest``) -that starts the test. The shell script that starts the test must not -contain the actual test --- the script only starts the test. On the other -hand, the test can be anything from a simple shell script that runs a -binary and checks the output to an elaborate system of test binaries and -data files. - -The test generates output in the format used by Automake:: - - result: testname - -where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and -the testname can be any identifying string. - -For a list of Yocto Project recipes that are already enabled with ptest, -see the :yocto_wiki:`Ptest ` wiki page. - -.. note:: - - A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest` - class. - -Adding ptest to Your Build --------------------------- - -To add package testing to your build, add the :term:`DISTRO_FEATURES` and -:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which -is found in the :term:`Build Directory`:: - - DISTRO_FEATURES:append = " ptest" - EXTRA_IMAGE_FEATURES += "ptest-pkgs" - -Once your build is complete, the ptest files are installed into the -``/usr/lib/package/ptest`` directory within the image, where ``package`` -is the name of the package. - -Running ptest -------------- - -The ``ptest-runner`` package installs a shell script that loops through -all installed ptest test suites and runs them in sequence. Consequently, -you might want to add this package to your image. - -Getting Your Package Ready --------------------------- - -In order to enable a recipe to run installed ptests on target hardware, -you need to prepare the recipes that build the packages you want to -test. Here is what you have to do for each recipe: - -- *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:* - Include the following line in each recipe:: - - inherit ptest - -- *Create run-ptest:* This script starts your test. Locate the - script where you will refer to it using - :term:`SRC_URI`. Here is an - example that starts a test for ``dbus``:: - - #!/bin/sh - cd test - make -k runtest-TESTS - -- *Ensure dependencies are met:* If the test adds build or runtime - dependencies that normally do not exist for the package (such as - requiring "make" to run the test suite), use the - :term:`DEPENDS` and - :term:`RDEPENDS` variables in - your recipe in order for the package to meet the dependencies. Here - is an example where the package has a runtime dependency on "make":: - - RDEPENDS:${PN}-ptest += "make" - -- *Add a function to build the test suite:* Not many packages support - cross-compilation of their test suites. Consequently, you usually - need to add a cross-compilation function to the package. - - Many packages based on Automake compile and run the test suite by - using a single command such as ``make check``. However, the host - ``make check`` builds and runs on the same computer, while - cross-compiling requires that the package is built on the host but - executed for the target architecture (though often, as in the case - for ptest, the execution occurs on the host). The built version of - Automake that ships with the Yocto Project includes a patch that - separates building and execution. Consequently, packages that use the - unaltered, patched version of ``make check`` automatically - cross-compiles. - - Regardless, you still must add a ``do_compile_ptest`` function to - build the test suite. Add a function similar to the following to your - recipe:: - - do_compile_ptest() { - oe_runmake buildtest-TESTS - } - -- *Ensure special configurations are set:* If the package requires - special configurations prior to compiling the test code, you must - insert a ``do_configure_ptest`` function into the recipe. - -- *Install the test suite:* The :ref:`ref-classes-ptest` class - automatically copies the file ``run-ptest`` to the target and then runs make - ``install-ptest`` to run the tests. If this is not enough, you need - to create a ``do_install_ptest`` function and make sure it gets - called after the "make install-ptest" completes. +See the :ref:`test-manual/ptest:Testing Packages With ptest` section of the +Yocto Project Test Environment Manual. Creating Node Package Manager (NPM) Packages ============================================ diff --git a/documentation/migration-guides/migration-1.6.rst b/documentation/migration-guides/migration-1.6.rst index c50786a36d..ecd4f2c1af 100644 --- a/documentation/migration-guides/migration-1.6.rst +++ b/documentation/migration-guides/migration-1.6.rst @@ -220,7 +220,7 @@ Package Test (ptest) Package Tests (ptest) are built but not installed by default. For information on using Package Tests, see the -":ref:`dev-manual/packages:testing packages with ptest`" +":ref:`test-manual/ptest:testing packages with ptest`" section in the Yocto Project Development Tasks Manual. For information on the ``ptest`` class, see the ":ref:`ref-classes-ptest`" section. diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst index 52a77a11be..b609c3af66 100644 --- a/documentation/ref-manual/classes.rst +++ b/documentation/ref-manual/classes.rst @@ -2478,7 +2478,7 @@ runtime tests for recipes that build software that provides these tests. This class is intended to be inherited by individual recipes. However, the class' functionality is largely disabled unless "ptest" appears in :term:`DISTRO_FEATURES`. See the -":ref:`dev-manual/packages:testing packages with ptest`" +":ref:`test-manual/ptest:testing packages with ptest`" section in the Yocto Project Development Tasks Manual for more information on ptest. @@ -2491,7 +2491,7 @@ Enables package tests (ptests) specifically for GNOME packages, which have tests intended to be executed with ``gnome-desktop-testing``. For information on setting up and running ptests, see the -":ref:`dev-manual/packages:testing packages with ptest`" +":ref:`test-manual/ptest:testing packages with ptest`" section in the Yocto Project Development Tasks Manual. .. _ref-classes-python3-dir: diff --git a/documentation/ref-manual/features.rst b/documentation/ref-manual/features.rst index bd393abd07..4d5f900ab8 100644 --- a/documentation/ref-manual/features.rst +++ b/documentation/ref-manual/features.rst @@ -157,7 +157,7 @@ metadata: - *ptest:* Enables building the package tests where supported by individual recipes. For more information on package tests, see the - ":ref:`dev-manual/packages:testing packages with ptest`" section + ":ref:`test-manual/ptest:testing packages with ptest`" section in the Yocto Project Development Tasks Manual. - *smbfs:* Include SMB networks client support (for mounting diff --git a/documentation/ref-manual/qa-checks.rst b/documentation/ref-manual/qa-checks.rst index 1e5fe21b02..92d2a42afb 100644 --- a/documentation/ref-manual/qa-checks.rst +++ b/documentation/ref-manual/qa-checks.rst @@ -789,7 +789,6 @@ Errors and Warnings use a relative path rather than an absolute one, or to pick up the path from runtime configuration or environment variables. - Configuring and Disabling QA Checks =================================== diff --git a/documentation/ref-manual/release-process.rst b/documentation/ref-manual/release-process.rst index 5733dd70ff..89f79812e8 100644 --- a/documentation/ref-manual/release-process.rst +++ b/documentation/ref-manual/release-process.rst @@ -175,7 +175,7 @@ consists of the following pieces: operation and functions. However, the test can also use the IP address of a machine to test. -- :ref:`ptest `: +- :ref:`ptest `: Runs tests against packages produced during the build for a given piece of software. The test allows the packages to be run within a target image. diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 7f0d09903b..8ab7391f98 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst @@ -6434,7 +6434,7 @@ system and gives an overview of their function and contents. :term:`PTEST_ENABLED` Specifies whether or not :ref:`Package - Test ` (ptest) + Test ` (ptest) functionality is enabled when building a recipe. You should not set this variable directly. Enabling and disabling building Package Tests at build time should be done by adding "ptest" to (or removing it diff --git a/documentation/test-manual/index.rst b/documentation/test-manual/index.rst index 86a2f436ea..ad71f37910 100644 --- a/documentation/test-manual/index.rst +++ b/documentation/test-manual/index.rst @@ -12,6 +12,7 @@ Yocto Project Test Environment Manual intro test-process + ptest understand-autobuilder reproducible-builds yocto-project-compatible diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index bb267d2d88..ff9b6d3794 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst @@ -141,7 +141,7 @@ the following types of tests: - *Package Testing:* A Package Test (ptest) runs tests against packages built by the OpenEmbedded build system on the target machine. See the :ref:`Testing Packages With - ptest ` section + ptest ` section in the Yocto Project Development Tasks Manual and the ":yocto_wiki:`Ptest `" Wiki page for more information on Ptest. diff --git a/documentation/test-manual/ptest.rst b/documentation/test-manual/ptest.rst new file mode 100644 index 0000000000..dea1bad23b --- /dev/null +++ b/documentation/test-manual/ptest.rst @@ -0,0 +1,114 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +*************************** +Testing Packages With ptest +*************************** + +A Package Test (ptest) runs tests against packages built by the +OpenEmbedded build system on the target machine. A ptest contains at +least two items: the actual test, and a shell script (``run-ptest``) +that starts the test. The shell script that starts the test must not +contain the actual test --- the script only starts the test. On the other +hand, the test can be anything from a simple shell script that runs a +binary and checks the output to an elaborate system of test binaries and +data files. + +The test generates output in the format used by Automake:: + + result: testname + +where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and +the testname can be any identifying string. + +For a list of Yocto Project recipes that are already enabled with ptest, +see the :yocto_wiki:`Ptest ` wiki page. + +.. note:: + + A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest` + class. + +Adding ptest to Your Build +========================== + +To add package testing to your build, add the :term:`DISTRO_FEATURES` and +:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which +is found in the :term:`Build Directory`:: + + DISTRO_FEATURES:append = " ptest" + EXTRA_IMAGE_FEATURES += "ptest-pkgs" + +Once your build is complete, the ptest files are installed into the +``/usr/lib/package/ptest`` directory within the image, where ``package`` +is the name of the package. + +Running ptest +============= + +The ``ptest-runner`` package installs a shell script that loops through +all installed ptest test suites and runs them in sequence. Consequently, +you might want to add this package to your image. + +Getting Your Package Ready +========================== + +In order to enable a recipe to run installed ptests on target hardware, +you need to prepare the recipes that build the packages you want to +test. Here is what you have to do for each recipe: + +- *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:* + Include the following line in each recipe:: + + inherit ptest + +- *Create run-ptest:* This script starts your test. Locate the + script where you will refer to it using + :term:`SRC_URI`. Here is an + example that starts a test for ``dbus``:: + + #!/bin/sh + cd test + make -k runtest-TESTS + +- *Ensure dependencies are met:* If the test adds build or runtime + dependencies that normally do not exist for the package (such as + requiring "make" to run the test suite), use the + :term:`DEPENDS` and + :term:`RDEPENDS` variables in + your recipe in order for the package to meet the dependencies. Here + is an example where the package has a runtime dependency on "make":: + + RDEPENDS:${PN}-ptest += "make" + +- *Add a function to build the test suite:* Not many packages support + cross-compilation of their test suites. Consequently, you usually + need to add a cross-compilation function to the package. + + Many packages based on Automake compile and run the test suite by + using a single command such as ``make check``. However, the host + ``make check`` builds and runs on the same computer, while + cross-compiling requires that the package is built on the host but + executed for the target architecture (though often, as in the case + for ptest, the execution occurs on the host). The built version of + Automake that ships with the Yocto Project includes a patch that + separates building and execution. Consequently, packages that use the + unaltered, patched version of ``make check`` automatically + cross-compiles. + + Regardless, you still must add a ``do_compile_ptest`` function to + build the test suite. Add a function similar to the following to your + recipe:: + + do_compile_ptest() { + oe_runmake buildtest-TESTS + } + +- *Ensure special configurations are set:* If the package requires + special configurations prior to compiling the test code, you must + insert a ``do_configure_ptest`` function into the recipe. + +- *Install the test suite:* The :ref:`ref-classes-ptest` class + automatically copies the file ``run-ptest`` to the target and then runs make + ``install-ptest`` to run the tests. If this is not enough, you need + to create a ``do_install_ptest`` function and make sure it gets + called after the "make install-ptest" completes.