Commit Graph

3255 Commits

Author SHA1 Message Date
Florian Bezdeka
fce639f1eb wic: Warn if an ext filesystem affected by the Y2038 problem is used
We are getting closer and closer to the year 2038 where the 32 bit
time_t overflow will happen. While products (= embedded systems) with an
expected life time of 15 years are still save the situation may change
if your system has to survive the next 20 years.

ext2 and ext3 filesystems are always affected by the time overflow, so
let's warn the user if these filesystems are still being used.

If ext4 is affected depends on the inode size chosen during filesystem
creation. At least 256 bytes are necessary to be safe. As ext4 is
used very often (and partitions may be created small first and extended
later) this might be an issue for many users.

Some filesystems created during CI runs were already affected by the Y2038
problem. By using `--mkfs-extraopts "-T default"` we tell mke2fs not to
auto-detect the usage type based on the filesystem size. mke2fs will use
the default values for tuning parameters instead. The inode size is one
of these parameters.

(From OE-Core rev: d30283d8090ff21be51433572b7afb542ecea592)

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eecbe625558406680121d2a7e84917fea45ea9dc)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-10 00:24:26 +00:00
Dorinda
c08762baeb oe-pkgdata-util: Check if environment script is initialized
Tinfoil doesn't behave well if environment is not initialized, this check ensures a proper error log if environment is not initialized.

[YOCTO #12096]

(From OE-Core rev: 6cc45e917e371d572cbacf05f1a46af9e833d0cb)

Signed-off-by: Dorinda Bassey <dorindabassey@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e88073e16f1b4cfd0f97c81a988640a84adad674)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 15:30:39 +00:00
Lee Chee Yang
e63e8599d8 wic: debug mode to keep tmp directory
files in wic tmp directory can be usefull for debugging, so do not remove
tmp directory when wic create run with debugging mode (-D or --debug).

also update wic.Wic.test_debug_short and wic.Wic.test_debug_long to
check for tmp directory.

[YOCTO#14216]

(From OE-Core rev: 04d334d5bc5353c39c5059d25d97f111a9e368d9)

Signed-off-by: Lee Chee Yang <Chee.Yang.Lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a122e2418b67d38f691edcf8dd846c167d6b4fa9)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-26 15:30:39 +00:00
Kamel Bouhara
6a35ab437f recipetool: create: only add npmsw url if required
Before adding a npmsw fetcher to a recipe we
should first check if the generated shrinkwrap file
contains dependencies.

(From OE-Core rev: 6bcd2e242ef85cf875a2ea1a7c208f8c30fbdffa)

Signed-off-by: Kamel Bouhara <kamel.bouhara@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ef153ad36d0299e83a03af8f207686d0d8a238b3)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-11 17:46:11 +00:00
Tomasz Dziendzielski
8e23c6ec37 devtool: Fix file:// fetcher symlink directory structure
Add relative path if file is under directory.

[YOCTO #13738]

(From OE-Core rev: 19ddacc1b38f9ebb86a9359963ccc3c707f7125e)

(From OE-Core rev: a6a4e1350c6c8170648a9e910b06e6534d11f926)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3a220f1e41)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-05 23:35:18 +00:00
Tomasz Dziendzielski
1a0b7ca08e devtool: Fix source extraction for gcc shared source
If do_patch task is disabled then prepare do_configure dependencies to
fetch external sources and create symlink to ${S} in devtool workspace.

[YOCTO #13036]

(From OE-Core rev: 9a1bcf47f0242215610ddf917e8bd87e86fdf419)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9e47319bfe62d289b90f7545a64dbdc1cbde7f1d)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Paul Barker
58e578f7b6 wic: Optimise fstab modification for ext2/3/4 and msdos partitions
The fix for [Yocto #13994] required the rootfs directory to be copied
(using hardlinks if possible) when modifying the fstab file under wic.

We can optimise this copy away for filesystems where we have the tools
to modify the contents of the partition image after it is created. For
ext2/3/4 filesystems we have the debugfs tool and for msdos/vfat
filesystems we have the mcopy tool. So for any of these filesystems we
skip the modification of the fstab file in the rootfs directory (and
skip the associated copy unless it is otherwise necessary) and update
the contents of fstab directly in the partition image.

(From OE-Core rev: a56b9dbfeb410d235b5af2c5a846b3f1360d2d1d)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5fb8ae0e9159597d7eaa9307a3a8543800bf9405)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Paul Barker
d2c89a6f15 wic: Copy rootfs dir if fstab needs updating
By default, wic updates the /etc/fstab in the rootfs to include details
of additional partitions described in the selected wks file. If this
modification is performed in place, other tasks which create an image
file from the rootfs directory (e.g. do_image_tar and do_image_ext4)
will pick up the modified fstab file which would not be appropriate for
those images as they do not include the additional partitions described
in the wks file. wic does undo modifications to the fstab file once it
has finished creating the filesystem image, however this leaves open a
race condition if one of the other tasks reads the contents of the fstab
file from the rootfs directory between the point where wic modifies the
fstab file and the point where wic restores the files original content.

This could be solved by adding a lockfile for tasks which use the rootfs
directory to ensure that no other such task is reading the rootfs
directory while do_image_wic is running. This would serialize several
do_image_* tasks and result in slower builds, especially for large
images. Another drawback of this solution is that it is hard to
selectively optimise - adding lockfiles to do_image_* tasks would result
in these tasks always being serialized even if no fstab modification
will take place.

An alternative solution is to copy the rootfs directory when fstab needs
to be modified. The code to do this in wic already exists as it is
needed when including or excluding content in the rootfs. This still
results in an impact on build times but the copy uses hardlinks if
possible (so little data is actually copied) and we can make selective
optimisations to improve things. The rootfs copy will only take place if
fstab modification is required (or if it was already needed to include
or exclude rootfs content). We can also follow up with further
optimisations after this commit. So this second solution is chosen.

Fixes [Yocto #13994]

(From OE-Core rev: 39f98ef8c1f6b0259deb43ea902f5ac8e5b069cd)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ce682a73b7447652f898ce1d1d0416a456df5416)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Paul Barker
7e9fc45c23 wic: Update pseudo db when excluding content from rootfs
To exclude content from the rootfs, wic makes a copy (using hardlinks if
possible) of the rootfs directory and associated pseudo db, then removes
files & directories as needed. However if these files and directories
are removed using the python functions os.remove and shutil.rmtree, the
copied pseudo db will not be updated correctly. For files copied from
the original rootfs, if hardlinks were used successfully when copying
the rootfs this should mean that the relevant inodes can't be reused and
so the risk of pseudo aborts should be avoided. However, this logic
doesn't apply for directories (as they can't be hardlinked) or for files
added via the '--include-path' argument (as they weren't present in the
original rootfs) and so there remains some risk of inodes being reused
and the pseudo db becoming corrupted.

To fix this, use the 'rm' command under pseudo when removing files &
directories from the copied rootfs to ensure that the copied pseudo db
is updated.

(From OE-Core rev: 9f33aa6f1dd66b1b6990b256ad822fa4cf8daa94)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d5db7e268947f0392c2126137571a44acd29ccd6)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Paul Barker
d8b7d59e66 wic: Allow exec_native_cmd to run HOSTTOOLS
This allows programs from HOSTTOOLS (e.g. 'install', 'rm', 'mv', etc) to
be more easily executed by wic. Without this change only programs from
an actual *-native recipe built by bitbake can be executed by wic.

(From OE-Core rev: 00474d990174c3c4af6e0a6058871f925380750a)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8eb186acdecfbb3151c9a0ab148358e3fe5cce39)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Paul Barker
c9009b4554 wic: Ensure internal workdir is not reused
If a path is specified for the internal wic working directory using
the -w/--workdir argument then it must not already exist. Re-using a
previous workdir could easily result in rootfs and intermediate files
from a previous build being added to the current image.

(From OE-Core rev: 365b10a95ac0b04ca48a53e55cf5a0a330f1bc2f)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2e40c8d4109024ff704c5ce40d98050ca7f34dd5)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Paul Barker
b3bb553816 wic: Add workdir argument
This allows the path for the temporary workdir used by wic to be set
when running wic from bitbake or directly from the command line.

(From OE-Core rev: 6136e1312fc69b1b6196aa2fb8986d99ad46e7a1)

Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2e7314ac3a3cab89de93c932e8efc75de0feb0a7)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Adrian Herrera
f643bfd10e scripts: oe-run-native, fix *-native directories
This fixes a crash with "find" when running a native tool and *-native
directories do not exist under the binary directory in the sysroot.
This happened because the directory wildcard was passed as part of the
root directory.
The directory wildcard is now passed by "-name", which returns an empty
result if no matching directory.

(From OE-Core rev: ebc71592b57eba514588693d76057d90f92d6f62)

Signed-off-by: Adrian Herrera <adrian.herrera@arm.com>
Change-Id: Iba7acd8bbd7e0beb4d25c984f6af7a4fd21486e6
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f6c90ed0ad24b7d4f892e22e088b1578824eb1d3)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-27 09:32:45 +00:00
Peter Kjellerstedt
44bc055c8c wic: Pass canonicalized paths in PSEUDO_IGNORE_PATHS
(From OE-Core rev: 9aec2fdda1235f069cbb365634a3f0fcfc846682)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 55618cbd58d6784a82e773f323723be6f722232f)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-12 14:25:14 +00:00
Lee Chee Yang
67925a6731 wic/direct/kparser: ensure fsuuid for vfat and msdos align with format
vfat/msdos filesystem should have fsuuid in format 0xYYYYYYYY where "0x"
in front follow with 8 hexadecimal number in uppercase. In wic, when using
custom fsuuid for vfat/msdos partition in wks, it is able to set the value
in any length, with or without leading "0x". This can cause fsuuid
missaligned when fstab updates, fstab expect exactly 10 character
fsuuid for vfat/msdos partition and all in uppercase.

if custom fsuuid for vfat/msdos is set, check the length and format,
error if it exceed the format size. Amend it so it is align with format
0xYYYYYYYY. This is done before image create and fstab update to ensure the
fsuuid are same in all followup process. if custom fsuuid length less than
expected, fill in "0".

[YOCTO #14161]

(From OE-Core rev: e71365fe128fe5e7d01232a1da62a3989e0d8eb2)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d9686ae511ef10a504becfd81bfe296b788e1456)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-12 14:25:13 +00:00
Alexander Kanavin
ad798f5ecf devtool: gitsm:// should be handled same as git:// in upgrades
(From OE-Core rev: 086584d5e60bc7daec914ab4384fdda776f27103)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e7076f1742bb83e76aa4e90aad861546536166cf)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-12 14:25:13 +00:00
Mingli Yu
e2312cd887 update_udev_hwdb: clean hwdb.bin
Steps to reproduce:
echo "IMAGE_INSTALL_append = \" udev-hwdb lib32-udev-hwdb\"" >> conf/local.conf

When install both udev-hwdb and lib32-udev-hwdb as above,
there comes below do_populate_sdk error:
 $ bitbake core-image-sato  -c populate_sdk
 ERROR: Task (/path/core-image-sato.bb:do_populate_sdk) failed with exit code '134'
 NOTE: Tasks Summary: Attempted 5554 tasks of which 0 didn't need to be rerun and 1 failed.

 $ cat /path/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r5/pseudo/pseudo.log
 [snip]
 inode mismatch: '/path/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r5/sdk/image/usr/local/oecore-x86_64/sysroots/core2-64-poky-linux/lib/udev/hwdb.bin' ino 427383040 in db, 427383042 in request.
 [snip]

It is because both udev-hwdb and lib32-udev-hwdb will generate
${SDK_OUTPUT}/${SDKTARGETSYSROOT}/lib/udev/hwdb.bin during do_populate_sdk
and it triggers pseudo error.

So clean hwdb.bin before generate hwdb.bin to avoid conflict to
fix the above do_populate_sdk error.

(From OE-Core rev: c7472925feb53ce92c1799feba2b7a9104e3f38f)

(From OE-Core rev: 93e59a78da3dab56c91f423b2c0f29a8ebaf2700)

Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 994ca65e6f)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-12 13:06:28 +00:00
Ross Burton
92e46629b0 scripts/install-buildtools: Update to 3.2 M3 buildtools
(From OE-Core rev: 89a94ad07b641434089b0684bec0acd8fa8f97c1)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-20 11:11:45 +01:00
Roland Hieber
a6c9ee99b3 devtool: make sure .git/info exists before writing to .git/info/excludes
If nothing else is specified, 'git init' uses its default repository
template from the install location (e.g. /usr/share/git-core/templates),
which already includes an info/ subdirectory. However, when setting
init.templateDir to a different template path in ~/.gitconfig, this
isn't necessarily the case, and it can lead to setup_git_repo() failing
with stack traces like:

    File: '.../scripts/lib/devtool/__init__.py', lineno: 234, function: setup_git_repo
         0230:        pass
         0231:    if 'singletask.lock\n' not in excludes:
         0232:        excludes.append('singletask.lock\n')
         0233:    bb.warn("try writing excludefile")
     *** 0234:    with open(excludefile, 'w') as f:
         0235:        for line in excludes:
         0236:            f.write(line)
         0237:
         0238:    bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
    Exception: FileNotFoundError: [Errno 2] No such file or directory: '.../devtooltmp-6m36b181/workdir/foobar-1.0.1/.git/info/exclude'

Fix this edge case by creating the .git/info/ directory first.

Fixes: 334ba846c795fc0d8c73 (2018-02-01, "devtool: set up git repos so that singletask.lock is ignored")
(From OE-Core rev: 148a23e4d5ceaf655ccacb52deca4ba501f12975)

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-17 12:34:29 +01:00
Richard Purdie
57a1ccaa19 oe-build-perf-report: Ensure correct data is shown for multiple branch options
If --branch and --branch2 are the same, there is extra confusing data plotted
onto the graph. Only do that if the branches are actually different.

(From OE-Core rev: c263d810b7eb47ee90f2adeb5ab6decf7332bec2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-10 13:37:56 +01:00
Richard Purdie
5088e3b64b scripts/oe-build-perf-report: Allow operation with no buildstats
If buildstats is missing, avoid a backtrace.

(From OE-Core rev: 23c4b21f947c7aae1303ca6526fc5aaaa9fc7bb8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-10 13:37:56 +01:00
Ross Burton
197cec77f5 devtool: remove unused variable
This variable isn't used at all.

(From OE-Core rev: 9676ee24d190b144cb50be6ebde3990644a6df2a)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-08 08:08:32 +01:00
Richard Purdie
c194e5fac6 wic: Handle new PSEUDO_IGNORE_PATHS variable
Adjust wic to correctly handle the new PSEUDO_IGNORE_PATH variable and avoid
inode corruption issues.

(From OE-Core rev: 13500f5234361385c365c7c35e83f99435500481)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-07 11:18:38 +01:00
Richard Purdie
a2db4fa127 base/bitbake.conf: Enable pseudo path filtering
This is a pretty big change to the way pseudo operates when used in OpenEmbedded.
Normally, pseudo monitors and logs (adds to its database) any file created or
modified whilst in a fakeroot environment. There are large numbers of files
we simply don't care about the permissions of whilst in that fakeroot context,
for example ${S}, ${B}, ${T}, ${SSTATE_DIR}, the central sstate control
directories,

This change uses new functionality in pseudo to ignore these directory trees,
resulting in a cleaner database with less chance of "stray" mismatches if files
are modified outside pseudo context. It also should reduce some overhead from
pseudo as the interprocess round trip to the server is avoided.

There is a possible complication where some existing recipe may break, for
example, we found a recipe which was writing to "${B}/install" for
"make install" in do_install and since we listed ${B} as not to be tracked,
there were errors trying to chown root for files in this location.

This patch fixes a few corner cases in OE-Core when used with this new
ignore list:

* The archiver directory matched a "${WORKDIR}/deploy*" pattern so was renamed
  to something else since that directory does need its root permissions
* The ${S} and ${B} ignoring is conditional on them being different to ${WORKDIR}
* package_write_* task output (the debs/rpms/ipks) are now owned by the build
  user so we don't want the file ownership information in the hashequiv outhash
  calculation even if they are built under pseudo.
* The fontcache postinstall intercept is run under qemu outside of pseudo context
  so delete files it may delete up front where pseudo can see this.
* SSTATE_DIR is in PSEUDO_PATHS_IGNORE, which is in FAKEROOTENV which is cached
  by bitbake. We therefore need to trigger reparsing if this changes, which means
  SSTATE_DIR can be in BB_HASHBASE_WHITELIST but not BB_HASHCONFIG_WHITELIST.
  Rework the variables to handle this. This otherwise breaks some of our sstate
  tests in oe-selftest.
* Ignore the temp directory wic uses for rebuilding rootfs.

(From OE-Core rev: ad8f5532ffaead9a5ad13e1034fe9e5e1b7979f4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-07 11:18:38 +01:00
Khem Raj
b6b985a223 qemuboot: Add QB_RNG variable
RNG passthru has been enabled on all qemu machines but its being added
to each one of them, with this patch its turned into QB variables which
defaults to host passthru, yet it can be overridden if needed via
machine or config metadata if needed.

(From OE-Core rev: 26dd24506ef36088e17f999ce5489dc4b72194e8)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-24 22:32:49 +01:00
Richard Purdie
da4fb13f6d scripts/oe-build-perf-report: Use python3 from the environment
This means ot matches the rest of the system rather than causing confusing errors
where multiple python versions are mixed.

(From OE-Core rev: ef4138b6d8575fb4a0e9dfbaec2cb766f3705efb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-20 12:47:48 +01:00
Ross Burton
4064756e20 buildhistory-diff: use BUILDDIR to know where buildhistory is
Instead of assuming that the buildhistory is stored in ./buildhistory
unless told otherwise, use the environment variable BUILDDIR if present
(which is set by oe-init-build-env) to locate the buildhistory
directory.

(From OE-Core rev: 18ca377df98e53b84ae8ec9ba561392c776c10df)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-17 09:47:46 +01:00
Khairul Rohaizzat Jamaluddin
85574ce0cf wic/bootimg-efi: IMAGE_EFI_BOOT_FILES variable added to separate bootimg-efi and bootimg-partition
Due to recent changes in bootimg-efi to include IMAGE_BOOT_FILES,
when both bootimg-partition and bootimg-efi occur in a single .wks
and IMAGE_BOOT_FILES are defined, files listed in IMAGE_BOOT_FILES
will be duplicated in both partition.
Since IMAGE_BOOT_FILES are crucial for bootimg-partition, but
optional for bootimg-efi, hence allowing bootimg-efi to have the option
to ignore it.

The new variable, IMAGE_EFI_BOOT_FILES, was added to help handle this
issue. Its basic usage is the same as IMAGE_BOOT_FILES.
Usage example:
        ${IMGDEPLOYDIR}/${IMAGE_BASENAME}-${MACHINE}.ext4;rootfs.img \

This commit is also squashed with the updated testcase to cover for
this change.

[YOCTO #14011]

(From OE-Core rev: 945339e06b273df1935cfd784f548ef57e0b7f4c)

Signed-off-by: Khairul Rohaizzat Jamaluddin <khairul.rohaizzat.jamaluddin@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-15 11:53:28 +01:00
Adrian Freihofer
8d7b5e56ad oe-publish-sdk: add --keep-orig option
Support publishing the sdk-update as well as the sdk installer in one
go.

(From OE-Core rev: 97f1544132ba3bb0c85cdafff6a870c7ba0ef2e4)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-08 16:43:40 +01:00
Adrian Freihofer
da44b89f66 oe-publish-sdk: fix layers init via ssh
Escaping does not work in my use case. It must be escaped for
python, ssh and shell as well as for different versions of echo.
Let's try it a little less elegant, but hopefully more reliable.

(From OE-Core rev: 5cc1ae332eb6b05d83802c8d64ab2767c7079412)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-08 16:43:40 +01:00
Richard Purdie
f19f252e2b runqemu: Add a hook to allow it to renice
We have an issue where qemu is being starved of resources on our autobuilders.
We can't raise its priority without special capacilties, therefore add a hook
which if present can allow this to happen using an executable
"~/runqemu-renice".

(From OE-Core rev: 141a3c9ce93bc3d526303021ecf0460c6e9fea8a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-06 09:58:50 +01:00
Michael Tretter
fdc6ef9673 devtool: deploy-target: Fix size calculation for hard links
If a package contains hard links to a file, the file size is added for
each hard link instead of once for the file. Therefore, the calculated
size may be much larger than the actual package size.

For example, the mesa-megadriver package contains several hard links to
the same library.

Keep track of the inode numbers when listing the files that are
installed and use the actual size only for the first occurrence of an
inode. All further hard links to the same inode are added to the file
list, but accounted with size 0.

All file names need to be added to the file list, because the list is
used for preserving the files/hard links on the target.

(From OE-Core rev: 17e92572278980d1a7f06de9d72c68baf57698f1)

Signed-off-by: Michael Tretter <m.tretter@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-02 16:00:46 +01:00
Vijai Kumar K
435ec90a67 wic: misc: Add /bin to the list of searchpaths
/bin is also a valid path where one can find executables. Add
that to the search path.

(From OE-Core rev: ca0a6025351cb2135e87cecf828633cf12aa34c6)

Signed-off-by: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-02 16:00:46 +01:00
Martin Jansa
b67303460c devtool: expand SRC_URI when guessing recipe update mode
* I have recipes which use variable inside SRC_URI, e.g.:
  ROS_BRANCH ?= "branch=release/melodic/swri_nodelet"
  SRC_URI = "git://github.com/swri-robotics-gbp/marti_common-release;${ROS_BRANCH};protocol=https"

  and devtool modify works fine, but devtool finish fails with:

  $ devtool finish --force-patch-refresh swri-nodelet meta-ros/meta-ros1-melodic/
  ...
  Traceback (most recent call last):
    File "/OE/openembedded-core/scripts/devtool", line 334, in <module>
      ret = main()
    File "/OE/openembedded-core/scripts/devtool", line 321, in main
      ret = args.func(args, config, basepath, workspace)
    File "/OE/openembedded-core/scripts/lib/devtool/standard.py", line 2082, in finish
      updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
    File "/OE/openembedded-core/scripts/lib/devtool/standard.py", line 1737, in _update_recipe
      mode = _guess_recipe_update_mode(srctree, rd)
    File "/OE/openembedded-core/scripts/lib/devtool/standard.py", line 1721, in _guess_recipe_update_mode
      params = bb.fetch.decodeurl(uri)[5]
    File "/OE/bitbake/lib/bb/fetch2/__init__.py", line 390, in decodeurl
      raise MalformedUrl(url, "The URL: '%s' is invalid: parameter %s does not specify a value (missing '=')" % (url, s))
  bb.fetch2.MalformedUrl: The URL: 'git://github.com/swri-robotics-gbp/marti_common-release;${ROS_BRANCH};protocol=https' is invalid: parameter ${ROS_BRANCH} does not specify a value (missing '=')

  let it expand the SRC_URI before trying to decode it.

(From OE-Core rev: 3535cfdbf3d77f550b804276f957acf859da484f)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-27 08:28:03 +01:00
Richard Purdie
ee98c6ab03 runqemu: Show an error for conflicting graphics options
The autobuilder managed to pass the nographic option with "sdl gl" due
to a problem elsewhere. It would have been useful for runqemu to have
errored rather than passing conflicting options to qemu. Add an
error for this invalid usecase.

(From OE-Core rev: 377a6ffbcba0d3c5ede470c989756c4d1636873f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26 14:33:57 +01:00
Joshua Watt
5fa3c3d2e3 wic: Add 512 Byte alignment to --offset
Allows the --offset argument to use the "s" or "S" suffix to specify
that it is reporting the number of 512 byte sectors.

This is required for some SoCs where the mask ROM looks for an item at a
sector that isn't aligned to a 1KB boundary.

(From OE-Core rev: 938595d1dc4abaf5f7f3a7900add3f0492b805d0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-22 15:46:55 +01:00
Vasyl Vavrychuk
21c13f75b5 runqemu: Check gtk or sdl option is passed together with gl or gl-es options.
runqemu help reports that gtk or sdl option is needed with gl or gl-es
option. But if user forgot to add gtk or sdl option, then gl or gl-es
options were silently skipped.

(From OE-Core rev: 4d1e93d4bf013bb0c48032bfda43f77c5aba9ecf)

Signed-off-by: Vasyl Vavrychuk <vvavrychuk@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-19 10:57:44 +01:00
Chen Qi
98549fd68c runqemu: fix permission check of /dev/vhost-net
The code should check /dev/vhost-net instead of /dev/kvm. And
the permission check logically belongs to the part when kvm-vhost
is enabled. Also fix the URL to point to the correct wiki page.

(From OE-Core rev: 53bdf2d20077921bdf83641c365d60d85174f5b5)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-11 20:35:39 +01:00
Daniel Ammann
f39694933a wic: fix typo
(From OE-Core rev: 9ab4c4571d899d2eefcd2fc53af0851863f29008)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-06 15:12:39 +01:00
Lee Chee Yang
4cbea10197 checklayer: check layer in BBLAYERS before test
layer under test should absent from BBLAYERS when running
yocto-check-layer. This allow to get signatures before layer
under test. There are existing steps to add the layer under
test to BBLAYERS after getting initial signatures.

add steps to check for layer under test in BBLAYERS before
running any test, skip test for the layer if the layer under
test exist in BBLAYERS.

[YOCTO #13176]

(From OE-Core rev: be02e8dbfb0d1decce125322f9f1e11a649756c0)

Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-27 10:37:58 +01:00
Khasim Mohammed
801f07fc00 wic/bootimg-efi: Add support for IMAGE_BOOT_FILES
List of files defined using IMAGE_BOOT_FILES are installed into
the boot partition when preparing an image using the wic tool with
the bootimg-efi source plugin.

The code snippet introduced is taken as is from bootimg-partition.py

Change-Id: I8dbb6b4e7c24870f587a6f31e6ae4a87d7033782
Issue-Id: PLATFORMS-3134
(From OE-Core rev: a44ab3a4ee5b3c57812909c6194456f299d6ba7f)

Signed-off-by: Khasim Syed Mohammed <khasim.mohammed@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-25 15:11:05 +01:00
Kevin Hao
b617cd5b47 wic/filemap: Fall back to standard copy when no way to get the block map
For some filesystems, such as aufs which may be used by docker container,
don't support either the SEEK_DATA/HOLE or FIEMAP to get the block
map. So add a FileNobmap class to fall back to standard copy when there
is no way to get the block map.

[Yocto #12988]

(From OE-Core rev: 7934ed49179242f15b413c0275040a3bb6b68876)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-18 11:06:31 +01:00
Kevin Hao
2212d938c5 wic/filemap: Drop the unused get_unmapped_ranges()
This method is not used by any code, so drop it.

(From OE-Core rev: 6513fd9302b9989f97fc9d95e76e06ad5d266774)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-18 11:06:31 +01:00
Kevin Hao
9818a04678 wic/filemap: Drop the unused block_is_unmapped()
This method is not used by any code, so drop it.

(From OE-Core rev: fb62a15349597ee026c67a0bb0a6ca2cc9bfe420)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-18 11:06:31 +01:00
Tim Orling
e25b982000 scripts/lib/recipetool/create.py: fix regex strings
Python now expects regex strings to be prepended with r.
Silence pylint/autopep8 and similar warnings by identifying
these regex patterns as... regex patterns.

(From OE-Core rev: 0eae98a369d80340e48dc690d09a1364cde97973)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-12 22:39:41 +01:00
Bjarne Michelsen
5c2381f9eb devtool: default to empty string, if LIC_FILES_CHKSUM is not available
[Bug 13971] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13971

A recipe using LICENSE=CLOSED would normally not have LIC_FILES_CHKSUM
set.
If LIC_FILES_CHKSUM was not set, it would cause `devtool upgrade` to fail.

This fixes the issue by defaulting to an empty string if
LIC_FILES_CHKSUM is not found.

[YOCTO #13971]

(From OE-Core rev: 95e112f0f465868e98285509ea0d1a624a439ac2)

Signed-off-by: Bjarne Michelsen <bjarne_michelsen@hotmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-08 10:37:12 +01:00
Nicolas Dechesne
c0a16eaae0 checklayer: parse LAYERDEPENDS with bb.utils.explode_dep_versions2()
LAYERDEPENDS is a string of this format:
"DEPEND1 (optional version) DEPEND2 (optional version) ..."

However when we parse LAYERDEPENDS in _get_layer_collections() we
parse it as a simple string, and if any optional versions are there the
'depends' field is wrong. For example, running yocto-check-layer
might result in such errors:

ERROR: Layer meta-python depends on (>= and isn't found.
ERROR: Layer meta-python depends on 12) and isn't found.

Let's use bb.utils.explode_dep_versions2() to parse LAYERDEPENDS, and
create a string that contains all dependencies, effectively
skipping/ignoring any optional versions.

[YOCTO #13957]

(From OE-Core rev: f81f07afc200fe06c5c06ea81a4f8a3a43436faf)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-28 08:36:00 +01:00
Jason Wessel
157e44bdba runqemu: If using a vmtype image do not add the -no-reboot flag
There is no way to use runqemu and turn off the no-reboot flag with a
command line argument.  If someone really wants it back it can be
added with the qemuparams="" argument.

Also if you use the "halt -p" from user space, the qemu will exit when
it is complete.  It is impossible to test self deploying image with
runqemu if you cannot reboot the device.  One might argue that you can
run runqemu in a loop, but that defeats the purpose of having a
wrapper around a tool that already does what is needed in the first
place.

For the vmtype images, the -no-reboot flag should not be added.

(From OE-Core rev: 6d07a72356f9a8ea76b6cda9159d3ca4743028d9)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-28 08:36:00 +01:00
Richard Purdie
132952c59c scripts/install-buildtools: Handle new format checksum files
Autobuilder generated checksum files only have a single space between the sum and the
filename, tweak it to account for this.

(From OE-Core rev: d1e71bf5b399372166eb40bb0d99c8fb52231600)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-25 10:23:37 +01:00
Richard Purdie
1dc6c9f5dc scripts/install-buildtools: Update to 3.2 M1 buildtools
This fixes issues with openssl certs not working properly which meant error
reporting to an error report server was failing.

Also, all our downloads are now standarised on "sha256sum" so adjust for that.

(From OE-Core rev: bc8b44e19a05f499f5cef049eedbed1fede2e765)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-25 10:23:37 +01:00