The code was changing the timestamps of the files in the do_package output,
particularly the files added for debug sources. This was to do two things:
a) make do_package sstate more reproducible
b) ensure better hash equivalence matching
Unfortuately the debug source files are hardlinks into the source tree for
efficiency so touching these, touches a lot of files in ${B} and ${S}. This
causes unpredictable effects if compile is run again for example, or could
cause compiling in the install task.
The hash equivalence matching is of key importance but we can mimic that
using clamping of the file timestamps in the depsig output used to generate
the hashes.
This patch drops the global timestamp clamping, instead allowing the files
to retain their creation timestamps into sstate. This makes do_package sstate
slightly less reproducibile. We could clamp the sstate timestamps but that
would lead to two different sets of timestamps depending on whether the
data came from sstate or not. I'd prefer to have consistent code behaviour,
rather than differing behavhour depending on whether data came from sstate
or not.
If we wanted to have reproducibiliy and fix the "corruption" of S/B and have
consistent codepaths, the only other option would be two copies of the
sources, which could end up huge and seems the least desireable option.
This patch therefore drops the timestamp clamping in the sstate files
and tweaks the depsig data generation to clamp the timestamps for do_package
instead since this seems the best compromise.
I validated that rpm/deb/ipk files still generate correctly as before.
(From OE-Core rev: 0c93bb692b39af51f0ca109dfd1f949abe7eea9c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 475759fdab7200488b2a568b2ba1aa31a456d113)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
We planned to drop SSTATEPOSTINSTFUNC some time ago with the introduction of
postfuncs. Finally get around to doing that which should make the buildhistory
code a little more readable.
Unfortunately ordering the buildhistory function calls after the sstate ones is
difficult without coding that into the sstate class. This patch does that to
ensure everything functions as expected until we can find a better way. This is
still likely preferable than the generic sstate postfuncs support since the function
flow is much more readable.
(From OE-Core rev: 78ca086441b21dedd9c471a3d3200c24fd9ec8d2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c9e2a8fa2f0305ef1247ec405555612326f798f8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
do_deploy_archives tasks don't need their dependencies so we can optimistion
this as we do for some other tasks.
(From OE-Core rev: 1e32ad0849f06aee92fe844e8eaee73c5935bfb0)
Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 3dd9f6e398844380d3765c54d35afe0d2ccf82e7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The autobuilder sstate was corrupted via incorrect equiavlences caused by
the security fix to git and the poor interaction that had with SCM version
checks under fakeroot/pseudo. Bump the versions to enable a clean slate
to work off.
(From OE-Core rev: 69f2d0822462e77d09b4781dcec41a0747e4d387)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3b6672730372e130d4d72b683fc3150911964745)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_create_spdx tasks don't need their dependencies so we can optimistion
this as we do for some other tasks.
(From OE-Core rev: 6723a045c3a46537bb76111f8306b5960e532522)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When inside the threadedpool we make a copy of the localdata
to avoid some race condition, so we need to use this new
localdata2 and stop write the shared localdata.
(From OE-Core rev: 1fa763b2022822a76fde541724e83e1977833d03)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is redundant (target arch is already in PN), and breaks
compiling a cross-canadian toolchain, as that needs populating the
sysroot with two different native-hosted toolchains built from
cross recipes. Inserting TARGET_ARCH allows only one or the other.
(From OE-Core rev: 33fc1792cd782feb8dbb4285e3006bb588f7978f)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The threading code here can race as the fetcher changes the environment which is
shared between the threads. By setting it up in advance, it isn't changed and
therefore no longer races.
(From OE-Core rev: cabc3cc2eac5916e63340c18d1074411b377ced4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should print the actual stack trace, for real :)
(From OE-Core rev: 9fbc7d6a2b2388e70a76cb97285e2f60bdc79624)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Autobuilder has been showing things like:
Checking sstate mirror object availability...ERROR: SState: cannot test file://universal/d9/bc/sstate:xz-native:x86_64-linux:5.2.5:r0:x86_64:8:d9bced04b194d5fc8d778eb8a0d674fa7375a42c8c50a9237e6d7672e9e7a00c_deploy_source_date_epoch.tar.zst: TimeoutError('timed out')
ERROR: SState: cannot test file://37/a0/sstate:libgcc-initial:core2-64-poky-linux:11.2.0:r0:core2-64:8:37a0a5aec105a0822df098f15ff2b67d0e7220204742b5d2b1f7958dda6fa5ce_deploy_source_date_epoch.tar.zst: TimeoutError('timed out')
ERROR: SState: cannot test file://universal/11/a4/sstate:libpciaccess-native:x86_64-linux:0.16:r0:x86_64:8:11a4d6c3a2e147ef7dd5f31c0ff2a91271dad49b561d8aa24849115081cf1842_deploy_source_date_epoch.tar.zst: TimeoutError('timed out')
done.
which is not helpful. To find out what really happened and where, the original
traceback is needed too.
(From OE-Core rev: 80a9052221fb2a12e7c652f2d1764101202fdb90)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the hash equivalence fix, we need to bump the sstate and hash equivalence
version numbers to ensure older task hashes aren't matched into the new namespace.
(From OE-Core rev: 3f229267e0588c747265e849ee19724033cc6a80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Target build output, stored in do_populate_sysroot or do_package can depend
not only upon direct dependencies but also indirect ones. A good example is
linux-libc-headers. The toolchain depends on this but most target recipes do
not. There are some headers which are not used by the toolchain build and do
not change the toolchain task output, hence the task hashes can change without
changing the sysroot output of that recipe yet they can influence others.
A specific example is rtc.h which can change rtcwake.c in util-linux but is not
used in the glibc or gcc build. To account for this, we need to account for the
populate_sysroot hashes in the task output hashes.
(From OE-Core rev: 1cf62882bbac543960e4815d117ffce0e53bda07)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This continues where commit676757f "sstate: fix touching files inside
pseudo" and commit 29fc8599 "sstate: another fix for touching files
inside pseudo" left off.
The previous changes switched from trying to check if the sstate file is
writable before touching it, to always touching the sstate file and
ignoring any errors. However, if the sstate file is actually a symbolic
link that links to nothing, this would actually result in an empty
sstate file being created. And this in turn leads to that future
setscene tasks will fail when they try to unpack the empty file.
Change the code so that if an sstate file linking to nothing already
exists, it is overwritten with the new sstate file. Also change it so
that the temporary file that is used is always removed, even if ln
fails to link the sstate file to it.
(From OE-Core rev: 552197a0c4c9f75a9177c00b197ea91296ed9fc4)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bb.fatal() case where sstate failed to find/use an archive in setcene tasks
is suboptimal. Bitbakes handling of setscene tasks will be to warn but the fatal
will turn this into an error, despite the real task being rerun.
In these failure cases other messages would usually have been printed so turn
this into a warning and raise a handled exception status so that bitbake knows
to fail the task but not print more messages.
(From OE-Core rev: 7875baeca7a3ed216b2442fb8771e51efbfa5a4f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Errors like
SState: cannot test file://universal/5f/10/sstate:m4-native:x86_64-linux:1.4.19:r0:x86_64:7:5f108b175274798b17950b5ee686bf61445e90fb204d7f795e5b879b5603e88b_deploy_source_date_epoch.tar.zst: 'GIT_SSL_CAINFO'
aren't useful, try and improve on this.
(From OE-Core rev: 1701d40536db098adb26508799bae13733f1a6fd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is done by default when tar is run by the superuser, but for native
recipes the corresponding task is not run as root under pseudo, so pass
the flag explicitly.
Suggested-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: 32f01e57e1e915e8121609fb48b0b0254e625957)
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the newly added network task flag against tasks where network
access is expected. This is do_fetch, do_checkuri, do_testimage, do_testsdk
and do_testsdkext.
We can't disable networking in sstate tasks due to sstate downloads and
also so we can report hash equivalence to the server so network access
is enabled in sstate tasks.
Access within build-appliance do_image is also allowed due to the use
of pip, this is a poor example made rather obvious now and needs to be reworked.
Network access anywhere else in any other task isn't allowed.
(From OE-Core rev: 7ce1e88a3ad85bbb925bb9f7167dc0a5fd1c27f4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Give this thread pool a name for clarity.
(From OE-Core rev: 57ea50fcfe81f47b93b9302d1aab2e81dcdd3105)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existence of missed files on the local sstate cache is the
condition to search for it on the sstate mirror.
The missed collection are populated when we search the files
on the local sstate cache.
- When we have sstate mirrors we don't need to iterate all files
again because we already know what's missing.
- When we found a file on the sstate mirror is because of they
are missing on the local sstate mirror.
So avoid checking if the hash exists on the missed collection.
- When we can't find the file on the sstate mirror we don't need
to add the hash again to the missed collection as it already exists.
(From OE-Core rev: 46e1f9a900e2be1abeda0dc540fc14040ecd0d7c)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only have the progress bar when there are more than 100 objects.
So we can check for this and store the result to show the progress bar.
(From OE-Core rev: 18e7a77aa1626e11820804ad64b86a32f97da2ca)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is a fixup for 676757f "sstate: fix touching files inside pseudo"
running the 'id' command inside the sstate_unpack_package
function shows that this funcion run inside the pseudo:
uid=0(root) gid=0(root) groups=0(root)
The check for [ -w ${SSTATE_PKG} ] and [ -O ${SSTATE_PKG}.siginfo ]
will always return true and the touch can fail when the real user
don't have permission or in readonly filesystem.
As the documentation refers:
- the file test operator "-w" check if the file has write permission
(for the user running the test).
- the file test operator "-O" check if you are owner of file
We can avoid this test running the touch and mask any return errors
that we have.
(From OE-Core rev: 29fc85997ade490ae46ffca37ef8e1a56957c876)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, when shortening sstate filenames, the reserved
characters for .siginfo were not considered, when siginfo=False,
resulting in differently shortend filenames for the sstate and siginfo
files. With this change, the filenames of the truncated sstate and
siginfo files have the same basename, just as is already the case for
untruncated filenames.
Making sure that the .siginfo files always have the filename of the
corresponding sstate file plus its .siginfo suffix, also when being
truncated, makes it easier to manage the sstate cache and an sstate
mirror outside of Bitbake/Yocto.
(From OE-Core rev: c2e0e43b7123cf5149833e0072c8edaea3629112)
Signed-off-by: Manuel Leonhardt <mleonhardt@arri.de>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
running the 'id' command inside the sstate_create_package
function shows that this funcion run inside the pseudo:
uid=0(root) gid=0(root) groups=0(root)
The check for touch files [ ! -w ${SSTATE_PKG} ]
will always return true and the touch can fail
when the real user don't have permission or
in readonly filesystem.
As the documentation refers, the file test operator "-w"
check if the file has write permission (for the user running the test).
We can avoid this test running the touch and mask any return errors
that we have.
(From OE-Core rev: f6e7445c94443544e92fda97a017ce93393c5f84)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need extra python collections to count the found files
on the sstate cache and sstate mirrors.
The main found collections provides all the files that were found.
Then we only need to store the nunber of files that found on the
local sstate cache and the files that found on the sstate cache
mirror is derived from that.
(From OE-Core rev: 0fe71f31593e87b9188b388860d2ba94bf8266bc)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If checkstatus returns an exception we should silently handle
FetchError, as this means the fetch failed for 'normal' reasons such as
file not found. However, other exceptions may be raised, and these
should be made visible.
(From OE-Core rev: 50d99faf88a1d82cbd939b9bd6e33ebed2b1ffd8)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow a user to validate sstate objects against a list of keys, instead
of just any known key in the user's keychain.
(From OE-Core rev: 52ba0c5e6e2e3d5d01dc3f01404f0ab1bb29b3b5)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch switches the compressor from Gzip to ZStandard for ssate cache
files.
Zstandard compression provides a significant improvement in
decompression speed as well as improvement in compression speed and disk
usage over the 'tgz' format in use. Furthermore, its configurable
compression level offers a trade-off between time spent compressing
sstate cache files and disk space used by those files. The reduced disk
usage also contributes to saving network traffic for those sharing their
sstate cache with others.
Zstandard should therefore be a good choice when:
* disk space is at a premium
* network speed / resources are limited
* the CI server can sstate packages can be created at high compression
* less CPU on the build server should be used for sstate decompression
(From OE-Core rev: 0710e98f40ec7394567ede1f88f7c7bae306b925)
Signed-off-by: Henry Kleynhans <hkleynhans@fb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We hardlink some files into the build, such as licence files in
do_populate_lic tasks. Depending on the umask that the source tree
was checked out with, the group permissions would vary. This
results in inconsistent task outhashes.
Avoid this by ignoring the group/other bits unless we're under
pseudo context.
Bump the ABI numbers to ensure we don't see cache corruption from
earlier builds.
(From OE-Core rev: 2bd9c806de7e5197168360e3bd1d223a04a92291)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At this point the hash equivalence and sstate is 'junk' on the autobuilder
unforuntately due to the volume of fixes and also the volume of slighly
not quite right patches tested during the development of the fixes.
In order to try and help any remaining sanity I might have, bump the
version numbers to start with a clean slate so we're working from a known
good baseline rather than risk chasing phantom issues. For those
upgrading, there wouldn't be much reuse anyway after the changes.
(From OE-Core rev: be32692c627a14509de5eb3834e7321c3c5faf25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some files in the populate_sysroot tasks have hardcoded paths in them,
particularly if they are postinst-useradd- files or crossscripts.
Add some filtering logic to remove these paths.
This means that the hashequiv "outhash" matches correcting in more
cases allowing for better build artefact reuse.
To make this work a new variable is added SSTATE_HASHEQUIV_FILEMAP
which maps file globbing to replacement patterns (paths or regex)
on a per sstate task basis. It is hoped this shouldn't be needed
in many cases. We are in the process to developing QA tests which
will better detect issues in this area to allow optimal sstate
reuse.
(From OE-Core rev: d9852ffbbe728dac33dc081538a08af98f52fd4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are several tasks which should only be triggered by direct
dependencies. Some are listed in the dep loop, some are not. They
should all be in both cases.
Fix this by making a list and using the list in both cases to fix
various inconsistencies.
(From OE-Core rev: 29d285bc07bd7732c7ea39c0fd03c0b3c0d93e52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sstate task is only needed when depended upon, it can be skipped
if there are no tasks running that directly depend upon it.
This reduced the number of sstate tasks in something like an image
build.
(From OE-Core rev: 601cee016da5c7505915e26641a085714de175ce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an image is built from sstate, the cross toolchain was being pulled in.
This was due to the sstate dependencies for deploy tasks not being considered
in the "depvalid" logic.
do_deploy tasks do not need their populate_sysroot dependencies when installed
from sstate so skip these. This reduces the sstate pulled in by an image
build from sstate significantly.
(From OE-Core rev: 8047c184f8266fc047c0e5a5524586d414528626)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating packages we build them with --clamp-mtime and use
SOURCE_DATE_EPOCH as the maximum mtime. This makes the end packages
reproducible. The data stored in sstate for do_package and the package
task doesn't benefit from this though and have varying timestamps.
This means their outhash varies and means hash equivalance isn't
effective at all and doesn't work as intended/desired.
We could create the sstate archives with the same clamping however
that would lead to different results depending on whether a task was
installed from sstate or not. Making that differ is a path to madness.
It also wouldn't fix the outhash of the task to be determninistic
without clamping of the date in the hash calculation code.
Instead, iterate over the files in sstate output and clamp them at
the code level. This isn't ideal but does make the file timestamps
determnistic everywhere and means we don't have to change the hash
calculation code.
This issue can be clearly seen looking at the do_package outhash for
a recipe which you then re-run the package task for after adding
something like whitespace to the install task. The outhash shouldn't
change but currently does.
(From OE-Core rev: c3b3cc4745811b48b9193f83889946b2e1788932)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fetching sstate, SRCPV can be expanded withuin PV which breaks the
sstate fetcher as other variables like SRC_URI are changed by the fetcher
code. Expand the variable before doing this to avoid those problems and
hence avoid setscene task failures with remote sstate storage in git
recipes.
(From OE-Core rev: 0d9d7acebe0be42d4cf8a30ecbbbbcb2dfd16a4b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- bitbake BB_NUMBER_THREADS uses cpu_count from oe utils that uses
the python os.sched_getaffinity and it is more acurrate.
grep -nH ^BB_NUMBER_THREADS meta/conf/bitbake.conf
meta/conf/bitbake.conf:806:BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
- multiprocessing.cpu_count() returns the number of CPUs on the host,
not the number of usable CPUs on the host. If the user is using
scheduler affinity then the number of usable CPUs may be less,
so when determining how many cores we can use check the affinity instead.
(From OE-Core rev: 45c52f08a289c6eb2329de50634a0406204d1d8e)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit dd555537fc
'sstate.bbclass: fix errors about read-only sstate mirrors'
adds an additional exception handler to silently mask read
only rootfs errors thrown during the touch.
The exception handler checks the error type with the python module errno
but this module needs to be imported as it don't exist.
Example of the error:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_task_postfunc(d)
0003:
File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 778, function: sstate_task_postfunc
0774:
0775: omask = os.umask(0o002)
0776: if omask != 0o002:
0777: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask)
*** 0778: sstate_package(shared_state, d)
0779: os.umask(omask)
0780:
0781: sstateinst = d.getVar("SSTATE_INSTDIR")
0782: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir'])
File: '/home/builder/src/base/poky/meta/classes/sstate.bbclass', lineno: 708, function: sstate_package
0704: except PermissionError:
0705: pass
0706: except OSError as e:
0707: # Handle read-only file systems gracefully
*** 0708: if e.errno != errno.EROFS:
0709: raise e
0710:
0711: return
0712:
Exception: NameError: name 'errno' is not defined
(From OE-Core rev: 15f30ad144fbe25e9a5e71bc7e42e746d2039992)
Signed-off-by: Jose Quaresma <quaresma.jose@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When adding a layer which changed SSTATE_EXCLUDEDEPS_SYSROOT, the state
hashes were changing when they should not. This was caused by wider use
of setscene_depvalid which means the dependency on the variable was seen
when it was previously not.
Exclude the variable since this should be be included in the hashes.
(From OE-Core rev: 09725a29365c69ccbd603fe3a1de72189f26d5ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that pseudo-native always includes statx support and we have sanity
checks on pseudo-native to ensure it always contains a minimum feature set,
we no longer need to mark pseudo-native as distro specific. This fixes
eSDK build problems.
[YOCTO #14428]
(From OE-Core rev: 3279e43fcb469edb63c7c4eb60fdc565d5751f9d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sstate code tries to be careful about racing around directory creation.
In particular, the copyhardlinktree code creates the directory tree first
allowing for "already exists" errors and ignoring them, then hardlinks the
files in.
Unfortunately the sstate removal code can race against this since it
will try and remove empty directories. If there is some bad timing,
a newly created directory can be removed before it was populated, leading
to build failures.
We could try and add locking but this would damage performance, we've been
there before. It is also unclear where to actually place locks just based on
the contents of a manifest file which may cover multiple sstate install
locations for a given task.
Instead, lets disable directory removal in the problematic "shared" core
path. This could result in a few more empty directories being left on disk
but those should be harmless and better than locking hurting performance
or rare build races.
[YOCTO #13999]
[YOCTO #14379]
(From OE-Core rev: 4f94d9296394bc7ce241439f00df86eb5912875f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a read-only sstate mirror is used in conjunction with hash equiv,
then OSError will be raised when an sstate-cache hit is achieved.
This is because sstate_task_postfunc will try to "touch" the symlinks
that point to the read-only sstate mirror when sstate_report_unihash
has changed SSTATE_PKG.
This commit adds an additional exception handler to silently mask read
only rootfs errors thrown during the touch.
The fix is also duplicated to sstate_eventhandler as the code is very
similar but it may not be needed there.
Example of the error:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:sstate_task_postfunc(d)
0003:
File: '/poky/meta/classes/sstate.bbclass', lineno: 774, function: sstate_task_postfunc
0770:
0771: omask = os.umask(0o002)
0772: if omask != 0o002:
0773: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask)
*** 0774: sstate_package(shared_state, d)
0775: os.umask(omask)
0776:
0777: sstateinst = d.getVar("SSTATE_INSTDIR")
0778: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir'])
File: '/poky/meta/classes/sstate.bbclass', lineno: 703, function: sstate_package
0699: if not os.path.exists(siginfo):
0700: bb.siggen.dump_this_task(siginfo, d)
0701: else:
0702: try:
*** 0703: os.utime(siginfo, None)
0704: except PermissionError:
0705: pass
0706:
0707: return
Exception: OSError: [Errno 30] Read-only file system
(From OE-Core rev: 244b3be0358a66e0cca4016fe26144e3d7323390)
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
What key is used to sign sstate artefacts should not affect the hash of
the object, otherwise everyone would need to use the same signing key.
(From OE-Core rev: 57cc9429dba4f9bd23127633dbc1f57dc2d5dd16)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain build patterns, warnings about missing manifests can appear. These
are real issues where the manifest was removed and shouldn't have been.
Martin Jansa was able to find a reproducer of:
MACHINE=qemux86 bitbake zlib-native
echo 'PR = "r1"' >> meta/recipes-core/zlib/zlib_1.2.11.bb
MACHINE=qemux86-64 bitbake zlib-native
MACHINE=qemux86 bitbake zlib-native
<the zlib-native manifest is now removed along with the sysroot-components contents>
The code maintains a per machine list of stamps but a per PACAGE_ARCH list of
stamp/manifest/workdir mappings. The latter is only appended to for speed with
the assumption that once stamps are gone, the code wouldn't trigger.
The code only ever appends to the mapping list (for speed/efficency under lock)
meaning that multiple entries can result where the stamp/workdir differs due to
version changes but the manifest remains the same.
By switching MACHINE part way through the build, the older stamp is referenced
and the manifest is incorrectly removed as it matches an now obsolete entry in
the mapping file.
There are two possible fixes, one is to rewrite the mapping file every time
which means adding regexs, iterating and generally complicating that code. The
second option is to only use the last mapping entry in the file for a given
manifest and ignore any earlier ones. This patch implments the latter.
Also drop the stale entries if we are rewriting it.
(From OE-Core rev: 63da9a4f889c5b0e41bc8ec08abe0acea1546479)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Incremental build in Docker fails with:
OSError: [Errno 18] Invalid cross-device link
when source and destination are on different overlay filesystems.
Rather than adding fallback code to every call site, use a new wrapper
in bitbake which detects this case and falls back to shutil.move
which is slower but will handtle the overlay docker filesystems correctly.
[YOCTO #14301]
(From OE-Core rev: 656a65b2b84e7d529b89cf5de7eb838f902d84a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It isn't immediately obvious what these do, add some better examples
to document them. Also improve the naming for one of the handlers.
This will help make it clear in any tracebacks which code is being
used.
(From OE-Core rev: 2b5c553744804f44a17b61c345e482027e45b71e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The split of util-linux-uuid out from util-linux caused some interesting
sstate file overlap errors on existing build directories. This is a
challenge to handle since util-linux depends on util-linux-uuid and has
overlapping files in package data and deploy/packages directories.
The util-linux build happens later and is what would clean up those files
but it happens too late for uuid.
Fixing this is hard as we don't know the taskhashes until the task
graph is calculated. Once that is ready, we can compare the hashes
with the existing hashes and know which sstate tasks are "stale".
This patch adds a handler which iterates the sstate manifests looking
for matching stamp paths and then removes the manifests along with the
associated stamp files.
(From OE-Core rev: 60e77b1777c6c304aa1d629c4cfdabe0daa22eb1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
