From git 2.48 release notes:
"""
When "git fetch $remote" notices that refs/remotes/$remote/HEAD is
missing and discovers what branch the other side points with its
HEAD, refs/remotes/$remote/HEAD is updated to point to it.
"""
This means with git 2.48 onwards, there is a mystery "HEAD" revision
appearing in some of our shallow clone tests. We can avoid this by
using the same canonicalization as used for the reference revisions.
This resolves autobuilder failures on the Fedora 40 workers.
(Bitbake rev: c83444d1210740e27b1744d3aa7c5cad4e28db2f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, providers are set on a global config basis. This change allows
for a select set of providers configured in BB_RECIPE_VIRTUAL_PROVIDERS to
be selected on a per recipe basis. This would allow for the selection of
virtual/cross-cc as gcc or clang for example in OE-Core.
DEPENDS and task flag [depends] values are processed.
(Bitbake rev: fb119c7888ae8a749aa824f8c51780176af077f9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
??= operator works for variable value by defining a "_defaultval" flag.
Use something similar for flags: For the default value of the flag
"flag_name", define a flag "_defaultval_flag_flagname" that is used when
reading flag_name but no other value has been set.
Fixes [YOCTO #15685]
(Bitbake rev: 0329a7e3ac694737f2d2c1861f65492551360663)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Reviewed-by: Ghislain Mangé <ghislain.mange@smile.fr>
Suggested-by: Ola Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Run the test with:
$ bitbake-selftest bb.tests.parse.ParseTest.test_parse_defaulttest
.
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
This is a test case for [YOCTO #15685]
(Bitbake rev: d12c80125d810df54379cc4f019a95ab9064ad2c)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Reviewed-by: Ghislain Mangé <ghislain.mange@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the following warning when executing to bitbake linux-yocto:
bitbake/lib/bb/fetch2/__init__.py:464: DeprecationWarning: 'count' is passed as positional argument
This is because the 4th parameter of re.sub(pattern, repl, string, count=0, flags=0)
is a keyword parameter. We use keyword arguments for parameters that are not positional.
(Bitbake rev: 4bd62639f7023eec9f55ad7909fa9250538de936)
Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a basic test for the goh1_file function.
(Bitbake rev: 3dc4d28a9eed962876c148dbe69d9f521bf42287)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Go mod h1 hash. The hash is based on the Go dirhash
package. The package defines hashes over directory trees and is uses for
Go mod files and zip archives.
(Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The websockets library throws a number of exceptions which are currently
not caught leading to unhandled exceptions in the idle loop.
Fix this by catching them and reexposing them as a `ConnectionError`
which is the exception expected by users of `asyncrpc`.
(Bitbake rev: 41d62911a480283287265fe063696d2acd5904aa)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The transaction model was only introduced in git 2.27 whereas Ubuntu focal
(20.04) has 2.25. This causes failures. We don't need the transations here
so simply drop the commit piece, fixing on older git versions.
Credit to Nick Owens <nick.owens@eero.com> for working out how to fix it.
(Bitbake rev: 0723ec9d4cd7c9b2d46904c3a038be123feea374)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
when there are many refs to delete, using xargs to exec git can take a very long
time. make this faster by only running git update-ref with stdin mode.
for a repo with over 34000 git tags this makes git-make-shallow finish
in 2 seconds instead of 3 minutes for me.
(Bitbake rev: 2b815e42ec074a7f8667bbfaccaa69fc4a0ba788)
Signed-off-by: Nick Owens <nick.owens@eero.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher support entries with an embedded checksum like 'sha256sum'
in the SRC_URI. It adds the parameter 'name' as prefix to the checksums
if the parameter is set. This behavior is unexpected and leads to hacks
in fetchers. Fallback to the checksum without the useless prefix and
set the parameter 'name' in the gomod fetcher unconditional.
(Bitbake rev: 7a86c5a20ea2586f1ae240613644e065e7b21683)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace [url] with urls to simplify future modifications.
(Bitbake rev: 212f9406585a2a6ce6e128bb5975ea79f47c5522)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove support for the old lockfile format. The old lockfile format is
required by npm 6 / Node.js 14 which is out of maintenance [2].
[1] https://docs.npmjs.com/cli/v6/configuring-npm/package-lock-json
[2] https://nodejs.org/en/about/previous-releases
(Bitbake rev: 7824e19483d9b60a259d6e3a4c7068fade94f2bf)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update npmsw test cases to new lockfile format. The old lockfile format
is required by npm 6 / Node.js 14 which is out of maintenance [2].
[1] https://docs.npmjs.com/cli/v6/configuring-npm/package-lock-json
[2] https://nodejs.org/en/about/previous-releases
(Bitbake rev: bb59d4fdaf8fb81aa3529e431dcdd4c3d665d742)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add Styhead as a supported release manual, and move Dunfell to the
outdated releases manuals.
(Bitbake rev: d726f4537f16d99512b21ab0a0f476cef832b955)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix Sphinx warnings (treated as errors on local builds):
bitbake-user-manual-metadata.rst:979: WARNING: Bullet list ends without a blank line; unexpected unindent.
bitbake-user-manual-metadata.rst:982: WARNING: Bullet list ends without a blank line; unexpected unindent.
bitbake-user-manual-metadata.rst:985: WARNING: Bullet list ends without a blank line; unexpected unindent.
(Bitbake rev: 51d89ce0fc8d188bf10a8bcc6a112466268899ae)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ud.path has been unescaped (eg %20 is space) but as we're reconstructing
a URL we should re-escape it. For example, unzip has a SRC_URI
containing "UnZip%206.x%20%28latest%29/UnZip%206.0/unzip60.tar.gz" which
then throws exceptions if the unescaped string " (latest)" is used.
Also, this code uses the extracted ud.host and ud.path variables. These
are unescaped but potentially stale as eg the cargo fetcher subclasses
Wget() and reassigns ud.url on construction.
Simplify the code by reconstructing a URL from ud.url directly instead
of bouncing through intermediate variables that may be wrong or
unescaped.
(Bitbake rev: c9906bb289dcfd9ae41f10bd5399ccc17a4cc437)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
urllib2.HTTPRedirectHandler.redirect_request doesn't handle HTTP reponse
code 308 (Permanent Redirect). This was fixed in c379bc5 but can't be
worked around without copying the entire redirect_request() method.
When we can depend on Python 3.13, FixedHTTPRedirectHandler can be
removed.
(Bitbake rev: 365829a2803b954ee6cb0364749551a91d806075)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The npm package lock and shrinkwrap file list bundled dependencies which
are supplied together with the parent dependency. The bundled
dependencies are marked by a flag. The flag and thereby test depends on
the lock file version. The old lock file version uses a `bundled` flag
and stores dependencies in the `dependencies` list. The new lock file
version uses an `inBundle` flag and stores dependencies in the
`packages` list.
(Bitbake rev: 34fd8ea6abe755e04220fe70b082aa620ae15f86)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The npm package lock and shrinkwrap file list bundled dependencies which
are supplied together with the parent dependency. The flag depends on
the lock file version. Add the missing `inBundle` flag to packages list
processing.
(Bitbake rev: 042d7121ddc93c18306bf03d26ea774f66379a90)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The npmsw fetcher doesn't use the npm command. Remove the
`@skipIfNoNpm()` decorator from plain npmsw tests which don't use the
npm fetcher.
(Bitbake rev: 48f9964a19b8e2e3a73667c9889908eceecf7719)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adapt the npmsw tests to commit 84f102954e10 ("fetch/npmsw: The fetcher
shouldn't have any knowledge of S").
(Bitbake rev: 8194178291d9b47efb2079fe81b704ba6171efc1)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases it would be helpful to be able to have an include file
in a standard location which is included in all layers that are added
to the system. The intent is for these to provide configuration tweaks
of specific types so that a given file pattern can be adopted more widely
for such configuration.
The code will search for any named configuration file within BBPATH, so
a configuration directive of:
include_all conf/distro/include/myinc.conf
would include the myinc.conf file in that subpath if present in any
directory in BBPATH. Multiple files will be included if present.
(Bitbake rev: d01d5593e7829ac60f37bc23cb87dc6917026471)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, the cooker skiplist was shared across multiconfigs
(including default ''). If you had a recipe that was incompatible with
several multiconfigs for different reasons, then the displayed reason
(i.e. the "ERROR: Nothing PROVIDES" and "* was skipped" messages) might
vary across invocations of bitbake. This was caused by the random order
in which recipes are parsed under different multiconfig contexts, with
each skip reason overwriting the previously assigned reason.
I hit this specificially when using COMPATIBLE_MACHINE, but
COMPATIBLE_HOST (or anything using bb.parse.SkipRecipe) would have done it too.
(Bitbake rev: c51f01a35ed9a928402eab0899598b5c59602eef)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This alias is intended for managing specific local configs and it
is prompted by adding support for config fragments (in a separate
commit to oe-core); after some deliberation I concluded there should be
a separate tool, as bitbake-layers is already somewhat over-stuffed,
and this will give space for more build/conf/* operations in the future
that anyone can come up with (such as tweaking site-specific items
in site.conf etc.)
The alias completely reuses existing code via symlink and
the difference is in where it looks for plugins.
(Bitbake rev: ba90fe673aa87cb0cda9b2e465ebe2063551f527)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new listtasks() function (introduced in 185c4b) to avoid
accessing a private variable.
Also use assertSequenceEqual() directly as we don't really care about
the actual type returned, only that it's a sequence.
(Bitbake rev: 452289efc6d3608ceca321286fd5bf417a6e1bed)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was suggested that using flags in fragment content to
mark fragment-specific metadata (such as descriptions and
dependencies) is prone to quiet regressions when fragments
are renamed or moved, and it clutters the fragment content.
With this change allowed fragment metadata variables must
be explicitly listed in a variable that is given to addfragment
directive, and parser will add the flag containing the fragment name
to them.
(Bitbake rev: ed9a3ca9426500511feb77f41a146953dbfe9af7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It takes two parameters:
- location prefix for fragments
- name of variable that holds the list of enabled fragments, each of them prefixed
by layer id
Implementation of this directive essentially expands the fragment list
obtained from the variable into absolute fragment paths and hands them to the
implementation of 'require'.
(Bitbake rev: f687746703e7b096c5480668fd4f49bd4951182b)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is beneficial for config fragments, as their names
are specified via flags, and those names can include slashes:
BB_CONF_FRAGMENT_SUMMARY[init/systemd] = "This fragment enables systemd as an init manager"
(Bitbake rev: 80805988492b35593067230f68782a5687c8f557)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
enum was introduced in Python 3.4
(Bitbake rev: 35b71a94f8757fcca830f972a42edab1dd000c16)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the user forgets to set MACHINE, bitbake just exits without
printing anything.
This is because BB_CONSOLELOG ends up with an unexpanded '${MACHINE}', which
bb.utils.mkdirhier tries to report using bb.fatal. But bb.fatal utilizes the
very logging infrastructure that this code was trying to setup.
(Bitbake rev: 7d3f3655b2f610f76898c84b8b97ef2e26529c41)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Analysis shows that "bitbake core-image-ptest-all" spends a lot of
time in scenequeue_updatecounters and much of it is rebuilding a set
which doens't change. Reorder the code to avoid that performance
glitch.
(Bitbake rev: 923c19b9713e398d8e66e6d4422dfd4c18a03486)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than looping through things we looped through on the previous execution,
start looping where we left off for setscene processing. This gives speed
improvements depending on the kind of build being executed.
(Bitbake rev: 00f4d932e3af0eeb333339cbe942010fc76dee0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been complaints about the performance of large multiconfig builds
for a while. The key missing data point was that the builds needed to have large
overlaps in sstate objects. This can be simulated by building the same things with
just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
deferred tasks.
The issue is that the expensive checks in the setscene loop were hit every time
through runqueue's execute function before the check on deferred tasks. This leads
to task execution starvation as that only happens once per iteration.
Move the skip check earlier in the function which speeds things up enormously
and should improve performance of such builds for users.
(Bitbake rev: 9c6c506757f2b3e28c8b20513b45da6b4659c95f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when
uncompressing a single 'file.zst', zstd command line returns the error
'unexpected end of file', and the input file 'file.zst' becomes empty.
The command line is 'zstd --decompress --stdout file.zst > file.zst'.
So the issue is that the input and output filenames are the same.
Fix the output filename when using zstd for a single file.
(Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8)
Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removes the code in bitbake to show custom backtrace formatting for
exceptions. In particular, the bitbake exception code prints function
arguments, which while helpful is a security problem when passwords and
other secrets can be passed as function arguments.
As it turns out, the handling of the custom serialized exception stack
frames was pretty much made obsolete by d7db75020ed ("event/msg: Pass
formatted exceptions"), which changed the events to pass a preformatted
stacktrack list of strings, but the passing of the serialized data was
never removed.
Change all the code to use the python traceback API to format exceptions
instead of the custom code; conveniently traceback.format_exception()
also returns a list of stack trace strings, so it can be used as a drop
in replacement for bb.exception.format_exception()
(Bitbake rev: 2cda75a185aaf8f657f072dac34f8cef9d75f63a)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no easy way to list all of the tasks in a recipe, you can either
look at __BBTASKS (internal variable, shouldn't be used) or iterate all
items in the datastore looking for variables with the 'task' flag set
(which is slow).
Solve this problem by adding a bb.build.listtasks() function that
returns an immutable copy of the __BBTASSK variable.
(Bitbake rev: 185c4b803962b20ba65a7d885dfe1a14e68736ef)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was added in 2.7.2. Since using inherit_defer is safer that inherit
when inheriting conditionally, move the instructions about that in
inherit_defer.
Fixes [YOCTO #15640].
Reported-by: Yoann Congal <yoann.congal@smile.fr>
(Bitbake rev: eb10df5a9619e243e28e0f4cd6122c24ed668f52)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the recipe provided by (-r, --recipe) is not found
tinfoil raises an exception that is not catched for
readability, example:
Traceback (most recent call last):
File "/.../poky/bitbake/bin/bitbake-getvar", line 45, in <module>
d = tinfoil.parse_recipe(args.recipe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../poky/bitbake/lib/bb/tinfoil.py", line 633, in parse_recipe
fn = self.get_recipe_file(pn)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../poky/bitbake/lib/bb/tinfoil.py", line 550, in get_recipe_file
raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
bb.providers.NoProvider: Unable to find any recipe file matching "aaa"
(Bitbake rev: 06aa6c292813a28c84736193b550fb2d18884d43)
Signed-off-by: Talel BELHAJSALEM <bhstalel@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Testing shows the worst case CDN response time can be up to 100s. The wget fetcher
is used for accessing sstate from the CDN so increase our timeouts there to match
our worst case repsonse times.
(Bitbake rev: 7aca591529e115bc277f93811d8c586630acc8c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default pipe size is 64kb on builds, which can be inefficient
for larger log files from workers. Increase the pipe size to 512kb
since build systems have decent amounts of memory and this is a more
efficient way of batching the data.
Tweak the default read sizes to match the pipe size for efficiency.
Since the contstant is only present in python 3.10 onwards, add
some compatibility code.
(Bitbake rev: 69c14e46600ba5ae9703f67704ab2548875ae6d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are large amounts of data being transferred to the cooker
from the worker, recreating the bytearray becomes inefficient as it
happens for every pipesize block of data, defaulting to 64kb.
Instead we can use the deletion API for bytearrays to make this more
efficient and avoid the object recreation.
We noticed this with a strace ptest image taking days to complete the
build after having 6GB of data in the testimage log. Whilst there are
other issues there, making this code more efficient doesn't hurt.
(Bitbake rev: a4a72b7edb368f352784c856a647236a887010dd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DANGLINGAPPENDS_WARNONLY is a very poorly designed interface and
is commonly abused. The challenge is that once it is set, by any
layer, it applies everywhere.
Some layers rely on this to get notification they need to update
bbappend files and having the behaviour change from inclusion of an
antisocial layer is not good.
In addition, showing warnings as an accepted thing on the console
devalues them and trains the user to ignore them. I want to steer us
away from this mindset.
We could extend the functionality and make it apply only to certain
layers, or only to certain appends but then we've basically re-invented
BBMASK.
Given all the above, we should drop support for BB_DANGLINGAPPENDS_WARNONLY
and direct anyone with issues to BBMASK instead.
https://lists.openembedded.org/g/openembedded-architecture/message/2029
[YOCTO #14870]
(Bitbake rev: fca9c9e3cb6f8e9f99bf51dc5e8a8d83f4c84c69)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
