When running bitbake command with Python 3.6.5 always result in
import error causing by the change of distutils module.
This patch replaces the method to search executable in PATH by
"/usr/bin/env <command>".
(Bitbake rev: 0193e1f5b2128602ad8a80c82a3dd4d2cdb6e4a7)
Signed-off-by: Tzu Hsiang Lin <t9360341@ntut.org.tw>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Changhyeok Bae <changhyeok.bae@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We saw builds where runtime providers were sometimes changing order and the
build result was therefore non-deterministic. For example it could show:
DEBUG: providers for lib32-initd-functions are: ['lib32-lsbinitscripts', 'lib32-initscripts']
or
DEBUG: providers for lib32-initd-functions are: ['lib32-initscripts', 'lib32-lsbinitscripts']
which could cause a test to pass or fail.
This change ensures we don't rely on the random order of dictonaries in
memory and act deterministically.
(Bitbake rev: da85da9b58ed252c7b80a437fb5babff9e6dde48)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Someone hacked the http://hambedded site or it was moved and some
links to that site in the BB manual had been hijacked to point to
an entry portal for a pornography site. Replaced the link with an
archived version that restores the integrity of the links.
(Bitbake rev: 894814cca59f14b0f808680ea8fdf63063f01e7d)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a superfluous section in toaster start/stop script that was
setting the TOASTER_DIR and had a set of comments around that setting.
This was done in two places and only the last one was effective. This
patch removes the spurious section to make it clearer what was
happening and what TOASTER_DIR was actually getting set to.
(Bitbake rev: aeed732ff6e8c1009c5efad3aadbbd3b0bbaeea7)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The toaster.sqlite database was located in TOASTER_DIR/build. This
meant that if you named your build directory something else (like cow),
Toaster would fail to make/find the database. TOASTER_DIR is on the
whitelist unlike BUILDDIR and we need to be able to write there anyway
given our current layout so this should not disrupt anything.
[YOCTO #9992]
(Bitbake rev: 32b13d52ae37be343962cc1cf504f3d569fb1153)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the tinfoil2 changes resulted in TOASTER_DIR being cleared by
the memory resident bitbake server toaster starts up. This prevented
toaster from being able to connect to its sqlite database. Adding
TOASTER_DIR to the BB_ENV_EXTRAWHITE list stops the cooker from clearing
out the TOASTER_DIR variable.
[YOCTO #9252]
(Bitbake rev: 30da2715cd6a114faf99b09c74e73c5d84767b91)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the fetcher retrieves file:// URLs, there is no lock file being
used. This means that in case two separate tasks (typically from two
concurrent invocations of bitbake) want to download the same file://
URL at the same time, there is a very small chance that they also end
up wanting to create a symbolic link to the file at the same time.
This would previously lead to one of the tasks failing as the other
task would have created the link.
(Bitbake rev: 5f5e13bacde95a93633f621ec6b94a022c476a58)
Signed-off-by: Peter Kjellerstedt <pkj@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data we read from an HTTPConnection comes in the form of bytes, but
we need it as a string, so in Python 3 we need to decode it (missed in
the Python 3 migration).
(Bitbake rev: e75c68819086788bac1c4ca8a35aebeaf604e624)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since 2017-08-17 (git version 2.14.1.473.g3ec7d702a) using deprecated
git branch parameter "--set-upstream" causes a fetcher error. Replace
it by "--set-upstream-to".
https://git.kernel.org/pub/scm/git/git.git/commit/?id=52668846ea2d41ffbd87cda7cb8e492dea9f2c4d
says, it's deprecated since 2012-08-30 so hopefully all still supported
host distributions have new enough git to support "--set-upstream-to".
ERROR: PACKAGE do_unpack: Fetcher failure: ...;
git -c core.fsyncobjectfiles=0 branch --set-upstream master origin/master failed with exit code 128, output:
fatal: the '--set-upstream' option is no longer supported. Please use '--track' or '--set-upstream-to' instead.
ERROR: PACKAGE do_unpack: Function failed: base_do_unpack
(Bitbake rev: 991b533f1d61042a7b3edd1fbf3dea0bf9991606)
Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The newly added file in the last commit was corrupted, fix it.
(Bitbake rev: 7c9370f7fafc98e4c836255e72ff8acff1138997)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove "prettify.js" and "prettify.css" due to license issues with Apache2.
Replace with "highlight.pack.js" with its BSD3 License.
[YOCTO #12206]
(Bitbake rev: 5732046a48779be5f42616ec08b32ae53b33b230)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Brian Avery <brian.avery@intel.com>
Signed-off-by: David Reyna <david.reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
URL decoding was improved in the core a while ago and this looks like
a leftover from those times which caused urls needing a user/password to
fail. Use the parameters from the core instead of the broken split
implementation.
[YOCTO #11262]
(Bitbake rev: 32a249cf33f79b579a2cf6224c7d23cdb20d376f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For shell constructs like
echo hello & wait $!
the process_tokens() method ended up with a situation where "token"
in the "name, value = token" assignment was a list of tuples
and not the expected tuple, causing the assignment to fail.
There were already two for loops (one in _parse_shell(), one in
process_tokens()) which iterated over token lists. Apparently the
actual nesting can also be deeper.
Now there is just one such loop in process_token_list() which calls
itself recursively when it detects that a list entry is another list.
As a side effect (improvement?!) of the loop removal in
_parse_shell(), the local function definitions in process_tokens() get
executed less often.
Fixes: [YOCTO #10668]
(Bitbake rev: 887ea6d25cee5114365dfbf1130603599e13ee80)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code in these two functions is meant to be equivlanet in behaviour
but isn't. Add in code to ensure files that don't exist are handled
consistently by both functions. Users did report being able to generate
tracebacks otherwise.
(Bitbake rev: df69c0257aaa9f4e2f4a20d130b7bc0ce2ca0848)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As of html5lib 0.9999999/1.0b8 (released on July 14, 2016), some modules
have moved from _base to base. Handle this, while staying compatible
with earlier versions.
(Bitbake rev: bdf7a362ecdafd47f309b7a21feac4b94624f287)
Signed-off-by: Daniel Lublin <daniel@lublin.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also run sanity check for all additional multiconfig configurations
as listed in BBMULTICONFIG, not just the "default" configuration.
[YOCTO#10810]
(Bitbake rev: 0f7fdaa8212a8b9f1f089d32e652400d6c4b4822)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently you cannot build a target for all the configured multiconfigs without
specifying a list. The list can be quite long, requiring to type several lines
of text.
This enhancement is to support globbing so that you can do this,
e.g. instead of:
$ bitbake multiconfig:A:bash multiconfig:B:bash bash
you can do:
$ bitbake multiconfig:*:bash
There are real world use cases where it is desirable to use multiconfig with
two different tasks. For example: SDKs with multiple toolchains but also
containing set of additional host tools, or multiconfig builds requiring one image for
the main CPU(s) and a different co-image for a companion CPU.
For this reason, two variations of the new syntax are supported.
For example, the following:
$ bitbake multiconfig:*:meta-toolhchain
would expand to:
$ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain meta-toolchain
However the following:
$ bitbake multiconfig:*:meta-toolhchain hosttools
would expand to:
$ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain hosttools
In other words, if the user specified the "default" task explicitly, it replaces the implicit
"default" task.
[YOCTO#10680]
(Bitbake rev: 3e80d47bea51b64ed6c8bffc033f2d11a630481e)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If was pointed out that if we have:
XXX = " A"
XXX_remove_inactive-override = "YY"
then XXX can become "A" and the leading space can be removed. This is because
the remove override code changes the variable value even when there is no
removals active. In the process it dirties the cache.
We don't really need to do this so tweak the code accordingly.
(Bitbake rev: f1ee6dfd3d193a9055320bdd555c1dbaa63f9475)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During normal bitbake execution, the environment is cleaned of variables
not on a whitelist while starting up cooker, and then restored
afterwards. Prior to the tinfoil2 rework in master we were taking a
number of shortcuts within tinfoil and one of those was not doing this
environment cleaning. However, prior to OE-Core rev
3d39ca5c91dbb62fb43199f916bd390cd6212e3d we didn't have any code (as far
as I'm aware) that was affected by this shortcut, hence why this wasn't
an issue up to now.
The result is the following error when attempting to run "devtool build"
in the eSDK, as CCACHE_PATH is allowed through from the eSDK's
environment setup script:
----------- snip -----------
ccache: error: Could not find compiler "gcc" in PATH
...
subprocess.CalledProcessError: Command 'gcc --version' returned
non-zero exit status 1
----------- snip -----------
We can fix this by simply doing the environment filtering while we are
starting up cooker, thus the environment when uninative.bbclass comes to
do the gcc version check it is not affected by CCACHE_PATH or other
variables in the external environment that should be filtered out.
For clarity, this patch is only applicable to the bitbake 1.32 branch
as used for the OE-Core morty branch - master uses the reworked
tinfoil2 and doesn't need this fix.
Fixes [YOCTO #10961].
(Bitbake rev: a240f5ff71092cb209c44a071cd6fa07756ccfa0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I noiced builds where tasks seemed to be taking a surprisingly long time.
When I looked at the output of top/pstree, these tasks were no longer
running despite being listed in knotty. Some were in D/Z state waiting for
their exit code to be collected, others were simply not present at all.
strace showed communication problems between the worker and cooker, each
was trying to write to the other and nearly deadlocking. Eventually, timeouts
would allow them to echange 64kb of data but this was only happening every
few seconds.
Whilst this particularly affected builds on machines with large numbers
of cores (and hence highly parallal task execution) and in cases where
I had a lot of debug enabled, this situation is clearly bad in general.
This patch introduces a thread to the worker which is used to write data
back to cooker. This means that the deadlock can't occur and data flows
much more freely and effectively.
(Bitbake rev: f48befe1163147b02a9926ee38af0f7258a477e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
People are struggling with multiconfig as the point the conf file
is injected into the data store is not what people expect. We can't
really use a post config since that is too late and we can't really
use a pre config file since that is too early. In OE terms, we need
something right around the local.conf point so it behaves in a similar
way.
A way to handle this is to set the new variable BB_CURRENT_MC to be the
currently selected multiconfig, then the metadata itself can choose
when to inject the approriate configuration.
(Bitbake rev: 1469828fa747da0aaaa3e964954ff17f2b3180fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fakeroot workers usually have dependencies that need to be ready before they
can be started. Starting them as a block therefore doesn't work as the
dependencies may or may not have been built. Therefore start the multiconfig
fakeworkers individually upon demand.
[YOCTO #10344]
(Bitbake rev: ac5ea74152b011256209c8b5664216f290b123e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In multiconfig, mutliple tasks can execute which share the same stamp file. These
must not execute in parallel, the idea is the first should execute, the subsequent
ones should see a valid stamp and get skipped.
The normal task execution code has stamps code to handle this, this adds similar
code to the setscene execute() function to handle the issue there too.
(Bitbake rev: df8408a6b54fc908d4de81529b34477b8924d181)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setscene was being entirelu skipped for multiconfig variants as the tasks
were simply not being spotted. If the default config was also being built
it masked the problem. When this was fixed by using taskfn instead of fn
in lookups against dataCache, several other instances of this problem were
highlighted.
This goes through and corrects the setscene code to correclty use taskfn
instead of fn in the appropriate places meaning setscene tasks for multiconfig
now work correctly.
(Bitbake rev: a5d81eefe9106f2080001b7313e2b15ab21ea55b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tasknames can now start with "multiconfig:" which broke the virtual: comparison code and
lead to unpredictable checksums with nativesdk recipes. This adds in handling for
the new additional prefix which unbreaks nativesdk builds when using multiconfig.
(Bitbake rev: 0ca6b8438624d892ee7ef3b42df0024604b64567)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the UI shows X is building, possibly multiple times but doesn't
say which of the multilibs that might be. This adds a prefix to the task
name so the mulitconfig being built can be identified.
(Bitbake rev: dfb775c67a96a79f3b85104870c0ade46ef2a9ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The location of the fakeroot command and the various environmental values need
to be taken from the right multiconfig datastore, not the shared one. This
patch ensures the right one is used for cases like a split TMPDIR.
[YOCTO #10344]
(Bitbake rev: c241f16670cada2cdf45ecddb4961e16edb83486)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If many files change and the inotify queue overflows, rather than print
a traceback, invalidate the caches and warn the user.
[YOCTO #10676]
(Bitbake rev: 058f8517c041b80e8b591ad7d34a68281b2d03fc)
(Bitbake rev: 4fafb6c6d261de78dd1bc3824a1389d191b70321)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The processing of the "do_" prefix to tasks is currently inconsistent
and has resulted in "bitbake world -g" being broken as task prefixes
don't get handled correctly.
Make the "do_" task prefix handling consistent through various codepaths.
[YOCTO #10651]
(Bitbake rev: 3d7186353e804c9410096c408bc337a98c8b33fe)
(Bitbake rev: 100439e715841ecfd6460d59cd51c831184b328d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As of Django 1.8.16, Django is rejecting any HTTP_HOST header that is
not on the ALLOWED_HOST list. We often need to reference the
toaster server via a fqdn, if we start it via webport=0.0.0.0:8000 for
instance, and are hitting the server from a laptop. This change does
reduce the protection from a DNS rebinding attack, however, if you are
running the toaster server outside a protected network, you should be
using the production instance.
[YOCTO #10578]
(Bitbake rev: 59a3f391ac5ac194f30d11a39676356464269d55)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Print full traceback instead of just the exception message in the
child() function inside fork_off_task(). This makes debugging a lot
easier as the function catches a generic "Exception" and the exception
message alone might not give much information.
[YOCTO #10393]
(Bitbake rev: 61cc397a5b7136afb37052a2860c6c39a176ddab)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bug where a totally wrong value of a variable would be exported if
an exception happened during d.getVar(). Also, print a warning if an
exception happends instead of silently ignoring it. It would probably be
best just to raise the exception, instead, but use the warning for now
in order to avoid breaking existing builds.
[YOCTO #10393]
(Bitbake rev: 59c606cfc6e0a4f367344d4e3def6017fb560d75)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We recalculate the taskhash to ensure the version we have matches
what we think it should be. When we write out a sigdata file, use
the calculated value so that we don't overwrite any existing file.
This leaves any original taskhash sigdata file intact to allow a
debugging comparison.
(Bitbake rev: dac68af6f4add9c99cb7adcf23b2ae89b96ca075)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake can parse metadata in the cooker and in the worker during builds. If
the metadata isn't deterministic, it can change between these two parses and
this confuses things a lot. It turns out to be hard to debug these issues
currently.
This patch ensures the basehashes from the original parsing are passed into
the workers and that these are checked when reparsing for consistency. The user
is shown an error message if inconsistencies are found.
There is debug code in siggen.py (see the "Slow but can be useful for debugging
mismatched basehashes" commented code), we don't enable this by default due to
performance issues. If you run into this message, enable this code and you will
find "sigbasedata" files in tmp/stamps which should correspond to the hashes
shown in this error message. bitbake-diffsigs on the files should show which
variables are changing.
(Bitbake rev: 46207262ee6cdd2e49c4765481a6a24702ca4843)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't remove sigdata files, we also shouldn't remove sigbasedata files
as this hinders debugging.
(Bitbake rev: 06e7c00f2e1ddda6a2632ec2354a3c8f5c34562d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were presuming that all the layer dependency information was of the
form "^/path/to/layer" to we were just stripping the leading "^" off of
the layer information when we were matching the layer priorities to the
toaster database. This patch splits out the priorities layer match which
gets a regex from the task/recipe match which is gets a path.
(Bitbake rev: 82775c80d169266cc18ca2b2065a05c79dc6fbfc)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set Morty to be the default release in toaster for the Morty release
when creating new projects.
(Bitbake rev: 00f79096f639ce3a9c0b7c72cfb36f14e264733d)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the poky fixture to the Morty release. This removes the
master branch from the release and limits it to the morty branch.
Normally, we would also support at least one past branch but the change
from Python 2.7 -> Python 3 makes that infeasible.
(Bitbake rev: 2674ca33b900f4f3f16be504d7c67d0fc69c1c2d)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the oe-core fixture to the Morty release. This removes the
master branch from the release and limits it to the morty branch.
Normally, we would also support at least one past branch but the change
from Python 2.7 -> Python 3 makes that infeasible.
(Bitbake rev: 93f1e6c3d022b1f12a230879160efa941cb1e250)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The BB_SETSCENE_VERIFY_FUNCTION variable has effectively changed
to BB_SETSCENE_VERIFY_FUNCTION2. I changed the three areas in the
book. Basically a name change.
(Bitbake rev: 41d9cd41d40b04746c82b4a940dca47df02514fc)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #10401]
Applied some edits to various areas of the section. These
were minor in nature.
(Bitbake rev: ec99f168fd0d6feca67f2890917ba4829933c73b)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #10401]
Added some wording to clarify that setting these flags to "1"
causes the desired action. Also, provided a cautionary note
about tasks depending on any [nostamp] task causes the task
to always be executed and could cause unnecessary rebuild time.
(Bitbake rev: b6a4a6147b9e455cc1fec37553fb577f187a2d73)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #10401]
The "addtask" documentation was rewritten to tighten up the
introductory section and to flesh out the actual examples.
(Bitbake rev: c3373399c5d565de033c40a39e6f6f9399bb782e)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #10400]
No information existed for cases when a variable did not exist.
I added this information to each of the variables in the table.
(Bitbake rev: 466b2c30f31754a7b6a2478e359f687ec6888e0d)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The whitelist checks for BB_SETSCENE_ENFORCE were running for every call
to execute(). Since the task list doesn't change for each call into
execute, the checks only need to be ran once.
[YOCTO #10369]
(Bitbake rev: f65e631ab6705dfd9188f19ee423eca33bca7d7d)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this the UI just sits there doing nothing. Showing an
infobar in-UI would be nicer but not much more useful since currently
user couldn't do anything in-UI to fix the situation. Implementation
is based on the one in knotty.
Fixes [YOCTO #9288]
(Bitbake rev: eee9231a543f1d0b9ef3cd8377fc46fd23afb97b)
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
