The difference between bb-layers and bb-layers-relative is unclear as both
are relative paths. Rename one to "file-relative" which makes it clear it
is relative to the current file, without becomming a long name.
https://lists.openembedded.org/g/bitbake-devel/message/18296
Based on a patch from Alexander Kanavin <alex@linutronix.de> but
with different naming.
(Bitbake rev: dcb17758b99767ab6da4172cf60eabc9269082dd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
{THISDIR} is a special value token that can be used in the list of enabled
layers to specify the layer location relative to the confguration file:
https://git.openembedded.org/bitbake/commit/?id=b3153be29de8b8570b0c184369bd41f4c646cf92
This replaces the token with an explicit separate keyword for such layers:
so that special processing to determine the final value can be avoided, and
the feature can be formalized in the json schema:
instead of
"bb-layers": [
"{THISDIR}/meta-my-project"
]
this allows
"bb-layers-relative": [
"meta-my-project"
Going forward I think we should strive to avoid any further special value tokens.
(Bitbake rev: 90da82bd2bfcfd5590c9ae06015737b616074b56)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When initializing a build setup from a conf.json that only sets 'rev'
to a tag or branch, the actual revision would not be captured or
logged.
To capture the current layer state after an 'init' or 'update', the
checkout_layers function is extended to store the revision the
bb.fetch.Fetch pulled, and write that information into a
sources-fixed-revisions.json file. This file can then be fed back into
bitbake-setup init as: --sources-overrides
This new 'sources-fixed-revisions.json' is written during 'update_build' and
stored alongside the 'config-upstream.json' in the config dir. And put
with the later under version control by calling 'commit_config" after
'update_build'.
The use of 'deepcopy' is necessary to not modify the original input
data - which python passes around as reference.
(Bitbake rev: 95866ff03f78e987ae7e47daad053bc0f353eea4)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All places that called into 'check_setupdir_files' did the same
preparation step to load the upstream-config.json and then pass it
into the function.
Since the 'setuppath' is already passed into the function, and the
name and relative location of the upstream-config.json is fixed,
constructing the file path and loading the json could be done in the
function.
De-duplicate code by loading the json inside the function instead.
(Bitbake rev: 16d77c83ae3ce92ddab84d714a93fd3bb7def5e2)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Construct the 'src_uri' separately, and then pass either variant into
one call that creates the Fetch. Making use of format-strings to
shorten/simplify the code.
Also: using 'proto' instead of 'type' for a variable name, to avoid
the protected keyword.
(Bitbake rev: 4ad70e05ceca19c1e903dafc33386a82b1176bba)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename the function to reflect which (of the possibly many) files in
the build/config directory it writes to.
(Bitbake rev: 7f9cd4c1e91d110085d74f9b9f12884f97f4e8dd)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting a key=value in a new section would raise a key-error when
using in-memory settings, e.g:
./bin/bitbake-setup --setting default top-dir-prefix /tmp/bitbake-setup --setting default top-dir-name gs settings foo bar baz
Loading settings from
/tmp/bitbake-setup/gs/settings.conf
Traceback (most recent call last):
File "/tmp/bitbake-setup/bitbake/./bin/bitbake-setup", line 853, in <module>
main()
File "/tmp/bitbake-setup/bitbake/./bin/bitbake-setup", line 838, in main
all_settings = merge_settings(builtin_settings, global_settings, topdir_settings, args.setting)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/bitbake-setup/bitbake/./bin/bitbake-setup", line 733, in merge_settings
all_settings[section][setting] = value
~~~~~~~~~~~~^^^^^^^^^
KeyError: 'foo'
(Bitbake rev: 78ab0d15dff5ccf64b0bf681185370779e6cabaf)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename the function to align with the corresponding
'global_settings_path' function, and move it down just above the
later.
This is done to differentiate from the built-in default settings, and
to align with the other places in the code that use 'topdir_settings'
(or 'global_settings')
(Bitbake rev: 3df994f773abbd1d0240e721f5fd29d4b021bfb5)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds support for the new bitbake-setup:
* Maintain backwards compatibility for poky-classic releases
* Assume bitbake-setup for all releases 'master' and forward
* When no ".templateconf" assume Poky and not OE
* Generate and use "init-build-env" script instead of 'oe-init-build-env'
directly, but "cd" to the clone and not the local installation.
* Include "bblayers.conf" because "cookerdata.py" now explicitly
required it, and let it be over-ridden by the usual "toaster-bblayers.conf".
* Force the value "TOPDIR" in "toaster.conf" to the local project's directory.
Normally "cookerdata.py" sets TOPDIR from getcwd(), but somehow in bitbake-setup
mode the CWD at this point has become ".../layers/openembedded-core", so
"bitbake.conf and all other files cannot be found. This non-obvious problem was
hard to discover. Until this is understood, the forced value will make things work.
[YOCTO #16012]
(Bitbake rev: 22b3fd418efc10da7e4a8c1725de285714d2396b)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After a terminology review by Antonin we brainstormed
something less confusing than 'build' and 'build directory'
for the place where bitbake-setup clones layers and
creates a *bitbake* build directory in. People are bound
to get these two confused and mix them up, and 'setup' is
much more distinct and aligns nicely with 'bitbake-setup'.
It's also not claimed by anything else in OE/Yocto.
So before:
top-dir -> build-dir (can be several) -> bitbake build dir, layer dir, config dir
Now:
top-dir -> setup-dir (can be several) -> bitbake build dir, layer dir, config dir
This also updates the respective command line options, I understand it's
a breaking change, but as before the tweaks are simple and we need to get the
terminology right for the users, and now is the time to do it.
(Bitbake rev: eeb81a35bf0304451f7612950d5156ea7ff18bad)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extend the current documentation of PERSISTENT_DIR:
- Mention that it also contains the hash equivalence database when
BB_HASHSERVE = auto.
- Mention that it should not be shared between builds.
[YOCTO #15727]
(Bitbake rev: 682c2dccb67a57ecbf93aac8e9deea88bff10155)
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding documentation for having an empty first component for the
SRCREV_FORMAT.
(Bitbake rev: 37ef26cfd5441ce97ff7ec0d5026c92091224c07)
Signed-off-by: Patrick Vogelaar <patrick.vogelaar.dev@mailbox.org>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current wording makes it difficult to understand what
exactly is done with INHERIT-ed anonymous functions and
when.
(Bitbake rev: f06af69458083735e69f115edf73f783f18bc08c)
Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
Reviewed-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix encountered style issues in the fetching section.
Mention that "subdir" and fetcher specific parameters can stack-up.
(Bitbake rev: 5cb35734c7056aa183bc15d454e125b75267449b)
Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
Reviewed-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Describe the roles of AUTOREV and SRCREV_FORMAT in the fetching step.
Add their missing descriptions to the variable reference section.
Fixes [YOCTO #14498]
(Bitbake rev: fbf0a8c5711f56bd64ae4c9d2c1d5ba884508c9f)
Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
Reviewed-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
--global was defined twice (for settings command as a whole, and specifically
for set/unset sub-commands), which caused 'settings --global set ...' to
be allowed, and at the same time not have any effect. 'settings set --global'
is the only valid form after this change.
(Bitbake rev: 2f14cf68f6c653b6da90180a99a22c58855b0cef)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Tested-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The actual variable name used in oe-core is OE_FRAGMENTS_BUILTIN,
which is in a consistent format with other OE_FRAGMENTS_XXX vars.
(Bitbake rev: 86c5f6aa59980d04264bc526976a49fdf749d1af)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The regex used to detect bitbake files was improved in patch 9.1.1732.
Update the plugin to match that logic and fix a bug where the wrong
expression was used to set the file type
(Bitbake rev: 7ce0c9b44968f80a6060f9f9ff2cc00823b772cc)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously the sub-command 'settings' would take any number of
arguments and then silently do nothing if the number wasn't three.
The help text was also not clear about this, marking the positionals
separately as optional:
usage: bitbake-setup settings [-h] [--global] [--unset UNSET UNSET] [-l] [section] [key] [value]
The '--unset SECTION SETTING' also did not integrate too well, as it
had its own positional arguments for section+setting.
For a bit more consistency and a explorable help, a sub-subparser is
added, that provides the commands:
bitbake-setup settings list
bitbake-setup settings set foo bar baz
bitbake-setup settings unset foo bar
with a '--global' that is added from a stand-alone parent parser, so
that it shows up in all sub-command help texts.
The new help text now reads:
usage: bitbake-setup settings [-h] [--global] {list,set,unset} ...
and the respective sub commands:
usage: bitbake-setup settings list [-h] [--global]
usage: bitbake-setup settings set [-h] [--global] <section> <setting> <value>
usage: bitbake-setup settings unset [-h] [--global] <section> <setting>
(Bitbake rev: 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a metavar to the argparse options to have a self-descriptive help text.
Otherwise argpase defaults to use the argument name in all-uppercase.
Before:
usage: bitbake-setup [-h] [-d] [-q] [--color COLOR] [--no-network] [--global-settings GLOBAL_SETTINGS] [--setting SETTING SETTING SETTING]
{list,init,status,update,install-buildtools,settings} ...
After:
usage: bitbake-setup [-h] [-d] [-q] [--color COLOR] [--no-network] [--global-settings PATH] [--setting SECTION SETTING VALUE]
{list,init,status,update,install-buildtools,settings} ...
(Bitbake rev: 83cecc9356a0684f90249d527fe372298ae92719)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To align the storage destination with the internally used variable
name. This makes room for having another option use 'args.setting'
(Bitbake rev: 14d8535309abc78ee30cfdb51bba2e00b474f443)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In cases where the first call to 'init' failed or was aborted before
creating the 'build/init-build-env' has been created, a user can get
stuck: a second call to init aborts, suggesting 'status' or 'update'
but these to refuse because the --build-dir is not valid.
Guide the user by adding a suggestion to start over from scratch.
(Bitbake rev: 11b2740c3e19e0c6680229c6bbce3691c73746a8)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After some further feedback, additional changes are made:
1. 'setting' command is renamed to 'settings' to better reflect
that it is an interface to various ways of managing settings.
2. This command now has a -l/--list option to list all settings
with their values (same as 'git config -l').
3. A new level of settings (built-in defaults) is added,
and used as a last resort after command line options, top dir
settings file and global settings file.
4. This means bitbake-setup does not have to write and use a
global settings file, and it no longer does so when initializing
a build, avoiding default 'pollution' of ~/.config/bitbake-setup/
which can be problematic or unwelcome.
A global settings file is still created if a setting is explicitly
requested to be placed into it.
5. 'install-global-settins' is removed as the use case for it
(tweak default settings before using them to initialize a build)
can be achieved by setting the settings individually.
5. Similarly, a top dir settings file is no longer created by default
and only appears if a setting needs to be written into it.
6. Default dl-dir is again created inside a top directory and not
in ~/.cache/ to make default builds fully contained in the top
directory (which was also asked about).
(Bitbake rev: 664f8ec48d42d2ddc5f234c4f7d590fa597f489a)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the outcome of various discussions, suggestions and pull
requests on github.
What has specifically changed?
1. The sources for the settings are no longer separated, but are stacked and given priorities,
from highest to lowest:
a. '--setting section key value' on the command line
b. a settings file in the top directory
c. a global settings file in ~/.config/bitbake-setup/ (or in a file pointed to by --global-settings)
Any setting can be in any of these three locations (other than top dir name and prefix which do not
make sense in the settings file in the top directory).
2. A global settings file must contain all of the needed settings, while a settings file
in the top directory can be empty (and this is how they are written out if they do not exist).
Specifically, both dl-dir and registry settings have been relocated to the global file,
and dl-dir defaults to ~/.cache/bitbake-setup/downloads, rather than somewhere in top dir.
3. The file name for both global and top dir settings is now 'settings.conf'.
4. --top-dir-prefix and --top-dir-name options have been removed and superseded by
a generic, universal --setting option.
5. 'install-settings' command has been removed, as it is no longer does anything useful,
and is superseded by the 'setting' command (see below).
'install-global-settings' has been retained, to be able to have a set of global defaults
that can be changed without initializing a build.
6. 'change-setting', 'change-global-setting' and 'install-settings' have all been replaced
by a single 'setting' command that mimics 'git config' in its parameters:
a. Changing a setting: bitbake-setup setting [--global] default dl-dir /path/to/downloads
b. Removing a setting: bitbake-setup setting [--global] --unset default dl-dir
(Bitbake rev: 713e7f213c6d4a620be9ce34d5f4396af48e1d69)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a test configuration to cover the 'bb-env-passthrough-additions'
conf.json key, and add it to the test routine.
(Bitbake rev: 24f12b68692f9ebb5d3813bc3b1771e43298f640)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-setup conf.json can be purely template driven, to setup older
yocto LTS (e.g. scarthgap) based layer-collections - where fragment
support was not yet present in oe-core+bitbake.
Add a configuration for that scenario and add it to the test routine.
(Bitbake rev: 23e121befa0779fbb1f342984b583c04ccc637a0)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds support for extending the BB_ENV_PASSTHROUGH_ADDITIONS
environment variable from within the `init-build-env` wrapper script -
generated by either oe-core's oe-setup-build, or by `bitbake-setup` -
based on per-configuration JSON settings.
This enables CI workflows to inject environment-specific data - such
as build number, host, build type, or credentials required to fetch
from certain SRC_URIs - which cannot be captured via configuration
fragments alone. These variables are now handled early in the setup
process and exported directly into the build environment.
Example:
"bb-env-passthrough-additions": [
"ACME_DIR",
"ARTIFACTORY_TOKEN",
"ARTIFACTORY_USERNAME",
"GITHUB_TOKEN",
"GITHUB_PROTOCOL",
"KEY"
]
<snip>
the resulting 'init-build-env' would then be:
# environment passthrough added by bitbake-setup
export BB_ENV_PASSTHROUGH_ADDITIONS=" \
$BB_ENV_PASSTHROUGH_ADDITIONS \
ACME_DIR \
ARTIFACTORY_TOKEN \
ARTIFACTORY_USERNAME \
GITHUB_TOKEN \
GITHUB_PROTOCOL \
KEY"
# init-build-env wrapper created by bitbake-setup
. /tmp/acme_master-acme-distro_acme-machine_bang/layers/openembedded-core/oe-init-build-env /tmp/bitbake-setup/gs/acme_master-acme-distro_acme-machine_bang/build
(Bitbake rev: 782ab99e7a04fba43bdcf5763a6280785944ae3f)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In autobuilder testing a use case arised where
- the available choices in configuration file for choosing a machine are incomplete
- putting every possible machine choice into that configuration is undesirable/not possible
- autobuilder code can write a machine selection into the bitbake config
after the fact.
So this --skip-selection option is intended for advanced users that know what they're doing
and is generally not recommended as it requires manually tweaking the bitbake config to
make it usable.
(Bitbake rev: 8cb2372bdad381179969d2ecbba7decaf03a7c5f)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using sstate mirror places much higher requirements on both network robustness
and network bandwidth than just fetching source code. When the network doesn't meet
those, the user experience can be very frustrating as errors can be cryptic and
intermittent, or bitbake would just seemingly do nothing for a long time.
Let's make sstate an opt-in, and provide a caution for using it. When it does
work, it works wonderfully and does accelerate builds significantly.
(Bitbake rev: fe88a8e9cd5ccbc26508cf524a2f71a06d3df03a)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is common practice to put the default choice in upper case for
yes/no interactive prompts, so that when people just hit enter, they
know what they are getting. An example that linux users are probably
familiar with is "sensors-detect" from the "lm-sensors" package.
Unify all the prompts to be the same and indicate that the default
answer from hitting enter is a no with an upper case N. No functional
changes.
(Bitbake rev: 7d6225722e21b116ae164fbaae2a918534a5107b)
Signed-off-by: Paul Gortmaker <paulg@kernel.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This implement the ability to use "{THISDIR}/my-layer" in the
"bb-layers" list. "{THISDIR}" is remplaced by the directory containing
the configuration file.
In small projects, we try to keep the setup a simple as possible: a
single git repo containing both the build confguration (e.g.
a bitbake-setup configuration file) and the meta layer with project
recipes/machine/distro.
This change allows this kind of setup:
├── meta-my-project/ # the project layer
└── my-project.conf.json # the bb-setup configuration file
by writing, in my-project.conf.json:
"bitbake-setup": {
"configurations": [{
"bb-layers": [
"{THISDIR}/meta-my-project"
Note: in this case meta-my-project is not present as a "source", so, not
handled by bb-setup update/status. It is expected of the user to handle
this on their own (is our case, a simple git workflow).
(Bitbake rev: b3153be29de8b8570b0c184369bd41f4c646cf92)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-setup deduces top directory from BBPATH, which, if set, interferes with
the tests' own setup.
(Bitbake rev: e974d42eb5229f755e14b46a00ad06b23b53e143)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes maintaining and extending them easier.
(Bitbake rev: 16dc8e3dad7dde7e7651cce13549e61574cafba1)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being minimalist, dash does not support the (non-POSIX) feature of
passing an argument while sourcing a script. Like in
. <some path>/oe-init-build-env <build dir>
With dash, one must use:
set <build dir> # puts <build dir> in $1
cd <some path>
. ./oe-init-build-env # can only be called from its directory in dash
To do this:
* Instead of a symlink to oe-init-build-env, keep a symlink to the
directory containing it (called "oe-init-build-env-dir")
* Generate a init-build-env script that dash can source using the above
snippet.
(Bitbake rev: 442b41c7949e1522212b66b16811f6b64b089b23)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"." is in POSIX standard[0], whereas "source" is only supported in more
feature-full shells (bash, zsh, ...)
[0]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#dot
(Bitbake rev: 22c5fe7b2de74841e86d28a81143bd1a717518d9)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Particularly:
- ensure global settings command line argument is always expanded to
full path
- ensure any errors that happen when loading settings are reported
at that point, otherwise we get an empty dictionary and cryptic
key errors later
(Bitbake rev: 578afa2f05dfa6727952365918df703875070f64)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously bitbake-setup was checking out 'detached commits' using
fetcher's nobranch feature, as that is the only option when only a revision is in the config.
Branches are optional, but beneficial, as
- checkout directory will be on a branch, making it easier for users
to understand where they are if they need to make changes (also
bitbake will print branch information instead of saying 'HEAD:sha').
- supply chain security! Enforcing a branch means any specified revision
has to be on it, and no one can sneak in (accidentally or deliberately!)
some dangling commit, or something from their private branch in the same repo.
(Bitbake rev: 45ed9b9faebdaa8cb7cc8dd2a6d51ec8eea06e73)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the directory already exists but hasn't been setup, the current test
can fail so improve it.
(Bitbake rev: dac27bd5acbde1807b9637f809fd0ee5bf424286)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch the url to be the default internal registry rather than a private
repo which was intended for testing.
(Bitbake rev: e031b75b5b92552d812d2305a35ce90eb6c68b78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than forcing every user to have to access a separate repository
for data for some key defaults, add those for nodistro and poky. This
gives us some commonly used setups and also something to test against too.
We will need to come up with a criteria for adding these, most likely based
on community usage/demand with the OE TSC governing that policy.
(Bitbake rev: 562de93d5b0e7cf0a9a43714d0563e5bcf6e9931)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is useful for bitbake-setup to support local paths without access through
the fetcher so that internal data to the bitbake repository can be used as
a default.
(Bitbake rev: ec82a6d402a0bec5704310c66c6f4206c75e4fc9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This basically calls install-buildtools from oe-core/poky, but
it ensures via command line parameters that the installation
location is stable and the downloads are preserved for reproducibility:
$ bin/bitbake-setup install-buildtools
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf
======
Buildtools archive is downloaded into /home/alex/bitbake-builds/yocto-master-testing/buildtools-downloads/20250319141333 and its content installed into /home/alex/bitbake-builds/yocto-master-testing/buildtools
... (output from install-buildtools script)
======
It also detects when buildtools are already installed, and will direct
users what to do:
======
alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup install-buildtools
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf
Buildtools are already installed in /home/alex/bitbake-builds/yocto-master-testing/buildtools.
If you wish to use them, you need to source the the environment setup script e.g.
$ . /home/alex/bitbake-builds/yocto-master-testing/buildtools/environment-setup-x86_64-pokysdk-linux
You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation
======
This commits includes fixes by Gyorgy Sarvari <skandigraun@gmail.com>
https://github.com/kanavin/bitbake/pull/2
(Bitbake rev: 3fe3096847046110c72b23fce37fb4a459b1d748)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Run like this:
alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup
test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok
----------------------------------------------------------------------
Ran 1 test in 9.223s
OK
The test does a basic run-through of init, then status/update
on an unchanged configuration, then status/update on a
configuration changed via new commits to the test layer,
then status/update on configuration changed via the top
level json config file.
Note that nothing whatsoever is fetched from the network;
the test relies entirely on synthetic data contained inside
itself, including minimal stubs for oe-setup-build and
bitbake-config-build. This data is used to create temporary
git repositories then clone them via local filesystem URIs.
Later on this can be supplemented by an oe-selftest that
tests bitbake-setup against real config files in the
official configuration repository and real layers,
templates and fragments.
(Bitbake rev: e3aa3eb46bd3196fa5415fa36e3737636fd6a1c0)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Preamble
========
The latest iteration of this patchset is available at
https://github.com/kanavin/bitbake
I recommend taking the patches from there to ensure that
you are not trying out outdated code.
For the rationale and design guidelines please see this message:
https://lists.openembedded.org/g/openembedded-architecture/message/1913
Left out for now but will be done later:
- official configuration repository
- documentation
Amble *scratch* HOWTO
=====================
1. If you don't know where to start, run 'bitbake-setup init'.
Bitbake-setup will ask a few questions about available configuration choices and set up a build.
Note: 'init' sub-command can also take a path or a URL with a configuration file directly.
You can see how those files look like here:
https://github.com/kanavin/bitbake-setup-configurations
2. You can then source the bitbake environment and run bitbake to perform builds as usual:
$ . /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env
Also, subsequent status/update commands will not require a separate --build-dir argument telling
bitbake-setup where the build is.
3. To check if the build configuration needs to be updated, run:
===
$ bin/bitbake-setup status
...
Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed.
===
If the configuration has changed, you will see the difference as a diff.
...
- "rev": "akanavin/sstate-for-all"
+ "rev": "akanavin/bitbake-setup-testing"
...
If the configuration has not changed, but layer revisions referred to it have (for example
if the configuration specifies a tip of a branch), you will see that too:
===
...
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de
===
4. If the configuration has changed, you can bring it in sync with:
$ bin/bitbake-setup update
Note that it will also rename/preserve the existing build/conf directory, and print changes
in bitbake configuration (diff of content of build/conf/) if that has changed. I can't
at the moment think of anything more clever that is also not much more brittle or complex
to implement, but open to suggestions.
Terminology
===========
- 'top directory' means the place under which bitbake-setup reads and
writes everything. bitbake-setup makes a promise to not touch anything outside of
that, unless otherwise directed to by entries in settings (currently
there is one such setting for fetcher downloads for layers and config
registries). Top directory can be selected by an environment variable, a command line option,
or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment
(e.g. we are in a bitbake environment), then the top directory is
deduced from that and doesn't need to be specified by hand.
- 'settings' means bitbake-setup operational parameters that are
global to all builds under a top directory. E.g. the location of
configuration registry, or where the bitbake fetcher should place the
downloads (DL_DIR setting). Settings are stored in a .conf file in ini
format just under the top directory.
- 'build' means a tree structure set up by 'bitbake-setup init',
consisting of, at least, a layers checkout, and a bitbake
build. It maps 1:1 to the json data it was constructed from, which is
called 'build configuration'. Build configurations are constructed from
generic configurations that may involve making one or more choices
about available options in them. Generic configurations are files, URLs
or are obtained from git repositories called 'config
registries', in which case they can be listed with 'bitbake-setup
list'. There can be multiple 'builds' under a top directory. Here are
two example generic configurations that showcase this:
https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-options.conf.jsonhttps://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-nested-configs.conf.json
- 'bitbake-setup status' will tell if a build is in sync with
the generic configuration it was made from. 'bitbake-setup update' will bring a build
in sync with a configuration if needed.
- 'bitbake build' means a particular sub-tree inside a build that
bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR
by oe-init-build-env. conf/* in that tree is 'bitbake configuration'.
Bitbake configurations are constructed from templates and fragments,
with existing mechanisms provided by oe-core. The configuration file
format is specified such that other mechanisms to set up a
bitbake build can be added; there was a mention of ability to specify
local.conf content and a set of layers directly in a configuration. I
think that scales poorly compared to templates and fragments, but I
made sure alternative ways to configure a bitbake build are possible
to add in the future :)
- 'source override' is a json file that can be used to modify revisions
and origins of layers that need to be checkout into a build (e.g.
when master branches need to be changed to master-next for purposes
of testing). Such a file is specified with a command-line option to 'init'
and an example can be seen here:
https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-next.override.json
This commit includes fixes by
Ryan Eatmon <reatmon@ti.com> https://github.com/kanavin/bitbake/pull/1
Gyorgy Sarvari <skandigraun@gmail.com> https://github.com/kanavin/bitbake/pull/2
Johannes Schneider <johannes.schneider@leica-geosystems.com> https://github.com/kanavin/bitbake/pull/3https://github.com/kanavin/bitbake/pull/5
(Bitbake rev: b96154aeb1fc89184ac245e0d68e6e726fe80c04)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lastpids has not been used for almost 10 years.
* There is no longer any need to use running_pids to keep track of the
order the pids in running_tasks were added as dicts are guaranteed to
remember the insertion order since Python 3.7.
(Bitbake rev: 562b5bf0be2883144391e7030a9dce6a233802ac)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having multiple builtin config fragments for the same variable
(eg OE_FRAGMENTS = "... machine/A ... machine/B") is not supported.
Warn the user to make them fix this but continue with the normal
variable evaluation : the last affectation "wins".
Added warning looks like:
WARNING: Multiple builtin fragments are enabled for machine via variable OE_FRAGMENTS: machine/qemux86-64 machine/test machine/qemux86-64. This likely points to a mis-configuration in the metadata, as only one of them should be set. The build will use the last value.
(Bitbake rev: 1c12aa23f6678dc289fc0e0d8b4dad311bd39c35)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
