Setting a key=value in a new section would raise a key-error when
using in-memory settings, e.g:
./bin/bitbake-setup --setting default top-dir-prefix /tmp/bitbake-setup --setting default top-dir-name gs settings foo bar baz
Loading settings from
/tmp/bitbake-setup/gs/settings.conf
Traceback (most recent call last):
File "/tmp/bitbake-setup/bitbake/./bin/bitbake-setup", line 853, in <module>
main()
File "/tmp/bitbake-setup/bitbake/./bin/bitbake-setup", line 838, in main
all_settings = merge_settings(builtin_settings, global_settings, topdir_settings, args.setting)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/bitbake-setup/bitbake/./bin/bitbake-setup", line 733, in merge_settings
all_settings[section][setting] = value
~~~~~~~~~~~~^^^^^^^^^
KeyError: 'foo'
(Bitbake rev: 78ab0d15dff5ccf64b0bf681185370779e6cabaf)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename the function to align with the corresponding
'global_settings_path' function, and move it down just above the
later.
This is done to differentiate from the built-in default settings, and
to align with the other places in the code that use 'topdir_settings'
(or 'global_settings')
(Bitbake rev: 3df994f773abbd1d0240e721f5fd29d4b021bfb5)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After a terminology review by Antonin we brainstormed
something less confusing than 'build' and 'build directory'
for the place where bitbake-setup clones layers and
creates a *bitbake* build directory in. People are bound
to get these two confused and mix them up, and 'setup' is
much more distinct and aligns nicely with 'bitbake-setup'.
It's also not claimed by anything else in OE/Yocto.
So before:
top-dir -> build-dir (can be several) -> bitbake build dir, layer dir, config dir
Now:
top-dir -> setup-dir (can be several) -> bitbake build dir, layer dir, config dir
This also updates the respective command line options, I understand it's
a breaking change, but as before the tweaks are simple and we need to get the
terminology right for the users, and now is the time to do it.
(Bitbake rev: eeb81a35bf0304451f7612950d5156ea7ff18bad)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
--global was defined twice (for settings command as a whole, and specifically
for set/unset sub-commands), which caused 'settings --global set ...' to
be allowed, and at the same time not have any effect. 'settings set --global'
is the only valid form after this change.
(Bitbake rev: 2f14cf68f6c653b6da90180a99a22c58855b0cef)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Tested-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously the sub-command 'settings' would take any number of
arguments and then silently do nothing if the number wasn't three.
The help text was also not clear about this, marking the positionals
separately as optional:
usage: bitbake-setup settings [-h] [--global] [--unset UNSET UNSET] [-l] [section] [key] [value]
The '--unset SECTION SETTING' also did not integrate too well, as it
had its own positional arguments for section+setting.
For a bit more consistency and a explorable help, a sub-subparser is
added, that provides the commands:
bitbake-setup settings list
bitbake-setup settings set foo bar baz
bitbake-setup settings unset foo bar
with a '--global' that is added from a stand-alone parent parser, so
that it shows up in all sub-command help texts.
The new help text now reads:
usage: bitbake-setup settings [-h] [--global] {list,set,unset} ...
and the respective sub commands:
usage: bitbake-setup settings list [-h] [--global]
usage: bitbake-setup settings set [-h] [--global] <section> <setting> <value>
usage: bitbake-setup settings unset [-h] [--global] <section> <setting>
(Bitbake rev: 8b582ef8dd0cef0192d4c0104bcd9b5d642d132c)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a metavar to the argparse options to have a self-descriptive help text.
Otherwise argpase defaults to use the argument name in all-uppercase.
Before:
usage: bitbake-setup [-h] [-d] [-q] [--color COLOR] [--no-network] [--global-settings GLOBAL_SETTINGS] [--setting SETTING SETTING SETTING]
{list,init,status,update,install-buildtools,settings} ...
After:
usage: bitbake-setup [-h] [-d] [-q] [--color COLOR] [--no-network] [--global-settings PATH] [--setting SECTION SETTING VALUE]
{list,init,status,update,install-buildtools,settings} ...
(Bitbake rev: 83cecc9356a0684f90249d527fe372298ae92719)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To align the storage destination with the internally used variable
name. This makes room for having another option use 'args.setting'
(Bitbake rev: 14d8535309abc78ee30cfdb51bba2e00b474f443)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In cases where the first call to 'init' failed or was aborted before
creating the 'build/init-build-env' has been created, a user can get
stuck: a second call to init aborts, suggesting 'status' or 'update'
but these to refuse because the --build-dir is not valid.
Guide the user by adding a suggestion to start over from scratch.
(Bitbake rev: 11b2740c3e19e0c6680229c6bbce3691c73746a8)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After some further feedback, additional changes are made:
1. 'setting' command is renamed to 'settings' to better reflect
that it is an interface to various ways of managing settings.
2. This command now has a -l/--list option to list all settings
with their values (same as 'git config -l').
3. A new level of settings (built-in defaults) is added,
and used as a last resort after command line options, top dir
settings file and global settings file.
4. This means bitbake-setup does not have to write and use a
global settings file, and it no longer does so when initializing
a build, avoiding default 'pollution' of ~/.config/bitbake-setup/
which can be problematic or unwelcome.
A global settings file is still created if a setting is explicitly
requested to be placed into it.
5. 'install-global-settins' is removed as the use case for it
(tweak default settings before using them to initialize a build)
can be achieved by setting the settings individually.
5. Similarly, a top dir settings file is no longer created by default
and only appears if a setting needs to be written into it.
6. Default dl-dir is again created inside a top directory and not
in ~/.cache/ to make default builds fully contained in the top
directory (which was also asked about).
(Bitbake rev: 664f8ec48d42d2ddc5f234c4f7d590fa597f489a)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the outcome of various discussions, suggestions and pull
requests on github.
What has specifically changed?
1. The sources for the settings are no longer separated, but are stacked and given priorities,
from highest to lowest:
a. '--setting section key value' on the command line
b. a settings file in the top directory
c. a global settings file in ~/.config/bitbake-setup/ (or in a file pointed to by --global-settings)
Any setting can be in any of these three locations (other than top dir name and prefix which do not
make sense in the settings file in the top directory).
2. A global settings file must contain all of the needed settings, while a settings file
in the top directory can be empty (and this is how they are written out if they do not exist).
Specifically, both dl-dir and registry settings have been relocated to the global file,
and dl-dir defaults to ~/.cache/bitbake-setup/downloads, rather than somewhere in top dir.
3. The file name for both global and top dir settings is now 'settings.conf'.
4. --top-dir-prefix and --top-dir-name options have been removed and superseded by
a generic, universal --setting option.
5. 'install-settings' command has been removed, as it is no longer does anything useful,
and is superseded by the 'setting' command (see below).
'install-global-settings' has been retained, to be able to have a set of global defaults
that can be changed without initializing a build.
6. 'change-setting', 'change-global-setting' and 'install-settings' have all been replaced
by a single 'setting' command that mimics 'git config' in its parameters:
a. Changing a setting: bitbake-setup setting [--global] default dl-dir /path/to/downloads
b. Removing a setting: bitbake-setup setting [--global] --unset default dl-dir
(Bitbake rev: 713e7f213c6d4a620be9ce34d5f4396af48e1d69)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds support for extending the BB_ENV_PASSTHROUGH_ADDITIONS
environment variable from within the `init-build-env` wrapper script -
generated by either oe-core's oe-setup-build, or by `bitbake-setup` -
based on per-configuration JSON settings.
This enables CI workflows to inject environment-specific data - such
as build number, host, build type, or credentials required to fetch
from certain SRC_URIs - which cannot be captured via configuration
fragments alone. These variables are now handled early in the setup
process and exported directly into the build environment.
Example:
"bb-env-passthrough-additions": [
"ACME_DIR",
"ARTIFACTORY_TOKEN",
"ARTIFACTORY_USERNAME",
"GITHUB_TOKEN",
"GITHUB_PROTOCOL",
"KEY"
]
<snip>
the resulting 'init-build-env' would then be:
# environment passthrough added by bitbake-setup
export BB_ENV_PASSTHROUGH_ADDITIONS=" \
$BB_ENV_PASSTHROUGH_ADDITIONS \
ACME_DIR \
ARTIFACTORY_TOKEN \
ARTIFACTORY_USERNAME \
GITHUB_TOKEN \
GITHUB_PROTOCOL \
KEY"
# init-build-env wrapper created by bitbake-setup
. /tmp/acme_master-acme-distro_acme-machine_bang/layers/openembedded-core/oe-init-build-env /tmp/bitbake-setup/gs/acme_master-acme-distro_acme-machine_bang/build
(Bitbake rev: 782ab99e7a04fba43bdcf5763a6280785944ae3f)
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In autobuilder testing a use case arised where
- the available choices in configuration file for choosing a machine are incomplete
- putting every possible machine choice into that configuration is undesirable/not possible
- autobuilder code can write a machine selection into the bitbake config
after the fact.
So this --skip-selection option is intended for advanced users that know what they're doing
and is generally not recommended as it requires manually tweaking the bitbake config to
make it usable.
(Bitbake rev: 8cb2372bdad381179969d2ecbba7decaf03a7c5f)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is common practice to put the default choice in upper case for
yes/no interactive prompts, so that when people just hit enter, they
know what they are getting. An example that linux users are probably
familiar with is "sensors-detect" from the "lm-sensors" package.
Unify all the prompts to be the same and indicate that the default
answer from hitting enter is a no with an upper case N. No functional
changes.
(Bitbake rev: 7d6225722e21b116ae164fbaae2a918534a5107b)
Signed-off-by: Paul Gortmaker <paulg@kernel.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This implement the ability to use "{THISDIR}/my-layer" in the
"bb-layers" list. "{THISDIR}" is remplaced by the directory containing
the configuration file.
In small projects, we try to keep the setup a simple as possible: a
single git repo containing both the build confguration (e.g.
a bitbake-setup configuration file) and the meta layer with project
recipes/machine/distro.
This change allows this kind of setup:
├── meta-my-project/ # the project layer
└── my-project.conf.json # the bb-setup configuration file
by writing, in my-project.conf.json:
"bitbake-setup": {
"configurations": [{
"bb-layers": [
"{THISDIR}/meta-my-project"
Note: in this case meta-my-project is not present as a "source", so, not
handled by bb-setup update/status. It is expected of the user to handle
this on their own (is our case, a simple git workflow).
(Bitbake rev: b3153be29de8b8570b0c184369bd41f4c646cf92)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being minimalist, dash does not support the (non-POSIX) feature of
passing an argument while sourcing a script. Like in
. <some path>/oe-init-build-env <build dir>
With dash, one must use:
set <build dir> # puts <build dir> in $1
cd <some path>
. ./oe-init-build-env # can only be called from its directory in dash
To do this:
* Instead of a symlink to oe-init-build-env, keep a symlink to the
directory containing it (called "oe-init-build-env-dir")
* Generate a init-build-env script that dash can source using the above
snippet.
(Bitbake rev: 442b41c7949e1522212b66b16811f6b64b089b23)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"." is in POSIX standard[0], whereas "source" is only supported in more
feature-full shells (bash, zsh, ...)
[0]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#dot
(Bitbake rev: 22c5fe7b2de74841e86d28a81143bd1a717518d9)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Particularly:
- ensure global settings command line argument is always expanded to
full path
- ensure any errors that happen when loading settings are reported
at that point, otherwise we get an empty dictionary and cryptic
key errors later
(Bitbake rev: 578afa2f05dfa6727952365918df703875070f64)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously bitbake-setup was checking out 'detached commits' using
fetcher's nobranch feature, as that is the only option when only a revision is in the config.
Branches are optional, but beneficial, as
- checkout directory will be on a branch, making it easier for users
to understand where they are if they need to make changes (also
bitbake will print branch information instead of saying 'HEAD:sha').
- supply chain security! Enforcing a branch means any specified revision
has to be on it, and no one can sneak in (accidentally or deliberately!)
some dangling commit, or something from their private branch in the same repo.
(Bitbake rev: 45ed9b9faebdaa8cb7cc8dd2a6d51ec8eea06e73)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the directory already exists but hasn't been setup, the current test
can fail so improve it.
(Bitbake rev: dac27bd5acbde1807b9637f809fd0ee5bf424286)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch the url to be the default internal registry rather than a private
repo which was intended for testing.
(Bitbake rev: e031b75b5b92552d812d2305a35ce90eb6c68b78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is useful for bitbake-setup to support local paths without access through
the fetcher so that internal data to the bitbake repository can be used as
a default.
(Bitbake rev: ec82a6d402a0bec5704310c66c6f4206c75e4fc9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This basically calls install-buildtools from oe-core/poky, but
it ensures via command line parameters that the installation
location is stable and the downloads are preserved for reproducibility:
$ bin/bitbake-setup install-buildtools
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf
======
Buildtools archive is downloaded into /home/alex/bitbake-builds/yocto-master-testing/buildtools-downloads/20250319141333 and its content installed into /home/alex/bitbake-builds/yocto-master-testing/buildtools
... (output from install-buildtools script)
======
It also detects when buildtools are already installed, and will direct
users what to do:
======
alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup install-buildtools
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf
Buildtools are already installed in /home/alex/bitbake-builds/yocto-master-testing/buildtools.
If you wish to use them, you need to source the the environment setup script e.g.
$ . /home/alex/bitbake-builds/yocto-master-testing/buildtools/environment-setup-x86_64-pokysdk-linux
You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation
======
This commits includes fixes by Gyorgy Sarvari <skandigraun@gmail.com>
https://github.com/kanavin/bitbake/pull/2
(Bitbake rev: 3fe3096847046110c72b23fce37fb4a459b1d748)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Preamble
========
The latest iteration of this patchset is available at
https://github.com/kanavin/bitbake
I recommend taking the patches from there to ensure that
you are not trying out outdated code.
For the rationale and design guidelines please see this message:
https://lists.openembedded.org/g/openembedded-architecture/message/1913
Left out for now but will be done later:
- official configuration repository
- documentation
Amble *scratch* HOWTO
=====================
1. If you don't know where to start, run 'bitbake-setup init'.
Bitbake-setup will ask a few questions about available configuration choices and set up a build.
Note: 'init' sub-command can also take a path or a URL with a configuration file directly.
You can see how those files look like here:
https://github.com/kanavin/bitbake-setup-configurations
2. You can then source the bitbake environment and run bitbake to perform builds as usual:
$ . /home/alex/bitbake-builds/yocto-master-options-poky-distro_poky-machine_qemux86-64/build/init-build-env
Also, subsequent status/update commands will not require a separate --build-dir argument telling
bitbake-setup where the build is.
3. To check if the build configuration needs to be updated, run:
===
$ bin/bitbake-setup status
...
Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed.
===
If the configuration has changed, you will see the difference as a diff.
...
- "rev": "akanavin/sstate-for-all"
+ "rev": "akanavin/bitbake-setup-testing"
...
If the configuration has not changed, but layer revisions referred to it have (for example
if the configuration specifies a tip of a branch), you will see that too:
===
...
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de
===
4. If the configuration has changed, you can bring it in sync with:
$ bin/bitbake-setup update
Note that it will also rename/preserve the existing build/conf directory, and print changes
in bitbake configuration (diff of content of build/conf/) if that has changed. I can't
at the moment think of anything more clever that is also not much more brittle or complex
to implement, but open to suggestions.
Terminology
===========
- 'top directory' means the place under which bitbake-setup reads and
writes everything. bitbake-setup makes a promise to not touch anything outside of
that, unless otherwise directed to by entries in settings (currently
there is one such setting for fetcher downloads for layers and config
registries). Top directory can be selected by an environment variable, a command line option,
or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment
(e.g. we are in a bitbake environment), then the top directory is
deduced from that and doesn't need to be specified by hand.
- 'settings' means bitbake-setup operational parameters that are
global to all builds under a top directory. E.g. the location of
configuration registry, or where the bitbake fetcher should place the
downloads (DL_DIR setting). Settings are stored in a .conf file in ini
format just under the top directory.
- 'build' means a tree structure set up by 'bitbake-setup init',
consisting of, at least, a layers checkout, and a bitbake
build. It maps 1:1 to the json data it was constructed from, which is
called 'build configuration'. Build configurations are constructed from
generic configurations that may involve making one or more choices
about available options in them. Generic configurations are files, URLs
or are obtained from git repositories called 'config
registries', in which case they can be listed with 'bitbake-setup
list'. There can be multiple 'builds' under a top directory. Here are
two example generic configurations that showcase this:
https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-options.conf.jsonhttps://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-nested-configs.conf.json
- 'bitbake-setup status' will tell if a build is in sync with
the generic configuration it was made from. 'bitbake-setup update' will bring a build
in sync with a configuration if needed.
- 'bitbake build' means a particular sub-tree inside a build that
bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR
by oe-init-build-env. conf/* in that tree is 'bitbake configuration'.
Bitbake configurations are constructed from templates and fragments,
with existing mechanisms provided by oe-core. The configuration file
format is specified such that other mechanisms to set up a
bitbake build can be added; there was a mention of ability to specify
local.conf content and a set of layers directly in a configuration. I
think that scales poorly compared to templates and fragments, but I
made sure alternative ways to configure a bitbake build are possible
to add in the future :)
- 'source override' is a json file that can be used to modify revisions
and origins of layers that need to be checkout into a build (e.g.
when master branches need to be changed to master-next for purposes
of testing). Such a file is specified with a command-line option to 'init'
and an example can be seen here:
https://github.com/kanavin/bitbake-setup-configurations/blob/main/yocto-master-next.override.json
This commit includes fixes by
Ryan Eatmon <reatmon@ti.com> https://github.com/kanavin/bitbake/pull/1
Gyorgy Sarvari <skandigraun@gmail.com> https://github.com/kanavin/bitbake/pull/2
Johannes Schneider <johannes.schneider@leica-geosystems.com> https://github.com/kanavin/bitbake/pull/3https://github.com/kanavin/bitbake/pull/5
(Bitbake rev: b96154aeb1fc89184ac245e0d68e6e726fe80c04)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
