Improve the runqueue init code to call unihash queries in parallel since
this is faster and more efficient, particularly on slower links with longer
round trip times.
The call to the function from cooker is unneeded since that function calls
prepare() and hence this functionality will already have run, so drop
that obsolete call.
(Bitbake rev: e0486054c7a4c637446c18608e9983cc8dc4d7fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Report which task dependencies in BB_TASKDEPDATA are included in the
taskhash. This allows tasks to identify which tasks dependencies may
change without the task re-running. Knowing this information is
important for tasks that want to transfer information from dependencies
(such as SPDX)
(Bitbake rev: 853423661779023763a87462b623b6e9ff2798b2)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The initial hash serve loop exits in the case where interrupts are present
but probably checks a bit too often. Tweak that and also allow the slow
rehash loop to break on interrupt, improving bitbake Ctrl+C response.
(Bitbake rev: ca2f63d84b90cbd99936c9d7815d912d9019a594)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
With hashserve enabled, there are two slow paths/loops, one at initial runqueue
generation and also during the rehash process when new outhashes are found.
Add timing information at the hashserve log level for when these loops
take longer than 30s or 60s overall. This will leave evidence in the logs when
things are running particularly slowly.
(Bitbake rev: 9ee503c79936b13f1d45f9e43211f77a528cdbfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
This adds the missing import. This bug was introduced with 1ab1d36c.
(Bitbake rev: b5159c0373e2e7d403aed16e096ad655f38b1fa7)
Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.
We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.
This hopefully fixes [YOCTO #15318].
(Bitbake rev: 2aa519fec30ecf0e55202d6421fbba2f0320e211)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d9e5d313c79500e3c70ab9c3239b6b2180194f67)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Handles ImportError when creating a hash equivalence to ping the server.
This notifies user earlier with a more precise error if websockets can't
be used, and also prevents passing a known bad upstream value to the
local server
(Bitbake rev: 93190565fd1251e8f47d9a6291739f8b8ed5ec87)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit aa80b3cfc5d16dfba13ca7fb9b78bae179ce3b74)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
The cooker attempts to connect to the upstream hash equivalent server to
warn the user early if it is misconfigured. However, this was making the
assumption that it was a raw TCP connection and failed when attempting
to use a websocket upstream server. Fix this by creating an hash client
and using the ping API to check the server instead of using a raw
socket.
(Bitbake rev: 8f90d10f9efc9a32e13f6bd031992aece79fe7cc)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Careful reading of the code can contrive cases where poorly timed
ConnectionError's will result in the client mode being incorrectly reset
to MODE_NORMAL when it should actual be a stream mode for the current
command. Fix this by no longer attempting to restore the mode when the
connection is setup. Instead, attempt to set the stream mode inside the
send wrapper for the stream data, which means that it should always end
up in the correct mode before continuing.
Also, factor out the transition to normal mode into a invoke() override
so it doesn't need to be specified over and over again.
(Bitbake rev: 0cd276fd98eeca463518d4a42675fffb18d6b3de)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the bitbake hash server supports SSL connections, we need to
capture a few environment variables which can affect the ability to
connect via SSL. Note that the variables are only put in place to affect
the environment while actually invoking the server
[RP: Tweak to use BB_ORIGENV as well]
[RP: Tweak to handle os.environ restore correctly]
(Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A function accidentally defined as:
somefunction() {
:
}
which is unclosed due to the space at the end, would currently silently
cause breakage. Have the parser throw and error for this.
[YOCTO #15470]
(Bitbake rev: a7dce72da6be626734486808f1b731247697e638)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In db.py, the ifnull() statement guarantees that the SQL request will
return a value. It's therefore unnecessary to test the case when no
value is found.
(Bitbake rev: e4ae5177861c9a27e93e5a2d3a6c393baecd6416)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
according to db.py, prserv.NotFoundError is returned here when
adding a new value to the database failed
(Bitbake rev: 4cc4069987edd14f51715dfaf0c6e1a3aa307106)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This exception handler is already present in db.py's get_value() code.
(Bitbake rev: 2fd38b1bb685ec441f0eb0f28f3d84ba252ba90b)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Useful for connecting a PR server to an upstream one
- "test-package" checks whether the specified package
version and arch is known in the database.
- "test-pr" checks a specified output hash is found in the database.
Otherwise it returns 'None' instead of a new value.
- "max-package-pr" returns the highest PR number for
(version, arch) entries in the database, and None if not found
Add new DB functions supporting the above, plus test_value()
which tells whether a given value is available for the specified
package and architecture.
(Bitbake rev: 0f1474a30f741b760ca81c19dd1d8f3bd5647251)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Choosing only one style of capitalization
Add extra space after some commas too
Remove idle spaces
(Bitbake rev: daad17bccec8cb98ef2fca4262641167500bd46e)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moving the code and related definitions from
hashserv/__init__.py to asyncrpc/client.py,
allowing this function to be used in other asyncrpc clients.
(Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In both the PRServerClient and PRClient objects.
This aligns with what is done in hashserv/server.py and makes it
possible to benefit from possible specializations of the logger
in the corresponding super classes, instead of using
always the global logger.
(Bitbake rev: 5fc6d2b1a5db617e16c1eb9fbd25e821237611d8)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
optparse is deprecated since Python 2.7
Note that this is neither supposed to change the options
supported by bitbake-prserv nor the way they are interpreted.
Note that in the "--help" output, long options are now reported
for example as "--host HOST" instead of "--host=HOST" but
both are equivalent anyway, as they already were with optparse.
(Bitbake rev: 434cd00a9e5a8ef32088f1a587005adf910a92eb)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To aligh with the hashserv code
(Bitbake rev: 7a6999750791659eaffe49aabfbfba9f37f51913)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv.py: simplify the PRServerClient() interface by passing the
server object instead of multiple arguments, and then retrieving
the data through this object.
This replicates what is done for ServerClient() in hashserv/server.py
(Bitbake rev: d3be073218feb4d6e68a751832da4936da485dbc)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Cc: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Document BB_LOADFACTOR_MAX which was recently added.
(Bitbake rev: 833b76e9333e317cab5f17d6f7daaecc89c69547)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is inspired by the same section in the yocto-docs. It aims to provide
information in what contexts(recipes, .conf, bbclass,...) a variable is usually
used. For that I tried to group similar variables, so that a short overview is
given. This was inspired by [YOCTO #14072], but of course does not implement a
warning if a variable is used in an unintended context.
(Bitbake rev: 5ced476685376b1a32b7fdd9546f9b61c5962aa0)
Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fedora 40 introduces wget2 as a drop-in replacement for wget. This
rewrite does not currently have support for FTP. This causes
the wget fetcher to fail complaining about an unrecognized option.
Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem. It also gives us
an opportunity to build the older wget as a host tool.
(Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is an error parsing .netrc, warn the user on stderr
(Bitbake rev: 6366ea8d9c284d10bb8f4129004b55239d9022c0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for hashserver credentials to be specified in the
SignatureGenerator
(Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Git uses a lock file to prevent concurrent modifications to the global
config, so if unpack tasks for different recipes try to run "git lfs
install" simultaneously the operation can fail:
error: could not lock config file /home/build/.gitconfig: File exists exit status 255
Run `git lfs install --force` to reset Git configuration.
Adding "--local" sets the smudge and clean filters in the local
repository's config instead of modifying the user's global config.
(Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7)
Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch addresses an issue in bitbake-worker where stdout,
reserved for status reporting, is improperly accessed by child processes.
The problem occurs during the execution of parseRecipe,
which calls anonymous functions. If these functions use print-like operations,
they can inadvertently output data to stdout. This unexpected data can cause
the runqueue to hang silently, if the stdout buffer is flushed
before exec_task is executed.
To prevent this, the patch redirects stdout to /dev/null and ensures it is
flushed prior to the execution of exec_task.
(Bitbake rev: 08f3e677d6af27a41a918aaa9da9c1c9b20a0b95)
Signed-off-by: Yang Xu <yang.xu@mediatek.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running in a cgroup which is limited to a subset of cpus (via
cpuset.cpus), cpu_count() should return the number of cpus that can be
used instead of the number of cpus the system has.
This also aligns the semantics with nproc.
(Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a command is run with a non-null log, errors are only output to the
log and are not returned in the exception. In that case direct users to
that logfile instead of telling the command had no output.
(Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a)
Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.
For example: http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:
File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
self.query = self._param_str_split(urlp.query, "&")
File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)
In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.
The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.
Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().
(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Red text on a black background can make it difficult for people with
visual impairments to read the text of error messages. Respect the
presence of a non-empty NO_COLOR environment variable as an indication
the user doesn't want colorization to be automatically enabled.
See: https://no-color.org/
(Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85)
Signed-off-by: Peter A. Bigot <pab@pabigot.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FIXES [YOCTO #15404]
When using git fetcher on a repo with parentheses in its URL, the
invocation of the git clone command will fail. The clone directory
is not quoted thus the shell will return an error and won't execute
the command.
(Bitbake rev: b5624ee564)
Cc: Philippe Rivest <privest@genetec.com>
(Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6)
Signed-off-by: Philippe Rivest <technophil98@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Limit the split to key & value (2 items) instead of the n items one
can get if there are '=' characters in the value.
Fixes [YOCTO #15447]
(Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb)
Signed-off-by: david d zuhn <david.zuhn@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* In the current implementation, "umask" variable is initially set to
None and overwritten with user-specified value. However, in the worker
implementation, a faulty if clause would only check whether the
variable contains a value that evaluates to True, and not whether
the variable is defined, so the value of 0 would lead to umask not
being changed.
This bug makes it impossible to have a task set its umask to value 0,
for any possible reason it may want to.
Fix this bug by extending the condition checked in the worker
implementation.
(Bitbake rev: 19f9df6c750c592316a0fa18165b68636281fe3e)
Signed-off-by: Sava Jakovljev <sjakovljev@outlook.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
preserved_envvars() is used when the BB_ENV_PASSTHROUGH
environment variable is not set. Therefore, its code shouldn't
return this variable.
(Bitbake rev: 0a33b560233b983456178541603ab96fea22238b)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A race condition has happened where the exiting server removed the PID
file between the existence check and the removal, resulting in a
FileNotFoundError exception.
The fix is to ignore the FileNotFoundError exception, the existence
check is now redundant so remove it to simplify.
Fixes [YOCTO #14341]
(Bitbake rev: 40d00bf9308e0bf73a00134a99a012a292daa1c5)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently when git fetches fail, it destroys all the existing local clone data.
For large repositories this can introduce long build delays when for example,
you just typo'd the git revision hash.
The git fetcher should be able to recover most directories so when the fetch is
for a git repo, avoid removing things unless clean is explicitly called
(e.g. a -c cleanall task).
(Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop the ".py" file extension from the "-u" option's argument that has been
overlooked while applying the original patch (see [1]) to make the example work.
While at it sort the recipes' names consistently with respect to what is found
in the self-test examples below.
[1] https://lore.kernel.org/bitbake-devel/6f2645a7c4db2ae149d387544d2b94209cfed3f4.camel@linuxfoundation.org/
(Bitbake rev: 1f4d517b7a0389f78d1f791135f8dc9120e9912b)
Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for asyncrpc servers to send connection headers to clients
on connection. Since this is a breaking protocol change, clients must
opt-in to expect headers from the server, corresponding to a version
bump in the client protocol.
(Bitbake rev: 1cb2b8be6cc5269553f549285592e47b7d29db03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a LayerBranch (a specific layer at a specific release) does not
exist in the layerindex database ("Layerindex Metadata"), the dependency
would throw a backtrace. Instead fail early and provide an error message.
Since layerindexlib will also check the local layers, inform the user that
a local checkout might resolve the situation. Recommend that they reach
out to the layer maintainers and layer index admins to properly fix it for
everyone.
While we are here, remove some trailing whitespace.
[YOCTO #15365]
[YOCTO #13954]
(Bitbake rev: 96cbe8f87209a927c157ebcf469f8b9d54fcf92e)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).
Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.
This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.
(Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a very rare case where the maxval is improperly computed
initially for cache loading progress, and the value will go over.
Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:progress()
# we might have calculated incorrect total size because a file
# might've been written out just after we checked its size
In that case, progressbar will receive a value over the initial maxval.
This results in a ValueError stack trace as well as bitbake returning 1.
Traceback (most recent call last):
File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main
cacheprogress.update(event.current)
File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in update
raise ValueError('Value out of range')
ValueError: Value out of range
This fix mirrors the behavior of MulticonfigCache and accepts the new
value as the new maxval. This is also what the percentage printout
is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:percentage()
I encountered this issue randomly while working on a project with
VSCode saving files while commands where fired.
Note: This file is a fork from python-progressbar. It hasn't been
refreshed in 8 years. We did only two commits, 5 years ago with minor
modifications. This new change is also not how the upstream project is
behaving.
(Bitbake rev: 7cea7f7a87da041fc1ad370c5c3d15aabad3a0d4)
Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some ditros don't enable /proc/pressure and it tends to be those which we
see bitbake timeout issues on, seemingly as load gets too high and the bitbake
processes don't get scheduled in for minutes at a time.
Add support for stopping running extra tasks if the system load average goes
above a certain threshold by setting BB_LOADFACTOR_MAX.
The value used is scaled by CPU number, so a value of 1 would be when
the load average equals the number of cpu cores of the system, under one
only starts tasks when the load average is below the number of cores.
This means you can centrally set a value such as 1.5 which will then
scale correctly to different sized machines with differing numbers
of CPUs.
The pressure regulation is probably more accurate and responsive, however
our graphs do show singificant load spikes on some workers and this
patch is aimed at trying to avoid those.
Pressure regulation is used where available in preference to this load
factor regulation when both are set.
(Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Re-enable connection pooling in case `postgresql+psygopg` driver
is used. Async connection pooling is supported in psycopg 3 [psycopg]
driver in SQLAlchemy. Allow the connection pool to grow to
arbitrary size.
(Bitbake rev: 4fe05513b5314c201725e3f8ad54f58d70c56258)
Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 987ab2a446.
There's been a report that this breaks downloads from Jfrog Artifactory
as self.user_agent is set to 'Mozilla Firefox', and when Artifactory
sees that, it sends a response tailored for showing in an interactive browser
(which in my opinion it has every right to).
If we're using wget, we should say so via wget's default; handling uncooperative
servers should be done on per-recipe basis, and ideally with tickets
to admins of those servers.
(Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before, everything up to the last slash was removed when extracting the
names of the tags. This would lead to that a tag such as "agent/11.0.0"
would be incorrectly identified as "11.0.0", which would then be treated
as a correct version matching "^(?P<pver>\d+(\.\d+)+)".
(Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is mostly preparations for the next commit.
(Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
