The websockets library throws a number of exceptions which are currently
not caught leading to unhandled exceptions in the idle loop.
Fix this by catching them and reexposing them as a `ConnectionError`
which is the exception expected by users of `asyncrpc`.
(Bitbake rev: 41d62911a480283287265fe063696d2acd5904aa)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default connection timeout for websockets is 10 seconds, so use the
provided client timeout instead (which defaults to 30 seconds).
(Bitbake rev: 23681775e5941e54ebead469addf708fca1e6beb)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Batching support on the client side has proven to be a much more
effective way of dealing with server latency than multiple client
connections and is also much nicer on the server, so drop the client
pool support from asyncrpc and the hash server
(Bitbake rev: 6f80560f1c7010d09fe5448fdde616aef8468102)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: We can call multiprocessing close() unconditionally and tweak a
comment give 3.8 is now the minimum version.
lib/bb: We can drop the logger addition code only needed before 3.6
asyncrpc/hashserv: Since the minimum version is 3.8, we can drop the
conditional code.
(Bitbake rev: 16f4386400f88ba50605307961c248bef09895c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python 3.10 support is only available in websockets 10.0 and later:
08d8011132
Update the version for this case. This avoids failures on Ubuntu 22.04.
(Bitbake rev: 0e4767c4a880408750e1a6855270c5a4eef8383d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Checks that the minimum version of the websockets module is present, and
if not raises an ImportError. This allows the user to get earlier
feedback if using websockets is going to succeed
(Bitbake rev: 330ea6914aad65dc8b34c986c44779820c392f03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moving the code and related definitions from
hashserv/__init__.py to asyncrpc/client.py,
allowing this function to be used in other asyncrpc clients.
(Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for asyncrpc servers to send connection headers to clients
on connection. Since this is a breaking protocol change, clients must
opt-in to expect headers from the server, corresponding to a version
bump in the client protocol.
(Bitbake rev: 1cb2b8be6cc5269553f549285592e47b7d29db03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an abstract base class that can be used to implement a pool of
client connections. The class implements a thread that runs an async
event loop, and allows derived classes to schedule work on the loop and
wait for the work to be finished.
(Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds an API to explicitly disconnect a client. This can be useful for
testing the auto-reconnect behavior of clients
(Bitbake rev: cb2fec7eaca20608faf4b8ff11ec8590ac7f2229)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Invalidate the loop in the client close() call so that it is not closed
twice (which is an error in the asyncio code)
(Bitbake rev: ef22f8ee82c242383625f078baafb629e45dad7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support for Invocation Errors (that is, errors raised by the actual
RPC call instead of at the protocol level) to propagate across the
connection. If a server RPC call raises an InvokeError, it will be sent
across the connection and then raised on the client side also. The
connection is still terminated on this error.
(Bitbake rev: 50ee68175e7cf20a32bfbb176db2c47d7859da04)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds context manager API for the asyncrcp client class which allow
writing code that will automatically close the connection like so:
with hashserv.create_client(address) as client:
...
Rework the bitbake-hashclient tool and PR server to use this new API to
fix warnings about unclosed event loops when exiting
(Bitbake rev: d01d684a0f6398270fe35ed59b7d28f3fd9b7e41)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds support to the hash equivalence client and server to communicate
over websockets. Since websockets are message orientated instead of
stream orientated, and new connection class is needed to handle them.
Note that websocket support does require the 3rd party websockets python
module be installed on the host, but it should not be required unless
websockets are actually being used.
(Bitbake rev: 56dd2fdbfb6350a9eef43a12aa529c8637887a7e)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrites the asyncrpc client and server code to make it possible to have
other transport backends that are not stream based (e.g. websockets
which are message based). The connection handling classes are now shared
between both the client and server to make it easier to implement new
transport mechanisms
(Bitbake rev: 2aaeae53696e4c2f13a169830c3b7089cbad6eca)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect_unix() call had a bug where if a relative path to a socket
was passed (which the non-async client always does), and the current
working directory was changed after the initial call, it would fail to
reconnect if it became disconnected, since the socket couldn't be found
relative to the new current working directory.
To work around this, change the socket connection for UNIX domain
sockets to be synchronous and change current working before connecting.
This isn't ideal since the connection could block the entire event loop,
but in practice this shouldn't happen since the socket are local files
anyway.
Help debugging and resolving from Joshua Watt.
(Bitbake rev: 5964bb67bb20df7f411ee0650cf189504a05cf25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where copyright headers were not present, add them to make things
clear.
(Bitbake rev: 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevents `ResourceWarning: unclosed event loop` warnings when using the
synchronous client and python exits
(Bitbake rev: 8b95972bc04ce52a98c7780184af15a5e95f987b)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
asyncio in older Python 3.x (seen with 3.7) can seemingly hang if
new_event_loop is called repeatedly in the same process. The
reuse of processes in the Bitbake thread pool during parsing seems
to be able to trigger this with the PR server export selftest.
It appears that calling set_event_loop with the new loop avoids the
issue, so that is now done in the asyncrpc Client initializer (with
an explanatory comment). This should be revisited when the day
arrives that Python 3.9 becomes the minimum required for BitBake.
Additionally, it was discovered that using get_event_loop in the
asyncrpc server initialization can trigger hangs in the hashserv
unittests when the second test is run. To avoid this, switch to
calling new_event_loop + set_event_loop in the initialization code
there as well.
(Bitbake rev: bb9a36563505652294b20b4c88199b24fa208342)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is an issue with the format of the reply given by the server then we
should print this reply line in the error message. Printing the message which
the client sent doesn't illuminate anything here.
(Bitbake rev: bd8f8d7b055da15cd7bdd0b383061852a0f54cb7)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This method is needed to support startup of the prservice. As it is so
generic we can add it to the common asyncrpc module.
(Bitbake rev: 25ccd697ea76f66b813be2296866b2d3405b079c)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The hashserv module implements a flexible RPC mechanism based on sending
json formatted messages over unix or tcp sockets and uses Python's
asyncio features to build an efficient message loop on both the client
and server side. Much of this implementation is not specific to the
hash equivalency service and can be extracted into a new module for
easy re-use elsewhere in bitbake.
(Bitbake rev: 4105ffd967fa86154ad67366aaf0f898abf78d14)
Signed-off-by: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
