In multilib builds python3-config gets renamed to eg python3-config-lib64 but
this ends up being packaged in python3-core not python3-dev.
The manifest uses an extended glob to package all python* binaries that are not
python-config into python3-core:
"${bindir}/python*[!-config]",
However, this doesn't do what was intended, as [] is a range match.
Replace the globs with more verbose but precise matches, and clear out
FILES_${PN} to ensure that new binaries don't end up in ${PN} (which shouldn't
exist).
[ YOCTO #13592 ]
(From OE-Core rev: c780f21c33b0684dafff3421600cd1c11ddd0c48)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3-config uses distutils:
Traceback (most recent call last):
File "/usr/bin/python3-config", line 9, in <module>
from distutils import sysconfig
ModuleNotFoundError: No module named 'distutils'
Add the dependency so that distutils is always present.
[ YOCTO #13592 ]
(From OE-Core rev: 04136dbac48986dce5b2b872b2c0b46c673c44f2)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The use case is building a gpl3-free image, without having
to rely on outdated recipes from meta-gplv2 layer.
(From OE-Core rev: 02eb487c8145e0f3d957c39cf16f6f805e95e536)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runpy module is used to implement 'python3 -m foo', so move it to
python3-core as it's an essential part of the CLI.
(From OE-Core rev: eba857d3e52f83d426e95fa8373799da058f9484)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this patch:
# python3 -m test -v test_locale
[snip]
test_getsetlocale_issue1813 (test.test_locale.TestMiscellaneous) ... testing with ('tr_TR', 'ISO8859-9') ok
[snip]
After this patch:
# python3 -m test -v test_locale
[snip]
test_getsetlocale_issue1813 (test.test_locale.TestMiscellaneous) ... testing with ('tr_TR', 'ISO8859-9')... ok
[snip]
Make the test ended with "... ok" is common in python
unittest world, we should make it keep consistent
with other test cases in case it may be ignored to
record in the report if we use the common filter
"... ok".
[YOCTO #13298]
(From OE-Core rev: 4e698cf8f3a9c661f5b67c6001ad4d92b574d136)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Needing the Windows distutils installer stubs is quite a niche requirement, so
put them into a separate package and just recommend it.
This can save both space and legal pain, as the installers embed an old zlib
that has known CVEs.
(From OE-Core rev: 617331a42fc5bde0c2d8d0f4dd8df652daa28778)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
distutils doesn't ship any static libraries, and the files this should pick up
do not exist. Copy/paste gone wrong maybe?
(From OE-Core rev: afbca8da4ddd6c3dd01bb44af43aee9e80cae512)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Applies a patch to python that makes the pre-compiled .pyc files
generated during the build reproducible.
(From OE-Core rev: 148d54f91f43147f31b16a7c2cb1ade385832366)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reformats the sysconfig file when packaging. This file is output by
using the python pprint function. This function will wrap long lines at
80 characters by default, and will even split strings at whitespace
boundaries to do so, e.g.:
'A': 'B is really'
' long'
This causes a problem for reproducibility however because there might be
lines of differing lengths depending on the build path. These
non-reproducible paths are removed, but their effect on string wrapping
from pprint remains.
To correct this, reformat the entire sysconfig file by re-printing using
pprint with an (effectively) unlimited line length.
(From OE-Core rev: ec8a2b310d5f0b42f60898a5c6d239949842b34c)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building x86->x86 the system will try to execute .so and related items
from the default PYTHONPATH. This will fail if the target CPU contains
instructions that the host CPU does not have, add CROSSPYTHONPATH
into PYTHONPATH so we can prepend the list to find correct libs.
Fixes:
Illegal instruction (core dumped)
Makefile:625: recipe for target 'sharedmods' failed
make: *** [sharedmods] Error 132
make: *** Waiting for unfinished jobs....
(From OE-Core rev: 2106a567820bad438ff78d54a49e3d87da428dcf)
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch originally only meant to correct the python3 build for mips
with softfloat, as the original test only checked for mips hardfloat.
Replaced custom C Program for triplet detection with autotools triplet
detection.
(From OE-Core rev: 203336486c84528e8779db93f64cc28e4b372aab)
Signed-off-by: Matthias Schoepfer <matthias.schoepfer@ithinx.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using make -j with the 'install' target, it's possible for altbininstall
(which normally creates BINDIR) and libainstall (which doesn't, though it
installs python-config there) to race, resulting in a failure due to
attempting to install python-config into a nonexistent BINDIR. Ensure it also
exists in the libainstall target.
(From OE-Core rev: 07aeaa4f3f88f575e4674145a7faab3ba8e97fad)
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The newer python3 recipe no longer includes ntpath.py in core, leading
to failures in pkg_resources:
Traceback (most recent call last):
File "/usr/bin/bmaptool", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 42, in <module>
import ntpath
ModuleNotFoundError: No module named 'ntpath'
(From OE-Core rev: 9ff61fa83a0a4f2a7b5b0376b6c48fb1173c9ac7)
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the kernel TCP backlog queue coalescing bug fixed in the 5.0 kernels
we don't need this patch anymore and can run the tests.
(From OE-Core rev: 21bfbef23e1f4c90070017e19f70e488e6ff64be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding back the python wrapper and adding a patch to use OEPYTHON3HOME
instead of PYTHONHOME if set, for python3.
If we add back the wrapper as is, we would see the following error that
we also see in Thud:
ImportError: No module named site
OpenEmbedded requires 'python' to be python v2 (>= 2.7.3), not python
v3.
Please upgrade your python v2
This is because python3 would've set PYTHONHOME to use nativesdk
python3 libraries but when the oe-buildenv-internal script tries to call
python2 for the py_v27_check, there will be no python2 libraries in the
PYTHONHOME directory.
In other words, bitbake needs host python2 and the env variable set from
the wrapper contaminates the env and host python2 won't be able to find
its libraries
Creating another variable OEPYTHON3HOME and using this in the python3
wrapper to allow for a way to set a different paths for python3 and
python2
[YOCTO #13208]
(From OE-Core rev: 75d2a85e24ef9a2bf0e218521944523f0ff281e0)
Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
License-update: copyright years
(From OE-Core rev: a215296bb1b6a87da3692ef0e6176e56310b05f2)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A previous fix to python added -W and removed -v. This reverts that part of the change
since we're no longer getting ptest results at all. This change back to more
verbose output means we start getting test results again.
(From OE-Core rev: d70c77e4d5ec1d7cb4f134377df11e6475f31062)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are python tests which hang with recent kernels, 5.0 onwards. This causes
ptest to timeout for python3. Disable the problematic test until we better understand
the real cause and fix of the issue (discussions are happening with upstream).
See the patch for details/links.
(From OE-Core rev: 2a48df02a2871635f8235645bfd7f7a3ff0aef31)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to run sed with the -u option to ensure the output is unbuffered else
ptest-runner may timeout thinkig things were idle. Busybox doesn't have the -u
option so we need to RDEPEND on sed (which is a good thing to do if we use it
anyway).
Alex Kanavin should get credit for discovering the problem.
(From OE-Core rev: d3ffbebf43c23faa43af81c9ecf6fcaef36d675b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add missing runtime dependencies: the test suite needs proper chmod from
coreutils, and the Europe/Minsk timezone.
Also change run-ptest to use -W (run verbosely on failure) instead of -v (always
verbose) to hopefully reduce the noise in the ptest reports.
(From OE-Core rev: 7574f6850797fb7556d2f8077860b7c1d26ae8ec)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I took the same approach as the recent perl upgrade: write recipe from scratch,
taking the pieces from the old recipe only when they were proven to be necessary.
The pgo, manifest and ptest features are all preserved.
New features:
- native and target recipes are now unified into one recipe
- check_build_completeness.py runs right after do_compile() and verifies that
all optional modules have been built (a notorious source of regressions)
- a new approach to sysconfig.py and distutils/sysconfig.py returning values
appropriate for native or target builds: we copy the configuration file to a
separate folder, add that folder to sys.path (through environment variable
that differs between native and target builds), and point python to the file
through another environment variable.
There were a few other patches where it was difficult to decide if the patch
is still relevant, and how to test that it works correctly; please add those
as-needed by testing the new python.
(From OE-Core rev: 02714c105426b0d687620913c1a7401b386428b6)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that wide-char functions are accssible.
[YOCTO #13142]
(From OE-Core rev: 042187a65d0622fbd721346b4e55146616c87fd8)
Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When working path contains "clang"/"gcc"/"icc", it might be part of $CC
because of the "--sysroot" parameter. That could cause judgement error
about clang/gcc/icc compilers.
eg: if build under /yocto/builds/xicc/, bitbake python, $CC will contains
xicc, will make $CC match *icc, but actuall xicc just folder name.
When "*icc" is matched, below errors are reported when
compiling python/python3:
x86_64-wrs-linux-gcc: error: strict: No such file or directory
x86_64-wrs-linux-gcc: error: unrecognized command line option '-fp-model'
Here use cc_basename to replace CC for checking compiler to avoid such
kind of issue.
(From OE-Core rev: 96383efa2726ed1bf7893332d726112a8552fc24)
Signed-off-by: Li Zhou <li.zhou@windriver.com>
Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building for powerpc 32bit with musl following error triggered
from do_configure:
checking for the platform triplet based on compiler characteristics... powerpc-linux-gnu
configure: error: internal configure error for the platform triplet, please file a bug report
This is caused by PLATFORM_TRIPLET != MULTIARCH mismatch since MULTIARCH
in case of musl is powerpc-linux-musl. Since triplet is used as part
module name as described in PEP-3149 to make fix less intrusive alias
powerpc-linux-musl to powerpc-linux-gnu to avoid possible runtime
(e.g. tests) incompatibilities later.
Fix was inspired by commit cda0ef61d373 ("python3: fix do_configure
check platform triplet error").
(From OE-Core rev: b2ec91e12088afa0560aecede587d0970fc64110)
Signed-off-by: Serhey Popovych <serhe.popovych@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating a new python3 manifest, there is a corner case on which
the filepath for a certain dependency that was found, could contain
the path of an existing folder, e.g. ${libdir}/python3/xmlrpclib.py
module path contains ${libdir}/python3/xml, this causes an issue where
the dependency doesnt get eventually added on FILES for that module.
This patch checks if the dependency that was found is a directory, if it
is, it checks if it matches one of the existing directories on the
manifest, if it is not, then it checks if the dependency's path (without
the filename) matches one of the directories.
(From OE-Core rev: 59db12fdf294cfab5c1730337d092a75867658f7)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch altered the clean target's behaviour to skip the ipkg-install
directory. However this directory isn't created by opkg, opkg-utils, or the
package_ipk class; and we don't invoke the clean target as we perform
out-of-tree builds.
(From OE-Core rev: 9f8bd475701e5d797d3ffc1ba97647101ba0b9b0)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While there is a bit of documentation regarding building a new
manifest file for python, it seems that users usually only read
the manifest file.
The manifest file is in JSON format which doesn't allow comments,
hence why instructions were initially put elsewhere.
This patch hacks the call to open the JSON manifest file by using a
marker to trick it into reading only part of the file as the manifest
itself, and keep the other part as comments, which contain instructions
for the user to run the create_manifest task after an upgrade or when
adding a new package.
(From OE-Core rev: 3eab24c6dc095fd2305b9be8467aab1191141e35)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support to enable tk via PACKAGECONFIG.
before this patch:
# python3
Python 3.5.6 (default, Nov 8 2018, 04:53:45)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.5/tkinter/__init__.py", line 35, in <module>
import _tkinter # If this fails your Python may not be configured for Tk
ImportError: No module named '_tkinter'
>>>
After this patch, if enable tk in PACKAGECONFIG, then
# python3
Python 3.5.6 (default, Nov 8 2018, 03:15:52)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
>>>
(From OE-Core rev: 4b781d545e7e0b084201cd6a8fad953b0f231513)
Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pyvenv is just a small script that uses venv to create virtual
environments.
https://www.python.org/dev/peps/pep-0405/#creating-virtual-environments
This patch adds the python3-venv module as a self-contained package which
python3-pyvenv must depend on at run-time.
The patch also provides the package python3-pyvenv from the pyhton3-venv
package.This is good for future-proofing since python3-pyvenv has been
deprecated and only python3-venv is now available in Python 3.6.
https://docs.python.org/3/library/venv.html.
Without this patch python3-pyvenv is broken because it is missing the
venv module at run-time. This patch specifies the newly created
python3-venv as a run-time dependency of python3-pyvenv.
(From OE-Core rev: effa141bfce55aab25142ee578c95383c755ad73)
Signed-off-by: Hugues Kamba <hugues.kamba@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Copy the Python 2 run-ptest script to execute the Python 3 test suite.
(From OE-Core rev: d371ff04628bdf7bac66341aeb9a66d145f66416)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sqlite3/__init__.py was accidentally included in python3-misc.
(From OE-Core rev: 4e397e16996662bce51608cf5e20448b9803c476)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This package doesn't exist anymore so the manifest tool doesn't need to handle
it specially.
(From OE-Core rev: 30eadb119a64d8561b946d5b8ee30244caeaf134)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the bulk of the tests in python3-tests, some more in
python3-sqlite3-tests, and others in their parent module (such as
python3-ctypes). This is pointless space usage if we're not planning on running
the tests, so consolidate all the tests into python3-tests.
(From OE-Core rev: 554d3690dab49fee3c93b6b677eb3ef58cff64c1)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of sorting the entire manifest when it is updated, use OrderedDict to
preserve the order of fields. This means that packages can be ordered in the
manifest to allow non-trivial FILES assignments (such as a package that picks up
pieces of other packages)
The manifest has been regenerated with the new stable ordering, and
distutils-staticdev moved above distutils so the packaging rules work as
expected.
(From OE-Core rev: 1c67c2146e3644a26367a32885d27a4378f17ac6)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to human error an older revision of the SSL patch was merged.
(From OE-Core rev: 325af0f4a821971a7aeeca35b10e3558f86029e0)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
glibc 2.28 slightly changed the behaviour of sigaddset() which broke
multiprocessing. Backport a patch from Python 3.6 to solve this.
(From OE-Core rev: df1874eda517070dab86270ffafd5fe9a307e76b)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch dates back to the addition of the Python 3 recipe to oe-core, and as
listxattr is never added to supports_follow_symlinks the extended attribute
support will never be enabled.
(From OE-Core rev: 9806e141e85b4e4c38eb7b45e6f2fbc2d2aed29d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
None of the Python invocations that this changes are actually called, and
there's no need to provide a HOSTPGEN variable when the recipe can just override
PGEN directly.
(From OE-Core rev: 47a8602171428b7ce5d897f7e2c2f26b203b8b63)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With OpenSSL 1.1.x TLS 1.3 can be used, so backport a patch from Python 3.6 to
fix the ftplib unit test.
(From OE-Core rev: a31047bec6b7c368674d4620e70e526ac211b936)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
