Commit Graph

2761 Commits

Author SHA1 Message Date
Daniel Díaz
40a817a038 multilib_header: recognize BPF as a target
When building with `clang -target bpf` using the
multilib_header, a recursion was unavoidable because
bits/wordsize.h would #include itself, still lacking
a definition for __MHWORDSIZE or __WORDSIZE.

(From OE-Core rev: 70b41b3c335a80b4ac243f468f22331d261299db)

(From OE-Core rev: 58abe666d3bad7a915c244c61085482e94b3d549)

(From OE-Core rev: edb02f0d1d313fb026e39cc320e63be7e0132e7b)

Signed-off-by: Daniel Díaz <daniel.diaz@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-13 10:55:38 +01:00
Chen Qi
3259b70497 devtool/sdk.py: error out in case of downloading file failure
It's possible that downloading file from updateserver fails. In
this case, we should error out instead of continue.

We have users reporting unexpected behavior of 'devtool sdk-update'.
When an invalid url is supplied, e.g., `devtool sdk-update http://invalid',
the program reports 'Note: Already up-to-date'.

This is obviously not expected. We should error out in such case.

(From OE-Core rev: 449564783dfb162536a2f772b3a8704973221e0f)

(From OE-Core rev: 4b022a62998e38dbefe1f882bcb9a229485ac9da)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15 10:22:46 +01:00
Dogukan Ergun
53d5f01d2d wic: if we can't get from ioctl, try from os.stat()
Under some conditions, ioctl FIGETBSZ can't return real value.
We can try to use fallback via os.stat() to get block size.

Source of patch:
17365f4fe9

(From OE-Core rev: d8f7cf2d38934c248be91101236f7537d0d31ea7)

(From OE-Core rev: 4f528a93352c6301d57a9aaeb082d0cc313d91c3)

Signed-off-by: Dogukan Ergun <dogukan.ergun@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-15 10:22:44 +01:00
Martin Jansa
b0488ff67a scripts/test-dependencies.sh: remove
* with RSS used in pyro this script isn't very useful anymore
* RSS makes sure that the dependencies are almost always deterministic
  the only case known to me where dependencies are different based on
  what was already built in TMPDIR are runtime dependencies resolved
  by shlibs code in package.bbclass (which is using global pkgdata, not
  specific to given recipe and its RSS) as described here:
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=9217#c4
  but for this case it's not worth running complete test-dependencies.sh
  runs

(From OE-Core rev: 522005e722ceb1d1447826e6d7a36d43e49d0450)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-07 15:57:37 +01:00
Anuj Mittal
9e5d96c2ab checklayer: remove reference to undefined class
LayerError doesn't exist and will lead to an error when this failure
code path is hit.

(From OE-Core rev: 7780482772d005c77825dc3e99e63f00911156bf)

(From OE-Core rev: d06cf17109a2ca310c270ce4a27a15cb6190e2ff)

Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-26 15:07:12 +01:00
Markus Lehtonen
3fae046acf scripts/oe-build-perf-report: fix comparing arbitrary commits
Fix a crash when generating a txt report and the two commits to be
compared were not consecutive (but there were some tested commits
between them).

(From OE-Core rev: f3afd2c47f4c740df52dfd80e208ce721d5ebf6e)

(From OE-Core rev: 00b197b2fb2253f2c5c534b16cc2d3d55aad6352)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-26 15:07:11 +01:00
Richard Purdie
92e9b46f3c runqemu: Add workaround for APIC hang on pre 4.15 kernels on qemux86
On pre 4.15 host kernels, an APIC window emulation bug can cause qemu
to hang. On 64 bit we can use the x2apic, for 32 bit, we just have to
disable the other timer sources and rely on kvm-clock.

[YOCTO #12301]

(From OE-Core rev: 04646ed7edf020b73c6639a1a7445252cf7aad5e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 82e67b82ea8e12aa0b7b9db1d84fec0436dec71b)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-06 10:13:57 +00:00
Adrian Fiergolski
e5d7b7d802 wic: Fix a path to a psuedo state directory (PSEUDO_LOCALSTATEDIR).
In case of 'new_rootfs' the psuedo directory is not copied. Thus
PSEUDO_LOCALSTATEDIR should still point to the dsa
'native_sysroot'/../pseudo. Otherwise PSEUDO_LOCALSTATEDIR points to a not
existing director ('new_rootfs'/../pseudo) and UID and GUID attributes are not
applied to files of the image.

(From OE-Core rev: 4c1741ad08c809c83cd9670b14850687b598e728)

Signed-off-by: Adrian Fiergolski <adrian.fiergolski@cern.ch>
Signed-off-by: Ross Burton <ross.burton@intel.com>
(cherry picked from commit 19642e2d6e015072e4a413f4f57aee65df757cb9)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-06 10:13:57 +00:00
Chang Rebecca Swee Fun
b443c17416 recipetool: create: fix conflict between SRCREV and tag
If you specify 'tag=' for a git URL and passed to recipetool create, you
will get into Bitbake expansion error shown below:

----- snip -----
$ devtool add --version 2.4.2 mbedtls "git://github.com/ARMmbed/mbedtls;tag=mbedtls-2.4.2"
...
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Conflicting revisions (abeccb9dbd7e19ae91ac50e1edd3803111c5f9b6 from SRCREV and mbedtls-2.4.2 from the url) found, please specify one valid value
----- snip -----

Assuming the tag is valid, we should get the tag commit hash and
drop the usage of 'tag=' from SRC_URI. By using a commit hash
corresponding to the tag will prevent bitbake from accessing
remote repository in order to expand SRCPV.

(From OE-Core rev: 4953d54dd06ff7025406d2df6a71c95162cc1ec7)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
(cherry picked from commit 53f8effa3eb07dc7035ff9933e7918318f242579)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-06 10:13:56 +00:00
Martin Hundebøll
aa5127d279 wic: support filesystem label for rawcopy
The '--label' argument should work for '--source rawcopy' as it does for
'--source rootfs', so add a method in RawCopyPlugin to update the label
on the temporary filesystem images.

(From OE-Core rev: 913b31985b2fc8b7e17bc29d668168ab25ff989e)

Signed-off-by: Martin Hundebøll <mnhu@prevas.dk>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-06 10:13:55 +00:00
Robert Yang
7de56ebc2a runqemu: print command search result when not found
This makes debug easier.

(From OE-Core rev: b99ba567cd8089a9a3ca01704f6ba6c42d390e9f)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
(cherry picked from commit a453639e19fb2a9f9fb63fddd0b3ee26c0116d91)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:24:00 +00:00
Richard Purdie
d0640da88e runqemu: Also specialcase resolution of '.' to the file's location
Similarly to handling "../", handle "." to resovle to the qemuconf
file's current directory.

(From OE-Core rev: 9870247d0dc33357988d9636c8ff8db35490752e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 33418ed064fe9cff5b4803f09135a81d9170c189)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:24:00 +00:00
Richard Purdie
968145b24e runqemu: Improve relative path handling in qemuconf files
If a variable starts with "../", its likely its a path and we want to
set it to an absolute path relative to the qemuconf file.

This means we don't have to use bitbake as often to figure out variables.

(From OE-Core rev: 61c449857f056d7c6c29530aa11bf8353b113638)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dfc7940900d798aa47716288338107e1d46a3972)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:59 +00:00
Richard Purdie
65d09a7d1e runqemu: Ensure we process all tap devices
The regexp in the script misses some tap devices, e.g. we see output like:

runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap25.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap26.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap27.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap28.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap40.lock failed: [Errno 11] Resource temporarily unavailable
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap41.lock failed: [Errno 11] Resource temporarily unavailable

What happened to tap29 to tap39?

The issue is was we were missing devices with '0' in the number,
like "10:" and so on in the output from "ip link".

(From OE-Core rev: ec1481f7ad6f2b3d1420027327510bec94dd66a8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6447697a48e3b693ee38806bc2ba07c2a65c2bc8)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:59 +00:00
Paul Eggleton
2d9aecf044 recipetool: create: fix failure handling included dicts
If a setup dict in a python setup.py file pulled in the contents of
another dict (e.g.  **otherdict), then we got an error when mapping
the keys because the key is None in that case. Skip those keys to avoid
the error (we pick up the values directly in any case).

A quick reproducer for this issue:

recipetool create https://files.pythonhosted.org/packages/source/p/pyqtgraph/pyqtgraph-0.10.0.tar.gz

(From OE-Core rev: 49b2d571da88fb2afce71835276523ed3538d31f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
(cherry picked from commit ae62a9953e219df5147ed4a5ae3f4163d51cff28)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:59 +00:00
Paul Eggleton
e36cf9e621 recipetool: ignore incidental kernel module source
If the source tree happens to contain a kernel module as an example, a
test or under a "contrib" directory then we shouldn't be picking it up
and making the determination that the entire thing is a kernel module.

An example that triggered this is zstd, which ships a kernel module
under contrib/linux-kernel:

  https://github.com/facebook/zstd

(From OE-Core rev: 5c89bd0db1b327483f674802740ff21b909e0876)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:58 +00:00
Paul Eggleton
611e4b43d8 recipetool: pass absolute source tree path to plugins
We shouldn't be passing a relative path to the plugins if that's what's
been specified on the recipetool command line.

(From OE-Core rev: 821742f48723a66fdafe5406bb57188b2f88889a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:58 +00:00
Markus Lehtonen
852c71956b oe-build-perf-report-email.py: add images as MIME objects
Add images as separate MIME objects instead of directly embedding images
in the html (as base64 encoded pngs). This makes the emails better
suited for certain email servers/clients.

(From OE-Core rev: fbbc84d9919d9cc18add03fc617637330721f5d9)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:57 +00:00
Tom Rini
de78322f16 wic: Update canned-wks for systemd to use UUID everywhere
With systemd, the mounting of the swap partition is handled via systemd
and will mount it, regardless of if PARTUUID is parsed or not.  systemd
has a runtime dependency on util-linux-mount so PARTUUID for regular
mount points will be handled correctly.  Make all partitions that we add
to the image make use of UUIDs for maximum portability.

(From OE-Core rev: 8bf0e3ee85b22fdd4d8940878b4d99cccff1efd5)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:57 +00:00
Tom Rini
61f319db67 wic: When using --use-uuid make sure that we update the fstab with PARTUUID
When we have been told to use the UUID we should also update the fstab
to make use of PARTUUID instead of hard-coding the device in question.
This will make the resulting image much more portable.

(From OE-Core rev: 1d1fdcaf8702110783f2003cd3f8ae96c99a6d72)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:57 +00:00
Saul Wold
d40531211c wic: misc.py: Use mmd from mtools instead of syslinux
mtools already provides a suite of msdos utilities, switch to this
one also.  This could allow for future changes to reduce wic's
dependecies.

(From OE-Core rev: 493bbd9ae773d0713db9782b434ce9543e2266f3)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:56 +00:00
Thomas Perrot
8754f4779c runqemu: correct rootfs setup to boot an ide hddimg
vm_drive variable is malformed when the drive type is an ide device.

(From OE-Core rev: 02dbf124328eebdfdf62402588a41719953a22bf)

Signed-off-by: Thomas Perrot <thomas.perrot@tupi.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 88d7b17871fe8340ab7fd5c901d3a535ae098c3e)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:55 +00:00
Saul Wold
f483bdf9c4 wic: misc.py: Added more mtools binaries
This fixes the issue that if you don't have mtools installed on the host
thus causing host contamination, that the correct binaries would be selected
from the native sysroot.

[YOCTO #12173]

(From OE-Core rev: 9562669a4979bb31bbc27dc80c6a8d4f08500a49)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
(cherry picked from commit dca43c557449d3765fec9f8d159d5c9e4ea8b0cb)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:55 +00:00
Richard Purdie
87f8aafd53 oe-buildenv-internal: Fix finding build directory
The intent of the env setup scripts is to set BBPATH to point at the
build directory. This means if the user changes directory, bitbake can
still find the original build directory. The default bblayers.conf files
reset BBPATH to the correct components so this is safe and restores the
behaviour the script was intended to have.

[YOCTO #12163]

(From OE-Core rev: bfacf88f15a27db579d8790d92f8497d832961f8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 82eeb934997c9eaa6443079dfb649a89872a222c)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:54 +00:00
Artur Mądrzak
6709452171 wic: add 'part-name' argument for naming GPT partitions
The WIC's 'part' can now give a name for GPT partition in WKS file.
It's similar to '--label', but is naming partintions instead file systems.
It's required by some bootloaders to partitions have specified names.

Backport from master, without it WIC cannot be used on Qualcomm based machines.

(From OE-Core rev: 45aee3d57697f8dcc967120b5afd280d5ceadd21)

Signed-off-by: Artur Mądrzak <artur@madrzak.eu>
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9b60e3466ed7cff0cea10815851eb1304002eb52)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-04 17:23:54 +00:00
Peter Kjellerstedt
ede79dea7c devtool: standard: Expand SRCREV before using it in _update_recipe_srcrev
If SRCREV contains a variable reference, any devtool command that
would try to update it would fail. E.g., if SRCREV = "R${PV}", then
devtool finish without having committed any changes would fail with:

  oe.patch.CmdError: Command Error: 'sh -c 'git format-patch R${PV} -o
  /tmp/oepatchb_doareb -- .'' exited with 0 Output:
  fatal: bad revision 'R'

(From OE-Core rev: 094499c819722ad698ccb64ec65dd439b211c31c)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-10-16 23:52:44 +01:00
Ross Burton
6172b0131f oe-pkgdata-util: add unescape option to read-value
Some fields are multiline values which have been escaped, so add an option to
unescape the \n and \t.

(From OE-Core rev: 4165ec0057c6bbb24de681572034262351d9b34f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-10-07 23:20:40 +01:00
Fabio Berton
5c98f20f5c wic: Add option to not change fstab
Create an option to wic doesn't change fstab file, the final
fstab file will be same that in rootfs and wic doesn't update
file, e.g adding a new mount point.

Users can control the fstab file content in base-files recipe.
This is useful if you want to only create an partition but not
add fstab mount point or add new mount point using label e.g:

LABEL=recovery /recovery auto defaults 0  1

(From OE-Core rev: 00420ec42140c1b752132bda190dede85756d157)

Signed-off-by: Fabio Berton <fabio.berton@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-26 11:05:01 +01:00
Ming Liu
5c67faff20 scripts: drop True option to getVar calls
Search made with the following regex: getVar ?\((.*), True\).

(From OE-Core rev: b848c3cb495905605283c57c79f2ed8ca17758db)

Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-26 11:05:01 +01:00
Enrico Scholz
edcf39820f wic: allow multiple /boot partitions with different content
It can be useful to have multiple partitions with '--source bootimg-partition'
but different content.

E.g. for TI AM335x, one boot partition can contain an first stage
bootloader ("MLO"), while the real bootloader and kernel plus devicetree
are in another one.

Patch allows to specify multiple IMAGE_BOOT_FILES with optional "_label-XXX"
or "_uuid-XXX" overrides.

E.g. with this patch, a .wks file with

| part --source bootimg-partition ... --label=mlo --active
| part --source bootimg-partition ... --label=boot0
| part --source bootimg-partition ... --label=boot1

and a recipe with

| IMAGE_BOOT_FILES_label-mlo = "\
|   MLO-${MACHINE}.img;MLO \
| "
|
| IMAGE_BOOT_FILES_label-boot0 = "\
|   u-boot-${MACHINE}.img;u-boot.img \
|   zImage \
| "
|
| IMAGE_BOOT_FILES_label-boot1 = "${IMAGE_BOOT_FILES_label-boot0}"
|
| WICVARS += " \
|   IMAGE_BOOT_FILES_label-mlo \
|   IMAGE_BOOT_FILES_label-boot0 \
|   IMAGE_BOOT_FILES_label-boot1 \
| "

is possible.  It will create one partition with the MLO and two redundant
ones with the uboot + kernel.

(From OE-Core rev: 8c1dec627e9735260516fe8f0b2bfdb0ee70172b)

Signed-off-by: Enrico Scholz <enrico.scholz@ensc.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-25 14:14:16 +01:00
Joshua Lock
eac8a5cf42 oe-build-perf-report-email.py: add cc and bcc options
Enable carbon copy and blind carbon copy recipients for the performance
report emails.

(From OE-Core rev: df5ae8143ff1764b6ed5973ed3d6f1a83ecf45ee)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-21 23:16:55 +01:00
Paul Eggleton
b32174e58e scripts: rename yocto-compat-layer to remove "compatible" nomenclature
"Yocto Project Compatible" [1] is a programme which requires you meet
specific criteria including going through an application process - it is
not sufficient simply to run the script we have created here and have it
produce no warnings/errors. To avoid people being confused by the fact
that this script uses the term "compatible" or variations thereof,
substitute usage of that word with "check" instead. The functionality of
the script is unchanged.

[1] https://www.yoctoproject.org/ecosystem/yocto-project-branding-program

(From OE-Core rev: 2a6126a115f10750ea89f95629d3699ad41c5665)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-21 11:34:19 +01:00
Markus Lehtonen
6c222a5c11 scripts/oe-build-perf-report: show recipe version changes in html report
If buildstats are available (for a certain measurement), show recipe
version changes between the two builds that are being compared. The
information shown includes new and dropped recipes as well as changes in
recipe version, revision or epoch.

[YOCTO #11382]

(From OE-Core rev: 46eb839b51bb1466a9feeb09c9c437d6d45576cc)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
a80f5e761c scripts/buildstats-diff: move more code to lib/buildstats.py
More refactoring of buildstats-diff script. Move recipe version
comparison functionality to scripts/lib/buildstats.py. This patch also
compasses some wording changes, i.e. changing 'package' to 'recipe'.

[YOCTO #11382]

(From OE-Core rev: 2f8942d6830258fcbe1925f12ba1516def32d132)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
81aef784fd scripts/oe-build-perf-report: summary of task resource usage
Utilize buildstats, if available, and show a summary of the resource
usage of bitbake tasks in the html report. The details provided are:
- total number of tasks
- top 5 resource-hungry tasks (cputime)
- top 5 increase in resource usage (cputime)
- top 5 decrease in resource usage (cputime)

[YOCTO #11381]

(From OE-Core rev: ddd9443cb2432af2c15b358bfda708393fa3c417)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
b5fb3dd904 scripts/buildstats-diff: move code to lib/buildstats.py
Move over code from buildstats-diff to new scripts/lib/buildstats.py
module in order to share code related to buildstats processing.  Also,
refactor the code, introducing new classes to make the code readable,
maintainable and easier to debug.

[YOCTO #11381]

(From OE-Core rev: 8a2cd9afc95919737d8e75234e78bbc52e1494a1)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
873707489f scripts/oe-build-perf-report: tidy up html syntax
Fix some problems in the html syntax of the generated report:
- prevent empty rows in the summary table
- add one missing column in the results table

(From OE-Core rev: 10883bb49ad2f5309883fd352cf320b2e1648615)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
eaeb698129 scripts/oe-build-perf-report: provide valid title in the html report
(From OE-Core rev: 46312fde635a717e41d12e7ef01940838316ae17)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
93139fb66e scripts/oe-build-perf-report: remove dead code
Some leftover from an early prototype.

(From OE-Core rev: a28284fc1bbaa18162f65eee0ae2e94d28091b2f)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
0821f3b0f8 scripts/oe-build-perf-report: use --hostname in --list
Makes it possible to list test results for certain host only, instead of
always listing all results from all hosts.

(From OE-Core rev: 3c07f1f05440234243c570ebfb42dcda2f455a3d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
a7461ad2a2 scripts/oe-build-perf-report: fix handling of --history-length
Don't crash if 'left' revision is older than the range of commits
specified with '--history-length'. In this case the 'left' revision
takes precedence.

(From OE-Core rev: cbeb1fb27329f8eba4d779d22fcad56f0fb03947)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
062bdb044c scripts/oe-build-perf-report: add AggregateTestData class
Making the code a bit more readable.

(From OE-Core rev: 25351c7cac167b1a3e8b531e2cdf708192c6fa1f)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Markus Lehtonen
ec5a5f28e2 scripts/oe-build-perf-report: fix dumping buildstats
Fix a misbehavior when some of the buildstats are missing.

(From OE-Core rev: 2930515acb5cfdcd335a76ae36fd9f7189207aa3)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:30 +01:00
Paul Eggleton
3fde63363a devtool: ensure recipes devtool is working on are unlocked within the eSDK
Alongside reworking the way devtool extracts source, we now need to
ensure that within the extensible SDK where task signatures are locked,
the signatures of the tasks for the recipes being worked on get unlocked
at the right time or otherwise we'll now get taskhash mismatches when
running devtool modify on a recipe that was included in the eSDK such as
the kernel (due to a separate bug). The existing mechanism for
auto-unlocking recipes was a little weak and was happening too late, so
I've reimplemented it so that:
(a) it gets triggered immediately when the recipe/append is created
(b) we avoid writing to the unlocked signatures file unnecessarily
    (since it's a global configuration file) and
(c) within the eSDK configuration we whitelist SIGGEN_UNLOCKED_RECIPES
    to avoid unnecessary reparses every time we perform one of the
    devtool operations that does need to change this list.

Fixes [YOCTO #11883] (not the underlying cause, but this manifestation
of the issue).

(From OE-Core rev: 4e9a0be32fc30fb87d65da7cd1a4015c99533aff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:29 +01:00
Paul Eggleton
10af6d86b3 devtool: rework source extraction so that dependencies are handled
Since it was first implemented, devtool's source extraction (as used by
the devtool modify, extract and upgrade subcommands) ignored other recipe
dependencies - so for example if you ran devtool modify on a recipe that
fetches from svn or is compressed using xz then it would fail if those
dependencies hadn't been built first. Now that we can execute tasks in
the normal way (i.e. tinfoil.build_targets()) then we can rework it to
use that. This is slightly tricky in that the source extraction needs to
insert some logic in between tasks; luckily we can use a helper class
that conditionally adds prefuncs to make that possible.

Some side-effects / aspects of this change worth noting:
* Operations are a little slower because we have to go through the task
  dependency graph generation and other startup processing. There's not
  really any way to avoid this though.
* devtool extract didn't used to require a workspace, now it does
  because it needs to create a temporary bbappend for the recipe. (As
  with other commands the workspace be created on the fly if it doesn't
  already exist.)
* I want any existing sysroot files and stamps to be left alone during
  extraction since we are running the tasks off to the side, and
  especially devtool extract should be able to be used without touching
  these. However, this was hampered by the automatic removal process in
  sstate.bbclass triggered by bb.event.ReachableStamps when the task
  signatures change, thus I had to introduce a way to disable this
  removal on a per-recipe basis (we still want it to function for any
  dependencies that we aren't working on). To implement this I elected
  to use a file written to tmp/sstate-control which gets deleted
  automatically after reading so that there's less chance of stale files
  affecting future sessions. I could have used a variable but this would
  have needed to be whitelisted and I'd have to have poked its value in
  using the setVariable command.

Fixes [YOCTO #11198].

(From OE-Core rev: 830dbd66992cbb9e731b48d56fddf8f220349666)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:29 +01:00
Enrico Scholz
7324bc16a2 wic: accept '-' in bitbake variables
'-' is valid and common in bitbake variables (e.g. 'FOO_pn-bar'). Accept
it and other characters when reading the .env file.

Also, allow variables to be empty.

(From OE-Core rev: e688ac8e92d2bc451d8b2d437596f630bedccd2c)

(From OE-Core rev: 2a69250abf61e51f633033ddb672e8f459191899)

Signed-off-by: Enrico Scholz <enrico.scholz@ensc.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-18 11:07:29 +01:00
Juan M Cruz Alcaraz
688e0894aa devtool/standard: set a preferred provider when adding a new recipe with devtool
A recipe added with "devtool add" requires to be able to take precedence on recipes
previously defined with PREFERRED_PROVIDER.

By adding the parameter "--provides" to "devtool add" it is possible to specify
an element to be provided by the recipe. A devtool recipe can override a previous
PREFERRED_PROVIDER using the layer configuration file in the workspace.

E.g.
    devtool add my-libgl git@git://my-libgl-repository --provides virtual/libgl

[YOCTO #10415]

(From OE-Core rev: adeea2fe6895898a5e6006e798898f0f5dabd890)

Signed-off-by: Juan M Cruz Alcaraz <juan.m.cruz.alcaraz@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-13 22:07:42 +01:00
Paul Eggleton
6426136655 devtool: upgrade: check that user has configured git properly
If user.name or user.email haven't been set then git rebase can't really
work properly. Check that the user has set these and error out if not.
(Elsewhere we are relying on OE's git patch functionality which forces
a dummy OE value - that's OK there as it's completely under OE's control
and therefore it's OK for a dummy OE user to be the committer, but here
the rebase may require intervention so it's reasonable to have the
user's actual name and email on the operation.)

Fixes [YOCTO #11947].

(From OE-Core rev: 129a3be07e272013be2db17552c13b4d8cc2cf6e)

(From OE-Core rev: 802829f1c38d8c5eee11ba1d9ddd37cf02597f6e)

Signed-off-by: paul <paul@peggleto-mobl.ger.corp.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-11 17:30:30 +01:00
Daniela Plascencia
c72af57827 scripts/buildhistory-diff: use of argparse instead of optparse
Optparse is deprecated since version 2.7 and won't be developed further.
Argparse should be used instead as it provides better tools for parsing
and handling arguments.

[YOCTO #9635]

(From OE-Core rev: e67b40c01fd98048035ca18595d87ae1be050ab4)

Signed-off-by: Daniela Plascencia <daniela.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-11 17:30:30 +01:00
Ed Bartosh
cef80d0dfe wic: run bmaptool with native Python3
Modified wic code to run bmaptool using native Python3
from wic-tools native sysroot.

[YOCTO #11891]

(From OE-Core rev: 7fca44e03130c0860cc5df2093902773f426c774)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-11 17:30:29 +01:00