All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass
@ 2017-09-11 21:18 Otavio Salvador
  2017-09-11 21:18 ` [PATCH v7 2/2] kernel-devicetree.bbclass: Add support to generate append to kernel Otavio Salvador
  2017-09-12  8:34 ` [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass Richard Purdie
  0 siblings, 2 replies; 4+ messages in thread
From: Otavio Salvador @ 2017-09-11 21:18 UTC (permalink / raw)
  To: OpenEmbedded Core Mailing List; +Cc: Otavio Salvador

The Device Tree is commonly used but it is still kept as a .inc file
instead of a proper class. Instead now we move the Device Tree code to
a kernel-devicetree class and automatically enable it when the
KERNEL_DEVICETREE variable is set.

While converting to the class, we reworked the compile and install in
tasks as well as run the build of the Device Tree files in parallel,
if possible.

To avoid breakage in existing layers, we kept a linux-dtb.inc file
which raises a warning telling the user about the change so in next
release this can be removed.

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---

Changes in v7: None
Changes in v6:
- Remove linux-dtb.inc from linux-yocto recipes
- Always inherit kernel-devicetree class

Changes in v5: None
Changes in v4:
- Rebase on top of 1/3

Changes in v3:
- rework compile and install to tasks
- improve commit log

Changes in v2:
- Wrong version as the changes were not staged. Sorry!

 meta/classes/kernel-devicetree.bbclass    | 68 +++++++++++++++++++++++++++++++
 meta/classes/kernel.bbclass               |  3 ++
 meta/recipes-kernel/linux/linux-dtb.inc   | 66 +-----------------------------
 meta/recipes-kernel/linux/linux-yocto.inc |  1 -
 4 files changed, 73 insertions(+), 65 deletions(-)
 create mode 100644 meta/classes/kernel-devicetree.bbclass

diff --git a/meta/classes/kernel-devicetree.bbclass b/meta/classes/kernel-devicetree.bbclass
new file mode 100644
index 0000000000..04e2f2695e
--- /dev/null
+++ b/meta/classes/kernel-devicetree.bbclass
@@ -0,0 +1,68 @@
+# Support for device tree generation
+PACKAGES_append = " kernel-devicetree"
+FILES_kernel-devicetree = "/${KERNEL_IMAGEDEST}/*.dtb /${KERNEL_IMAGEDEST}/*.dtbo"
+
+normalize_dtb () {
+	DTB="$1"
+	if echo ${DTB} | grep -q '/dts/'; then
+		bbwarn "${DTB} contains the full path to the the dts file, but only the dtb name should be used."
+		DTB=`basename ${DTB} | sed 's,\.dts$,.dtb,g'`
+	fi
+	echo "${DTB}"
+}
+
+get_real_dtb_path_in_kernel () {
+	DTB="$1"
+	DTB_PATH="${B}/arch/${ARCH}/boot/dts/${DTB}"
+	if [ ! -e "${DTB_PATH}" ]; then
+		DTB_PATH="${B}/arch/${ARCH}/boot/${DTB}"
+	fi
+	echo "${DTB_PATH}"
+}
+
+do_compile_devicetree() {
+    DTBS=""
+	for dtb in ${KERNEL_DEVICETREE}; do
+		dtb=`normalize_dtb "${dtb}"`
+		DTBS="$DTBS $dtb"
+	done
+	oe_runmake -C ${B} ${PARALLEL_MAKE} ${DTBS}
+}
+addtask compile_devicetree after do_compile before do_install
+
+do_install_devicetree[dirs] = "${B}"
+do_install_devicetree[umask] = "022"
+fakeroot do_install_devicetree() {
+	for DTB in ${KERNEL_DEVICETREE}; do
+		DTB=`normalize_dtb "${DTB}"`
+		DTB_EXT=${DTB##*.}
+		DTB_PATH=`get_real_dtb_path_in_kernel "${DTB}"`
+		DTB_BASE_NAME=`basename ${DTB} ."${DTB_EXT}"`
+		install -m 0644 ${DTB_PATH} ${D}/${KERNEL_IMAGEDEST}/${DTB_BASE_NAME}.${DTB_EXT}
+		for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do
+			symlink_name=${type}"-"${KERNEL_IMAGE_SYMLINK_NAME}
+			DTB_SYMLINK_NAME=`echo ${symlink_name} | sed "s/${MACHINE}/${DTB_BASE_NAME}/g"`
+			ln -sf ${DTB_BASE_NAME}.${DTB_EXT} ${D}/${KERNEL_IMAGEDEST}/devicetree-${DTB_SYMLINK_NAME}.${DTB_EXT}
+		done
+	done
+}
+addtask install_devicetree after do_install before do_deploy
+
+do_deploy_append() {
+	for DTB in ${KERNEL_DEVICETREE}; do
+		DTB=`normalize_dtb "${DTB}"`
+		DTB_EXT=${DTB##*.}
+		DTB_BASE_NAME=`basename ${DTB} ."${DTB_EXT}"`
+		for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do
+			base_name=${type}"-"${KERNEL_IMAGE_BASE_NAME}
+			symlink_name=${type}"-"${KERNEL_IMAGE_SYMLINK_NAME}
+			DTB_NAME=`echo ${base_name} | sed "s/${MACHINE}/${DTB_BASE_NAME}/g"`
+			DTB_SYMLINK_NAME=`echo ${symlink_name} | sed "s/${MACHINE}/${DTB_BASE_NAME}/g"`
+			DTB_PATH=`get_real_dtb_path_in_kernel "${DTB}"`
+			install -d ${DEPLOYDIR}
+			install -m 0644 ${DTB_PATH} ${DEPLOYDIR}/${DTB_NAME}.${DTB_EXT}
+			ln -sf ${DTB_NAME}.${DTB_EXT} ${DEPLOYDIR}/${DTB_SYMLINK_NAME}.${DTB_EXT}
+			ln -sf ${DTB_NAME}.${DTB_EXT} ${DEPLOYDIR}/${DTB_BASE_NAME}.${DTB_EXT}
+		done
+	done
+}
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 02a5e961cb..0ad522d167 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -662,3 +662,6 @@ do_deploy[prefuncs] += "package_get_auto_pr"
 addtask deploy after do_populate_sysroot do_packagedata
 
 EXPORT_FUNCTIONS do_deploy
+
+# Add using Device Tree support
+inherit kernel-devicetree
diff --git a/meta/recipes-kernel/linux/linux-dtb.inc b/meta/recipes-kernel/linux/linux-dtb.inc
index ca92822d25..f1912775ca 100644
--- a/meta/recipes-kernel/linux/linux-dtb.inc
+++ b/meta/recipes-kernel/linux/linux-dtb.inc
@@ -1,65 +1,3 @@
-# Support for device tree generation
-FILES_kernel-devicetree = "/${KERNEL_IMAGEDEST}/*.dtb /${KERNEL_IMAGEDEST}/*.dtbo"
-
-python __anonymous () {
-    d.appendVar("PACKAGES", " kernel-devicetree")
-}
-
-normalize_dtb () {
-	DTB="$1"
-	if echo ${DTB} | grep -q '/dts/'; then
-		bbwarn "${DTB} contains the full path to the the dts file, but only the dtb name should be used."
-		DTB=`basename ${DTB} | sed 's,\.dts$,.dtb,g'`
-	fi
-	echo "${DTB}"
-}
-
-get_real_dtb_path_in_kernel () {
-	DTB="$1"
-	DTB_PATH="${B}/arch/${ARCH}/boot/dts/${DTB}"
-	if [ ! -e "${DTB_PATH}" ]; then
-		DTB_PATH="${B}/arch/${ARCH}/boot/${DTB}"
-	fi
-	echo "${DTB_PATH}"
-}
-
-do_compile_append() {
-	for DTB in ${KERNEL_DEVICETREE}; do
-		DTB=`normalize_dtb "${DTB}"`
-		oe_runmake ${DTB}
-	done
-}
-
-do_install_append() {
-	for DTB in ${KERNEL_DEVICETREE}; do
-		DTB=`normalize_dtb "${DTB}"`
-		DTB_EXT=${DTB##*.}
-		DTB_PATH=`get_real_dtb_path_in_kernel "${DTB}"`
-		DTB_BASE_NAME=`basename ${DTB} ."${DTB_EXT}"`
-		install -m 0644 ${DTB_PATH} ${D}/${KERNEL_IMAGEDEST}/${DTB_BASE_NAME}.${DTB_EXT}
-		for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do
-			symlink_name=${type}"-"${KERNEL_IMAGE_SYMLINK_NAME}
-			DTB_SYMLINK_NAME=`echo ${symlink_name} | sed "s/${MACHINE}/${DTB_BASE_NAME}/g"`
-			ln -sf ${DTB_BASE_NAME}.${DTB_EXT} ${D}/${KERNEL_IMAGEDEST}/devicetree-${DTB_SYMLINK_NAME}.${DTB_EXT}
-		done
-	done
-}
-
-do_deploy_append() {
-	for DTB in ${KERNEL_DEVICETREE}; do
-		DTB=`normalize_dtb "${DTB}"`
-		DTB_EXT=${DTB##*.}
-		DTB_BASE_NAME=`basename ${DTB} ."${DTB_EXT}"`
-		for type in ${KERNEL_IMAGETYPE_FOR_MAKE}; do
-			base_name=${type}"-"${KERNEL_IMAGE_BASE_NAME}
-			symlink_name=${type}"-"${KERNEL_IMAGE_SYMLINK_NAME}
-			DTB_NAME=`echo ${base_name} | sed "s/${MACHINE}/${DTB_BASE_NAME}/g"`
-			DTB_SYMLINK_NAME=`echo ${symlink_name} | sed "s/${MACHINE}/${DTB_BASE_NAME}/g"`
-			DTB_PATH=`get_real_dtb_path_in_kernel "${DTB}"`
-			install -d ${DEPLOYDIR}
-			install -m 0644 ${DTB_PATH} ${DEPLOYDIR}/${DTB_NAME}.${DTB_EXT}
-			ln -sf ${DTB_NAME}.${DTB_EXT} ${DEPLOYDIR}/${DTB_SYMLINK_NAME}.${DTB_EXT}
-			ln -sf ${DTB_NAME}.${DTB_EXT} ${DEPLOYDIR}/${DTB_BASE_NAME}.${DTB_EXT}
-		done
-	done
+python() {
+    bb.warn("You are using the linux-dtb.inc which is deprecated. You can safely remove it as the Device Tree support is automatically enabled when KERNEL_DEVICETREE is set.")
 }
diff --git a/meta/recipes-kernel/linux/linux-yocto.inc b/meta/recipes-kernel/linux/linux-yocto.inc
index 1b8ffd0c17..9c1f61be75 100644
--- a/meta/recipes-kernel/linux/linux-yocto.inc
+++ b/meta/recipes-kernel/linux/linux-yocto.inc
@@ -54,7 +54,6 @@ LINUX_VERSION_EXTENSION ??= "-yocto-${LINUX_KERNEL_TYPE}"
 # Pick up shared functions
 inherit kernel
 inherit kernel-yocto
-require linux-dtb.inc
 
 B = "${WORKDIR}/linux-${PACKAGE_ARCH}-${LINUX_KERNEL_TYPE}-build"
 
-- 
2.14.1



^ permalink raw reply related	[flat|nested] 4+ messages in thread
* [PATCH v7 2/2] kernel-devicetree.bbclass: Add support to generate append to kernel
  2017-09-11 21:18 [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass Otavio Salvador
@ 2017-09-11 21:18 ` Otavio Salvador
  2017-09-12  8:34 ` [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass Richard Purdie
  1 sibling, 0 replies; 4+ messages in thread
From: Otavio Salvador @ 2017-09-11 21:18 UTC (permalink / raw)
  To: OpenEmbedded Core Mailing List; +Cc: Otavio Salvador

The are use cases where the Device Tree appended to the kernel is
convinient, so we generate the bundle concatenating the kernel (and
potentionally the initramfs) and the Device Tree binaries.

To enable it, set KERNEL_DEVICETREE_BUNDLE variable to '1'

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---

Changes in v7:
- simplified code
- rename bundle to use .bin extension

Changes in v6: None
Changes in v5:
- add support for initramfs bundle

Changes in v4:
- new patch

Changes in v3: None
Changes in v2: None

 meta/classes/kernel-devicetree.bbclass | 48 +++++++++++++++++++++++++++++++++-
 1 file changed, 47 insertions(+), 1 deletion(-)

diff --git a/meta/classes/kernel-devicetree.bbclass b/meta/classes/kernel-devicetree.bbclass
index 04e2f2695e..50c2055b46 100644
--- a/meta/classes/kernel-devicetree.bbclass
+++ b/meta/classes/kernel-devicetree.bbclass
@@ -1,6 +1,13 @@
 # Support for device tree generation
-PACKAGES_append = " kernel-devicetree"
+PACKAGES_append = " \
+    kernel-devicetree \
+    ${@['kernel-image-zimage-bundle', ''][d.getVar('KERNEL_DEVICETREE_BUNDLE') != '1']} \
+"
 FILES_kernel-devicetree = "/${KERNEL_IMAGEDEST}/*.dtb /${KERNEL_IMAGEDEST}/*.dtbo"
+FILES_kernel-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"
+
+# Generate kernel+devicetree bundle
+KERNEL_DEVICETREE_BUNDLE ?= "0"
 
 normalize_dtb () {
 	DTB="$1"
@@ -20,6 +27,24 @@ get_real_dtb_path_in_kernel () {
 	echo "${DTB_PATH}"
 }
 
+
+do_configure_devicetree() {
+	if [ "${KERNEL_DEVICETREE_BUNDLE}" = "1" ]; then
+		if echo ${KERNEL_IMAGETYPE_FOR_MAKE} | grep -q 'zImage'; then
+			config="${B}/.config"
+			if ! grep -q 'CONFIG_ARM_APPENDED_DTB=y' $config; then
+				bbwarn 'CONFIG_ARM_APPENDED_DTB is NOT enabled in the kernel. Enabling it to allow the kernel to boot with the Device Tree appended!'
+				sed -i "/CONFIG_ARM_APPENDED_DTB[ =]/d" $config
+				echo "CONFIG_ARM_APPENDED_DTB=y" >> $config
+				echo "# CONFIG_ARM_ATAG_DTB_COMPAT is not set" >> $config
+			fi
+		else
+			bberror 'The KERNEL_DEVICETREE_BUNDLE requires the KERNEL_IMAGETYPE to contain zImage.'
+		fi
+	fi
+}
+addtask configure_devicetree after do_configure before do_compile
+
 do_compile_devicetree() {
     DTBS=""
 	for dtb in ${KERNEL_DEVICETREE}; do
@@ -43,6 +68,12 @@ fakeroot do_install_devicetree() {
 			symlink_name=${type}"-"${KERNEL_IMAGE_SYMLINK_NAME}
 			DTB_SYMLINK_NAME=`echo ${symlink_name} | sed "s/${MACHINE}/${DTB_BASE_NAME}/g"`
 			ln -sf ${DTB_BASE_NAME}.${DTB_EXT} ${D}/${KERNEL_IMAGEDEST}/devicetree-${DTB_SYMLINK_NAME}.${DTB_EXT}
+
+			if [ "$type" = "zImage" ] && [ "${KERNEL_DEVICETREE_BUNDLE}" = "1" ]; then
+				cat ${D}/${KERNEL_IMAGEDEST}/$type \
+					${D}/${KERNEL_IMAGEDEST}/${DTB_BASE_NAME}.${DTB_EXT} \
+					> ${D}/${KERNEL_IMAGEDEST}/$type-${DTB_BASE_NAME}.${DTB_EXT}.bin
+			fi
 		done
 	done
 }
@@ -63,6 +94,21 @@ do_deploy_append() {
 			install -m 0644 ${DTB_PATH} ${DEPLOYDIR}/${DTB_NAME}.${DTB_EXT}
 			ln -sf ${DTB_NAME}.${DTB_EXT} ${DEPLOYDIR}/${DTB_SYMLINK_NAME}.${DTB_EXT}
 			ln -sf ${DTB_NAME}.${DTB_EXT} ${DEPLOYDIR}/${DTB_BASE_NAME}.${DTB_EXT}
+
+			if [ "$type" = "zImage" ] && [ "${KERNEL_DEVICETREE_BUNDLE}" = "1" ]; then
+				cat ${DEPLOYDIR}/$type \
+					${DEPLOYDIR}/${DTB_NAME}.${DTB_EXT} \
+					> ${DEPLOYDIR}/${DTB_NAME}.${DTB_EXT}.bin
+				ln -sf ${DTB_NAME}.${DTB_EXT}.bin ${DEPLOYDIR}/$type-${DTB_BASE_NAME}.${DTB_EXT}.bin
+
+				if [ -e "${KERNEL_OUTPUT_DIR}/${type}.initramfs" ]; then
+					cat ${KERNEL_OUTPUT_DIR}/${type}.initramfs \
+						${DEPLOYDIR}/${DTB_NAME}.${DTB_EXT} \
+						> ${DEPLOYDIR}/${type}-${INITRAMFS_BASE_NAME}-${DTB_BASE_NAME}.${DTB_EXT}.bin
+					ln -sf ${type}-${INITRAMFS_BASE_NAME}-${DTB_BASE_NAME}.${DTB_EXT}.bin \
+					       ${DEPLOYDIR}/${type}-initramfs-${DTB_BASE_NAME}.${DTB_EXT}-${MACHINE}.bin
+				fi
+			fi
 		done
 	done
 }
-- 
2.14.1



^ permalink raw reply related	[flat|nested] 4+ messages in thread
* Re: [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass
  2017-09-11 21:18 [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass Otavio Salvador
  2017-09-11 21:18 ` [PATCH v7 2/2] kernel-devicetree.bbclass: Add support to generate append to kernel Otavio Salvador
@ 2017-09-12  8:34 ` Richard Purdie
  2017-09-12 11:44   ` Otavio Salvador
  1 sibling, 1 reply; 4+ messages in thread
From: Richard Purdie @ 2017-09-12  8:34 UTC (permalink / raw)
  To: Otavio Salvador, OpenEmbedded Core Mailing List

On Mon, 2017-09-11 at 18:18 -0300, Otavio Salvador wrote:
> The Device Tree is commonly used but it is still kept as a .inc file
> instead of a proper class. Instead now we move the Device Tree code
> to
> a kernel-devicetree class and automatically enable it when the
> KERNEL_DEVICETREE variable is set.
> 
> While converting to the class, we reworked the compile and install in
> tasks as well as run the build of the Device Tree files in parallel,
> if possible.
> 
> To avoid breakage in existing layers, we kept a linux-dtb.inc file
> which raises a warning telling the user about the change so in next
> release this can be removed.

Sorry but this patchset doesn't appear to be well tested and has big
issues. For example:

https://autobuilder.yocto.io/builders/nightly-x86/builds/497/steps/Buil
dImages/logs/stdio

I'm going to guess that do_compile_devicetree is racing
against do_compile_kernelmodules. Debugging these kinds of races is
tremendously hard and taking a patchset on a key component like the
kernel at this point in M4 where this hasn't been considered is
worrying/risky.

https://autobuilder.yocto.io/builders/nightly-mips-lsb/builds/461/steps
/BuildImages/logs/stdio

another form of the same race? The autobuilder is full of these
failures :(

I think if this patchset is to go anywhere it needs to be split into
more incremental changes, one to move to the new file structure but not
change the functionality, another looking at parallelisation etc. as
the current approach of "this patch does X + Y + Z" makes review harder
and makes it harder to merge any piece of it.

Cheers,

Richard






^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass
  2017-09-12  8:34 ` [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass Richard Purdie
@ 2017-09-12 11:44   ` Otavio Salvador
  0 siblings, 0 replies; 4+ messages in thread
From: Otavio Salvador @ 2017-09-12 11:44 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Otavio Salvador, OpenEmbedded Core Mailing List

Hello Richard,

On Tue, Sep 12, 2017 at 5:34 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Mon, 2017-09-11 at 18:18 -0300, Otavio Salvador wrote:
>> The Device Tree is commonly used but it is still kept as a .inc file
>> instead of a proper class. Instead now we move the Device Tree code
>> to
>> a kernel-devicetree class and automatically enable it when the
>> KERNEL_DEVICETREE variable is set.
>>
>> While converting to the class, we reworked the compile and install in
>> tasks as well as run the build of the Device Tree files in parallel,
>> if possible.
>>
>> To avoid breakage in existing layers, we kept a linux-dtb.inc file
>> which raises a warning telling the user about the change so in next
>> release this can be removed.
>
> Sorry but this patchset doesn't appear to be well tested and has big
> issues. For example:
>
> https://autobuilder.yocto.io/builders/nightly-x86/builds/497/steps/Buil
> dImages/logs/stdio
>
> I'm going to guess that do_compile_devicetree is racing
> against do_compile_kernelmodules. Debugging these kinds of races is
> tremendously hard and taking a patchset on a key component like the
> kernel at this point in M4 where this hasn't been considered is
> worrying/risky.
>
> https://autobuilder.yocto.io/builders/nightly-mips-lsb/builds/461/steps
> /BuildImages/logs/stdio
>
> another form of the same race? The autobuilder is full of these
> failures :(

Ok.

> I think if this patchset is to go anywhere it needs to be split into
> more incremental changes, one to move to the new file structure but not
> change the functionality, another looking at parallelisation etc. as
> the current approach of "this patch does X + Y + Z" makes review harder
> and makes it harder to merge any piece of it.

I will do; I will split to the patchset so we have a more granular change.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750


^ permalink raw reply	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-09-12 11:44 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-11 21:18 [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass Otavio Salvador
2017-09-11 21:18 ` [PATCH v7 2/2] kernel-devicetree.bbclass: Add support to generate append to kernel Otavio Salvador
2017-09-12  8:34 ` [PATCH v7 1/2] kernel: Move Device Tree support to kernel.bbclass Richard Purdie
2017-09-12 11:44   ` Otavio Salvador

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.