[ANNOUNCE] v5.9-rc2-rt1

[ANNOUNCE] v5.9-rc2-rt1

[Sebastian Andrzej Siewior]

Dear RT folks!

I'm pleased to announce the v5.9-rc2-rt1 patch set. 

Changes since v5.6.19-rt12:

  - Rebase to v5.9-rc2

  - The seqcount related patches have been replaced on top of the
    seqcount series by Ahmed S. Darwis which landed mainline. 

  - The posix-timer patches have been dropped because upstream changes
    cover all of was needed on RT's side. As a result RT relies on
    HAVE_POSIX_CPU_TIMERS_TASK_WORK. This is provided only by x86.
    The RT patch provides this option for ARM/ARM64/POWERPC as long as
    KVM is disabled. The reason is that the task work must be handled
    before KVM returns to guest.

Known issues
     - It has been pointed out that due to changes to the printk code the
       internal buffer representation changed. This is only an issue if tools
       like `crash' are used to extract the printk buffer from a kernel memory
       image.

You can get this release via the git tree at:

    git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v5.9-rc2-rt1

The RT patch against v5.9-rc2 can be found here:

    https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.9/older/patch-5.9-rc2-rt1.patch.xz

The split quilt queue is available at:

    https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.9/older/patches-5.9-rc2-rt1.tar.xz

Sebastian

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

Hi Sebastian,

On Mon, Aug 24, 2020 at 05:46:05PM +0200, Sebastian Andrzej Siewior wrote:
> I'm pleased to announce the v5.9-rc2-rt1 patch set.

I gave it a quick run on my test system. Can't boot the system at this
point. Didn't look closer at it, maybe it's something obvious...

 stack segment: 0000 [#1] PREEMPT_RT SMP PTI
 CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.9.0-rc2-rt1 #1
 Hardware name: wortmann G31M-ES2L/G31M-S2L, BIOS F10 09/29/2009
 RIP: 0010:intel_pipe_update_start+0x139/0x5f0
 Code: ed 7c a1 ff 48 8d 83 b8 05 00 00 48 89 c5 48 89 04 24 e8 ea 75 51 00 89 c0 48 03 2c c5 a0 26 0e a0 65 48 8b 04 25 80 6c 01 00 <48> 39 45 30 0f 84 4a 01 00 00 48 89 ef e8 35 e2 51 00 8b 35 b7 42
 RSP: 0000:ffffbc3d800179c8 EFLAGS: 00010287
 
 RAX: ffff9c91ab220000 RBX: ffff9c91aaac6800 RCX: 0000000000000000
 RDX: 0000000000000001 RSI: ffffffffa0015367 RDI: ffffffffa003f1de
 RBP: ffff3923566c6db8 R08: 0000000000000000 R09: 0000000000000001
 R10: ffff9c91ab292b90 R11: 00000000000000ba R12: 00000000000001e8
 R13: ffff9c91a9e9cc30 R14: 00000000000001e4 R15: 00000000000001e7
 FS:  0000000000000000(0000) GS:ffff9c91abc00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000000 CR3: 00000000ac41a000 CR4: 00000000000406f0
 Call Trace:
  ? preempt_count_add+0x68/0xa0
  ? _raw_spin_lock+0x13/0x30
  ? wait_woken+0x90/0x90
  intel_update_crtc+0xa7/0x360
  ? wait_woken+0x90/0x90
  intel_commit_modeset_enables+0x5e/0x80
  intel_atomic_commit_tail+0x311/0x1210
  ? __queue_work+0x372/0x540
  ? migrate_enable+0x11b/0x430
  intel_atomic_commit+0x357/0x3e0
  intel_modeset_init+0x84d/0x1e50
  i915_driver_probe+0x95d/0xe00
  i915_pci_probe+0x44/0x120
  local_pci_probe+0x26/0x50
  pci_device_probe+0xd5/0x160
  really_probe+0xdb/0x2e0
  device_driver_attach+0x53/0x60
  __driver_attach+0x4c/0xc0
  ? device_driver_attach+0x60/0x60
  bus_for_each_dev+0x7b/0xc0
  bus_add_driver+0x17a/0x1c0
  driver_register+0x6c/0xc0
  ? mipi_dsi_bus_init+0x11/0x11
  i915_init+0x58/0x6b
  do_one_initcall+0x46/0x274
  kernel_init_freeable+0x199/0x1dc
  ? rest_init+0xba/0xba
  kernel_init+0xa/0x106
  ret_from_fork+0x22/0x30
 Modules linked in:
 
 Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
 Kernel Offset: 0x1de00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
 ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

Thanks,
Daniel

Re: [ANNOUNCE] v5.9-rc2-rt1

[Sebastian Andrzej Siewior]

On 2020-08-26 10:08:02 [+0200], Daniel Wagner wrote:
> Hi Sebastian,
Hi,

> On Mon, Aug 24, 2020 at 05:46:05PM +0200, Sebastian Andrzej Siewior wrote:
> > I'm pleased to announce the v5.9-rc2-rt1 patch set.
> 
> I gave it a quick run on my test system. Can't boot the system at this
> point. Didn't look closer at it, maybe it's something obvious...

Carsten reported it the other day, but didn't Cc: the list. I've sent
him this to test:

diff --git a/drivers/gpu/drm/i915/display/intel_sprite.c b/drivers/gpu/drm/i915/display/intel_sprite.c
index 24baa5f2047bb..cc435d0a51215 100644
--- a/drivers/gpu/drm/i915/display/intel_sprite.c
+++ b/drivers/gpu/drm/i915/display/intel_sprite.c
@@ -118,8 +118,6 @@ void intel_pipe_update_start(const struct intel_crtc_state *new_crtc_state)
 			"PSR idle timed out 0x%x, atomic update may fail\n",
 			psr_status);
 
-	local_lock_irq(&crtc->pipe_update_lock);
-
 	crtc->debug.min_vbl = min;
 	crtc->debug.max_vbl = max;
 	trace_intel_pipe_update_start(crtc);
@@ -143,11 +141,7 @@ void intel_pipe_update_start(const struct intel_crtc_state *new_crtc_state)
 			break;
 		}
 
-		local_unlock_irq(&crtc->pipe_update_lock);
-
 		timeout = schedule_timeout(timeout);
-
-		local_lock_irq(&crtc->pipe_update_lock);
 	}
 
 	finish_wait(wq, &wait);
@@ -180,7 +174,6 @@ void intel_pipe_update_start(const struct intel_crtc_state *new_crtc_state)
 	return;
 
 irq_disable:
-	local_lock_irq(&crtc->pipe_update_lock);
 }
 
 /**
@@ -218,8 +211,6 @@ void intel_pipe_update_end(struct intel_crtc_state *new_crtc_state)
 		new_crtc_state->uapi.event = NULL;
 	}
 
-	local_unlock_irq(&crtc->pipe_update_lock);
-
 	if (intel_vgpu_active(dev_priv))
 		return;
 

He complained about a stale label and so. 
The RT patch replaced a local-irq-disable with a local-lock which broke.
I intend to get rid of this local-irq-disable on RT, the local-lock is
just duct-tape to make it look the same. If this works for everyone then
I will think of something…

> Thanks,
> Daniel

Sebastian

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

On Wed, Aug 26, 2020 at 10:12:11AM +0200, Sebastian Andrzej Siewior wrote:
> The RT patch replaced a local-irq-disable with a local-lock which broke.
> I intend to get rid of this local-irq-disable on RT, the local-lock is
> just duct-tape to make it look the same. If this works for everyone then
> I will think of something…

Yep, this patch helps. The system boots now. I give it a complete test
run, just to make sure.

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

On Wed, Aug 26, 2020 at 11:05:18AM +0200, Daniel Wagner wrote:
> Yep, this patch helps. The system boots now. I give it a complete test
> run, just to make sure.

All looks good, no crash and all tests do pass on x86_64. Firing up the
ARM boards now.

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

On Wed, Aug 26, 2020 at 12:43:26PM +0200, Daniel Wagner wrote:
> All looks good, no crash and all tests do pass on x86_64. Firing up the
> ARM boards now.

All test pass on the BeagleBone Black.

Something is a bit weird with my RPi3 in 64bit mode. uboot loads
the the dtb file via ftp and then does a booti. For all non PREEMPT_RT
kernels (PREEMPT, NONE, SMP=n, ...) everything works fine. For the
PREEMPT_RT kernel uboot complains with

  U-Boot> booti 0x00080000 - 0x02600000
  bootloader-commands: Wait for prompt Starting kernel (timeout 00:03:43)
  booti 0x00080000 - 0x02600000
  ERROR: Did not find a cmdline Flattened Device Tree
  Starting kernel ...

It's the same dtb in all cases. Not totally sure what is upsetting uboot
here, maybe the load addresses don't work anymore?

In short, I can't really say if v5.9-rt for ARMv8 works for me.

Re: [ANNOUNCE] v5.9-rc2-rt1

[Sebastian Andrzej Siewior]

On 2020-08-27 11:19:10 [+0200], Daniel Wagner wrote:
> the the dtb file via ftp and then does a booti. For all non PREEMPT_RT
> kernels (PREEMPT, NONE, SMP=n, ...) everything works fine. For the
> PREEMPT_RT kernel uboot complains with
> 
>   U-Boot> booti 0x00080000 - 0x02600000
>   bootloader-commands: Wait for prompt Starting kernel (timeout 00:03:43)
>   booti 0x00080000 - 0x02600000
>   ERROR: Did not find a cmdline Flattened Device Tree
>   Starting kernel ...
> 
> It's the same dtb in all cases. Not totally sure what is upsetting uboot
> here, maybe the load addresses don't work anymore?

So v5.9-rc2-rt1 with PREEMPT_RT=y enabled leads to the problem and
v5.9-rc2-rt1 with PREEMPT_PREEMPT=y boots fine?

> In short, I can't really say if v5.9-rt for ARMv8 works for me.

Sebastian

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

On Thu, Aug 27, 2020 at 11:27:43AM +0200, Sebastian Andrzej Siewior wrote:
> So v5.9-rc2-rt1 with PREEMPT_RT=y enabled leads to the problem and
> v5.9-rc2-rt1 with PREEMPT_PREEMPT=y boots fine?

Yes. But it must be something with uboot related since it's not the
kernel printing the error message. It's likely just an infrastructure
problem on my side. Properly related to the kernel size:

-rw-r--r-- 1 wagi users 31910400 Aug 26 12:54 rpi3-image-ll-v5.9-rc2-rt1-rebase-1-gdf6b97f22351
-rw-r--r-- 1 wagi users 31910400 Aug 26 12:49 rpi3-image-nohz-v5.9-rc2-rt1-rebase-1-gdf6b97f22351
-rw-r--r-- 1 wagi users 32891392 Aug 26 12:47 rpi3-image-none-v5.9-rc2-rt1-rebase-1-gdf6b97f22351
-rw-r--r-- 1 wagi users 38224384 Aug 26 12:45 rpi3-image-rt-v5.9-rc2-rt1-rebase-1-gdf6b97f22351
-rw-r--r-- 1 wagi users 31910400 Aug 26 12:54 rpi3-image-up-v5.9-rc2-rt1-rebase-1-gdf6b97f22351
-rw-r--r-- 1 wagi users 32891392 Aug 26 12:52 rpi3-image-vp-v5.9-rc2-rt1-rebase-1-gdf6b97f22351

The -rt kernel is roughly 6MB larger. Just need to check the memory
ranges u-boot is using.

Re: [ANNOUNCE] v5.9-rc2-rt1

[Sebastian Andrzej Siewior]

On 2020-08-27 12:16:22 [+0200], Daniel Wagner wrote:
> The -rt kernel is roughly 6MB larger. Just need to check the memory
> ranges u-boot is using.

so that 6MiB sounded bad but then it is ~36MiB in total so….
Is this full debug and so on?

Sebastian

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

On Thu, Aug 27, 2020 at 12:28:40PM +0200, Sebastian Andrzej Siewior wrote:
> On 2020-08-27 12:16:22 [+0200], Daniel Wagner wrote:
> > The -rt kernel is roughly 6MB larger. Just need to check the memory
> > ranges u-boot is using.
> 
> so that 6MiB sounded bad but then it is ~36MiB in total so….
> Is this full debug and so on?

I didn't really try to minimize the kernel. I haven't checked yet
if 5.6-rt is also showing this size increase. At least, our SUSE spin
off from v5.4-rt doesn't have this size increase with the same config.

Would be interesting to see the size numbers for v5.6-rt? Hmm, I'll
just start the compiler. It's all scripted anyway :)

Anyway, the config is:

  make defconfig

and

#
# Networking
#
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y

# NFS
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_ROOT_NFS=y

#
# Debugging
#
CONFIG_DEBUG_INFO=y
CONFIG_PRINTK_TIME=y
CONFIG_DEBUG_KERNEL=y
CONFIG_EARLY_PRINTK=y
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=7

# Embedded config to kernel. /proc/config.gz
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y

CONFIG_KEXEC=y

# Default settings
#
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCKDEP is not set

CONFIG_DEBUG_ATOMIC_SLEEP=y

# CONFIG_CPU_FREQ is not set
# CONFIG_CPU_IDLE is not set

# CONFIG_NO_HZ is not set
CONFIG_HZ_PERIODIC=y

CONFIG_HZ_250=y
CONFIG_HZ=250

# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
# CONFIG_PM is not set

# cyclicdeadline dependency
CONFIG_SCHED_DEBUG=y

CONFIG_RCU_EXPERT=y
CONFIG_RCU_NOCB_CPU=y

# tracing
CONFIG_EMBEDDED=y
CONFIG_EXPERT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_SCHED_TRACER=y

CONFIG_PREEMPT_RT=y

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

> Would be interesting to see the size numbers for v5.6-rt? Hmm, I'll
> just start the compiler. It's all scripted anyway :)

v5.6-rt:

-rw-r--r-- 1 wagi users 28688896 Aug 27 13:38 rpi3-image-ll-v5.6.19-rt12
-rw-r--r-- 1 wagi users 28688896 Aug 27 13:34 rpi3-image-nohz-v5.6.19-rt12
-rw-r--r-- 1 wagi users 29669888 Aug 27 13:32 rpi3-image-none-v5.6.19-rt12
-rw-r--r-- 1 wagi users 33438208 Aug 27 13:30 rpi3-image-rt-v5.6.19-rt12
-rw-r--r-- 1 wagi users 28688896 Aug 27 13:38 rpi3-image-up-v5.6.19-rt12
-rw-r--r-- 1 wagi users 29669888 Aug 27 13:36 rpi3-image-vp-v5.6.19-rt12

v5.4-rt:

-rw-r--r-- 1 wagi users 27525632 Aug 27 14:40 rpi3-image-ll-v5.4.59-rt36
-rw-r--r-- 1 wagi users 27525632 Aug 27 14:36 rpi3-image-nohz-v5.4.59-rt36
-rw-r--r-- 1 wagi users 28506624 Aug 27 14:34 rpi3-image-none-v5.4.59-rt36
-rw-r--r-- 1 wagi users 32360960 Aug 27 14:32 rpi3-image-rt-v5.4.59-rt36
-rw-r--r-- 1 wagi users 27525632 Aug 27 14:40 rpi3-image-up-v5.4.59-rt36
-rw-r--r-- 1 wagi users 28506624 Aug 27 14:38 rpi3-image-vp-v5.4.59-rt36

So in previous releases the size offset was roughly around 4MB.

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Vacek]

On Thu, Aug 27, 2020 at 2:49 PM Daniel Wagner <wagi@monom.org> wrote:
>
> > Would be interesting to see the size numbers for v5.6-rt? Hmm, I'll
> > just start the compiler. It's all scripted anyway :)
>
> v5.6-rt:
>
> -rw-r--r-- 1 wagi users 28688896 Aug 27 13:38 rpi3-image-ll-v5.6.19-rt12
> -rw-r--r-- 1 wagi users 28688896 Aug 27 13:34 rpi3-image-nohz-v5.6.19-rt12
> -rw-r--r-- 1 wagi users 29669888 Aug 27 13:32 rpi3-image-none-v5.6.19-rt12
> -rw-r--r-- 1 wagi users 33438208 Aug 27 13:30 rpi3-image-rt-v5.6.19-rt12
> -rw-r--r-- 1 wagi users 28688896 Aug 27 13:38 rpi3-image-up-v5.6.19-rt12
> -rw-r--r-- 1 wagi users 29669888 Aug 27 13:36 rpi3-image-vp-v5.6.19-rt12
>
> v5.4-rt:
>
> -rw-r--r-- 1 wagi users 27525632 Aug 27 14:40 rpi3-image-ll-v5.4.59-rt36
> -rw-r--r-- 1 wagi users 27525632 Aug 27 14:36 rpi3-image-nohz-v5.4.59-rt36
> -rw-r--r-- 1 wagi users 28506624 Aug 27 14:34 rpi3-image-none-v5.4.59-rt36
> -rw-r--r-- 1 wagi users 32360960 Aug 27 14:32 rpi3-image-rt-v5.4.59-rt36
> -rw-r--r-- 1 wagi users 27525632 Aug 27 14:40 rpi3-image-up-v5.4.59-rt36
> -rw-r--r-- 1 wagi users 28506624 Aug 27 14:38 rpi3-image-vp-v5.4.59-rt36
>
> So in previous releases the size offset was roughly around 4MB.

Maybe the threshold is 33554432, eg. 32 megs...

--nX

Re: [ANNOUNCE] v5.9-rc2-rt1

[Daniel Wagner]

On Thu, Aug 27, 2020 at 05:50:24PM +0200, Daniel Vacek wrote:
> Maybe the threshold is 33554432, eg. 32 megs...

I've rearranged the load addresses for the kernel and the dtb and now
the board is booting again. Starting with a full test run now.

The size list was mainly to see if the v5.9 tree is showing abnormal
size regression. It doesn't look that's the case. All RT kernels are
bigger than the rest.

Re: [ANNOUNCE] v5.9-rc2-rt1

[Sebastian Andrzej Siewior]

On 2020-08-28 09:36:50 [+0200], Daniel Wagner wrote:
> On Thu, Aug 27, 2020 at 05:50:24PM +0200, Daniel Vacek wrote:
> > Maybe the threshold is 33554432, eg. 32 megs...
> 
> I've rearranged the load addresses for the kernel and the dtb and now
> the board is booting again. Starting with a full test run now.
> 
> The size list was mainly to see if the v5.9 tree is showing abnormal
> size regression. It doesn't look that's the case. All RT kernels are
> bigger than the rest.

Looking at the size increase, the v5.6 -> 5.9 increased way more than
5.4 -> 5.6. However this is also true for the ll config (and is not
limited to RT):

rpi3-image-ll-v5.4.59-rt36 -> rpi3-image-ll-v5.6.19-rt12 -> rpi3-image-ll-v5.9-rc2-rt1
0                          -> + 1.1 MiB (4.2%)           -> + 3.1 MiB (11.2%)

rpi3-image-rt-v5.4.59-rt36 -> rpi3-image-rt-v5.6.19-rt12 -> rpi3-image-rt-v5.9-rc2-rt1
0                          -> + 1.0 MiB (3.3%)           -> + 4.6 MiB (14.3%)

Sebastian