Re: linux-next: Tree for Jun 23 (objtool (2))

Re: linux-next: Tree for Jun 23 (objtool (2))

[Miroslav Benes]

On Thu, 2 Jul 2020, Josh Poimboeuf wrote:

> On Tue, Jun 23, 2020 at 08:06:07AM -0700, Randy Dunlap wrote:
> > On 6/22/20 11:28 PM, Stephen Rothwell wrote:
> > > Hi all,
> > > 
> > > Changes since 20200622:
> > > 
> > 
> > on x86_64:
> > 
> > arch/x86/kernel/cpu/mce/core.o: warning: objtool: mce_timed_out()+0x24: unreachable instruction
> > kernel/exit.o: warning: objtool: __x64_sys_exit_group()+0x14: unreachable instruction
> > 
> > Full randconfig file is attached.
> 
> More livepatch...

Correct.

Both are known and I thought Josh had fixes queued somewhere for both, but 
my memory fails me quite often. See below.

However, I think it is time to decide how to approach this whole saga. It 
seems that there are not so many places in the kernel in need of 
__noreturn annotation in the end and as jikos argued at least some of 
those should be fixed regardless. Josh, should I prepare proper patches 
and submit them to relevant maintainers to see where this path is going?

It would be much better to fix it in GCC, but it has been like banging 
one's head against a wall so far. Josh, you wanted to create a bug 
for GCC in this respect in the past? Has that happened?

If I remember correctly, we discussed briefly a possibility to cope with 
that in objtool, but no solution was presented.

Removing -flive-patching is also a possibility. I don't like it much, but 
we discussed it with Petr M. a couple of months ago and it might be a way 
too.

Thanks
Miroslav

---

diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index 14e4b4d17ee5..469a71ecea3c 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -279,7 +279,7 @@ static int fake_panic;
 static atomic_t mce_fake_panicked;
 
 /* Panic in progress. Enable interrupts and wait for final IPI */
-static void wait_for_panic(void)
+static void __noreturn wait_for_panic(void)
 {
        long timeout = PANIC_TIMEOUT*USEC_PER_SEC;
 
diff --git a/kernel/exit.c b/kernel/exit.c
index 727150f28103..570649152e7f 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -877,7 +877,7 @@ SYSCALL_DEFINE1(exit, int, error_code)
  * as well as by sys_exit_group (below).
  */
 void
-do_group_exit(int exit_code)
+__noreturn do_group_exit(int exit_code)
 {
        struct signal_struct *sig = current->signal;

Re: linux-next: Tree for Jun 23 (objtool (2))

[Josh Poimboeuf]

On Tue, Jul 14, 2020 at 12:56:21PM +0200, Miroslav Benes wrote:
> On Thu, 2 Jul 2020, Josh Poimboeuf wrote:
> 
> > On Tue, Jun 23, 2020 at 08:06:07AM -0700, Randy Dunlap wrote:
> > > On 6/22/20 11:28 PM, Stephen Rothwell wrote:
> > > > Hi all,
> > > > 
> > > > Changes since 20200622:
> > > > 
> > > 
> > > on x86_64:
> > > 
> > > arch/x86/kernel/cpu/mce/core.o: warning: objtool: mce_timed_out()+0x24: unreachable instruction
> > > kernel/exit.o: warning: objtool: __x64_sys_exit_group()+0x14: unreachable instruction
> > > 
> > > Full randconfig file is attached.
> > 
> > More livepatch...
> 
> Correct.
> 
> Both are known and I thought Josh had fixes queued somewhere for both, but 
> my memory fails me quite often. See below.

I did have fixes for some of them in a stash somewhere, but I never
finished them because I decided it's a GCC bug.

> However, I think it is time to decide how to approach this whole saga. It 
> seems that there are not so many places in the kernel in need of 
> __noreturn annotation in the end and as jikos argued at least some of 
> those should be fixed regardless.

I would agree that global functions like do_group_exit() deserve a
__noreturn annotation, though it should be in the header file.  But
static functions shouldn't need it.

> Josh, should I prepare proper patches and submit them to relevant
> maintainers to see where this path is going?

If that's how you want to handle it, ok, but it doesn't seem right to
me, for the static functions at least.

> It would be much better to fix it in GCC, but it has been like banging 
> one's head against a wall so far. Josh, you wanted to create a bug 
> for GCC in this respect in the past? Has that happened?

I didn't open a bug, but I could, if you think that would help.  I
haven't had a lot of success with GCC bugs in the past.

> If I remember correctly, we discussed briefly a possibility to cope with 
> that in objtool, but no solution was presented.

That would also feel like a GCC workaround and might impede objtool's
ability to find bugs like this one, and possibly more serious bugs.

> Removing -flive-patching is also a possibility. I don't like it much, but 
> we discussed it with Petr M. a couple of months ago and it might be a way 
> too.

-flive-patching has many problems which I outlined before.  None of them
have been addressed.  I still feel the same way, that it should be
reverted until it's ready.  Otherwise it's a drain on upstream.

Also, if the GCC developers won't acknowledge this bug then it doesn't
give me confidence in their ability to keep the feature working as
optimizations are added or changed.

I still think a potential alternative exists: objtool could be used as a
simple tree-wide object diff tool by generating a checksum for each
function.  Then the patch can be applied and built to see exactly which
functions have changed, based on the changed checksums.  In which case
this feature would no longer be needed anyway, would you agree?

I also think that could be a first step for converging our patch
creation processes.

-- 
Josh

Re: linux-next: Tree for Jun 23 (objtool (2))

[Miroslav Benes]

On Tue, 14 Jul 2020, Josh Poimboeuf wrote:

> On Tue, Jul 14, 2020 at 12:56:21PM +0200, Miroslav Benes wrote:
> > On Thu, 2 Jul 2020, Josh Poimboeuf wrote:
> > 
> > > On Tue, Jun 23, 2020 at 08:06:07AM -0700, Randy Dunlap wrote:
> > > > On 6/22/20 11:28 PM, Stephen Rothwell wrote:
> > > > > Hi all,
> > > > > 
> > > > > Changes since 20200622:
> > > > > 
> > > > 
> > > > on x86_64:
> > > > 
> > > > arch/x86/kernel/cpu/mce/core.o: warning: objtool: mce_timed_out()+0x24: unreachable instruction
> > > > kernel/exit.o: warning: objtool: __x64_sys_exit_group()+0x14: unreachable instruction
> > > > 
> > > > Full randconfig file is attached.
> > > 
> > > More livepatch...
> > 
> > Correct.
> > 
> > Both are known and I thought Josh had fixes queued somewhere for both, but 
> > my memory fails me quite often. See below.
> 
> I did have fixes for some of them in a stash somewhere, but I never
> finished them because I decided it's a GCC bug.

Same here.
 
> > However, I think it is time to decide how to approach this whole saga. It 
> > seems that there are not so many places in the kernel in need of 
> > __noreturn annotation in the end and as jikos argued at least some of 
> > those should be fixed regardless.
> 
> I would agree that global functions like do_group_exit() deserve a
> __noreturn annotation, though it should be in the header file.  But
> static functions shouldn't need it.

Agreed. I'll post the patches for global functions eventually, but see 
below first.

> > Josh, should I prepare proper patches and submit them to relevant
> > maintainers to see where this path is going?
> 
> If that's how you want to handle it, ok, but it doesn't seem right to
> me, for the static functions at least.
> 
> > It would be much better to fix it in GCC, but it has been like banging 
> > one's head against a wall so far. Josh, you wanted to create a bug 
> > for GCC in this respect in the past? Has that happened?
> 
> I didn't open a bug, but I could, if you think that would help.  I
> haven't had a lot of success with GCC bugs in the past.

Understood.

> > If I remember correctly, we discussed briefly a possibility to cope with 
> > that in objtool, but no solution was presented.
> 
> That would also feel like a GCC workaround and might impede objtool's
> ability to find bugs like this one, and possibly more serious bugs.
> 
> > Removing -flive-patching is also a possibility. I don't like it much, but 
> > we discussed it with Petr M. a couple of months ago and it might be a way 
> > too.
> 
> -flive-patching has many problems which I outlined before.  None of them
> have been addressed.  I still feel the same way, that it should be
> reverted until it's ready.  Otherwise it's a drain on upstream.
> 
> Also, if the GCC developers won't acknowledge this bug then it doesn't
> give me confidence in their ability to keep the feature working as
> optimizations are added or changed.

I must admit that I've started to share the sentiment recently. And it is 
probably the main reason for changing my mind about the whole thing.

> I still think a potential alternative exists: objtool could be used as a
> simple tree-wide object diff tool by generating a checksum for each
> function.  Then the patch can be applied and built to see exactly which
> functions have changed, based on the changed checksums.  In which case
> this feature would no longer be needed anyway, would you agree?

Yes.

> I also think that could be a first step for converging our patch
> creation processes.

Yes again.

Petr, would you agree to revert -flive-patching due to reasons above? Is 
there anything you want to add?

Miroslav

Re: linux-next: Tree for Jun 23 (objtool (2))

[Petr Mladek]

On Wed 2020-07-15 13:11:14, Miroslav Benes wrote:
> Petr, would you agree to revert -flive-patching.

Yes, I agree.

Best Regards,
Petr

Re: linux-next: Tree for Jun 23 (objtool (2))

[Petr Mladek]

On Wed 2020-07-15 14:10:54, Petr Mladek wrote:
> On Wed 2020-07-15 13:11:14, Miroslav Benes wrote:
> > Petr, would you agree to revert -flive-patching.
> 
> Yes, I agree.

Or better to say that I will not block it.

I see that it causes maintenance burden. There are questions about
reliability and performance impact. I do not have a magic solution
in the pocket.

Anyway, we need a solution to know what functions need to get livepatched.
I do not have experience with comparing the assembly, so I do not know
how it is complicated and reliable.

Best Regards,
Petr

Re: linux-next: Tree for Jun 23 (objtool (2))

[Josh Poimboeuf]

On Wed, Jul 15, 2020 at 03:41:55PM +0200, Petr Mladek wrote:
> On Wed 2020-07-15 14:10:54, Petr Mladek wrote:
> > On Wed 2020-07-15 13:11:14, Miroslav Benes wrote:
> > > Petr, would you agree to revert -flive-patching.
> > 
> > Yes, I agree.
> 
> Or better to say that I will not block it.
> 
> I see that it causes maintenance burden. There are questions about
> reliability and performance impact. I do not have a magic solution
> in the pocket.
> 
> Anyway, we need a solution to know what functions need to get livepatched.
> I do not have experience with comparing the assembly, so I do not know
> how it is complicated and reliable.

Thanks Petr/Miroslav.  I can do the revert patch.  It doesn't have to be
a permanent revert.  I'd certainly be willing to ACK it again in the
future if/when it becomes more ready.

Yannick has agreed to start looking at the objtool function diff
feature.  It's purely theoretical at the moment, we'll see how it works
in practice.  We already do something similar in kpatch-build -- it
differs from the objtool model, but it at least shows that something
similar is possible.

-- 
Josh

Re: linux-next: Tree for Jun 23 (objtool (2))

[Miroslav Benes]

On Wed, 15 Jul 2020, Josh Poimboeuf wrote:

> On Wed, Jul 15, 2020 at 03:41:55PM +0200, Petr Mladek wrote:
> > On Wed 2020-07-15 14:10:54, Petr Mladek wrote:
> > > On Wed 2020-07-15 13:11:14, Miroslav Benes wrote:
> > > > Petr, would you agree to revert -flive-patching.
> > > 
> > > Yes, I agree.
> > 
> > Or better to say that I will not block it.
> > 
> > I see that it causes maintenance burden. There are questions about
> > reliability and performance impact. I do not have a magic solution
> > in the pocket.
> > 
> > Anyway, we need a solution to know what functions need to get livepatched.
> > I do not have experience with comparing the assembly, so I do not know
> > how it is complicated and reliable.
> 
> Thanks Petr/Miroslav.  I can do the revert patch.  It doesn't have to be
> a permanent revert.  I'd certainly be willing to ACK it again in the
> future if/when it becomes more ready.

Ok.

> Yannick has agreed to start looking at the objtool function diff
> feature.  It's purely theoretical at the moment, we'll see how it works
> in practice.  We already do something similar in kpatch-build -- it
> differs from the objtool model, but it at least shows that something
> similar is possible.

Great. I'm looking forward to seeing that.

Thanks
Miroslav