Re: [GIT PULL] x86/urgent for v5.11-rc7

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Josh Poimboeuf]

On Mon, Feb 08, 2021 at 10:02:06AM -0500, Steven Rostedt wrote:
> On Sun, 7 Feb 2021 16:45:40 -0600
> Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> 
> > > I do suspect involved people should start thinking about how they want
> > > to deal with functions starting with
> > > 
> > >         endbr64
> > >         call __fentry__
> > > 
> > > instead of the call being at the very top of the function.  
> > 
> > FWIW, objtool's already fine with it (otherwise we would have discovered
> > the need to disable fcf-protection much sooner).
> 
> And this doesn't really affect tracing (note, another user that might be
> affected is live kernel patching).

Good point, livepatch is indeed affected.  Is there a better way to get
the "call __fentry__" address for a given function?


/*
 * Convert a function address into the appropriate ftrace location.
 *
 * Usually this is just the address of the function, but on some architectures
 * it's more complicated so allow them to provide a custom behaviour.
 */
#ifndef klp_get_ftrace_location
static unsigned long klp_get_ftrace_location(unsigned long faddr)
{
	return faddr;
}
#endif

-- 
Josh

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Peter Zijlstra]

On Mon, Feb 08, 2021 at 09:33:00AM -0600, Josh Poimboeuf wrote:
> On Mon, Feb 08, 2021 at 10:02:06AM -0500, Steven Rostedt wrote:
> > On Sun, 7 Feb 2021 16:45:40 -0600
> > Josh Poimboeuf <jpoimboe@redhat.com> wrote:
> > 
> > > > I do suspect involved people should start thinking about how they want
> > > > to deal with functions starting with
> > > > 
> > > >         endbr64
> > > >         call __fentry__
> > > > 
> > > > instead of the call being at the very top of the function.  
> > > 
> > > FWIW, objtool's already fine with it (otherwise we would have discovered
> > > the need to disable fcf-protection much sooner).
> > 
> > And this doesn't really affect tracing (note, another user that might be
> > affected is live kernel patching).
> 
> Good point, livepatch is indeed affected.  Is there a better way to get
> the "call __fentry__" address for a given function?
> 
> 
> /*
>  * Convert a function address into the appropriate ftrace location.
>  *
>  * Usually this is just the address of the function, but on some architectures
>  * it's more complicated so allow them to provide a custom behaviour.
>  */
> #ifndef klp_get_ftrace_location
> static unsigned long klp_get_ftrace_location(unsigned long faddr)
> {
> 	return faddr;
> }
> #endif

I suppose the trivial fix is to see if it points to endbr64 and if so,
increment the addr by the length of that.

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Steven Rostedt]

On Mon, 8 Feb 2021 16:47:05 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> > /*
> >  * Convert a function address into the appropriate ftrace location.
> >  *
> >  * Usually this is just the address of the function, but on some architectures
> >  * it's more complicated so allow them to provide a custom behaviour.
> >  */
> > #ifndef klp_get_ftrace_location
> > static unsigned long klp_get_ftrace_location(unsigned long faddr)
> > {
> > 	return faddr;
> > }
> > #endif  
> 
> I suppose the trivial fix is to see if it points to endbr64 and if so,
> increment the addr by the length of that.

I thought of that too. But one thing that may be possible, is to use
kallsym. I believe you can get the range of a function (start and end of
the function) from kallsyms. Then ask ftrace for the addr in that range
(there should only be one).

-- Steve

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Miroslav Benes]

On Mon, 8 Feb 2021, Steven Rostedt wrote:

> On Mon, 8 Feb 2021 16:47:05 +0100
> Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > > /*
> > >  * Convert a function address into the appropriate ftrace location.
> > >  *
> > >  * Usually this is just the address of the function, but on some architectures
> > >  * it's more complicated so allow them to provide a custom behaviour.
> > >  */
> > > #ifndef klp_get_ftrace_location
> > > static unsigned long klp_get_ftrace_location(unsigned long faddr)
> > > {
> > > 	return faddr;
> > > }
> > > #endif  

powerpc has this

static inline unsigned long klp_get_ftrace_location(unsigned long faddr)                                               
{                                                                                                                      
        /*                                                                                                             
         * Live patch works only with -mprofile-kernel on PPC. In this case,                                           
         * the ftrace location is always within the first 16 bytes.                                                    
         */                                                                                                            
        return ftrace_location_range(faddr, faddr + 16);                                                               
}                                                                                                                      

> > I suppose the trivial fix is to see if it points to endbr64 and if so,
> > increment the addr by the length of that.
> 
> I thought of that too. But one thing that may be possible, is to use
> kallsym. I believe you can get the range of a function (start and end of
> the function) from kallsyms. Then ask ftrace for the addr in that range
> (there should only be one).

And we can do this if a hard-coded value live above is not welcome. If I 
remember correctly, we used to have exactly this in the old versions of 
kGraft. We walked through all ftrace records, called 
kallsyms_lookup_size_offset() on every record's ip and if the offset+ip 
matched faddr (in this case), we returned the ip.

Miroslav

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Steven Rostedt]

On Tue, 9 Feb 2021 09:32:34 +0100 (CET)
Miroslav Benes <mbenes@suse.cz> wrote:

> powerpc has this
> 
> static inline unsigned long klp_get_ftrace_location(unsigned long faddr)                                               
> {                                                                                                                      
>         /*                                                                                                             
>          * Live patch works only with -mprofile-kernel on PPC. In this case,                                           
>          * the ftrace location is always within the first 16 bytes.                                                    
>          */                                                                                                            
>         return ftrace_location_range(faddr, faddr + 16);                                                               
> }                                                                                                                      
> 
> > > I suppose the trivial fix is to see if it points to endbr64 and if so,
> > > increment the addr by the length of that.  
> > 
> > I thought of that too. But one thing that may be possible, is to use
> > kallsym. I believe you can get the range of a function (start and end of
> > the function) from kallsyms. Then ask ftrace for the addr in that range
> > (there should only be one).  
> 
> And we can do this if a hard-coded value live above is not welcome. If I 
> remember correctly, we used to have exactly this in the old versions of 
> kGraft. We walked through all ftrace records, called 
> kallsyms_lookup_size_offset() on every record's ip and if the offset+ip 
> matched faddr (in this case), we returned the ip.

Either way is fine. Question is, should we just wait till CET is
implemented for the kernel before making any of these changes? Just knowing
that we have a solution to handle it may be good enough for now.

-- Steve

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Miroslav Benes]

On Tue, 9 Feb 2021, Steven Rostedt wrote:

> On Tue, 9 Feb 2021 09:32:34 +0100 (CET)
> Miroslav Benes <mbenes@suse.cz> wrote:
> 
> > powerpc has this
> > 
> > static inline unsigned long klp_get_ftrace_location(unsigned long faddr)                                               
> > {                                                                                                                      
> >         /*                                                                                                             
> >          * Live patch works only with -mprofile-kernel on PPC. In this case,                                           
> >          * the ftrace location is always within the first 16 bytes.                                                    
> >          */                                                                                                            
> >         return ftrace_location_range(faddr, faddr + 16);                                                               
> > }                                                                                                                      
> > 
> > > > I suppose the trivial fix is to see if it points to endbr64 and if so,
> > > > increment the addr by the length of that.  
> > > 
> > > I thought of that too. But one thing that may be possible, is to use
> > > kallsym. I believe you can get the range of a function (start and end of
> > > the function) from kallsyms. Then ask ftrace for the addr in that range
> > > (there should only be one).  
> > 
> > And we can do this if a hard-coded value live above is not welcome. If I 
> > remember correctly, we used to have exactly this in the old versions of 
> > kGraft. We walked through all ftrace records, called 
> > kallsyms_lookup_size_offset() on every record's ip and if the offset+ip 
> > matched faddr (in this case), we returned the ip.
> 
> Either way is fine. Question is, should we just wait till CET is
> implemented for the kernel before making any of these changes? Just knowing
> that we have a solution to handle it may be good enough for now.

I'd prefer it to be a part of CET enablement patch set.

Miroslav

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Alexei Starovoitov]

On Tue, Feb 9, 2021 at 6:49 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 9 Feb 2021 09:32:34 +0100 (CET)
> Miroslav Benes <mbenes@suse.cz> wrote:
>
> > powerpc has this
> >
> > static inline unsigned long klp_get_ftrace_location(unsigned long faddr)
> > {
> >         /*
> >          * Live patch works only with -mprofile-kernel on PPC. In this case,
> >          * the ftrace location is always within the first 16 bytes.
> >          */
> >         return ftrace_location_range(faddr, faddr + 16);
> > }
> >
> > > > I suppose the trivial fix is to see if it points to endbr64 and if so,
> > > > increment the addr by the length of that.
> > >
> > > I thought of that too. But one thing that may be possible, is to use
> > > kallsym. I believe you can get the range of a function (start and end of
> > > the function) from kallsyms. Then ask ftrace for the addr in that range
> > > (there should only be one).
> >
> > And we can do this if a hard-coded value live above is not welcome. If I
> > remember correctly, we used to have exactly this in the old versions of
> > kGraft. We walked through all ftrace records, called
> > kallsyms_lookup_size_offset() on every record's ip and if the offset+ip
> > matched faddr (in this case), we returned the ip.
>
> Either way is fine. Question is, should we just wait till CET is
> implemented for the kernel before making any of these changes? Just knowing
> that we have a solution to handle it may be good enough for now.

I think the issue is more fundamental than what appears on the surface.
According to endbr64 documentation it's not just any instruction.
The cpu will wait for it and if it's replaced with int3 or not seen at
the branch target the cpu will throw an exception.
If I understood the doc correctly it means that endbr64 can never be
replaced with a breakpoint. If that's the case text_poke_bp and kprobe
need to do extra safety checks.

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Andy Lutomirski]

> On Feb 9, 2021, at 8:45 AM, Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> 
> On Tue, Feb 9, 2021 at 6:49 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>> 
>>> On Tue, 9 Feb 2021 09:32:34 +0100 (CET)
>>> Miroslav Benes <mbenes@suse.cz> wrote:
>>> 
>>> powerpc has this
>>> 
>>> static inline unsigned long klp_get_ftrace_location(unsigned long faddr)
>>> {
>>>        /*
>>>         * Live patch works only with -mprofile-kernel on PPC. In this case,
>>>         * the ftrace location is always within the first 16 bytes.
>>>         */
>>>        return ftrace_location_range(faddr, faddr + 16);
>>> }
>>> 
>>>>> I suppose the trivial fix is to see if it points to endbr64 and if so,
>>>>> increment the addr by the length of that.
>>>> 
>>>> I thought of that too. But one thing that may be possible, is to use
>>>> kallsym. I believe you can get the range of a function (start and end of
>>>> the function) from kallsyms. Then ask ftrace for the addr in that range
>>>> (there should only be one).
>>> 
>>> And we can do this if a hard-coded value live above is not welcome. If I
>>> remember correctly, we used to have exactly this in the old versions of
>>> kGraft. We walked through all ftrace records, called
>>> kallsyms_lookup_size_offset() on every record's ip and if the offset+ip
>>> matched faddr (in this case), we returned the ip.
>> 
>> Either way is fine. Question is, should we just wait till CET is
>> implemented for the kernel before making any of these changes? Just knowing
>> that we have a solution to handle it may be good enough for now.
> 
> I think the issue is more fundamental than what appears on the surface.
> According to endbr64 documentation it's not just any instruction.
> The cpu will wait for it and if it's replaced with int3 or not seen at
> the branch target the cpu will throw an exception.
> If I understood the doc correctly it means that endbr64 can never be
> replaced with a breakpoint. If that's the case text_poke_bp and kprobe
> need to do extra safety checks.

Ugh.

Or we hack up #CP to handle this case. I don’t quite know how I feel about this.

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Linus Torvalds]

On Tue, Feb 9, 2021 at 8:55 AM Andy Lutomirski <luto@amacapital.net> wrote:
>
> Or we hack up #CP to handle this case. I don’t quite know how I feel about this.

I think that's the sane model - if we've replaced the instruction with
'int3', and we end up getting #CP due to that, just do the #BP
handling.

Anything else would just be insanely complicated, I feel.

             Linus

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Andy Lutomirski]

> On Feb 9, 2021, at 10:09 AM, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
> On Tue, Feb 9, 2021 at 8:55 AM Andy Lutomirski <luto@amacapital.net> wrote:
>> 
>> Or we hack up #CP to handle this case. I don’t quite know how I feel about this.
> 
> I think that's the sane model - if we've replaced the instruction with
> 'int3', and we end up getting #CP due to that, just do the #BP
> handling.
> 
> Anything else would just be insanely complicated, I feel.

The other model is “don’t do that then.”

I suppose a nice property of patching ENDBR to INT3 is that, not only is it atomic, but ENDBR is sort of a NOP, so we don’t need to replace the ENDBR with anything.

> 
>             Linus

Re: [GIT PULL] x86/urgent for v5.11-rc7

[Linus Torvalds]

On Tue, Feb 9, 2021 at 10:26 AM Andy Lutomirski <luto@amacapital.net> wrote:
> >
> > Anything else would just be insanely complicated, I feel.
>
> The other model is “don’t do that then.”

Hmm. I guess all the code that does int3 patching could just be taught
to always go to the next instruction instead.

I don't think advancing the rewriting is an option for the asm
alternative() logic or the static call infrastructure, but those
should never be about endbr anyway, so presumably that's not an issue.

So if it ends up being _only_ about kprobes, then the "don't do that
then" might work fine.

                 Linus