ipipe-arm64 kernel issue

ipipe-arm64 kernel issue

[Steve Pavao]

I have 2 different, basic apps that oops in a very similar way on ipipe-arm64 running on R Pi 3B.  One of them is rpm, another is my own program.

Here is the partial backtrace for the ‘rpm’ oops.  The backtrace for my other program is almost identical.  (I’ve left out interim calls to keep this email concise.)

[  124.816179] [<ffffff8008260480>] __find_get_block+0x2a0/0x400
...
[  124.816311] [<ffffff8008778b88>] do_page_fault+0x268/0x4e0
[  124.816320] [<ffffff8008778e6c>] do_translation_fault+0x6c/0x7c
[  124.816330] [<ffffff80080814ec>] do_mem_abort+0x54/0xc8

The BUG finally occurs at line 1279 of buffer.c:

static inline void check_irqs_on(void)
{
#ifdef irqs_disabled
	BUG_ON(irqs_disabled());   //  <———— here
#endif
}

The above BUG happens during the page fault recovery process.  (FYI, some of the calls between do_page_fault_and __find_get_block are ext4 calls.)

In both apps, the oops happens around the time of file creation on the SD card:  For rpm, it is during an install of a valid rpm package, for the other app, it occurs at the time a sqlite database is opened.  This is basic stuff that should just work.

I am looking for ideas about what could be wrong with the setup/configuration of my ipipe-arm64 system, such that some apps that read/write to the SD card are oopsing this way when doing page fault recovery.

- Steve Pavao
Korg R&D

Re: ipipe-arm64 kernel issue

[Philippe Gerum]

On 12/11/18 10:25 PM, Steve Pavao via Xenomai wrote:
> I have 2 different, basic apps that oops in a very similar way on ipipe-arm64 running on R Pi 3B.  One of them is rpm, another is my own program.
> 
> Here is the partial backtrace for the ‘rpm’ oops.  The backtrace for my other program is almost identical.  (I’ve left out interim calls to keep this email concise.)
> 
> [  124.816179] [<ffffff8008260480>] __find_get_block+0x2a0/0x400
> ...
> [  124.816311] [<ffffff8008778b88>] do_page_fault+0x268/0x4e0
> [  124.816320] [<ffffff8008778e6c>] do_translation_fault+0x6c/0x7c
> [  124.816330] [<ffffff80080814ec>] do_mem_abort+0x54/0xc8
> 
> The BUG finally occurs at line 1279 of buffer.c:
> 
> static inline void check_irqs_on(void)
> {
> #ifdef irqs_disabled
> 	BUG_ON(irqs_disabled());   //  <———— here
> #endif
> }
> 
> The above BUG happens during the page fault recovery process.  (FYI, some of the calls between do_page_fault_and __find_get_block are ext4 calls.)
> 
> In both apps, the oops happens around the time of file creation on the SD card:  For rpm, it is during an install of a valid rpm package, for the other app, it occurs at the time a sqlite database is opened.  This is basic stuff that should just work.
> 
> I am looking for ideas about what could be wrong with the setup/configuration of my ipipe-arm64 system, such that some apps that read/write to the SD card are oopsing this way when doing page fault recovery.

This is a bug in the I-pipe layer. Assuming you are still running kernel
4.9, please try 4.14 from [1], it fixes several issues which still exist
in the 4.9 pipeline.

[1] https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64


-- 
Philippe.

Re: ipipe-arm64 kernel issue

[Steve Pavao]

I'm actually using 4.1.14 at change Id 133bbf97672ff6e72fc21784b8ae44b194b76626 in master branch.

I will build at the latest head, however, to see if the problem goes away.

Thank you.

- Steve Pavao
Korg R&D


> On Dec 12, 2018, at 5:21 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> 
> On 12/11/18 10:25 PM, Steve Pavao via Xenomai wrote:
>> I have 2 different, basic apps that oops in a very similar way on ipipe-arm64 running on R Pi 3B.  One of them is rpm, another is my own program.
>> 
>> Here is the partial backtrace for the ‘rpm’ oops.  The backtrace for my other program is almost identical.  (I’ve left out interim calls to keep this email concise.)
>> 
>> [  124.816179] [<ffffff8008260480>] __find_get_block+0x2a0/0x400
>> ...
>> [  124.816311] [<ffffff8008778b88>] do_page_fault+0x268/0x4e0
>> [  124.816320] [<ffffff8008778e6c>] do_translation_fault+0x6c/0x7c
>> [  124.816330] [<ffffff80080814ec>] do_mem_abort+0x54/0xc8
>> 
>> The BUG finally occurs at line 1279 of buffer.c:
>> 
>> static inline void check_irqs_on(void)
>> {
>> #ifdef irqs_disabled
>> 	BUG_ON(irqs_disabled());   //  <———— here
>> #endif
>> }
>> 
>> The above BUG happens during the page fault recovery process.  (FYI, some of the calls between do_page_fault_and __find_get_block are ext4 calls.)
>> 
>> In both apps, the oops happens around the time of file creation on the SD card:  For rpm, it is during an install of a valid rpm package, for the other app, it occurs at the time a sqlite database is opened.  This is basic stuff that should just work.
>> 
>> I am looking for ideas about what could be wrong with the setup/configuration of my ipipe-arm64 system, such that some apps that read/write to the SD card are oopsing this way when doing page fault recovery.
> 
> This is a bug in the I-pipe layer. Assuming you are still running kernel
> 4.9, please try 4.14 from [1], it fixes several issues which still exist
> in the 4.9 pipeline.
> 
> [1] https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64 <https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64>
> 
> 
> -- 
> Philippe.

Re: ipipe-arm64 kernel issue

[Steve Pavao]

> On Dec 12, 2018, at 5:21 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> 
> On 12/11/18 10:25 PM, Steve Pavao via Xenomai wrote:
>> I have 2 different, basic apps that oops in a very similar way on ipipe-arm64 running on R Pi 3B.  One of them is rpm, another is my own program.
>> 
>> Here is the partial backtrace for the ‘rpm’ oops.  The backtrace for my other program is almost identical.  (I’ve left out interim calls to keep this email concise.)
>> 
>> [  124.816179] [<ffffff8008260480>] __find_get_block+0x2a0/0x400
>> ...
>> [  124.816311] [<ffffff8008778b88>] do_page_fault+0x268/0x4e0
>> [  124.816320] [<ffffff8008778e6c>] do_translation_fault+0x6c/0x7c
>> [  124.816330] [<ffffff80080814ec>] do_mem_abort+0x54/0xc8
>> 
>> The BUG finally occurs at line 1279 of buffer.c:
>> 
>> static inline void check_irqs_on(void)
>> {
>> #ifdef irqs_disabled
>> 	BUG_ON(irqs_disabled());   //  <———— here
>> #endif
>> }
>> 
>> The above BUG happens during the page fault recovery process.  (FYI, some of the calls between do_page_fault_and __find_get_block are ext4 calls.)
>> 
>> In both apps, the oops happens around the time of file creation on the SD card:  For rpm, it is during an install of a valid rpm package, for the other app, it occurs at the time a sqlite database is opened.  This is basic stuff that should just work.
>> 
>> I am looking for ideas about what could be wrong with the setup/configuration of my ipipe-arm64 system, such that some apps that read/write to the SD card are oopsing this way when doing page fault recovery.
> 
> This is a bug in the I-pipe layer. Assuming you are still running kernel
> 4.9, please try 4.14 from [1], it fixes several issues which still exist
> in the 4.9 pipeline.
> 
> [1] https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64 <https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64>
> 
> 
> -- 
> Philippe.

Thank you very much!  That code does fix the kernel oops problem.

As I continue to bring up my app, I notice I’m having a segfault (SIGSEGV) trying to access the general purpose IO pins.

I notice that /dev/gpiomem is missing from the environment.

Would this be due to an out-of-date or too lean dtb file for the Pi 3B, or might it not be available for some other reason?

I am using the dtb from the ipipe-arm64 build.

Steve Pavao
Korg R&D

Re: ipipe-arm64 kernel issue

[Steve Pavao]

> On Dec 13, 2018, at 7:50 AM, Steve Pavao <stevep@korgrd.com> wrote:

> As I continue to bring up my app, I notice I’m having a segfault (SIGSEGV) trying to access the general purpose IO pins.
> 
> I notice that /dev/gpiomem is missing from the environment.

> I am using the dtb from the ipipe-arm64 build.
> 
> Steve Pavao
> Korg R&D

I learned there is a kernel config option to load a driver for this.  The config option is named CONFIG_BCM2835_DEVGPIOMEM.  It is set to Y in the config.  In the raspberrypi-linux kernel, the driver is here https://github.com/raspberrypi/linux/tree/rpi-4.14.y/drivers/char/broadcom , but I don’t see the file in the ipipe-arm64 repo.

Do the ipipe-arm repos tend to allow support for vendor-specific drivers for features like this, and if so, would the Xenomai team be willing to add the gpiomem driver to the ipipe-arm64 kernel build?  I don’t know the policy for something like this, so thought I would ask to add support for it, and see what is customary.

Steve Pavao
Korg R&D

Re: ipipe-arm64 kernel issue

[Greg Gallagher]

Hi Steve,

On Thu, Dec 13, 2018 at 3:04 PM Steve Pavao via Xenomai
<xenomai@xenomai.org> wrote:
>
> > On Dec 13, 2018, at 7:50 AM, Steve Pavao <stevep@korgrd.com> wrote:
>
> > As I continue to bring up my app, I notice I’m having a segfault (SIGSEGV) trying to access the general purpose IO pins.
> >
> > I notice that /dev/gpiomem is missing from the environment.
>
> > I am using the dtb from the ipipe-arm64 build.
> >
> > Steve Pavao
> > Korg R&D
>
> I learned there is a kernel config option to load a driver for this.  The config option is named CONFIG_BCM2835_DEVGPIOMEM.  It is set to Y in the config.  In the raspberrypi-linux kernel, the driver is here https://github.com/raspberrypi/linux/tree/rpi-4.14.y/drivers/char/broadcom , but I don’t see the file in the ipipe-arm64 repo.

If it's not in the mainline repo then it won't be in the arm64 ipipe tree.
>
> Do the ipipe-arm repos tend to allow support for vendor-specific drivers for features like this, and if so, would the Xenomai team be willing to add the gpiomem driver to the ipipe-arm64 kernel build?  I don’t know the policy for something like this, so thought I would ask to add support for it, and see what is customary.
>
Do the pins need realtime response?  If you don't need realtime
response have you looked at using the /dev/gpiochipX character driver?
 If they need to be realtime we have a rtdm gpio driver that can be
used to control the pins.

> Steve Pavao
> Korg R&D
>
>
>

-Greg

Re: ipipe-arm64 kernel issue

[Pintu Agarwal]

On Thu, Dec 13, 2018 at 6:20 PM Steve Pavao via Xenomai
<xenomai@xenomai.org> wrote:
>
>
> > On Dec 12, 2018, at 5:21 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> >
> > On 12/11/18 10:25 PM, Steve Pavao via Xenomai wrote:
> >> I have 2 different, basic apps that oops in a very similar way on ipipe-arm64 running on R Pi 3B.  One of them is rpm, another is my own program.
> >>
> >> Here is the partial backtrace for the ‘rpm’ oops.  The backtrace for my other program is almost identical.  (I’ve left out interim calls to keep this email concise.)
> >>
> >> [  124.816179] [<ffffff8008260480>] __find_get_block+0x2a0/0x400
> >> ...
> >> [  124.816311] [<ffffff8008778b88>] do_page_fault+0x268/0x4e0
> >> [  124.816320] [<ffffff8008778e6c>] do_translation_fault+0x6c/0x7c
> >> [  124.816330] [<ffffff80080814ec>] do_mem_abort+0x54/0xc8
> >>
> >> The BUG finally occurs at line 1279 of buffer.c:
> >>
> >> static inline void check_irqs_on(void)
> >> {
> >> #ifdef irqs_disabled
> >>      BUG_ON(irqs_disabled());   //  <———— here
> >> #endif
> >> }
> >>
> >> The above BUG happens during the page fault recovery process.  (FYI, some of the calls between do_page_fault_and __find_get_block are ext4 calls.)
> >>
> >> In both apps, the oops happens around the time of file creation on the SD card:  For rpm, it is during an install of a valid rpm package, for the other app, it occurs at the time a sqlite database is opened.  This is basic stuff that should just work.
> >>
> >> I am looking for ideas about what could be wrong with the setup/configuration of my ipipe-arm64 system, such that some apps that read/write to the SD card are oopsing this way when doing page fault recovery.
> >
> > This is a bug in the I-pipe layer. Assuming you are still running kernel
> > 4.9, please try 4.14 from [1], it fixes several issues which still exist
> > in the 4.9 pipeline.
> >
> > [1] https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64 <https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64>
> >
> >
> > --
> > Philippe.
>
> Thank you very much!  That code does fix the kernel oops problem.
>

Hello Steve,
I am also facing the similar issue.
Can you point me to the exact patch that fixes this issue.
Actually I am using a vendor kernel repo, so I cannot upgrade the kernel.
I can only apply the fix patches.

Re: ipipe-arm64 kernel issue

[Pintu Agarwal]

Dear Steve,

Can you help me in identifying the patches for this issue?
Can you tell me which ipipe repo you used for your Dragon board ?

Thanks,
Pintu

On Fri, Dec 21, 2018 at 6:18 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> On Thu, Dec 13, 2018 at 6:20 PM Steve Pavao via Xenomai
> <xenomai@xenomai.org> wrote:
> >
> >
> > > On Dec 12, 2018, at 5:21 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> > >
> > > On 12/11/18 10:25 PM, Steve Pavao via Xenomai wrote:
> > >> I have 2 different, basic apps that oops in a very similar way on ipipe-arm64 running on R Pi 3B.  One of them is rpm, another is my own program.
> > >>
> > >> Here is the partial backtrace for the ‘rpm’ oops.  The backtrace for my other program is almost identical.  (I’ve left out interim calls to keep this email concise.)
> > >>
> > >> [  124.816179] [<ffffff8008260480>] __find_get_block+0x2a0/0x400
> > >> ...
> > >> [  124.816311] [<ffffff8008778b88>] do_page_fault+0x268/0x4e0
> > >> [  124.816320] [<ffffff8008778e6c>] do_translation_fault+0x6c/0x7c
> > >> [  124.816330] [<ffffff80080814ec>] do_mem_abort+0x54/0xc8
> > >>
> > >> The BUG finally occurs at line 1279 of buffer.c:
> > >>
> > >> static inline void check_irqs_on(void)
> > >> {
> > >> #ifdef irqs_disabled
> > >>      BUG_ON(irqs_disabled());   //  <———— here
> > >> #endif
> > >> }
> > >>
> > >> The above BUG happens during the page fault recovery process.  (FYI, some of the calls between do_page_fault_and __find_get_block are ext4 calls.)
> > >>
> > >> In both apps, the oops happens around the time of file creation on the SD card:  For rpm, it is during an install of a valid rpm package, for the other app, it occurs at the time a sqlite database is opened.  This is basic stuff that should just work.
> > >>
> > >> I am looking for ideas about what could be wrong with the setup/configuration of my ipipe-arm64 system, such that some apps that read/write to the SD card are oopsing this way when doing page fault recovery.
> > >
> > > This is a bug in the I-pipe layer. Assuming you are still running kernel
> > > 4.9, please try 4.14 from [1], it fixes several issues which still exist
> > > in the 4.9 pipeline.
> > >
> > > [1] https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64 <https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64>
> > >
> > >
> > > --
> > > Philippe.
> >
> > Thank you very much!  That code does fix the kernel oops problem.
> >
>
> Hello Steve,
> I am also facing the similar issue.
> Can you point me to the exact patch that fixes this issue.
> Actually I am using a vendor kernel repo, so I cannot upgrade the kernel.
> I can only apply the fix patches.

Re: ipipe-arm64 kernel issue

[Steve Pavao]

Hi Pintu,

1. I am not using a Dragon board.  I mentioned I was using a R Pi 3B.  You can see it is mentioned just a few lines below.

2. I did not do any special patches.  I am simply using the branch that Phillipe suggested: https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64 <https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64>
That is also mentioned below.

- Steve Pavao


> On Jan 8, 2019, at 7:58 AM, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> 
> Dear Steve,
> 
> Can you help me in identifying the patches for this issue?
> Can you tell me which ipipe repo you used for your Dragon board ?
> 
> Thanks,
> Pintu
> 
> On Fri, Dec 21, 2018 at 6:18 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
>> 
>> On Thu, Dec 13, 2018 at 6:20 PM Steve Pavao via Xenomai
>> <xenomai@xenomai.org> wrote:
>>> 
>>> 
>>>> On Dec 12, 2018, at 5:21 AM, Philippe Gerum <rpm@xenomai.org> wrote:
>>>> 
>>>> On 12/11/18 10:25 PM, Steve Pavao via Xenomai wrote:
>>>>> I have 2 different, basic apps that oops in a very similar way on ipipe-arm64 running on R Pi 3B.  One of them is rpm, another is my own program.
>>>>> 
>>>>> Here is the partial backtrace for the ‘rpm’ oops.  The backtrace for my other program is almost identical.  (I’ve left out interim calls to keep this email concise.)
>>>>> 
>>>>> [  124.816179] [<ffffff8008260480>] __find_get_block+0x2a0/0x400
>>>>> ...
>>>>> [  124.816311] [<ffffff8008778b88>] do_page_fault+0x268/0x4e0
>>>>> [  124.816320] [<ffffff8008778e6c>] do_translation_fault+0x6c/0x7c
>>>>> [  124.816330] [<ffffff80080814ec>] do_mem_abort+0x54/0xc8
>>>>> 
>>>>> The BUG finally occurs at line 1279 of buffer.c:
>>>>> 
>>>>> static inline void check_irqs_on(void)
>>>>> {
>>>>> #ifdef irqs_disabled
>>>>>     BUG_ON(irqs_disabled());   //  <———— here
>>>>> #endif
>>>>> }
>>>>> 
>>>>> The above BUG happens during the page fault recovery process.  (FYI, some of the calls between do_page_fault_and __find_get_block are ext4 calls.)
>>>>> 
>>>>> In both apps, the oops happens around the time of file creation on the SD card:  For rpm, it is during an install of a valid rpm package, for the other app, it occurs at the time a sqlite database is opened.  This is basic stuff that should just work.
>>>>> 
>>>>> I am looking for ideas about what could be wrong with the setup/configuration of my ipipe-arm64 system, such that some apps that read/write to the SD card are oopsing this way when doing page fault recovery.
>>>> 
>>>> This is a bug in the I-pipe layer. Assuming you are still running kernel
>>>> 4.9, please try 4.14 from [1], it fixes several issues which still exist
>>>> in the 4.9 pipeline.
>>>> 
>>>> [1] https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64 <https://gitlab.denx.de/Xenomai/ipipe-arm64/commits/stable/4.14.71-arm64>
>>>> 
>>>> 
>>>> --
>>>> Philippe.
>>> 
>>> Thank you very much!  That code does fix the kernel oops problem.
>>> 
>> 
>> Hello Steve,
>> I am also facing the similar issue.
>> Can you point me to the exact patch that fixes this issue.
>> Actually I am using a vendor kernel repo, so I cannot upgrade the kernel.
>> I can only apply the fix patches.