Occasional hung with UM after enable VMAP_STACK

Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi all,

I noticed that after "um: enable VMAP_STACK" [1] I experienced some 
occasional hung in my Gitlab CI jobs that use user-mode-linux to build 
distro images. The configuration use for the building is the one 
available from Debian, see [2] and [3]. Unfortunately I don't have much 
information, when the issue arises the job seems to hang and it is 
cancelled by Gitlab due to a job timeout.

So far I did some bisect to get the commit which introduces the issue 
and a quick check with latest Linux stable branch master to confirm the 
issue is still present. I know this is not a proper bug report but I 
thought it will be better to at least raise a comment.

Thanks in advance,

Walter

[1] http://lists.infradead.org/pipermail/linux-um/2021-August/001657.html

[2] https://packages.debian.org/bookworm/user-mode-linux

[3] https://salsa.debian.org/uml-team/user-mode-linux

-- 
Walter Lozano
Collabora Ltd.

Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi all,

I noticed that after "um: enable VMAP_STACK" [1] I experienced some 
occasional hung in my Gitlab CI jobs that use user-mode-linux to build 
distro images. The configuration use for the building is the one 
available from Debian, see [2] and [3]. Unfortunately I don't have much 
information, when the issue arises the job seems to hang and it is 
cancelled by Gitlab due to a job timeout.

So far I did some bisect to get the commit which introduces the issue 
and a quick check with latest Linux stable branch master to confirm the 
issue is still present. I know this is not a proper bug report but I 
thought it will be better to at least raise a comment.

Thanks in advance,

Walter

[1] http://lists.infradead.org/pipermail/linux-um/2021-August/001657.html

[2] https://packages.debian.org/bookworm/user-mode-linux

[3] https://salsa.debian.org/uml-team/user-mode-linux

-- 
Walter Lozano
Collabora Ltd.


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: Occasional hung with UM after enable VMAP_STACK

[Johannes Berg]

On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
> Hi all,
> 
> I noticed that after "um: enable VMAP_STACK" [1] I experienced some 
> occasional hung in my Gitlab CI jobs that use user-mode-linux to build 
> distro images.
> 

Did you actually *enable* VMAP_STACK in the config as well? The commit
just makes it *possible* to enable it, you still have to set it
yourself. So you should be able to easily check with/without that
setting.

johannes

Re: Occasional hung with UM after enable VMAP_STACK

[Johannes Berg]

On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
> Hi all,
> 
> I noticed that after "um: enable VMAP_STACK" [1] I experienced some 
> occasional hung in my Gitlab CI jobs that use user-mode-linux to build 
> distro images.
> 

Did you actually *enable* VMAP_STACK in the config as well? The commit
just makes it *possible* to enable it, you still have to set it
yourself. So you should be able to easily check with/without that
setting.

johannes

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi Johannes,

On 1/4/22 16:04, Johannes Berg wrote:
> On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
>> Hi all,
>>
>> I noticed that after "um: enable VMAP_STACK" [1] I experienced some
>> occasional hung in my Gitlab CI jobs that use user-mode-linux to build
>> distro images.
>>
> Did you actually *enable* VMAP_STACK in the config as well? The commit
> just makes it *possible* to enable it, you still have to set it
> yourself. So you should be able to easily check with/without that
> setting.

Thank you for your quick response. The Debian configuration on package 
user-mode-linux have these settings

CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y


as you can see in [1]. I did run some tests disabling those settings, 
which passed without any hung.

Unfortunately the "occasional" behavior makes this issue a bit tricky to 
debug.

Regards,

Walter

[1] 
https://salsa.debian.org/uml-team/user-mode-linux/-/blob/master/config.amd64#L321 


-- 
Walter Lozano
Collabora Ltd.

Re: Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi Johannes,

On 1/4/22 16:04, Johannes Berg wrote:
> On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
>> Hi all,
>>
>> I noticed that after "um: enable VMAP_STACK" [1] I experienced some
>> occasional hung in my Gitlab CI jobs that use user-mode-linux to build
>> distro images.
>>
> Did you actually *enable* VMAP_STACK in the config as well? The commit
> just makes it *possible* to enable it, you still have to set it
> yourself. So you should be able to easily check with/without that
> setting.

Thank you for your quick response. The Debian configuration on package 
user-mode-linux have these settings

CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y


as you can see in [1]. I did run some tests disabling those settings, 
which passed without any hung.

Unfortunately the "occasional" behavior makes this issue a bit tricky to 
debug.

Regards,

Walter

[1] 
https://salsa.debian.org/uml-team/user-mode-linux/-/blob/master/config.amd64#L321 


-- 
Walter Lozano
Collabora Ltd.


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: Occasional hung with UM after enable VMAP_STACK

[Johannes Berg]

On Tue, 2022-01-04 at 16:26 -0300, Walter Lozano wrote:
> 
> Thank you for your quick response. The Debian configuration on package 
> user-mode-linux have these settings
> 
> CONFIG_HAVE_ARCH_VMAP_STACK=y
> CONFIG_VMAP_STACK=y

OK, so it actually _is_ enabled.

> as you can see in [1]. I did run some tests disabling those settings, 
> which passed without any hung.
> 
> Unfortunately the "occasional" behavior makes this issue a bit tricky to 
> debug.
> 

Right.

Hm. I've been running our tests with it for about three months and
haven't observed any hangs, but I guess that doesn't mean much.

To be honest, I have no particular reason to even want it, other than
that it catches accidental DMA from stack more easily ... so I guess if
we can't find anything, we might as well revert it.

Feels like it _should_ work though, since it's just a different location
for the stack.

johannes

Re: Occasional hung with UM after enable VMAP_STACK

[Johannes Berg]

On Tue, 2022-01-04 at 16:26 -0300, Walter Lozano wrote:
> 
> Thank you for your quick response. The Debian configuration on package 
> user-mode-linux have these settings
> 
> CONFIG_HAVE_ARCH_VMAP_STACK=y
> CONFIG_VMAP_STACK=y

OK, so it actually _is_ enabled.

> as you can see in [1]. I did run some tests disabling those settings, 
> which passed without any hung.
> 
> Unfortunately the "occasional" behavior makes this issue a bit tricky to 
> debug.
> 

Right.

Hm. I've been running our tests with it for about three months and
haven't observed any hangs, but I guess that doesn't mean much.

To be honest, I have no particular reason to even want it, other than
that it catches accidental DMA from stack more easily ... so I guess if
we can't find anything, we might as well revert it.

Feels like it _should_ work though, since it's just a different location
for the stack.

johannes

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: Occasional hung with UM after enable VMAP_STACK

[Anton Ivanov]

On 04/01/2022 19:26, Walter Lozano wrote:
> Hi Johannes,
> 
> On 1/4/22 16:04, Johannes Berg wrote:
>> On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
>>> Hi all,
>>>
>>> I noticed that after "um: enable VMAP_STACK" [1] I experienced some
>>> occasional hung in my Gitlab CI jobs that use user-mode-linux to build
>>> distro images.
>>>
>> Did you actually *enable* VMAP_STACK in the config as well? The commit
>> just makes it *possible* to enable it, you still have to set it
>> yourself. So you should be able to easily check with/without that
>> setting.
> 
> Thank you for your quick response. The Debian configuration on package 
> user-mode-linux have these settings
> 
> CONFIG_HAVE_ARCH_VMAP_STACK=y
> CONFIG_VMAP_STACK=y
> 
> 
> as you can see in [1]. I did run some tests disabling those settings, 
> which passed without any hung.
> 
> Unfortunately the "occasional" behavior makes this issue a bit tricky to 
> debug.
> 
> Regards,
> 
> Walter
> 
> [1] 
> https://salsa.debian.org/uml-team/user-mode-linux/-/blob/master/config.amd64#L321 
> 
> 

Just to narrow things down - 64 bit or 32 bit?

-- 
Anton R. Ivanov
https://www.kot-begemot.co.uk/

Re: Occasional hung with UM after enable VMAP_STACK

[Anton Ivanov]

On 04/01/2022 19:26, Walter Lozano wrote:
> Hi Johannes,
> 
> On 1/4/22 16:04, Johannes Berg wrote:
>> On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
>>> Hi all,
>>>
>>> I noticed that after "um: enable VMAP_STACK" [1] I experienced some
>>> occasional hung in my Gitlab CI jobs that use user-mode-linux to build
>>> distro images.
>>>
>> Did you actually *enable* VMAP_STACK in the config as well? The commit
>> just makes it *possible* to enable it, you still have to set it
>> yourself. So you should be able to easily check with/without that
>> setting.
> 
> Thank you for your quick response. The Debian configuration on package 
> user-mode-linux have these settings
> 
> CONFIG_HAVE_ARCH_VMAP_STACK=y
> CONFIG_VMAP_STACK=y
> 
> 
> as you can see in [1]. I did run some tests disabling those settings, 
> which passed without any hung.
> 
> Unfortunately the "occasional" behavior makes this issue a bit tricky to 
> debug.
> 
> Regards,
> 
> Walter
> 
> [1] 
> https://salsa.debian.org/uml-team/user-mode-linux/-/blob/master/config.amd64#L321 
> 
> 

Just to narrow things down - 64 bit or 32 bit?

-- 
Anton R. Ivanov
https://www.kot-begemot.co.uk/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi Johannes,

On 1/4/22 16:49, Johannes Berg wrote:
> On Tue, 2022-01-04 at 16:26 -0300, Walter Lozano wrote:
>> Thank you for your quick response. The Debian configuration on package
>> user-mode-linux have these settings
>>
>> CONFIG_HAVE_ARCH_VMAP_STACK=y
>> CONFIG_VMAP_STACK=y
> OK, so it actually _is_ enabled.
>
>> as you can see in [1]. I did run some tests disabling those settings,
>> which passed without any hung.
>>
>> Unfortunately the "occasional" behavior makes this issue a bit tricky to
>> debug.
>>
> Right.
>
> Hm. I've been running our tests with it for about three months and
> haven't observed any hangs, but I guess that doesn't mean much.

The issue is very rare at least in my setup, I had to prepare a test 
environment to stress the system in order to be able to reproduce it 
more or less consistently. I will continue investigating and share any 
useful information.

> To be honest, I have no particular reason to even want it, other than
> that it catches accidental DMA from stack more easily ... so I guess if
> we can't find anything, we might as well revert it.

I hope we will be able to find the root cause of the issue.

> Feels like it _should_ work though, since it's just a different location
> for the stack.


I see. Thank you again for your quick reply, I will keep you updated.

Regards,

Walter

-- 
Walter Lozano
Collabora Ltd.

Re: Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi Johannes,

On 1/4/22 16:49, Johannes Berg wrote:
> On Tue, 2022-01-04 at 16:26 -0300, Walter Lozano wrote:
>> Thank you for your quick response. The Debian configuration on package
>> user-mode-linux have these settings
>>
>> CONFIG_HAVE_ARCH_VMAP_STACK=y
>> CONFIG_VMAP_STACK=y
> OK, so it actually _is_ enabled.
>
>> as you can see in [1]. I did run some tests disabling those settings,
>> which passed without any hung.
>>
>> Unfortunately the "occasional" behavior makes this issue a bit tricky to
>> debug.
>>
> Right.
>
> Hm. I've been running our tests with it for about three months and
> haven't observed any hangs, but I guess that doesn't mean much.

The issue is very rare at least in my setup, I had to prepare a test 
environment to stress the system in order to be able to reproduce it 
more or less consistently. I will continue investigating and share any 
useful information.

> To be honest, I have no particular reason to even want it, other than
> that it catches accidental DMA from stack more easily ... so I guess if
> we can't find anything, we might as well revert it.

I hope we will be able to find the root cause of the issue.

> Feels like it _should_ work though, since it's just a different location
> for the stack.


I see. Thank you again for your quick reply, I will keep you updated.

Regards,

Walter

-- 
Walter Lozano
Collabora Ltd.


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi Anton,

On 1/4/22 18:39, Anton Ivanov wrote:
> On 04/01/2022 19:26, Walter Lozano wrote:
>> Hi Johannes,
>>
>> On 1/4/22 16:04, Johannes Berg wrote:
>>> On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
>>>> Hi all,
>>>>
>>>> I noticed that after "um: enable VMAP_STACK" [1] I experienced some
>>>> occasional hung in my Gitlab CI jobs that use user-mode-linux to build
>>>> distro images.
>>>>
>>> Did you actually *enable* VMAP_STACK in the config as well? The commit
>>> just makes it *possible* to enable it, you still have to set it
>>> yourself. So you should be able to easily check with/without that
>>> setting.
>>
>> Thank you for your quick response. The Debian configuration on 
>> package user-mode-linux have these settings
>>
>> CONFIG_HAVE_ARCH_VMAP_STACK=y
>> CONFIG_VMAP_STACK=y
>>
>>
>> as you can see in [1]. I did run some tests disabling those settings, 
>> which passed without any hung.
>>
>> Unfortunately the "occasional" behavior makes this issue a bit tricky 
>> to debug.
>>
>> Regards,
>>
>> Walter
>>
>> [1] 
>> https://salsa.debian.org/uml-team/user-mode-linux/-/blob/master/config.amd64#L321 
>>
>>
>
> Just to narrow things down - 64 bit or 32 bit?
>
Thank you for commenting on this thread. All my tests were done with 64 
bits.

Regards,

-- 
Walter Lozano
Collabora Ltd.

Re: Occasional hung with UM after enable VMAP_STACK

[Walter Lozano]

Hi Anton,

On 1/4/22 18:39, Anton Ivanov wrote:
> On 04/01/2022 19:26, Walter Lozano wrote:
>> Hi Johannes,
>>
>> On 1/4/22 16:04, Johannes Berg wrote:
>>> On Tue, 2022-01-04 at 15:10 -0300, Walter Lozano wrote:
>>>> Hi all,
>>>>
>>>> I noticed that after "um: enable VMAP_STACK" [1] I experienced some
>>>> occasional hung in my Gitlab CI jobs that use user-mode-linux to build
>>>> distro images.
>>>>
>>> Did you actually *enable* VMAP_STACK in the config as well? The commit
>>> just makes it *possible* to enable it, you still have to set it
>>> yourself. So you should be able to easily check with/without that
>>> setting.
>>
>> Thank you for your quick response. The Debian configuration on 
>> package user-mode-linux have these settings
>>
>> CONFIG_HAVE_ARCH_VMAP_STACK=y
>> CONFIG_VMAP_STACK=y
>>
>>
>> as you can see in [1]. I did run some tests disabling those settings, 
>> which passed without any hung.
>>
>> Unfortunately the "occasional" behavior makes this issue a bit tricky 
>> to debug.
>>
>> Regards,
>>
>> Walter
>>
>> [1] 
>> https://salsa.debian.org/uml-team/user-mode-linux/-/blob/master/config.amd64#L321 
>>
>>
>
> Just to narrow things down - 64 bit or 32 bit?
>
Thank you for commenting on this thread. All my tests were done with 64 
bits.

Regards,

-- 
Walter Lozano
Collabora Ltd.


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um