DOS by unprivileged user

DOS by unprivileged user

[Ferry Toth]

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo 
(SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
+ hyperthreading. But I will be easy enough to get the same effect with more 
RAM, other fs etc.

In this case gitk allocates more and more memory (until my system freezes 
6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files 
(can't investigate as there is no time until it freezes) and the io wait 
goes to 100% on all cores. At this point it is impossible to login from 
remote and local keyboard and mouse are frozen. Hard reset is the only way 
out at this point.

IMHO there is something wrong in how the kernel hands out resources, in this 
case memory, CPU time and disk accesses. It should be easily measurable that 
a single application is allocating all memory and consequently all CPU time 
is spent in io wait. I'm pretty sure that if the kernel would stop or reduce 
allocating CPU time to the hogging (user) process causing excessive io wait, 
the machine would continue just fine. And probably a single process should 
not even be allowed to allocate this amount of memory. Or at least should 
not force other applications to get swapped out.

In effect gitk should have beeb forced to run slow or be killed off (which 
doesn't happen because probably the kernel itself doesn't receive enough CPU 
time).

As a positive side effect maybe someone would dig into gitk and make it less 
memory hungry (qgit is half as hungry, still a lot).

-- 
Ferry Toth

Re: DOS by unprivileged user

[Mike Galbraith]

On Thu, 2018-04-19 at 21:13 +0200, Ferry Toth wrote:
> It appears any ordinary user can easily create a DOS on linux.
> 
> One sure way to reproduce this is to open gitk on the linux kernel repo 
> (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> + hyperthreading. But I will be easy enough to get the same effect with more 
> RAM, other fs etc.
> 
> In this case gitk allocates more and more memory (until my system freezes 
> 6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files 
> (can't investigate as there is no time until it freezes) and the io wait 
> goes to 100% on all cores. At this point it is impossible to login from 
> remote and local keyboard and mouse are frozen. Hard reset is the only way 
> out at this point.

datapoint: my i4790/ext4 box running master.yesterday booted mem=8G
became highly unpleasant to use, but I retained control, and the all
cores going to 100% thing did not happen at any time.

I didn't try constraining on the gitk user, just turned it loose a few
times to see if it managed to render box effectively dead.  It failed
to kill my box, but (expectedly) did make it suck rocks.

	-Mike

Re: DOS by unprivileged user

[Ferry Toth]

Op vrijdag 20 april 2018 06:46:58 CEST schreef Mike Galbraith:
> On Thu, 2018-04-19 at 21:13 +0200, Ferry Toth wrote:
> > It appears any ordinary user can easily create a DOS on linux.
> > 
> > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > + hyperthreading. But I will be easy enough to get the same effect with more 
> > RAM, other fs etc.
> > 
> > In this case gitk allocates more and more memory (until my system freezes 
> > 6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files 
> > (can't investigate as there is no time until it freezes) and the io wait 
> > goes to 100% on all cores. At this point it is impossible to login from 
> > remote and local keyboard and mouse are frozen. Hard reset is the only way 
> > out at this point.
> 
> datapoint: my i4790/ext4 box running master.yesterday booted mem=8G
> became highly unpleasant to use, but I retained control, and the all
> cores going to 100% thing did not happen at any time.
> 
> I didn't try constraining on the gitk user, just turned it loose a few
> times to see if it managed to render box effectively dead.  It failed
> to kill my box, but (expectedly) did make it suck rocks.
> 
> 	-Mike
> 

Yes, might be less dramatic with ext4 than with btrfs (COW icw fsync on hdd'\0s destroys performance 
of things like virtualbox, databases, dpkg).

Nevertheless I feel one process should not be allowed to harm other processes by denying them resources. Even if when btrfs makes it easy abuse I think the scheduler should have throttled gitk.

Re: DOS by unprivileged user

[Mike Galbraith]

On Fri, 2018-04-20 at 10:39 +0200, Ferry Toth wrote:
> 
> Nevertheless I feel one process should not be allowed to harm other
> processes by denying them resources. Even if when btrfs makes it easy
> abuse I think the scheduler should have throttled gitk.

Memory management is not in it's job description.

	-Mike

Re: DOS by unprivileged user

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> It appears any ordinary user can easily create a DOS on linux.
> 
> One sure way to reproduce this is to open gitk on the linux kernel repo 
> (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> + hyperthreading. But I will be easy enough to get the same effect with more 
> RAM, other fs etc.

You may want to disable swap.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: DOS by unprivileged user

[vcaputo]

On Sun, Apr 22, 2018 at 12:16:54PM +0200, Pavel Machek wrote:
> On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> > It appears any ordinary user can easily create a DOS on linux.
> > 
> > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > + hyperthreading. But I will be easy enough to get the same effect with more 
> > RAM, other fs etc.
> 
> You may want to disable swap.
> 

I run without swap on my laptops, and still observe long periods of
thrashing on the road towards OOM.  What seems to occur is the active
file-backed mappings of executables/libraries become a sort of swap
area, repeatedly being discarded and faulted back in as the context
switches occur.

If there's any good way to prevent this, I'd like to know.

Regards,
Vito Caputo

Re: DOS by unprivileged user

[Michal Hocko]

On Sun 22-04-18 10:43:00, vcaputo@pengaru.com wrote:
> On Sun, Apr 22, 2018 at 12:16:54PM +0200, Pavel Machek wrote:
> > On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> > > It appears any ordinary user can easily create a DOS on linux.
> > > 
> > > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > > + hyperthreading. But I will be easy enough to get the same effect with more 
> > > RAM, other fs etc.
> > 
> > You may want to disable swap.
> > 
> 
> I run without swap on my laptops, and still observe long periods of
> thrashing on the road towards OOM.  What seems to occur is the active
> file-backed mappings of executables/libraries become a sort of swap
> area, repeatedly being discarded and faulted back in as the context
> switches occur.
> 
> If there's any good way to prevent this, I'd like to know.

I am afraid there is none yet. Johannes had some ground work for
page cache trashing detection https://marc.info/?i=20170727153010.23347-1-hannes%40cmpxchg.org
but there was no version of the patchseries for quite some time and
there was no integration into the oom detection which would be
non-trivial as well.

I realize this sucks. But the reality is that this is far from trivial
to resolve without introducing pre-mature OOM killer invocations.
-- 
Michal Hocko
SUSE Labs

Re: DOS by unprivileged user

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

On Sun 2018-04-22 18:27:38, Michal Hocko wrote:
> On Sun 22-04-18 10:43:00, vcaputo@pengaru.com wrote:
> > On Sun, Apr 22, 2018 at 12:16:54PM +0200, Pavel Machek wrote:
> > > On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> > > > It appears any ordinary user can easily create a DOS on linux.
> > > > 
> > > > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > > > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > > > + hyperthreading. But I will be easy enough to get the same effect with more 
> > > > RAM, other fs etc.
> > > 
> > > You may want to disable swap.
> > > 
> > 
> > I run without swap on my laptops, and still observe long periods of
> > thrashing on the road towards OOM.  What seems to occur is the active
> > file-backed mappings of executables/libraries become a sort of swap
> > area, repeatedly being discarded and faulted back in as the context
> > switches occur.
> > 
> > If there's any good way to prevent this, I'd like to know.
> 
> I am afraid there is none yet. Johannes had some ground work for
> page cache trashing detection https://marc.info/?i=20170727153010.23347-1-hannes%40cmpxchg.org
> but there was no version of the patchseries for quite some time and
> there was no integration into the oom detection which would be
> non-trivial as well.
> 
> I realize this sucks. But the reality is that this is far from trivial
> to resolve without introducing pre-mature OOM killer invocations.

Another problem is that what "unusable machine" in X/web browser
situation may be normal load for build server...

I guess one way would be "hey, this is my X server; if it is waiting
for disk for more than 10 seconds, you probably want to OOM kill
someone. Ouch and same goes for my window manager". 

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: DOS by unprivileged user

[Mike Galbraith]

On Sun, 2018-04-22 at 21:37 +0200, Ferry Toth wrote:
> > Yes your memory hog scenario thoroughly wrecks the user experience, but
> > the process scheduler in not the source of that wreckage, it's a memory
> > management issue.  With no constraints in place, anybody can just keep
> > on allocating until the entire system starts grinding itself to dust.
> > 
> > 	-Mike
>  
> That is exactly the issue I think. It is not just a user experience,
> they is no distinction between crashing the kernel and grinding it to
> dust. The effect we have is: any user on a multi user system can
> crash the system.
>  
> Memory management / constraints can't solve the problem either: it
> might be intentional to alloc such amounts of memory.

Constraints definitely can solve this particular problem instance. 
Plain old ulimit can stop the memory hog (wish) dead in its tracks, or
you can use memory cgroup controller to constrain it in a more friendly
manner.  I started gitk in a 4G constrained cgroup, which redirected
its greedy fingers to the swap bin for the remainder of its needs.

But yes, there is currently no wonderful one size fits all fully
automatic solution to running low on memory that doesn't involve
running to the store.

> I think memory allocation and io waits can't be decoupled from
> scheduling as they are now.

The scheduler is not decoupled from either, it is intimately involved
in both.  However, none of the decision making smarts for either reside
in the scheduler, nor should they.

	-Mike

Re: DOS by unprivileged user

[Alan Cox]

> > I think memory allocation and io waits can't be decoupled from
> > scheduling as they are now.  
> 
> The scheduler is not decoupled from either, it is intimately involved
> in both.  However, none of the decision making smarts for either reside
> in the scheduler, nor should they.

It belongs in both.

Classical Unix systems never had this problem because they respond to
thrashing by ensuring that all processes consumed CPU and made some
progress. Linux handles it by thrashing itself to dealth while BSD always
handled it by moving from paging more towards swapping and behaving like
a swap bound batch machine.

Linux thrashes itself to death, the classic BSD algorithn instead throws
fairness out of the window under extreme load to prevent it. It might take
a few seconds but at least you will get your prompt back.

Alan

Re: DOS by unprivileged user

[Mike Galbraith]

On Wed, 2018-04-25 at 15:54 +0100, Alan Cox wrote:
> > > I think memory allocation and io waits can't be decoupled from
> > > scheduling as they are now.  
> > 
> > The scheduler is not decoupled from either, it is intimately involved
> > in both.  However, none of the decision making smarts for either reside
> > in the scheduler, nor should they.
> 
> It belongs in both.

If mm decision making belongs within the process scheduler, it follows
that IO requests, dirty page writeback etc. do as well.  Nope, I don't
think we want to create a squid-uler, with tentacles extending all over
the dang kernel.

The thrashing problem could use some attention, but we'll have to agree
to disagree about the scheduler growing mm, io (etc) smarts.

	-Mike

Re: DOS by unprivileged user

[Mike Galbraith]

On Wed, 2018-04-25 at 15:54 +0100, Alan Cox wrote:
> 
> Classical Unix systems never had this problem because they respond to
> thrashing by ensuring that all processes consumed CPU and made some
> progress. Linux handles it by thrashing itself to dealth while BSD always
> handled it by moving from paging more towards swapping and behaving like
> a swap bound batch machine.

Memcg constrained the gitk hog nicely, forcing it do dig into swap
after hitting its limit.  Dunno if there are any userspace bits that
can use it wisely though.

	-Mike

Re: DOS by unprivileged user

[Ferry Toth]

Op woensdag 25 april 2018 16:54:59 CEST schreef Alan Cox:
> > > I think memory allocation and io waits can't be decoupled from
> > > scheduling as they are now.  
> > 
> > The scheduler is not decoupled from either, it is intimately involved
> > in both.  However, none of the decision making smarts for either reside
> > in the scheduler, nor should they.
> 
> It belongs in both.
> 
> Classical Unix systems never had this problem because they respond to
> thrashing by ensuring that all processes consumed CPU and made some
> progress. Linux handles it by thrashing itself to dealth while BSD always
> handled it by moving from paging more towards swapping and behaving like
> a swap bound batch machine.
> 
> Linux thrashes itself to death, the classic BSD algorithn instead throws
> fairness out of the window under extreme load to prevent it. It might take
> a few seconds but at least you will get your prompt back.
> 
> Alan
> 
I haven\0t tried BSD.

But when I was young I allocated 10MB on a HP9000 (UX) with 1MB of RAM. People wanted to launch me out of the window (18th floor).

I did not want to say Unix was better, only with so much emphasis on security I'\0m surprised how easy it is for a regular user to bring linux to on it\0s knees.

Re: DOS by unprivileged user

[Miguel Ojeda]

On Mon, Apr 30, 2018 at 12:00 PM, Ferry Toth <ftoth@telfort.nl> wrote:
> Op woensdag 25 april 2018 16:54:59 CEST schreef Alan Cox:
>> > > I think memory allocation and io waits can't be decoupled from
>> > > scheduling as they are now.
>> >
>> > The scheduler is not decoupled from either, it is intimately involved
>> > in both.  However, none of the decision making smarts for either reside
>> > in the scheduler, nor should they.
>>
>> It belongs in both.
>>
>> Classical Unix systems never had this problem because they respond to
>> thrashing by ensuring that all processes consumed CPU and made some
>> progress. Linux handles it by thrashing itself to dealth while BSD always
>> handled it by moving from paging more towards swapping and behaving like
>> a swap bound batch machine.
>>
>> Linux thrashes itself to death, the classic BSD algorithn instead throws
>> fairness out of the window under extreme load to prevent it. It might take
>> a few seconds but at least you will get your prompt back.
>>
>> Alan
>>
> I haven t tried BSD.
>
> But when I was young I allocated 10MB on a HP9000 (UX) with 1MB of RAM. People wanted to launch me out of the window (18th floor).
>
> I did not want to say Unix was better, only with so much emphasis on security I' m surprised how easy it is for a regular user to bring linux to on it s knees.

While it is true that things can be improved/tweaked for typical
desktop/single user usage; this isn't really a security issue. For
shared systems, there are a few ways to soft/hard limit resources:
nice, *limit, cgroups, systemd limits, containers/VMs...

Cheers,
Miguel