archive mirror
 help / color / mirror / Atom feed
From: Alex Belits <>
To: Christoph Lameter <>, Marcelo Tosatti <>
Cc: "" <>,
	"" <>,
	"" <>,
	"" <>,
	"" <>,
	"" <>,
	"" <>,
	Juri Lelli <>,
	Daniel Bristot de Oliveira <>
Subject: Re: [RFC] tentative prctl task isolation interface
Date: Mon, 1 Feb 2021 04:47:04 -0800	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On 2/1/21 02:48, Christoph Lameter wrote:
>> Notifications:
>> -------------
>> Notification mode of isolation breakage can be configured as follows:
>> - None (default): No notification is performed by the kernel on isolation
>>    breakage.
>> - Syslog: Isolation breakage is reported to syslog.

Syslog is intended for humans, and isn't useful for userspace software 
processing. Since there are at least some cases then isolation breaking 
is unavoidable on startup (benign race of isolation entering with 
isolation-breaking event, register-mapping page fault), I would rather 
allow completely automated processing of those events. Signal interface 
does that now, however I think, it would help to associate 
software-handled events with either software-identifiable "cause type" 
(ex: "scheduling timer" or "page fault") or more verbose human-readable 
"cause description" (ex: IPI received, and here is the sender CPU's 
stack dump that led to this IPI being sent).

The former ("cause") may be important for software (for example, it may 
want to have special processing of page faults for device registers), 
while the latter ("description") is more useful when it can be 
associated with particular event in userspace without manual log timing 
comparison and guesswork.

> - Abort with core dump

I would use an existing signal interface for that, with user-defined 
signal. The user can choose to handle the signal, ignore it, let it kill 
the task with or without a core dump.

Oh, and if user wants, he can use ptrace() to delegate this signal to 
some other process.

> This is useful for debugging and for hard core bare metalers that never
> want any interrupts.
> One particular issue are page faults.  One would have to prefault the
> binary executable functions in order to avoid "interruptions" through page
> faults. Are these proper interrutions of the code? Certainly major faults
> are but minor faults may be ok? Dunno.
> In practice what I have often seen in such apps is that there is a "warm"
> up mode where all critical functions are executed, all important variables
> are touched and dummy I/Os are performed in order to populate the caches
> and prefault all the data.I guess one would run these without isolation
> first and then switch on some sort of isolation mode after warm up. So far
> I think most people relied on the timer interrupt etc etc to be turned off
> after a few secs of just running throught a polling loop without any OS
> activities.

This is usually done not as much for page preloading but for cache. 
There is mlock() and mlockall() that load and lock pages explicitly. One 
exception is device registers -- they may remain unmapped until accessed.

I can often see a pattern when application enters isolation, calls 
low-level library such as ODP, gets a page fault, leaves and re-enters 
isolation, and then everything is running perfectly because everything 
is mapped. However in those cases mlockall() is done before entering 
isolation, so regular memory mapping is already there.

>>> I ended up implementing a manager/helper task that talks to tasks over a
>>> socket (when they are not isolated) and over ring buffers in shared memory
>>> (when they are isolated). While the current implementation is rather
>>> limited, the intention is to delegate to it everything that isolated task
>>> either can't do at all (like, writing logs) or that it would be cumbersome
>>> to implement (like monitoring the state of task, determining presence of
>>> deferred work after the task returned to userspace), etc.
>> Interesting. Are you considering opensourcing such library? Seems like a
>> generic problem.

It's already open source,

It still needs some work. At the moment it does more than I would prefer 
because it tries to detect possible problems, such as running timers, 
and at the same time does not provide some obviously useful things like 
asynchronous interface to arbitrary file I/O.

I also want to allow the use of some generic interface to triggering 
interrupts from isolated task to the manager (through, say, a sacrifice 
of a single GPIO), so if this option is available, the manager won't 
have to do all that polling.

> Well everyone swears on having the right implementation. The people I know
> would not do any thing with a socket in such situations. They would only
> use shared memory and direct access to I/O devices via SPDK and DPDK or
> the RDMA subsystem.

Same applies to me. My library uses sockets to communicate when the task 
is not isolated, and it will be necessary if we want to have a dedicated 
manager process instead of a manager thread in every process. I would 
prefer initiating a connection with a manager through a socket, and only 
after that succeeds, assume that I can use any particular part of shared 
memory (because it means that manager allocated it for me, and no one 
else will race with me trying to touch it).

>>>> Blocking? The app should fail if any deferred actions are triggered as a
>>>> result of syscalls. It would give a warning with _WARN
>>> There are many supposedly innocent things, nowhere at the scale of CPU
>>> hotplug, that happen in a system and result in synchronization implemented
>>> as an IPI to every online CPU. We should consider them to be an ordinary
>>> occurrence, so there is a choice:
>>> 1. Ignore them completely and allow them in isolated mode. This will delay
>>> userspace with no indication and no isolation breaking.
>>> 2. Allow them, and notify userspace afterwards (through vdso or through
>>> userspace helper/manager over shared memory). This may be useful in those
>>> rare situations when the consequences of delay can be mitigated afterwards.
>>> 3. Make them break isolation, with userspace being notified normally (ex:
>>> with a signal in the current implementation). I guess, can be used if
>>> somehow most of the causes will be eliminated.
>>> 4. Prevent them from reaching the target CPU and make sure that whatever
>>> synchronization they are intended to cause, will happen when intended target
>>> CPU will enter to kernel later. Since we may have to synchronize things like
>>> code modification, some of this synchronization has to happen very early on
>>> kernel entry.
> Or move the actions to a different victim processor like done with rcu and
> vmstat etc etc.

If possible. For most of those things everything can be moved to other 
CPUs when entering for isolation, or not allowed on CPUs intended for 
isolation in the first place (how it's mostly done now). The troublesome 
sources of interruption are things that are legitimately supposed to be 
done on all CPUs at once to synchronize some important kind of state, 
and now we want to delay them on some CPUs until the end of isolation.

>>> I am most interested in (4), so this is what was implemented in my version
>>> of the patch (and currently I am trying to achieve completeness and, if
>>> possible, elegance of the implementation).
>> Agree. (3) will be necessary as intermediate step. The proposed
>> improvement to Christoph's reply, in this thread, separates notification
>> and syscall blockage.
> I guess the notification mode will take care of the way we handle these
> interruptions.

I think, development should go in parallel -- to have a "delayed 
synchronization on entry" mechanism that allows "no-interruption mode" 
(4) to work given that all interruptions are dealt with (that won't work 
perfectly at first because there are still "unprocessed" sources of 
interruptions) and a notification mechanism that will allow us to find 
and properly process them as (3), so we can exclude them and allow (4). 
Since (4) still requires somewhat intrusive architecture-specific 
changes, there may be some time when (4) will be only available on some 
CPUs, but (3) will work on everything.


  reply	other threads:[~2021-02-01 12:47 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-17 16:28 [PATCH] mm: introduce sysctl file to flush per-cpu vmstat statistics Marcelo Tosatti
2020-11-17 18:03 ` Matthew Wilcox
2020-11-17 19:06   ` Christopher Lameter
2020-11-17 19:09     ` Matthew Wilcox
2020-11-20 18:04       ` Christopher Lameter
2020-11-17 20:23     ` Marcelo Tosatti
2020-11-20 18:02       ` Marcelo Tosatti
2020-11-20 18:20       ` Christopher Lameter
2020-11-23 18:02         ` Marcelo Tosatti
2020-11-24 17:12         ` Vlastimil Babka
2020-11-24 19:52           ` Marcelo Tosatti
2020-11-27 15:48         ` Marcelo Tosatti
2020-11-28  3:49           ` [EXT] " Alex Belits
2020-11-30 18:18             ` Marcelo Tosatti
2020-11-30 18:29               ` Marcelo Tosatti
2020-12-03 22:47                 ` Alex Belits
2020-12-03 22:21               ` Alex Belits
2020-11-30  9:31           ` Christoph Lameter
2020-12-02 12:43             ` Marcelo Tosatti
2020-12-02 15:57             ` Thomas Gleixner
2020-12-02 17:43               ` Christoph Lameter
2020-12-03  3:17                 ` Thomas Gleixner
2020-12-07  8:08                   ` Christoph Lameter
2020-12-07 16:09                     ` Thomas Gleixner
2020-12-07 19:01                       ` Thomas Gleixner
2020-12-02 18:38               ` Marcelo Tosatti
2020-12-04  0:20                 ` Frederic Weisbecker
2020-12-04 13:31                   ` Marcelo Tosatti
2020-12-04  1:43               ` [EXT] " Alex Belits
2021-01-13 12:15                 ` [RFC] tentative prctl task isolation interface Marcelo Tosatti
2021-01-14  9:22                   ` Christoph Lameter
2021-01-14 19:34                     ` Marcelo Tosatti
2021-01-15 13:24                       ` Christoph Lameter
2021-01-15 18:35                         ` Alex Belits
2021-01-21 15:51                           ` Marcelo Tosatti
2021-01-21 16:20                             ` Marcelo Tosatti
2021-01-22 13:05                               ` Marcelo Tosatti
2021-02-01 10:48                             ` Christoph Lameter
2021-02-01 12:47                               ` Alex Belits [this message]
2021-02-01 18:20                               ` Marcelo Tosatti
2021-01-18 15:18                         ` Marcelo Tosatti
2020-11-24  5:02 ` [mm] e655d17ffa: BUG:using_smp_processor_id()in_preemptible kernel test robot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [RFC] tentative prctl task isolation interface' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox