archive mirror
 help / color / mirror / Atom feed
From: Alex Belits <>
To: "" <>
Cc: "" <>,
	"" <>,
	"" <>,
	"" <>,
	"" <>,
	"" <>,
	"" <>,
	"" <>
Subject: Re: [EXT] Re: [PATCH] mm: introduce sysctl file to flush per-cpu vmstat statistics
Date: Thu, 3 Dec 2020 22:21:47 +0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <20201130181858.GA5924@fuller.cnet>

On Mon, 2020-11-30 at 15:18 -0300, Marcelo Tosatti wrote:

> Two questions:
> What is the reason for not allowing _any_ interruption? Because 
> it might be the case (and it is in the vRAN use cases our customers
> target), that some amount of interruption is tolerable.
> For example, for one use case 20us (of interruptions) every 1ms of 
> elapsed time is tolerable.

I agree that for many applications that would be sufficient, and if so,
we can avoid the rest of the mechanism, However I know three types of
applications where full isolation is absolutely necessary.

1. Physical process control. A program continuously polls data from
sensors, processes it, determines the state of controlled device or
process, and issues commands or tunes parameters, controlling it. An
unpredictable delay caused by an interrupt or other
performance-affecting event can break the timing of this mechanism.

Even taking into account that usually safety requirements demand that
some low-latency reaction should be performed without CPU involvement,
some of the control logic may demand involvement of a CPU usually
within a closed control loop. For example, tracking the state of an RF
connection and adjusting the signal, analysis of measurements and
control of a mechanical system, including elasticity and vibration in
industrial and transportation applications, all require a large amount
of calculations, and benefit from reduced amount of buffering and
delays. In case of RF this allows to maintain more reliable
connections. Mechanical systems can have greater precision and lower
stress on the components because such system can easily implement
dampening and compensation of deformation and deterioration.

Usually when such processing is necessary, the "fast" part of it is
performed on a microcontroller, separate CPU core built into an ASIC or
even a "soft CPU" implemented on an FPGA, while "slow" calculations are
performed by a large CPU or SoC. This, however, places severe
limitations on the software design and requires engineers to waste
"expensive" resources such as including PCIe links into
microcontrollers, building ASICs, implementing soft CPU cores on FPGA,
etc. while the use of CPU cores of modern CPUs and SoC is limited.
Sometimes a large processing mechanism is built out of DSP blocks of
FPGA and its built-in static RAM, while more capable SIMD of a CPU
remains under-used.

2. Audio processing. A program implements a set of filters and mixers,
processing multiple audio streams in a live performance. Processing
should be synchronized not only between individual microphones,
speakers and processors, but, when used as a part of sound
reinforcement system, also with propagation of sound through the
environment that bypasses the audio system but is subject to the
acoustics of the building. The amount of data is large, however
calculations are always the same, so it is possible to tune all delays
to match the projected or measured timing. Program runs at the sample
rate of the signals, with processing algorithms already creating
delays, so additional buffering would place total delay out of the
available range.

This can be seen as a subset of controlling the mechanical system,
except usually a control loop, if present, is much slower (such as
adapting to changing acoustical properties), however processing of
incoming data includes very large amounts of calculations that should
be performed on a full-featured CPU. The total amount of data that
should be available to a CPU core at a time, usually can fit into its
cache, and if sufficient cache allocation or partitioning is available,
all necessary calculations (digital filters, mixing, etc.) can be
reliably performed with minimal latency. Interrupts break this model,
causing unpredictable cache invalidation, so design has to take into
account not just the time spent processing the interrupt but also
delays for re-filling the caches with data that normally remains there.

3. Parallel handling of network packets. Modern network-oriented SoCs
have hardware scheduling mechanisms that can distribute packets
processing, including packet processing within the same flow, to
multiple CPU cores, form processing pipelines, etc. Flow-related state,
if present, can be kept within the scheduling mechanism, and not
handled by CPU cores. When CPUs work in a predictable manner, this
allows fast processing of network traffic regardless of the
distribution of the amount of data between flows or other details, save
for situations that warrant some kind of additional per-flow processing
that can't be performed in a stateless manner (and that may be still
offloaded to dedicated CPU cores for "slow path"). This is
fundamentally different from the model in which each flow is kept on a
single CPU for stateful processing and buffering.

When interrupt happens, a core is temporarily excluded from this
orchestrated process, and the packets that it handles stay in buffer,
possibly keeping the whole flows waiting for reordering before the
output. Every time this happens, buffers end up holding more data than
they would be without interruption, so the whole system can provide
lower throughput before it has to drop packets. Handling network
buffers and CPU access to them is usually extremely optimized in such
system, CPU usually only accesses a subset of header data in network
buffers, then makes whatever decisions that are not already done by a
complex system of hardware header parsers, CAMs and queues that precede
and follow this CPU-based processing.

In many cases all three types applications suffer from insufficiently
flexible cache management, however recently many CPUs and SoCs are
built specifically for the above mentioned purposes, and more complex
cache management can be easily added to them. When implementing those
it's much easier to allow optimization for running an application with
predictable limits on access, that will fall back into un-optimized
state when a predictable pattern is broken. This means, designers of
both hardware and software can breathe much easier if OS can guarantee
interrupt-free environment for timing-critical tasks. Outside of those
tasks it would be completely harmless for cache to get out of the
optimized state as long as consistency is maintained by hardware, so
there is no reason to expect any incompatibility or violation of
specifications that would require any special software support. However
as a person working on the software side of a hardware company, I can't
request such a feature when I know that whatever they will build, will
have to crash down like a Jenga tower every second, leave alone

To support this kind of tasks, various companies used either full-
featured RTOS or "bare metal environments" that are basically
specialized extremely reduced OSes, designed to run up to one
application per CPU core with some minimal initialization and resource
allocation. While I agree that this is not the place to evaluate merit
of those approaches in general, I can say that my recent experience
with both developers and users, including but not limited to myself
being in both roles, shows that many would rather prefer if Linux
provided this environment when running some tasks after initialization
in a normal mode. If the task can go back and forth between isolated
and non-isolated mode, that would greatly simplify re-configuration,
and any mechanism for communicating outside of isolated environment can
be a bonus, however isolation itself is the important part.

This was not the case in early 2000s, when embedded systems were built
on single-core SoCs and microcontrollers. Then either everything had to
be integrated in a single task under "bare metal", or everything
depended on interrupt processing provided by the OS. Now, when network
and embedded-oriented chips are built with tens of cores, sometimes
it's better if OS prepared everything and left the task, along with its
core and devices that it handles, alone and undisturbed. And if this
can be done entirely within Linux, consistently and integrated with
other related features and design of its subsystems, I think, this is

> > Since I first and foremost care about eliminating all disturbances
> > for
> > a running userspace task,
> Why?

Mostly because I work on (3) from the list above. However experience
with other mentioned uses contributed to that, too. I think, some very
useful work is already done in Linux for limiting interrupts and
disturbances in general and for isolated CPUs in particular. One
additional step that will allow to provide a disturbances-free
environment and greatly expand possible applications of Linux kernel
looks to me like a useful applications of more efforts.

> > my approach is to allow disabling everything
> > including "unavoidable" synchronization IPIs, and make kernel entry
> > procedure recognize that some delayed synchronization is necessary
> > while avoiding race conditions. As far as I can tell, not everyone
> > wants to go that far, 
> Suppose it depends on how long each interruption takes. Upon the 
> suggestion from Thomas and Frederic, i've been thinking it should 
> be possible, with the tracing framework, to record the length of 
> all interruptions to a given CPU and, every second check how many 
> have happened (and whether the sum of interruptions exceeds the 
> acceptable threshold).
> Its not as nice as the task isolation patch (in terms of spotting
> the culprit, since one won't get a backtrace on the originating
> CPU in case of an IPI for example), but it should be possible 
> to verify that the latency threshold is not broken (which is
> what the application is interested in, isnt it?).

With the current version of the isolation patch I have completely
removed the diagnostics part of the code. I want to re-introduce a much
cleaner implementation of "interrupt cause diagnostics" on top of it,
mostly for the purpose of development -- at least I think, it will give
me a better tool to chase the root causes of interrupts that task
isolation can see.

Now that I have functions where all kernel entry and exit passes for
isolated tasks, and some easily readable indication of an isolated CPU,
it should be easy to collect both "We are going to send an IPI to a
task in this special state because..." and "We are doing something
specific in kernel, and it happens, our task was in this special
state". It may be that just using tracing this can be done as well,
however if we will expand conditions for cause recording beyond just
fully isolated task, a separate recording mechanism would be able to
collect useful information and statistics with less of its own impact
on latency and performance. At least, I think, it makes sense to try
separately from task isolation implementation itself, that no longer
depends on knowing why kernel was entered.

As for measuring latency, as I have mentioned before, it's not just the
impact of time being spent in kernel in interrupt processing that
matters, it's also all the memory that is being accessed in the process
of doing so, and whatever performance-affecting changes, such as TLB
flushes or expected delayed work, that made the call necessary in the
first place. And for non-isolated tasks there is also a possibility of
preemption that can further complicate the picture with timing.

> > and it may make sense to allow "almost isolated
> > tasks" that still receive normal interrupts, including IPIs and
> > page
> > faults. 
> > 
> > While that would be useless for the purposes that task
> > isolation patch was developed for, I recognize that some might
> > prefer
> > that to be one of the options set by the same prctl call. This
> > still
> > remains close enough to the design of task isolation -- same idea
> > of
> > something that affects CPU but being tied to a given task (and
> > dying
> > with it), same model of handling attributes, etc.
> > 
> > Maybe there can be a mask of what we do and don't want to avoid for
> > the
> > task. Say, some may want to only allow page faults or syscalls. Or
> > re-
> > enter isolation on breaking without notifying the userspace.
> OK, i will try to come up with an interface that allows additional 
> attributes - please review and let me know if it works for task
> isolation patchset.

I think, in addition to enable/disable and configuring a signal we will
need some flags that enable and disable features, such as full
isolation and avoidance of specific events and causes, logging (if
supported), etc.

> Can you talk a little about the signal handling part? What type
> of applications are expected to perform once isolation is broken?

First and foremost, a task may be killed because a signal was sent to
it by another task, terminal, socket or pipe. Then it's completely
normal signal processing, possibly a userspace cleanup, reloading,
etc., except isolation breaking happens on kernel entry.

Second, application may encounter an early exit from isolation
immediately after entering because some kernel event that was supposed
to cause IPIs to all non-isolated tasks, happened while it was
entering. Then all it has to do is re-enter.

Third, and this is what I do in libtmc, isolation breaking is handled
by notifying a separate isolation manager thread or process, so it can
check the state of tasks and timers, and decide if and when it should
attempt re-enter, ask the task to leave and re-enter the CPU, etc. I
will be happier if those actions became unnecessary, however for now
it's the easiest way to check if anything and what in particular may
prevent the task from entering isolation. Maybe it will make sense to
make libtmc use ptrace() on a task from isolation manager, so isolation
manager will handle that signal when necessary, and then notify the

As an alternative I have added a simple readable isolation mask that
can be checked by processes (ones that are not isolated) such as
isolation manager, and if there will be a separate mechanism for
reporting isolation breaking, it may be used instead of the signal (and
then isolation manager will be able to communicate with the task using
shared memory, signal or other means).

The original patch used a signal, and I can't predict what else users
may want to do on isolation breaking, so I have kept that interface


  parent reply	other threads:[~2020-12-03 22:22 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-17 16:28 Marcelo Tosatti
2020-11-17 18:03 ` Matthew Wilcox
2020-11-17 19:06   ` Christopher Lameter
2020-11-17 19:09     ` Matthew Wilcox
2020-11-20 18:04       ` Christopher Lameter
2020-11-17 20:23     ` Marcelo Tosatti
2020-11-20 18:02       ` Marcelo Tosatti
2020-11-20 18:20       ` Christopher Lameter
2020-11-23 18:02         ` Marcelo Tosatti
2020-11-24 17:12         ` Vlastimil Babka
2020-11-24 19:52           ` Marcelo Tosatti
2020-11-27 15:48         ` Marcelo Tosatti
2020-11-28  3:49           ` [EXT] " Alex Belits
2020-11-30 18:18             ` Marcelo Tosatti
2020-11-30 18:29               ` Marcelo Tosatti
2020-12-03 22:47                 ` Alex Belits
2020-12-03 22:21               ` Alex Belits [this message]
2020-11-30  9:31           ` Christoph Lameter
2020-12-02 12:43             ` Marcelo Tosatti
2020-12-02 15:57             ` Thomas Gleixner
2020-12-02 17:43               ` Christoph Lameter
2020-12-03  3:17                 ` Thomas Gleixner
2020-12-07  8:08                   ` Christoph Lameter
2020-12-07 16:09                     ` Thomas Gleixner
2020-12-07 19:01                       ` Thomas Gleixner
2020-12-02 18:38               ` Marcelo Tosatti
2020-12-04  0:20                 ` Frederic Weisbecker
2020-12-04 13:31                   ` Marcelo Tosatti
2020-12-04  1:43               ` [EXT] " Alex Belits
2021-01-13 12:15                 ` [RFC] tentative prctl task isolation interface Marcelo Tosatti
2021-01-14  9:22                   ` Christoph Lameter
2021-01-14 19:34                     ` Marcelo Tosatti
2021-01-15 13:24                       ` Christoph Lameter
2021-01-15 18:35                         ` Alex Belits
2021-01-21 15:51                           ` Marcelo Tosatti
2021-01-21 16:20                             ` Marcelo Tosatti
2021-01-22 13:05                               ` Marcelo Tosatti
2021-02-01 10:48                             ` Christoph Lameter
2021-02-01 12:47                               ` Alex Belits
2021-02-01 18:20                               ` Marcelo Tosatti
2021-01-18 15:18                         ` Marcelo Tosatti
2020-11-24  5:02 ` [mm] e655d17ffa: BUG:using_smp_processor_id()in_preemptible kernel test robot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [EXT] Re: [PATCH] mm: introduce sysctl file to flush per-cpu vmstat statistics' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).