linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v7 00/10] mm: reparent slab memory on cgroup removal
@ 2019-06-11 23:18 Roman Gushchin
  2019-06-11 23:18 ` [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Roman Gushchin
                   ` (9 more replies)
  0 siblings, 10 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

# Why do we need this?

We've noticed that the number of dying cgroups is steadily growing on most
of our hosts in production. The following investigation revealed an issue
in the userspace memory reclaim code [1], accounting of kernel stacks [2],
and also the main reason: slab objects.

The underlying problem is quite simple: any page charged to a cgroup holds
a reference to it, so the cgroup can't be reclaimed unless all charged pages
are gone. If a slab object is actively used by other cgroups, it won't be
reclaimed, and will prevent the origin cgroup from being reclaimed.

Slab objects, and first of all vfs cache, is shared between cgroups, which are
using the same underlying fs, and what's even more important, it's shared
between multiple generations of the same workload. So if something is running
periodically every time in a new cgroup (like how systemd works), we do
accumulate multiple dying cgroups.

Strictly speaking pagecache isn't different here, but there is a key difference:
we disable protection and apply some extra pressure on LRUs of dying cgroups,
and these LRUs contain all charged pages.
My experiments show that with the disabled kernel memory accounting the number
of dying cgroups stabilizes at a relatively small number (~100, depends on
memory pressure and cgroup creation rate), and with kernel memory accounting
it grows pretty steadily up to several thousands.

Memory cgroups are quite complex and big objects (mostly due to percpu stats),
so it leads to noticeable memory losses. Memory occupied by dying cgroups
is measured in hundreds of megabytes. I've even seen a host with more than 100Gb
of memory wasted for dying cgroups. It leads to a degradation of performance
with the uptime, and generally limits the usage of cgroups.

My previous attempt [3] to fix the problem by applying extra pressure on slab
shrinker lists caused a regressions with xfs and ext4, and has been reverted [4].
The following attempts to find the right balance [5, 6] were not successful.

So instead of trying to find a maybe non-existing balance, let's do reparent
accounted slab caches to the parent cgroup on cgroup removal.


# Implementation approach

There is however a significant problem with reparenting of slab memory:
there is no list of charged pages. Some of them are in shrinker lists,
but not all. Introducing of a new list is really not an option.

But fortunately there is a way forward: every slab page has a stable pointer
to the corresponding kmem_cache. So the idea is to reparent kmem_caches
instead of slab pages.

It's actually simpler and cheaper, but requires some underlying changes:
1) Make kmem_caches to hold a single reference to the memory cgroup,
   instead of a separate reference per every slab page.
2) Stop setting page->mem_cgroup pointer for memcg slab pages and use
   page->kmem_cache->memcg indirection instead. It's used only on
   slab page release, so performance overhead shouldn't be a big issue.
3) Introduce a refcounter for non-root slab caches. It's required to
   be able to destroy kmem_caches when they become empty and release
   the associated memory cgroup.

There is a bonus: currently we release all memcg kmem_caches all together
with the memory cgroup itself. This patchset allows individual kmem_caches
to be released as soon as they become inactive and free.

Some additional implementation details are provided in corresponding
commit messages.


# Results

Below is the average number of dying cgroups on two groups of our production
hosts. They do run some sort of web frontend workload, the memory pressure
is moderate. As we can see, with the kernel memory reparenting the number
stabilizes in 60s range; however with the original version it grows almost
linearly and doesn't show any signs of plateauing. The difference in slab
and percpu usage between patched and unpatched versions also grows linearly.
In 7 days it exceeded 200Mb.

day           0    1    2    3    4    5    6    7
original     56  362  628  752 1070 1250 1490 1560
patched      23   46   51   55   60   57   67   69
mem diff(Mb) 22   74  123  152  164  182  214  241


# History

v7:
  1) refined cover letter and some commit logs
  2) dropped patch 1
  3) dropped the dying check on kmem_cache creation path
  4) dropped __rcu annotation in patch 10, switched to READ_ONCE()/WRITE_ONCE()
     where is necessary

v6:
  1) split biggest patches into parts to make the review easier
  2) changed synchronization around the dying flag
  3) sysfs entry removal on deactivation is back
  4) got rid of redundant rcu wait on kmem_cache release
  5) fixed getting memcg pointer in mem_cgroup_from_kmem()
  5) fixed missed smp_rmb()
  6) removed redundant CONFIG_SLOB
  7) some renames and cosmetic fixes

v5:
  1) fixed a compilation warning around missing kmemcg_queue_cache_shutdown()
  2) s/rcu_read_lock()/rcu_read_unlock() in memcg_kmem_get_cache()

v4:
  1) removed excessive memcg != parent check in memcg_deactivate_kmem_caches()
  2) fixed rcu_read_lock() usage in memcg_charge_slab()
  3) fixed synchronization around dying flag in kmemcg_queue_cache_shutdown()
  4) refreshed test results data
  5) reworked PageTail() checks in memcg_from_slab_page()
  6) added some comments in multiple places

v3:
  1) reworked memcg kmem_cache search on allocation path
  2) fixed /proc/kpagecgroup interface

v2:
  1) switched to percpu kmem_cache refcounter
  2) a reference to kmem_cache is held during the allocation
  3) slabs stats are fixed for !MEMCG case (and the refactoring
     is separated into a standalone patch)
  4) kmem_cache reparenting is performed from deactivatation context

v1:
  https://lkml.org/lkml/2019/4/17/1095


# Links

[1]: commit 68600f623d69 ("mm: don't miss the last page because of
round-off error")
[2]: commit 9b6f7e163cd0 ("mm: rework memcg kernel stack accounting")
[3]: commit 172b06c32b94 ("mm: slowly shrink slabs with a relatively
small number of objects")
[4]: commit a9a238e83fbb ("Revert "mm: slowly shrink slabs
with a relatively small number of objects")
[5]: https://lkml.org/lkml/2019/1/28/1865
[6]: https://marc.info/?l=linux-mm&m=155064763626437&w=2


Roman Gushchin (10):
  mm: postpone kmem_cache memcg pointer initialization to
    memcg_link_cache()
  mm: rename slab delayed deactivation functions and fields
  mm: generalize postponed non-root kmem_cache deactivation
  mm: introduce __memcg_kmem_uncharge_memcg()
  mm: unify SLAB and SLUB page accounting
  mm: don't check the dying flag on kmem_cache creation
  mm: synchronize access to kmem_cache dying flag using a spinlock
  mm: rework non-root kmem_cache lifecycle management
  mm: stop setting page->mem_cgroup pointer for slab pages
  mm: reparent memcg kmem_caches on cgroup removal

 include/linux/memcontrol.h |  10 +++
 include/linux/slab.h       |  11 +--
 mm/list_lru.c              |   3 +-
 mm/memcontrol.c            | 101 ++++++++++++++++-------
 mm/slab.c                  |  25 ++----
 mm/slab.h                  | 143 +++++++++++++++++++++++---------
 mm/slab_common.c           | 164 ++++++++++++++++++++++---------------
 mm/slub.c                  |  24 +-----
 8 files changed, 301 insertions(+), 180 deletions(-)

-- 
2.21.0


^ permalink raw reply	[flat|nested] 36+ messages in thread

* [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache()
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-13  2:04   ` Andrew Morton
  2019-06-11 23:18 ` [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields Roman Gushchin
                   ` (8 subsequent siblings)
  9 siblings, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Initialize kmem_cache->memcg_params.memcg pointer in
memcg_link_cache() rather than in init_memcg_params().

Once kmem_cache will hold a reference to the memory cgroup,
it will simplify the refcounting.

For non-root kmem_caches memcg_link_cache() is always called
before the kmem_cache becomes visible to a user, so it's safe.

Signed-off-by: Roman Gushchin <guro@fb.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/slab.c        |  2 +-
 mm/slab.h        |  5 +++--
 mm/slab_common.c | 14 +++++++-------
 mm/slub.c        |  2 +-
 4 files changed, 12 insertions(+), 11 deletions(-)

diff --git a/mm/slab.c b/mm/slab.c
index 9e3eee5568b6..a4091f8b3655 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -1239,7 +1239,7 @@ void __init kmem_cache_init(void)
 				  nr_node_ids * sizeof(struct kmem_cache_node *),
 				  SLAB_HWCACHE_ALIGN, 0, 0);
 	list_add(&kmem_cache->list, &slab_caches);
-	memcg_link_cache(kmem_cache);
+	memcg_link_cache(kmem_cache, NULL);
 	slab_state = PARTIAL;
 
 	/*
diff --git a/mm/slab.h b/mm/slab.h
index 739099af6cbb..86f7ede21203 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -289,7 +289,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,
 }
 
 extern void slab_init_memcg_params(struct kmem_cache *);
-extern void memcg_link_cache(struct kmem_cache *s);
+extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg);
 extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
 				void (*deact_fn)(struct kmem_cache *));
 
@@ -344,7 +344,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s)
 {
 }
 
-static inline void memcg_link_cache(struct kmem_cache *s)
+static inline void memcg_link_cache(struct kmem_cache *s,
+				    struct mem_cgroup *memcg)
 {
 }
 
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 58251ba63e4a..6e00bdf8618d 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s)
 }
 
 static int init_memcg_params(struct kmem_cache *s,
-		struct mem_cgroup *memcg, struct kmem_cache *root_cache)
+			     struct kmem_cache *root_cache)
 {
 	struct memcg_cache_array *arr;
 
 	if (root_cache) {
 		s->memcg_params.root_cache = root_cache;
-		s->memcg_params.memcg = memcg;
 		INIT_LIST_HEAD(&s->memcg_params.children_node);
 		INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node);
 		return 0;
@@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs)
 	return ret;
 }
 
-void memcg_link_cache(struct kmem_cache *s)
+void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg)
 {
 	if (is_root_cache(s)) {
 		list_add(&s->root_caches_node, &slab_root_caches);
 	} else {
+		s->memcg_params.memcg = memcg;
 		list_add(&s->memcg_params.children_node,
 			 &s->memcg_params.root_cache->memcg_params.children);
 		list_add(&s->memcg_params.kmem_caches_node,
@@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s)
 }
 #else
 static inline int init_memcg_params(struct kmem_cache *s,
-		struct mem_cgroup *memcg, struct kmem_cache *root_cache)
+				    struct kmem_cache *root_cache)
 {
 	return 0;
 }
@@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name,
 	s->useroffset = useroffset;
 	s->usersize = usersize;
 
-	err = init_memcg_params(s, memcg, root_cache);
+	err = init_memcg_params(s, root_cache);
 	if (err)
 		goto out_free_cache;
 
@@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name,
 
 	s->refcount = 1;
 	list_add(&s->list, &slab_caches);
-	memcg_link_cache(s);
+	memcg_link_cache(s, memcg);
 out:
 	if (err)
 		return ERR_PTR(err);
@@ -997,7 +997,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name,
 
 	create_boot_cache(s, name, size, flags, useroffset, usersize);
 	list_add(&s->list, &slab_caches);
-	memcg_link_cache(s);
+	memcg_link_cache(s, NULL);
 	s->refcount = 1;
 	return s;
 }
diff --git a/mm/slub.c b/mm/slub.c
index 1802c87799ff..9cb2eef62a37 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4213,7 +4213,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache)
 	}
 	slab_init_memcg_params(s);
 	list_add(&s->list, &slab_caches);
-	memcg_link_cache(s);
+	memcg_link_cache(s, NULL);
 	return s;
 }
 
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
  2019-06-11 23:18 ` [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-25 18:17   ` Shakeel Butt
  2019-06-11 23:18 ` [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation Roman Gushchin
                   ` (7 subsequent siblings)
  9 siblings, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

The delayed work/rcu deactivation infrastructure of non-root
kmem_caches can be also used for asynchronous release of these
objects. Let's get rid of the word "deactivation" in corresponding
names to make the code look better after generalization.

It's easier to make the renaming first, so that the generalized
code will look consistent from scratch.

Let's rename struct memcg_cache_params fields:
  deact_fn -> work_fn
  deact_rcu_head -> rcu_head
  deact_work -> work

And RCU/delayed work callbacks in slab common code:
  kmemcg_deactivate_rcufn -> kmemcg_rcufn
  kmemcg_deactivate_workfn -> kmemcg_workfn

This patch contains no functional changes, only renamings.

Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
---
 include/linux/slab.h |  6 +++---
 mm/slab.h            |  2 +-
 mm/slab_common.c     | 30 +++++++++++++++---------------
 3 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 9449b19c5f10..47923c173f30 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -642,10 +642,10 @@ struct memcg_cache_params {
 			struct list_head children_node;
 			struct list_head kmem_caches_node;
 
-			void (*deact_fn)(struct kmem_cache *);
+			void (*work_fn)(struct kmem_cache *);
 			union {
-				struct rcu_head deact_rcu_head;
-				struct work_struct deact_work;
+				struct rcu_head rcu_head;
+				struct work_struct work;
 			};
 		};
 	};
diff --git a/mm/slab.h b/mm/slab.h
index 86f7ede21203..7ef695b91919 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -291,7 +291,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,
 extern void slab_init_memcg_params(struct kmem_cache *);
 extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg);
 extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
-				void (*deact_fn)(struct kmem_cache *));
+				void (*work_fn)(struct kmem_cache *));
 
 #else /* CONFIG_MEMCG_KMEM */
 
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 6e00bdf8618d..99489d82ba78 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -691,17 +691,17 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,
 	put_online_cpus();
 }
 
-static void kmemcg_deactivate_workfn(struct work_struct *work)
+static void kmemcg_workfn(struct work_struct *work)
 {
 	struct kmem_cache *s = container_of(work, struct kmem_cache,
-					    memcg_params.deact_work);
+					    memcg_params.work);
 
 	get_online_cpus();
 	get_online_mems();
 
 	mutex_lock(&slab_mutex);
 
-	s->memcg_params.deact_fn(s);
+	s->memcg_params.work_fn(s);
 
 	mutex_unlock(&slab_mutex);
 
@@ -712,36 +712,36 @@ static void kmemcg_deactivate_workfn(struct work_struct *work)
 	css_put(&s->memcg_params.memcg->css);
 }
 
-static void kmemcg_deactivate_rcufn(struct rcu_head *head)
+static void kmemcg_rcufn(struct rcu_head *head)
 {
 	struct kmem_cache *s = container_of(head, struct kmem_cache,
-					    memcg_params.deact_rcu_head);
+					    memcg_params.rcu_head);
 
 	/*
-	 * We need to grab blocking locks.  Bounce to ->deact_work.  The
+	 * We need to grab blocking locks.  Bounce to ->work.  The
 	 * work item shares the space with the RCU head and can't be
 	 * initialized eariler.
 	 */
-	INIT_WORK(&s->memcg_params.deact_work, kmemcg_deactivate_workfn);
-	queue_work(memcg_kmem_cache_wq, &s->memcg_params.deact_work);
+	INIT_WORK(&s->memcg_params.work, kmemcg_workfn);
+	queue_work(memcg_kmem_cache_wq, &s->memcg_params.work);
 }
 
 /**
  * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a
  *					   sched RCU grace period
  * @s: target kmem_cache
- * @deact_fn: deactivation function to call
+ * @work_fn: deactivation function to call
  *
- * Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex
+ * Schedule @work_fn to be invoked with online cpus, mems and slab_mutex
  * held after a sched RCU grace period.  The slab is guaranteed to stay
- * alive until @deact_fn is finished.  This is to be used from
+ * alive until @work_fn is finished.  This is to be used from
  * __kmemcg_cache_deactivate().
  */
 void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
-					   void (*deact_fn)(struct kmem_cache *))
+					   void (*work_fn)(struct kmem_cache *))
 {
 	if (WARN_ON_ONCE(is_root_cache(s)) ||
-	    WARN_ON_ONCE(s->memcg_params.deact_fn))
+	    WARN_ON_ONCE(s->memcg_params.work_fn))
 		return;
 
 	if (s->memcg_params.root_cache->memcg_params.dying)
@@ -750,8 +750,8 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
 	/* pin memcg so that @s doesn't get destroyed in the middle */
 	css_get(&s->memcg_params.memcg->css);
 
-	s->memcg_params.deact_fn = deact_fn;
-	call_rcu(&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn);
+	s->memcg_params.work_fn = work_fn;
+	call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn);
 }
 
 void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
  2019-06-11 23:18 ` [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Roman Gushchin
  2019-06-11 23:18 ` [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-25 18:17   ` Shakeel Butt
  2019-06-11 23:18 ` [PATCH v7 04/10] mm: introduce __memcg_kmem_uncharge_memcg() Roman Gushchin
                   ` (6 subsequent siblings)
  9 siblings, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Currently SLUB uses a work scheduled after an RCU grace period
to deactivate a non-root kmem_cache. This mechanism can be reused
for kmem_caches release, but requires generalization for SLAB
case.

Introduce kmemcg_cache_deactivate() function, which calls
allocator-specific __kmem_cache_deactivate() and schedules
execution of __kmem_cache_deactivate_after_rcu() with all
necessary locks in a worker context after an rcu grace period.

Here is the new calling scheme:
  kmemcg_cache_deactivate()
    __kmemcg_cache_deactivate()                  SLAB/SLUB-specific
    kmemcg_rcufn()                               rcu
      kmemcg_workfn()                            work
        __kmemcg_cache_deactivate_after_rcu()    SLAB/SLUB-specific

instead of:
  __kmemcg_cache_deactivate()                    SLAB/SLUB-specific
    slab_deactivate_memcg_cache_rcu_sched()      SLUB-only
      kmemcg_rcufn()                             rcu
        kmemcg_workfn()                          work
          kmemcg_cache_deact_after_rcu()         SLUB-only

For consistency, all allocator-specific functions start with "__".

Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
---
 mm/slab.c        |  4 ++++
 mm/slab.h        |  3 +--
 mm/slab_common.c | 27 ++++++++-------------------
 mm/slub.c        |  8 +-------
 4 files changed, 14 insertions(+), 28 deletions(-)

diff --git a/mm/slab.c b/mm/slab.c
index a4091f8b3655..4b865393ebb4 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -2252,6 +2252,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep)
 {
 	__kmem_cache_shrink(cachep);
 }
+
+void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s)
+{
+}
 #endif
 
 int __kmem_cache_shutdown(struct kmem_cache *cachep)
diff --git a/mm/slab.h b/mm/slab.h
index 7ef695b91919..dc83583ee9dd 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *);
 void __kmem_cache_release(struct kmem_cache *);
 int __kmem_cache_shrink(struct kmem_cache *);
 void __kmemcg_cache_deactivate(struct kmem_cache *s);
+void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s);
 void slab_kmem_cache_release(struct kmem_cache *);
 
 struct seq_file;
@@ -290,8 +291,6 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,
 
 extern void slab_init_memcg_params(struct kmem_cache *);
 extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg);
-extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
-				void (*work_fn)(struct kmem_cache *));
 
 #else /* CONFIG_MEMCG_KMEM */
 
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 99489d82ba78..5e7638f495d1 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -708,7 +708,7 @@ static void kmemcg_workfn(struct work_struct *work)
 	put_online_mems();
 	put_online_cpus();
 
-	/* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */
+	/* done, put the ref from kmemcg_cache_deactivate() */
 	css_put(&s->memcg_params.memcg->css);
 }
 
@@ -726,31 +726,21 @@ static void kmemcg_rcufn(struct rcu_head *head)
 	queue_work(memcg_kmem_cache_wq, &s->memcg_params.work);
 }
 
-/**
- * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a
- *					   sched RCU grace period
- * @s: target kmem_cache
- * @work_fn: deactivation function to call
- *
- * Schedule @work_fn to be invoked with online cpus, mems and slab_mutex
- * held after a sched RCU grace period.  The slab is guaranteed to stay
- * alive until @work_fn is finished.  This is to be used from
- * __kmemcg_cache_deactivate().
- */
-void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
-					   void (*work_fn)(struct kmem_cache *))
+static void kmemcg_cache_deactivate(struct kmem_cache *s)
 {
 	if (WARN_ON_ONCE(is_root_cache(s)) ||
 	    WARN_ON_ONCE(s->memcg_params.work_fn))
 		return;
 
+	__kmemcg_cache_deactivate(s);
+
 	if (s->memcg_params.root_cache->memcg_params.dying)
 		return;
 
 	/* pin memcg so that @s doesn't get destroyed in the middle */
 	css_get(&s->memcg_params.memcg->css);
 
-	s->memcg_params.work_fn = work_fn;
+	s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu;
 	call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn);
 }
 
@@ -773,7 +763,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
 		if (!c)
 			continue;
 
-		__kmemcg_cache_deactivate(c);
+		kmemcg_cache_deactivate(c);
 		arr->entries[idx] = NULL;
 	}
 	mutex_unlock(&slab_mutex);
@@ -866,11 +856,10 @@ static void flush_memcg_workqueue(struct kmem_cache *s)
 	mutex_unlock(&slab_mutex);
 
 	/*
-	 * SLUB deactivates the kmem_caches through call_rcu. Make
+	 * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make
 	 * sure all registered rcu callbacks have been invoked.
 	 */
-	if (IS_ENABLED(CONFIG_SLUB))
-		rcu_barrier();
+	rcu_barrier();
 
 	/*
 	 * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB
diff --git a/mm/slub.c b/mm/slub.c
index 9cb2eef62a37..ae3b1e49ecec 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4022,7 +4022,7 @@ int __kmem_cache_shrink(struct kmem_cache *s)
 }
 
 #ifdef CONFIG_MEMCG
-static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s)
+void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s)
 {
 	/*
 	 * Called with all the locks held after a sched RCU grace period.
@@ -4048,12 +4048,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s)
 	 */
 	slub_set_cpu_partial(s, 0);
 	s->min_partial = 0;
-
-	/*
-	 * s->cpu_partial is checked locklessly (see put_cpu_partial), so
-	 * we have to make sure the change is visible before shrinking.
-	 */
-	slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu);
 }
 #endif	/* CONFIG_MEMCG */
 
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 04/10] mm: introduce __memcg_kmem_uncharge_memcg()
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
                   ` (2 preceding siblings ...)
  2019-06-11 23:18 ` [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-11 23:18 ` [PATCH v7 05/10] mm: unify SLAB and SLUB page accounting Roman Gushchin
                   ` (5 subsequent siblings)
  9 siblings, 0 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Let's separate the page counter modification code out of
__memcg_kmem_uncharge() in a way similar to what
__memcg_kmem_charge() and __memcg_kmem_charge_memcg() work.

This will allow to reuse this code later using a new
memcg_kmem_uncharge_memcg() wrapper, which calls
__memcg_kmem_uncharge_memcg() if memcg_kmem_enabled()
check is passed.

Signed-off-by: Roman Gushchin <guro@fb.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
---
 include/linux/memcontrol.h | 10 ++++++++++
 mm/memcontrol.c            | 25 +++++++++++++++++--------
 2 files changed, 27 insertions(+), 8 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 3ca57bacfdd2..9abf31bbe53a 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1304,6 +1304,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order);
 void __memcg_kmem_uncharge(struct page *page, int order);
 int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order,
 			      struct mem_cgroup *memcg);
+void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
+				 unsigned int nr_pages);
 
 extern struct static_key_false memcg_kmem_enabled_key;
 extern struct workqueue_struct *memcg_kmem_cache_wq;
@@ -1345,6 +1347,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp,
 		return __memcg_kmem_charge_memcg(page, gfp, order, memcg);
 	return 0;
 }
+
+static inline void memcg_kmem_uncharge_memcg(struct page *page, int order,
+					     struct mem_cgroup *memcg)
+{
+	if (memcg_kmem_enabled())
+		__memcg_kmem_uncharge_memcg(memcg, 1 << order);
+}
+
 /*
  * helper for accessing a memcg's index. It will be used as an index in the
  * child cache array in kmem_cache, and also to derive its name. This function
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index be9344777b29..8eaf553b67f1 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2812,6 +2812,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order)
 	css_put(&memcg->css);
 	return ret;
 }
+
+/**
+ * __memcg_kmem_uncharge_memcg: uncharge a kmem page
+ * @memcg: memcg to uncharge
+ * @nr_pages: number of pages to uncharge
+ */
+void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
+				 unsigned int nr_pages)
+{
+	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		page_counter_uncharge(&memcg->kmem, nr_pages);
+
+	page_counter_uncharge(&memcg->memory, nr_pages);
+	if (do_memsw_account())
+		page_counter_uncharge(&memcg->memsw, nr_pages);
+}
 /**
  * __memcg_kmem_uncharge: uncharge a kmem page
  * @page: page to uncharge
@@ -2826,14 +2842,7 @@ void __memcg_kmem_uncharge(struct page *page, int order)
 		return;
 
 	VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page);
-
-	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
-		page_counter_uncharge(&memcg->kmem, nr_pages);
-
-	page_counter_uncharge(&memcg->memory, nr_pages);
-	if (do_memsw_account())
-		page_counter_uncharge(&memcg->memsw, nr_pages);
-
+	__memcg_kmem_uncharge_memcg(memcg, nr_pages);
 	page->mem_cgroup = NULL;
 
 	/* slab pages do not have PageKmemcg flag set */
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 05/10] mm: unify SLAB and SLUB page accounting
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
                   ` (3 preceding siblings ...)
  2019-06-11 23:18 ` [PATCH v7 04/10] mm: introduce __memcg_kmem_uncharge_memcg() Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-11 23:18 ` [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation Roman Gushchin
                   ` (4 subsequent siblings)
  9 siblings, 0 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Currently the page accounting code is duplicated in SLAB and SLUB
internals. Let's move it into new (un)charge_slab_page helpers
in the slab_common.c file. These helpers will be responsible
for statistics (global and memcg-aware) and memcg charging.
So they are replacing direct memcg_(un)charge_slab() calls.

Signed-off-by: Roman Gushchin <guro@fb.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Acked-by: Christoph Lameter <cl@linux.com>
Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
---
 mm/slab.c | 19 +++----------------
 mm/slab.h | 25 +++++++++++++++++++++++++
 mm/slub.c | 14 ++------------
 3 files changed, 30 insertions(+), 28 deletions(-)

diff --git a/mm/slab.c b/mm/slab.c
index 4b865393ebb4..b417824a9b15 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -1360,7 +1360,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
 								int nodeid)
 {
 	struct page *page;
-	int nr_pages;
 
 	flags |= cachep->allocflags;
 
@@ -1370,17 +1369,11 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
 		return NULL;
 	}
 
-	if (memcg_charge_slab(page, flags, cachep->gfporder, cachep)) {
+	if (charge_slab_page(page, flags, cachep->gfporder, cachep)) {
 		__free_pages(page, cachep->gfporder);
 		return NULL;
 	}
 
-	nr_pages = (1 << cachep->gfporder);
-	if (cachep->flags & SLAB_RECLAIM_ACCOUNT)
-		mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, nr_pages);
-	else
-		mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, nr_pages);
-
 	__SetPageSlab(page);
 	/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */
 	if (sk_memalloc_socks() && page_is_pfmemalloc(page))
@@ -1395,12 +1388,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
 static void kmem_freepages(struct kmem_cache *cachep, struct page *page)
 {
 	int order = cachep->gfporder;
-	unsigned long nr_freed = (1 << order);
-
-	if (cachep->flags & SLAB_RECLAIM_ACCOUNT)
-		mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, -nr_freed);
-	else
-		mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, -nr_freed);
 
 	BUG_ON(!PageSlab(page));
 	__ClearPageSlabPfmemalloc(page);
@@ -1409,8 +1396,8 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page)
 	page->mapping = NULL;
 
 	if (current->reclaim_state)
-		current->reclaim_state->reclaimed_slab += nr_freed;
-	memcg_uncharge_slab(page, order, cachep);
+		current->reclaim_state->reclaimed_slab += 1 << order;
+	uncharge_slab_page(page, order, cachep);
 	__free_pages(page, order);
 }
 
diff --git a/mm/slab.h b/mm/slab.h
index dc83583ee9dd..46623a576a3c 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -205,6 +205,12 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer,
 void __kmem_cache_free_bulk(struct kmem_cache *, size_t, void **);
 int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **);
 
+static inline int cache_vmstat_idx(struct kmem_cache *s)
+{
+	return (s->flags & SLAB_RECLAIM_ACCOUNT) ?
+		NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE;
+}
+
 #ifdef CONFIG_MEMCG_KMEM
 
 /* List of all root caches. */
@@ -361,6 +367,25 @@ static inline struct kmem_cache *virt_to_cache(const void *obj)
 	return page->slab_cache;
 }
 
+static __always_inline int charge_slab_page(struct page *page,
+					    gfp_t gfp, int order,
+					    struct kmem_cache *s)
+{
+	int ret = memcg_charge_slab(page, gfp, order, s);
+
+	if (!ret)
+		mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order);
+
+	return ret;
+}
+
+static __always_inline void uncharge_slab_page(struct page *page, int order,
+					       struct kmem_cache *s)
+{
+	mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order));
+	memcg_uncharge_slab(page, order, s);
+}
+
 static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
 {
 	struct kmem_cache *cachep;
diff --git a/mm/slub.c b/mm/slub.c
index ae3b1e49ecec..6a5174b51cd6 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1488,7 +1488,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,
 	else
 		page = __alloc_pages_node(node, flags, order);
 
-	if (page && memcg_charge_slab(page, flags, order, s)) {
+	if (page && charge_slab_page(page, flags, order, s)) {
 		__free_pages(page, order);
 		page = NULL;
 	}
@@ -1681,11 +1681,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	if (!page)
 		return NULL;
 
-	mod_lruvec_page_state(page,
-		(s->flags & SLAB_RECLAIM_ACCOUNT) ?
-		NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE,
-		1 << oo_order(oo));
-
 	inc_slabs_node(s, page_to_nid(page), page->objects);
 
 	return page;
@@ -1719,18 +1714,13 @@ static void __free_slab(struct kmem_cache *s, struct page *page)
 			check_object(s, page, p, SLUB_RED_INACTIVE);
 	}
 
-	mod_lruvec_page_state(page,
-		(s->flags & SLAB_RECLAIM_ACCOUNT) ?
-		NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE,
-		-pages);
-
 	__ClearPageSlabPfmemalloc(page);
 	__ClearPageSlab(page);
 
 	page->mapping = NULL;
 	if (current->reclaim_state)
 		current->reclaim_state->reclaimed_slab += pages;
-	memcg_uncharge_slab(page, order, s);
+	uncharge_slab_page(page, order, s);
 	__free_pages(page, order);
 }
 
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
                   ` (4 preceding siblings ...)
  2019-06-11 23:18 ` [PATCH v7 05/10] mm: unify SLAB and SLUB page accounting Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-16 16:26   ` Vladimir Davydov
  2019-06-25 18:31   ` Shakeel Butt
  2019-06-11 23:18 ` [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Roman Gushchin
                   ` (3 subsequent siblings)
  9 siblings, 2 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

There is no point in checking the root_cache->memcg_params.dying
flag on kmem_cache creation path. New allocations shouldn't be
performed using a dead root kmem_cache, so no new memcg kmem_cache
creation can be scheduled after the flag is set. And if it was
scheduled before, flush_memcg_workqueue() will wait for it anyway.

So let's drop this check to simplify the code.

Signed-off-by: Roman Gushchin <guro@fb.com>
---
 mm/slab_common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 5e7638f495d1..9383104651cd 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -640,7 +640,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,
 	 * The memory cgroup could have been offlined while the cache
 	 * creation work was pending.
 	 */
-	if (memcg->kmem_state != KMEM_ONLINE || root_cache->memcg_params.dying)
+	if (memcg->kmem_state != KMEM_ONLINE)
 		goto out_unlock;
 
 	idx = memcg_cache_id(memcg);
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
                   ` (5 preceding siblings ...)
  2019-06-11 23:18 ` [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-16 16:27   ` Vladimir Davydov
  2019-06-25 18:33   ` Shakeel Butt
  2019-06-11 23:18 ` [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management Roman Gushchin
                   ` (2 subsequent siblings)
  9 siblings, 2 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Currently the memcg_params.dying flag and the corresponding
workqueue used for the asynchronous deactivation of kmem_caches
is synchronized using the slab_mutex.

It makes impossible to check this flag from the irq context,
which will be required in order to implement asynchronous release
of kmem_caches.

So let's switch over to the irq-save flavor of the spinlock-based
synchronization.

Signed-off-by: Roman Gushchin <guro@fb.com>
---
 mm/slab_common.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 9383104651cd..1e5eaf84bf08 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,
 #ifdef CONFIG_MEMCG_KMEM
 
 LIST_HEAD(slab_root_caches);
+static DEFINE_SPINLOCK(memcg_kmem_wq_lock);
 
 void slab_init_memcg_params(struct kmem_cache *s)
 {
@@ -734,14 +735,22 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s)
 
 	__kmemcg_cache_deactivate(s);
 
+	/*
+	 * memcg_kmem_wq_lock is used to synchronize memcg_params.dying
+	 * flag and make sure that no new kmem_cache deactivation tasks
+	 * are queued (see flush_memcg_workqueue() ).
+	 */
+	spin_lock_irq(&memcg_kmem_wq_lock);
 	if (s->memcg_params.root_cache->memcg_params.dying)
-		return;
+		goto unlock;
 
 	/* pin memcg so that @s doesn't get destroyed in the middle */
 	css_get(&s->memcg_params.memcg->css);
 
 	s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu;
 	call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn);
+unlock:
+	spin_unlock_irq(&memcg_kmem_wq_lock);
 }
 
 void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
@@ -851,9 +860,9 @@ static int shutdown_memcg_caches(struct kmem_cache *s)
 
 static void flush_memcg_workqueue(struct kmem_cache *s)
 {
-	mutex_lock(&slab_mutex);
+	spin_lock_irq(&memcg_kmem_wq_lock);
 	s->memcg_params.dying = true;
-	mutex_unlock(&slab_mutex);
+	spin_unlock_irq(&memcg_kmem_wq_lock);
 
 	/*
 	 * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
                   ` (6 preceding siblings ...)
  2019-06-11 23:18 ` [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-25 23:57   ` Shakeel Butt
  2019-11-21 11:17   ` WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management) Christian Borntraeger
  2019-06-11 23:18 ` [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages Roman Gushchin
  2019-06-11 23:18 ` [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal Roman Gushchin
  9 siblings, 2 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Currently each charged slab page holds a reference to the cgroup to
which it's charged. Kmem_caches are held by the memcg and are released
all together with the memory cgroup. It means that none of kmem_caches
are released unless at least one reference to the memcg exists, which
is very far from optimal.

Let's rework it in a way that allows releasing individual kmem_caches
as soon as the cgroup is offline, the kmem_cache is empty and there
are no pending allocations.

To make it possible, let's introduce a new percpu refcounter for
non-root kmem caches. The counter is initialized to the percpu mode,
and is switched to the atomic mode during kmem_cache deactivation. The
counter is bumped for every charged page and also for every running
allocation. So the kmem_cache can't be released unless all allocations
complete.

To shutdown non-active empty kmem_caches, let's reuse the work queue,
previously used for the kmem_cache deactivation. Once the reference
counter reaches 0, let's schedule an asynchronous kmem_cache release.

* I used the following simple approach to test the performance
(stolen from another patchset by T. Harding):

    time find / -name fname-no-exist
    echo 2 > /proc/sys/vm/drop_caches
    repeat 10 times

Results:

        orig		patched

real	0m1.455s	real	0m1.355s
user	0m0.206s	user	0m0.219s
sys	0m0.855s	sys	0m0.807s

real	0m1.487s	real	0m1.699s
user	0m0.221s	user	0m0.256s
sys	0m0.806s	sys	0m0.948s

real	0m1.515s	real	0m1.505s
user	0m0.183s	user	0m0.215s
sys	0m0.876s	sys	0m0.858s

real	0m1.291s	real	0m1.380s
user	0m0.193s	user	0m0.198s
sys	0m0.843s	sys	0m0.786s

real	0m1.364s	real	0m1.374s
user	0m0.180s	user	0m0.182s
sys	0m0.868s	sys	0m0.806s

real	0m1.352s	real	0m1.312s
user	0m0.201s	user	0m0.212s
sys	0m0.820s	sys	0m0.761s

real	0m1.302s	real	0m1.349s
user	0m0.205s	user	0m0.203s
sys	0m0.803s	sys	0m0.792s

real	0m1.334s	real	0m1.301s
user	0m0.194s	user	0m0.201s
sys	0m0.806s	sys	0m0.779s

real	0m1.426s	real	0m1.434s
user	0m0.216s	user	0m0.181s
sys	0m0.824s	sys	0m0.864s

real	0m1.350s	real	0m1.295s
user	0m0.200s	user	0m0.190s
sys	0m0.842s	sys	0m0.811s

So it looks like the difference is not noticeable in this test.

Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
---
 include/linux/slab.h |  3 +-
 mm/memcontrol.c      | 50 +++++++++++++++++++++-------
 mm/slab.h            | 44 +++++++-----------------
 mm/slab_common.c     | 79 ++++++++++++++++++++++++++------------------
 4 files changed, 99 insertions(+), 77 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 47923c173f30..1b54e5f83342 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -16,6 +16,7 @@
 #include <linux/overflow.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
+#include <linux/percpu-refcount.h>
 
 
 /*
@@ -152,7 +153,6 @@ int kmem_cache_shrink(struct kmem_cache *);
 
 void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *);
 void memcg_deactivate_kmem_caches(struct mem_cgroup *);
-void memcg_destroy_kmem_caches(struct mem_cgroup *);
 
 /*
  * Please use this macro to create slab caches. Simply specify the
@@ -641,6 +641,7 @@ struct memcg_cache_params {
 			struct mem_cgroup *memcg;
 			struct list_head children_node;
 			struct list_head kmem_caches_node;
+			struct percpu_ref refcnt;
 
 			void (*work_fn)(struct kmem_cache *);
 			union {
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 8eaf553b67f1..43a42bc3ed3f 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2672,12 +2672,13 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg,
 {
 	struct memcg_kmem_cache_create_work *cw;
 
+	if (!css_tryget_online(&memcg->css))
+		return;
+
 	cw = kmalloc(sizeof(*cw), GFP_NOWAIT | __GFP_NOWARN);
 	if (!cw)
 		return;
 
-	css_get(&memcg->css);
-
 	cw->memcg = memcg;
 	cw->cachep = cachep;
 	INIT_WORK(&cw->work, memcg_kmem_cache_create_func);
@@ -2712,6 +2713,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep)
 {
 	struct mem_cgroup *memcg;
 	struct kmem_cache *memcg_cachep;
+	struct memcg_cache_array *arr;
 	int kmemcg_id;
 
 	VM_BUG_ON(!is_root_cache(cachep));
@@ -2719,14 +2721,28 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep)
 	if (memcg_kmem_bypass())
 		return cachep;
 
-	memcg = get_mem_cgroup_from_current();
+	rcu_read_lock();
+
+	if (unlikely(current->active_memcg))
+		memcg = current->active_memcg;
+	else
+		memcg = mem_cgroup_from_task(current);
+
+	if (!memcg || memcg == root_mem_cgroup)
+		goto out_unlock;
+
 	kmemcg_id = READ_ONCE(memcg->kmemcg_id);
 	if (kmemcg_id < 0)
-		goto out;
+		goto out_unlock;
+
+	arr = rcu_dereference(cachep->memcg_params.memcg_caches);
 
-	memcg_cachep = cache_from_memcg_idx(cachep, kmemcg_id);
-	if (likely(memcg_cachep))
-		return memcg_cachep;
+	/*
+	 * Make sure we will access the up-to-date value. The code updating
+	 * memcg_caches issues a write barrier to match the data dependency
+	 * barrier inside READ_ONCE() (see memcg_create_kmem_cache()).
+	 */
+	memcg_cachep = READ_ONCE(arr->entries[kmemcg_id]);
 
 	/*
 	 * If we are in a safe context (can wait, and not in interrupt
@@ -2739,10 +2755,20 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep)
 	 * memcg_create_kmem_cache, this means no further allocation
 	 * could happen with the slab_mutex held. So it's better to
 	 * defer everything.
+	 *
+	 * If the memcg is dying or memcg_cache is about to be released,
+	 * don't bother creating new kmem_caches. Because memcg_cachep
+	 * is ZEROed as the fist step of kmem offlining, we don't need
+	 * percpu_ref_tryget_live() here. css_tryget_online() check in
+	 * memcg_schedule_kmem_cache_create() will prevent us from
+	 * creation of a new kmem_cache.
 	 */
-	memcg_schedule_kmem_cache_create(memcg, cachep);
-out:
-	css_put(&memcg->css);
+	if (unlikely(!memcg_cachep))
+		memcg_schedule_kmem_cache_create(memcg, cachep);
+	else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt))
+		cachep = memcg_cachep;
+out_unlock:
+	rcu_read_unlock();
 	return cachep;
 }
 
@@ -2753,7 +2779,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep)
 void memcg_kmem_put_cache(struct kmem_cache *cachep)
 {
 	if (!is_root_cache(cachep))
-		css_put(&cachep->memcg_params.memcg->css);
+		percpu_ref_put(&cachep->memcg_params.refcnt);
 }
 
 /**
@@ -3300,7 +3326,7 @@ static void memcg_free_kmem(struct mem_cgroup *memcg)
 		memcg_offline_kmem(memcg);
 
 	if (memcg->kmem_state == KMEM_ALLOCATED) {
-		memcg_destroy_kmem_caches(memcg);
+		WARN_ON(!list_empty(&memcg->kmem_caches));
 		static_branch_dec(&memcg_kmem_enabled_key);
 		WARN_ON(page_counter_read(&memcg->kmem));
 	}
diff --git a/mm/slab.h b/mm/slab.h
index 46623a576a3c..5d2b8511e6fb 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -248,31 +248,6 @@ static inline const char *cache_name(struct kmem_cache *s)
 	return s->name;
 }
 
-/*
- * Note, we protect with RCU only the memcg_caches array, not per-memcg caches.
- * That said the caller must assure the memcg's cache won't go away by either
- * taking a css reference to the owner cgroup, or holding the slab_mutex.
- */
-static inline struct kmem_cache *
-cache_from_memcg_idx(struct kmem_cache *s, int idx)
-{
-	struct kmem_cache *cachep;
-	struct memcg_cache_array *arr;
-
-	rcu_read_lock();
-	arr = rcu_dereference(s->memcg_params.memcg_caches);
-
-	/*
-	 * Make sure we will access the up-to-date value. The code updating
-	 * memcg_caches issues a write barrier to match this (see
-	 * memcg_create_kmem_cache()).
-	 */
-	cachep = READ_ONCE(arr->entries[idx]);
-	rcu_read_unlock();
-
-	return cachep;
-}
-
 static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s)
 {
 	if (is_root_cache(s))
@@ -284,14 +259,25 @@ static __always_inline int memcg_charge_slab(struct page *page,
 					     gfp_t gfp, int order,
 					     struct kmem_cache *s)
 {
+	int ret;
+
 	if (is_root_cache(s))
 		return 0;
-	return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg);
+
+	ret = memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg);
+	if (ret)
+		return ret;
+
+	percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order);
+
+	return 0;
 }
 
 static __always_inline void memcg_uncharge_slab(struct page *page, int order,
 						struct kmem_cache *s)
 {
+	if (!is_root_cache(s))
+		percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order);
 	memcg_kmem_uncharge(page, order);
 }
 
@@ -323,12 +309,6 @@ static inline const char *cache_name(struct kmem_cache *s)
 	return s->name;
 }
 
-static inline struct kmem_cache *
-cache_from_memcg_idx(struct kmem_cache *s, int idx)
-{
-	return NULL;
-}
-
 static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s)
 {
 	return s;
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 1e5eaf84bf08..6b7750f7ea33 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -132,6 +132,8 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,
 LIST_HEAD(slab_root_caches);
 static DEFINE_SPINLOCK(memcg_kmem_wq_lock);
 
+static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref);
+
 void slab_init_memcg_params(struct kmem_cache *s)
 {
 	s->memcg_params.root_cache = NULL;
@@ -146,6 +148,12 @@ static int init_memcg_params(struct kmem_cache *s,
 	struct memcg_cache_array *arr;
 
 	if (root_cache) {
+		int ret = percpu_ref_init(&s->memcg_params.refcnt,
+					  kmemcg_cache_shutdown,
+					  0, GFP_KERNEL);
+		if (ret)
+			return ret;
+
 		s->memcg_params.root_cache = root_cache;
 		INIT_LIST_HEAD(&s->memcg_params.children_node);
 		INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node);
@@ -171,6 +179,8 @@ static void destroy_memcg_params(struct kmem_cache *s)
 {
 	if (is_root_cache(s))
 		kvfree(rcu_access_pointer(s->memcg_params.memcg_caches));
+	else
+		percpu_ref_exit(&s->memcg_params.refcnt);
 }
 
 static void free_memcg_params(struct rcu_head *rcu)
@@ -226,6 +236,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg)
 	if (is_root_cache(s)) {
 		list_add(&s->root_caches_node, &slab_root_caches);
 	} else {
+		css_get(&memcg->css);
 		s->memcg_params.memcg = memcg;
 		list_add(&s->memcg_params.children_node,
 			 &s->memcg_params.root_cache->memcg_params.children);
@@ -241,6 +252,7 @@ static void memcg_unlink_cache(struct kmem_cache *s)
 	} else {
 		list_del(&s->memcg_params.children_node);
 		list_del(&s->memcg_params.kmem_caches_node);
+		css_put(&s->memcg_params.memcg->css);
 	}
 }
 #else
@@ -678,7 +690,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg,
 	}
 
 	/*
-	 * Since readers won't lock (see cache_from_memcg_idx()), we need a
+	 * Since readers won't lock (see memcg_kmem_get_cache()), we need a
 	 * barrier here to ensure nobody will see the kmem_cache partially
 	 * initialized.
 	 */
@@ -703,14 +715,12 @@ static void kmemcg_workfn(struct work_struct *work)
 	mutex_lock(&slab_mutex);
 
 	s->memcg_params.work_fn(s);
+	s->memcg_params.work_fn = NULL;
 
 	mutex_unlock(&slab_mutex);
 
 	put_online_mems();
 	put_online_cpus();
-
-	/* done, put the ref from kmemcg_cache_deactivate() */
-	css_put(&s->memcg_params.memcg->css);
 }
 
 static void kmemcg_rcufn(struct rcu_head *head)
@@ -727,10 +737,39 @@ static void kmemcg_rcufn(struct rcu_head *head)
 	queue_work(memcg_kmem_cache_wq, &s->memcg_params.work);
 }
 
+static void kmemcg_cache_shutdown_fn(struct kmem_cache *s)
+{
+	WARN_ON(shutdown_cache(s));
+}
+
+static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref)
+{
+	struct kmem_cache *s = container_of(percpu_ref, struct kmem_cache,
+					    memcg_params.refcnt);
+	unsigned long flags;
+
+	spin_lock_irqsave(&memcg_kmem_wq_lock, flags);
+	if (s->memcg_params.root_cache->memcg_params.dying)
+		goto unlock;
+
+	WARN_ON(s->memcg_params.work_fn);
+	s->memcg_params.work_fn = kmemcg_cache_shutdown_fn;
+	INIT_WORK(&s->memcg_params.work, kmemcg_workfn);
+	queue_work(memcg_kmem_cache_wq, &s->memcg_params.work);
+
+unlock:
+	spin_unlock_irqrestore(&memcg_kmem_wq_lock, flags);
+}
+
+static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s)
+{
+	__kmemcg_cache_deactivate_after_rcu(s);
+	percpu_ref_kill(&s->memcg_params.refcnt);
+}
+
 static void kmemcg_cache_deactivate(struct kmem_cache *s)
 {
-	if (WARN_ON_ONCE(is_root_cache(s)) ||
-	    WARN_ON_ONCE(s->memcg_params.work_fn))
+	if (WARN_ON_ONCE(is_root_cache(s)))
 		return;
 
 	__kmemcg_cache_deactivate(s);
@@ -744,10 +783,8 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s)
 	if (s->memcg_params.root_cache->memcg_params.dying)
 		goto unlock;
 
-	/* pin memcg so that @s doesn't get destroyed in the middle */
-	css_get(&s->memcg_params.memcg->css);
-
-	s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu;
+	WARN_ON_ONCE(s->memcg_params.work_fn);
+	s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu;
 	call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn);
 unlock:
 	spin_unlock_irq(&memcg_kmem_wq_lock);
@@ -781,28 +818,6 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
 	put_online_cpus();
 }
 
-void memcg_destroy_kmem_caches(struct mem_cgroup *memcg)
-{
-	struct kmem_cache *s, *s2;
-
-	get_online_cpus();
-	get_online_mems();
-
-	mutex_lock(&slab_mutex);
-	list_for_each_entry_safe(s, s2, &memcg->kmem_caches,
-				 memcg_params.kmem_caches_node) {
-		/*
-		 * The cgroup is about to be freed and therefore has no charges
-		 * left. Hence, all its caches must be empty by now.
-		 */
-		BUG_ON(shutdown_cache(s));
-	}
-	mutex_unlock(&slab_mutex);
-
-	put_online_mems();
-	put_online_cpus();
-}
-
 static int shutdown_memcg_caches(struct kmem_cache *s)
 {
 	struct memcg_cache_array *arr;
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
                   ` (7 preceding siblings ...)
  2019-06-11 23:18 ` [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-26  0:15   ` Shakeel Butt
  2019-06-11 23:18 ` [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal Roman Gushchin
  9 siblings, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Every slab page charged to a non-root memory cgroup has a pointer
to the memory cgroup and holds a reference to it, which protects
a non-empty memory cgroup from being released. At the same time
the page has a pointer to the corresponding kmem_cache, and also
hold a reference to the kmem_cache. And kmem_cache by itself
holds a reference to the cgroup.

So there is clearly some redundancy, which allows to stop setting
the page->mem_cgroup pointer and rely on getting memcg pointer
indirectly via kmem_cache. Further it will allow to change this
pointer easier, without a need to go over all charged pages.

So let's stop setting page->mem_cgroup pointer for slab pages,
and stop using the css refcounter directly for protecting
the memory cgroup from going away. Instead rely on kmem_cache
as an intermediate object.

Make sure that vmstats and shrinker lists are working as previously,
as well as /proc/kpagecgroup interface.

Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>
---
 mm/list_lru.c   |  3 +-
 mm/memcontrol.c | 12 ++++----
 mm/slab.h       | 74 ++++++++++++++++++++++++++++++++++++++++---------
 3 files changed, 70 insertions(+), 19 deletions(-)

diff --git a/mm/list_lru.c b/mm/list_lru.c
index 927d85be32f6..0f1f6b06b7f3 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -12,6 +12,7 @@
 #include <linux/slab.h>
 #include <linux/mutex.h>
 #include <linux/memcontrol.h>
+#include "slab.h"
 
 #ifdef CONFIG_MEMCG_KMEM
 static LIST_HEAD(list_lrus);
@@ -63,7 +64,7 @@ static __always_inline struct mem_cgroup *mem_cgroup_from_kmem(void *ptr)
 	if (!memcg_kmem_enabled())
 		return NULL;
 	page = virt_to_head_page(ptr);
-	return page->mem_cgroup;
+	return memcg_from_slab_page(page);
 }
 
 static inline struct list_lru_one *
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 43a42bc3ed3f..25e72779fd33 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -486,7 +486,10 @@ ino_t page_cgroup_ino(struct page *page)
 	unsigned long ino = 0;
 
 	rcu_read_lock();
-	memcg = READ_ONCE(page->mem_cgroup);
+	if (PageHead(page) && PageSlab(page))
+		memcg = memcg_from_slab_page(page);
+	else
+		memcg = READ_ONCE(page->mem_cgroup);
 	while (memcg && !(memcg->css.flags & CSS_ONLINE))
 		memcg = parent_mem_cgroup(memcg);
 	if (memcg)
@@ -2807,9 +2810,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order,
 		cancel_charge(memcg, nr_pages);
 		return -ENOMEM;
 	}
-
-	page->mem_cgroup = memcg;
-
 	return 0;
 }
 
@@ -2832,8 +2832,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order)
 	memcg = get_mem_cgroup_from_current();
 	if (!mem_cgroup_is_root(memcg)) {
 		ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg);
-		if (!ret)
+		if (!ret) {
+			page->mem_cgroup = memcg;
 			__SetPageKmemcg(page);
+		}
 	}
 	css_put(&memcg->css);
 	return ret;
diff --git a/mm/slab.h b/mm/slab.h
index 5d2b8511e6fb..7ead47cb9338 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -255,30 +255,67 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s)
 	return s->memcg_params.root_cache;
 }
 
+/*
+ * Expects a pointer to a slab page. Please note, that PageSlab() check
+ * isn't sufficient, as it returns true also for tail compound slab pages,
+ * which do not have slab_cache pointer set.
+ * So this function assumes that the page can pass PageHead() and PageSlab()
+ * checks.
+ */
+static inline struct mem_cgroup *memcg_from_slab_page(struct page *page)
+{
+	struct kmem_cache *s;
+
+	s = READ_ONCE(page->slab_cache);
+	if (s && !is_root_cache(s))
+		return s->memcg_params.memcg;
+
+	return NULL;
+}
+
+/*
+ * Charge the slab page belonging to the non-root kmem_cache.
+ * Can be called for non-root kmem_caches only.
+ */
 static __always_inline int memcg_charge_slab(struct page *page,
 					     gfp_t gfp, int order,
 					     struct kmem_cache *s)
 {
+	struct mem_cgroup *memcg;
+	struct lruvec *lruvec;
 	int ret;
 
-	if (is_root_cache(s))
-		return 0;
-
-	ret = memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg);
+	memcg = s->memcg_params.memcg;
+	ret = memcg_kmem_charge_memcg(page, gfp, order, memcg);
 	if (ret)
 		return ret;
 
+	lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg);
+	mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order);
+
+	/* transer try_charge() page references to kmem_cache */
 	percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order);
+	css_put_many(&memcg->css, 1 << order);
 
 	return 0;
 }
 
+/*
+ * Uncharge a slab page belonging to a non-root kmem_cache.
+ * Can be called for non-root kmem_caches only.
+ */
 static __always_inline void memcg_uncharge_slab(struct page *page, int order,
 						struct kmem_cache *s)
 {
-	if (!is_root_cache(s))
-		percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order);
-	memcg_kmem_uncharge(page, order);
+	struct mem_cgroup *memcg;
+	struct lruvec *lruvec;
+
+	memcg = s->memcg_params.memcg;
+	lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg);
+	mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order));
+	memcg_kmem_uncharge_memcg(page, order, memcg);
+
+	percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order);
 }
 
 extern void slab_init_memcg_params(struct kmem_cache *);
@@ -314,6 +351,11 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s)
 	return s;
 }
 
+static inline struct mem_cgroup *memcg_from_slab_page(struct page *page)
+{
+	return NULL;
+}
+
 static inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order,
 				    struct kmem_cache *s)
 {
@@ -351,18 +393,24 @@ static __always_inline int charge_slab_page(struct page *page,
 					    gfp_t gfp, int order,
 					    struct kmem_cache *s)
 {
-	int ret = memcg_charge_slab(page, gfp, order, s);
-
-	if (!ret)
-		mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order);
+	if (is_root_cache(s)) {
+		mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s),
+				    1 << order);
+		return 0;
+	}
 
-	return ret;
+	return memcg_charge_slab(page, gfp, order, s);
 }
 
 static __always_inline void uncharge_slab_page(struct page *page, int order,
 					       struct kmem_cache *s)
 {
-	mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order));
+	if (is_root_cache(s)) {
+		mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s),
+				    -(1 << order));
+		return;
+	}
+
 	memcg_uncharge_slab(page, order, s);
 }
 
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal
  2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
                   ` (8 preceding siblings ...)
  2019-06-11 23:18 ` [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages Roman Gushchin
@ 2019-06-11 23:18 ` Roman Gushchin
  2019-06-16 16:29   ` Vladimir Davydov
  2019-06-26  0:15   ` Shakeel Butt
  9 siblings, 2 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-11 23:18 UTC (permalink / raw)
  To: Andrew Morton, Vladimir Davydov
  Cc: linux-mm, linux-kernel, kernel-team, Johannes Weiner,
	Shakeel Butt, Waiman Long, Roman Gushchin

Let's reparent non-root kmem_caches on memcg offlining. This allows us
to release the memory cgroup without waiting for the last outstanding
kernel object (e.g. dentry used by another application).

Since the parent cgroup is already charged, everything we need to do
is to splice the list of kmem_caches to the parent's kmem_caches list,
swap the memcg pointer, drop the css refcounter for each kmem_cache
and adjust the parent's css refcounter.

Please, note that kmem_cache->memcg_params.memcg isn't a stable
pointer anymore. It's safe to read it under rcu_read_lock(),
cgroup_mutex held, or any other way that protects the memory cgroup
from being released.

We can race with the slab allocation and deallocation paths. It's not
a big problem: parent's charge and slab global stats are always
correct, and we don't care anymore about the child usage and global
stats. The child cgroup is already offline, so we don't use or show it
anywhere.

Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE)
aren't used anywhere except count_shadow_nodes(). But even there it
won't break anything: after reparenting "nodes" will be 0 on child
level (because we're already reparenting shrinker lists), and on
parent level page stats always were 0, and this patch won't change
anything.

Signed-off-by: Roman Gushchin <guro@fb.com>
---
 include/linux/slab.h |  2 +-
 mm/memcontrol.c      | 14 ++++++++------
 mm/slab.h            | 26 ++++++++++++++++++++------
 mm/slab_common.c     | 19 +++++++++++++++++--
 4 files changed, 46 insertions(+), 15 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 1b54e5f83342..fecf40b7be69 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *);
 int kmem_cache_shrink(struct kmem_cache *);
 
 void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *);
-void memcg_deactivate_kmem_caches(struct mem_cgroup *);
+void memcg_deactivate_kmem_caches(struct mem_cgroup *, struct mem_cgroup *);
 
 /*
  * Please use this macro to create slab caches. Simply specify the
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 25e72779fd33..db46a9dc37ab 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3289,15 +3289,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)
 	 */
 	memcg->kmem_state = KMEM_ALLOCATED;
 
-	memcg_deactivate_kmem_caches(memcg);
-
-	kmemcg_id = memcg->kmemcg_id;
-	BUG_ON(kmemcg_id < 0);
-
 	parent = parent_mem_cgroup(memcg);
 	if (!parent)
 		parent = root_mem_cgroup;
 
+	memcg_deactivate_kmem_caches(memcg, parent);
+
+	kmemcg_id = memcg->kmemcg_id;
+	BUG_ON(kmemcg_id < 0);
+
 	/*
 	 * Change kmemcg_id of this cgroup and all its descendants to the
 	 * parent's id, and then move all entries from this cgroup's list_lrus
@@ -3330,7 +3330,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg)
 	if (memcg->kmem_state == KMEM_ALLOCATED) {
 		WARN_ON(!list_empty(&memcg->kmem_caches));
 		static_branch_dec(&memcg_kmem_enabled_key);
-		WARN_ON(page_counter_read(&memcg->kmem));
 	}
 }
 #else
@@ -4777,6 +4776,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 
 	/* The following stuff does not apply to the root */
 	if (!parent) {
+#ifdef CONFIG_MEMCG_KMEM
+		INIT_LIST_HEAD(&memcg->kmem_caches);
+#endif
 		root_mem_cgroup = memcg;
 		return &memcg->css;
 	}
diff --git a/mm/slab.h b/mm/slab.h
index 7ead47cb9338..a4c9b9d042de 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -261,6 +261,9 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s)
  * which do not have slab_cache pointer set.
  * So this function assumes that the page can pass PageHead() and PageSlab()
  * checks.
+ *
+ * The kmem_cache can be reparented asynchronously. The caller must ensure
+ * the memcg lifetime, e.g. by taking rcu_read_lock() or cgroup_mutex.
  */
 static inline struct mem_cgroup *memcg_from_slab_page(struct page *page)
 {
@@ -268,7 +271,7 @@ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page)
 
 	s = READ_ONCE(page->slab_cache);
 	if (s && !is_root_cache(s))
-		return s->memcg_params.memcg;
+		return READ_ONCE(s->memcg_params.memcg);
 
 	return NULL;
 }
@@ -285,10 +288,18 @@ static __always_inline int memcg_charge_slab(struct page *page,
 	struct lruvec *lruvec;
 	int ret;
 
-	memcg = s->memcg_params.memcg;
+	rcu_read_lock();
+	memcg = READ_ONCE(s->memcg_params.memcg);
+	while (memcg && !css_tryget_online(&memcg->css))
+		memcg = parent_mem_cgroup(memcg);
+	rcu_read_unlock();
+
+	if (unlikely(!memcg))
+		return true;
+
 	ret = memcg_kmem_charge_memcg(page, gfp, order, memcg);
 	if (ret)
-		return ret;
+		goto out;
 
 	lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg);
 	mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order);
@@ -296,8 +307,9 @@ static __always_inline int memcg_charge_slab(struct page *page,
 	/* transer try_charge() page references to kmem_cache */
 	percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order);
 	css_put_many(&memcg->css, 1 << order);
-
-	return 0;
+out:
+	css_put(&memcg->css);
+	return ret;
 }
 
 /*
@@ -310,10 +322,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order,
 	struct mem_cgroup *memcg;
 	struct lruvec *lruvec;
 
-	memcg = s->memcg_params.memcg;
+	rcu_read_lock();
+	memcg = READ_ONCE(s->memcg_params.memcg);
 	lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg);
 	mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order));
 	memcg_kmem_uncharge_memcg(page, order, memcg);
+	rcu_read_unlock();
 
 	percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order);
 }
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 6b7750f7ea33..91e8c739dc97 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -252,7 +252,8 @@ static void memcg_unlink_cache(struct kmem_cache *s)
 	} else {
 		list_del(&s->memcg_params.children_node);
 		list_del(&s->memcg_params.kmem_caches_node);
-		css_put(&s->memcg_params.memcg->css);
+		mem_cgroup_put(s->memcg_params.memcg);
+		WRITE_ONCE(s->memcg_params.memcg, NULL);
 	}
 }
 #else
@@ -790,11 +791,13 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s)
 	spin_unlock_irq(&memcg_kmem_wq_lock);
 }
 
-void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
+void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg,
+				  struct mem_cgroup *parent)
 {
 	int idx;
 	struct memcg_cache_array *arr;
 	struct kmem_cache *s, *c;
+	unsigned int nr_reparented;
 
 	idx = memcg_cache_id(memcg);
 
@@ -812,6 +815,18 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
 		kmemcg_cache_deactivate(c);
 		arr->entries[idx] = NULL;
 	}
+	nr_reparented = 0;
+	list_for_each_entry(s, &memcg->kmem_caches,
+			    memcg_params.kmem_caches_node) {
+		WRITE_ONCE(s->memcg_params.memcg, parent);
+		css_put(&memcg->css);
+		nr_reparented++;
+	}
+	if (nr_reparented) {
+		list_splice_init(&memcg->kmem_caches,
+				 &parent->kmem_caches);
+		css_get_many(&parent->css, nr_reparented);
+	}
 	mutex_unlock(&slab_mutex);
 
 	put_online_mems();
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache()
  2019-06-11 23:18 ` [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Roman Gushchin
@ 2019-06-13  2:04   ` Andrew Morton
  2019-06-13 16:25     ` Roman Gushchin
  0 siblings, 1 reply; 36+ messages in thread
From: Andrew Morton @ 2019-06-13  2:04 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Vladimir Davydov, linux-mm, linux-kernel, kernel-team,
	Johannes Weiner, Shakeel Butt, Waiman Long

On Tue, 11 Jun 2019 16:18:04 -0700 Roman Gushchin <guro@fb.com> wrote:

> Subject: [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache()]

I think mm is too large a place for patches to be described as
affecting simply "mm".  So I'll irritatingly rewrite all these titles to
"mm: memcg/slab:".

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache()
  2019-06-13  2:04   ` Andrew Morton
@ 2019-06-13 16:25     ` Roman Gushchin
  0 siblings, 0 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-06-13 16:25 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Vladimir Davydov, linux-mm, linux-kernel, Kernel Team,
	Johannes Weiner, Shakeel Butt, Waiman Long

On Wed, Jun 12, 2019 at 07:04:23PM -0700, Andrew Morton wrote:
> On Tue, 11 Jun 2019 16:18:04 -0700 Roman Gushchin <guro@fb.com> wrote:
> 
> > Subject: [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache()]
> 
> I think mm is too large a place for patches to be described as
> affecting simply "mm".  So I'll irritatingly rewrite all these titles to
> "mm: memcg/slab:".

Works for me. Thanks!

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation
  2019-06-11 23:18 ` [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation Roman Gushchin
@ 2019-06-16 16:26   ` Vladimir Davydov
  2019-06-25 18:31   ` Shakeel Butt
  1 sibling, 0 replies; 36+ messages in thread
From: Vladimir Davydov @ 2019-06-16 16:26 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, linux-mm, linux-kernel, kernel-team,
	Johannes Weiner, Shakeel Butt, Waiman Long

On Tue, Jun 11, 2019 at 04:18:09PM -0700, Roman Gushchin wrote:
> There is no point in checking the root_cache->memcg_params.dying
> flag on kmem_cache creation path. New allocations shouldn't be
> performed using a dead root kmem_cache, so no new memcg kmem_cache
> creation can be scheduled after the flag is set. And if it was
> scheduled before, flush_memcg_workqueue() will wait for it anyway.
> 
> So let's drop this check to simplify the code.
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>

Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
  2019-06-11 23:18 ` [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Roman Gushchin
@ 2019-06-16 16:27   ` Vladimir Davydov
  2019-06-25 18:33   ` Shakeel Butt
  1 sibling, 0 replies; 36+ messages in thread
From: Vladimir Davydov @ 2019-06-16 16:27 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, linux-mm, linux-kernel, kernel-team,
	Johannes Weiner, Shakeel Butt, Waiman Long

On Tue, Jun 11, 2019 at 04:18:10PM -0700, Roman Gushchin wrote:
> Currently the memcg_params.dying flag and the corresponding
> workqueue used for the asynchronous deactivation of kmem_caches
> is synchronized using the slab_mutex.
> 
> It makes impossible to check this flag from the irq context,
> which will be required in order to implement asynchronous release
> of kmem_caches.
> 
> So let's switch over to the irq-save flavor of the spinlock-based
> synchronization.
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>

Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal
  2019-06-11 23:18 ` [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal Roman Gushchin
@ 2019-06-16 16:29   ` Vladimir Davydov
  2019-06-26  0:15   ` Shakeel Butt
  1 sibling, 0 replies; 36+ messages in thread
From: Vladimir Davydov @ 2019-06-16 16:29 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, linux-mm, linux-kernel, kernel-team,
	Johannes Weiner, Shakeel Butt, Waiman Long

On Tue, Jun 11, 2019 at 04:18:13PM -0700, Roman Gushchin wrote:
> Let's reparent non-root kmem_caches on memcg offlining. This allows us
> to release the memory cgroup without waiting for the last outstanding
> kernel object (e.g. dentry used by another application).
> 
> Since the parent cgroup is already charged, everything we need to do
> is to splice the list of kmem_caches to the parent's kmem_caches list,
> swap the memcg pointer, drop the css refcounter for each kmem_cache
> and adjust the parent's css refcounter.
> 
> Please, note that kmem_cache->memcg_params.memcg isn't a stable
> pointer anymore. It's safe to read it under rcu_read_lock(),
> cgroup_mutex held, or any other way that protects the memory cgroup
> from being released.
> 
> We can race with the slab allocation and deallocation paths. It's not
> a big problem: parent's charge and slab global stats are always
> correct, and we don't care anymore about the child usage and global
> stats. The child cgroup is already offline, so we don't use or show it
> anywhere.
> 
> Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE)
> aren't used anywhere except count_shadow_nodes(). But even there it
> won't break anything: after reparenting "nodes" will be 0 on child
> level (because we're already reparenting shrinker lists), and on
> parent level page stats always were 0, and this patch won't change
> anything.
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>

Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields
  2019-06-11 23:18 ` [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields Roman Gushchin
@ 2019-06-25 18:17   ` Shakeel Butt
  0 siblings, 0 replies; 36+ messages in thread
From: Shakeel Butt @ 2019-06-25 18:17 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Vladimir Davydov, Linux MM, LKML, Kernel Team,
	Johannes Weiner, Waiman Long

On Tue, Jun 11, 2019 at 4:18 PM Roman Gushchin <guro@fb.com> wrote:
>
> The delayed work/rcu deactivation infrastructure of non-root
> kmem_caches can be also used for asynchronous release of these
> objects. Let's get rid of the word "deactivation" in corresponding
> names to make the code look better after generalization.
>
> It's easier to make the renaming first, so that the generalized
> code will look consistent from scratch.
>
> Let's rename struct memcg_cache_params fields:
>   deact_fn -> work_fn
>   deact_rcu_head -> rcu_head
>   deact_work -> work
>
> And RCU/delayed work callbacks in slab common code:
>   kmemcg_deactivate_rcufn -> kmemcg_rcufn
>   kmemcg_deactivate_workfn -> kmemcg_workfn
>
> This patch contains no functional changes, only renamings.
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation
  2019-06-11 23:18 ` [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation Roman Gushchin
@ 2019-06-25 18:17   ` Shakeel Butt
  0 siblings, 0 replies; 36+ messages in thread
From: Shakeel Butt @ 2019-06-25 18:17 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Vladimir Davydov, Linux MM, LKML, Kernel Team,
	Johannes Weiner, Waiman Long

On Tue, Jun 11, 2019 at 4:18 PM Roman Gushchin <guro@fb.com> wrote:
>
> Currently SLUB uses a work scheduled after an RCU grace period
> to deactivate a non-root kmem_cache. This mechanism can be reused
> for kmem_caches release, but requires generalization for SLAB
> case.
>
> Introduce kmemcg_cache_deactivate() function, which calls
> allocator-specific __kmem_cache_deactivate() and schedules
> execution of __kmem_cache_deactivate_after_rcu() with all
> necessary locks in a worker context after an rcu grace period.
>
> Here is the new calling scheme:
>   kmemcg_cache_deactivate()
>     __kmemcg_cache_deactivate()                  SLAB/SLUB-specific
>     kmemcg_rcufn()                               rcu
>       kmemcg_workfn()                            work
>         __kmemcg_cache_deactivate_after_rcu()    SLAB/SLUB-specific
>
> instead of:
>   __kmemcg_cache_deactivate()                    SLAB/SLUB-specific
>     slab_deactivate_memcg_cache_rcu_sched()      SLUB-only
>       kmemcg_rcufn()                             rcu
>         kmemcg_workfn()                          work
>           kmemcg_cache_deact_after_rcu()         SLUB-only
>
> For consistency, all allocator-specific functions start with "__".
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation
  2019-06-11 23:18 ` [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation Roman Gushchin
  2019-06-16 16:26   ` Vladimir Davydov
@ 2019-06-25 18:31   ` Shakeel Butt
  1 sibling, 0 replies; 36+ messages in thread
From: Shakeel Butt @ 2019-06-25 18:31 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Vladimir Davydov, Linux MM, LKML, Kernel Team,
	Johannes Weiner, Waiman Long

On Tue, Jun 11, 2019 at 4:18 PM Roman Gushchin <guro@fb.com> wrote:
>
> There is no point in checking the root_cache->memcg_params.dying
> flag on kmem_cache creation path. New allocations shouldn't be
> performed using a dead root kmem_cache,

Yes, it's the user's responsibility to synchronize the kmem cache
destruction and allocations.

> so no new memcg kmem_cache
> creation can be scheduled after the flag is set. And if it was
> scheduled before, flush_memcg_workqueue() will wait for it anyway.
>
> So let's drop this check to simplify the code.
>
> Signed-off-by: Roman Gushchin <guro@fb.com>

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock
  2019-06-11 23:18 ` [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Roman Gushchin
  2019-06-16 16:27   ` Vladimir Davydov
@ 2019-06-25 18:33   ` Shakeel Butt
  1 sibling, 0 replies; 36+ messages in thread
From: Shakeel Butt @ 2019-06-25 18:33 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Vladimir Davydov, Linux MM, LKML, Kernel Team,
	Johannes Weiner, Waiman Long

On Tue, Jun 11, 2019 at 4:18 PM Roman Gushchin <guro@fb.com> wrote:
>
> Currently the memcg_params.dying flag and the corresponding
> workqueue used for the asynchronous deactivation of kmem_caches
> is synchronized using the slab_mutex.
>
> It makes impossible to check this flag from the irq context,
> which will be required in order to implement asynchronous release
> of kmem_caches.
>
> So let's switch over to the irq-save flavor of the spinlock-based
> synchronization.
>
> Signed-off-by: Roman Gushchin <guro@fb.com>

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management
  2019-06-11 23:18 ` [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management Roman Gushchin
@ 2019-06-25 23:57   ` Shakeel Butt
  2019-11-21 11:17   ` WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management) Christian Borntraeger
  1 sibling, 0 replies; 36+ messages in thread
From: Shakeel Butt @ 2019-06-25 23:57 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Vladimir Davydov, Linux MM, LKML, Kernel Team,
	Johannes Weiner, Waiman Long

On Tue, Jun 11, 2019 at 4:18 PM Roman Gushchin <guro@fb.com> wrote:
>
> Currently each charged slab page holds a reference to the cgroup to
> which it's charged. Kmem_caches are held by the memcg and are released
> all together with the memory cgroup. It means that none of kmem_caches
> are released unless at least one reference to the memcg exists, which
> is very far from optimal.
>
> Let's rework it in a way that allows releasing individual kmem_caches
> as soon as the cgroup is offline, the kmem_cache is empty and there
> are no pending allocations.
>
> To make it possible, let's introduce a new percpu refcounter for
> non-root kmem caches. The counter is initialized to the percpu mode,
> and is switched to the atomic mode during kmem_cache deactivation. The
> counter is bumped for every charged page and also for every running
> allocation. So the kmem_cache can't be released unless all allocations
> complete.
>
> To shutdown non-active empty kmem_caches, let's reuse the work queue,
> previously used for the kmem_cache deactivation. Once the reference
> counter reaches 0, let's schedule an asynchronous kmem_cache release.
>
> * I used the following simple approach to test the performance
> (stolen from another patchset by T. Harding):
>
>     time find / -name fname-no-exist
>     echo 2 > /proc/sys/vm/drop_caches
>     repeat 10 times
>
> Results:
>
>         orig            patched
>
> real    0m1.455s        real    0m1.355s
> user    0m0.206s        user    0m0.219s
> sys     0m0.855s        sys     0m0.807s
>
> real    0m1.487s        real    0m1.699s
> user    0m0.221s        user    0m0.256s
> sys     0m0.806s        sys     0m0.948s
>
> real    0m1.515s        real    0m1.505s
> user    0m0.183s        user    0m0.215s
> sys     0m0.876s        sys     0m0.858s
>
> real    0m1.291s        real    0m1.380s
> user    0m0.193s        user    0m0.198s
> sys     0m0.843s        sys     0m0.786s
>
> real    0m1.364s        real    0m1.374s
> user    0m0.180s        user    0m0.182s
> sys     0m0.868s        sys     0m0.806s
>
> real    0m1.352s        real    0m1.312s
> user    0m0.201s        user    0m0.212s
> sys     0m0.820s        sys     0m0.761s
>
> real    0m1.302s        real    0m1.349s
> user    0m0.205s        user    0m0.203s
> sys     0m0.803s        sys     0m0.792s
>
> real    0m1.334s        real    0m1.301s
> user    0m0.194s        user    0m0.201s
> sys     0m0.806s        sys     0m0.779s
>
> real    0m1.426s        real    0m1.434s
> user    0m0.216s        user    0m0.181s
> sys     0m0.824s        sys     0m0.864s
>
> real    0m1.350s        real    0m1.295s
> user    0m0.200s        user    0m0.190s
> sys     0m0.842s        sys     0m0.811s
>
> So it looks like the difference is not noticeable in this test.
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal
  2019-06-11 23:18 ` [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal Roman Gushchin
  2019-06-16 16:29   ` Vladimir Davydov
@ 2019-06-26  0:15   ` Shakeel Butt
  1 sibling, 0 replies; 36+ messages in thread
From: Shakeel Butt @ 2019-06-26  0:15 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Vladimir Davydov, Linux MM, LKML, Kernel Team,
	Johannes Weiner, Waiman Long

On Tue, Jun 11, 2019 at 4:18 PM Roman Gushchin <guro@fb.com> wrote:
>
> Let's reparent non-root kmem_caches on memcg offlining. This allows us
> to release the memory cgroup without waiting for the last outstanding
> kernel object (e.g. dentry used by another application).
>
> Since the parent cgroup is already charged, everything we need to do
> is to splice the list of kmem_caches to the parent's kmem_caches list,
> swap the memcg pointer, drop the css refcounter for each kmem_cache
> and adjust the parent's css refcounter.
>
> Please, note that kmem_cache->memcg_params.memcg isn't a stable
> pointer anymore. It's safe to read it under rcu_read_lock(),
> cgroup_mutex held, or any other way that protects the memory cgroup
> from being released.
>
> We can race with the slab allocation and deallocation paths. It's not
> a big problem: parent's charge and slab global stats are always
> correct, and we don't care anymore about the child usage and global
> stats. The child cgroup is already offline, so we don't use or show it
> anywhere.
>
> Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE)
> aren't used anywhere except count_shadow_nodes(). But even there it
> won't break anything: after reparenting "nodes" will be 0 on child
> level (because we're already reparenting shrinker lists), and on
> parent level page stats always were 0, and this patch won't change
> anything.
>
> Signed-off-by: Roman Gushchin <guro@fb.com>

The reparenting of top level memcg and "return true" is fixed in the
later patch.

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages
  2019-06-11 23:18 ` [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages Roman Gushchin
@ 2019-06-26  0:15   ` Shakeel Butt
  0 siblings, 0 replies; 36+ messages in thread
From: Shakeel Butt @ 2019-06-26  0:15 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Vladimir Davydov, Linux MM, LKML, Kernel Team,
	Johannes Weiner, Waiman Long

On Tue, Jun 11, 2019 at 4:18 PM Roman Gushchin <guro@fb.com> wrote:
>
> Every slab page charged to a non-root memory cgroup has a pointer
> to the memory cgroup and holds a reference to it, which protects
> a non-empty memory cgroup from being released. At the same time
> the page has a pointer to the corresponding kmem_cache, and also
> hold a reference to the kmem_cache. And kmem_cache by itself
> holds a reference to the cgroup.
>
> So there is clearly some redundancy, which allows to stop setting
> the page->mem_cgroup pointer and rely on getting memcg pointer
> indirectly via kmem_cache. Further it will allow to change this
> pointer easier, without a need to go over all charged pages.
>
> So let's stop setting page->mem_cgroup pointer for slab pages,
> and stop using the css refcounter directly for protecting
> the memory cgroup from going away. Instead rely on kmem_cache
> as an intermediate object.
>
> Make sure that vmstats and shrinker lists are working as previously,
> as well as /proc/kpagecgroup interface.
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Acked-by: Vladimir Davydov <vdavydov.dev@gmail.com>

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 36+ messages in thread

* WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-06-11 23:18 ` [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management Roman Gushchin
  2019-06-25 23:57   ` Shakeel Butt
@ 2019-11-21 11:17   ` Christian Borntraeger
  2019-11-21 13:08     ` Christian Borntraeger
  2019-11-21 16:58     ` Roman Gushchin
  1 sibling, 2 replies; 36+ messages in thread
From: Christian Borntraeger @ 2019-11-21 11:17 UTC (permalink / raw)
  To: guro
  Cc: akpm, hannes, kernel-team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm,
	Christian Borntraeger

Folks,

I do get errors like the following when running a new testcase in our KVM CI.
The test basically unloads kvm, reloads with with hpage=1 (enable huge page
support for guests on s390) start a guest with libvirt and hugepages, shut the
guest down and unload the kvm module. 

WARNING: CPU: 8 PID: 208 at lib/percpu-refcount.c:108 percpu_ref_exit+0x50/0x58
Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE bonding xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_na>
CPU: 8 PID: 208 Comm: kworker/8:1 Not tainted 5.2.0+ #66
Hardware name: IBM 2964 NC9 712 (LPAR)
Workqueue: events sysfs_slab_remove_workfn
Krnl PSW : 0704e00180000000 0000001529746850 (percpu_ref_exit+0x50/0x58)
           R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
Krnl GPRS: 00000000ffff8808 0000001529746740 000003f4e30e8e18 0036008100000000
           0000001f00000000 0035008100000000 0000001fb3573ab8 0000000000000000
           0000001fbdb6de00 0000000000000000 0000001529f01328 0000001fb3573b00
           0000001fbb27e000 0000001fbdb69300 000003e009263d00 000003e009263cd0
Krnl Code: 0000001529746842: f0a0000407fe        srp        4(11,%r0),2046,0
           0000001529746848: 47000700            bc         0,1792
          #000000152974684c: a7f40001            brc        15,152974684e
          >0000001529746850: a7f4fff2            brc        15,1529746834
           0000001529746854: 0707                bcr        0,%r7
           0000001529746856: 0707                bcr        0,%r7
           0000001529746858: eb8ff0580024        stmg       %r8,%r15,88(%r15)
           000000152974685e: a738ffff            lhi        %r3,-1
Call Trace:
([<000003e009263d00>] 0x3e009263d00)
 [<00000015293252ea>] slab_kmem_cache_release+0x3a/0x70 
 [<0000001529b04882>] kobject_put+0xaa/0xe8 
 [<000000152918cf28>] process_one_work+0x1e8/0x428 
 [<000000152918d1b0>] worker_thread+0x48/0x460 
 [<00000015291942c6>] kthread+0x126/0x160 
 [<0000001529b22344>] ret_from_fork+0x28/0x30 
 [<0000001529b2234c>] kernel_thread_starter+0x0/0x10 
Last Breaking-Event-Address:
 [<000000152974684c>] percpu_ref_exit+0x4c/0x58
---[ end trace b035e7da5788eb09 ]---

I have bisected this to
# first bad commit: [f0a3a24b532d9a7e56a33c5112b2a212ed6ec580] mm: memcg/slab: rework non-root kmem_cache lifecycle management

unmounting /sys/fs/cgroup/memory/ before the test makes the problem go away so
it really seems to be related to the new percpu-refs from this patch.
 
Any ideas?

Christian


^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 11:17   ` WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management) Christian Borntraeger
@ 2019-11-21 13:08     ` Christian Borntraeger
  2019-11-21 16:58     ` Roman Gushchin
  1 sibling, 0 replies; 36+ messages in thread
From: Christian Borntraeger @ 2019-11-21 13:08 UTC (permalink / raw)
  To: guro
  Cc: akpm, hannes, kernel-team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm



On 21.11.19 12:17, Christian Borntraeger wrote:
> Folks,
> 
> I do get errors like the following when running a new testcase in our KVM CI.
> The test basically unloads kvm, reloads with with hpage=1 (enable huge page
> support for guests on s390) start a guest with libvirt and hugepages, shut the
> guest down and unload the kvm module. 

It also crashes without large pages. The trigger is really that the time between 
"guest is going away" and rmmod kvm is really short.

i
> 
> WARNING: CPU: 8 PID: 208 at lib/percpu-refcount.c:108 percpu_ref_exit+0x50/0x58
> Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE bonding xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_na>
> CPU: 8 PID: 208 Comm: kworker/8:1 Not tainted 5.2.0+ #66
> Hardware name: IBM 2964 NC9 712 (LPAR)
> Workqueue: events sysfs_slab_remove_workfn
> Krnl PSW : 0704e00180000000 0000001529746850 (percpu_ref_exit+0x50/0x58)
>            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
> Krnl GPRS: 00000000ffff8808 0000001529746740 000003f4e30e8e18 0036008100000000
>            0000001f00000000 0035008100000000 0000001fb3573ab8 0000000000000000
>            0000001fbdb6de00 0000000000000000 0000001529f01328 0000001fb3573b00
>            0000001fbb27e000 0000001fbdb69300 000003e009263d00 000003e009263cd0
> Krnl Code: 0000001529746842: f0a0000407fe        srp        4(11,%r0),2046,0
>            0000001529746848: 47000700            bc         0,1792
>           #000000152974684c: a7f40001            brc        15,152974684e
>           >0000001529746850: a7f4fff2            brc        15,1529746834
>            0000001529746854: 0707                bcr        0,%r7
>            0000001529746856: 0707                bcr        0,%r7
>            0000001529746858: eb8ff0580024        stmg       %r8,%r15,88(%r15)
>            000000152974685e: a738ffff            lhi        %r3,-1
> Call Trace:
> ([<000003e009263d00>] 0x3e009263d00)
>  [<00000015293252ea>] slab_kmem_cache_release+0x3a/0x70 
>  [<0000001529b04882>] kobject_put+0xaa/0xe8 
>  [<000000152918cf28>] process_one_work+0x1e8/0x428 
>  [<000000152918d1b0>] worker_thread+0x48/0x460 
>  [<00000015291942c6>] kthread+0x126/0x160 
>  [<0000001529b22344>] ret_from_fork+0x28/0x30 
>  [<0000001529b2234c>] kernel_thread_starter+0x0/0x10 
> Last Breaking-Event-Address:
>  [<000000152974684c>] percpu_ref_exit+0x4c/0x58
> ---[ end trace b035e7da5788eb09 ]---
> 
> I have bisected this to
> # first bad commit: [f0a3a24b532d9a7e56a33c5112b2a212ed6ec580] mm: memcg/slab: rework non-root kmem_cache lifecycle management
> 
> unmounting /sys/fs/cgroup/memory/ before the test makes the problem go away so
> it really seems to be related to the new percpu-refs from this patch.
>  
> Any ideas?
> 
> Christian
> 


^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 11:17   ` WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management) Christian Borntraeger
  2019-11-21 13:08     ` Christian Borntraeger
@ 2019-11-21 16:58     ` Roman Gushchin
  2019-11-21 16:59       ` Christian Borntraeger
  1 sibling, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-11-21 16:58 UTC (permalink / raw)
  To: Christian Borntraeger
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm

On Thu, Nov 21, 2019 at 12:17:39PM +0100, Christian Borntraeger wrote:
> Folks,
> 
> I do get errors like the following when running a new testcase in our KVM CI.
> The test basically unloads kvm, reloads with with hpage=1 (enable huge page
> support for guests on s390) start a guest with libvirt and hugepages, shut the
> guest down and unload the kvm module. 
> 
> WARNING: CPU: 8 PID: 208 at lib/percpu-refcount.c:108 percpu_ref_exit+0x50/0x58
> Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE bonding xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_na>
> CPU: 8 PID: 208 Comm: kworker/8:1 Not tainted 5.2.0+ #66
> Hardware name: IBM 2964 NC9 712 (LPAR)
> Workqueue: events sysfs_slab_remove_workfn
> Krnl PSW : 0704e00180000000 0000001529746850 (percpu_ref_exit+0x50/0x58)
>            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
> Krnl GPRS: 00000000ffff8808 0000001529746740 000003f4e30e8e18 0036008100000000
>            0000001f00000000 0035008100000000 0000001fb3573ab8 0000000000000000
>            0000001fbdb6de00 0000000000000000 0000001529f01328 0000001fb3573b00
>            0000001fbb27e000 0000001fbdb69300 000003e009263d00 000003e009263cd0
> Krnl Code: 0000001529746842: f0a0000407fe        srp        4(11,%r0),2046,0
>            0000001529746848: 47000700            bc         0,1792
>           #000000152974684c: a7f40001            brc        15,152974684e
>           >0000001529746850: a7f4fff2            brc        15,1529746834
>            0000001529746854: 0707                bcr        0,%r7
>            0000001529746856: 0707                bcr        0,%r7
>            0000001529746858: eb8ff0580024        stmg       %r8,%r15,88(%r15)
>            000000152974685e: a738ffff            lhi        %r3,-1
> Call Trace:
> ([<000003e009263d00>] 0x3e009263d00)
>  [<00000015293252ea>] slab_kmem_cache_release+0x3a/0x70 
>  [<0000001529b04882>] kobject_put+0xaa/0xe8 
>  [<000000152918cf28>] process_one_work+0x1e8/0x428 
>  [<000000152918d1b0>] worker_thread+0x48/0x460 
>  [<00000015291942c6>] kthread+0x126/0x160 
>  [<0000001529b22344>] ret_from_fork+0x28/0x30 
>  [<0000001529b2234c>] kernel_thread_starter+0x0/0x10 
> Last Breaking-Event-Address:
>  [<000000152974684c>] percpu_ref_exit+0x4c/0x58
> ---[ end trace b035e7da5788eb09 ]---
> 
> I have bisected this to
> # first bad commit: [f0a3a24b532d9a7e56a33c5112b2a212ed6ec580] mm: memcg/slab: rework non-root kmem_cache lifecycle management
> 
> unmounting /sys/fs/cgroup/memory/ before the test makes the problem go away so
> it really seems to be related to the new percpu-refs from this patch.
>  
> Any ideas?

Hello, Christian!

It seems to be a race between releasing of the root kmem_cache (caused by rmmod)
and a memcg kmem_cache. Does delaying rmmod for say, a minute, "resolve" the
issue?

I'll take a look at this code path.

Thanks!

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 16:58     ` Roman Gushchin
@ 2019-11-21 16:59       ` Christian Borntraeger
  2019-11-21 18:45         ` Roman Gushchin
  0 siblings, 1 reply; 36+ messages in thread
From: Christian Borntraeger @ 2019-11-21 16:59 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm



On 21.11.19 17:58, Roman Gushchin wrote:
> On Thu, Nov 21, 2019 at 12:17:39PM +0100, Christian Borntraeger wrote:
>> Folks,
>>
>> I do get errors like the following when running a new testcase in our KVM CI.
>> The test basically unloads kvm, reloads with with hpage=1 (enable huge page
>> support for guests on s390) start a guest with libvirt and hugepages, shut the
>> guest down and unload the kvm module. 
>>
>> WARNING: CPU: 8 PID: 208 at lib/percpu-refcount.c:108 percpu_ref_exit+0x50/0x58
>> Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE bonding xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_na>
>> CPU: 8 PID: 208 Comm: kworker/8:1 Not tainted 5.2.0+ #66
>> Hardware name: IBM 2964 NC9 712 (LPAR)
>> Workqueue: events sysfs_slab_remove_workfn
>> Krnl PSW : 0704e00180000000 0000001529746850 (percpu_ref_exit+0x50/0x58)
>>            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
>> Krnl GPRS: 00000000ffff8808 0000001529746740 000003f4e30e8e18 0036008100000000
>>            0000001f00000000 0035008100000000 0000001fb3573ab8 0000000000000000
>>            0000001fbdb6de00 0000000000000000 0000001529f01328 0000001fb3573b00
>>            0000001fbb27e000 0000001fbdb69300 000003e009263d00 000003e009263cd0
>> Krnl Code: 0000001529746842: f0a0000407fe        srp        4(11,%r0),2046,0
>>            0000001529746848: 47000700            bc         0,1792
>>           #000000152974684c: a7f40001            brc        15,152974684e
>>           >0000001529746850: a7f4fff2            brc        15,1529746834
>>            0000001529746854: 0707                bcr        0,%r7
>>            0000001529746856: 0707                bcr        0,%r7
>>            0000001529746858: eb8ff0580024        stmg       %r8,%r15,88(%r15)
>>            000000152974685e: a738ffff            lhi        %r3,-1
>> Call Trace:
>> ([<000003e009263d00>] 0x3e009263d00)
>>  [<00000015293252ea>] slab_kmem_cache_release+0x3a/0x70 
>>  [<0000001529b04882>] kobject_put+0xaa/0xe8 
>>  [<000000152918cf28>] process_one_work+0x1e8/0x428 
>>  [<000000152918d1b0>] worker_thread+0x48/0x460 
>>  [<00000015291942c6>] kthread+0x126/0x160 
>>  [<0000001529b22344>] ret_from_fork+0x28/0x30 
>>  [<0000001529b2234c>] kernel_thread_starter+0x0/0x10 
>> Last Breaking-Event-Address:
>>  [<000000152974684c>] percpu_ref_exit+0x4c/0x58
>> ---[ end trace b035e7da5788eb09 ]---
>>
>> I have bisected this to
>> # first bad commit: [f0a3a24b532d9a7e56a33c5112b2a212ed6ec580] mm: memcg/slab: rework non-root kmem_cache lifecycle management
>>
>> unmounting /sys/fs/cgroup/memory/ before the test makes the problem go away so
>> it really seems to be related to the new percpu-refs from this patch.
>>  
>> Any ideas?
> 
> Hello, Christian!
> 
> It seems to be a race between releasing of the root kmem_cache (caused by rmmod)
> and a memcg kmem_cache. Does delaying rmmod for say, a minute, "resolve" the
> issue?

Yes, rmmod has to be called directly after the guest shutdown to see the issue.
See my 2nd mail.


^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 16:59       ` Christian Borntraeger
@ 2019-11-21 18:45         ` Roman Gushchin
  2019-11-21 20:43           ` Rik van Riel
  2019-11-22 16:28           ` Christian Borntraeger
  0 siblings, 2 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-11-21 18:45 UTC (permalink / raw)
  To: Christian Borntraeger
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm

On Thu, Nov 21, 2019 at 05:59:54PM +0100, Christian Borntraeger wrote:
> 
> 
> On 21.11.19 17:58, Roman Gushchin wrote:
> > On Thu, Nov 21, 2019 at 12:17:39PM +0100, Christian Borntraeger wrote:
> >> Folks,
> >>
> >> I do get errors like the following when running a new testcase in our KVM CI.
> >> The test basically unloads kvm, reloads with with hpage=1 (enable huge page
> >> support for guests on s390) start a guest with libvirt and hugepages, shut the
> >> guest down and unload the kvm module. 
> >>
> >> WARNING: CPU: 8 PID: 208 at lib/percpu-refcount.c:108 percpu_ref_exit+0x50/0x58
> >> Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE bonding xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_na>
> >> CPU: 8 PID: 208 Comm: kworker/8:1 Not tainted 5.2.0+ #66
> >> Hardware name: IBM 2964 NC9 712 (LPAR)
> >> Workqueue: events sysfs_slab_remove_workfn
> >> Krnl PSW : 0704e00180000000 0000001529746850 (percpu_ref_exit+0x50/0x58)
> >>            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
> >> Krnl GPRS: 00000000ffff8808 0000001529746740 000003f4e30e8e18 0036008100000000
> >>            0000001f00000000 0035008100000000 0000001fb3573ab8 0000000000000000
> >>            0000001fbdb6de00 0000000000000000 0000001529f01328 0000001fb3573b00
> >>            0000001fbb27e000 0000001fbdb69300 000003e009263d00 000003e009263cd0
> >> Krnl Code: 0000001529746842: f0a0000407fe        srp        4(11,%r0),2046,0
> >>            0000001529746848: 47000700            bc         0,1792
> >>           #000000152974684c: a7f40001            brc        15,152974684e
> >>           >0000001529746850: a7f4fff2            brc        15,1529746834
> >>            0000001529746854: 0707                bcr        0,%r7
> >>            0000001529746856: 0707                bcr        0,%r7
> >>            0000001529746858: eb8ff0580024        stmg       %r8,%r15,88(%r15)
> >>            000000152974685e: a738ffff            lhi        %r3,-1
> >> Call Trace:
> >> ([<000003e009263d00>] 0x3e009263d00)
> >>  [<00000015293252ea>] slab_kmem_cache_release+0x3a/0x70 
> >>  [<0000001529b04882>] kobject_put+0xaa/0xe8 
> >>  [<000000152918cf28>] process_one_work+0x1e8/0x428 
> >>  [<000000152918d1b0>] worker_thread+0x48/0x460 
> >>  [<00000015291942c6>] kthread+0x126/0x160 
> >>  [<0000001529b22344>] ret_from_fork+0x28/0x30 
> >>  [<0000001529b2234c>] kernel_thread_starter+0x0/0x10 
> >> Last Breaking-Event-Address:
> >>  [<000000152974684c>] percpu_ref_exit+0x4c/0x58
> >> ---[ end trace b035e7da5788eb09 ]---
> >>
> >> I have bisected this to
> >> # first bad commit: [f0a3a24b532d9a7e56a33c5112b2a212ed6ec580] mm: memcg/slab: rework non-root kmem_cache lifecycle management
> >>
> >> unmounting /sys/fs/cgroup/memory/ before the test makes the problem go away so
> >> it really seems to be related to the new percpu-refs from this patch.
> >>  
> >> Any ideas?
> > 
> > Hello, Christian!
> > 
> > It seems to be a race between releasing of the root kmem_cache (caused by rmmod)
> > and a memcg kmem_cache. Does delaying rmmod for say, a minute, "resolve" the
> > issue?
> 
> Yes, rmmod has to be called directly after the guest shutdown to see the issue.
> See my 2nd mail.

I see. Do you know, which kmem_cache it is? If not, can you, please,
figure it out?

I tried to reproduce the issue, but wasn't successful so far. So I wonder
what can make your case special.

Thanks!

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 18:45         ` Roman Gushchin
@ 2019-11-21 20:43           ` Rik van Riel
  2019-11-21 20:55             ` Roman Gushchin
  2019-11-22 16:28           ` Christian Borntraeger
  1 sibling, 1 reply; 36+ messages in thread
From: Rik van Riel @ 2019-11-21 20:43 UTC (permalink / raw)
  To: Roman Gushchin, Christian Borntraeger
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm

On Thu, 2019-11-21 at 13:45 -0500, Roman Gushchin wrote:
> On Thu, Nov 21, 2019 at 05:59:54PM +0100, Christian Borntraeger
> wrote:
> > 
> > 
> > Yes, rmmod has to be called directly after the guest shutdown to
> > see the issue.
> > See my 2nd mail.
> 
> I see. Do you know, which kmem_cache it is? If not, can you, please,
> figure it out?
> 
> I tried to reproduce the issue, but wasn't successful so far. So I
> wonder
> what can make your case special.

I do not know either, but have a guess.

My guess would be that either the slab object or the
slab page is RCU freed, and the kmem_cache destruction
is called before that RCU callback has completed.


^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 20:43           ` Rik van Riel
@ 2019-11-21 20:55             ` Roman Gushchin
  2019-11-21 22:09               ` Roman Gushchin
  0 siblings, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-11-21 20:55 UTC (permalink / raw)
  To: Rik van Riel
  Cc: Christian Borntraeger, akpm, hannes, Kernel Team, linux-kernel,
	linux-mm, longman, shakeelb, vdavydov.dev, Heiko Carstens,
	Janosch Frank, kvm

On Thu, Nov 21, 2019 at 12:43:01PM -0800, Rik van Riel wrote:
> On Thu, 2019-11-21 at 13:45 -0500, Roman Gushchin wrote:
> > On Thu, Nov 21, 2019 at 05:59:54PM +0100, Christian Borntraeger
> > wrote:
> > > 
> > > 
> > > Yes, rmmod has to be called directly after the guest shutdown to
> > > see the issue.
> > > See my 2nd mail.
> > 
> > I see. Do you know, which kmem_cache it is? If not, can you, please,
> > figure it out?
> > 
> > I tried to reproduce the issue, but wasn't successful so far. So I
> > wonder
> > what can make your case special.
> 
> I do not know either, but have a guess.
> 
> My guess would be that either the slab object or the
> slab page is RCU freed, and the kmem_cache destruction
> is called before that RCU callback has completed.
> 

I've a reproducer, but it requires SLAB_TYPESAFE_BY_RCU to panic.
The only question is if it's the same or different issues.
As soon as I'll have a fix, I'll post it here to test.

Thanks!

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 20:55             ` Roman Gushchin
@ 2019-11-21 22:09               ` Roman Gushchin
  2019-11-22 13:00                 ` Christian Borntraeger
  0 siblings, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-11-21 22:09 UTC (permalink / raw)
  To: Rik van Riel
  Cc: Christian Borntraeger, akpm, hannes, Kernel Team, linux-kernel,
	linux-mm, longman, shakeelb, vdavydov.dev, Heiko Carstens,
	Janosch Frank, kvm

On Thu, Nov 21, 2019 at 12:55:28PM -0800, Roman Gushchin wrote:
> On Thu, Nov 21, 2019 at 12:43:01PM -0800, Rik van Riel wrote:
> > On Thu, 2019-11-21 at 13:45 -0500, Roman Gushchin wrote:
> > > On Thu, Nov 21, 2019 at 05:59:54PM +0100, Christian Borntraeger
> > > wrote:
> > > > 
> > > > 
> > > > Yes, rmmod has to be called directly after the guest shutdown to
> > > > see the issue.
> > > > See my 2nd mail.
> > > 
> > > I see. Do you know, which kmem_cache it is? If not, can you, please,
> > > figure it out?
> > > 
> > > I tried to reproduce the issue, but wasn't successful so far. So I
> > > wonder
> > > what can make your case special.
> > 
> > I do not know either, but have a guess.
> > 
> > My guess would be that either the slab object or the
> > slab page is RCU freed, and the kmem_cache destruction
> > is called before that RCU callback has completed.
> > 
> 
> I've a reproducer, but it requires SLAB_TYPESAFE_BY_RCU to panic.
> The only question is if it's the same or different issues.
> As soon as I'll have a fix, I'll post it here to test.

Ah, no, the issue I've reproduced is already fixed by commit b749ecfaf6c5
("mm: memcg/slab: fix panic in __free_slab() caused by premature memcg pointer release").

Christian, can you, please, confirm that you have this one in your tree?

Also, can you, please, provide you config?
And you mentioned some panics, but didn't send any dmesg messages.
Can you, please, provide them?

Thanks!

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 22:09               ` Roman Gushchin
@ 2019-11-22 13:00                 ` Christian Borntraeger
  0 siblings, 0 replies; 36+ messages in thread
From: Christian Borntraeger @ 2019-11-22 13:00 UTC (permalink / raw)
  To: Roman Gushchin, Rik van Riel
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm

[-- Attachment #1: Type: text/plain, Size: 1767 bytes --]



On 21.11.19 23:09, Roman Gushchin wrote:
> On Thu, Nov 21, 2019 at 12:55:28PM -0800, Roman Gushchin wrote:
>> On Thu, Nov 21, 2019 at 12:43:01PM -0800, Rik van Riel wrote:
>>> On Thu, 2019-11-21 at 13:45 -0500, Roman Gushchin wrote:
>>>> On Thu, Nov 21, 2019 at 05:59:54PM +0100, Christian Borntraeger
>>>> wrote:
>>>>>
>>>>>
>>>>> Yes, rmmod has to be called directly after the guest shutdown to
>>>>> see the issue.
>>>>> See my 2nd mail.
>>>>
>>>> I see. Do you know, which kmem_cache it is? If not, can you, please,
>>>> figure it out?

I can try to figure out.

>>>>
>>>> I tried to reproduce the issue, but wasn't successful so far. So I
>>>> wonder
>>>> what can make your case special.
>>>
>>> I do not know either, but have a guess.
>>>
>>> My guess would be that either the slab object or the
>>> slab page is RCU freed, and the kmem_cache destruction
>>> is called before that RCU callback has completed.
>>>
>>
>> I've a reproducer, but it requires SLAB_TYPESAFE_BY_RCU to panic.
>> The only question is if it's the same or different issues.
>> As soon as I'll have a fix, I'll post it here to test.
> 
> Ah, no, the issue I've reproduced is already fixed by commit b749ecfaf6c5
> ("mm: memcg/slab: fix panic in __free_slab() caused by premature memcg pointer release").
> 
> Christian, can you, please, confirm that you have this one in your tree?

The problem still reproduces on the latest next kernel.
> 
> Also, can you, please, provide you config?

I can but this is an s390 config. 

> And you mentioned some panics, but didn't send any dmesg messages.
> Can you, please, provide them?

Those were all the WARN_ONs from percpu_ref_exit, I just had panic_on_warn enabled.

I attached my config and dmesg just in case for the next kernel of today.

[-- Attachment #2: config.txt --]
[-- Type: text/plain, Size: 79462 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/s390 5.4.0-rc8 Kernel Configuration
#

#
# Compiler: gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
#
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=90201
CONFIG_CLANG_VERSION=0
CONFIG_CC_CAN_LINK=y
CONFIG_CC_HAS_ASM_GOTO=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_WARN_MAYBE_UNINITIALIZED=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_BUILD_SALT=""
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_HAVE_KERNEL_UNCOMPRESSED=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
# CONFIG_KERNEL_UNCOMPRESSED is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_CROSS_MEMORY_ATTACH=y
# CONFIG_USELIB is not set
CONFIG_AUDIT=y
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
CONFIG_AUDITSYSCALL=y

#
# IRQ subsystem
#
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
CONFIG_SPARSE_IRQ=y
# CONFIG_GENERIC_IRQ_DEBUGFS is not set
# end of IRQ subsystem

CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
# end of Timers subsystem

CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set

#
# CPU/Task time and stats accounting
#
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
# CONFIG_PSI is not set
# end of CPU/Task time and stats accounting

CONFIG_CPU_ISOLATION=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
# end of RCU Subsystem

CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_IKHEADERS is not set
CONFIG_LOG_BUF_SHIFT=17
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13

#
# Scheduler features
#
# end of Scheduler features

CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_CC_HAS_INT128=y
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_CGROUPS=y
CONFIG_PAGE_COUNTER=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
CONFIG_MEMCG_KMEM=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_WRITEBACK=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_RDMA=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_BPF=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_MULTIUSER=y
# CONFIG_SGETMASK_SYSCALL is not set
# CONFIG_SYSFS_SYSCALL is not set
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
CONFIG_PRINTK_NMI=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_FUTEX_PI=y
CONFIG_HAVE_FUTEX_CMPXCHG=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_IO_URING=y
CONFIG_ADVISE_SYSCALLS=y
CONFIG_MEMBARRIER=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_USERFAULTFD=y
CONFIG_RSEQ=y
# CONFIG_DEBUG_RSEQ is not set
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y
# CONFIG_PC104 is not set

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
# end of Kernel Performance Events And Counters

CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLUB_MEMCG_SYSFS_ON is not set
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_SLAB_MERGE_DEFAULT=y
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
# CONFIG_SHUFFLE_PAGE_ALLOCATOR is not set
CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_SYSTEM_DATA_VERIFICATION=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
# end of General setup

CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_CPU_BIG_ENDIAN=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_PGSTE=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_AUDIT_ARCH=y
CONFIG_NO_IOPORT_MAP=y
# CONFIG_PCI_QUIRKS is not set
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_S390=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_PGTABLE_LEVELS=5
CONFIG_HAVE_LIVEPATCH=y
CONFIG_LIVEPATCH=y

#
# Processor type and features
#
CONFIG_HAVE_MARCH_Z900_FEATURES=y
CONFIG_HAVE_MARCH_Z990_FEATURES=y
CONFIG_HAVE_MARCH_Z9_109_FEATURES=y
CONFIG_HAVE_MARCH_Z10_FEATURES=y
CONFIG_HAVE_MARCH_Z196_FEATURES=y
# CONFIG_MARCH_Z900 is not set
# CONFIG_MARCH_Z990 is not set
# CONFIG_MARCH_Z9_109 is not set
# CONFIG_MARCH_Z10 is not set
CONFIG_MARCH_Z196=y
# CONFIG_MARCH_ZEC12 is not set
# CONFIG_MARCH_Z13 is not set
# CONFIG_MARCH_Z14 is not set
CONFIG_MARCH_ZEC12_TUNE=y
# CONFIG_TUNE_DEFAULT is not set
# CONFIG_TUNE_Z900 is not set
# CONFIG_TUNE_Z990 is not set
# CONFIG_TUNE_Z9_109 is not set
# CONFIG_TUNE_Z10 is not set
# CONFIG_TUNE_Z196 is not set
CONFIG_TUNE_ZEC12=y
# CONFIG_TUNE_Z13 is not set
# CONFIG_TUNE_Z14 is not set
CONFIG_64BIT=y
CONFIG_COMPAT=y
CONFIG_COMPAT_VDSO=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_SMP=y
CONFIG_NR_CPUS=512
CONFIG_HOTPLUG_CPU=y
CONFIG_NODES_SPAN_OTHER_NODES=y
CONFIG_NUMA=y
CONFIG_NODES_SHIFT=4

#
# Select NUMA modes
#
# CONFIG_NUMA_EMU is not set
# end of Select NUMA modes

CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_BOOK=y
CONFIG_SCHED_DRAWER=y
CONFIG_SCHED_TOPOLOGY=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_ARCH_HAS_KEXEC_PURGATORY=y
CONFIG_KEXEC_SIG=y
CONFIG_ARCH_RANDOM=y
# CONFIG_KERNEL_NOBP is not set
CONFIG_EXPOLINE=y
# CONFIG_EXPOLINE_OFF is not set
CONFIG_EXPOLINE_AUTO=y
# CONFIG_EXPOLINE_FULL is not set
CONFIG_RELOCATABLE=y
CONFIG_RANDOMIZE_BASE=y
# end of Processor type and features

#
# Memory setup
#
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_FORCE_MAX_ZONEORDER=9
CONFIG_MAX_PHYSMEM_BITS=46
CONFIG_PACK_STACK=y
# CONFIG_WARN_DYNAMIC_STACK is not set
# end of Memory setup

#
# I/O subsystem
#
CONFIG_QDIO=y
CONFIG_PCI_NR_FUNCTIONS=128
CONFIG_HAS_IOMEM=y
CONFIG_CHSC_SCH=y
CONFIG_SCM_BUS=y
CONFIG_EADM_SCH=m
CONFIG_VFIO_CCW=m
CONFIG_VFIO_AP=m
# end of I/O subsystem

#
# Dump support
#
CONFIG_CRASH_DUMP=y
# end of Dump support

CONFIG_SECCOMP=y

#
# Power Management
#
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_ARCH_SAVE_PAGE_KEYS=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_ADVANCED_DEBUG is not set
CONFIG_PM_SLEEP_DEBUG=y
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
# end of Power Management

CONFIG_CCW=y
CONFIG_HAVE_PNETID=y

#
# Virtualization
#
# CONFIG_PROTECTED_VIRTUALIZATION_GUEST is not set
CONFIG_PFAULT=y
CONFIG_CMM=m
CONFIG_CMM_IUCV=y
CONFIG_APPLDATA_BASE=y
CONFIG_APPLDATA_MEM=m
CONFIG_APPLDATA_OS=m
CONFIG_APPLDATA_NET_SUM=m
CONFIG_S390_HYPFS_FS=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_KVM_ASYNC_PF_SYNC=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_HAVE_KVM_INVALID_WAKEUPS=y
CONFIG_HAVE_KVM_VCPU_ASYNC_IOCTL=y
CONFIG_HAVE_KVM_NO_POLL=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
# CONFIG_KVM_S390_UCONTROL is not set
CONFIG_VHOST_NET=m
CONFIG_VHOST_VSOCK=m
CONFIG_VHOST=m
# CONFIG_VHOST_CROSS_ENDIAN_LEGACY is not set
CONFIG_S390_GUEST=y
# end of Virtualization

#
# General architecture-dependent options
#
CONFIG_CRASH_CORE=y
CONFIG_KEXEC_CORE=y
CONFIG_OPROFILE=m
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
# CONFIG_STATIC_KEYS_SELFTEST is not set
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_NMI=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_HAS_FORTIFY_SOURCE=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
CONFIG_HAVE_ASM_MODVERSIONS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_RSEQ=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
CONFIG_HAVE_RCU_TABLE_FREE=y
CONFIG_HAVE_MMU_GATHER_NO_GATHER=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING=y
CONFIG_ARCH_HAS_SCALED_CPUTIME=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_COPY_THREAD_TLS=y
CONFIG_CLONE_BACKWARDS2=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_OLD_SIGACTION=y
CONFIG_COMPAT_OLD_SIGACTION=y
CONFIG_COMPAT_32BIT_TIME=y
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_STRICT_MODULE_RWX=y
# CONFIG_REFCOUNT_FULL is not set
# CONFIG_LOCK_EVENT_COUNTS is not set
CONFIG_ARCH_HAS_MEM_ENCRYPT=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# end of GCOV-based kernel profiling

CONFIG_PLUGIN_HOSTCC="g++"
CONFIG_HAVE_GCC_PLUGINS=y
CONFIG_GCC_PLUGINS=y

#
# GCC plugins
#
# CONFIG_GCC_PLUGIN_CYC_COMPLEXITY is not set
# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
# CONFIG_GCC_PLUGIN_RANDSTRUCT is not set
# end of GCC plugins
# end of General architecture-dependent options

CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULE_SIG_FORMAT=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_ASM_MODVERSIONS=y
CONFIG_MODULE_REL_CRCS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
CONFIG_MODULE_SIG_ALL=y
# CONFIG_MODULE_SIG_SHA1 is not set
# CONFIG_MODULE_SIG_SHA224 is not set
CONFIG_MODULE_SIG_SHA256=y
# CONFIG_MODULE_SIG_SHA384 is not set
# CONFIG_MODULE_SIG_SHA512 is not set
CONFIG_MODULE_SIG_HASH="sha256"
# CONFIG_MODULE_COMPRESS is not set
# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set
CONFIG_UNUSED_SYMBOLS=y
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
CONFIG_BLK_RQ_ALLOC_TIME=y
CONFIG_BLK_SCSI_REQUEST=y
CONFIG_BLK_CGROUP_RWSTAT=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
# CONFIG_BLK_DEV_ZONED is not set
CONFIG_BLK_DEV_THROTTLING=y
# CONFIG_BLK_DEV_THROTTLING_LOW is not set
# CONFIG_BLK_CMDLINE_PARSER is not set
CONFIG_BLK_WBT=y
CONFIG_BLK_CGROUP_IOLATENCY=y
CONFIG_BLK_CGROUP_IOCOST=y
CONFIG_BLK_WBT_MQ=y
CONFIG_BLK_DEBUG_FS=y
# CONFIG_BLK_SED_OPAL is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
CONFIG_IBM_PARTITION=y
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set
# end of Partition Types

CONFIG_BLOCK_COMPAT=y
CONFIG_BLK_MQ_PCI=y
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_BLK_MQ_RDMA=y
CONFIG_BLK_PM=y

#
# IO Schedulers
#
CONFIG_MQ_IOSCHED_DEADLINE=y
CONFIG_MQ_IOSCHED_KYBER=y
CONFIG_IOSCHED_BFQ=y
CONFIG_BFQ_GROUP_IOSCHED=y
# CONFIG_BFQ_CGROUP_DEBUG is not set
# end of IO Schedulers

CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_ASN1=y
CONFIG_ARCH_INLINE_SPIN_TRYLOCK=y
CONFIG_ARCH_INLINE_SPIN_TRYLOCK_BH=y
CONFIG_ARCH_INLINE_SPIN_LOCK=y
CONFIG_ARCH_INLINE_SPIN_LOCK_BH=y
CONFIG_ARCH_INLINE_SPIN_LOCK_IRQ=y
CONFIG_ARCH_INLINE_SPIN_LOCK_IRQSAVE=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK_BH=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK_IRQRESTORE=y
CONFIG_ARCH_INLINE_READ_TRYLOCK=y
CONFIG_ARCH_INLINE_READ_LOCK=y
CONFIG_ARCH_INLINE_READ_LOCK_BH=y
CONFIG_ARCH_INLINE_READ_LOCK_IRQ=y
CONFIG_ARCH_INLINE_READ_LOCK_IRQSAVE=y
CONFIG_ARCH_INLINE_READ_UNLOCK=y
CONFIG_ARCH_INLINE_READ_UNLOCK_BH=y
CONFIG_ARCH_INLINE_READ_UNLOCK_IRQ=y
CONFIG_ARCH_INLINE_READ_UNLOCK_IRQRESTORE=y
CONFIG_ARCH_INLINE_WRITE_TRYLOCK=y
CONFIG_ARCH_INLINE_WRITE_LOCK=y
CONFIG_ARCH_INLINE_WRITE_LOCK_BH=y
CONFIG_ARCH_INLINE_WRITE_LOCK_IRQ=y
CONFIG_ARCH_INLINE_WRITE_LOCK_IRQSAVE=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK_BH=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE=y
CONFIG_INLINE_SPIN_TRYLOCK=y
CONFIG_INLINE_SPIN_TRYLOCK_BH=y
CONFIG_INLINE_SPIN_LOCK=y
CONFIG_INLINE_SPIN_LOCK_BH=y
CONFIG_INLINE_SPIN_LOCK_IRQ=y
CONFIG_INLINE_SPIN_LOCK_IRQSAVE=y
CONFIG_INLINE_SPIN_UNLOCK_BH=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE=y
CONFIG_INLINE_READ_TRYLOCK=y
CONFIG_INLINE_READ_LOCK=y
CONFIG_INLINE_READ_LOCK_BH=y
CONFIG_INLINE_READ_LOCK_IRQ=y
CONFIG_INLINE_READ_LOCK_IRQSAVE=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_BH=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK_IRQRESTORE=y
CONFIG_INLINE_WRITE_TRYLOCK=y
CONFIG_INLINE_WRITE_LOCK=y
CONFIG_INLINE_WRITE_LOCK_BH=y
CONFIG_INLINE_WRITE_LOCK_IRQ=y
CONFIG_INLINE_WRITE_LOCK_IRQSAVE=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_BH=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_ARCH_HAS_SYSCALL_WRAPPER=y
CONFIG_FREEZER=y

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_STATE=y
CONFIG_ELFCORE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_SCRIPT=y
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y
# end of Executable file formats

#
# Memory Management options
#
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_HAVE_MEMBLOCK_PHYS_MAP=y
CONFIG_HAVE_FAST_GUP=y
CONFIG_ARCH_KEEP_MEMBLOCK=y
CONFIG_MEMORY_ISOLATION=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_MEMORY_BALLOON=y
CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_CONTIG_ALLOC=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_BOUNCE=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_TRANSPARENT_HUGE_PAGECACHE=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_CMA=y
# CONFIG_CMA_DEBUG is not set
# CONFIG_CMA_DEBUGFS is not set
CONFIG_CMA_AREAS=7
CONFIG_MEM_SOFT_DIRTY=y
CONFIG_ZSWAP=y
CONFIG_ZPOOL=y
CONFIG_ZBUD=m
# CONFIG_Z3FOLD is not set
CONFIG_ZSMALLOC=m
# CONFIG_PGTABLE_MAPPING is not set
CONFIG_ZSMALLOC_STAT=y
CONFIG_DEFERRED_STRUCT_PAGE_INIT=y
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_PERCPU_STATS=y
CONFIG_GUP_BENCHMARK=y
# CONFIG_READ_ONLY_THP_FOR_FS is not set
CONFIG_ARCH_HAS_PTE_SPECIAL=y
# end of Memory Management options

CONFIG_NET=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
CONFIG_SKB_EXTENSIONS=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_SCM=y
CONFIG_UNIX_DIAG=m
# CONFIG_TLS is not set
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
# CONFIG_XFRM_INTERFACE is not set
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
# CONFIG_NET_KEY_MIGRATE is not set
CONFIG_IUCV=y
CONFIG_AFIUCV=m
CONFIG_SMC=m
CONFIG_SMC_DIAG=m
# CONFIG_XDP_SOCKETS is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_IP_FIB_TRIE_STATS is not set
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IP_TUNNEL=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE_COMMON=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y
CONFIG_NET_IPVTI=m
CONFIG_NET_UDP_TUNNEL=m
# CONFIG_NET_FOU is not set
# CONFIG_NET_FOU_IP_TUNNELS is not set
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
# CONFIG_INET_ESP_OFFLOAD is not set
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_INET_UDP_DIAG=m
# CONFIG_INET_RAW_DIAG is not set
# CONFIG_INET_DIAG_DESTROY is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
# CONFIG_TCP_CONG_NV is not set
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
# CONFIG_TCP_CONG_DCTCP is not set
# CONFIG_TCP_CONG_CDG is not set
# CONFIG_TCP_CONG_BBR is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=y
CONFIG_IPV6_ROUTER_PREF=y
# CONFIG_IPV6_ROUTE_INFO is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
# CONFIG_INET6_ESP_OFFLOAD is not set
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=m
# CONFIG_IPV6_ILA is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_IPV6_VTI=m
CONFIG_IPV6_SIT=m
# CONFIG_IPV6_SIT_6RD is not set
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_GRE=m
CONFIG_IPV6_MULTIPLE_TABLES=y
CONFIG_IPV6_SUBTREES=y
# CONFIG_IPV6_MROUTE is not set
# CONFIG_IPV6_SEG6_LWTUNNEL is not set
# CONFIG_IPV6_SEG6_HMAC is not set
# CONFIG_NETLABEL is not set
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=m

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_INGRESS=y
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_FAMILY_BRIDGE=y
CONFIG_NETFILTER_FAMILY_ARP=y
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NETFILTER_NETLINK_OSF=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_LOG_COMMON=m
# CONFIG_NF_LOG_NETDEV is not set
CONFIG_NETFILTER_CONNCOUNT=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
# CONFIG_NF_CONNTRACK_ZONES is not set
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CONNTRACK_TIMEOUT=y
CONFIG_NF_CONNTRACK_TIMESTAMP=y
CONFIG_NF_CONNTRACK_LABELS=y
CONFIG_NF_CT_PROTO_DCCP=y
CONFIG_NF_CT_PROTO_GRE=y
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NF_CT_NETLINK_TIMEOUT=m
# CONFIG_NETFILTER_NETLINK_GLUE_CT is not set
CONFIG_NF_NAT=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_MASQUERADE=y
CONFIG_NF_TABLES=m
# CONFIG_NF_TABLES_SET is not set
# CONFIG_NF_TABLES_INET is not set
# CONFIG_NF_TABLES_NETDEV is not set
# CONFIG_NFT_NUMGEN is not set
CONFIG_NFT_CT=m
CONFIG_NFT_COUNTER=m
# CONFIG_NFT_CONNLIMIT is not set
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
# CONFIG_NFT_MASQ is not set
# CONFIG_NFT_REDIR is not set
CONFIG_NFT_NAT=m
# CONFIG_NFT_TUNNEL is not set
# CONFIG_NFT_OBJREF is not set
# CONFIG_NFT_QUEUE is not set
# CONFIG_NFT_QUOTA is not set
# CONFIG_NFT_REJECT is not set
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
# CONFIG_NFT_XFRM is not set
# CONFIG_NFT_SOCKET is not set
# CONFIG_NFT_OSF is not set
# CONFIG_NFT_TPROXY is not set
# CONFIG_NFT_SYNPROXY is not set
# CONFIG_NF_FLOW_TABLE is not set
CONFIG_NETFILTER_XTABLES=m

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_SET=m

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_HMARK=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_NAT=m
# CONFIG_NETFILTER_XT_TARGET_NETMAP is not set
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
# CONFIG_NETFILTER_XT_TARGET_NOTRACK is not set
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
# CONFIG_NETFILTER_XT_TARGET_REDIRECT is not set
CONFIG_NETFILTER_XT_TARGET_MASQUERADE=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_NETFILTER_XT_MATCH_CGROUP is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
# CONFIG_NETFILTER_XT_MATCH_IPCOMP is not set
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_L2TP=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
# CONFIG_NETFILTER_XT_MATCH_SOCKET is not set
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
# end of Core Netfilter Configuration

CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=m
CONFIG_IP_SET_BITMAP_IPMAC=m
CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
# CONFIG_IP_SET_HASH_IPMARK is not set
CONFIG_IP_SET_HASH_IPPORT=m
CONFIG_IP_SET_HASH_IPPORTIP=m
CONFIG_IP_SET_HASH_IPPORTNET=m
# CONFIG_IP_SET_HASH_IPMAC is not set
# CONFIG_IP_SET_HASH_MAC is not set
CONFIG_IP_SET_HASH_NETPORTNET=m
CONFIG_IP_SET_HASH_NET=m
CONFIG_IP_SET_HASH_NETNET=m
CONFIG_IP_SET_HASH_NETPORT=m
CONFIG_IP_SET_HASH_NETIFACE=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m
# CONFIG_IP_VS_IPV6 is not set
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
# CONFIG_IP_VS_PROTO_SCTP is not set

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
# CONFIG_IP_VS_FO is not set
# CONFIG_IP_VS_OVF is not set
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
# CONFIG_IP_VS_MH is not set
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS MH scheduler
#
CONFIG_IP_VS_MH_TAB_INDEX=12

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
# CONFIG_NF_SOCKET_IPV4 is not set
CONFIG_NF_TPROXY_IPV4=m
CONFIG_NF_TABLES_IPV4=y
# CONFIG_NFT_DUP_IPV4 is not set
# CONFIG_NFT_FIB_IPV4 is not set
CONFIG_NF_TABLES_ARP=y
CONFIG_NF_DUP_IPV4=m
# CONFIG_NF_LOG_ARP is not set
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
# CONFIG_IP_NF_TARGET_SYNPROXY is not set
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
# CONFIG_IP_NF_TARGET_NETMAP is not set
# CONFIG_IP_NF_TARGET_REDIRECT is not set
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# end of IP: Netfilter Configuration

#
# IPv6: Netfilter Configuration
#
# CONFIG_NF_SOCKET_IPV6 is not set
CONFIG_NF_TPROXY_IPV6=m
CONFIG_NF_TABLES_IPV6=y
# CONFIG_NFT_DUP_IPV6 is not set
# CONFIG_NFT_FIB_IPV6 is not set
CONFIG_NF_DUP_IPV6=m
CONFIG_NF_REJECT_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RPFILTER=m
CONFIG_IP6_NF_MATCH_RT=m
# CONFIG_IP6_NF_MATCH_SRH is not set
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
# CONFIG_IP6_NF_TARGET_SYNPROXY is not set
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
CONFIG_IP6_NF_NAT=m
CONFIG_IP6_NF_TARGET_MASQUERADE=m
# CONFIG_IP6_NF_TARGET_NPT is not set
# end of IPv6: Netfilter Configuration

CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_TABLES_BRIDGE=m
# CONFIG_NFT_BRIDGE_META is not set
# CONFIG_NF_LOG_BRIDGE is not set
# CONFIG_NF_CONNTRACK_BRIDGE is not set
# CONFIG_BRIDGE_NF_EBTABLES is not set
# CONFIG_BPFILTER is not set
# CONFIG_IP_DCCP is not set
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_OBJCNT is not set
CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5=y
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1 is not set
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set
CONFIG_SCTP_COOKIE_HMAC_MD5=y
# CONFIG_SCTP_COOKIE_HMAC_SHA1 is not set
CONFIG_INET_SCTP_DIAG=m
CONFIG_RDS=m
CONFIG_RDS_RDMA=m
CONFIG_RDS_TCP=m
# CONFIG_RDS_DEBUG is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_BRIDGE_VLAN_FILTERING is not set
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
# CONFIG_VLAN_8021Q_MVRP is not set
# CONFIG_DECNET is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_6LOWPAN is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
# CONFIG_NET_SCH_CBS is not set
# CONFIG_NET_SCH_ETF is not set
# CONFIG_NET_SCH_TAPRIO is not set
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_MQPRIO=m
# CONFIG_NET_SCH_SKBPRIO is not set
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_QFQ=m
CONFIG_NET_SCH_CODEL=m
CONFIG_NET_SCH_FQ_CODEL=m
# CONFIG_NET_SCH_CAKE is not set
# CONFIG_NET_SCH_FQ is not set
# CONFIG_NET_SCH_HHF is not set
# CONFIG_NET_SCH_PIE is not set
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_SCH_PLUG=m
# CONFIG_NET_SCH_DEFAULT is not set

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_CLS_BPF=m
# CONFIG_NET_CLS_FLOWER is not set
# CONFIG_NET_CLS_MATCHALL is not set
# CONFIG_NET_EMATCH is not set
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
# CONFIG_NET_ACT_SAMPLE is not set
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
# CONFIG_NET_ACT_MPLS is not set
# CONFIG_NET_ACT_VLAN is not set
# CONFIG_NET_ACT_BPF is not set
# CONFIG_NET_ACT_CONNMARK is not set
# CONFIG_NET_ACT_CTINFO is not set
# CONFIG_NET_ACT_SKBMOD is not set
# CONFIG_NET_ACT_IFE is not set
# CONFIG_NET_ACT_TUNNEL_KEY is not set
# CONFIG_NET_ACT_CT is not set
# CONFIG_NET_TC_SKB_EXT is not set
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
CONFIG_OPENVSWITCH=m
CONFIG_OPENVSWITCH_GRE=m
CONFIG_OPENVSWITCH_VXLAN=m
CONFIG_VSOCKETS=m
CONFIG_VSOCKETS_DIAG=m
CONFIG_VIRTIO_VSOCKETS=m
CONFIG_VIRTIO_VSOCKETS_COMMON=m
CONFIG_NETLINK_DIAG=m
CONFIG_MPLS=y
CONFIG_NET_MPLS_GSO=m
# CONFIG_MPLS_ROUTING is not set
CONFIG_NET_NSH=m
# CONFIG_HSR is not set
# CONFIG_NET_SWITCHDEV is not set
# CONFIG_NET_L3_MASTER_DEV is not set
# CONFIG_NET_NCSI is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
CONFIG_BPF_JIT=y
# CONFIG_BPF_STREAM_PARSER is not set
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_DROP_MONITOR is not set
# end of Network testing
# end of Networking options

# CONFIG_CAN is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
CONFIG_FIB_RULES=y
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
CONFIG_CEPH_LIB=m
# CONFIG_CEPH_LIB_PRETTYDEBUG is not set
# CONFIG_CEPH_LIB_USE_DNS_RESOLVER is not set
# CONFIG_NFC is not set
# CONFIG_PSAMPLE is not set
# CONFIG_NET_IFE is not set
# CONFIG_LWTUNNEL is not set
CONFIG_DST_CACHE=y
CONFIG_GRO_CELLS=y
CONFIG_NET_DEVLINK=y
CONFIG_PAGE_POOL=y
CONFIG_FAILOVER=m
CONFIG_HAVE_EBPF_JIT=y

#
# Device Drivers
#
CONFIG_HAVE_PCI=y
CONFIG_PCI=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_POWER_SUPERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
# CONFIG_PCIE_PTM is not set
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
# CONFIG_PCI_IOV is not set
# CONFIG_PCI_PRI is not set
# CONFIG_PCI_PASID is not set
CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_CPCI is not set
# CONFIG_HOTPLUG_PCI_SHPC is not set
CONFIG_HOTPLUG_PCI_S390=y

#
# PCI controller drivers
#

#
# DesignWare PCI Core Support
#
# CONFIG_PCIE_DW_PLAT_HOST is not set
# CONFIG_PCI_MESON is not set
# end of DesignWare PCI Core Support

#
# Cadence PCIe controllers support
#
# end of Cadence PCIe controllers support
# end of PCI controller drivers

#
# PCI Endpoint
#
# CONFIG_PCI_ENDPOINT is not set
# end of PCI Endpoint

#
# PCI switch controller drivers
#
# CONFIG_PCI_SW_SWITCHTEC is not set
# end of PCI switch controller drivers

# CONFIG_PCCARD is not set
# CONFIG_RAPIDIO is not set

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER=y
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
# CONFIG_DEVTMPFS_MOUNT is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y

#
# Firmware loader
#
CONFIG_FW_LOADER=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_FW_LOADER_USER_HELPER is not set
# CONFIG_FW_LOADER_COMPRESS is not set
CONFIG_FW_CACHE=y
# end of Firmware loader

CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not set
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_SYS_HYPERVISOR=y
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_GENERIC_CPU_VULNERABILITIES=y
CONFIG_DMA_SHARED_BUFFER=y
# CONFIG_DMA_FENCE_TRACE is not set
# end of Generic Driver Options

#
# Bus devices
#
# end of Bus devices

CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_GNSS is not set
# CONFIG_MTD is not set
# CONFIG_OF is not set
# CONFIG_PARPORT is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_NULL_BLK is not set
CONFIG_CDROM=m
# CONFIG_BLK_DEV_PCIESSD_MTIP32XX is not set
CONFIG_ZRAM=m
# CONFIG_ZRAM_WRITEBACK is not set
# CONFIG_ZRAM_MEMORY_TRACKING is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
# CONFIG_DRBD_FAULT_INJECTION is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SKD is not set
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=32768
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set

#
# S/390 block device drivers
#
# CONFIG_BLK_DEV_XPRAM is not set
CONFIG_DCSSBLK=m
CONFIG_DASD=y
CONFIG_DASD_PROFILE=y
CONFIG_DASD_ECKD=y
CONFIG_DASD_FBA=y
CONFIG_DASD_DIAG=y
CONFIG_DASD_EER=y
CONFIG_SCM_BLOCK=m
CONFIG_VIRTIO_BLK=y
# CONFIG_VIRTIO_BLK_SCSI is not set
CONFIG_BLK_DEV_RBD=m
# CONFIG_BLK_DEV_RSXX is not set

#
# NVME Support
#
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
# CONFIG_NVME_MULTIPATH is not set
# CONFIG_NVME_RDMA is not set
# CONFIG_NVME_FC is not set
# CONFIG_NVME_TCP is not set
# CONFIG_NVME_TARGET is not set
# end of NVME Support

#
# Misc devices
#
# CONFIG_WATCH_QUEUE is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_PHANTOM is not set
# CONFIG_TIFM_CORE is not set
CONFIG_ENCLOSURE_SERVICES=m
# CONFIG_HP_ILO is not set
# CONFIG_SRAM is not set
# CONFIG_PCI_ENDPOINT_TEST is not set
# CONFIG_XILINX_SDFEC is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_93CX6 is not set
# end of EEPROM support

# CONFIG_CB710_CORE is not set

#
# Texas Instruments shared transport line discipline
#
# end of Texas Instruments shared transport line discipline

#
# Altera FPGA firmware download module (requires I2C)
#

#
# Intel MIC & related support
#
# CONFIG_VOP_BUS is not set
# end of Intel MIC & related support

CONFIG_GENWQE=m
CONFIG_GENWQE_PLATFORM_ERROR_RECOVERY=0
# CONFIG_ECHO is not set
# CONFIG_MISC_ALCOR_PCI is not set
# CONFIG_MISC_RTSX_PCI is not set
# CONFIG_HABANA_AI is not set
# end of Misc devices

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m
# end of SCSI Transports

CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
# CONFIG_ISCSI_BOOT_SYSFS is not set
# CONFIG_SCSI_CXGB3_ISCSI is not set
# CONFIG_SCSI_CXGB4_ISCSI is not set
# CONFIG_SCSI_BNX2_ISCSI is not set
# CONFIG_BE2ISCSI is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_HPSA is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_3W_SAS is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_MVUMI is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_SCSI_ESAS2R is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_MPT3SAS is not set
# CONFIG_SCSI_MPT2SAS is not set
# CONFIG_SCSI_UFSHCD is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_MYRB is not set
# CONFIG_LIBFC is not set
# CONFIG_SCSI_SNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_FDOMAIN_PCI is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_WD719X is not set
CONFIG_SCSI_DEBUG=m
CONFIG_ZFCP=y
# CONFIG_SCSI_PMCRAID is not set
# CONFIG_SCSI_PM8001 is not set
# CONFIG_SCSI_BFA_FC is not set
CONFIG_SCSI_VIRTIO=m
# CONFIG_SCSI_CHELSIO_FCOE is not set
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=m
CONFIG_SCSI_DH_HP_SW=m
CONFIG_SCSI_DH_EMC=m
CONFIG_SCSI_DH_ALUA=m
# end of SCSI device support

# CONFIG_ATA is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_MD_CLUSTER=m
CONFIG_BCACHE=m
# CONFIG_BCACHE_DEBUG is not set
# CONFIG_BCACHE_CLOSURES_DEBUG is not set
CONFIG_BLK_DEV_DM_BUILTIN=y
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_BUFIO=m
# CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING is not set
CONFIG_DM_BIO_PRISON=m
CONFIG_DM_PERSISTENT_DATA=m
CONFIG_DM_UNSTRIPED=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
# CONFIG_DM_CACHE is not set
CONFIG_DM_WRITECACHE=m
# CONFIG_DM_ERA is not set
CONFIG_DM_CLONE=m
CONFIG_DM_MIRROR=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_RAID=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
CONFIG_DM_DELAY=m
# CONFIG_DM_DUST is not set
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m
CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG=y
# CONFIG_DM_VERITY_FEC is not set
CONFIG_DM_SWITCH=m
# CONFIG_DM_LOG_WRITES is not set
CONFIG_DM_INTEGRITY=m
# CONFIG_TARGET_CORE is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_FIREWIRE_NOSY is not set
# end of IEEE 1394 (FireWire) support

CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
CONFIG_EQUALIZER=m
# CONFIG_NET_FC is not set
CONFIG_IFB=m
# CONFIG_NET_TEAM is not set
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
# CONFIG_IPVLAN is not set
CONFIG_VXLAN=m
# CONFIG_GENEVE is not set
# CONFIG_GTP is not set
# CONFIG_MACSEC is not set
# CONFIG_NETCONSOLE is not set
CONFIG_TUN=m
CONFIG_TAP=m
# CONFIG_TUN_VNET_CROSS_LE is not set
CONFIG_VETH=m
CONFIG_VIRTIO_NET=m
CONFIG_NLMON=m
# CONFIG_VSOCKMON is not set
# CONFIG_ARCNET is not set
CONFIG_ETHERNET=y
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
# CONFIG_NET_VENDOR_AGERE is not set
# CONFIG_NET_VENDOR_ALACRITECH is not set
# CONFIG_NET_VENDOR_ALTEON is not set
# CONFIG_ALTERA_TSE is not set
# CONFIG_NET_VENDOR_AMAZON is not set
# CONFIG_NET_VENDOR_AMD is not set
# CONFIG_NET_VENDOR_AQUANTIA is not set
# CONFIG_NET_VENDOR_ARC is not set
# CONFIG_NET_VENDOR_ATHEROS is not set
# CONFIG_NET_VENDOR_AURORA is not set
# CONFIG_NET_VENDOR_BROADCOM is not set
# CONFIG_NET_VENDOR_BROCADE is not set
# CONFIG_NET_VENDOR_CADENCE is not set
# CONFIG_NET_VENDOR_CAVIUM is not set
# CONFIG_NET_VENDOR_CHELSIO is not set
# CONFIG_NET_VENDOR_CISCO is not set
# CONFIG_NET_VENDOR_CORTINA is not set
# CONFIG_DNET is not set
# CONFIG_NET_VENDOR_DEC is not set
# CONFIG_NET_VENDOR_DLINK is not set
# CONFIG_NET_VENDOR_EMULEX is not set
# CONFIG_NET_VENDOR_EZCHIP is not set
# CONFIG_NET_VENDOR_GOOGLE is not set
# CONFIG_NET_VENDOR_HUAWEI is not set
# CONFIG_NET_VENDOR_INTEL is not set
# CONFIG_JME is not set
# CONFIG_NET_VENDOR_MARVELL is not set
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=m
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_MLX4_CORE_GEN2=y
CONFIG_MLX5_CORE=m
# CONFIG_MLX5_FPGA is not set
CONFIG_MLX5_CORE_EN=y
CONFIG_MLX5_EN_ARFS=y
CONFIG_MLX5_EN_RXNFC=y
CONFIG_MLX5_MPFS=y
# CONFIG_MLX5_CORE_IPOIB is not set
# CONFIG_MLXSW_CORE is not set
# CONFIG_MLXFW is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_MICROCHIP is not set
# CONFIG_NET_VENDOR_MICROSEMI is not set
# CONFIG_NET_VENDOR_MYRI is not set
# CONFIG_FEALNX is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
# CONFIG_NET_VENDOR_NETERION is not set
# CONFIG_NET_VENDOR_NETRONOME is not set
# CONFIG_NET_VENDOR_NI is not set
# CONFIG_NET_VENDOR_NVIDIA is not set
# CONFIG_NET_VENDOR_OKI is not set
# CONFIG_ETHOC is not set
# CONFIG_NET_VENDOR_PACKET_ENGINES is not set
# CONFIG_NET_VENDOR_PENSANDO is not set
# CONFIG_NET_VENDOR_QLOGIC is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RDC is not set
# CONFIG_NET_VENDOR_REALTEK is not set
# CONFIG_NET_VENDOR_RENESAS is not set
# CONFIG_NET_VENDOR_ROCKER is not set
# CONFIG_NET_VENDOR_SAMSUNG is not set
# CONFIG_NET_VENDOR_SEEQ is not set
# CONFIG_NET_VENDOR_SOLARFLARE is not set
# CONFIG_NET_VENDOR_SILAN is not set
# CONFIG_NET_VENDOR_SIS is not set
# CONFIG_NET_VENDOR_SMSC is not set
# CONFIG_NET_VENDOR_SOCIONEXT is not set
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SUN is not set
# CONFIG_NET_VENDOR_SYNOPSYS is not set
# CONFIG_NET_VENDOR_TEHUTI is not set
# CONFIG_NET_VENDOR_TI is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_NET_VENDOR_XILINX=y
# CONFIG_XILINX_AXI_EMAC is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_MDIO_DEVICE is not set
# CONFIG_PHYLIB is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPPOE=m
CONFIG_PPTP=m
CONFIG_PPPOL2TP=m
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
# CONFIG_SLIP is not set
CONFIG_SLHC=m

#
# S/390 network device drivers
#
CONFIG_LCS=m
CONFIG_CTCM=m
CONFIG_NETIUCV=m
CONFIG_SMSGIUCV=m
CONFIG_SMSGIUCV_EVENT=m
CONFIG_QETH=y
CONFIG_QETH_L2=y
CONFIG_QETH_L3=y
CONFIG_CCWGROUP=y
CONFIG_ISM=m
# end of S/390 network device drivers

#
# Host-side USB support is needed for USB Network Adapter support
#

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
# CONFIG_VMXNET3 is not set
# CONFIG_NETDEVSIM is not set
CONFIG_NET_FAILOVER=m
# CONFIG_NVM is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
# CONFIG_INPUT_SPARSEKMAP is not set
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
# CONFIG_INPUT_KEYBOARD is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
# CONFIG_SERIO is not set
# CONFIG_GAMEPORT is not set
# end of Hardware I/O ports
# end of Input device support

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=0
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
CONFIG_NULL_TTY=m
CONFIG_LDISC_AUTOLOAD=y
CONFIG_DEVMEM=y
# CONFIG_DEVKMEM is not set

#
# Serial drivers
#

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_UARTLITE is not set
# CONFIG_SERIAL_JSM is not set
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_FSL_LINFLEXUART is not set
# end of Serial drivers

# CONFIG_SERIAL_DEV_BUS is not set
# CONFIG_TTY_PRINTK is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IUCV=y
CONFIG_VIRTIO_CONSOLE=y
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=m
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_HW_RANDOM_S390=m
# CONFIG_APPLICOM is not set
CONFIG_RAW_DRIVER=m
CONFIG_MAX_RAW_DEVS=256
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=y
# CONFIG_TCG_VTPM_PROXY is not set
CONFIG_DEVPORT=y

#
# S/390 character device drivers
#
CONFIG_TN3270=y
CONFIG_TN3270_TTY=y
CONFIG_TN3270_FS=y
CONFIG_TN3270_CONSOLE=y
CONFIG_TN3215=y
CONFIG_TN3215_CONSOLE=y
CONFIG_CCW_CONSOLE=y
CONFIG_SCLP_TTY=y
CONFIG_SCLP_CONSOLE=y
CONFIG_SCLP_VT220_TTY=y
CONFIG_SCLP_VT220_CONSOLE=y
CONFIG_HMC_DRV=m
# CONFIG_SCLP_OFB is not set
CONFIG_S390_TAPE=m

#
# S/390 tape hardware support
#
CONFIG_S390_TAPE_34XX=m
CONFIG_S390_TAPE_3590=m
CONFIG_VMLOGRDR=m
CONFIG_VMCP=y
CONFIG_VMCP_CMA_SIZE=4
CONFIG_MONREADER=m
CONFIG_MONWRITER=m
CONFIG_S390_VMUR=m
# CONFIG_XILLYBUS is not set
# end of Character devices

# CONFIG_RANDOM_TRUST_CPU is not set
# CONFIG_RANDOM_TRUST_BOOTLOADER is not set

#
# I2C support
#
# CONFIG_I2C is not set
# end of I2C support

# CONFIG_I3C is not set
# CONFIG_SPI is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
# CONFIG_PPS is not set

#
# PTP clock support
#
# CONFIG_PTP_1588_CLOCK is not set

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# end of PTP clock support

# CONFIG_PINCTRL is not set
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
# CONFIG_POWER_AVS is not set
# CONFIG_POWER_RESET is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_WATCHDOG_NOWAYOUT=y
CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
CONFIG_WATCHDOG_OPEN_TIMEOUT=0
# CONFIG_WATCHDOG_SYSFS is not set

#
# Watchdog Pretimeout Governors
#
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
# CONFIG_XILINX_WATCHDOG is not set
# CONFIG_CADENCE_WATCHDOG is not set
# CONFIG_DW_WATCHDOG is not set
# CONFIG_MAX63XX_WATCHDOG is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_I6300ESB_WDT is not set
CONFIG_DIAG288_WATCHDOG=m

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
# CONFIG_BCMA is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_MADERA is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_LPC_ICH is not set
# CONFIG_LPC_SCH is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_TQMX86 is not set
# CONFIG_MFD_VX855 is not set
# end of Multifunction device drivers

# CONFIG_REGULATOR is not set
# CONFIG_RC_CORE is not set
# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
# CONFIG_DRM is not set
# CONFIG_DRM_DP_CEC is not set

#
# ARM devices
#
# end of ARM devices

#
# ACP (Audio CoProcessor) Configuration
#
# end of ACP (Audio CoProcessor) Configuration

#
# Frame buffer Devices
#
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_MODE_HELPERS is not set
# CONFIG_FB_TILEBLITTING is not set

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_UVESA is not set
# CONFIG_FB_OPENCORES is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I740 is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_IBM_GXT4500 is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_SIMPLE is not set
# CONFIG_FB_SM712 is not set
# end of Frame buffer Devices

#
# Backlight & LCD device support
#
# CONFIG_LCD_CLASS_DEVICE is not set
# CONFIG_BACKLIGHT_CLASS_DEVICE is not set
# end of Backlight & LCD device support

#
# Console display driver support
#
CONFIG_DUMMY_CONSOLE=y
CONFIG_DUMMY_CONSOLE_COLUMNS=80
CONFIG_DUMMY_CONSOLE_ROWS=25
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER is not set
# end of Console display driver support

# CONFIG_LOGO is not set
# end of Graphics support

# CONFIG_SOUND is not set

#
# HID support
#
# CONFIG_HID is not set
# end of HID support

CONFIG_USB_OHCI_LITTLE_ENDIAN=y
# CONFIG_USB_SUPPORT is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
CONFIG_INFINIBAND=m
# CONFIG_INFINIBAND_USER_MAD is not set
CONFIG_INFINIBAND_USER_ACCESS=m
# CONFIG_INFINIBAND_EXP_LEGACY_VERBS_NEW_UAPI is not set
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ON_DEMAND_PAGING=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS=y
# CONFIG_INFINIBAND_MTHCA is not set
CONFIG_MLX4_INFINIBAND=m
CONFIG_MLX5_INFINIBAND=m
# CONFIG_INFINIBAND_OCRDMA is not set
# CONFIG_RDMA_RXE is not set
# CONFIG_RDMA_SIW is not set
# CONFIG_INFINIBAND_IPOIB is not set
# CONFIG_INFINIBAND_SRP is not set
# CONFIG_INFINIBAND_ISER is not set
# CONFIG_DMADEVICES is not set

#
# DMABUF options
#
CONFIG_SYNC_FILE=y
# CONFIG_SW_SYNC is not set
# CONFIG_UDMABUF is not set
# CONFIG_DMABUF_SELFTESTS is not set
# end of DMABUF options

# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO_VIRQFD=m
CONFIG_VFIO=m
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_PCI=m
CONFIG_VFIO_MDEV=m
CONFIG_VFIO_MDEV_DEVICE=m
CONFIG_IRQ_BYPASS_MANAGER=m
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTIO_INPUT=y
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
# end of Microsoft Hyper-V guest support

# CONFIG_GREYBUS is not set
# CONFIG_STAGING is not set
# CONFIG_GOLDFISH is not set
# CONFIG_HWSPINLOCK is not set

#
# Clock Source drivers
#
# end of Clock Source drivers

# CONFIG_MAILBOX is not set
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y

#
# Generic IOMMU Pagetable Support
#
# end of Generic IOMMU Pagetable Support

# CONFIG_IOMMU_DEBUGFS is not set
# CONFIG_IOMMU_DEFAULT_PASSTHROUGH is not set
CONFIG_S390_IOMMU=y
CONFIG_S390_CCW_IOMMU=y
CONFIG_S390_AP_IOMMU=y

#
# Remoteproc drivers
#
# CONFIG_REMOTEPROC is not set
# end of Remoteproc drivers

#
# Rpmsg drivers
#
# CONFIG_RPMSG_VIRTIO is not set
# end of Rpmsg drivers

#
# SOC (System On Chip) specific Drivers
#

#
# Amlogic SoC drivers
#
# end of Amlogic SoC drivers

#
# Aspeed SoC drivers
#
# end of Aspeed SoC drivers

#
# Broadcom SoC drivers
#
# end of Broadcom SoC drivers

#
# NXP/Freescale QorIQ SoC drivers
#
# end of NXP/Freescale QorIQ SoC drivers

#
# i.MX SoC drivers
#
# end of i.MX SoC drivers

#
# Qualcomm SoC drivers
#
# end of Qualcomm SoC drivers

# CONFIG_SOC_TI is not set

#
# Xilinx SoC drivers
#
# CONFIG_XILINX_VCU is not set
# end of Xilinx SoC drivers
# end of SOC (System On Chip) specific Drivers

# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_NTB is not set
# CONFIG_VME_BUS is not set
# CONFIG_PWM is not set

#
# IRQ chip support
#
# end of IRQ chip support

# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set

#
# PHY Subsystem
#
# CONFIG_GENERIC_PHY is not set
# CONFIG_BCM_KONA_USB2_PHY is not set
# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
# end of PHY Subsystem

# CONFIG_POWERCAP is not set
# CONFIG_MCB is not set

#
# Performance monitor support
#
# end of Performance monitor support

# CONFIG_RAS is not set

#
# Android
#
# CONFIG_ANDROID is not set
# end of Android

# CONFIG_LIBNVDIMM is not set
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
# CONFIG_DEV_DAX is not set
# CONFIG_NVMEM is not set

#
# HW tracing support
#
# CONFIG_STM is not set
# CONFIG_INTEL_TH is not set
# end of HW tracing support

# CONFIG_FPGA is not set
# CONFIG_SIOX is not set
# CONFIG_SLIMBUS is not set
# CONFIG_INTERCONNECT is not set
# CONFIG_COUNTER is not set
# end of Device Drivers

#
# File systems
#
# CONFIG_VALIDATE_FS_PARSER is not set
CONFIG_FS_IOMAP=y
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT2=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
CONFIG_JBD2_DEBUG=y
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
CONFIG_JFS_STATISTICS=y
CONFIG_XFS_FS=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
# CONFIG_XFS_ONLINE_SCRUB is not set
# CONFIG_XFS_WARN is not set
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_BTRFS_FS=y
CONFIG_BTRFS_FS_POSIX_ACL=y
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
# CONFIG_BTRFS_ASSERT is not set
# CONFIG_BTRFS_FS_REF_VERIFY is not set
CONFIG_NILFS2_FS=m
# CONFIG_F2FS_FS is not set
CONFIG_FS_DAX=y
CONFIG_FS_DAX_LIMITED=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
CONFIG_EXPORTFS_BLOCK_OPS=y
CONFIG_FILE_LOCKING=y
CONFIG_MANDATORY_FILE_LOCKING=y
CONFIG_FS_ENCRYPTION=y
CONFIG_FS_VERITY=y
# CONFIG_FS_VERITY_DEBUG is not set
CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=m
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_AUTOFS4_FS=m
CONFIG_AUTOFS_FS=m
CONFIG_FUSE_FS=y
CONFIG_CUSE=m
CONFIG_VIRTIO_FS=m
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

#
# Caches
#
CONFIG_FSCACHE=m
# CONFIG_FSCACHE_STATS is not set
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set
# end of Caches

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
# end of CD-ROM/DVD Filesystems

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_FAT_DEFAULT_UTF8 is not set
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y
# end of DOS/FAT/NT Filesystems

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
# CONFIG_PROC_VMCORE_DEVICE_DUMP is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_PROC_CHILDREN=y
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_MEMFD_CREATE=y
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
CONFIG_CONFIGFS_FS=m
# end of Pseudo filesystems

CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ORANGEFS_FS is not set
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
CONFIG_ECRYPT_FS=m
# CONFIG_ECRYPT_FS_MESSAGING is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=m
CONFIG_CRAMFS_BLOCKDEV=y
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_FILE_CACHE=y
# CONFIG_SQUASHFS_FILE_DIRECT is not set
CONFIG_SQUASHFS_DECOMP_SINGLE=y
# CONFIG_SQUASHFS_DECOMP_MULTI is not set
# CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU is not set
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_LZ4=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_ZSTD=y
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
CONFIG_ROMFS_FS=m
CONFIG_ROMFS_BACKED_BY_BLOCK=y
CONFIG_ROMFS_ON_BLOCK=y
# CONFIG_PSTORE is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_EROFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=m
CONFIG_NFS_V3=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
CONFIG_NFS_SWAP=y
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_FSCACHE is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_BLOCKLAYOUT is not set
# CONFIG_NFSD_SCSILAYOUT is not set
# CONFIG_NFSD_FLEXFILELAYOUT is not set
CONFIG_NFSD_V4_SECURITY_LABEL=y
CONFIG_GRACE_PERIOD=m
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_SWAP=y
CONFIG_RPCSEC_GSS_KRB5=m
# CONFIG_SUNRPC_DISABLE_INSECURE_ENCTYPES is not set
# CONFIG_SUNRPC_DEBUG is not set
CONFIG_SUNRPC_XPRT_RDMA=m
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
CONFIG_CIFS_STATS2=y
CONFIG_CIFS_ALLOW_INSECURE_LEGACY=y
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG is not set
CONFIG_CIFS_DFS_UPCALL=y
# CONFIG_CIFS_SMB_DIRECT is not set
# CONFIG_CIFS_FSCACHE is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=m
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_MAC_ROMAN is not set
# CONFIG_NLS_MAC_CELTIC is not set
# CONFIG_NLS_MAC_CENTEURO is not set
# CONFIG_NLS_MAC_CROATIAN is not set
# CONFIG_NLS_MAC_CYRILLIC is not set
# CONFIG_NLS_MAC_GAELIC is not set
# CONFIG_NLS_MAC_GREEK is not set
# CONFIG_NLS_MAC_ICELAND is not set
# CONFIG_NLS_MAC_INUIT is not set
# CONFIG_NLS_MAC_ROMANIAN is not set
# CONFIG_NLS_MAC_TURKISH is not set
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
# CONFIG_DLM_DEBUG is not set
CONFIG_UNICODE=y
# CONFIG_UNICODE_NORMALIZATION_SELFTEST is not set
CONFIG_IO_WQ=y
# end of File systems

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_COMPAT=y
# CONFIG_KEYS_REQUEST_CACHE is not set
CONFIG_PERSISTENT_KEYRINGS=y
CONFIG_BIG_KEYS=y
# CONFIG_TRUSTED_KEYS is not set
CONFIG_ENCRYPTED_KEYS=m
# CONFIG_KEY_DH_OPERATIONS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_WRITABLE_HOOKS=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
# CONFIG_SECURITY_INFINIBAND is not set
# CONFIG_SECURITY_NETWORK_XFRM is not set
# CONFIG_SECURITY_PATH is not set
CONFIG_LSM_MMAP_MIN_ADDR=65536
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
# CONFIG_HARDENED_USERCOPY is not set
# CONFIG_FORTIFY_SOURCE is not set
# CONFIG_STATIC_USERMODEHELPER is not set
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=0
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_SECURITY_APPARMOR is not set
# CONFIG_SECURITY_LOADPIN is not set
# CONFIG_SECURITY_YAMA is not set
# CONFIG_SECURITY_SAFESETID is not set
CONFIG_SECURITY_LOCKDOWN_LSM=y
CONFIG_SECURITY_LOCKDOWN_LSM_EARLY=y
CONFIG_LOCK_DOWN_KERNEL_FORCE_NONE=y
# CONFIG_LOCK_DOWN_KERNEL_FORCE_INTEGRITY is not set
# CONFIG_LOCK_DOWN_KERNEL_FORCE_CONFIDENTIALITY is not set
CONFIG_INTEGRITY=y
CONFIG_INTEGRITY_SIGNATURE=y
CONFIG_INTEGRITY_ASYMMETRIC_KEYS=y
CONFIG_INTEGRITY_TRUSTED_KEYRING=y
CONFIG_INTEGRITY_AUDIT=y
CONFIG_IMA=y
CONFIG_IMA_MEASURE_PCR_IDX=10
CONFIG_IMA_LSM_RULES=y
# CONFIG_IMA_TEMPLATE is not set
CONFIG_IMA_NG_TEMPLATE=y
# CONFIG_IMA_SIG_TEMPLATE is not set
CONFIG_IMA_DEFAULT_TEMPLATE="ima-ng"
# CONFIG_IMA_DEFAULT_HASH_SHA1 is not set
CONFIG_IMA_DEFAULT_HASH_SHA256=y
# CONFIG_IMA_DEFAULT_HASH_SHA512 is not set
CONFIG_IMA_DEFAULT_HASH="sha256"
CONFIG_IMA_WRITE_POLICY=y
CONFIG_IMA_READ_POLICY=y
CONFIG_IMA_APPRAISE=y
# CONFIG_IMA_ARCH_POLICY is not set
# CONFIG_IMA_APPRAISE_BUILD_POLICY is not set
CONFIG_IMA_APPRAISE_BOOTPARAM=y
# CONFIG_IMA_APPRAISE_MODSIG is not set
CONFIG_IMA_TRUSTED_KEYRING=y
# CONFIG_IMA_BLACKLIST_KEYRING is not set
# CONFIG_IMA_LOAD_X509 is not set
# CONFIG_EVM is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_LSM="yama,loadpin,safesetid,integrity,selinux,smack,tomoyo,apparmor"

#
# Kernel hardening options
#

#
# Memory initialization
#
CONFIG_INIT_STACK_NONE=y
# CONFIG_GCC_PLUGIN_STRUCTLEAK_USER is not set
# CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF is not set
# CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL is not set
# CONFIG_INIT_ON_ALLOC_DEFAULT_ON is not set
# CONFIG_INIT_ON_FREE_DEFAULT_ON is not set
# end of Memory initialization
# end of Kernel hardening options
# end of Security options

CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_SKCIPHER=y
CONFIG_CRYPTO_SKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=y
CONFIG_CRYPTO_AKCIPHER2=y
CONFIG_CRYPTO_AKCIPHER=y
CONFIG_CRYPTO_KPP2=y
CONFIG_CRYPTO_KPP=m
CONFIG_CRYPTO_ACOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
# CONFIG_CRYPTO_MANAGER_DISABLE_TESTS is not set
# CONFIG_CRYPTO_MANAGER_EXTRA_TESTS is not set
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=y
CONFIG_CRYPTO_NULL2=y
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_ENGINE=m

#
# Public-key cryptography
#
CONFIG_CRYPTO_RSA=y
CONFIG_CRYPTO_DH=m
CONFIG_CRYPTO_ECC=m
CONFIG_CRYPTO_ECDH=m
CONFIG_CRYPTO_ECRDSA=m
# CONFIG_CRYPTO_CURVE25519 is not set

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=y
CONFIG_CRYPTO_CHACHA20POLY1305=m
CONFIG_CRYPTO_AEGIS128=m
CONFIG_CRYPTO_SEQIV=y
CONFIG_CRYPTO_ECHAINIV=m

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CFB=m
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_OFB=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=y
CONFIG_CRYPTO_KEYWRAP=m
CONFIG_CRYPTO_NHPOLY1305=m
CONFIG_CRYPTO_ADIANTUM=m
CONFIG_CRYPTO_ESSIV=m

#
# Hash modes
#
CONFIG_CRYPTO_CMAC=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32=m
CONFIG_CRYPTO_XXHASH=y
# CONFIG_CRYPTO_BLAKE2B is not set
# CONFIG_CRYPTO_BLAKE2S is not set
CONFIG_CRYPTO_CRCT10DIF=y
CONFIG_CRYPTO_GHASH=y
CONFIG_CRYPTO_POLY1305=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_SHA3=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_STREEBOG=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_TI=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_CHACHA20=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SM4=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_842=m
CONFIG_CRYPTO_LZ4=m
CONFIG_CRYPTO_LZ4HC=m
CONFIG_CRYPTO_ZSTD=m

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_DRBG_MENU=y
CONFIG_CRYPTO_DRBG_HMAC=y
# CONFIG_CRYPTO_DRBG_HASH is not set
# CONFIG_CRYPTO_DRBG_CTR is not set
CONFIG_CRYPTO_DRBG=y
CONFIG_CRYPTO_JITTERENTROPY=y
CONFIG_CRYPTO_USER_API=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_STATS=y
CONFIG_CRYPTO_HASH_INFO=y

#
# Crypto library routines
#
CONFIG_CRYPTO_LIB_AES=y
CONFIG_CRYPTO_LIB_ARC4=m
# CONFIG_CRYPTO_LIB_BLAKE2S is not set
CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
# CONFIG_CRYPTO_LIB_CHACHA is not set
# CONFIG_CRYPTO_LIB_CURVE25519 is not set
CONFIG_CRYPTO_LIB_DES=m
CONFIG_CRYPTO_LIB_POLY1305_RSIZE=1
CONFIG_CRYPTO_LIB_POLY1305_GENERIC=m
# CONFIG_CRYPTO_LIB_POLY1305 is not set
# CONFIG_CRYPTO_LIB_CHACHA20POLY1305 is not set
CONFIG_CRYPTO_LIB_SHA256=y
CONFIG_CRYPTO_HW=y
CONFIG_ZCRYPT=m
CONFIG_ZCRYPT_MULTIDEVNODES=y
CONFIG_PKEY=m
CONFIG_CRYPTO_PAES_S390=m
CONFIG_CRYPTO_SHA1_S390=m
CONFIG_CRYPTO_SHA256_S390=m
CONFIG_CRYPTO_SHA512_S390=m
CONFIG_CRYPTO_SHA3_256_S390=m
CONFIG_CRYPTO_SHA3_512_S390=m
CONFIG_CRYPTO_DES_S390=m
CONFIG_CRYPTO_AES_S390=m
CONFIG_S390_PRNG=m
CONFIG_CRYPTO_GHASH_S390=m
CONFIG_CRYPTO_CRC32_S390=y
# CONFIG_CRYPTO_DEV_NITROX_CNN55XX is not set
CONFIG_CRYPTO_DEV_VIRTIO=m
# CONFIG_CRYPTO_DEV_SAFEXCEL is not set
# CONFIG_CRYPTO_DEV_AMLOGIC_GXL is not set
CONFIG_ASYMMETRIC_KEY_TYPE=y
CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
CONFIG_X509_CERTIFICATE_PARSER=y
# CONFIG_PKCS8_PRIVATE_KEY_PARSER is not set
CONFIG_PKCS7_MESSAGE_PARSER=y
# CONFIG_PKCS7_TEST_KEY is not set
# CONFIG_SIGNED_PE_FILE_VERIFICATION is not set

#
# Certificates for signature checking
#
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS=""
# CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set
# CONFIG_SECONDARY_TRUSTED_KEYRING is not set
# CONFIG_SYSTEM_BLACKLIST_KEYRING is not set
# end of Certificates for signature checking

CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=y
CONFIG_RAID6_PQ_BENCHMARK=y
# CONFIG_PACKING is not set
CONFIG_BITREVERSE=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_CORDIC=m
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
CONFIG_CRC64=m
CONFIG_CRC4=m
CONFIG_CRC7=m
CONFIG_LIBCRC32C=y
CONFIG_CRC8=m
CONFIG_XXHASH=y
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_842_COMPRESS=m
CONFIG_842_DECOMPRESS=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_COMPRESS=m
CONFIG_LZ4HC_COMPRESS=m
CONFIG_LZ4_DECOMPRESS=y
CONFIG_ZSTD_COMPRESS=y
CONFIG_ZSTD_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_DECOMPRESS_LZ4=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_INTERVAL_TREE=y
CONFIG_XARRAY_MULTI=y
CONFIG_ASSOCIATIVE_ARRAY=y
CONFIG_HAS_DMA=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_ARCH_HAS_FORCE_DMA_UNENCRYPTED=y
CONFIG_SWIOTLB=y
CONFIG_DMA_CMA=y

#
# Default contiguous memory area size:
#
CONFIG_CMA_SIZE_MBYTES=0
CONFIG_CMA_SIZE_SEL_MBYTES=y
# CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set
# CONFIG_CMA_SIZE_SEL_MIN is not set
# CONFIG_CMA_SIZE_SEL_MAX is not set
CONFIG_CMA_ALIGNMENT=8
# CONFIG_DMA_API_DEBUG is not set
CONFIG_SGL_ALLOC=y
CONFIG_IOMMU_HELPER=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_GLOB=y
# CONFIG_GLOB_SELFTEST is not set
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_CLZ_TAB=y
CONFIG_IRQ_POLL=y
CONFIG_MPILIB=y
CONFIG_SIGNATURE=y
CONFIG_DIMLIB=y
CONFIG_OID_REGISTRY=y
CONFIG_FONT_SUPPORT=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_SG_POOL=y
CONFIG_ARCH_STACKWALK=y
CONFIG_SBITMAP=y
# CONFIG_STRING_SELFTEST is not set
# end of Library routines

#
# Kernel hacking
#

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
# CONFIG_PRINTK_CALLER is not set
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
CONFIG_CONSOLE_LOGLEVEL_QUIET=4
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
# CONFIG_DYNAMIC_DEBUG is not set
CONFIG_SYMBOLIC_ERRNAME=y
CONFIG_DEBUG_BUGVERBOSE=y
# end of printk and dmesg options

#
# Compile-time checks and compiler options
#
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
# CONFIG_DEBUG_INFO_BTF is not set
CONFIG_GDB_SCRIPTS=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=1024
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_READABLE_ASM is not set
# CONFIG_HEADERS_INSTALL is not set
CONFIG_OPTIMIZE_INLINING=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# end of Compile-time checks and compiler options

#
# Generic Kernel Debugging Instruments
#
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_MAGIC_SYSRQ_SERIAL=y
CONFIG_DEBUG_FS=y
CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
# CONFIG_UBSAN is not set
CONFIG_UBSAN_ALIGNMENT=y
# end of Generic Kernel Debugging Instruments

CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_MISC=y

#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_OWNER is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_PAGE_REF is not set
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_SCHED_STACK_END_CHECK is not set
# CONFIG_DEBUG_VM is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_CC_HAS_KASAN_GENERIC=y
# CONFIG_KASAN is not set
CONFIG_KASAN_STACK=1
# end of Memory Debugging

# CONFIG_DEBUG_SHIRQ is not set

#
# Debug Oops, Lockups and Hangs
#
CONFIG_PANIC_ON_OOPS=y
CONFIG_PANIC_ON_OOPS_VALUE=1
CONFIG_PANIC_TIMEOUT=0
# CONFIG_DETECT_HUNG_TASK is not set
# CONFIG_WQ_WATCHDOG is not set
# end of Debug Oops, Lockups and Hangs

#
# Scheduler Debugging
#
CONFIG_SCHED_DEBUG=y
CONFIG_SCHED_INFO=y
CONFIG_SCHEDSTATS=y
# end of Scheduler Debugging

# CONFIG_DEBUG_TIMEKEEPING is not set

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_LOCK_DEBUGGING_SUPPORT=y
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
# CONFIG_DEBUG_RWSEMS is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_WW_MUTEX_SELFTEST is not set
# end of Lock Debugging (spinlocks, mutexes, etc...)

CONFIG_STACKTRACE=y
# CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set
# CONFIG_DEBUG_KOBJECT is not set

#
# Debug kernel data structures
#
CONFIG_DEBUG_LIST=y
# CONFIG_DEBUG_PLIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
CONFIG_BUG_ON_DATA_CORRUPTION=y
# end of Debug kernel data structures

# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
CONFIG_TORTURE_TEST=m
# CONFIG_RCU_PERF_TEST is not set
CONFIG_RCU_TORTURE_TEST=m
CONFIG_RCU_CPU_STALL_TIMEOUT=60
CONFIG_RCU_TRACE=y
# CONFIG_RCU_EQS_DEBUG is not set
# end of RCU Debugging

# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_CPU_HOTPLUG_STATE_CONTROL is not set
CONFIG_LATENCYTOP=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_NOP_MCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_PREEMPTIRQ_EVENTS is not set
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
# CONFIG_HWLAT_TRACER is not set
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENTS=y
CONFIG_UPROBE_EVENTS=y
CONFIG_BPF_EVENTS=y
CONFIG_DYNAMIC_EVENTS=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
CONFIG_TRACING_MAP=y
CONFIG_HIST_TRIGGERS=y
# CONFIG_TRACEPOINT_BENCHMARK is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_RING_BUFFER_STARTUP_TEST is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
# CONFIG_KUNIT is not set
# CONFIG_SAMPLES is not set
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
# CONFIG_STRICT_DEVMEM is not set
# CONFIG_DEBUG_AID_FOR_SYZBOT is not set

#
# s390 Debugging
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_S390_PTDUMP=y
CONFIG_EARLY_PRINTK=y
# end of s390 Debugging

#
# Kernel Testing and Coverage
#
CONFIG_ARCH_HAS_KCOV=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_KCOV is not set
CONFIG_RUNTIME_TESTING_MENU=y
CONFIG_LKDTM=m
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_TEST_SORT is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_RBTREE_TEST is not set
# CONFIG_REED_SOLOMON_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
CONFIG_PERCPU_TEST=m
CONFIG_ATOMIC64_SELFTEST=y
# CONFIG_ASYNC_RAID6_TEST is not set
# CONFIG_TEST_HEXDUMP is not set
# CONFIG_TEST_STRING_HELPERS is not set
# CONFIG_TEST_STRSCPY is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_TEST_PRINTF is not set
# CONFIG_TEST_BITMAP is not set
# CONFIG_TEST_BITFIELD is not set
# CONFIG_TEST_UUID is not set
# CONFIG_TEST_XARRAY is not set
# CONFIG_TEST_OVERFLOW is not set
# CONFIG_TEST_RHASHTABLE is not set
# CONFIG_TEST_HASH is not set
# CONFIG_TEST_IDA is not set
# CONFIG_TEST_LKM is not set
# CONFIG_TEST_VMALLOC is not set
# CONFIG_TEST_USER_COPY is not set
CONFIG_TEST_BPF=m
# CONFIG_TEST_BLACKHOLE_DEV is not set
# CONFIG_FIND_BIT_BENCHMARK is not set
# CONFIG_TEST_FIRMWARE is not set
# CONFIG_TEST_SYSCTL is not set
# CONFIG_TEST_UDELAY is not set
# CONFIG_TEST_STATIC_KEYS is not set
# CONFIG_TEST_KMOD is not set
# CONFIG_TEST_MEMCAT_P is not set
# CONFIG_TEST_STACKINIT is not set
# CONFIG_TEST_MEMINIT is not set
# CONFIG_MEMTEST is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
# end of Kernel Testing and Coverage
# end of Kernel hacking

[-- Attachment #3: dmesg.txt --]
[-- Type: text/plain, Size: 56836 bytes --]

[    0.153849] Linux version 5.4.0-rc8-next-20191122 (cborntra@s38lp08.lnxne.boe) (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #70 SMP Fri Nov 22 08:59:29 CET 2019
[    0.153851] setup: Linux is running natively in 64-bit mode
[    0.153891] setup: The maximum memory size is 131072MB
[    0.153897] setup: Reserving 1024MB of memory at 130048MB for crashkernel (System RAM: 130048MB)
[    0.153909] numa: NUMA mode: plain
[    0.153975] cpu: 32 configured CPUs, 0 standby CPUs
[    0.154005] cpu: The CPU configuration topology of the machine is: 0 0 4 2 3 8 / 4
[    0.154735] Write protected kernel read-only data: 13316k
[    0.156243] Zone ranges:
[    0.156244]   DMA      [mem 0x0000000000000000-0x000000007fffffff]
[    0.156246]   Normal   [mem 0x0000000080000000-0x0000001fffffffff]
[    0.156248] Movable zone start for each node
[    0.156248] Early memory node ranges
[    0.156250]   node   0: [mem 0x0000000000000000-0x0000001fffffffff]
[    0.156257] Initmem setup node 0 [mem 0x0000000000000000-0x0000001fffffffff]
[    0.156258] On node 0 totalpages: 33554432
[    0.156259]   DMA zone: 8192 pages used for memmap
[    0.156260]   DMA zone: 0 pages reserved
[    0.156260]   DMA zone: 524288 pages, LIFO batch:63
[    0.172415]   Normal zone: 516096 pages used for memmap
[    0.172415]   Normal zone: 33030144 pages, LIFO batch:63
[    0.188681] percpu: Embedded 33 pages/cpu s97024 r8192 d29952 u135168
[    0.188689] pcpu-alloc: s97024 r8192 d29952 u135168 alloc=33*4096
[    0.188690] pcpu-alloc: [0] 000 [0] 001 [0] 002 [0] 003 
[    0.188691] pcpu-alloc: [0] 004 [0] 005 [0] 006 [0] 007 
[    0.188693] pcpu-alloc: [0] 008 [0] 009 [0] 010 [0] 011 
[    0.188694] pcpu-alloc: [0] 012 [0] 013 [0] 014 [0] 015 
[    0.188695] pcpu-alloc: [0] 016 [0] 017 [0] 018 [0] 019 
[    0.188696] pcpu-alloc: [0] 020 [0] 021 [0] 022 [0] 023 
[    0.188698] pcpu-alloc: [0] 024 [0] 025 [0] 026 [0] 027 
[    0.188699] pcpu-alloc: [0] 028 [0] 029 [0] 030 [0] 031 
[    0.188700] pcpu-alloc: [0] 032 [0] 033 [0] 034 [0] 035 
[    0.188701] pcpu-alloc: [0] 036 [0] 037 [0] 038 [0] 039 
[    0.188703] pcpu-alloc: [0] 040 [0] 041 [0] 042 [0] 043 
[    0.188704] pcpu-alloc: [0] 044 [0] 045 [0] 046 [0] 047 
[    0.188705] pcpu-alloc: [0] 048 [0] 049 [0] 050 [0] 051 
[    0.188706] pcpu-alloc: [0] 052 [0] 053 [0] 054 [0] 055 
[    0.188707] pcpu-alloc: [0] 056 [0] 057 [0] 058 [0] 059 
[    0.188709] pcpu-alloc: [0] 060 [0] 061 [0] 062 [0] 063 
[    0.188710] pcpu-alloc: [0] 064 [0] 065 [0] 066 [0] 067 
[    0.188711] pcpu-alloc: [0] 068 [0] 069 [0] 070 [0] 071 
[    0.188712] pcpu-alloc: [0] 072 [0] 073 [0] 074 [0] 075 
[    0.188714] pcpu-alloc: [0] 076 [0] 077 [0] 078 [0] 079 
[    0.188715] pcpu-alloc: [0] 080 [0] 081 [0] 082 [0] 083 
[    0.188716] pcpu-alloc: [0] 084 [0] 085 [0] 086 [0] 087 
[    0.188717] pcpu-alloc: [0] 088 [0] 089 [0] 090 [0] 091 
[    0.188718] pcpu-alloc: [0] 092 [0] 093 [0] 094 [0] 095 
[    0.188720] pcpu-alloc: [0] 096 [0] 097 [0] 098 [0] 099 
[    0.188721] pcpu-alloc: [0] 100 [0] 101 [0] 102 [0] 103 
[    0.188722] pcpu-alloc: [0] 104 [0] 105 [0] 106 [0] 107 
[    0.188723] pcpu-alloc: [0] 108 [0] 109 [0] 110 [0] 111 
[    0.188725] pcpu-alloc: [0] 112 [0] 113 [0] 114 [0] 115 
[    0.188726] pcpu-alloc: [0] 116 [0] 117 [0] 118 [0] 119 
[    0.188727] pcpu-alloc: [0] 120 [0] 121 [0] 122 [0] 123 
[    0.188728] pcpu-alloc: [0] 124 [0] 125 [0] 126 [0] 127 
[    0.188729] pcpu-alloc: [0] 128 [0] 129 [0] 130 [0] 131 
[    0.188731] pcpu-alloc: [0] 132 [0] 133 [0] 134 [0] 135 
[    0.188732] pcpu-alloc: [0] 136 [0] 137 [0] 138 [0] 139 
[    0.188733] pcpu-alloc: [0] 140 [0] 141 [0] 142 [0] 143 
[    0.188734] pcpu-alloc: [0] 144 [0] 145 [0] 146 [0] 147 
[    0.188735] pcpu-alloc: [0] 148 [0] 149 [0] 150 [0] 151 
[    0.188737] pcpu-alloc: [0] 152 [0] 153 [0] 154 [0] 155 
[    0.188738] pcpu-alloc: [0] 156 [0] 157 [0] 158 [0] 159 
[    0.188739] pcpu-alloc: [0] 160 [0] 161 [0] 162 [0] 163 
[    0.188740] pcpu-alloc: [0] 164 [0] 165 [0] 166 [0] 167 
[    0.188742] pcpu-alloc: [0] 168 [0] 169 [0] 170 [0] 171 
[    0.188743] pcpu-alloc: [0] 172 [0] 173 [0] 174 [0] 175 
[    0.188744] pcpu-alloc: [0] 176 [0] 177 [0] 178 [0] 179 
[    0.188745] pcpu-alloc: [0] 180 [0] 181 [0] 182 [0] 183 
[    0.188746] pcpu-alloc: [0] 184 [0] 185 [0] 186 [0] 187 
[    0.188748] pcpu-alloc: [0] 188 [0] 189 [0] 190 [0] 191 
[    0.188749] pcpu-alloc: [0] 192 [0] 193 [0] 194 [0] 195 
[    0.188750] pcpu-alloc: [0] 196 [0] 197 [0] 198 [0] 199 
[    0.188751] pcpu-alloc: [0] 200 [0] 201 [0] 202 [0] 203 
[    0.188752] pcpu-alloc: [0] 204 [0] 205 [0] 206 [0] 207 
[    0.188754] pcpu-alloc: [0] 208 [0] 209 [0] 210 [0] 211 
[    0.188755] pcpu-alloc: [0] 212 [0] 213 [0] 214 [0] 215 
[    0.188756] pcpu-alloc: [0] 216 [0] 217 [0] 218 [0] 219 
[    0.188757] pcpu-alloc: [0] 220 [0] 221 [0] 222 [0] 223 
[    0.188759] pcpu-alloc: [0] 224 [0] 225 [0] 226 [0] 227 
[    0.188760] pcpu-alloc: [0] 228 [0] 229 [0] 230 [0] 231 
[    0.188761] pcpu-alloc: [0] 232 [0] 233 [0] 234 [0] 235 
[    0.188762] pcpu-alloc: [0] 236 [0] 237 [0] 238 [0] 239 
[    0.188763] pcpu-alloc: [0] 240 [0] 241 [0] 242 [0] 243 
[    0.188765] pcpu-alloc: [0] 244 [0] 245 [0] 246 [0] 247 
[    0.188766] pcpu-alloc: [0] 248 [0] 249 [0] 250 [0] 251 
[    0.188767] pcpu-alloc: [0] 252 [0] 253 [0] 254 [0] 255 
[    0.188768] pcpu-alloc: [0] 256 [0] 257 [0] 258 [0] 259 
[    0.188769] pcpu-alloc: [0] 260 [0] 261 [0] 262 [0] 263 
[    0.188771] pcpu-alloc: [0] 264 [0] 265 [0] 266 [0] 267 
[    0.188772] pcpu-alloc: [0] 268 [0] 269 [0] 270 [0] 271 
[    0.188773] pcpu-alloc: [0] 272 [0] 273 [0] 274 [0] 275 
[    0.188774] pcpu-alloc: [0] 276 [0] 277 [0] 278 [0] 279 
[    0.188776] pcpu-alloc: [0] 280 [0] 281 
[    0.188796] Built 1 zonelists, mobility grouping on.  Total pages: 33030144
[    0.188797] Policy zone: Normal
[    0.188798] Kernel command line: root=/dev/disk/by-path/ccw-0.0.241c-part1 rd.dasd=0.0.241c rd.dasd=0.0.241d rd.dasd=0.0.241e cio_ignore=all,!condev rd.znet=qeth,0.0.e400,0.0.e401,0.0.e402,layer2=1,portno=0,portname=OSAPORT zfcp.allow_lun_scan=1 crashkernel=1G spectre_v2=off BOOT_IMAGE=0
[    0.189869] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.189869] printk: log_buf_len total cpu_extra contributions: 1150976 bytes
[    0.189870] printk: log_buf_len min size: 131072 bytes
[    0.190164] printk: log_buf_len: 2097152 bytes
[    0.190165] printk: early log buf free: 124672(95%)
[    0.199225] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
[    0.203790] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.203792] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.240221] Memory: 2314512K/134217728K available (10264K kernel code, 2020K rwdata, 3052K rodata, 3888K init, 856K bss, 3348492K reserved, 0K cma-reserved)
[    0.240611] SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=282, Nodes=1
[    0.240642] ftrace: allocating 31258 entries in 123 pages
[    0.245932] ftrace: allocated 123 pages with 6 groups
[    0.246508] rcu: Hierarchical RCU implementation.
[    0.246509] rcu: 	RCU event tracing is enabled.
[    0.246510] rcu: 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=282.
[    0.246510] 	Tasks RCU enabled.
[    0.246511] rcu: RCU calculated value of scheduler-enlistment delay is 11 jiffies.
[    0.246512] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=282
[    0.249499] NR_IRQS: 3, nr_irqs: 3, preallocated irqs: 3
[    0.249645] clocksource: tod: mask: 0xffffffffffffffff max_cycles: 0x3b0a9be803b0a9, max_idle_ns: 1805497147909793 ns
[    0.249844] Console: colour dummy device 80x25
[    0.371333] printk: console [ttyS0] enabled
[    0.526531] Calibrating delay loop (skipped)... 20325.00 BogoMIPS preset
[    0.526532] pid_max: default: 288768 minimum: 2256
[    0.526656] LSM: Security Framework initializing
[    0.526675] SELinux:  Initializing.
[    0.526876] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.526992] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.527686] rcu: Hierarchical SRCU implementation.
[    0.529583] smp: Bringing up secondary CPUs ...
[    0.535016] smp: Brought up 1 node, 32 CPUs
[    1.464986] node 0 initialised, 32138681 pages in 930ms
[    1.490567] devtmpfs: initialized
[    1.492229] random: get_random_u32 called from bucket_table_alloc.isra.0+0x82/0x120 with crng_init=0
[    1.492699] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    1.493390] futex hash table entries: 131072 (order: 13, 33554432 bytes, vmalloc)
[    1.498838] xor: automatically using best checksumming function   xc        
[    1.498988] NET: Registered protocol family 16
[    1.499033] audit: initializing netlink subsys (disabled)
[    1.499093] audit: type=2000 audit(1574427501.087:1): state=initialized audit_enabled=0 res=1
[    1.502384] HugeTLB registered 1.00 MiB page size, pre-allocated 0 pages
[    1.545283] random: fast init done
[    1.749690] raid6: vx128x8  gen() 20468 MB/s
[    1.919658] raid6: vx128x8  xor() 11713 MB/s
[    1.919660] raid6: using algorithm vx128x8 gen() 20468 MB/s
[    1.919661] raid6: .... xor() 11713 MB/s, rmw enabled
[    1.919662] raid6: using s390xc recovery algorithm
[    1.919882] iommu: Default domain type: Translated 
[    1.920037] SCSI subsystem initialized
[    1.926005] PCI host bridge to bus 0000:00
[    1.926010] pci_bus 0000:00: root bus resource [mem 0x8000000000000000-0x80000000007fffff 64bit pref]
[    1.926013] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[    1.926209] pci 0000:00:00.0: [15b3:1004] type 00 class 0x020000
[    1.927382] pci 0000:00:00.0: reg 0x18: [mem 0xffffcc0001000000-0xffffcc00017fffff 64bit pref]
[    1.930105] pci 0000:00:00.0: Adding to iommu group 0
[    1.930114] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[    2.600903] VFS: Disk quotas dquot_6.6.0
[    2.600958] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    2.602623] NET: Registered protocol family 2
[    2.603156] tcp_listen_portaddr_hash hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    2.603731] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc)
[    2.605808] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    2.606321] TCP: Hash tables configured (established 524288 bind 65536)
[    2.606600] UDP hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[    2.607624] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[    2.608948] NET: Registered protocol family 1
[    2.609143] Trying to unpack rootfs image as initramfs...
[    3.270624] Freeing initrd memory: 44180K
[    3.271994] alg: No test for crc32be (crc32be-vx)
[    3.275961] Initialise system trusted keyrings
[    3.276005] workingset: timestamp_bits=42 max_order=25 bucket_order=0
[    3.277201] fuse: init (API version 7.31)
[    3.277264] SGI XFS with ACLs, security attributes, realtime, quota, no debug enabled
[    3.283706] Key type asymmetric registered
[    3.283707] Asymmetric key parser 'x509' registered
[    3.283714] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    3.283850] io scheduler mq-deadline registered
[    3.283851] io scheduler kyber registered
[    3.283876] io scheduler bfq registered
[    3.284217] atomic64_test: passed
[    3.284329] hvc_iucv: The z/VM IUCV HVC device driver cannot be used without z/VM
[    3.289528] brd: module loaded
[    3.289819] cio: Channel measurement facility initialized using format extended (mode autodetected)
[    3.290048] Discipline DIAG cannot be used without z/VM
[    4.646249] sclp_sd: No data is available for the config data entity
[    4.813487] qeth: loading core functions
[    4.813534] qeth: register layer 2 discipline
[    4.813536] qeth: register layer 3 discipline
[    4.813875] NET: Registered protocol family 10
[    4.814674] Segment Routing with IPv6
[    4.814692] NET: Registered protocol family 17
[    4.814704] Key type dns_resolver registered
[    4.814786] registered taskstats version 1
[    4.814790] Loading compiled-in X.509 certificates
[    4.814911] alg: No test for pkcs1pad(rsa,sha1) (pkcs1pad(rsa-generic,sha1))
[    4.816834] Loaded X.509 cert 'Build time autogenerated kernel key: f59e0411a915bdd2e8aa85c3d95a9bbf8eeef38d'
[    4.817510] zswap: default zpool zbud not available
[    4.817512] zswap: pool creation failed
[    4.817805] Key type ._fscrypt registered
[    4.817806] Key type .fscrypt registered
[    4.818116] Btrfs loaded, crc32c=crc32c-vx
[    4.823104] Key type big_key registered
[    4.823111] ima: No TPM chip found, activating TPM-bypass!
[    4.823115] ima: Allocated hash algorithm: sha256
[    4.823122] ima: No architecture policies found
[    4.824160] Freeing unused kernel memory: 3888K
[    4.889756] Write protected read-only-after-init data: 68k
[    4.889759] Run /init as init process
[    4.947324] systemd[1]: Inserted module 'autofs4'
[    4.949140] systemd[1]: systemd v241-12.git323cdf4.fc30 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[    4.949615] systemd[1]: Detected architecture s390x.
[    4.949617] systemd[1]: Running in initial RAM disk.
[    5.024189] systemd[1]: Set hostname to <s38lp08.lnxne.boe>.
[    5.025147] systemd[1]: Couldn't move remaining userspace processes, ignoring: Input/output error
[    5.058854] random: systemd: uninitialized urandom read (16 bytes read)
[    5.058895] systemd[1]: Listening on Journal Audit Socket.
[    5.058947] random: systemd: uninitialized urandom read (16 bytes read)
[    5.058983] systemd[1]: Listening on Journal Socket (/dev/log).
[    5.059005] random: systemd: uninitialized urandom read (16 bytes read)
[    5.059029] systemd[1]: Listening on udev Kernel Socket.
[    5.059085] systemd[1]: Listening on Journal Socket.
[    5.062310] systemd[1]: Starting Load Kernel Modules...
[    5.300072] audit: type=1130 audit(1574427504.897:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    5.509734] random: crng init done
[    5.509737] random: 7 urandom warning(s) missed due to ratelimiting
[    7.273226] audit: type=1130 audit(1574427506.867:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.316074] device-mapper: uevent: version 1.0.3
[    7.316158] device-mapper: ioctl: 4.41.0-ioctl (2019-09-16) initialised: dm-devel@redhat.com
[    7.340398] alua: device handler registered
[    7.345145] emc: device handler registered
[    7.350191] rdac: device handler registered
[    7.364131] audit: type=1130 audit(1574427506.957:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-udev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.364719] audit: type=1334 audit(1574427506.957:5): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=1 comm="systemd" exe="/usr/lib/systemd/systemd" prog-id=5 event=LOAD
[    7.364776] audit: type=1334 audit(1574427506.957:6): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=1 comm="systemd" exe="/usr/lib/systemd/systemd" prog-id=6 event=LOAD
[    7.592918] audit: type=1130 audit(1574427507.187:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.597370] qeth 0.0.e400: Priority Queueing not supported
[    7.598076] qeth 0.0.e400: portname is deprecated and is ignored
[    7.600120] dasd-eckd 0.0.241c: A channel path to the device has become operational
[    7.601851] dasd-eckd 0.0.241e: A channel path to the device has become operational
[    7.602230] dasd-eckd 0.0.241d: A channel path to the device has become operational
[    7.603067] dasd-eckd 0.0.241c: New DASD 3390/0C (CU 3990/01) with 30051 cylinders, 15 heads, 224 sectors
[    7.603291] scsi host0: zfcp
[    7.604226] dasd-eckd 0.0.241e: New DASD 3390/0C (CU 3990/01) with 30051 cylinders, 15 heads, 224 sectors
[    7.604557] dasd-eckd 0.0.241d: New DASD 3390/0C (CU 3990/01) with 30051 cylinders, 15 heads, 224 sectors
[    7.604860] qdio: 0.0.e402 OSA on SC 3e6 using AI:1 QEBSM:0 PRI:1 TDD:1 SIGA: W AP
[    7.605810] qdio: 0.0.1900 ZFCP on SC 147 using AI:1 QEBSM:0 PRI:1 TDD:1 SIGA: W AP
[    7.606718] dasd-eckd 0.0.241c: DASD with 4 KB/block, 21636720 KB total size, 48 KB/track, compatible disk layout
[    7.607984]  dasda:VOL1/  0X241C: dasda1
[    7.608530] dasd-eckd 0.0.241e: DASD with 4 KB/block, 21636720 KB total size, 48 KB/track, compatible disk layout
[    7.608760] dasd-eckd 0.0.241d: DASD with 4 KB/block, 21636720 KB total size, 48 KB/track, compatible disk layout
[    7.609385] scsi host1: zfcp
[    7.609974]  dasdb:VOL1/  0X3333: dasdb1 dasdb2
[    7.610043]  dasdc:VOL1/  0X241D: dasdc1
[    7.611359] qdio: 0.0.1940 ZFCP on SC 148 using AI:1 QEBSM:0 PRI:1 TDD:1 SIGA: W AP
[    7.633794] qeth 0.0.e400: QDIO data connection isolation is deactivated
[    7.634345] qeth 0.0.e400: The device represents a Bridge Capable Port
[    7.635362] qeth 0.0.e400: MAC address 56:2b:61:4f:36:50 successfully registered
[    7.635964] qeth 0.0.e400: Device is a OSD Express card (level: 0774)
               with link type OSD_10GIG.
[    7.639822] audit: type=1130 audit(1574427507.237:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.646577] qeth 0.0.e400: MAC address 8a:ec:e6:ac:7f:ac successfully registered
[    7.647104] qeth 0.0.e400 ence400: renamed from eth0
[    7.655531] audit: type=1130 audit(1574427507.247:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=plymouth-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    7.723280] mlx4_core: Mellanox ConnectX core driver v4.0-0
[    7.723295] mlx4_core: Initializing 0000:00:00.0
[    7.723376] mlx4_core 0000:00:00.0: enabling device (0000 -> 0002)
[    7.724446] mlx4_core 0000:00:00.0: Detected virtual function - running in slave mode
[    7.724484] mlx4_core 0000:00:00.0: Sending reset
[    7.725868] mlx4_core 0000:00:00.0: Sending vhcr0
[    7.731304] mlx4_core 0000:00:00.0: Requested number of MACs is too much for port 1, reducing to 64
[    7.731305] mlx4_core 0000:00:00.0: Requested number of VLANs is too much for port 1, reducing to 1
[    7.731498] mlx4_core 0000:00:00.0: HCA minimum page size:512
[    7.732931] mlx4_core 0000:00:00.0: Timestamping is not supported in slave mode
[    7.827951] dasdconf.sh Warning: 0.0.241d is already online, not configuring
[    7.835698] dasdconf.sh Warning: 0.0.241e is already online, not configuring
[    7.947875] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[    7.948376] mlx4_en 0000:00:00.0: Activating port:1
[    7.959194] dasdconf.sh Warning: 0.0.241c is already online, not configuring
[    7.988608] mlx4_en: 0000:00:00.0: Port 1: Using 32 TX rings
[    7.988611] mlx4_en: 0000:00:00.0: Port 1: Using 8 RX rings
[    7.989106] mlx4_en: 0000:00:00.0: Port 1: Initializing port
[    7.989765] mlx4_en 0000:00:00.0: Activating port:2
[    8.025371] mlx4_en: 0000:00:00.0: Port 2: Using 32 TX rings
[    8.025373] mlx4_en: 0000:00:00.0: Port 2: Using 8 RX rings
[    8.025846] mlx4_en: 0000:00:00.0: Port 2: Initializing port
[    8.030272] mlx4_core 0000:00:00.0 ens12: renamed from eth0
[    8.119874] mlx4_core 0000:00:00.0 ens12d1: renamed from eth1
[    8.697897] scsi 0:0:0:1074675840: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.698587] scsi 0:0:0:1074675840: alua: supports implicit TPGS
[    8.698593] scsi 0:0:0:1074675840: alua: device naa.6005076305ffc1ae000000000000800e port group 0 rel port 101
[    8.698830] sd 0:0:0:1074675840: Attached scsi generic sg0 type 0
[    8.698876] sd 0:0:0:1074675840: Power-on or device reset occurred
[    8.699712] scsi 0:0:0:1074741376: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.700310] scsi 0:0:0:1074741376: alua: supports implicit TPGS
[    8.700313] scsi 0:0:0:1074741376: alua: device naa.6005076305ffc1ae000000000000800f port group 0 rel port 101
[    8.700444] sd 0:0:0:1074741376: Attached scsi generic sg1 type 0
[    8.700541] sd 0:0:0:1074741376: Power-on or device reset occurred
[    8.701469] scsi 0:0:0:1074675841: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.702096] scsi 0:0:0:1074675841: alua: supports implicit TPGS
[    8.702100] scsi 0:0:0:1074675841: alua: device naa.6005076305ffc1ae000000000000810e port group 0 rel port 101
[    8.702269] sd 0:0:0:1074675841: Attached scsi generic sg2 type 0
[    8.702362] sd 0:0:0:1074675841: Power-on or device reset occurred
[    8.703072] scsi 0:0:0:1074741377: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.703659] scsi 0:0:0:1074741377: alua: supports implicit TPGS
[    8.703662] scsi 0:0:0:1074741377: alua: device naa.6005076305ffc1ae000000000000810f port group 0 rel port 101
[    8.703759] sd 0:0:0:1074741377: Attached scsi generic sg3 type 0
[    8.703971] sd 0:0:0:1074741377: Power-on or device reset occurred
[    8.704983] scsi 0:0:1:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.705646] sd 0:0:1:0: Attached scsi generic sg4 type 0
[    8.705838] sd 0:0:1:0: Power-on or device reset occurred
[    8.706756] scsi 0:0:2:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.706758] sd 0:0:1:0: [sde] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.706760] sd 0:0:1:0: [sde] 4096-byte physical blocks
[    8.706925] sd 0:0:1:0: [sde] Write Protect is off
[    8.706930] sd 0:0:1:0: [sde] Mode Sense: a7 00 00 08
[    8.707149] sd 0:0:1:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.707352] sd 0:0:1:0: [sde] Optimal transfer size 65536 bytes
[    8.707424] sd 0:0:2:0: Attached scsi generic sg5 type 0
[    8.707471] sd 0:0:2:0: Power-on or device reset occurred
[    8.708286] sd 0:0:2:0: [sdf] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.708289] sd 0:0:2:0: [sdf] 4096-byte physical blocks
[    8.708423] scsi 0:0:3:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.708448] sd 0:0:2:0: [sdf] Write Protect is off
[    8.708451] sd 0:0:2:0: [sdf] Mode Sense: a7 00 00 08
[    8.708678] sd 0:0:2:0: [sdf] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.708963] sd 0:0:2:0: [sdf] Optimal transfer size 65536 bytes
[    8.709130] sd 0:0:3:0: Attached scsi generic sg6 type 0
[    8.709383] sd 0:0:3:0: Power-on or device reset occurred
[    8.709899] sd 0:0:0:1074675840: alua: transition timeout set to 60 seconds
[    8.709902] sd 0:0:0:1074675840: alua: port group 00 state A preferred supports tolusnA
[    8.710350] scsi 0:0:4:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.710354] sd 0:0:3:0: [sdg] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.710357] sd 0:0:3:0: [sdg] 4096-byte physical blocks
[    8.710533] sd 0:0:3:0: [sdg] Write Protect is off
[    8.710535] sd 0:0:3:0: [sdg] Mode Sense: a7 00 00 08
[    8.710753] sd 0:0:3:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.710823] sd 0:0:0:1074675840: [sda] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.710977] sd 0:0:0:1074675840: [sda] Write Protect is off
[    8.710981] sd 0:0:0:1074675840: [sda] Mode Sense: ed 00 00 08
[    8.711013] sd 0:0:4:0: Attached scsi generic sg7 type 0
[    8.711024] sd 0:0:3:0: [sdg] Optimal transfer size 65536 bytes
[    8.711116] sd 0:0:4:0: Power-on or device reset occurred
[    8.711308] sd 0:0:0:1074675840: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.711915] sd 0:0:4:0: [sdh] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.711919] sd 0:0:4:0: [sdh] 4096-byte physical blocks
[    8.711929] scsi 0:0:5:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.712093] sd 0:0:4:0: [sdh] Write Protect is off
[    8.712095] sd 0:0:4:0: [sdh] Mode Sense: a7 00 00 08
[    8.712329] sd 0:0:4:0: [sdh] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.712581] sd 0:0:4:0: [sdh] Optimal transfer size 65536 bytes
[    8.712635] sd 0:0:5:0: Attached scsi generic sg8 type 0
[    8.712747] sd 0:0:5:0: Power-on or device reset occurred
[    8.713471] sd 0:0:5:0: [sdi] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.713473] sd 0:0:5:0: [sdi] 4096-byte physical blocks
[    8.713523] scsi 0:0:6:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.713634] sd 0:0:5:0: [sdi] Write Protect is off
[    8.713636] sd 0:0:5:0: [sdi] Mode Sense: a7 00 00 08
[    8.713854] sd 0:0:5:0: [sdi] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.714111] sd 0:0:5:0: [sdi] Optimal transfer size 65536 bytes
[    8.714165] sd 0:0:6:0: Attached scsi generic sg9 type 0
[    8.714297] sd 0:0:6:0: Power-on or device reset occurred
[    8.715053] sd 0:0:6:0: [sdj] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.715056] sd 0:0:6:0: [sdj] 4096-byte physical blocks
[    8.715120] scsi 0:0:7:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.715220] sd 0:0:6:0: [sdj] Write Protect is off
[    8.715223] sd 0:0:6:0: [sdj] Mode Sense: a7 00 00 08
[    8.715458] sd 0:0:6:0: [sdj] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.715691] sd 0:0:6:0: [sdj] Optimal transfer size 65536 bytes
[    8.715763] sd 0:0:7:0: Attached scsi generic sg10 type 0
[    8.715856] sd 0:0:7:0: Power-on or device reset occurred
[    8.716630] sd 0:0:7:0: [sdk] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.716634] sd 0:0:7:0: [sdk] 4096-byte physical blocks
[    8.716681] scsi 0:0:8:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.716793] sd 0:0:7:0: [sdk] Write Protect is off
[    8.716796] sd 0:0:7:0: [sdk] Mode Sense: a7 00 00 08
[    8.717034] sd 0:0:7:0: [sdk] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.717311] sd 0:0:8:0: Attached scsi generic sg11 type 0
[    8.717331] sd 0:0:7:0: [sdk] Optimal transfer size 65536 bytes
[    8.717629] sd 0:0:8:0: Power-on or device reset occurred
[    8.718345] sd 0:0:8:0: [sdl] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.718349] sd 0:0:8:0: [sdl] 4096-byte physical blocks
[    8.718526] sd 0:0:8:0: [sdl] Write Protect is off
[    8.718528] sd 0:0:8:0: [sdl] Mode Sense: a7 00 00 08
[    8.718740] sd 0:0:8:0: [sdl] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.718999] sd 0:0:8:0: [sdl] Optimal transfer size 65536 bytes
[    8.719890] sd 0:0:0:1074675841: alua: transition timeout set to 60 seconds
[    8.719895] sd 0:0:0:1074675841: alua: port group 00 state A preferred supports tolusnA
[    8.719965] sd 0:0:0:1074741376: alua: transition timeout set to 60 seconds
[    8.719969] sd 0:0:0:1074741376: alua: port group 00 state A preferred supports tolusnA
[    8.720574] sd 0:0:0:1074675841: [sdc] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.720645] sd 0:0:0:1074741376: [sdb] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.720709] sd 0:0:0:1074675841: [sdc] Write Protect is off
[    8.720711] sd 0:0:0:1074675841: [sdc] Mode Sense: ed 00 00 08
[    8.720793] sd 0:0:0:1074741376: [sdb] Write Protect is off
[    8.720795] sd 0:0:0:1074741376: [sdb] Mode Sense: ed 00 00 08
[    8.721001] sd 0:0:0:1074675841: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.721086] sd 0:0:0:1074741376: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.739952] sd 0:0:0:1074741377: alua: transition timeout set to 60 seconds
[    8.739957] sd 0:0:0:1074741377: alua: port group 00 state A preferred supports tolusnA
[    8.740654] sd 0:0:0:1074741377: [sdd] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.740829] sd 0:0:0:1074741377: [sdd] Write Protect is off
[    8.740831] sd 0:0:0:1074741377: [sdd] Mode Sense: ed 00 00 08
[    8.741125] sd 0:0:0:1074741377: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.775521] scsi 1:0:0:1074675840: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.776156] scsi 1:0:0:1074675840: alua: supports implicit TPGS
[    8.776160] scsi 1:0:0:1074675840: alua: device naa.6005076305ffc1ae000000000000800e port group 0 rel port 31
[    8.776287] sd 1:0:0:1074675840: Attached scsi generic sg12 type 0
[    8.776651] sd 1:0:0:1074675840: Power-on or device reset occurred
[    8.777259] scsi 1:0:0:1074741376: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.777843] scsi 1:0:0:1074741376: alua: supports implicit TPGS
[    8.777847] scsi 1:0:0:1074741376: alua: device naa.6005076305ffc1ae000000000000800f port group 0 rel port 31
[    8.777962] sd 1:0:0:1074741376: Attached scsi generic sg13 type 0
[    8.778061] sd 1:0:0:1074741376: Power-on or device reset occurred
[    8.778842] scsi 1:0:0:1074675841: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.779476] scsi 1:0:0:1074675841: alua: supports implicit TPGS
[    8.779479] scsi 1:0:0:1074675841: alua: device naa.6005076305ffc1ae000000000000810e port group 0 rel port 31
[    8.779558] sd 1:0:0:1074675841: Attached scsi generic sg14 type 0
[    8.779719] sd 1:0:0:1074675841: Power-on or device reset occurred
[    8.780565] scsi 1:0:0:1074741377: Direct-Access     IBM      2107900          4200 PQ: 0 ANSI: 5
[    8.781230] scsi 1:0:0:1074741377: alua: supports implicit TPGS
[    8.781234] scsi 1:0:0:1074741377: alua: device naa.6005076305ffc1ae000000000000810f port group 0 rel port 31
[    8.781340] sd 1:0:0:1074741377: Attached scsi generic sg15 type 0
[    8.781466] sd 1:0:0:1074741377: Power-on or device reset occurred
[    8.782303] scsi 1:0:1:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.782931] sd 1:0:1:0: Attached scsi generic sg16 type 0
[    8.783074] sd 1:0:1:0: Power-on or device reset occurred
[    8.783859] sd 1:0:1:0: [sdq] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.783862] sd 1:0:1:0: [sdq] 4096-byte physical blocks
[    8.784026] sd 1:0:1:0: [sdq] Write Protect is off
[    8.784028] sd 1:0:1:0: [sdq] Mode Sense: a7 00 00 08
[    8.784035] scsi 1:0:2:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.784281] sd 1:0:1:0: [sdq] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.784529] sd 1:0:1:0: [sdq] Optimal transfer size 65536 bytes
[    8.784772] sd 1:0:2:0: Attached scsi generic sg17 type 0
[    8.784826] sd 1:0:2:0: Power-on or device reset occurred
[    8.785620] sd 1:0:2:0: [sdr] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.785623] sd 1:0:2:0: [sdr] 4096-byte physical blocks
[    8.785725] scsi 1:0:3:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.785801] sd 1:0:2:0: [sdr] Write Protect is off
[    8.785805] sd 1:0:2:0: [sdr] Mode Sense: a7 00 00 08
[    8.786034] sd 1:0:2:0: [sdr] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.786283] sd 1:0:2:0: [sdr] Optimal transfer size 65536 bytes
[    8.786467] sd 1:0:3:0: Attached scsi generic sg18 type 0
[    8.786527] sd 1:0:3:0: Power-on or device reset occurred
[    8.787347] sd 1:0:3:0: [sds] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.787351] sd 1:0:3:0: [sds] 4096-byte physical blocks
[    8.787470] scsi 1:0:4:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.787502] sd 1:0:3:0: [sds] Write Protect is off
[    8.787504] sd 1:0:3:0: [sds] Mode Sense: a7 00 00 08
[    8.787741] sd 1:0:3:0: [sds] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.787996] sd 1:0:3:0: [sds] Optimal transfer size 65536 bytes
[    8.788258] sd 1:0:4:0: Attached scsi generic sg19 type 0
[    8.788301] sd 1:0:4:0: Power-on or device reset occurred
[    8.788602] audit: type=1130 audit(1574427508.377:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-settle comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    8.789076] sd 1:0:4:0: [sdt] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.789079] sd 1:0:4:0: [sdt] 4096-byte physical blocks
[    8.789265] sd 1:0:4:0: [sdt] Write Protect is off
[    8.789269] sd 1:0:4:0: [sdt] Mode Sense: a7 00 00 08
[    8.789385] scsi 1:0:5:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.789532] sd 1:0:4:0: [sdt] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.789766] sd 1:0:4:0: [sdt] Optimal transfer size 65536 bytes
[    8.789987] sd 1:0:0:1074741376: alua: port group 00 state A preferred supports tolusnA
[    8.790048] sd 1:0:0:1074675840: alua: port group 00 state A preferred supports tolusnA
[    8.790165] sd 1:0:5:0: Attached scsi generic sg20 type 0
[    8.790332] sd 1:0:5:0: Power-on or device reset occurred
[    8.790644] audit: type=1130 audit(1574427508.387:11): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    8.790805] sd 1:0:0:1074741376: [sdn] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.790815] sd 1:0:0:1074675840: [sdm] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.790972] sd 1:0:0:1074741376: [sdn] Write Protect is off
[    8.790975] sd 1:0:0:1074741376: [sdn] Mode Sense: ed 00 00 08
[    8.790980] sd 1:0:0:1074675840: [sdm] Write Protect is off
[    8.790983] sd 1:0:0:1074675840: [sdm] Mode Sense: ed 00 00 08
[    8.791106] sd 1:0:5:0: [sdu] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.791109] sd 1:0:5:0: [sdu] 4096-byte physical blocks
[    8.791291] sd 1:0:5:0: [sdu] Write Protect is off
[    8.791293] sd 1:0:5:0: [sdu] Mode Sense: a7 00 00 08
[    8.791314] sd 1:0:0:1074741376: [sdn] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.791315] sd 1:0:0:1074675840: [sdm] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.791426] scsi 1:0:6:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.791546] sd 1:0:5:0: [sdu] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.791792] sd 1:0:5:0: [sdu] Optimal transfer size 65536 bytes
[    8.792288] sd 1:0:6:0: Power-on or device reset occurred
[    8.792325] sd 1:0:6:0: Attached scsi generic sg21 type 0
[    8.793031] audit: type=1130 audit(1574427508.387:12): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=multipathd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    8.793082] sd 1:0:6:0: [sdv] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.793086] sd 1:0:6:0: [sdv] 4096-byte physical blocks
[    8.793242] sd 1:0:6:0: [sdv] Write Protect is off
[    8.793248] sd 1:0:6:0: [sdv] Mode Sense: a7 00 00 08
[    8.793337] scsi 1:0:7:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.793463] sd 1:0:6:0: [sdv] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.793691] sd 1:0:6:0: [sdv] Optimal transfer size 65536 bytes
[    8.794072] sd 1:0:7:0: Power-on or device reset occurred
[    8.794086] sd 1:0:7:0: Attached scsi generic sg22 type 0
[    8.794847] sd 1:0:7:0: [sdw] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.794850] sd 1:0:7:0: [sdw] 4096-byte physical blocks
[    8.795018] sd 1:0:7:0: [sdw] Write Protect is off
[    8.795021] sd 1:0:7:0: [sdw] Mode Sense: a7 00 00 08
[    8.795260] sd 1:0:7:0: [sdw] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.795294] scsi 1:0:8:0: Direct-Access     IBM      FlashSystem-9840 1510 PQ: 0 ANSI: 5
[    8.795496] sd 1:0:7:0: [sdw] Optimal transfer size 65536 bytes
[    8.795969] sd 1:0:8:0: Attached scsi generic sg23 type 0
[    8.796209] sd 1:0:8:0: Power-on or device reset occurred
[    8.796930] sd 1:0:8:0: [sdx] 4294967296 512-byte logical blocks: (2.20 TB/2.00 TiB)
[    8.796934] sd 1:0:8:0: [sdx] 4096-byte physical blocks
[    8.797091] sd 1:0:8:0: [sdx] Write Protect is off
[    8.797092] sd 1:0:8:0: [sdx] Mode Sense: a7 00 00 08
[    8.797324] sd 1:0:8:0: [sdx] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    8.797602] sd 1:0:8:0: [sdx] Optimal transfer size 65536 bytes
[    8.809916] sd 1:0:0:1074741377: alua: port group 00 state A preferred supports tolusnA
[    8.809925] sd 1:0:0:1074675841: alua: port group 00 state A preferred supports tolusnA
[    8.810710] sd 1:0:0:1074675841: [sdo] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.810715] sd 1:0:0:1074741377: [sdp] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB)
[    8.810879] sd 1:0:0:1074675841: [sdo] Write Protect is off
[    8.810880] sd 1:0:0:1074675841: [sdo] Mode Sense: ed 00 00 08
[    8.810894] sd 1:0:0:1074741377: [sdp] Write Protect is off
[    8.810903] sd 1:0:0:1074741377: [sdp] Mode Sense: ed 00 00 08
[    8.811183] sd 1:0:0:1074675841: [sdo] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.811186] sd 1:0:0:1074741377: [sdp] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.813873]  sdl: sdl1 sdl2 sdl3
[    8.813882]  sde: sde1 sde2 sde3
[    8.813909]  sdj: sdj1 sdj2 sdj3
[    8.819049] sd 0:0:1:0: [sde] Attached SCSI disk
[    8.820072] sd 0:0:8:0: [sdl] Attached SCSI disk
[    8.820081] sd 0:0:6:0: [sdj] Attached SCSI disk
[    8.823682]  sdh: sdh1 sdh2 sdh3
[    8.823759]  sdk: sdk1 sdk2 sdk3
[    8.823768]  sdi: sdi1 sdi2 sdi3
[    8.823809]  sdf: sdf1 sdf2 sdf3
[    8.827323] EXT4-fs (dasda1): mounted filesystem with ordered data mode. Opts: (null)
[    8.828410] sd 0:0:4:0: [sdh] Attached SCSI disk
[    8.829447] sd 0:0:7:0: [sdk] Attached SCSI disk
[    8.829660] sd 0:0:2:0: [sdf] Attached SCSI disk
[    8.830120] sd 0:0:5:0: [sdi] Attached SCSI disk
[    8.842418]  sdn: sdn1 sdn2
[    8.843310]  sdb: sdb1 sdb2
[    8.843991]  sds: sds1 sds2 sds3
[    8.844254]  sdu: sdu1 sdu2 sdu3
[    8.844288]  sdq: sdq1 sdq2 sdq3
[    8.844982]  sdg: sdg1 sdg2 sdg3
[    8.845138]  sdx: sdx1 sdx2 sdx3
[    8.850072] sd 1:0:0:1074675840: [sdm] Attached SCSI disk
[    8.850352] sd 1:0:0:1074741376: [sdn] Attached SCSI disk
[    8.853228] sd 0:0:0:1074741377: [sdd] Attached SCSI disk
[    8.853374] sd 0:0:0:1074675840: [sda] Attached SCSI disk
[    8.853411]  sdr: sdr1 sdr2 sdr3
[    8.856368] sd 0:0:0:1074741376: [sdb] Attached SCSI disk
[    8.856603] sd 0:0:0:1074675841: [sdc] Attached SCSI disk
[    8.858293] sd 1:0:3:0: [sds] Attached SCSI disk
[    8.858396] sd 0:0:3:0: [sdg] Attached SCSI disk
[    8.858783] sd 1:0:5:0: [sdu] Attached SCSI disk
[    8.858823] sd 1:0:1:0: [sdq] Attached SCSI disk
[    8.858902] sd 1:0:8:0: [sdx] Attached SCSI disk
[    8.860584] sd 1:0:2:0: [sdr] Attached SCSI disk
[    8.874043]  sdv: sdv1 sdv2 sdv3
[    8.876853] sd 1:0:0:1074675841: [sdo] Attached SCSI disk
[    8.880523] sd 1:0:6:0: [sdv] Attached SCSI disk
[    8.883637]  sdw: sdw1 sdw2 sdw3
[    8.883678]  sdt: sdt1 sdt2 sdt3
[    8.887384] sd 1:0:0:1074741377: [sdp] Attached SCSI disk
[    8.890168] sd 1:0:4:0: [sdt] Attached SCSI disk
[    8.890370] sd 1:0:7:0: [sdw] Attached SCSI disk
[    9.452996] systemd-journald[348]: Received SIGTERM from PID 1 (systemd).
[    9.472233] printk: systemd: 20 output lines suppressed due to ratelimiting
[    9.860661] SELinux:  Permission watch in class filesystem not defined in policy.
[    9.860668] SELinux:  Permission watch in class file not defined in policy.
[    9.860669] SELinux:  Permission watch_mount in class file not defined in policy.
[    9.860670] SELinux:  Permission watch_sb in class file not defined in policy.
[    9.860671] SELinux:  Permission watch_with_perm in class file not defined in policy.
[    9.860672] SELinux:  Permission watch_reads in class file not defined in policy.
[    9.860675] SELinux:  Permission watch in class dir not defined in policy.
[    9.860676] SELinux:  Permission watch_mount in class dir not defined in policy.
[    9.860677] SELinux:  Permission watch_sb in class dir not defined in policy.
[    9.860679] SELinux:  Permission watch_with_perm in class dir not defined in policy.
[    9.860680] SELinux:  Permission watch_reads in class dir not defined in policy.
[    9.860683] SELinux:  Permission watch in class lnk_file not defined in policy.
[    9.860684] SELinux:  Permission watch_mount in class lnk_file not defined in policy.
[    9.860685] SELinux:  Permission watch_sb in class lnk_file not defined in policy.
[    9.860687] SELinux:  Permission watch_with_perm in class lnk_file not defined in policy.
[    9.860688] SELinux:  Permission watch_reads in class lnk_file not defined in policy.
[    9.860690] SELinux:  Permission watch in class chr_file not defined in policy.
[    9.860715] SELinux:  Permission watch_mount in class chr_file not defined in policy.
[    9.860717] SELinux:  Permission watch_sb in class chr_file not defined in policy.
[    9.860718] SELinux:  Permission watch_with_perm in class chr_file not defined in policy.
[    9.860719] SELinux:  Permission watch_reads in class chr_file not defined in policy.
[    9.860721] SELinux:  Permission watch in class blk_file not defined in policy.
[    9.860722] SELinux:  Permission watch_mount in class blk_file not defined in policy.
[    9.860724] SELinux:  Permission watch_sb in class blk_file not defined in policy.
[    9.860725] SELinux:  Permission watch_with_perm in class blk_file not defined in policy.
[    9.860726] SELinux:  Permission watch_reads in class blk_file not defined in policy.
[    9.860729] SELinux:  Permission watch in class sock_file not defined in policy.
[    9.860730] SELinux:  Permission watch_mount in class sock_file not defined in policy.
[    9.860731] SELinux:  Permission watch_sb in class sock_file not defined in policy.
[    9.860732] SELinux:  Permission watch_with_perm in class sock_file not defined in policy.
[    9.860733] SELinux:  Permission watch_reads in class sock_file not defined in policy.
[    9.860735] SELinux:  Permission watch in class fifo_file not defined in policy.
[    9.860737] SELinux:  Permission watch_mount in class fifo_file not defined in policy.
[    9.860738] SELinux:  Permission watch_sb in class fifo_file not defined in policy.
[    9.860739] SELinux:  Permission watch_with_perm in class fifo_file not defined in policy.
[    9.860740] SELinux:  Permission watch_reads in class fifo_file not defined in policy.
[    9.860834] SELinux:  Class perf_event not defined in policy.
[    9.860835] SELinux: the above unknown classes and permissions will be allowed
[    9.860838] SELinux:  policy capability network_peer_controls=1
[    9.860839] SELinux:  policy capability open_perms=1
[    9.860840] SELinux:  policy capability extended_socket_class=1
[    9.860841] SELinux:  policy capability always_check_network=0
[    9.860842] SELinux:  policy capability cgroup_seclabel=1
[    9.860842] SELinux:  policy capability nnp_nosuid_transition=1
[    9.985323] systemd[1]: Successfully loaded SELinux policy in 349.326ms.
[   10.102452] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 23.805ms.
[   10.104319] systemd[1]: systemd v241-12.git323cdf4.fc30 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[   10.104985] systemd[1]: Detected architecture s390x.
[   10.106198] systemd[1]: Set hostname to <s38lp08.lnxne.boe>.
[   10.108987] systemd[1]: Couldn't move remaining userspace processes, ignoring: Input/output error
[   10.217550] systemd[1]: /usr/lib/systemd/system/gssproxy.service:9: PIDFile= references path below legacy directory /var/run/, updating /var/run/gssproxy.pid → /run/gssproxy.pid; please update the unit file accordingly.
[   10.247504] systemd[1]: /usr/lib/systemd/system/sssd.service:11: PIDFile= references path below legacy directory /var/run/, updating /var/run/sssd.pid → /run/sssd.pid; please update the unit file accordingly.
[   10.251473] systemd[1]: /usr/lib/systemd/system/mdmonitor.service:6: PIDFile= references path below legacy directory /var/run/, updating /var/run/mdadm/mdadm.pid → /run/mdadm/mdadm.pid; please update the unit file accordingly.
[   10.252505] systemd[1]: /usr/lib/systemd/system/iscsid.service:10: PIDFile= references path below legacy directory /var/run/, updating /var/run/iscsid.pid → /run/iscsid.pid; please update the unit file accordingly.
[   10.252747] systemd[1]: /usr/lib/systemd/system/iscsiuio.service:12: PIDFile= references path below legacy directory /var/run/, updating /var/run/iscsiuio.pid → /run/iscsiuio.pid; please update the unit file accordingly.
[   10.327019] systemd[1]: initrd-switch-root.service: Succeeded.
[   10.335802] EXT4-fs (dasda1): re-mounted. Opts: (null)
[   10.529721] Adding 21635084k swap on /dev/dasdc1.  Priority:-2 extents:1 across:21635084k SSFS
[   10.598942] systemd-journald[1048]: Received request to flush runtime journal from PID 1
[   11.130303] VFIO - User Level meta-driver version: 0.3
[   11.266349] qeth 0.0.e400 enccw0.0.e400: renamed from ence400
[   11.278773] dasdconf.sh Warning: 0.0.241c is already online, not configuring
[   11.281981] dasdconf.sh Warning: 0.0.241d is already online, not configuring
[   11.282514] dasdconf.sh Warning: 0.0.241e is already online, not configuring
[   11.798179] scsi host2: zfcp
[   11.799357] dasd-eckd 0.0.241f: A channel path to the device has become operational
[   11.800366] qdio: 0.0.3c00 ZFCP on SC 24b using AI:1 QEBSM:0 PRI:1 TDD:1 SIGA: W AP
[   11.801689] dasd-eckd 0.0.241f: New DASD 3390/0C (CU 3990/01) with 30051 cylinders, 15 heads, 224 sectors
[   11.805190] dasd-eckd 0.0.241f: DASD with 4 KB/block, 21636720 KB total size, 48 KB/track, compatible disk layout
[   11.806695]  dasdd:VOL1/  0X3333: dasdd1 dasdd2
[   11.813044] qdio: 0.0.bd02 OSA on SC 3cb using AI:1 QEBSM:0 PRI:1 TDD:1 SIGA: W AP
[   11.843878] qeth 0.0.bd00: QDIO data connection isolation is deactivated
[   11.844417] qeth 0.0.bd00: The device represents a Bridge Capable Port
[   11.845432] qeth 0.0.bd00: MAC address 3e:a9:e4:18:c1:3b successfully registered
[   11.846045] qeth 0.0.bd00: Device is a OSD Express card (level: 0774)
               with link type OSD_10GIG.
[   11.848707] qeth 0.0.bd00: MAC address 3a:a4:04:94:65:31 successfully registered
[   11.849429] qeth 0.0.bd00 enccw0.0.bd00: renamed from eth0
[   12.053175] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[   12.055871] <mlx4_ib> mlx4_ib_add: counter index 255 for port 1 allocated 1
[   12.055996] <mlx4_ib> mlx4_ib_add: counter index 255 for port 2 allocated 1
[   12.070959] mlx4_core 0000:00:00.0: mlx4_ib: multi-function enabled
[   12.070961] mlx4_core 0000:00:00.0: mlx4_ib: operating in qp1 tunnel mode
[   12.832508] RPC: Registered named UNIX socket transport module.
[   12.832512] RPC: Registered udp transport module.
[   12.832513] RPC: Registered tcp transport module.
[   12.832514] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   12.937392] RPC: Registered rdma transport module.
[   12.937394] RPC: Registered rdma backchannel transport module.
[   12.938939] kauditd_printk_skb: 89 callbacks suppressed
[   12.938940] audit: type=1130 audit(1574427512.527:102): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rdma-load-modules@rdma comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[   13.501763] audit: type=1130 audit(1574427513.097:103): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-pvscan@94:6 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   20.761726] audit: type=1130 audit(1574427520.357:104): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udev-settle comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   20.838019] audit: type=1130 audit(1574427520.427:105): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rdma comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   20.906337] device-mapper: multipath service-time: version 0.3.0 loaded
[   21.129899] sd 0:0:0:1074675840: alua: port group 00 state A preferred supports tolusnA
[   21.130078] sd 0:0:0:1074675840: alua: port group 00 state A preferred supports tolusnA
[   21.239701] Adding 15728636k swap on /dev/mapper/mpathk.  Priority:1 extents:1 across:15728636k FS
[   21.319910] sd 0:0:0:1074675841: alua: port group 00 state A preferred supports tolusnA
[   21.320092] sd 0:0:0:1074675841: alua: port group 00 state A preferred supports tolusnA
[   21.419737] Adding 15728636k swap on /dev/mapper/mpathl.  Priority:1 extents:1 across:15728636k FS
[   21.509927] sd 0:0:0:1074741376: alua: port group 00 state A preferred supports tolusnA
[   21.510104] sd 0:0:0:1074741376: alua: port group 00 state A preferred supports tolusnA
[   21.669922] sd 0:0:0:1074741377: alua: port group 00 state A preferred supports tolusnA
[   21.670100] sd 0:0:0:1074741377: alua: port group 00 state A preferred supports tolusnA
[   21.691123] audit: type=1130 audit(1574427521.287:106): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-pvscan@252:9 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[   21.739744] Adding 15728636k swap on /dev/mapper/mpathn.  Priority:1 extents:1 across:15728636k FS
[   21.840628] audit: type=1130 audit(1574427521.437:107): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=multipathd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   21.902512] EXT4-fs (dm-11): mounted filesystem with ordered data mode. Opts: (null)
[   21.904787] EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null)
[   22.089686] Adding 1027604476k swap on /dev/mapper/mpathae2.  Priority:1 extents:1 across:1027604476k SSFS
[   22.106879] audit: type=1130 audit(1574427521.697:108): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dracut-shutdown comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   22.120853] audit: type=1130 audit(1574427521.717:109): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plymouth-read-write comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   22.120857] audit: type=1131 audit(1574427521.717:110): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plymouth-read-write comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   22.122930] audit: type=1130 audit(1574427521.717:111): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   22.162986] audit: type=1130 audit(1574427521.757:112): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   23.447581] mlx4_en: ens12: Steering Mode 1
[   23.525866] mlx4_en: ens12: Link Up
[   23.530471] IPv6: ADDRCONF(NETDEV_CHANGE): ens12: link becomes ready
[   23.545351] mlx4_en: ens12d1: Steering Mode 1
[   23.669616] Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
[   23.707898] qeth 0.0.bd00: MAC address 02:38:08:00:76:00 successfully registered
[   23.709373] qeth 0.0.e400: MAC address 02:38:08:00:98:00 successfully registered
[   23.713864] qeth 0.0.bd00: MAC address 3a:a4:04:94:65:31 successfully registered
[   23.762803] qeth 0.0.bd00: MAC address 02:38:08:00:76:00 successfully registered
[   23.980344] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   24.021545] tun: Universal TUN/TAP device driver, 1.6
[   24.022460] virbr0: port 1(virbr0-nic) entered blocking state
[   24.022461] virbr0: port 1(virbr0-nic) entered disabled state
[   24.022532] device virbr0-nic entered promiscuous mode
[   24.265330] bond0: (slave enccw0.0.bd00): making interface the new active one
[   24.265435] bond0: (slave enccw0.0.bd00): Enslaving as an active interface with an up link
[   24.265662] IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[   24.472270] virbr0: port 1(virbr0-nic) entered blocking state
[   24.472273] virbr0: port 1(virbr0-nic) entered listening state
[   24.500413] virbr0: port 1(virbr0-nic) entered disabled state
[   34.272547] ------------[ cut here ]------------
[   34.272557] WARNING: CPU: 21 PID: 275 at lib/percpu-refcount.c:111 percpu_ref_exit+0x50/0x58
[   34.272558] Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 tun xt_conntrack bridge ip6table_nat ip6table_mangle ip6table_raw stp ip6table_security llc iptable_nat nf_nat iptable_mangle iptable_raw iptable_security bonding nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ip6table_filter ip6_tables iptable_filter dm_service_time rpcrdma sunrpc rdma_ucm rdma_cm iw_cm ib_cm configfs ghash_s390 mlx4_ib prng des_s390 libdes ib_uverbs ib_core sha512_s390 sha1_s390 vfio_ccw vfio_mdev mdev eadm_sch vfio_iommu_type1 zcrypt_cex4 vfio sch_fq_codel ip_tables x_tables mlx4_en mlx4_core sha256_s390 sha_common dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt rng_core autofs4
[   34.272593] CPU: 21 PID: 275 Comm: kworker/21:1 Not tainted 5.4.0-rc8-next-20191122 #70
[   34.272594] Hardware name: IBM 2964 NC9 712 (LPAR)
[   34.272598] Workqueue: events sysfs_slab_remove_workfn
[   34.272599] Krnl PSW : 0704e00180000000 000000136693a358 (percpu_ref_exit+0x50/0x58)
[   34.272602]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
[   34.272603] Krnl GPRS: 00000000fffff008 000000136693a230 000003f320603e68 0000001fa66f7000
[   34.272604]            0000001f00000000 002c008100000000 0000001f97b406b8 0000000000000000
[   34.272606]            0000001fbde29200 0000000000000000 00000013670d7828 0000001f97b40700
[   34.272631]            0000001fb5380000 0000001fbde24700 000003e00cb8fd00 000003e00cb8fcd0
[   34.272639] Krnl Code: 000000136693a34a: f0a0000407fe	srp	4(11,%r0),2046,0
                          000000136693a350: 47000700		bc	0,1792
                         #000000136693a354: a7f40001		brc	15,000000136693a356
                         >000000136693a358: a7f4fff2		brc	15,000000136693a33c
                          000000136693a35c: 0707		bcr	0,%r7
                          000000136693a35e: 0707		bcr	0,%r7
                          000000136693a360: eb8ff0580024	stmg	%r8,%r15,88(%r15)
                          000000136693a366: a738ffff		lhi	%r3,-1
[   34.272651] Call Trace:
[   34.272653] ([<000003e00cb8fd00>] 0x3e00cb8fd00)
[   34.272658]  [<00000013664e04b0>] slab_kmem_cache_release+0x30/0x58 
[   34.272661]  [<0000001366cab9c2>] kobject_put+0xaa/0xe8 
[   34.272663]  [<00000013663314d4>] process_one_work+0x1fc/0x440 
[   34.272665]  [<0000001366331760>] worker_thread+0x48/0x460 
[   34.272667]  [<0000001366338cae>] kthread+0x126/0x160 
[   34.272669]  [<0000001366cc9f9c>] ret_from_fork+0x28/0x30 
[   34.272671]  [<0000001366cc9fa4>] kernel_thread_starter+0x0/0x10 
[   34.272671] Last Breaking-Event-Address:
[   34.272673]  [<000000136693a354>] percpu_ref_exit+0x4c/0x58
[   34.272674] ---[ end trace 9556bf4ae9878b8f ]---

^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-21 18:45         ` Roman Gushchin
  2019-11-21 20:43           ` Rik van Riel
@ 2019-11-22 16:28           ` Christian Borntraeger
  2019-11-24  0:39             ` Roman Gushchin
  1 sibling, 1 reply; 36+ messages in thread
From: Christian Borntraeger @ 2019-11-22 16:28 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm

On 21.11.19 19:45, Roman Gushchin wrote:
> I see. Do you know, which kmem_cache it is? If not, can you, please,
> figure it out?

The release function for that ref is kmemcg_cache_shutdown. 


^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-22 16:28           ` Christian Borntraeger
@ 2019-11-24  0:39             ` Roman Gushchin
  2019-11-25  8:00               ` Christian Borntraeger
  0 siblings, 1 reply; 36+ messages in thread
From: Roman Gushchin @ 2019-11-24  0:39 UTC (permalink / raw)
  To: Christian Borntraeger
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm

On Fri, Nov 22, 2019 at 05:28:46PM +0100, Christian Borntraeger wrote:
> On 21.11.19 19:45, Roman Gushchin wrote:
> > I see. Do you know, which kmem_cache it is? If not, can you, please,
> > figure it out?
> 
> The release function for that ref is kmemcg_cache_shutdown. 
> 

Hi Christian!

Can you, please, test if the following patch resolves the problem?

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 8afa188f6e20..628e5f0ee19e 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -888,6 +888,8 @@ static int shutdown_memcg_caches(struct kmem_cache *s)
 
 static void flush_memcg_workqueue(struct kmem_cache *s)
 {
+	bool wait_for_children;
+
 	spin_lock_irq(&memcg_kmem_wq_lock);
 	s->memcg_params.dying = true;
 	spin_unlock_irq(&memcg_kmem_wq_lock);
@@ -904,6 +906,13 @@ static void flush_memcg_workqueue(struct kmem_cache *s)
 	 * previous workitems on workqueue are processed.
 	 */
 	flush_workqueue(memcg_kmem_cache_wq);
+
+	mutex_lock(&slab_mutex);
+	wait_for_children = !list_empty(&s->memcg_params.children);
+	mutex_unlock(&slab_mutex);
+
+	if (wait_for_children)
+		rcu_barrier();
 }
 #else
 static inline int shutdown_memcg_caches(struct kmem_cache *s)

--

Thanks!

^ permalink raw reply related	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-24  0:39             ` Roman Gushchin
@ 2019-11-25  8:00               ` Christian Borntraeger
  2019-11-25 18:07                 ` Roman Gushchin
  0 siblings, 1 reply; 36+ messages in thread
From: Christian Borntraeger @ 2019-11-25  8:00 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm



On 24.11.19 01:39, Roman Gushchin wrote:
> On Fri, Nov 22, 2019 at 05:28:46PM +0100, Christian Borntraeger wrote:
>> On 21.11.19 19:45, Roman Gushchin wrote:
>>> I see. Do you know, which kmem_cache it is? If not, can you, please,
>>> figure it out?
>>
>> The release function for that ref is kmemcg_cache_shutdown. 
>>
> 
> Hi Christian!
> 
> Can you, please, test if the following patch resolves the problem?

Yes, it does.


> 
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 8afa188f6e20..628e5f0ee19e 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -888,6 +888,8 @@ static int shutdown_memcg_caches(struct kmem_cache *s)
>  
>  static void flush_memcg_workqueue(struct kmem_cache *s)
>  {
> +	bool wait_for_children;
> +
>  	spin_lock_irq(&memcg_kmem_wq_lock);
>  	s->memcg_params.dying = true;
>  	spin_unlock_irq(&memcg_kmem_wq_lock);
> @@ -904,6 +906,13 @@ static void flush_memcg_workqueue(struct kmem_cache *s)
>  	 * previous workitems on workqueue are processed.
>  	 */
>  	flush_workqueue(memcg_kmem_cache_wq);
> +
> +	mutex_lock(&slab_mutex);
> +	wait_for_children = !list_empty(&s->memcg_params.children);
> +	mutex_unlock(&slab_mutex);

Not sure if (for reading) we really need the mutex. 
> +
> +	if (wait_for_children)
> +		rcu_barrier();
>  }
>  #else
>  static inline int shutdown_memcg_caches(struct kmem_cache *s)
> 
> --
> 
> Thanks!
> 


^ permalink raw reply	[flat|nested] 36+ messages in thread

* Re: WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management)
  2019-11-25  8:00               ` Christian Borntraeger
@ 2019-11-25 18:07                 ` Roman Gushchin
  0 siblings, 0 replies; 36+ messages in thread
From: Roman Gushchin @ 2019-11-25 18:07 UTC (permalink / raw)
  To: Christian Borntraeger
  Cc: akpm, hannes, Kernel Team, linux-kernel, linux-mm, longman,
	shakeelb, vdavydov.dev, Heiko Carstens, Janosch Frank, kvm

On Mon, Nov 25, 2019 at 09:00:56AM +0100, Christian Borntraeger wrote:
> 
> 
> On 24.11.19 01:39, Roman Gushchin wrote:
> > On Fri, Nov 22, 2019 at 05:28:46PM +0100, Christian Borntraeger wrote:
> >> On 21.11.19 19:45, Roman Gushchin wrote:
> >>> I see. Do you know, which kmem_cache it is? If not, can you, please,
> >>> figure it out?
> >>
> >> The release function for that ref is kmemcg_cache_shutdown. 
> >>
> > 
> > Hi Christian!
> > 
> > Can you, please, test if the following patch resolves the problem?
> 
> Yes, it does.

Thanks for testing it!
I'll send the patch shortly.

> 
> 
> > 
> > diff --git a/mm/slab_common.c b/mm/slab_common.c
> > index 8afa188f6e20..628e5f0ee19e 100644
> > --- a/mm/slab_common.c
> > +++ b/mm/slab_common.c
> > @@ -888,6 +888,8 @@ static int shutdown_memcg_caches(struct kmem_cache *s)
> >  
> >  static void flush_memcg_workqueue(struct kmem_cache *s)
> >  {
> > +	bool wait_for_children;
> > +
> >  	spin_lock_irq(&memcg_kmem_wq_lock);
> >  	s->memcg_params.dying = true;
> >  	spin_unlock_irq(&memcg_kmem_wq_lock);
> > @@ -904,6 +906,13 @@ static void flush_memcg_workqueue(struct kmem_cache *s)
> >  	 * previous workitems on workqueue are processed.
> >  	 */
> >  	flush_workqueue(memcg_kmem_cache_wq);
> > +
> > +	mutex_lock(&slab_mutex);
> > +	wait_for_children = !list_empty(&s->memcg_params.children);
> > +	mutex_unlock(&slab_mutex);
> 
> Not sure if (for reading) we really need the mutex.

Good point!
At this moment the list of children caches can't grow, only shrink.
So if we're reading it without the slab mutex, the worst thing can
happen is that we'll make an excessive rcu_barrier() call.
Which is fine given that resulting code looks much simpler.

Thanks!

^ permalink raw reply	[flat|nested] 36+ messages in thread

end of thread, other threads:[~2019-11-25 18:08 UTC | newest]

Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-11 23:18 [PATCH v7 00/10] mm: reparent slab memory on cgroup removal Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Roman Gushchin
2019-06-13  2:04   ` Andrew Morton
2019-06-13 16:25     ` Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields Roman Gushchin
2019-06-25 18:17   ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation Roman Gushchin
2019-06-25 18:17   ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 04/10] mm: introduce __memcg_kmem_uncharge_memcg() Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 05/10] mm: unify SLAB and SLUB page accounting Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation Roman Gushchin
2019-06-16 16:26   ` Vladimir Davydov
2019-06-25 18:31   ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Roman Gushchin
2019-06-16 16:27   ` Vladimir Davydov
2019-06-25 18:33   ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management Roman Gushchin
2019-06-25 23:57   ` Shakeel Butt
2019-11-21 11:17   ` WARNING bisected (was Re: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management) Christian Borntraeger
2019-11-21 13:08     ` Christian Borntraeger
2019-11-21 16:58     ` Roman Gushchin
2019-11-21 16:59       ` Christian Borntraeger
2019-11-21 18:45         ` Roman Gushchin
2019-11-21 20:43           ` Rik van Riel
2019-11-21 20:55             ` Roman Gushchin
2019-11-21 22:09               ` Roman Gushchin
2019-11-22 13:00                 ` Christian Borntraeger
2019-11-22 16:28           ` Christian Borntraeger
2019-11-24  0:39             ` Roman Gushchin
2019-11-25  8:00               ` Christian Borntraeger
2019-11-25 18:07                 ` Roman Gushchin
2019-06-11 23:18 ` [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages Roman Gushchin
2019-06-26  0:15   ` Shakeel Butt
2019-06-11 23:18 ` [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal Roman Gushchin
2019-06-16 16:29   ` Vladimir Davydov
2019-06-26  0:15   ` Shakeel Butt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).