All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jia Zhu <zhujia.zj@bytedance.com>
To: linux-erofs@lists.ozlabs.org
Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
	yinxin.x@bytedance.com, jefflexu@linux.alibaba.com,
	Jia Zhu <zhujia.zj@bytedance.com>
Subject: [PATCH V5 0/6] Introduce erofs shared domain
Date: Fri, 16 Sep 2022 16:59:34 +0800	[thread overview]
Message-ID: <20220916085940.89392-1-zhujia.zj@bytedance.com> (raw)

Changes since V4:
0. Relinquish the volume after mutex unlock in erofs_fscache_domain_put().
1. Use kill_anon_super() instead of kill_litter_super() to umount pseudo mnt.
2. Extract erofs_fscache_relinquish_cookie() to reduce lines.
3. Add code comments.
4. Remove useless local variable initialization.
5. Add "Fixes" line to patch 1.
6. Add Reviewed-by lines from Jingbo Xu.

[Kernel Patchset]
===============
Git tree:
	https://github.com/userzj/linux.git zhujia/shared-domain-v5
Git web:
	https://github.com/userzj/linux/tree/zhujia/shared-domain-v5

[User Daemon for Quick Test]
============================
Git web:
	https://github.com/userzj/demand-read-cachefilesd/tree/shared-domain
More test cases will be added to:
	https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/log/?h=experimental-tests-fscache 

[E2E Container Demo for Quick Test]
===================================
[Issue]
	https://github.com/containerd/nydus-snapshotter/issues/161
[PR]
	https://github.com/containerd/nydus-snapshotter/pull/162

[Background]
============
In ondemand read mode, we use individual volume to present an erofs
mountpoint, cookies to present bootstrap and data blobs.

In which case, since cookies can't be shared between fscache volumes,
even if the data blobs between different mountpoints are exactly same,
they can't be shared.

[Introduction]
==============
Here we introduce erofs shared domain to resolve above mentioned case.
Several erofs filesystems can belong to one domain, and data blobs can
be shared among these erofs filesystems of same domain.

[Usage]
Users could specify 'domain_id' mount option to create or join into a
domain which reuses the same cookies(blobs).

[Design]
========
1. Use pseudo mnt to manage domain's lifecycle.
2. Use a linked list to maintain & traverse domains.
3. Use pseudo sb to create anonymous inode for recording cookie's info
   and manage cookies lifecycle.

[Flow Path]
===========
1. User specify a new 'domain_id' in mount option.
   1.1 Traverse domain list, compare domain_id with existing domain.[Miss]
   1.2 Create a new domain(volume), add it to domain list.
   1.3 Traverse pseudo sb's inode list, compare cookie name with
       existing cookies.[Miss]
   1.4 Alloc new anonymous inodes and cookies.

2. User specify an existing 'domain_id' in mount option and the data
   blob is existed in domain.
   2.1 Traverse domain list, compare domain_id with existing domain.[Hit]
   2.2 Reuse the domain and increase its refcnt.
   2.3 Traverse pseudo sb's inode list, compare cookie name with
   	   existing cookies.[Hit]
   2.4 Reuse the cookie and increase its refcnt.

RFC: https://lore.kernel.org/all/YxAlO%2FDHDrIAafR2@B-P7TQMD6M-0146.local/
V1: https://lore.kernel.org/all/20220902034748.60868-1-zhujia.zj@bytedance.com/
V2: https://lore.kernel.org/all/20220902105305.79687-1-zhujia.zj@bytedance.com/
V3: https://lore.kernel.org/all/20220914105041.42970-1-zhujia.zj@bytedance.com/
V4: https://lore.kernel.org/all/20220915124213.25767-1-zhujia.zj@bytedance.com/

Jia Zhu (6):
  erofs: use kill_anon_super() to kill super in fscache mode
  erofs: code clean up for fscache
  erofs: introduce fscache-based domain
  erofs: introduce a pseudo mnt to manage shared cookies
  erofs: Support sharing cookies in the same domain
  erofs: introduce 'domain_id' mount option

 fs/erofs/fscache.c  | 264 ++++++++++++++++++++++++++++++++++++++------
 fs/erofs/internal.h |  32 ++++--
 fs/erofs/super.c    |  73 +++++++++---
 fs/erofs/sysfs.c    |  19 +++-
 4 files changed, 325 insertions(+), 63 deletions(-)

-- 
2.20.1


WARNING: multiple messages have this Message-ID (diff)
From: Jia Zhu <zhujia.zj@bytedance.com>
To: linux-erofs@lists.ozlabs.org
Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
	yinxin.x@bytedance.com
Subject: [PATCH V5 0/6] Introduce erofs shared domain
Date: Fri, 16 Sep 2022 16:59:34 +0800	[thread overview]
Message-ID: <20220916085940.89392-1-zhujia.zj@bytedance.com> (raw)

Changes since V4:
0. Relinquish the volume after mutex unlock in erofs_fscache_domain_put().
1. Use kill_anon_super() instead of kill_litter_super() to umount pseudo mnt.
2. Extract erofs_fscache_relinquish_cookie() to reduce lines.
3. Add code comments.
4. Remove useless local variable initialization.
5. Add "Fixes" line to patch 1.
6. Add Reviewed-by lines from Jingbo Xu.

[Kernel Patchset]
===============
Git tree:
	https://github.com/userzj/linux.git zhujia/shared-domain-v5
Git web:
	https://github.com/userzj/linux/tree/zhujia/shared-domain-v5

[User Daemon for Quick Test]
============================
Git web:
	https://github.com/userzj/demand-read-cachefilesd/tree/shared-domain
More test cases will be added to:
	https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/log/?h=experimental-tests-fscache 

[E2E Container Demo for Quick Test]
===================================
[Issue]
	https://github.com/containerd/nydus-snapshotter/issues/161
[PR]
	https://github.com/containerd/nydus-snapshotter/pull/162

[Background]
============
In ondemand read mode, we use individual volume to present an erofs
mountpoint, cookies to present bootstrap and data blobs.

In which case, since cookies can't be shared between fscache volumes,
even if the data blobs between different mountpoints are exactly same,
they can't be shared.

[Introduction]
==============
Here we introduce erofs shared domain to resolve above mentioned case.
Several erofs filesystems can belong to one domain, and data blobs can
be shared among these erofs filesystems of same domain.

[Usage]
Users could specify 'domain_id' mount option to create or join into a
domain which reuses the same cookies(blobs).

[Design]
========
1. Use pseudo mnt to manage domain's lifecycle.
2. Use a linked list to maintain & traverse domains.
3. Use pseudo sb to create anonymous inode for recording cookie's info
   and manage cookies lifecycle.

[Flow Path]
===========
1. User specify a new 'domain_id' in mount option.
   1.1 Traverse domain list, compare domain_id with existing domain.[Miss]
   1.2 Create a new domain(volume), add it to domain list.
   1.3 Traverse pseudo sb's inode list, compare cookie name with
       existing cookies.[Miss]
   1.4 Alloc new anonymous inodes and cookies.

2. User specify an existing 'domain_id' in mount option and the data
   blob is existed in domain.
   2.1 Traverse domain list, compare domain_id with existing domain.[Hit]
   2.2 Reuse the domain and increase its refcnt.
   2.3 Traverse pseudo sb's inode list, compare cookie name with
   	   existing cookies.[Hit]
   2.4 Reuse the cookie and increase its refcnt.

RFC: https://lore.kernel.org/all/YxAlO%2FDHDrIAafR2@B-P7TQMD6M-0146.local/
V1: https://lore.kernel.org/all/20220902034748.60868-1-zhujia.zj@bytedance.com/
V2: https://lore.kernel.org/all/20220902105305.79687-1-zhujia.zj@bytedance.com/
V3: https://lore.kernel.org/all/20220914105041.42970-1-zhujia.zj@bytedance.com/
V4: https://lore.kernel.org/all/20220915124213.25767-1-zhujia.zj@bytedance.com/

Jia Zhu (6):
  erofs: use kill_anon_super() to kill super in fscache mode
  erofs: code clean up for fscache
  erofs: introduce fscache-based domain
  erofs: introduce a pseudo mnt to manage shared cookies
  erofs: Support sharing cookies in the same domain
  erofs: introduce 'domain_id' mount option

 fs/erofs/fscache.c  | 264 ++++++++++++++++++++++++++++++++++++++------
 fs/erofs/internal.h |  32 ++++--
 fs/erofs/super.c    |  73 +++++++++---
 fs/erofs/sysfs.c    |  19 +++-
 4 files changed, 325 insertions(+), 63 deletions(-)

-- 
2.20.1


             reply	other threads:[~2022-09-16  9:00 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-16  8:59 Jia Zhu [this message]
2022-09-16  8:59 ` [PATCH V5 0/6] Introduce erofs shared domain Jia Zhu
2022-09-16  8:59 ` [PATCH V5 1/6] erofs: use kill_anon_super() to kill super in fscache mode Jia Zhu
2022-09-16  8:59   ` Jia Zhu
2022-09-16  8:59 ` [PATCH V5 2/6] erofs: code clean up for fscache Jia Zhu
2022-09-16  8:59   ` Jia Zhu
2022-09-16  8:59 ` [PATCH V5 3/6] erofs: introduce fscache-based domain Jia Zhu
2022-09-16  8:59   ` Jia Zhu
2022-09-16  8:59 ` [PATCH V5 4/6] erofs: introduce a pseudo mnt to manage shared cookies Jia Zhu
2022-09-16  8:59   ` Jia Zhu
2022-09-16  8:59 ` [PATCH V5 5/6] erofs: Support sharing cookies in the same domain Jia Zhu
2022-09-16  8:59   ` Jia Zhu
2022-09-16  9:29   ` JeffleXu
2022-09-16  8:59 ` [PATCH V5 6/6] erofs: introduce 'domain_id' mount option Jia Zhu
2022-09-16  8:59   ` Jia Zhu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220916085940.89392-1-zhujia.zj@bytedance.com \
    --to=zhujia.zj@bytedance.com \
    --cc=jefflexu@linux.alibaba.com \
    --cc=linux-erofs@lists.ozlabs.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=yinxin.x@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.