From: NeilBrown <neilb@suse.com> To: Oleg Drokin <oleg.drokin@intel.com>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, James Simmons <jsimmons@infradead.org>, Andreas Dilger <andreas.dilger@intel.com> Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Lustre Development List <lustre-devel@lists.lustre.org> Subject: [PATCH 00/20] staging: lustre: convert to rhashtable Date: Thu, 12 Apr 2018 07:54:48 +1000 [thread overview] Message-ID: <152348312863.12394.11915752362061083241.stgit@noble> (raw) libcfs in lustre has a resizeable hashtable. Linux already has a resizeable hashtable, rhashtable, which is better is most metrics. See https://lwn.net/Articles/751374/ in a few days for an introduction to rhashtable. This series converts lustre to use rhashtable. This affects several different tables, and each is different is various ways. There are two outstanding issues. One is that a bug in rhashtable means that we cannot enable auto-shrinking in one of the tables. That is documented as appropriate and should be fixed soon. The other is that rhashtable has an atomic_t which counts the elements in a hash table. At least one table in lustre went to some trouble to avoid any table-wide atomics, so that could lead to a regression. I'm hoping that rhashtable can be enhanced with the option of a per-cpu counter, or similar. I have enabled automatic shrinking on all tables where it makes sense and doesn't trigger the bug. I have also removed all hints concerning min/max size - I cannot see how these could be useful. The dump_pgcache debugfs file provided some interesting challenges. I think I have cleaned it up enough so that it all makes sense. An extra pair of eyes examining that code in particular would be appreciated. This series passes all the same tests that pass before the patches are applied. Thanks, NeilBrown --- NeilBrown (20): staging: lustre: ptlrpc: convert conn_hash to rhashtable staging: lustre: convert lov_pool to use rhashtable staging: lustre: convert obd uuid hash to rhashtable staging: lustre: convert osc_quota hash to rhashtable staging: lustre: separate buckets from ldlm hash table staging: lustre: ldlm: add a counter to the per-namespace data staging: lustre: ldlm: store name directly in namespace. staging: lustre: simplify ldlm_ns_hash_defs[] staging: lustre: convert ldlm_resource hash to rhashtable. staging: lustre: make struct lu_site_bkt_data private staging: lustre: lu_object: discard extra lru count. staging: lustre: lu_object: factor out extra per-bucket data staging: lustre: lu_object: move retry logic inside htable_lookup staging: lustre: fold lu_object_new() into lu_object_find_at() staging: lustre: llite: use more private data in dump_pgcache staging: lustre: llite: remove redundant lookup in dump_pgcache staging: lustre: use call_rcu() to free lu_object_headers staging: lustre: change how "dump_page_cache" walks a hash table staging: lustre: convert lu_object cache to rhashtable staging: lustre: remove cfs_hash resizeable hashtable implementation. .../staging/lustre/include/linux/libcfs/libcfs.h | 1 .../lustre/include/linux/libcfs/libcfs_hash.h | 866 -------- drivers/staging/lustre/lnet/libcfs/Makefile | 2 drivers/staging/lustre/lnet/libcfs/hash.c | 2064 -------------------- drivers/staging/lustre/lnet/libcfs/module.c | 12 drivers/staging/lustre/lustre/include/lu_object.h | 55 - drivers/staging/lustre/lustre/include/lustre_dlm.h | 19 .../staging/lustre/lustre/include/lustre_export.h | 2 drivers/staging/lustre/lustre/include/lustre_net.h | 4 drivers/staging/lustre/lustre/include/obd.h | 11 .../staging/lustre/lustre/include/obd_support.h | 9 drivers/staging/lustre/lustre/ldlm/ldlm_request.c | 31 drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 370 +--- drivers/staging/lustre/lustre/llite/lcommon_cl.c | 8 drivers/staging/lustre/lustre/llite/vvp_dev.c | 332 +-- drivers/staging/lustre/lustre/llite/vvp_object.c | 9 drivers/staging/lustre/lustre/lov/lov_internal.h | 11 drivers/staging/lustre/lustre/lov/lov_obd.c | 12 drivers/staging/lustre/lustre/lov/lov_object.c | 8 drivers/staging/lustre/lustre/lov/lov_pool.c | 159 +- drivers/staging/lustre/lustre/lov/lovsub_object.c | 9 drivers/staging/lustre/lustre/obdclass/genops.c | 34 drivers/staging/lustre/lustre/obdclass/lu_object.c | 564 ++--- .../staging/lustre/lustre/obdclass/obd_config.c | 161 +- .../staging/lustre/lustre/obdecho/echo_client.c | 8 drivers/staging/lustre/lustre/osc/osc_internal.h | 5 drivers/staging/lustre/lustre/osc/osc_quota.c | 136 - drivers/staging/lustre/lustre/osc/osc_request.c | 12 drivers/staging/lustre/lustre/ptlrpc/connection.c | 164 +- 29 files changed, 870 insertions(+), 4208 deletions(-) delete mode 100644 drivers/staging/lustre/include/linux/libcfs/libcfs_hash.h delete mode 100644 drivers/staging/lustre/lnet/libcfs/hash.c -- Signature
WARNING: multiple messages have this Message-ID (diff)
From: NeilBrown <neilb@suse.com> To: Oleg Drokin <oleg.drokin@intel.com>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, James Simmons <jsimmons@infradead.org>, Andreas Dilger <andreas.dilger@intel.com> Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Lustre Development List <lustre-devel@lists.lustre.org> Subject: [lustre-devel] [PATCH 00/20] staging: lustre: convert to rhashtable Date: Thu, 12 Apr 2018 07:54:48 +1000 [thread overview] Message-ID: <152348312863.12394.11915752362061083241.stgit@noble> (raw) libcfs in lustre has a resizeable hashtable. Linux already has a resizeable hashtable, rhashtable, which is better is most metrics. See https://lwn.net/Articles/751374/ in a few days for an introduction to rhashtable. This series converts lustre to use rhashtable. This affects several different tables, and each is different is various ways. There are two outstanding issues. One is that a bug in rhashtable means that we cannot enable auto-shrinking in one of the tables. That is documented as appropriate and should be fixed soon. The other is that rhashtable has an atomic_t which counts the elements in a hash table. At least one table in lustre went to some trouble to avoid any table-wide atomics, so that could lead to a regression. I'm hoping that rhashtable can be enhanced with the option of a per-cpu counter, or similar. I have enabled automatic shrinking on all tables where it makes sense and doesn't trigger the bug. I have also removed all hints concerning min/max size - I cannot see how these could be useful. The dump_pgcache debugfs file provided some interesting challenges. I think I have cleaned it up enough so that it all makes sense. An extra pair of eyes examining that code in particular would be appreciated. This series passes all the same tests that pass before the patches are applied. Thanks, NeilBrown --- NeilBrown (20): staging: lustre: ptlrpc: convert conn_hash to rhashtable staging: lustre: convert lov_pool to use rhashtable staging: lustre: convert obd uuid hash to rhashtable staging: lustre: convert osc_quota hash to rhashtable staging: lustre: separate buckets from ldlm hash table staging: lustre: ldlm: add a counter to the per-namespace data staging: lustre: ldlm: store name directly in namespace. staging: lustre: simplify ldlm_ns_hash_defs[] staging: lustre: convert ldlm_resource hash to rhashtable. staging: lustre: make struct lu_site_bkt_data private staging: lustre: lu_object: discard extra lru count. staging: lustre: lu_object: factor out extra per-bucket data staging: lustre: lu_object: move retry logic inside htable_lookup staging: lustre: fold lu_object_new() into lu_object_find_at() staging: lustre: llite: use more private data in dump_pgcache staging: lustre: llite: remove redundant lookup in dump_pgcache staging: lustre: use call_rcu() to free lu_object_headers staging: lustre: change how "dump_page_cache" walks a hash table staging: lustre: convert lu_object cache to rhashtable staging: lustre: remove cfs_hash resizeable hashtable implementation. .../staging/lustre/include/linux/libcfs/libcfs.h | 1 .../lustre/include/linux/libcfs/libcfs_hash.h | 866 -------- drivers/staging/lustre/lnet/libcfs/Makefile | 2 drivers/staging/lustre/lnet/libcfs/hash.c | 2064 -------------------- drivers/staging/lustre/lnet/libcfs/module.c | 12 drivers/staging/lustre/lustre/include/lu_object.h | 55 - drivers/staging/lustre/lustre/include/lustre_dlm.h | 19 .../staging/lustre/lustre/include/lustre_export.h | 2 drivers/staging/lustre/lustre/include/lustre_net.h | 4 drivers/staging/lustre/lustre/include/obd.h | 11 .../staging/lustre/lustre/include/obd_support.h | 9 drivers/staging/lustre/lustre/ldlm/ldlm_request.c | 31 drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 370 +--- drivers/staging/lustre/lustre/llite/lcommon_cl.c | 8 drivers/staging/lustre/lustre/llite/vvp_dev.c | 332 +-- drivers/staging/lustre/lustre/llite/vvp_object.c | 9 drivers/staging/lustre/lustre/lov/lov_internal.h | 11 drivers/staging/lustre/lustre/lov/lov_obd.c | 12 drivers/staging/lustre/lustre/lov/lov_object.c | 8 drivers/staging/lustre/lustre/lov/lov_pool.c | 159 +- drivers/staging/lustre/lustre/lov/lovsub_object.c | 9 drivers/staging/lustre/lustre/obdclass/genops.c | 34 drivers/staging/lustre/lustre/obdclass/lu_object.c | 564 ++--- .../staging/lustre/lustre/obdclass/obd_config.c | 161 +- .../staging/lustre/lustre/obdecho/echo_client.c | 8 drivers/staging/lustre/lustre/osc/osc_internal.h | 5 drivers/staging/lustre/lustre/osc/osc_quota.c | 136 - drivers/staging/lustre/lustre/osc/osc_request.c | 12 drivers/staging/lustre/lustre/ptlrpc/connection.c | 164 +- 29 files changed, 870 insertions(+), 4208 deletions(-) delete mode 100644 drivers/staging/lustre/include/linux/libcfs/libcfs_hash.h delete mode 100644 drivers/staging/lustre/lnet/libcfs/hash.c -- Signature
next reply other threads:[~2018-04-11 21:54 UTC|newest] Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-04-11 21:54 NeilBrown [this message] 2018-04-11 21:54 ` [lustre-devel] [PATCH 00/20] staging: lustre: convert to rhashtable NeilBrown 2018-04-11 21:54 ` [PATCH 03/20] staging: lustre: convert obd uuid hash " NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 04/20] staging: lustre: convert osc_quota " NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 05/20] staging: lustre: separate buckets from ldlm hash table NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 09/20] staging: lustre: convert ldlm_resource hash to rhashtable NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 07/20] staging: lustre: ldlm: store name directly in namespace NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 10/20] staging: lustre: make struct lu_site_bkt_data private NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 02/20] staging: lustre: convert lov_pool to use rhashtable NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 12/20] staging: lustre: lu_object: factor out extra per-bucket data NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 08/20] staging: lustre: simplify ldlm_ns_hash_defs[] NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 01/20] staging: lustre: ptlrpc: convert conn_hash to rhashtable NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 06/20] staging: lustre: ldlm: add a counter to the per-namespace data NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 11/20] staging: lustre: lu_object: discard extra lru count NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 17/20] staging: lustre: use call_rcu() to free lu_object_headers NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 15/20] staging: lustre: llite: use more private data in dump_pgcache NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 16/20] staging: lustre: llite: remove redundant lookup " NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 14/20] staging: lustre: fold lu_object_new() into lu_object_find_at() NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 13/20] staging: lustre: lu_object: move retry logic inside htable_lookup NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 18/20] staging: lustre: change how "dump_page_cache" walks a hash table NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 20/20] staging: lustre: remove cfs_hash resizeable hashtable implementation NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-11 21:54 ` [PATCH 19/20] staging: lustre: convert lu_object cache to rhashtable NeilBrown 2018-04-11 21:54 ` [lustre-devel] " NeilBrown 2018-04-17 3:35 ` [PATCH 00/20] staging: lustre: convert " James Simmons 2018-04-17 3:35 ` [lustre-devel] " James Simmons 2018-04-18 3:17 ` NeilBrown 2018-04-18 3:17 ` [lustre-devel] " NeilBrown 2018-04-18 21:56 ` Simmons, James A. 2018-04-18 21:56 ` Simmons, James A. 2018-04-23 13:08 ` Greg Kroah-Hartman 2018-04-23 13:08 ` [lustre-devel] " Greg Kroah-Hartman
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=152348312863.12394.11915752362061083241.stgit@noble \ --to=neilb@suse.com \ --cc=andreas.dilger@intel.com \ --cc=gregkh@linuxfoundation.org \ --cc=jsimmons@infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=lustre-devel@lists.lustre.org \ --cc=oleg.drokin@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.