From: Tejun Heo <tj@kernel.org>
To: akpm@linux-foundation.org
Cc: linux-kernel@vger.kernel.org, rusty@rustcorp.com.au,
bfields@fieldses.org, skinsbursky@parallels.com,
ebiederm@xmission.com, jmorris@namei.org, axboe@kernel.dk
Subject: [PATCHSET] idr: implement idr_alloc() and convert existing users
Date: Sat, 2 Feb 2013 17:20:01 -0800 [thread overview]
Message-ID: <1359854463-2538-1-git-send-email-tj@kernel.org> (raw)
Hello,
* Andrew, I think this one is best routed through -mm together with
the previous series. Please read on.
* Bruce, I couldn't convert nfsd. Can you please help? More on it
later.
* Stanislav, Eric, James, can you please take a look at ipc/util.c
conversion? I moved allocation inside ipc_addid(). I *think* it's
correct but I'm not too familiar with the code.
* Jens, 0006 is fix for block layer regarding devt allocation. Given
how long this has been broken unnoticed and how late we're in the
release cycle, I think it would be better to route the fix with the
rest and let it get backported through -stable later on.
Currently, idr interface is a pain in the ass to use. It has a
mandatory pre-alloc mechanism which doesn't even guarantee the
following allocation wouldn't fail from memory shortage requiring its
users to loop on -EAGAIN. Also, there's no way to specify upper
limit. Combined, a user who wants allocate an ID with upper limit
ends up doing something like the following.
do {
if (!idr_pre_get(idr, GFP_KERNEL))
return -ENOMEM;
spin_lock(lock);
ret = idr_get_new_above(idr, ptr, lower_limit, &id);
if (!ret && id < upper_limit) {
idr_remove(idr, id);
ret = -ENOSPC;
}
spin_unlock(lock);
} while (ret == -EAGAIN);
if (ret)
return ret;
which is just crazy. Preloading is also very inefficient - each idr
ends up holding MAX_IDR_FREE idr_layers. Even on a fairly small
machine (my x230) w/o anything too heavy going on, it ends up
allocating over 1200 idr_layers with three quarters of them unused.
This also makes it difficult to make each layer larger to reduce the
depth of idr trees.
This patchset implements a new set of allocation interface for idr.
void idr_preload(gfp_t gfp_mask);
void idr_preload_end(void);
int idr_alloc(struct idr *idr, void *ptr, int start, int end, gfp_t gfp_mask);
idr_alloc() can be used w/o preloading if permissive enough @gfp_mask
can be used directly. If not, preloading, which uses per-cpu layer
buffer, can be used. Note that idr_preload() doesn't fail. It simply
guarnatees that the following idr_alloc() would behave as if it were
called with @gfp_mask specified at preloading time. The above example
with the old interface would become the following using the new
interface.
idr_preload(GFP_KERNEL);
spin_lock(lock);
id = idr_alloc(idr, ptr, lower_limit, upper_limit, GFP_NOWAIT);
spin_unlock(lock);
idr_preload_end();
if (id < 0)
return id;
Which is simpler and less error-prone. Also, there are many cases
where preloading isn't necessary (e.g. the section is protected by
outer mutex). For those, idr_alloc() can be used by itself.
id = idr_alloc(idr, ptr, lower_limit, upper_limit, GFP_KERNEL);
if (id < 0)
return id;
I converted all in-kernel users except nfsd and staging drivers. nfsd
splits preloading and actual id allocation in a way that per-cpu
preloading can't be used. I couldn't follow the control flow to
verify whether the current code is correct either. I think the best
way would be allocating ID upfront without installing the handle and
then later using idr_replace() to install the pointer when the ID
actually gets used. Bruce, would something like that be possible?
This patchset contains the following 62 patches.
0001-idr-cosmetic-updates-to-struct-initializer-definitio.patch
0002-idr-relocate-idr_for_each_entry-and-reorganize-id-r-.patch
0003-idr-remove-_idr_rc_to_errno-hack.patch
0004-idr-refactor-idr_get_new_above.patch
0005-idr-implement-idr_preload-_end-and-idr_alloc.patch
0006-block-fix-synchronization-and-limit-check-in-blk_all.patch
0007-block-convert-to-idr_alloc.patch
0008-block-loop-convert-to-idr_alloc.patch
0009-atm-nicstar-convert-to-idr_alloc.patch
0010-drbd-convert-to-idr_alloc.patch
0011-dca-convert-to-idr_alloc.patch
0012-dmaengine-convert-to-idr_alloc.patch
0013-firewire-convert-to-idr_alloc.patch
0014-gpio-convert-to-idr_alloc.patch
0015-drm-convert-to-idr_alloc.patch
0016-drm-exynos-convert-to-idr_alloc.patch
0017-drm-i915-convert-to-idr_alloc.patch
0018-drm-sis-convert-to-idr_alloc.patch
0019-drm-via-convert-to-idr_alloc.patch
0020-drm-vmwgfx-convert-to-idr_alloc.patch
0021-i2c-convert-to-idr_alloc.patch
0022-infiniband-core-convert-to-idr_alloc.patch
0023-infiniband-amso1100-convert-to-idr_alloc.patch
0024-infiniband-cxgb3-convert-to-idr_alloc.patch
0025-infiniband-cxgb4-convert-to-idr_alloc.patch
0026-infiniband-ehca-convert-to-idr_alloc.patch
0027-infiniband-ipath-convert-to-idr_alloc.patch
0028-infiniband-mlx4-convert-to-idr_alloc.patch
0029-infiniband-ocrdma-convert-to-idr_alloc.patch
0030-infiniband-qib-convert-to-idr_alloc.patch
0031-dm-convert-to-idr_alloc.patch
0032-memstick-convert-to-idr_alloc.patch
0033-mfd-convert-to-idr_alloc.patch
0034-misc-c2port-convert-to-idr_alloc.patch
0035-misc-tifm_core-convert-to-idr_alloc.patch
0036-mmc-convert-to-idr_alloc.patch
0037-mtd-convert-to-idr_alloc.patch
0038-i2c-convert-to-idr_alloc.patch
0039-ppp-convert-to-idr_alloc.patch
0040-power-convert-to-idr_alloc.patch
0041-pps-convert-to-idr_alloc.patch
0042-remoteproc-convert-to-idr_alloc.patch
0043-rpmsg-convert-to-idr_alloc.patch
0044-scsi-bfa-convert-to-idr_alloc.patch
0045-scsi-convert-to-idr_alloc.patch
0046-target-iscsi-convert-to-idr_alloc.patch
0047-scsi-lpfc-convert-to-idr_alloc.patch
0048-thermal-convert-to-idr_alloc.patch
0049-uio-convert-to-idr_alloc.patch
0050-vfio-convert-to-idr_alloc.patch
0051-dlm-convert-to-idr_alloc.patch
0052-inotify-convert-to-idr_alloc.patch
0053-ocfs2-convert-to-idr_alloc.patch
0054-ipc-convert-to-idr_alloc.patch
0055-cgroup-convert-to-idr_alloc.patch
0056-events-convert-to-idr_alloc.patch
0057-posix-timers-convert-to-idr_alloc.patch
0058-net-9p-convert-to-idr_alloc.patch
0059-mac80211-convert-to-idr_alloc.patch
0060-sctp-convert-to-idr_alloc.patch
0061-nfs4client-convert-to-idr_alloc.patch
0062-idr-deprecate-idr_pre_get-and-idr_get_new-_above.patch
0001-0005 implement the new idr interface. 0006 is idr related block
fix. 0007-0061 convert in-kernel users to idr_alloc(). 0062 marks
the old interface deprecated (note that this makes nfsd compilation
generates warning).
While I tried to be careful and reviewed the whole series a couple
times, my test coverage is unavoidably quite limited. I don't think
fallouts would be too difficult to spot during testing.
This patchset is on top of
[1] [PATCH] idr: fix a subtle bug in idr_get_next()
+ [2] [PATCHSET] idr: deprecate idr_remove_all()
and available in the following git branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git convert-to-idr_alloc
Excluding idr.[hc], which should become smaller once we remove the old
interface, this conversion sheds 482 lines across the kernel.
diffstat follows.
block/bsg.c | 26 --
block/genhd.c | 22 --
drivers/atm/nicstar.c | 27 +-
drivers/block/drbd/drbd_main.c | 29 +-
drivers/block/loop.c | 23 --
drivers/dca/dca-sysfs.c | 23 --
drivers/dma/dmaengine.c | 16 -
drivers/firewire/core-cdev.c | 16 -
drivers/firewire/core-device.c | 5
drivers/gpio/gpiolib.c | 11 -
drivers/gpu/drm/drm_context.c | 17 -
drivers/gpu/drm/drm_crtc.c | 15 -
drivers/gpu/drm/drm_gem.c | 35 +--
drivers/gpu/drm/drm_stub.c | 19 -
drivers/gpu/drm/exynos/exynos_drm_ipp.c | 16 -
drivers/gpu/drm/i915/i915_gem_context.c | 21 --
drivers/gpu/drm/sis/sis_mm.c | 13 -
drivers/gpu/drm/via/via_mm.c | 13 -
drivers/gpu/drm/vmwgfx/vmwgfx_resource.c | 17 -
drivers/i2c/i2c-core.c | 45 ----
drivers/infiniband/core/cm.c | 22 +-
drivers/infiniband/core/cma.c | 24 --
drivers/infiniband/core/sa_query.c | 15 -
drivers/infiniband/core/ucm.c | 16 -
drivers/infiniband/core/ucma.c | 32 ---
drivers/infiniband/core/uverbs_cmd.c | 17 -
drivers/infiniband/hw/amso1100/c2_qp.c | 19 +
drivers/infiniband/hw/cxgb3/iwch.h | 24 +-
drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 27 +-
drivers/infiniband/hw/ehca/ehca_cq.c | 27 --
drivers/infiniband/hw/ehca/ehca_qp.c | 34 +--
drivers/infiniband/hw/ipath/ipath_driver.c | 16 -
drivers/infiniband/hw/mlx4/cm.c | 32 +--
drivers/infiniband/hw/ocrdma/ocrdma_main.c | 14 -
drivers/infiniband/hw/qib/qib_init.c | 21 --
drivers/md/dm.c | 54 +----
drivers/memstick/core/memstick.c | 21 --
drivers/memstick/core/mspro_block.c | 17 -
drivers/mfd/rtsx_pcr.c | 13 -
drivers/misc/c2port/core.c | 22 --
drivers/misc/tifm_core.c | 11 -
drivers/mmc/core/host.c | 11 -
drivers/mtd/mtdcore.c | 9
drivers/net/macvtap.c | 21 --
drivers/net/ppp/ppp_generic.c | 33 ---
drivers/power/bq2415x_charger.c | 11 -
drivers/power/bq27x00_battery.c | 9
drivers/power/ds2782_battery.c | 9
drivers/pps/kapi.c | 2
drivers/pps/pps.c | 36 +--
drivers/remoteproc/remoteproc_core.c | 10
drivers/rpmsg/virtio_rpmsg_bus.c | 30 +-
drivers/scsi/bfa/bfad_im.c | 15 -
drivers/scsi/ch.c | 21 --
drivers/scsi/lpfc/lpfc_init.c | 12 -
drivers/scsi/sg.c | 43 +---
drivers/scsi/st.c | 27 --
drivers/target/iscsi/iscsi_target.c | 15 -
drivers/target/iscsi/iscsi_target_login.c | 15 -
drivers/thermal/cpu_cooling.c | 17 -
drivers/thermal/thermal_sys.c | 17 -
drivers/uio/uio.c | 19 -
drivers/vfio/vfio.c | 18 -
fs/dlm/lock.c | 18 -
fs/dlm/recover.c | 27 +-
fs/nfs/nfs4client.c | 13 -
fs/notify/inotify/inotify_user.c | 24 +-
fs/ocfs2/cluster/tcp.c | 32 +--
include/linux/idr.h | 135 +++++++++----
ipc/util.c | 30 --
kernel/cgroup.c | 27 --
kernel/events/core.c | 10
kernel/posix-timers.c | 18 -
lib/idr.c | 291 ++++++++++++++++++-----------
net/9p/util.c | 17 -
net/mac80211/main.c | 2
net/mac80211/tx.c | 18 -
net/sctp/associola.c | 27 +-
78 files changed, 816 insertions(+), 1160 deletions(-)
--
tejun
[1] https://lkml.org/lkml/2013/2/2/145
[2] https://lkml.org/lkml/2013/1/25/759
next reply other threads:[~2013-02-03 1:21 UTC|newest]
Thread overview: 128+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-02-03 1:20 Tejun Heo [this message]
2013-02-03 1:20 ` [PATCH 01/62] idr: cosmetic updates to struct / initializer definitions Tejun Heo
2013-02-03 1:20 ` [PATCH 02/62] idr: relocate idr_for_each_entry() and reorganize id[r|a]_get_new() Tejun Heo
2013-02-03 1:20 ` [PATCH 03/62] idr: remove _idr_rc_to_errno() hack Tejun Heo
2013-02-03 1:20 ` [PATCH 04/62] idr: refactor idr_get_new_above() Tejun Heo
2013-02-03 1:20 ` [PATCH 05/62] idr: implement idr_preload[_end]() and idr_alloc() Tejun Heo
2013-02-04 18:32 ` [PATCH v2 " Tejun Heo
2013-02-03 1:20 ` [PATCH 06/62] block: fix synchronization and limit check in blk_alloc_devt() Tejun Heo
2013-02-06 11:00 ` Jens Axboe
2013-02-03 1:20 ` [PATCH 07/62] block: convert to idr_alloc() Tejun Heo
2013-02-04 13:10 ` Jens Axboe
2013-02-03 1:20 ` [PATCH 08/62] block/loop: " Tejun Heo
2013-02-04 13:11 ` Jens Axboe
2013-02-03 1:20 ` [PATCH 09/62] atm/nicstar: " Tejun Heo
2013-02-04 14:04 ` chas williams - CONTRACTOR
2013-02-04 17:06 ` Tejun Heo
2013-02-04 18:06 ` chas williams - CONTRACTOR
2013-02-04 18:37 ` [PATCH v3 " Tejun Heo
2013-02-03 1:20 ` [PATCH 10/62] drbd: " Tejun Heo
2013-02-03 1:20 ` [PATCH 11/62] dca: " Tejun Heo
2013-02-03 1:20 ` [PATCH 12/62] dmaengine: " Tejun Heo
2013-02-03 1:20 ` [PATCH 13/62] firewire: " Tejun Heo
2013-02-03 11:03 ` Stefan Richter
2013-02-03 11:18 ` Stefan Richter
2013-02-04 16:57 ` [PATCH 12.5/62] firewire: add minor number range check to fw_device_init() Tejun Heo
2013-02-04 18:15 ` Stefan Richter
2013-02-04 16:58 ` [PATCH v2 13/62] firewire: convert to idr_alloc() Tejun Heo
2013-02-04 18:16 ` Stefan Richter
2013-02-03 1:20 ` [PATCH 14/62] gpio: " Tejun Heo
2013-02-04 20:40 ` Linus Walleij
2013-02-03 1:20 ` [PATCH 15/62] drm: " Tejun Heo
2013-02-03 1:20 ` [PATCH 16/62] drm/exynos: " Tejun Heo
2013-02-03 1:20 ` [PATCH 17/62] drm/i915: " Tejun Heo
2013-02-04 14:53 ` Daniel Vetter
2013-02-03 1:20 ` [PATCH 18/62] drm/sis: " Tejun Heo
2013-02-03 1:20 ` [PATCH 19/62] drm/via: " Tejun Heo
2013-02-03 1:20 ` [PATCH 20/62] drm/vmwgfx: " Tejun Heo
2013-02-03 1:20 ` [PATCH 21/62] i2c: " Tejun Heo
2013-02-03 1:20 ` [PATCH 22/62] infiniband/core: " Tejun Heo
2013-02-04 16:43 ` [PATCH v2 " Tejun Heo
2013-02-05 0:07 ` Hefty, Sean
2013-02-03 1:20 ` [PATCH 23/62] infiniband/amso1100: " Tejun Heo
2013-02-03 14:36 ` Steve Wise
2013-02-03 1:20 ` [PATCH 24/62] infiniband/cxgb3: " Tejun Heo
2013-02-03 14:37 ` Steve Wise
2013-02-03 1:20 ` [PATCH 25/62] infiniband/cxgb4: " Tejun Heo
2013-02-03 14:18 ` Steve Wise
2013-02-03 14:28 ` Tejun Heo
2013-02-04 15:32 ` Steve Wise
2013-02-03 1:20 ` [PATCH 26/62] infiniband/ehca: " Tejun Heo
2013-02-03 1:20 ` [PATCH 27/62] infiniband/ipath: " Tejun Heo
2013-02-04 16:15 ` Marciniszyn, Mike
2013-02-04 16:18 ` Tejun Heo
2013-02-03 1:20 ` [PATCH 28/62] infiniband/mlx4: " Tejun Heo
2013-02-03 1:20 ` [PATCH 29/62] infiniband/ocrdma: " Tejun Heo
2013-02-03 1:20 ` [PATCH 30/62] infiniband/qib: " Tejun Heo
2013-02-03 1:20 ` [PATCH 31/62] dm: " Tejun Heo
2013-02-03 1:20 ` [PATCH 32/62] memstick: " Tejun Heo
2013-02-03 1:20 ` [PATCH 33/62] mfd: " Tejun Heo
2013-02-03 22:32 ` Samuel Ortiz
2013-02-03 1:20 ` [PATCH 34/62] misc/c2port: " Tejun Heo
2013-02-03 1:20 ` [PATCH 35/62] misc/tifm_core: " Tejun Heo
2013-02-03 1:20 ` [PATCH 36/62] mmc: " Tejun Heo
2013-02-03 1:20 ` [PATCH 37/62] mtd: " Tejun Heo
2013-02-04 13:09 ` Ezequiel Garcia
2013-02-03 1:20 ` [PATCH 38/62] i2c: " Tejun Heo
2013-02-03 1:25 ` [PATCH UPDATED 38/62] macvtap: " Tejun Heo
2013-02-03 1:20 ` [PATCH 39/62] ppp: " Tejun Heo
2013-02-03 1:20 ` [PATCH 40/62] power: " Tejun Heo
2013-02-03 3:46 ` Anton Vorontsov
2013-02-03 1:20 ` [PATCH 41/62] pps: " Tejun Heo
2013-02-03 1:20 ` [PATCH 42/62] remoteproc: " Tejun Heo
2013-02-03 1:20 ` [PATCH 43/62] rpmsg: " Tejun Heo
2013-02-03 1:20 ` [PATCH 44/62] scsi/bfa: " Tejun Heo
2013-02-03 1:20 ` [PATCH 45/62] scsi: " Tejun Heo
2013-02-03 1:20 ` [PATCH 46/62] target/iscsi: " Tejun Heo
2013-02-03 1:20 ` [PATCH 47/62] scsi/lpfc: " Tejun Heo
2013-02-11 22:46 ` James Smart
2013-02-03 1:20 ` [PATCH 48/62] thermal: " Tejun Heo
2013-02-03 1:20 ` [PATCH 49/62] uio: " Tejun Heo
2013-02-03 7:05 ` Greg Kroah-Hartman
2013-02-03 1:20 ` [PATCH 50/62] vfio: " Tejun Heo
2013-02-04 16:06 ` Alex Williamson
2013-02-04 16:48 ` [PATCH v2 " Tejun Heo
2013-02-03 1:20 ` [PATCH 51/62] dlm: " Tejun Heo
2013-02-03 1:20 ` [PATCH 52/62] inotify: " Tejun Heo
2013-02-03 1:20 ` [PATCH 53/62] ocfs2: " Tejun Heo
2013-02-03 1:20 ` [PATCH 54/62] ipc: " Tejun Heo
2013-02-03 1:20 ` [PATCH 55/62] cgroup: " Tejun Heo
2013-02-04 2:49 ` Li Zefan
2013-02-03 1:20 ` [PATCH 56/62] events: " Tejun Heo
2013-02-03 1:20 ` [PATCH 57/62] posix-timers: " Tejun Heo
2013-02-03 1:20 ` [PATCH 58/62] net/9p: " Tejun Heo
2013-02-03 1:21 ` [PATCH 59/62] mac80211: " Tejun Heo
2013-02-04 17:40 ` Johannes Berg
2013-02-04 17:42 ` Tejun Heo
2013-02-03 1:21 ` [PATCH 60/62] sctp: " Tejun Heo
2013-02-03 16:55 ` Neil Horman
2013-02-04 16:22 ` Vlad Yasevich
2013-02-04 16:37 ` Tejun Heo
2013-02-04 16:42 ` [PATCH v2 " Tejun Heo
2013-02-05 15:22 ` Neil Horman
2013-02-03 1:21 ` [PATCH 61/62] nfs4client: " Tejun Heo
2013-02-03 1:21 ` [PATCH 62/62] idr: deprecate idr_pre_get() and idr_get_new[_above]() Tejun Heo
2013-02-03 13:41 ` [PATCHSET] idr: implement idr_alloc() and convert existing users Eric W. Biederman
2013-02-03 14:13 ` Tejun Heo
2013-02-03 15:24 ` Eric W. Biederman
2013-02-03 15:47 ` Tejun Heo
2013-02-03 17:02 ` J. Bruce Fields
2013-02-04 0:15 ` J. Bruce Fields
2013-02-04 17:10 ` Tejun Heo
2013-02-04 17:11 ` Tejun Heo
2013-02-04 18:15 ` J. Bruce Fields
2013-03-21 14:06 ` J. Bruce Fields
2013-03-21 18:35 ` Tejun Heo
2013-03-26 15:19 ` Jeff Layton
2013-03-26 15:26 ` Tejun Heo
2013-03-26 16:30 ` J. Bruce Fields
2013-03-26 16:33 ` Tejun Heo
2013-03-26 16:36 ` Tejun Heo
2013-03-26 16:52 ` Jeff Layton
2013-02-04 17:16 ` J. Bruce Fields
2013-02-04 19:17 ` Tejun Heo
2013-02-04 19:54 ` Marciniszyn, Mike
2013-02-04 21:42 ` Tejun Heo
2013-02-04 22:25 ` Marciniszyn, Mike
2013-02-04 20:08 ` Marciniszyn, Mike
2013-02-04 21:43 ` Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1359854463-2538-1-git-send-email-tj@kernel.org \
--to=tj@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=axboe@kernel.dk \
--cc=bfields@fieldses.org \
--cc=ebiederm@xmission.com \
--cc=jmorris@namei.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rusty@rustcorp.com.au \
--cc=skinsbursky@parallels.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).