All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ilya Dryomov <idryomov@gmail.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	stable@vger.kernel.org,
	Sergey Jerusalimov <wintchester@gmail.com>,
	Jeff Layton <jlayton@redhat.com>,
	linux-xfs@vger.kernel.org
Subject: Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations
Date: Wed, 29 Mar 2017 16:25:18 +0200	[thread overview]
Message-ID: <CAOi1vP_6HvHAGo4Neu=q_LY_m_NRmSRkkGsW=95xYctLUdag6A@mail.gmail.com> (raw)
In-Reply-To: <20170329111650.GI27994@dhcp22.suse.cz>

On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko <mhocko@kernel.org> wrote:
> On Wed 29-03-17 13:10:01, Ilya Dryomov wrote:
>> On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko <mhocko@kernel.org> wrote:
>> > On Wed 29-03-17 12:41:26, Michal Hocko wrote:
>> > [...]
>> >> > ceph_con_workfn
>> >> >   mutex_lock(&con->mutex)  # ceph_connection::mutex
>> >> >   try_write
>> >> >     ceph_tcp_connect
>> >> >       sock_create_kern
>> >> >         GFP_KERNEL allocation
>> >> >           allocator recurses into XFS, more I/O is issued
>> >
>> > One more note. So what happens if this is a GFP_NOIO request which
>> > cannot make any progress? Your IO thread is blocked on con->mutex
>> > as you write below but the above thread cannot proceed as well. So I am
>> > _really_ not sure this acutally helps.
>>
>> This is not the only I/O worker.  A ceph cluster typically consists of
>> at least a few OSDs and can be as large as thousands of OSDs.  This is
>> the reason we are calling sock_create_kern() on the writeback path in
>> the first place: pre-opening thousands of sockets isn't feasible.
>
> Sorry for being dense here but what actually guarantees the forward
> progress? My current understanding is that the deadlock is caused by
> con->mutext being held while the allocation cannot make a forward
> progress. I can imagine this would be possible if the other io flushers
> depend on this lock. But then NOIO vs. KERNEL allocation doesn't make
> much difference. What am I missing?

con->mutex is per-ceph_connection, osdc->request_mutex is global and is
the real problem here because we need both on the submit side, at least
in 3.18.  You are correct that even with GFP_NOIO this code may lock up
in theory, however I think it's very unlikely in practice.

We got rid of osdc->request_mutex in 4.7, so these workers are almost
independent in newer kernels and should be able to free up memory for
those blocked on GFP_NOIO retries with their respective con->mutex
held.  Using GFP_KERNEL and thus allowing the recursion is just asking
for an AA deadlock on con->mutex OTOH, so it does make a difference.

I'm a little confused by this discussion because for me this patch was
a no-brainer...  Locking aside, you said it was the stack trace in the
changelog that got your attention -- are you saying it's OK for a block
device to recurse back into the filesystem when doing I/O, potentially
generating more I/O?

Thanks,

                Ilya

  reply	other threads:[~2017-03-29 14:25 UTC|newest]

Thread overview: 106+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-03-28 12:29 [PATCH 4.4 00/76] 4.4.58-stable review Greg Kroah-Hartman
2017-03-28 12:29 ` [PATCH 4.4 01/76] net/openvswitch: Set the ipv6 source tunnel key address attribute correctly Greg Kroah-Hartman
2017-03-28 12:29 ` [PATCH 4.4 02/76] net: bcmgenet: Do not suspend PHY if Wake-on-LAN is enabled Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 03/76] net: properly release sk_frag.page Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 04/76] amd-xgbe: Fix jumbo MTU processing on newer hardware Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 05/76] net: unix: properly re-increment inflight counter of GC discarded candidates Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 06/76] net/mlx5: Increase number of max QPs in default profile Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 07/76] net/mlx5e: Count LRO packets correctly Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 08/76] net: bcmgenet: remove bcmgenet_internal_phy_setup() Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 09/76] ipv4: provide stronger user input validation in nl_fib_input() Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 10/76] socket, bpf: fix sk_filter use after free in sk_clone_lock Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 11/76] tcp: initialize icsk_ack.lrcvtime at session start time Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 12/76] Input: elan_i2c - add ASUS EeeBook X205TA special touchpad fw Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 13/76] Input: i8042 - add noloop quirk for Dell Embedded Box PC 3000 Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 14/76] Input: iforce - validate number of endpoints before using them Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 15/76] Input: ims-pcu " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 16/76] Input: hanwang " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 17/76] Input: yealink " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 18/76] Input: cm109 " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 19/76] Input: kbtab " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 20/76] Input: sur40 " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 21/76] ALSA: seq: Fix racy cell insertions during snd_seq_pool_done() Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 22/76] ALSA: ctxfi: Fix the incorrect check of dma_set_mask() call Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 23/76] ALSA: hda - Adding a group of pin definition to fix headset problem Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 24/76] USB: serial: option: add Quectel UC15, UC20, EC21, and EC25 modems Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 27/76] usb: gadget: f_uvc: Fix SuperSpeed companion descriptors wBytesPerInterval Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 28/76] usb-core: Add LINEAR_FRAME_INTR_BINTERVAL USB quirk Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 29/76] USB: uss720: fix NULL-deref at probe Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 30/76] USB: lvtest: " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 31/76] USB: idmouse: " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 32/76] USB: wusbcore: " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 33/76] usb: musb: cppi41: dont check early-TX-interrupt for Isoch transfer Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 34/76] usb: hub: Fix crash after failure to read BOS descriptor Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 35/76] uwb: i1480-dfu: fix NULL-deref at probe Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 36/76] uwb: hwa-rc: " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 37/76] mmc: ushc: " Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 38/76] iio: adc: ti_am335x_adc: fix fifo overrun recovery Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 39/76] iio: hid-sensor-trigger: Change get poll value function order to avoid sensor properties losing after resume from S3 Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 40/76] parport: fix attempt to write duplicate procfiles Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 41/76] ext4: mark inode dirty after converting inline directory Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 42/76] mmc: sdhci: Do not disable interrupts while waiting for clock Greg Kroah-Hartman
2017-04-04 16:50   ` Ben Hutchings
2017-04-06 12:12     ` Ludovic Desroches
2017-04-06 14:22       ` Ben Hutchings
2017-03-28 12:30 ` [PATCH 4.4 43/76] xen/acpi: upload PM state from init-domain to Xen Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 44/76] iommu/vt-d: Fix NULL pointer dereference in device_to_iommu Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 45/76] ARM: at91: pm: cpu_idle: switch DDR to power-down mode Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 46/76] ARM: dts: at91: sama5d2: add dma properties to UART nodes Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 47/76] cpufreq: Restore policy min/max limits on CPU online Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations Greg Kroah-Hartman
2017-03-28 12:43   ` Michal Hocko
2017-03-28 13:23     ` Ilya Dryomov
2017-03-28 13:30       ` Michal Hocko
2017-03-29  9:21         ` Ilya Dryomov
2017-03-29 10:41           ` Michal Hocko
2017-03-29 10:55             ` Michal Hocko
2017-03-29 11:10               ` Ilya Dryomov
2017-03-29 11:16                 ` Michal Hocko
2017-03-29 14:25                   ` Ilya Dryomov [this message]
2017-03-30  6:25                     ` Michal Hocko
2017-03-30 10:02                       ` Ilya Dryomov
2017-03-30 11:21                         ` Michal Hocko
2017-03-30 13:48                           ` Ilya Dryomov
2017-03-30 14:36                             ` Michal Hocko
2017-03-30 15:06                               ` Ilya Dryomov
2017-03-30 16:12                                 ` Michal Hocko
2017-03-30 17:19                                   ` Ilya Dryomov
2017-03-30 18:44                                     ` Michal Hocko
2017-03-30 13:53                       ` Ilya Dryomov
2017-03-30 13:59                         ` Michal Hocko
2017-03-29 11:05             ` Brian Foster
2017-03-29 11:14               ` Ilya Dryomov
2017-03-29 11:18                 ` Michal Hocko
2017-03-29 11:49                   ` Brian Foster
2017-03-29 14:30                     ` Ilya Dryomov
2017-03-28 12:30 ` [PATCH 4.4 49/76] raid10: increment write counter after bio is split Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 50/76] libceph: dont set weight to IN when OSD is destroyed Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 51/76] xfs: dont allow di_size with high bit set Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 52/76] xfs: fix up xfs_swap_extent_forks inline extent handling Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 53/76] nl80211: fix dumpit error path RTNL deadlocks Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 54/76] USB: usbtmc: add missing endpoint sanity check Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 55/76] xfs: clear _XBF_PAGES from buffers when readahead page Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 56/76] xen: do not re-use pirq number cached in pci device msi msg data Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 57/76] igb: Workaround for igb i210 firmware issue Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 58/76] igb: add i211 to i210 PHY workaround Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 59/76] x86/hyperv: Handle unknown NMIs on one CPU when unknown_nmi_panic Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 60/76] PCI: Separate VF BAR updates from standard BAR updates Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 61/76] PCI: Remove pci_resource_bar() and pci_iov_resource_bar() Greg Kroah-Hartman
2017-03-28 12:30 ` [PATCH 4.4 62/76] PCI: Add comments about ROM BAR updating Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 63/76] PCI: Decouple IORESOURCE_ROM_ENABLE and PCI_ROM_ADDRESS_ENABLE Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 64/76] PCI: Dont update VF BARs while VF memory space is enabled Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 65/76] PCI: Update BARs using property bits appropriate for type Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 66/76] PCI: Ignore BAR updates on virtual functions Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 67/76] PCI: Do any VF BAR updates before enabling the BARs Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 68/76] vfio/spapr: Postpone allocation of userspace version of TCE table Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 69/76] block: allow WRITE_SAME commands with the SG_IO ioctl Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 70/76] s390/zcrypt: Introduce CEX6 toleration Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 71/76] uvcvideo: uvc_scan_fallback() for webcams with broken chain Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 72/76] ACPI / blacklist: add _REV quirks for Dell Precision 5520 and 3520 Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 73/76] ACPI / blacklist: Make Dell Latitude 3350 ethernet work Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 74/76] serial: 8250_pci: Detach low-level driver during PCI error recovery Greg Kroah-Hartman
2017-04-04 20:26   ` Ben Hutchings
2017-03-28 12:31 ` [PATCH 4.4 75/76] fbcon: Fix vc attr at deinit Greg Kroah-Hartman
2017-03-28 12:31 ` [PATCH 4.4 76/76] crypto: algif_hash - avoid zero-sized array Greg Kroah-Hartman
2017-03-28 19:38 ` [PATCH 4.4 00/76] 4.4.58-stable review Shuah Khan
2017-03-29  2:58 ` Guenter Roeck

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAOi1vP_6HvHAGo4Neu=q_LY_m_NRmSRkkGsW=95xYctLUdag6A@mail.gmail.com' \
    --to=idryomov@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jlayton@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-xfs@vger.kernel.org \
    --cc=mhocko@kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=wintchester@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.