From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1A343C43334 for ; Fri, 3 Jun 2022 07:18:16 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-455-BrB8Ql_JPC6_9DcXraXa-A-1; Fri, 03 Jun 2022 03:18:11 -0400 X-MC-Unique: BrB8Ql_JPC6_9DcXraXa-A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F124C803B22; Fri, 3 Jun 2022 07:18:08 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1C35E2166B26; Fri, 3 Jun 2022 07:18:06 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id E6117194704A; Fri, 3 Jun 2022 07:18:05 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id DE65619466DF for ; Thu, 2 Jun 2022 11:04:42 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id B9EDB1731B; Thu, 2 Jun 2022 11:04:42 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast07.extmail.prod.ext.rdu2.redhat.com [10.11.55.23]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B55D58288A for ; Thu, 2 Jun 2022 11:04:42 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 955763C0D873 for ; Thu, 2 Jun 2022 11:04:42 +0000 (UTC) Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-625-aV9LzUk-OjOLJwQ30Fvyyw-1; Thu, 02 Jun 2022 07:04:40 -0400 X-MC-Unique: aV9LzUk-OjOLJwQ30Fvyyw-1 Received: by mail-qk1-f181.google.com with SMTP id o68so3341055qkf.13 for ; Thu, 02 Jun 2022 04:04:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ooS6qSi+E7j9qtF0cXby3AfocOFZxtMqCfiX07fRi1Q=; b=zcM03F/b6acOSO3zfCFeuEAipabNnurmcqwRLGkygC4/h6VawFt43od+w0e4W4M+gE yZlewq1Cry3b+uAOJyXAkSWP/9gX+x2FoqurRlJ9DZ8sp5G97ig82+N+DSJwzQWJlmjr vMvnZUyIBu11eGevqeEOxDX2UpLp0NlmL3y7a62zcYuhOfAQTwdR8biezoaJCE66No61 neSKf1VhDzarsN5X55SLFP6PVg3E9801s5/Bl9IhPaDIjGH9yV3uvPew4VPiB/ttqhmW VoyUFhBd/chomn3qYdXi0gLXsy1Nw1l7c1Mo4slL3BiOtDbo0UOq4j4olgxHC2CV4R9k 6/IQ== X-Gm-Message-State: AOAM530w94BcjaPVMB8vqrhXRIy5t7jS8+OU2yBVpiMpp9JbdvUgO7/O iJib6GXiHRGCQAKrzG+6gfFrDkriohX0xT7LRGDlUYZW X-Google-Smtp-Source: ABdhPJx1m38jZDbqnNt0zWFYp8HxtfgW73CIMY+1L1c/basIl3NOUrgjBkYYgQ8vAUI+ZYfQJPwlnrGxTuQsuf6URzE= X-Received: by 2002:a05:620a:2913:b0:6a0:750e:f938 with SMTP id m19-20020a05620a291300b006a0750ef938mr2959524qkp.200.1654167879793; Thu, 02 Jun 2022 04:04:39 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Roger Heflin Date: Thu, 2 Jun 2022 06:04:28 -0500 Message-ID: To: LVM general discussion and development X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Mailman-Approved-At: Fri, 03 Jun 2022 07:18:04 +0000 Subject: Re: [linux-lvm] lvm commands hanging when run from inside a kubernetes pod X-BeenThere: linux-lvm@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: LVM general discussion and development Errors-To: linux-lvm-bounces@redhat.com Sender: "linux-lvm" X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=linux-lvm-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/mixed; boundary="===============6758013024366020605==" --===============6758013024366020605== Content-Type: multipart/alternative; boundary="000000000000dfbe8705e074f86d" --000000000000dfbe8705e074f86d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Random thoughts. Make sure use_lvmetad is 0, and its systemd units for it are stopped/disabled. Are you mounting /proc and /sys and /dev into the /host chroot? /run may also be needed. you might add a "-ttt" to the strace command to give timing data. On Thu, Jun 2, 2022 at 1:41 AM Abhishek Agarwal < mragarwal.developer@gmail.com> wrote: > These are not different LVM processes. The container process is using the > LVM binary that the node itself has. We have achieved this by using scrip= ts > that point to the same lvm binary that is used by the node. > > Configmap(~shell script) used for the same has the following contents > where `/host` refers to the root directory of the node: > > get_bin_path: | > #!/bin/sh > bin_name=3D$1 > if [ -x /host/bin/which ]; then > echo $(chroot /host /bin/which $bin_name | cut -d ' ' -f 1) > elif [ -x /host/usr/bin/which ]; then > echo $(chroot /host /usr/bin/which $bin_name | cut -d ' ' -f 1) > else > $(chroot /host which $bin_name | cut -d ' ' -f 1) > fi > > lvcreate: | > #!/bin/sh > path=3D$(/sbin/lvm-eg/get_bin_path "lvcreate") > chroot /host $path "$@" > > Also, the above logs in the pastebin link have errors because the vg lock > has not been acquired and hence creation commands will fail. Once the loc= k > is acquired, the `strace -f` command gives the following output being > stuck. Check out this link for full details -> > https://pastebin.com/raw/DwQfdmr8 > > P.S: We at OpenEBS are trying to provide lvm storage to cloud native > workloads with the help of kubernetes CSI drivers and since all these > drivers run as pods and help dynamic provisioning of kubernetes > volumes(storage) for the application, the lvm commands needs to be run fr= om > inside the pod. Reference -> https://github.com/openebs/lvm-localpv > > Regards > > On Wed, 1 Jun 2022 at 13:06, Demi Marie Obenour < > demi@invisiblethingslab.com> wrote: > >> On Wed, Jun 01, 2022 at 12:20:32AM +0530, Abhishek Agarwal wrote: >> > Hi Roger. Thanks for your reply. I have rerun the command with `strace >> -f` >> > as you suggested. Here is the pastebin link containing the detailed >> output >> > of the command: https://pastebin.com/raw/VRuBbHBc >> >> Even if you can get LVM =E2=80=9Cworking=E2=80=9D, it is still likely to= cause data >> corruption at some point, as there is no guarantee that different LVM >> processes in different namespaces will see each others=E2=80=99 locks. >> >> Why do you need to run LVM in a container? What are you trying to >> accomplish? >> -- >> Sincerely, >> Demi Marie Obenour (she/her/hers) >> Invisible Things Lab >> _______________________________________________ >> linux-lvm mailing list >> linux-lvm@redhat.com >> https://listman.redhat.com/mailman/listinfo/linux-lvm >> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >> > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://listman.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > --000000000000dfbe8705e074f86d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Random thoughts.

Make sure= =C2=A0 use_lvmetad is 0, and its systemd units for it are stopped/disabled.=

Are you mounting /proc and /sys and /dev into the= /host chroot?

/run may also be needed.
=
you might add a "-ttt" to the strace command to gi= ve timing data.



On Thu, Jun 2, 2022 at 1= :41 AM Abhishek Agarwal <mragarwal.developer@gmail.com> wrote:
These are not different L= VM processes. The container process is using the LVM binary that the node i= tself has. We have achieved this by using scripts that point to the same lv= m binary that is used by the node.

Configmap(~shell scri= pt) used for the same has the following contents where `/host` refers to th= e root directory of the node:
get_bin_path<=
/span>: |
  #!/bin/sh
  bin_name=3D$1
  if [ -x /host/bin/which ];=
 then
    echo $(chroot /host /bin/which $bin_name | cut -d ' ' =
-f 1)
  elif [ -x /host/usr/bin/which ]; then
    echo $(chroot /host=
 /usr/bin/which $bin_name | cut -d ' ' -f 1)
  else
    $(chr=
oot /host which $bin_name | cut -d ' ' -f 1)
  fi
lvcreate: |
  #!/bin/sh
  path=3D$(/sb=
in/lvm-eg/get_bin_path "lvcreate")
  chroot /host $path "=
$@"
Also, the above logs in the pastebin link have err= ors because the vg lock has not been acquired and hence creation commands w= ill fail. Once the lock is acquired, the `strace -f` command gives the foll= owing=C2=A0output being stuck. Check out this link for full details ->= =C2=A0https= ://pastebin.com/raw/DwQfdmr8

P.S: We at OpenEB= S are trying to provide lvm storage to cloud native workloads with the help= of kubernetes CSI drivers and since all these drivers run as pods and help= dynamic provisioning of kubernetes volumes(storage) for the application, t= he lvm commands needs to be run from inside the pod. Reference ->=C2=A0<= a href=3D"https://github.com/openebs/lvm-localpv" target=3D"_blank">https:/= /github.com/openebs/lvm-localpv

Regards
<= /div>
O= n Wed, 1 Jun 2022 at 13:06, Demi Marie Obenour <demi@invisiblethingslab.com>= ; wrote:
On Wed,= Jun 01, 2022 at 12:20:32AM +0530, Abhishek Agarwal wrote:
> Hi Roger. Thanks for your reply. I have rerun the command with `strace= -f`
> as you suggested. Here is the pastebin link containing the detailed ou= tput
> of the command: https://pastebin.com/raw/VRuBbHBc

Even if you can get LVM =E2=80=9Cworking=E2=80=9D, it is still likely to ca= use data
corruption at some point, as there is no guarantee that different LVM
processes in different namespaces will see each others=E2=80=99 locks.

Why do you need to run LVM in a container?=C2=A0 What are you trying to
accomplish?
--
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.= com
https://listman.redhat.com/mailman/listinfo/lin= ux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.= com
https://listman.redhat.com/mailman/listinfo/lin= ux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
--000000000000dfbe8705e074f86d-- --===============6758013024366020605== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ --===============6758013024366020605==--