* [PATCH 0/1] V2: Replace retry logic in useradd with flock @ 2016-02-16 7:14 kai.kang 2016-02-16 7:14 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang 0 siblings, 1 reply; 18+ messages in thread From: kai.kang @ 2016-02-16 7:14 UTC (permalink / raw) To: ross.burton; +Cc: openembedded-core From: Kai Kang <kai.kang@windriver.com> V2: * Use $SYSROOT/etc as flock file The following changes since commit da13f0bce5d1b3c52f5716ae8d104372a820e5f9: buildhistory.bbclass: remove out-dated information on request (2016-02-15 17:47:08 +0000) are available in the git repository at: git://git.pokylinux.org/poky-contrib kangkai/flock http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/flock Kai Kang (1): useradd_base.bbclass: fix simultaneous with flock meta/classes/useradd.bbclass | 8 +- meta/classes/useradd_base.bbclass | 165 ++++++++++---------------------------- 2 files changed, 48 insertions(+), 125 deletions(-) -- 2.7.0 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-16 7:14 [PATCH 0/1] V2: Replace retry logic in useradd with flock kai.kang @ 2016-02-16 7:14 ` kai.kang 2016-02-16 12:09 ` Burton, Ross 0 siblings, 1 reply; 18+ messages in thread From: kai.kang @ 2016-02-16 7:14 UTC (permalink / raw) To: ross.burton; +Cc: openembedded-core From: Kai Kang <kai.kang@windriver.com> When perform useradd during populate sysroot, it locks files passwd.lock and group.lock at same time. And then it meets a dead lock issue randomly. Use flock to reslove it by using an universal lock file for all the user and group related operations. [YOCTO #9022] Signed-off-by: Kai Kang <kai.kang@windriver.com> --- meta/classes/useradd.bbclass | 8 +- meta/classes/useradd_base.bbclass | 165 ++++++++++---------------------------- 2 files changed, 48 insertions(+), 125 deletions(-) diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass index c960656..4503020 100644 --- a/meta/classes/useradd.bbclass +++ b/meta/classes/useradd.bbclass @@ -38,6 +38,8 @@ if test "x$D" != "x"; then export PSEUDO_PASSWD="$SYSROOT:${STAGING_DIR_NATIVE}" fi +FLOCK_FILE=$SYSROOT/etc + # If we're not doing a special SSTATE/SYSROOT install # then set the values, otherwise use the environment if test "x$UA_SYSROOT" = "x"; then @@ -57,7 +59,7 @@ if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1` remaining=`echo "$GROUPADD_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_groupadd "$SYSROOT" "$OPT $opts" 10 + perform_groupadd "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi @@ -73,7 +75,7 @@ if test "x`echo $USERADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$USERADD_PARAM" | cut -d ';' -f 1` remaining=`echo "$USERADD_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_useradd "$SYSROOT" "$OPT $opts" 10 + perform_useradd "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi @@ -89,7 +91,7 @@ if test "x`echo $GROUPMEMS_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 1` remaining=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_groupmems "$SYSROOT" "$OPT $opts" 10 + perform_groupmems "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi diff --git a/meta/classes/useradd_base.bbclass b/meta/classes/useradd_base.bbclass index ab3cd35..66a9848 100644 --- a/meta/classes/useradd_base.bbclass +++ b/meta/classes/useradd_base.bbclass @@ -4,7 +4,7 @@ # The following functions basically have similar logic. # *) Perform necessary checks before invoking the actual command -# *) Invoke the actual command, make retries if necessary +# *) Invoke the actual command with flock # *) Error out if an error occurs. # Note that before invoking these functions, make sure the global variable @@ -13,26 +13,16 @@ perform_groupadd () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupadd with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupadd with [$opts]" local groupname=`echo "$opts" | awk '{ print $NF }'` local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" if test "x$group_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO groupadd $opts || true - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" - if test "x$group_exists" = "x"; then - bbwarn "${PN}: groupadd command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupadd command $retries times without success, giving up" - fi - sleep $count - done + opts=`echo $opts | sed s/\'/\"/g` + eval flock -x -w 100 $FLOCK_FILE -c \'$PSEUDO groupadd $opts\' || true + group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" + if test "x$group_exists" = "x"; then + bbfatal "${PN}: groupadd command did not succeed." + fi else bbnote "${PN}: group $groupname already exists, not re-creating it" fi @@ -41,26 +31,16 @@ perform_groupadd () { perform_useradd () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing useradd with [$opts] and $retries times of retry" + bbnote "${PN}: Performing useradd with [$opts]" local username=`echo "$opts" | awk '{ print $NF }'` local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" if test "x$user_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO useradd $opts || true - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" - if test "x$user_exists" = "x"; then - bbwarn "${PN}: useradd command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running useradd command $retries times without success, giving up" - fi - sleep $count - done + opts=`echo $opts | sed s/\'/\"/g` + eval flock -x -w 100 $FLOCK_FILE -c \'$PSEUDO useradd $opts\' || true + user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" + if test "x$user_exists" = "x"; then + bbfatal "${PN}: useradd command did not succeed." + fi else bbnote "${PN}: user $username already exists, not re-creating it" fi @@ -69,8 +49,7 @@ perform_useradd () { perform_groupmems () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupmems with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupmems with [$opts]" local groupname=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-g" || $i == "--group") print $(i+1) }'` local username=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-a" || $i == "--add") print $(i+1) }'` bbnote "${PN}: Running groupmems command with group $groupname and user $username" @@ -84,25 +63,11 @@ perform_groupmems () { fi local mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" if test "x$mem_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO groupmems $opts || true - mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" - if test "x$mem_exists" = "x"; then - bbwarn "${PN}: groupmems command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - if test "x$gshadow" = "xno"; then - rm -f $rootdir${sysconfdir}/gshadow - rm -f $rootdir${sysconfdir}/gshadow- - fi - bbfatal "${PN}: Tried running groupmems command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $FLOCK_FILE -c \'$PSEUDO groupmems $opts\' || true + mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" + if test "x$mem_exists" = "x"; then + bbfatal "${PN}: groupmems command did not succeed." + fi else bbnote "${PN}: group $groupname already contains $username, not re-adding it" fi @@ -115,26 +80,15 @@ perform_groupmems () { perform_groupdel () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupdel with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupdel with [$opts]" local groupname=`echo "$opts" | awk '{ print $NF }'` local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" if test "x$group_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO groupdel $opts || true - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" - if test "x$group_exists" != "x"; then - bbwarn "${PN}: groupdel command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupdel command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $FLOCK_FILE -c \'$PSEUDO groupdel $opts\' || true + group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" + if test "x$group_exists" != "x"; then + bbfatal "${PN}: groupdel command did not succeed." + fi else bbnote "${PN}: group $groupname doesn't exist, not removing it" fi @@ -143,26 +97,15 @@ perform_groupdel () { perform_userdel () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing userdel with [$opts] and $retries times of retry" + bbnote "${PN}: Performing userdel with [$opts]" local username=`echo "$opts" | awk '{ print $NF }'` local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" if test "x$user_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO userdel $opts || true - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" - if test "x$user_exists" != "x"; then - bbwarn "${PN}: userdel command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running userdel command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $FLOCK_FILE -c \'$PSEUDO userdel $opts\' || true + user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" + if test "x$user_exists" != "x"; then + bbfatal "${PN}: userdel command did not succeed." + fi else bbnote "${PN}: user $username doesn't exist, not removing it" fi @@ -174,25 +117,14 @@ perform_groupmod () { set +e local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupmod with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupmod with [$opts]" local groupname=`echo "$opts" | awk '{ print $NF }'` local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" if test "x$group_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO groupmod $opts - if test $? != 0; then - bbwarn "${PN}: groupmod command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupmod command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $FLOCK_FILE -c \'$PSEUDO groupmod $opts\' + if test $? != 0; then + bbwarn "${PN}: groupmod command did not succeed." + fi else bbwarn "${PN}: group $groupname doesn't exist, unable to modify it" fi @@ -204,25 +136,14 @@ perform_usermod () { set +e local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing usermod with [$opts] and $retries times of retry" + bbnote "${PN}: Performing usermod with [$opts]" local username=`echo "$opts" | awk '{ print $NF }'` local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" if test "x$user_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO usermod $opts - if test $? != 0; then - bbwarn "${PN}: usermod command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running usermod command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $FLOCK_FILE -c \'$PSEUDO usermod $opts\' + if test $? != 0; then + bbfatal "${PN}: usermod command did not succeed." + fi else bbwarn "${PN}: user $username doesn't exist, unable to modify it" fi -- 2.7.0 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-16 7:14 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang @ 2016-02-16 12:09 ` Burton, Ross 2016-02-17 5:40 ` Kang Kai 0 siblings, 1 reply; 18+ messages in thread From: Burton, Ross @ 2016-02-16 12:09 UTC (permalink / raw) To: Kang Kai; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 621 bytes --] On 16 February 2016 at 07:14, <kai.kang@windriver.com> wrote: > +FLOCK_FILE=$SYSROOT/etc > useradd_base is also used by extrausers.bbclass, so it would be best if useradd_base decided where to use as a lock ($rootdir/${sysconfdir} sounds sensible) instead of expecting that the caller sets FLOCK_FILE and mysteriously failing if it wasn't set. Also if we're locking and waiting properly instead of using timeouts and loops, can we remove the "does it exist?" check before the operation (eg groupadd -f will exit 0 if the group exists) and after (with proper locking that shouldn't be happening). Ross [-- Attachment #2: Type: text/html, Size: 1109 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-16 12:09 ` Burton, Ross @ 2016-02-17 5:40 ` Kang Kai 2016-02-17 23:49 ` Burton, Ross 0 siblings, 1 reply; 18+ messages in thread From: Kang Kai @ 2016-02-17 5:40 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 985 bytes --] On 2016年02月16日 20:09, Burton, Ross wrote: > > On 16 February 2016 at 07:14, <kai.kang@windriver.com > <mailto:kai.kang@windriver.com>> wrote: > > +FLOCK_FILE=$SYSROOT/etc > > > useradd_base is also used by extrausers.bbclass, so it would be best > if useradd_base decided where to use as a lock ($rootdir/${sysconfdir} > sounds sensible) instead of expecting that the caller sets FLOCK_FILE > and mysteriously failing if it wasn't set. OK. Sounds good. > > Also if we're locking and waiting properly instead of using timeouts > and loops, can we remove the "does it exist?" check before the > operation (eg groupadd -f will exit 0 if the group exists) and after > (with proper locking that shouldn't be happening). I prefer to keep such check. There is more than one case that causes operation error such as pass wrong parameters. The messages are helpful for developers. Thanks, Kai > > Ross -- Regards, Neil | Kai Kang [-- Attachment #2: Type: text/html, Size: 2656 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-17 5:40 ` Kang Kai @ 2016-02-17 23:49 ` Burton, Ross 2016-02-22 8:41 ` Kang Kai 0 siblings, 1 reply; 18+ messages in thread From: Burton, Ross @ 2016-02-17 23:49 UTC (permalink / raw) To: Kang Kai; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 425 bytes --] On 17 February 2016 at 05:40, Kang Kai <Kai.Kang@windriver.com> wrote: > I prefer to keep such check. There is more than one case that causes > operation error such as pass wrong parameters. The messages are helpful > for developers. > In that case the groupadd will return an error code. What situations with proper locking will result in groupadd running successfully but the group not being added? Ross [-- Attachment #2: Type: text/html, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-17 23:49 ` Burton, Ross @ 2016-02-22 8:41 ` Kang Kai 0 siblings, 0 replies; 18+ messages in thread From: Kang Kai @ 2016-02-22 8:41 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 670 bytes --] On 2016年02月18日 07:49, Burton, Ross wrote: > > On 17 February 2016 at 05:40, Kang Kai <Kai.Kang@windriver.com > <mailto:Kai.Kang@windriver.com>> wrote: > > I prefer to keep such check. There is more than one case that > causes operation error such as pass wrong parameters. The messages > are helpful for developers. > > > In that case the groupadd will return an error code. What situations > with proper locking will result in groupadd running successfully but > the group not being added? Sorry for late reply. I think I understand your meanings. V3 will be sent. --Kai > > Ross -- Regards, Neil | Kai Kang [-- Attachment #2: Type: text/html, Size: 2065 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 0/1] V3: Replace retry logic in useradd with flock @ 2016-02-23 2:45 kai.kang 2016-02-23 2:45 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang 0 siblings, 1 reply; 18+ messages in thread From: kai.kang @ 2016-02-23 2:45 UTC (permalink / raw) To: ross.burton; +Cc: openembedded-core From: Kai Kang <kai.kang@windriver.com> V3: * remove checks before operations such as useradd/userdel etc. The following changes since commit ea8c34e976c11757ed9869c51b48f39050e25708: libnewt: Fix build with PIE flags (2016-02-21 09:32:43 +0000) are available in the git repository at: git://git.pokylinux.org/poky-contrib kangkai/flockv3 http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/flockv3 Kai Kang (1): useradd_base.bbclass: fix simultaneous with flock meta/classes/useradd.bbclass | 6 +- meta/classes/useradd_base.bbclass | 186 ++++++++------------------------------ 2 files changed, 40 insertions(+), 152 deletions(-) -- 2.6.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 2:45 [PATCH 0/1] V3: Replace retry logic in useradd " kai.kang @ 2016-02-23 2:45 ` kai.kang 2016-02-23 3:05 ` Khem Raj 2016-02-23 16:29 ` Burton, Ross 0 siblings, 2 replies; 18+ messages in thread From: kai.kang @ 2016-02-23 2:45 UTC (permalink / raw) To: ross.burton; +Cc: openembedded-core From: Kai Kang <kai.kang@windriver.com> When perform useradd during populate sysroot, it locks files passwd.lock and group.lock at same time. And then it meets a dead lock issue randomly. Use flock to reslove it by using an universal lock file for all the user and group related operations. [YOCTO #9022] Signed-off-by: Kai Kang <kai.kang@windriver.com> --- meta/classes/useradd.bbclass | 6 +- meta/classes/useradd_base.bbclass | 186 ++++++++------------------------------ 2 files changed, 40 insertions(+), 152 deletions(-) diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass index c960656..0a6f2be 100644 --- a/meta/classes/useradd.bbclass +++ b/meta/classes/useradd.bbclass @@ -57,7 +57,7 @@ if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1` remaining=`echo "$GROUPADD_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_groupadd "$SYSROOT" "$OPT $opts" 10 + perform_groupadd "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi @@ -73,7 +73,7 @@ if test "x`echo $USERADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$USERADD_PARAM" | cut -d ';' -f 1` remaining=`echo "$USERADD_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_useradd "$SYSROOT" "$OPT $opts" 10 + perform_useradd "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi @@ -89,7 +89,7 @@ if test "x`echo $GROUPMEMS_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 1` remaining=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_groupmems "$SYSROOT" "$OPT $opts" 10 + perform_groupmems "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi diff --git a/meta/classes/useradd_base.bbclass b/meta/classes/useradd_base.bbclass index ab3cd35..7c1cc13 100644 --- a/meta/classes/useradd_base.bbclass +++ b/meta/classes/useradd_base.bbclass @@ -4,7 +4,7 @@ # The following functions basically have similar logic. # *) Perform necessary checks before invoking the actual command -# *) Invoke the actual command, make retries if necessary +# *) Invoke the actual command with flock # *) Error out if an error occurs. # Note that before invoking these functions, make sure the global variable @@ -12,65 +12,32 @@ perform_groupadd () { local rootdir="$1" - local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupadd with [$opts] and $retries times of retry" + local opts=`echo $2 | sed s/\'/\"/g` + bbnote "${PN}: Performing groupadd with [$opts]" + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupadd $opts\' || true local groupname=`echo "$opts" | awk '{ print $NF }'` - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" + local group_exists=`grep "^$groupname:" $rootdir/etc/group` if test "x$group_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO groupadd $opts || true - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" - if test "x$group_exists" = "x"; then - bbwarn "${PN}: groupadd command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupadd command $retries times without success, giving up" - fi - sleep $count - done - else - bbnote "${PN}: group $groupname already exists, not re-creating it" + bbfatal "${PN}: groupadd command did not succeed." fi } perform_useradd () { local rootdir="$1" - local opts="$2" - local retries="$3" - bbnote "${PN}: Performing useradd with [$opts] and $retries times of retry" + local opts=`echo $2 | sed s/\'/\"/g` + bbnote "${PN}: Performing useradd with [$opts]" + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO useradd $opts\' || true local username=`echo "$opts" | awk '{ print $NF }'` - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" + local user_exists=`grep "^$username:" $rootdir/etc/passwd` if test "x$user_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO useradd $opts || true - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" - if test "x$user_exists" = "x"; then - bbwarn "${PN}: useradd command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running useradd command $retries times without success, giving up" - fi - sleep $count - done - else - bbnote "${PN}: user $username already exists, not re-creating it" + bbfatal "${PN}: useradd command did not succeed." fi } perform_groupmems () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupmems with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupmems with [$opts]" local groupname=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-g" || $i == "--group") print $(i+1) }'` local username=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-a" || $i == "--add") print $(i+1) }'` bbnote "${PN}: Running groupmems command with group $groupname and user $username" @@ -82,30 +49,13 @@ perform_groupmems () { gshadow="no" touch $rootdir${sysconfdir}/gshadow fi - local mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" + + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupmems $opts\' || true + local mem_exists=`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group` if test "x$mem_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO groupmems $opts || true - mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" - if test "x$mem_exists" = "x"; then - bbwarn "${PN}: groupmems command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - if test "x$gshadow" = "xno"; then - rm -f $rootdir${sysconfdir}/gshadow - rm -f $rootdir${sysconfdir}/gshadow- - fi - bbfatal "${PN}: Tried running groupmems command $retries times without success, giving up" - fi - sleep $count - done - else - bbnote "${PN}: group $groupname already contains $username, not re-adding it" + bbfatal "${PN}: groupmems command did not succeed." fi + if test "x$gshadow" = "xno"; then rm -f $rootdir${sysconfdir}/gshadow rm -f $rootdir${sysconfdir}/gshadow- @@ -115,56 +65,24 @@ perform_groupmems () { perform_groupdel () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupdel with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupdel with [$opts]" + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupdel $opts\' || true local groupname=`echo "$opts" | awk '{ print $NF }'` - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" + local group_exists=`grep "^$groupname:" $rootdir/etc/group` if test "x$group_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO groupdel $opts || true - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" - if test "x$group_exists" != "x"; then - bbwarn "${PN}: groupdel command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupdel command $retries times without success, giving up" - fi - sleep $count - done - else - bbnote "${PN}: group $groupname doesn't exist, not removing it" + bbfatal "${PN}: groupdel command did not succeed." fi } perform_userdel () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing userdel with [$opts] and $retries times of retry" + bbnote "${PN}: Performing userdel with [$opts]" + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO userdel $opts\' || true local username=`echo "$opts" | awk '{ print $NF }'` - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" + local user_exists=`grep "^$username:" $rootdir/etc/passwd` if test "x$user_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO userdel $opts || true - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" - if test "x$user_exists" != "x"; then - bbwarn "${PN}: userdel command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running userdel command $retries times without success, giving up" - fi - sleep $count - done - else - bbnote "${PN}: user $username doesn't exist, not removing it" + bbfatal "${PN}: userdel command did not succeed." fi } @@ -172,59 +90,29 @@ perform_groupmod () { # Other than the return value of groupmod, there's no simple way to judge whether the command # succeeds, so we disable -e option temporarily set +e + local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupmod with [$opts] and $retries times of retry" - local groupname=`echo "$opts" | awk '{ print $NF }'` - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" - if test "x$group_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO groupmod $opts - if test $? != 0; then - bbwarn "${PN}: groupmod command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupmod command $retries times without success, giving up" - fi - sleep $count - done - else - bbwarn "${PN}: group $groupname doesn't exist, unable to modify it" + bbnote "${PN}: Performing groupmod with [$opts]" + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupmod $opts\' + if test $? != 0; then + bbwarn "${PN}: groupmod command did not succeed." fi + set -e } perform_usermod () { # Same reason with groupmod, temporarily disable -e option set +e + local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing usermod with [$opts] and $retries times of retry" - local username=`echo "$opts" | awk '{ print $NF }'` - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" - if test "x$user_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO usermod $opts - if test $? != 0; then - bbwarn "${PN}: usermod command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running usermod command $retries times without success, giving up" - fi - sleep $count - done - else - bbwarn "${PN}: user $username doesn't exist, unable to modify it" + bbnote "${PN}: Performing usermod with [$opts]" + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO usermod $opts\' + if test $? != 0; then + bbfatal "${PN}: usermod command did not succeed." fi + set -e } -- 2.6.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 2:45 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang @ 2016-02-23 3:05 ` Khem Raj 2016-02-23 3:35 ` Kang Kai 2016-02-23 16:29 ` Burton, Ross 1 sibling, 1 reply; 18+ messages in thread From: Khem Raj @ 2016-02-23 3:05 UTC (permalink / raw) To: Kang Kai; +Cc: Patches and discussions about the oe-core layer Patch subject is unclear. Patch itself looks ok On Mon, Feb 22, 2016 at 9:45 PM, <kai.kang@windriver.com> wrote: > From: Kai Kang <kai.kang@windriver.com> > > When perform useradd during populate sysroot, it locks files passwd.lock > and group.lock at same time. And then it meets a dead lock issue > randomly. > > Use flock to reslove it by using an universal lock file for all the > user and group related operations. > > [YOCTO #9022] > > Signed-off-by: Kai Kang <kai.kang@windriver.com> > --- > meta/classes/useradd.bbclass | 6 +- > meta/classes/useradd_base.bbclass | 186 ++++++++------------------------------ > 2 files changed, 40 insertions(+), 152 deletions(-) > > diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass > index c960656..0a6f2be 100644 > --- a/meta/classes/useradd.bbclass > +++ b/meta/classes/useradd.bbclass > @@ -57,7 +57,7 @@ if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then > opts=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1` > remaining=`echo "$GROUPADD_PARAM" | cut -d ';' -f 2-` > while test "x$opts" != "x"; do > - perform_groupadd "$SYSROOT" "$OPT $opts" 10 > + perform_groupadd "$SYSROOT" "$OPT $opts" > if test "x$opts" = "x$remaining"; then > break > fi > @@ -73,7 +73,7 @@ if test "x`echo $USERADD_PARAM | tr -d '[:space:]'`" != "x"; then > opts=`echo "$USERADD_PARAM" | cut -d ';' -f 1` > remaining=`echo "$USERADD_PARAM" | cut -d ';' -f 2-` > while test "x$opts" != "x"; do > - perform_useradd "$SYSROOT" "$OPT $opts" 10 > + perform_useradd "$SYSROOT" "$OPT $opts" > if test "x$opts" = "x$remaining"; then > break > fi > @@ -89,7 +89,7 @@ if test "x`echo $GROUPMEMS_PARAM | tr -d '[:space:]'`" != "x"; then > opts=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 1` > remaining=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 2-` > while test "x$opts" != "x"; do > - perform_groupmems "$SYSROOT" "$OPT $opts" 10 > + perform_groupmems "$SYSROOT" "$OPT $opts" > if test "x$opts" = "x$remaining"; then > break > fi > diff --git a/meta/classes/useradd_base.bbclass b/meta/classes/useradd_base.bbclass > index ab3cd35..7c1cc13 100644 > --- a/meta/classes/useradd_base.bbclass > +++ b/meta/classes/useradd_base.bbclass > @@ -4,7 +4,7 @@ > > # The following functions basically have similar logic. > # *) Perform necessary checks before invoking the actual command > -# *) Invoke the actual command, make retries if necessary > +# *) Invoke the actual command with flock > # *) Error out if an error occurs. > > # Note that before invoking these functions, make sure the global variable > @@ -12,65 +12,32 @@ > > perform_groupadd () { > local rootdir="$1" > - local opts="$2" > - local retries="$3" > - bbnote "${PN}: Performing groupadd with [$opts] and $retries times of retry" > + local opts=`echo $2 | sed s/\'/\"/g` > + bbnote "${PN}: Performing groupadd with [$opts]" > + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupadd $opts\' || true > local groupname=`echo "$opts" | awk '{ print $NF }'` > - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" > + local group_exists=`grep "^$groupname:" $rootdir/etc/group` > if test "x$group_exists" = "x"; then > - local count=0 > - while true; do > - eval $PSEUDO groupadd $opts || true > - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" > - if test "x$group_exists" = "x"; then > - bbwarn "${PN}: groupadd command did not succeed. Retrying..." > - else > - break > - fi > - count=`expr $count + 1` > - if test $count = $retries; then > - bbfatal "${PN}: Tried running groupadd command $retries times without success, giving up" > - fi > - sleep $count > - done > - else > - bbnote "${PN}: group $groupname already exists, not re-creating it" > + bbfatal "${PN}: groupadd command did not succeed." > fi > } > > perform_useradd () { > local rootdir="$1" > - local opts="$2" > - local retries="$3" > - bbnote "${PN}: Performing useradd with [$opts] and $retries times of retry" > + local opts=`echo $2 | sed s/\'/\"/g` > + bbnote "${PN}: Performing useradd with [$opts]" > + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO useradd $opts\' || true > local username=`echo "$opts" | awk '{ print $NF }'` > - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" > + local user_exists=`grep "^$username:" $rootdir/etc/passwd` > if test "x$user_exists" = "x"; then > - local count=0 > - while true; do > - eval $PSEUDO useradd $opts || true > - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" > - if test "x$user_exists" = "x"; then > - bbwarn "${PN}: useradd command did not succeed. Retrying..." > - else > - break > - fi > - count=`expr $count + 1` > - if test $count = $retries; then > - bbfatal "${PN}: Tried running useradd command $retries times without success, giving up" > - fi > - sleep $count > - done > - else > - bbnote "${PN}: user $username already exists, not re-creating it" > + bbfatal "${PN}: useradd command did not succeed." > fi > } > > perform_groupmems () { > local rootdir="$1" > local opts="$2" > - local retries="$3" > - bbnote "${PN}: Performing groupmems with [$opts] and $retries times of retry" > + bbnote "${PN}: Performing groupmems with [$opts]" > local groupname=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-g" || $i == "--group") print $(i+1) }'` > local username=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-a" || $i == "--add") print $(i+1) }'` > bbnote "${PN}: Running groupmems command with group $groupname and user $username" > @@ -82,30 +49,13 @@ perform_groupmems () { > gshadow="no" > touch $rootdir${sysconfdir}/gshadow > fi > - local mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" > + > + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupmems $opts\' || true > + local mem_exists=`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group` > if test "x$mem_exists" = "x"; then > - local count=0 > - while true; do > - eval $PSEUDO groupmems $opts || true > - mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" > - if test "x$mem_exists" = "x"; then > - bbwarn "${PN}: groupmems command did not succeed. Retrying..." > - else > - break > - fi > - count=`expr $count + 1` > - if test $count = $retries; then > - if test "x$gshadow" = "xno"; then > - rm -f $rootdir${sysconfdir}/gshadow > - rm -f $rootdir${sysconfdir}/gshadow- > - fi > - bbfatal "${PN}: Tried running groupmems command $retries times without success, giving up" > - fi > - sleep $count > - done > - else > - bbnote "${PN}: group $groupname already contains $username, not re-adding it" > + bbfatal "${PN}: groupmems command did not succeed." > fi > + > if test "x$gshadow" = "xno"; then > rm -f $rootdir${sysconfdir}/gshadow > rm -f $rootdir${sysconfdir}/gshadow- > @@ -115,56 +65,24 @@ perform_groupmems () { > perform_groupdel () { > local rootdir="$1" > local opts="$2" > - local retries="$3" > - bbnote "${PN}: Performing groupdel with [$opts] and $retries times of retry" > + bbnote "${PN}: Performing groupdel with [$opts]" > + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupdel $opts\' || true > local groupname=`echo "$opts" | awk '{ print $NF }'` > - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" > + local group_exists=`grep "^$groupname:" $rootdir/etc/group` > if test "x$group_exists" != "x"; then > - local count=0 > - while true; do > - eval $PSEUDO groupdel $opts || true > - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" > - if test "x$group_exists" != "x"; then > - bbwarn "${PN}: groupdel command did not succeed. Retrying..." > - else > - break > - fi > - count=`expr $count + 1` > - if test $count = $retries; then > - bbfatal "${PN}: Tried running groupdel command $retries times without success, giving up" > - fi > - sleep $count > - done > - else > - bbnote "${PN}: group $groupname doesn't exist, not removing it" > + bbfatal "${PN}: groupdel command did not succeed." > fi > } > > perform_userdel () { > local rootdir="$1" > local opts="$2" > - local retries="$3" > - bbnote "${PN}: Performing userdel with [$opts] and $retries times of retry" > + bbnote "${PN}: Performing userdel with [$opts]" > + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO userdel $opts\' || true > local username=`echo "$opts" | awk '{ print $NF }'` > - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" > + local user_exists=`grep "^$username:" $rootdir/etc/passwd` > if test "x$user_exists" != "x"; then > - local count=0 > - while true; do > - eval $PSEUDO userdel $opts || true > - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" > - if test "x$user_exists" != "x"; then > - bbwarn "${PN}: userdel command did not succeed. Retrying..." > - else > - break > - fi > - count=`expr $count + 1` > - if test $count = $retries; then > - bbfatal "${PN}: Tried running userdel command $retries times without success, giving up" > - fi > - sleep $count > - done > - else > - bbnote "${PN}: user $username doesn't exist, not removing it" > + bbfatal "${PN}: userdel command did not succeed." > fi > } > > @@ -172,59 +90,29 @@ perform_groupmod () { > # Other than the return value of groupmod, there's no simple way to judge whether the command > # succeeds, so we disable -e option temporarily > set +e > + > local rootdir="$1" > local opts="$2" > - local retries="$3" > - bbnote "${PN}: Performing groupmod with [$opts] and $retries times of retry" > - local groupname=`echo "$opts" | awk '{ print $NF }'` > - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" > - if test "x$group_exists" != "x"; then > - local count=0 > - while true; do > - eval $PSEUDO groupmod $opts > - if test $? != 0; then > - bbwarn "${PN}: groupmod command did not succeed. Retrying..." > - else > - break > - fi > - count=`expr $count + 1` > - if test $count = $retries; then > - bbfatal "${PN}: Tried running groupmod command $retries times without success, giving up" > - fi > - sleep $count > - done > - else > - bbwarn "${PN}: group $groupname doesn't exist, unable to modify it" > + bbnote "${PN}: Performing groupmod with [$opts]" > + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupmod $opts\' > + if test $? != 0; then > + bbwarn "${PN}: groupmod command did not succeed." > fi > + > set -e > } > > perform_usermod () { > # Same reason with groupmod, temporarily disable -e option > set +e > + > local rootdir="$1" > local opts="$2" > - local retries="$3" > - bbnote "${PN}: Performing usermod with [$opts] and $retries times of retry" > - local username=`echo "$opts" | awk '{ print $NF }'` > - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" > - if test "x$user_exists" != "x"; then > - local count=0 > - while true; do > - eval $PSEUDO usermod $opts > - if test $? != 0; then > - bbwarn "${PN}: usermod command did not succeed. Retrying..." > - else > - break > - fi > - count=`expr $count + 1` > - if test $count = $retries; then > - bbfatal "${PN}: Tried running usermod command $retries times without success, giving up" > - fi > - sleep $count > - done > - else > - bbwarn "${PN}: user $username doesn't exist, unable to modify it" > + bbnote "${PN}: Performing usermod with [$opts]" > + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO usermod $opts\' > + if test $? != 0; then > + bbfatal "${PN}: usermod command did not succeed." > fi > + > set -e > } > -- > 2.6.1 > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 3:05 ` Khem Raj @ 2016-02-23 3:35 ` Kang Kai 2016-02-23 7:25 ` Richard Purdie 0 siblings, 1 reply; 18+ messages in thread From: Kang Kai @ 2016-02-23 3:35 UTC (permalink / raw) To: Khem Raj; +Cc: Patches and discussions about the oe-core layer On 2016年02月23日 11:05, Khem Raj wrote: > Patch subject is unclear. Patch itself looks ok How about use the title of Yocto #9022: useradd_base.bbclass: replace retry logic with flock Thanks, Kai > > On Mon, Feb 22, 2016 at 9:45 PM, <kai.kang@windriver.com> wrote: >> From: Kai Kang <kai.kang@windriver.com> >> >> When perform useradd during populate sysroot, it locks files passwd.lock >> and group.lock at same time. And then it meets a dead lock issue >> randomly. >> >> Use flock to reslove it by using an universal lock file for all the >> user and group related operations. >> >> [YOCTO #9022] >> >> Signed-off-by: Kai Kang <kai.kang@windriver.com> >> --- >> meta/classes/useradd.bbclass | 6 +- >> meta/classes/useradd_base.bbclass | 186 ++++++++------------------------------ >> 2 files changed, 40 insertions(+), 152 deletions(-) >> >> diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass >> index c960656..0a6f2be 100644 >> --- a/meta/classes/useradd.bbclass >> +++ b/meta/classes/useradd.bbclass >> @@ -57,7 +57,7 @@ if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then >> opts=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1` >> remaining=`echo "$GROUPADD_PARAM" | cut -d ';' -f 2-` >> while test "x$opts" != "x"; do >> - perform_groupadd "$SYSROOT" "$OPT $opts" 10 >> + perform_groupadd "$SYSROOT" "$OPT $opts" >> if test "x$opts" = "x$remaining"; then >> break >> fi >> @@ -73,7 +73,7 @@ if test "x`echo $USERADD_PARAM | tr -d '[:space:]'`" != "x"; then >> opts=`echo "$USERADD_PARAM" | cut -d ';' -f 1` >> remaining=`echo "$USERADD_PARAM" | cut -d ';' -f 2-` >> while test "x$opts" != "x"; do >> - perform_useradd "$SYSROOT" "$OPT $opts" 10 >> + perform_useradd "$SYSROOT" "$OPT $opts" >> if test "x$opts" = "x$remaining"; then >> break >> fi >> @@ -89,7 +89,7 @@ if test "x`echo $GROUPMEMS_PARAM | tr -d '[:space:]'`" != "x"; then >> opts=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 1` >> remaining=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 2-` >> while test "x$opts" != "x"; do >> - perform_groupmems "$SYSROOT" "$OPT $opts" 10 >> + perform_groupmems "$SYSROOT" "$OPT $opts" >> if test "x$opts" = "x$remaining"; then >> break >> fi >> diff --git a/meta/classes/useradd_base.bbclass b/meta/classes/useradd_base.bbclass >> index ab3cd35..7c1cc13 100644 >> --- a/meta/classes/useradd_base.bbclass >> +++ b/meta/classes/useradd_base.bbclass >> @@ -4,7 +4,7 @@ >> >> # The following functions basically have similar logic. >> # *) Perform necessary checks before invoking the actual command >> -# *) Invoke the actual command, make retries if necessary >> +# *) Invoke the actual command with flock >> # *) Error out if an error occurs. >> >> # Note that before invoking these functions, make sure the global variable >> @@ -12,65 +12,32 @@ >> >> perform_groupadd () { >> local rootdir="$1" >> - local opts="$2" >> - local retries="$3" >> - bbnote "${PN}: Performing groupadd with [$opts] and $retries times of retry" >> + local opts=`echo $2 | sed s/\'/\"/g` >> + bbnote "${PN}: Performing groupadd with [$opts]" >> + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupadd $opts\' || true >> local groupname=`echo "$opts" | awk '{ print $NF }'` >> - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" >> + local group_exists=`grep "^$groupname:" $rootdir/etc/group` >> if test "x$group_exists" = "x"; then >> - local count=0 >> - while true; do >> - eval $PSEUDO groupadd $opts || true >> - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" >> - if test "x$group_exists" = "x"; then >> - bbwarn "${PN}: groupadd command did not succeed. Retrying..." >> - else >> - break >> - fi >> - count=`expr $count + 1` >> - if test $count = $retries; then >> - bbfatal "${PN}: Tried running groupadd command $retries times without success, giving up" >> - fi >> - sleep $count >> - done >> - else >> - bbnote "${PN}: group $groupname already exists, not re-creating it" >> + bbfatal "${PN}: groupadd command did not succeed." >> fi >> } >> >> perform_useradd () { >> local rootdir="$1" >> - local opts="$2" >> - local retries="$3" >> - bbnote "${PN}: Performing useradd with [$opts] and $retries times of retry" >> + local opts=`echo $2 | sed s/\'/\"/g` >> + bbnote "${PN}: Performing useradd with [$opts]" >> + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO useradd $opts\' || true >> local username=`echo "$opts" | awk '{ print $NF }'` >> - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" >> + local user_exists=`grep "^$username:" $rootdir/etc/passwd` >> if test "x$user_exists" = "x"; then >> - local count=0 >> - while true; do >> - eval $PSEUDO useradd $opts || true >> - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" >> - if test "x$user_exists" = "x"; then >> - bbwarn "${PN}: useradd command did not succeed. Retrying..." >> - else >> - break >> - fi >> - count=`expr $count + 1` >> - if test $count = $retries; then >> - bbfatal "${PN}: Tried running useradd command $retries times without success, giving up" >> - fi >> - sleep $count >> - done >> - else >> - bbnote "${PN}: user $username already exists, not re-creating it" >> + bbfatal "${PN}: useradd command did not succeed." >> fi >> } >> >> perform_groupmems () { >> local rootdir="$1" >> local opts="$2" >> - local retries="$3" >> - bbnote "${PN}: Performing groupmems with [$opts] and $retries times of retry" >> + bbnote "${PN}: Performing groupmems with [$opts]" >> local groupname=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-g" || $i == "--group") print $(i+1) }'` >> local username=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-a" || $i == "--add") print $(i+1) }'` >> bbnote "${PN}: Running groupmems command with group $groupname and user $username" >> @@ -82,30 +49,13 @@ perform_groupmems () { >> gshadow="no" >> touch $rootdir${sysconfdir}/gshadow >> fi >> - local mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" >> + >> + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupmems $opts\' || true >> + local mem_exists=`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group` >> if test "x$mem_exists" = "x"; then >> - local count=0 >> - while true; do >> - eval $PSEUDO groupmems $opts || true >> - mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" >> - if test "x$mem_exists" = "x"; then >> - bbwarn "${PN}: groupmems command did not succeed. Retrying..." >> - else >> - break >> - fi >> - count=`expr $count + 1` >> - if test $count = $retries; then >> - if test "x$gshadow" = "xno"; then >> - rm -f $rootdir${sysconfdir}/gshadow >> - rm -f $rootdir${sysconfdir}/gshadow- >> - fi >> - bbfatal "${PN}: Tried running groupmems command $retries times without success, giving up" >> - fi >> - sleep $count >> - done >> - else >> - bbnote "${PN}: group $groupname already contains $username, not re-adding it" >> + bbfatal "${PN}: groupmems command did not succeed." >> fi >> + >> if test "x$gshadow" = "xno"; then >> rm -f $rootdir${sysconfdir}/gshadow >> rm -f $rootdir${sysconfdir}/gshadow- >> @@ -115,56 +65,24 @@ perform_groupmems () { >> perform_groupdel () { >> local rootdir="$1" >> local opts="$2" >> - local retries="$3" >> - bbnote "${PN}: Performing groupdel with [$opts] and $retries times of retry" >> + bbnote "${PN}: Performing groupdel with [$opts]" >> + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupdel $opts\' || true >> local groupname=`echo "$opts" | awk '{ print $NF }'` >> - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" >> + local group_exists=`grep "^$groupname:" $rootdir/etc/group` >> if test "x$group_exists" != "x"; then >> - local count=0 >> - while true; do >> - eval $PSEUDO groupdel $opts || true >> - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" >> - if test "x$group_exists" != "x"; then >> - bbwarn "${PN}: groupdel command did not succeed. Retrying..." >> - else >> - break >> - fi >> - count=`expr $count + 1` >> - if test $count = $retries; then >> - bbfatal "${PN}: Tried running groupdel command $retries times without success, giving up" >> - fi >> - sleep $count >> - done >> - else >> - bbnote "${PN}: group $groupname doesn't exist, not removing it" >> + bbfatal "${PN}: groupdel command did not succeed." >> fi >> } >> >> perform_userdel () { >> local rootdir="$1" >> local opts="$2" >> - local retries="$3" >> - bbnote "${PN}: Performing userdel with [$opts] and $retries times of retry" >> + bbnote "${PN}: Performing userdel with [$opts]" >> + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO userdel $opts\' || true >> local username=`echo "$opts" | awk '{ print $NF }'` >> - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" >> + local user_exists=`grep "^$username:" $rootdir/etc/passwd` >> if test "x$user_exists" != "x"; then >> - local count=0 >> - while true; do >> - eval $PSEUDO userdel $opts || true >> - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" >> - if test "x$user_exists" != "x"; then >> - bbwarn "${PN}: userdel command did not succeed. Retrying..." >> - else >> - break >> - fi >> - count=`expr $count + 1` >> - if test $count = $retries; then >> - bbfatal "${PN}: Tried running userdel command $retries times without success, giving up" >> - fi >> - sleep $count >> - done >> - else >> - bbnote "${PN}: user $username doesn't exist, not removing it" >> + bbfatal "${PN}: userdel command did not succeed." >> fi >> } >> >> @@ -172,59 +90,29 @@ perform_groupmod () { >> # Other than the return value of groupmod, there's no simple way to judge whether the command >> # succeeds, so we disable -e option temporarily >> set +e >> + >> local rootdir="$1" >> local opts="$2" >> - local retries="$3" >> - bbnote "${PN}: Performing groupmod with [$opts] and $retries times of retry" >> - local groupname=`echo "$opts" | awk '{ print $NF }'` >> - local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" >> - if test "x$group_exists" != "x"; then >> - local count=0 >> - while true; do >> - eval $PSEUDO groupmod $opts >> - if test $? != 0; then >> - bbwarn "${PN}: groupmod command did not succeed. Retrying..." >> - else >> - break >> - fi >> - count=`expr $count + 1` >> - if test $count = $retries; then >> - bbfatal "${PN}: Tried running groupmod command $retries times without success, giving up" >> - fi >> - sleep $count >> - done >> - else >> - bbwarn "${PN}: group $groupname doesn't exist, unable to modify it" >> + bbnote "${PN}: Performing groupmod with [$opts]" >> + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO groupmod $opts\' >> + if test $? != 0; then >> + bbwarn "${PN}: groupmod command did not succeed." >> fi >> + >> set -e >> } >> >> perform_usermod () { >> # Same reason with groupmod, temporarily disable -e option >> set +e >> + >> local rootdir="$1" >> local opts="$2" >> - local retries="$3" >> - bbnote "${PN}: Performing usermod with [$opts] and $retries times of retry" >> - local username=`echo "$opts" | awk '{ print $NF }'` >> - local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" >> - if test "x$user_exists" != "x"; then >> - local count=0 >> - while true; do >> - eval $PSEUDO usermod $opts >> - if test $? != 0; then >> - bbwarn "${PN}: usermod command did not succeed. Retrying..." >> - else >> - break >> - fi >> - count=`expr $count + 1` >> - if test $count = $retries; then >> - bbfatal "${PN}: Tried running usermod command $retries times without success, giving up" >> - fi >> - sleep $count >> - done >> - else >> - bbwarn "${PN}: user $username doesn't exist, unable to modify it" >> + bbnote "${PN}: Performing usermod with [$opts]" >> + eval flock -x -w 100 $rootdir${sysconfdir} -c \'$PSEUDO usermod $opts\' >> + if test $? != 0; then >> + bbfatal "${PN}: usermod command did not succeed." >> fi >> + >> set -e >> } >> -- >> 2.6.1 >> >> -- >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.openembedded.org/mailman/listinfo/openembedded-core -- Regards, Neil | Kai Kang ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 3:35 ` Kang Kai @ 2016-02-23 7:25 ` Richard Purdie 2016-02-23 7:31 ` Kang Kai 0 siblings, 1 reply; 18+ messages in thread From: Richard Purdie @ 2016-02-23 7:25 UTC (permalink / raw) To: Kang Kai, Khem Raj; +Cc: Patches and discussions about the oe-core layer On Tue, 2016-02-23 at 11:35 +0800, Kang Kai wrote: > On 2016年02月23日 11:05, Khem Raj wrote: > > Patch subject is unclear. Patch itself looks ok > > How about use the title of Yocto #9022: > > useradd_base.bbclass: replace retry logic with flock Sounds good to me, thanks! Richard ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 7:25 ` Richard Purdie @ 2016-02-23 7:31 ` Kang Kai 0 siblings, 0 replies; 18+ messages in thread From: Kang Kai @ 2016-02-23 7:31 UTC (permalink / raw) To: Richard Purdie, Khem Raj; +Cc: Patches and discussions about the oe-core layer On 2016年02月23日 15:25, Richard Purdie wrote: > On Tue, 2016-02-23 at 11:35 +0800, Kang Kai wrote: >> On 2016年02月23日 11:05, Khem Raj wrote: >>> Patch subject is unclear. Patch itself looks ok >> How about use the title of Yocto #9022: >> >> useradd_base.bbclass: replace retry logic with flock > Sounds good to me, thanks! Updated the git repo: git://git.pokylinux.org/poky-contrib kangkai/flockv3 Thanks. --Kai > > Richard > -- Regards, Neil | Kai Kang ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 2:45 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang 2016-02-23 3:05 ` Khem Raj @ 2016-02-23 16:29 ` Burton, Ross 2016-02-23 17:01 ` Burton, Ross 1 sibling, 1 reply; 18+ messages in thread From: Burton, Ross @ 2016-02-23 16:29 UTC (permalink / raw) To: Kang Kai; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 1234 bytes --] On 23 February 2016 at 02:45, <kai.kang@windriver.com> wrote: > When perform useradd during populate sysroot, it locks files passwd.lock > and group.lock at same time. And then it meets a dead lock issue > randomly. > > Use flock to reslove it by using an universal lock file for all the > user and group related operations. > I got an error when openssh's do_install was running useradds to the sysroot: | DEBUG: Executing shell function useradd_sysroot | Running useradd commands... | /data/poky-master/tmp-glibc/work/core2-32-poky-linux/openssh/7.1p2-r0/temp/run.useradd_sysroot.13825: 264: local: /data/poky-master/tmp-glibc/sysroots/intel-core2-32: bad variable name | WARNING: exit code 2 from a shell command. | ERROR: Function failed: useradd_sysroot (log file is located at /data/poky-master/tmp-glibc/work/core2-32-poky-linux/openssh/7.1p2-r0/temp/log.do_install.13825) ERROR: Task 2939 (/home/ross/Yocto/poky/meta/recipes-connectivity/openssh/ openssh_7.1p2.bb, do_install) failed with exit code '1' 262 perform_useradd() { 263 local rootdir="$1" 264 local opts=`echo $2 | sed s/\'/\"/g` 265 bbnote "openssh: Performing useradd with [$opts]" Ross [-- Attachment #2: Type: text/html, Size: 2250 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 16:29 ` Burton, Ross @ 2016-02-23 17:01 ` Burton, Ross 2016-02-23 17:10 ` Burton, Ross 0 siblings, 1 reply; 18+ messages in thread From: Burton, Ross @ 2016-02-23 17:01 UTC (permalink / raw) To: Kang Kai; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 1536 bytes --] On 23 February 2016 at 16:29, Burton, Ross <ross.burton@intel.com> wrote: > I got an error when openssh's do_install was running useradds to the > sysroot: > > | DEBUG: Executing shell function useradd_sysroot > | Running useradd commands... > | > /data/poky-master/tmp-glibc/work/core2-32-poky-linux/openssh/7.1p2-r0/temp/run.useradd_sysroot.13825: > 264: local: /data/poky-master/tmp-glibc/sysroots/intel-core2-32: bad > variable name > | WARNING: exit code 2 from a shell command. > | ERROR: Function failed: useradd_sysroot (log file is located at > /data/poky-master/tmp-glibc/work/core2-32-poky-linux/openssh/7.1p2-r0/temp/log.do_install.13825) > ERROR: Task 2939 (/home/ross/Yocto/poky/meta/recipes-connectivity/openssh/ > openssh_7.1p2.bb, do_install) failed with exit code '1' > > 262 perform_useradd() { > 263 local rootdir="$1" > 264 local opts=`echo $2 | sed s/\'/\"/g` > 265 bbnote "openssh: Performing useradd with [$opts]" > The *amazing* BB_VERBOSE_LOGS option (set to 1 in local.conf to get all shell scripts to do set -x) tells me this is due to missing quotes around the opts assignment: + local opts=--root /data/poky-master/tmp-glibc/sysroots/intel-core2-32 --system --no-create-home --home-dir /var/run/sshd --shell /bin/false --user-group sshd /data/poky-master/tmp-glibc/work/core2-32-poky-linux/openssh/7.1p2-r0/temp/run.useradd_sysroot.16570: 264: local: /data/poky-master/tmp-glibc/sysroots/intel-core2-32: bad variable name Ross [-- Attachment #2: Type: text/html, Size: 2583 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 17:01 ` Burton, Ross @ 2016-02-23 17:10 ` Burton, Ross 2016-02-24 8:51 ` Kang Kai 0 siblings, 1 reply; 18+ messages in thread From: Burton, Ross @ 2016-02-23 17:10 UTC (permalink / raw) To: Kang Kai; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 341 bytes --] On 23 February 2016 at 17:01, Burton, Ross <ross.burton@intel.com> wrote: > The *amazing* BB_VERBOSE_LOGS option (set to 1 in local.conf to get all > shell scripts to do set -x) tells me this is due to missing quotes around > the opts assignment: > Why do some of the opts assignments just do opts=$2, and others do a sed? Ross [-- Attachment #2: Type: text/html, Size: 744 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-23 17:10 ` Burton, Ross @ 2016-02-24 8:51 ` Kang Kai 0 siblings, 0 replies; 18+ messages in thread From: Kang Kai @ 2016-02-24 8:51 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 899 bytes --] On 2016年02月24日 01:10, Burton, Ross wrote: > > On 23 February 2016 at 17:01, Burton, Ross <ross.burton@intel.com > <mailto:ross.burton@intel.com>> wrote: > > The *amazing* BB_VERBOSE_LOGS option (set to 1 in local.conf to > get all shell scripts to do set -x) tells me this is due to > missing quotes around the opts assignment: > > > Why do some of the opts assignments just do opts=$2, and others do a sed? In the previous version, there are: local opts="$2" ... opts=`echo $opts | sed s/\'/\"/g` And I want to refactor them but lost the double quotes. Only do sed operations for useradd and groupadd that they may be passed in "opts" with space but others didn't. I'll restore to previous version. Sorry for inconvenience that I built pass on Ubuntu 14.04 so didn't find this error. --Kai > > Ross -- Regards, Neil | Kai Kang [-- Attachment #2: Type: text/html, Size: 2261 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 0/1] Replace retry logic in useradd with flock @ 2016-02-15 1:59 kai.kang 2016-02-15 1:59 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang 0 siblings, 1 reply; 18+ messages in thread From: kai.kang @ 2016-02-15 1:59 UTC (permalink / raw) To: ross.burton; +Cc: openembedded-core From: Kai Kang <kai.kang@windriver.com> Hi Ross, We met problems that useradd fails to add users. I found it a simultaneous issue, so created this patch. As a comparison of build time: Without flock: 1 $ time bitbake core-image-minimal real 38m8.625s user 318m43.262s sys 62m47.710s 2 $ time bitbake core-image-sato real 54m13.065s user 587m1.386s sys 122m16.949s With flock: 1 $ time bitbake core-image-minimal real 40m55.178s user 316m26.576s sys 58m53.322s 2 $ time bitbake core-image-sato real 55m11.312s user 580m16.787s sys 118m16.385s The following changes since commit 36b43b233c841b3c09fc17e04686b4b758780963: lib/oe/terminal: set workdir for konsole terminal (2016-02-14 11:42:19 +0000) are available in the git repository at: git://git.pokylinux.org/poky-contrib kangkai/flock http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/flock Kai Kang (1): useradd_base.bbclass: fix simultaneous with flock meta/classes/useradd.bbclass | 8 +- meta/classes/useradd_base.bbclass | 165 ++++++++++---------------------------- 2 files changed, 48 insertions(+), 125 deletions(-) -- 2.6.1 ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-15 1:59 [PATCH 0/1] Replace retry logic in useradd " kai.kang @ 2016-02-15 1:59 ` kai.kang 2016-02-15 12:31 ` Burton, Ross 0 siblings, 1 reply; 18+ messages in thread From: kai.kang @ 2016-02-15 1:59 UTC (permalink / raw) To: ross.burton; +Cc: openembedded-core From: Kai Kang <kai.kang@windriver.com> When perform useradd during populate sysroot, it locks files passwd.lock and group.lock at same time. And then it meets a dead lock issue. Use flock to reslove it by using an universal lock file for all the user and group related operations. The lock file is put in image dir /tmp. /tmp will be mounted as a tmpfs, so the lock file will not be visible when boot the target. [YOCTO #9022] Signed-off-by: Kai Kang <kai.kang@windriver.com> --- meta/classes/useradd.bbclass | 8 +- meta/classes/useradd_base.bbclass | 165 ++++++++++---------------------------- 2 files changed, 48 insertions(+), 125 deletions(-) diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass index c960656..db74123 100644 --- a/meta/classes/useradd.bbclass +++ b/meta/classes/useradd.bbclass @@ -48,6 +48,8 @@ if test "x$UA_SYSROOT" = "x"; then GROUPMEMS_PARAM="${GROUPMEMS_PARAM}" fi +mkdir -p $SYSROOT/tmp + # Perform group additions first, since user additions may depend # on these groups existing if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then @@ -57,7 +59,7 @@ if test "x`echo $GROUPADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPADD_PARAM" | cut -d ';' -f 1` remaining=`echo "$GROUPADD_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_groupadd "$SYSROOT" "$OPT $opts" 10 + perform_groupadd "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi @@ -73,7 +75,7 @@ if test "x`echo $USERADD_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$USERADD_PARAM" | cut -d ';' -f 1` remaining=`echo "$USERADD_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_useradd "$SYSROOT" "$OPT $opts" 10 + perform_useradd "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi @@ -89,7 +91,7 @@ if test "x`echo $GROUPMEMS_PARAM | tr -d '[:space:]'`" != "x"; then opts=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 1` remaining=`echo "$GROUPMEMS_PARAM" | cut -d ';' -f 2-` while test "x$opts" != "x"; do - perform_groupmems "$SYSROOT" "$OPT $opts" 10 + perform_groupmems "$SYSROOT" "$OPT $opts" if test "x$opts" = "x$remaining"; then break fi diff --git a/meta/classes/useradd_base.bbclass b/meta/classes/useradd_base.bbclass index ab3cd35..a5baceb 100644 --- a/meta/classes/useradd_base.bbclass +++ b/meta/classes/useradd_base.bbclass @@ -4,7 +4,7 @@ # The following functions basically have similar logic. # *) Perform necessary checks before invoking the actual command -# *) Invoke the actual command, make retries if necessary +# *) Invoke the actual command with flock # *) Error out if an error occurs. # Note that before invoking these functions, make sure the global variable @@ -13,26 +13,16 @@ perform_groupadd () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupadd with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupadd with [$opts]" local groupname=`echo "$opts" | awk '{ print $NF }'` local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" if test "x$group_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO groupadd $opts || true - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" - if test "x$group_exists" = "x"; then - bbwarn "${PN}: groupadd command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupadd command $retries times without success, giving up" - fi - sleep $count - done + opts=`echo $opts | sed s/\'/\"/g` + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO groupadd $opts\' || true + group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" + if test "x$group_exists" = "x"; then + bbfatal "${PN}: groupadd command did not succeed." + fi else bbnote "${PN}: group $groupname already exists, not re-creating it" fi @@ -41,26 +31,16 @@ perform_groupadd () { perform_useradd () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing useradd with [$opts] and $retries times of retry" + bbnote "${PN}: Performing useradd with [$opts]" local username=`echo "$opts" | awk '{ print $NF }'` local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" if test "x$user_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO useradd $opts || true - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" - if test "x$user_exists" = "x"; then - bbwarn "${PN}: useradd command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running useradd command $retries times without success, giving up" - fi - sleep $count - done + opts=`echo $opts | sed s/\'/\"/g` + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO useradd $opts\' || true + user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" + if test "x$user_exists" = "x"; then + bbfatal "${PN}: useradd command did not succeed." + fi else bbnote "${PN}: user $username already exists, not re-creating it" fi @@ -69,8 +49,7 @@ perform_useradd () { perform_groupmems () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupmems with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupmems with [$opts]" local groupname=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-g" || $i == "--group") print $(i+1) }'` local username=`echo "$opts" | awk '{ for (i = 1; i < NF; i++) if ($i == "-a" || $i == "--add") print $(i+1) }'` bbnote "${PN}: Running groupmems command with group $groupname and user $username" @@ -84,25 +63,11 @@ perform_groupmems () { fi local mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" if test "x$mem_exists" = "x"; then - local count=0 - while true; do - eval $PSEUDO groupmems $opts || true - mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" - if test "x$mem_exists" = "x"; then - bbwarn "${PN}: groupmems command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - if test "x$gshadow" = "xno"; then - rm -f $rootdir${sysconfdir}/gshadow - rm -f $rootdir${sysconfdir}/gshadow- - fi - bbfatal "${PN}: Tried running groupmems command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO groupmems $opts\' || true + mem_exists="`grep "^$groupname:[^:]*:[^:]*:\([^,]*,\)*$username\(,[^,]*\)*" $rootdir/etc/group || true`" + if test "x$mem_exists" = "x"; then + bbfatal "${PN}: groupmems command did not succeed." + fi else bbnote "${PN}: group $groupname already contains $username, not re-adding it" fi @@ -115,26 +80,15 @@ perform_groupmems () { perform_groupdel () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupdel with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupdel with [$opts]" local groupname=`echo "$opts" | awk '{ print $NF }'` local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" if test "x$group_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO groupdel $opts || true - group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" - if test "x$group_exists" != "x"; then - bbwarn "${PN}: groupdel command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupdel command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO groupdel $opts\' || true + group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" + if test "x$group_exists" != "x"; then + bbfatal "${PN}: groupdel command did not succeed." + fi else bbnote "${PN}: group $groupname doesn't exist, not removing it" fi @@ -143,26 +97,15 @@ perform_groupdel () { perform_userdel () { local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing userdel with [$opts] and $retries times of retry" + bbnote "${PN}: Performing userdel with [$opts]" local username=`echo "$opts" | awk '{ print $NF }'` local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" if test "x$user_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO userdel $opts || true - user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" - if test "x$user_exists" != "x"; then - bbwarn "${PN}: userdel command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running userdel command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO userdel $opts\' || true + user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" + if test "x$user_exists" != "x"; then + bbfatal "${PN}: userdel command did not succeed." + fi else bbnote "${PN}: user $username doesn't exist, not removing it" fi @@ -174,25 +117,14 @@ perform_groupmod () { set +e local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing groupmod with [$opts] and $retries times of retry" + bbnote "${PN}: Performing groupmod with [$opts]" local groupname=`echo "$opts" | awk '{ print $NF }'` local group_exists="`grep "^$groupname:" $rootdir/etc/group || true`" if test "x$group_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO groupmod $opts - if test $? != 0; then - bbwarn "${PN}: groupmod command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running groupmod command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO groupmod $opts\' + if test $? != 0; then + bbwarn "${PN}: groupmod command did not succeed." + fi else bbwarn "${PN}: group $groupname doesn't exist, unable to modify it" fi @@ -204,25 +136,14 @@ perform_usermod () { set +e local rootdir="$1" local opts="$2" - local retries="$3" - bbnote "${PN}: Performing usermod with [$opts] and $retries times of retry" + bbnote "${PN}: Performing usermod with [$opts]" local username=`echo "$opts" | awk '{ print $NF }'` local user_exists="`grep "^$username:" $rootdir/etc/passwd || true`" if test "x$user_exists" != "x"; then - local count=0 - while true; do - eval $PSEUDO usermod $opts - if test $? != 0; then - bbwarn "${PN}: usermod command did not succeed. Retrying..." - else - break - fi - count=`expr $count + 1` - if test $count = $retries; then - bbfatal "${PN}: Tried running usermod command $retries times without success, giving up" - fi - sleep $count - done + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO usermod $opts\' + if test $? != 0; then + bbfatal "${PN}: usermod command did not succeed." + fi else bbwarn "${PN}: user $username doesn't exist, unable to modify it" fi -- 2.6.1 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-15 1:59 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang @ 2016-02-15 12:31 ` Burton, Ross 2016-02-16 2:14 ` Kang Kai 0 siblings, 1 reply; 18+ messages in thread From: Burton, Ross @ 2016-02-15 12:31 UTC (permalink / raw) To: Kang Kai; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 369 bytes --] On 15 February 2016 at 01:59, <kai.kang@windriver.com> wrote: > + eval flock -x -w 100 $rootdir/tmp/user-and-group-ops.flock > -c \'$PSEUDO useradd $opts\' || true > This is a great patch, but why not just flock $rootdir/etc instead of new file in tmp? Surely images end up being created with /tmp/user-and-group-ops.flock in them? Ross [-- Attachment #2: Type: text/html, Size: 793 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] useradd_base.bbclass: fix simultaneous with flock 2016-02-15 12:31 ` Burton, Ross @ 2016-02-16 2:14 ` Kang Kai 0 siblings, 0 replies; 18+ messages in thread From: Kang Kai @ 2016-02-16 2:14 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 722 bytes --] On 2016年02月15日 20:31, Burton, Ross wrote: > > On 15 February 2016 at 01:59, <kai.kang@windriver.com > <mailto:kai.kang@windriver.com>> wrote: > > + eval flock -x -w 100 > $rootdir/tmp/user-and-group-ops.flock -c \'$PSEUDO useradd > $opts\' || true > > > This is a great patch, but why not just flock $rootdir/etc instead of > new file in tmp? Surely images end up being created with > /tmp/user-and-group-ops.flock in them? I didn't try to think take a directory as a lock and I'll try it. For the lock file in $rootdir/tmp/, it is ok that /tmp will be remounted and user won't see the lock file. Thanks, Kai > > Ross -- Regards, Neil | Kai Kang [-- Attachment #2: Type: text/html, Size: 2127 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2016-02-24 8:51 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-02-16 7:14 [PATCH 0/1] V2: Replace retry logic in useradd with flock kai.kang 2016-02-16 7:14 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang 2016-02-16 12:09 ` Burton, Ross 2016-02-17 5:40 ` Kang Kai 2016-02-17 23:49 ` Burton, Ross 2016-02-22 8:41 ` Kang Kai -- strict thread matches above, loose matches on Subject: below -- 2016-02-23 2:45 [PATCH 0/1] V3: Replace retry logic in useradd " kai.kang 2016-02-23 2:45 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang 2016-02-23 3:05 ` Khem Raj 2016-02-23 3:35 ` Kang Kai 2016-02-23 7:25 ` Richard Purdie 2016-02-23 7:31 ` Kang Kai 2016-02-23 16:29 ` Burton, Ross 2016-02-23 17:01 ` Burton, Ross 2016-02-23 17:10 ` Burton, Ross 2016-02-24 8:51 ` Kang Kai 2016-02-15 1:59 [PATCH 0/1] Replace retry logic in useradd " kai.kang 2016-02-15 1:59 ` [PATCH 1/1] useradd_base.bbclass: fix simultaneous " kai.kang 2016-02-15 12:31 ` Burton, Ross 2016-02-16 2:14 ` Kang Kai
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.