From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by mail.openembedded.org (Postfix) with ESMTP id A48A87FBB5 for ; Fri, 17 Jan 2020 11:59:55 +0000 (UTC) Received: by mail-yb1-f174.google.com with SMTP id l197so4373928ybf.7 for ; Fri, 17 Jan 2020 03:59:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=x93DWZM7Wm7SZvbzHXd2S8z2FkQvmk8wFg4yzktVRFA=; b=T3b6e/XxBp76dHUxeLUp5UbcaxKHTJPqeiLAI7itqHzq55AVaTbWWj6wrS+ZPs5T4q NzL9bgeSlSHAXYh8auWiyQRIl7qYqoPUs25++zRDvsotoPnZEQwxzt0Y0YfYMGRobfhD 6H5alVk4O1G4eHtSqQ8j5AMF5F7YjFcamUhJrXs7HOwql58aLqGwqUWHl0rpkOrZm8m8 I6W3hfzAWRolrxu8OxlEZq69qcQLoQZkGzjnak/pdO67hVYBQZ7el7KCnz8axDacbWjY HIqUaaOUgJ/fjdrZnnN5Cy2tPX00GKAjwiuEZ4HTO3bD+zaDCqDvYVuC9GEf549RpfEu fp3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=x93DWZM7Wm7SZvbzHXd2S8z2FkQvmk8wFg4yzktVRFA=; b=Vg84lNkDHCd1SK/zRFnAOoiT2sUNFI0Tn+KXcxMftMOpJ4J6nCcME/nACMR1apfSkI ABRny/vZGuyhqFbLV5Y04jVYkfyxv10kUNKF8EABTii5zzA33Xrl8Wq2V5oM3zxpRaXX q3cWY9XT0KywaQsxA7Op5yLsyWbTeJl0vDa3INBOVj+K8+uxrH9LmiFhPbqvhU3oPfIG 2S+yIUdGyxxnhSecM+Sq4A6EFjDXPA8hoYhEU/0FxIvdaAfxJ6//9o86nmql9YPCEodY HV2Blb27UPA+EZbABf7FCbYrVV6zSzbhNr80KKQQUqO3k1r7vhf/J5vE1ORZDLcB4IPb 1ytg== X-Gm-Message-State: APjAAAXUTWUIhsGtp5GgISk0bYgmgsDwNsINpKKahIgNyUUSxatMNfkO +BECleAgTWFPiNBVTaRAVc8RMQ2/Yqz4X+JbBd5Clw== X-Google-Smtp-Source: APXvYqy0eabSfjqVlZU4YKPKriJW7stm8czWS6yoLnVAMfUrhx3944yzWf3KcZU6Gj0XViDskPfnKLo0KFJMOzwXJQk= X-Received: by 2002:a5b:7cf:: with SMTP id t15mr19738938ybq.127.1579262396424; Fri, 17 Jan 2020 03:59:56 -0800 (PST) MIME-Version: 1.0 References: <20200113130827.7409-1-maxim.uvarov@linaro.org> In-Reply-To: From: Maxim Uvarov Date: Fri, 17 Jan 2020 14:59:45 +0300 Message-ID: To: Paul Barker Cc: openembedded-core Subject: Re: [PATCHv2] wic: fix images build in parallel X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2020 11:59:55 -0000 Content-Type: text/plain; charset="UTF-8" On Fri, 17 Jan 2020 at 13:18, Paul Barker wrote: > > On Mon, 13 Jan 2020 at 14:12, Maxim Uvarov wrote: > > > > On Mon, 13 Jan 2020 at 17:00, Paul Barker wrote: > > > > > > On Mon, 13 Jan 2020 at 13:57, Maxim Uvarov wrote: > > > > > > > > On Mon, 13 Jan 2020 at 16:31, Paul Barker wrote: > > > > > > > > > > On Mon, 13 Jan 2020 at 13:08, Maxim Uvarov wrote: > > > > > > > > > > > > OE wic plugins create temporary file with the index of the line > > > > > > tmp file name. This causes race in case several builds run in time. > > > > > > Add more entropy as timestamp to remove this race. > > > > > > > > > > How would two wic images to be built in parallel with the same work > > > > > directory? To my understanding an image recipe only supports building > > > > > a single wic image. > > > > > > > > > > Thanks, > > > > > Paul > > > > > > > > bitbake image1 image2 image3 > > > > all images build .wics and use about the same files, like firmware. > > > > Issue is similar to that: > > > > https://www.yoctoproject.org/pipermail/yocto/2018-June/041373.html > > > > > > Each image has its own work directory though. > > > > > > I'll take a look in more detail later today or tomorrow, if wic is > > > writing temporary files outside of the work directory then that's a > > > bug and should be fixed. > > > > Thanks. I saw bug in rawcopy plugin. All other places were patched due > > to the same code. I guess then for rawcopy temp files are in > > DEST_IMAGE_DIR (which is common) instead of WORKDIR. > > I can't see how these files can possibly collide across parallel image > builds. In the lines you changed in your patch, cr_workdir passed in > as an argument set to imager.workdir (in > scripts/lib/wic/plugins/imager/direct.py) which is a temporary > directory created under options.outdir. image_types_wic.bbclass passes > the -o option to wic to set the outdir to something under ${WORKDIR}. > And different images have different WORKDIR paths. So things should be > safe. > > Which branch & version of poky or oe-core are you using? > zeus > DEST_IMAGE_DIR is not defined in openembedded-core. Are you sure > that's the right variable name? > DEPLOY_DIR_IMAGE > Could you provide full bitbake output showing the exact error message > when this collision happens? See traceback here: https://ci.linaro.org/job/ledge-oe/357/DISTRO=rpb,MACHINE=ledge-stm32mp157c-dk2,label=docker-stretch-amd64/console > > And lastly if you want to look into this in more detail maybe add some > print statements just after each of those lines you patched to print > the full paths used so we can see if they step outside WORKDIR. I > think doing this on your side with your exact wks files that are > causing errors will give the most useful output. > Is there a good way to add debug messages and see then on console, other then rise WicError() ? Maxim. > Thanks, > Paul