From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by yocto-www.yoctoproject.org (Postfix, from userid 118) id 3756AE00BBF; Mon, 18 Sep 2017 08:41:54 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on yocto-www.yoctoproject.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_NONE, RCVD_IN_SORBS_SPAM,WEIRD_PORT autolearn=no version=3.3.1 X-Spam-HAM-Report: * -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no * trust * [209.85.214.51 listed in list.dnswl.org] * 0.5 RCVD_IN_SORBS_SPAM RBL: SORBS: sender is a spam source * [209.85.214.51 listed in dnsbl.sorbs.net] * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (katutxakurra[at]gmail.com) * 0.0 WEIRD_PORT URI: Uses non-standard port number for HTTP * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 HTML_MESSAGE BODY: HTML included in message * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's * domain * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature Received: from mail-it0-f51.google.com (mail-it0-f51.google.com [209.85.214.51]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 31E56E008D2 for ; Mon, 18 Sep 2017 08:41:51 -0700 (PDT) Received: by mail-it0-f51.google.com with SMTP id 4so1370347itv.4 for ; Mon, 18 Sep 2017 08:41:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=xWr3cJ3TOVc3u8HcGlDB7vKjsA3i/jf19fGyqSMxp8c=; b=eHnz2JW1KSmTa0lawY3AAzqcXQpwfZXah3o1qgTIpfEyYHwwauvDz3ScfmfumhftRr yoCldp5xu4NX/i/G3ptfYc2fOeM7al8X4LdyzhFGlWmqirF92ju/7t/A8meCCUDqV0px GgjkrSseo+aTx2LgaE2pjZj+qxAYHAa5KZLIDKfUEJLJOebMFOsUpEzRzgfukx6RT/Uh OavdIP8ks/rUI+Y6td8VyQt2lD0PZZDy+pfwFLElefGSY3xjPPppuVYqR+gFTmYtNCrl GYoQp0OzjvM8KGBYiwaNrKXbJBAKJMFFN3GdvmiClhJJZxt4FLsbtwqaNOiLjuFdeT0h Qq5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=xWr3cJ3TOVc3u8HcGlDB7vKjsA3i/jf19fGyqSMxp8c=; b=J4KLWRB7GzenKwrV8ZxpEACR9pFRCI+V6oRrnO40nHgZz51YFxQ3O9v78fvLLMPEzu 3eHNg7g5hqzU2bfuQM58lbX4JyIRsCsva0L2CXd7DxrFurJ9ILpJ+Wz2jEMd5O1VSAE8 Ifmn5dNBsVfKvHz+S2qDY6LoVFYASzRRNvoaPnax9/n5E5U+ZZaESFCJii5WQocFz9F3 n401kw5gHlH5zy6m8DIpCtX/2KA5KumyyfT1TRy6CMC+0cWDdXxs+JaqwkOn7VgLXKVK Hcbe9OmDD7wxiAMAkrxnhcnIR0JyWpAhCUmqgNv2h++QPviX0r6aTaiRdAX/ByPs68qO Lbsw== X-Gm-Message-State: AHPjjUhgepqgsB9NAqZVt78VLTlZkYx6KqghmTqR6h1+mF1v6ydPSa78 1XTciQIcbjv5egFGPmNN8gwI9p2lVE/9/snsO+k= X-Google-Smtp-Source: AOwi7QC/UsYyua8OLpKKqy2iR/UGhbIaXF5mFYEbL3tsqGgig4sA6KNqbtnBgB7WnqS72VJSab2Lj9WF3C4gJ18AOBw= X-Received: by 10.36.4.151 with SMTP id 145mr19400021itb.33.1505749311193; Mon, 18 Sep 2017 08:41:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.153.218 with HTTP; Mon, 18 Sep 2017 08:41:50 -0700 (PDT) In-Reply-To: References: <1593300.26TW5Pyds6@peggleto-mobl.ger.corp.intel.com> <20170807111740.ppsdinlvltioc3uq@kaim-eeyore.kodakalaris.net> <20170811125251.t2t3kj6kbpsvhwxe@kaim-eeyore.kodakalaris.net> <20170828142759.iiubfgywp2nwz5c6@kaim-eeyore.kodakalaris.net> <20170831175451.23mxpmpof3zdmnll@kaim-eeyore.kodakalaris.net> <20170914161742.wocdydrc2mhxyjy4@kaim-eeyore.kodakalaris.net> From: Katu Txakur Date: Mon, 18 Sep 2017 16:41:50 +0100 Message-ID: To: Andrew Bradford Cc: yocto@yoctoproject.org, Andrew Bradford Subject: Re: Perforce fetcher ignores module and label X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Sep 2017 15:41:54 -0000 Content-Type: multipart/alternative; boundary="001a11c01802d085f80559789497" --001a11c01802d085f80559789497 Content-Type: text/plain; charset="UTF-8" Hello again, Just to say that I applied your patch with git apply --ignore-space-change --ignore-whitespace 0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch and the old Perforce fetcher seems to be working as it used to. I haven't tested it much yet but I will report if I see any problems. Thanks for your help in this issue. Regards, Katu 2017-09-18 11:32 GMT+01:00 Katu Txakur : > Hi Andrew, > > I'm getting an error when I apply your patch. Do you now how can I work > around this? > > user@pc:~/yocto/pyro/poky$ git fetch > user@pc:~/yocto/pyro/poky$ git reset > user@pc:~/yocto/pyro/poky$ git clean -fd > user@pc:~/yocto/pyro/poky$ git reset --hard > HEAD is now at b51b4f5 gawk: Enable native building > user@pc:~/yocto/pyro/poky$ git branch > * pyro > user@pc:~/yocto/pyro/poky$ git apply ../0001-Revert-w- > modifications-bitbake-fetch2-perforce-Rewor.patch > ../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:28: > trailing whitespace. > 'DBUS_SESSION_BUS_ADDRESS'] > ../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:41: > trailing whitespace. > BitBake 'Fetch' implementations > ../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:42: > trailing whitespace. > > ../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:43: > trailing whitespace. > Classes for obtaining upstream sources for the > ../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch:44: > trailing whitespace. > BitBake build tools. > error: patch failed: bitbake/lib/bb/fetch2/__init__.py:817 > error: bitbake/lib/bb/fetch2/__init__.py: patch does not apply > error: patch failed: bitbake/lib/bb/fetch2/perforce.py:1 > error: bitbake/lib/bb/fetch2/perforce.py: patch does not apply > > Cheers, > Katu > > 2017-09-14 17:17 GMT+01:00 Andrew Bradford : > >> Hi Katu, >> >> Sorry for my delay in responding. >> >> On 09/14 11:13, Katu Txakur wrote: >> > Have you had time to look at this? I tried to go back to the old >> perforce >> > fetcher but I got a license error and I couldn't work around it. >> >> On the pyro branch of poky, if you revert these two commits, in this >> order: >> >> 35081f55185a8a9804c21b16cd4600ba70646a10 "bitbake: bitbake-user-manual: >> Addeds support for the Perforce Fetcher" >> 40f3099e8ec5c6f3a8b7f3d0e90f1c65c388ee77 "base.bbclass: p4 fetcher >> supports srcrev" >> >> And then apply the patch attached, this should bring the pyro branch of >> poky back to how the old perforce fetcher worked. I have tested having >> two different depot paths in a recipe SRC_URI and they both are fetched >> how I believe you want them to be, next to each other in WORKDIR. >> >> Just be aware, I believe this loses the ability to use >> SRC_REV="${AUTOREV}" to automatically find the newest changelist which >> my previous changes introduced. >> >> The format for SRC_URI that I tested was like: >> >> SRC_URI = "p4://username:password:server.hostname.example.com:1666@ >> depot/path/to/directory/...;cset=178374" >> >> You should also be able to specify P4PORT to set the host and port >> number but you may then lose the ability to have the username and >> password (although I haven't tested this and don't remember how it used >> to work). You should also be able to specify the P4DATE variable in >> your recipe to apply to all p4 fetches instead of using the "cset=" >> parameter, and instead of using the "cset=" param you should also be >> able to specify a "revision=" for a single file or a "label=" for a >> label. I've only tested using the "cset=" way as the others don't >> easily apply to how my team internally uses perforce, sorry. >> >> I think I understand how I could make the current perforce fetcher >> (without the above reverts or attached patch) do the multi-directory >> fetching that you want, but I don't personally want to do that in my >> workflow so I'm not going to spend a bunch of time implementing it now. >> But if you do implement it, I'd be happy to test patches for you. >> >> Thanks, >> Andrew >> >> > 2017-08-31 18:54 GMT+01:00 Andrew Bradford > >: >> > >> > > Hi Katu, >> > > >> > > On 08/28 17:43, Katu Txakur wrote: >> > > > Thanks for looking at this. See my recipe below. I've left the bits >> > > related >> > > > to systemd service but I don't think they matter for this. I'm >> using an >> > > old >> > > > implementation of Perforce (2010) in case this matters. I've tried >> going >> > > > back to the old perforce.py fetcher but I get a license error... do >> you >> > > > know if it would be easy to revert to the old version in my bitbake >> > > folder >> > > > until we make this work? >> > > >> > > Sorry, I've been swamped this week and haven't been able to look into >> > > this yet. This coming weekend is a holiday weekend in the USA, too. >> > > But I plan to look at this early next week, hope that's OK. >> > > >> > > We should be able to create a patch series to revert my changes so you >> > > can go back to the old perforce fetcher. It might also be worth >> > > investigating how to take the current perforce fetcher and enable some >> > > of the use cases that you have (but we can do this after we >> successfully >> > > revert). >> > > >> > > I'll try to send a patch to you next week for the reverting. >> > > Thanks, >> > > Andrew >> > > >> > > > DESCRIPTION = "Dummy recipe to fetch from Perforce" >> > > > SECTION = "PerforceRecipes" >> > > > LICENSE = "CLOSED" >> > > > PR = "r0" >> > > > >> > > > inherit cmake >> > > > DEPENDS = "boost alsa-tools" >> > > > >> > > > P4USER = "myuser" >> > > > P4PASSWD = "" >> > > > P4PORT = "192.168.1.55:1666" >> > > > FETCHCMD_p4 = "/usr/local/bin/p4" >> > > > >> > > > SRC_URI = " \ >> > > > p4://${P4USER}:${P4PASSWD}@myrepo/myfolder/...; \ >> > > > file://perforce-recipe.service \ >> > > > " >> > > > SRCREV = "${AUTOREV}" >> > > > S = "${WORKDIR}/p4" >> > > > >> > > > do_install_append () { >> > > > install -d ${D}${systemd_unitdir}/system >> > > > install -m 0644 ../perforce-recipe.service >> > > ${D}${systemd_unitdir}/system >> > > > } >> > > > >> > > > NATIVE_SYSTEMD_SUPPORT = "1" >> > > > SYSTEMD_PACKAGES = "perforce-recipe" >> > > > SYSTEMD_SERVICE_${PN} = "perforce-recipe.service" >> > > > >> > > > SYSTEMD_AUTO_ENABLE = "disable" >> > > > >> > > > FILES_${PN} = "${systemd_unitdir} ${bindir}" >> > > > >> > > > >> > > > 2017-08-28 15:27 GMT+01:00 Andrew Bradford < >> andrew@bradfordembedded.com >> > > >: >> > > > >> > > > > Hi Katu, >> > > > > >> > > > > On 08/28 08:56, Katu Txakur wrote: >> > > > > > 2017-08-11 13:52 GMT+01:00 Andrew Bradford < >> > > andrew@bradfordembedded.com >> > > > > >: >> > > > > > On 08/11 11:34, Katu Txakur wrote: >> > > > > > > > 2017-08-07 12:17 GMT+01:00 Andrew Bradford < >> > > > > andrew@bradfordembedded.com >> > > > > > > >: >> > > > > > > > > On 08/02 10:28, Paul Eggleton wrote: >> > > > > > > > > > On Wednesday, 2 August 2017 9:40:10 AM CEST Katu Txakur >> > > wrote: >> > > > > > > > > > > I'm still having problems fetching from Perforce. Is >> there >> > > any >> > > > > > > > > > > documentation based on the latest implementation of >> the >> > > > > > > > > > > poky/bitbake/lib/bb/fetch2/perforce.py file? >> > > > > > > > > >> > > > > > > > > There's some documentation now in the official Yocto >> Project >> > > > > > > > > documentation location for bitbake fetchers [1]. Prior >> to my >> > > > > changes >> > > > > > > > > there was very little if any documentation, afaik. >> > > > > > > > > >> > > > > > > > > [1]: http://www.yoctoproject.org/docs/2.3.1/bitbake-user- >> > > > > > > > > manual/bitbake-user-manual.html#perforce-fetcher >> > > > > > > > > >> > > > > > > > > > > 2017-07-25 10:05 GMT+01:00 Katu Txakur < >> > > katutxakurra@gmail.com >> > > > > >: >> > > > > > > > > > > > I'm upgrading a recipe that fetches the source code >> from >> > > > > > > Perforce. >> > > > > > > > > > > > >> > > > > > > > > > > > The old recipe was: >> > > > > > > > > > > > >> > > > > > > > > > > > SRC_URI = " \ >> > > > > > > > > > > > p4://${P4USER}:${P4PASSWD}:${ >> P4HOST}:${P4PORT}@Depot >> > > /path/ >> > > > > pe >> > > > > > > > > > > > rforce/...;module=local/path/relativeto/p4;label=${ >> > > > > P4CHANGELIST} >> > > > > > > \ >> > > > > > > > > > > > " >> > > > > > > > > > > > >> > > > > > > > > > > > With the new version of /lib/bb/fetch2/perforce.py, >> I >> > > had to >> > > > > set >> > > > > > > > > P4PORT >> > > > > > > > > > > > outside SRC_URI, leaving the recipe with: >> > > > > > > > > > > > >> > > > > > > > > > > > SRC_URI = " \ >> > > > > > > > > > > > p4://${P4USER}:${P4PASSWD}@De >> pot/path/perforce/...; >> > > module= >> > > > > > > > > > > > local/path/relativeto/p4;label=${P4CHANGELIST} \ >> > > > > > > > > > > > " >> > > > > > > > > > > > >> > > > > > > > > > > > The Perforce fetcher kind of works, but it seems to >> be >> > > > > ignoring >> > > > > > > the >> > > > > > > > > > > > "module" and the "label" attributes. After reading >> the >> > > Python >> > > > > > > script >> > > > > > > > > I can >> > > > > > > > > > > > see that after the "@", only the substring before >> the >> > > first >> > > > > ";" >> > > > > > > is >> > > > > > > > > used. >> > > > > > > > > >> > > > > > > > > Sorry for not making the label change more clear, but you >> > > should be >> > > > > > > able >> > > > > > > > > to simply set SRCREV="labelname" and have the >> functionality you >> > > > > desire. >> > > > > > > > > However, we don't use labels very often internally, so >> the use >> > > of >> > > > > > > labels >> > > > > > > > > in SRCREV isn't tested that often by me. >> > > > > > > > > >> > > > > > > > >> > > > > > > > I don't use labels that much either, however, I used to be >> able >> > > to >> > > > > write >> > > > > > > > the changeslist number as a label and it would >> > > > > > > > fetch the code for that changelist. This doesn't work using >> > > SRCREV = >> > > > > > > > "changelistnumber" >> > > > > > > >> > > > > > > This does work for me using oe-core morty branch and bitbake >> > > > > > > 1.32 which is the versions where these changes were first >> applied >> > > and >> > > > > > > where I did my testing. >> > > > > > > >> > > > > > > Like I have a recipe which I put: >> > > > > > > >> > > > > > > SRCREV = "184127" >> > > > > > > >> > > > > > > The p4 fetcher will go fetch changelist 184127 of the SRC_URI >> path >> > > in >> > > > > > > perforce. >> > > > > > > >> > > > > > > Or if I put: >> > > > > > > >> > > > > > > SRCREV = "${AUTOREV}" >> > > > > > > >> > > > > > > Then it'll fetch the HEAD/tip of the SRC_URI path in perforce. >> > > > > > > >> > > > > > > Is this not working the same way with poky now? >> > > > > > > >> > > > > > >> > > > > > I'm using yocto pyro with bitbake 1.34.0. When I first fetch the >> > > code for >> > > > > > my recipe, it gets the latest changelist. >> > > > > > I submit a change and the next time, the recipe thinks there >> nothing >> > > new >> > > > > to >> > > > > > fetch. Also, using SRCREV = "184127" >> > > > > > doesn't work for me, it always gets the latest changelist. >> > > > > >> > > > > Can you post the recipe where this is failing for you? >> > > > > >> > > > > I could then try to recreate this issue in a sandbox depot we >> have in >> > > > > our perforce server and see what's going on. >> > > > > >> > > > > > > >> > > > > > > > > This change to using labels, changelist numbers, and >> allowing >> > > the >> > > > > use >> > > > > > > of >> > > > > > > > > AUTOREV was made so that the perforce fetcher would act >> more >> > > like >> > > > > the >> > > > > > > > > other source code control system fetchers. Prior to my >> > > changes to >> > > > > the >> > > > > > > > > p4 fetcher, the way that recipes using p4 had to be >> written >> > > seemed >> > > > > > > > > rather awkward to me. The way it is now isn't perfect, >> but >> > > it's >> > > > > closer >> > > > > > > > > to how the other fetchers act, I think (and hope). >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > Is there a way to set module and label/changelist? I >> have >> > > > > several >> > > > > > > > > folders >> > > > > > > > > > > > per recipe that I need to map to different >> subfolders and >> > > > > with >> > > > > > > the >> > > > > > > > > current >> > > > > > > > > > > > script some of the folders don't get fetched. >> > > > > > > > > >> > > > > > > > > I'm not sure I understand enough about how you use >> perforce and >> > > > > what >> > > > > > > you >> > > > > > > > > mean by mapping different subfolders. Can you explain >> this in >> > > a >> > > > > more >> > > > > > > > > expanded way to me? Maybe with some examples? >> > > > > > > > > >> > > > > > > > >> > > > > > > > What I mean by this is taking folders in different >> locations in >> > > > > Perforce >> > > > > > > > and fetch them to a custom folder structure inside the >> > > {WORKDIR}/p4 >> > > > > > > folder. >> > > > > > > > This is an example of how I was doing this with the old >> version >> > > of >> > > > > the >> > > > > > > > fetcher: >> > > > > > > > >> > > > > > > > #leave blank, "", for latest revision >> > > > > > > > P4CHANGELIST = "${PR}" >> > > > > > > > S = "${WORKDIR}" >> > > > > > > > SRC_URI = " \ >> > > > > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot >> /project >> > > > > > > 1/folder1/...;module=subfolderunderp4/src/;label=${P4CHANGEL >> IST} >> > > > > > > > \ >> > > > > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot >> /project >> > > > > > > 2/folder3/...;module=subfolderunderp4/pkg/;label=${P4CHANGEL >> IST} >> > > > > > > > \ >> > > > > > > > " >> > > > > > > > >> > > > > > > > Where folder1 contains the src files and folder3 contains >> the pkg >> > > > > > > > information. These need to be in an specific folder >> structure >> > > for the >> > > > > > > > recipe to work. >> > > > > > > > I had some issue when adding more than one line because it >> was >> > > > > trying to >> > > > > > > > copy everything to the same p4 folder. >> > > > > > > >> > > > > > > Ah, yeah... Sorry, that is a bit of a shortcoming of my >> changes. >> > > The >> > > > > > > reason I made this change is because putting the given >> SRC_URI's >> > > > > > > repository directly into a directory like ${WORKDIR}/p4/ is >> how >> > > some of >> > > > > > > the other fetchers operate and it makes the automatic tasks >> just >> > > work >> > > > > > > correctly. This is how the git fetcher does things so I >> tried to >> > > > > mimmic >> > > > > > > that but maybe I missed some nuance in how the git fetcher >> deals >> > > with >> > > > > > > more than one SRC_URI line being a git repo. I hadn't >> realized >> > > that >> > > > > > > there was users of the p4 fetcher who did things like you are >> > > doing, >> > > > > > > sorry. >> > > > > > > >> > > > > > > I think it should be reasonably easy to make a patch to the p4 >> > > fetcher >> > > > > > > which will put each SRC_URI line which uses the p4 fetcher >> into >> > > its own >> > > > > > > directory within the ${WORKDIR}/p4/ directory such that it >> supports >> > > > > your >> > > > > > > style of fetching. This then will likely require your recipe >> to >> > > know >> > > > > > > that this is the directory structure and properly change into >> > > > > > > directories as needed, but it sounds like you already >> comprehend >> > > that. >> > > > > > > And for users of the p4 fetcher who only use one SRC_URI >> line, I >> > > > > believe >> > > > > > > they would just need to set the S variable correctly to the >> > > directory >> > > > > > > name within ${WORKDIR}/p4/ to get the current operation. >> > > > > > >> > > > > > I have been looking at this and it wasn't straight forward for >> me... >> > > I >> > > > > > don't have much experience with this. I will continue trying >> but I >> > > might >> > > > > > need some help. >> > > > > > It's not only not being able to fetch more than one SRC_URI. We >> are >> > > also >> > > > > > loosing the flexibility of fetching different changelists for >> each >> > > line. >> > > > > > I'm not sure if this can be solved in this new fetcher whilst >> > > keeping git >> > > > > > and p4 "similar" as git doesn't work this way afaik. >> > > > > >> > > > > It sounds like you really just want to revert to how the p4 >> fetcher >> > > used >> > > > > to work. My changes were because I wanted the p4 fetcher to work >> more >> > > > > like the git and svn fetchers. But that doesn't sound like what >> you >> > > > > want. >> > > > > >> > > > > My p4 fetcher changes which are impacting you on poky's pyro are >> these: >> > > > > >> > > > > 35081f55185a8a9804c21b16cd4600ba70646a10 bitbake: >> bitbake-user-manual: >> > > > > Addeds support for the Perforce Fetcher >> > > > > 40f3099e8ec5c6f3a8b7f3d0e90f1c65c388ee77 base.bbclass: p4 fetcher >> > > > > supports srcrev >> > > > > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: >> fetch2/perforce: >> > > Rework >> > > > > to support SRCREV and P4CONFIG >> > > > > >> > > > > But that last one will run into conflicts if you just try to >> revert it >> > > > > directly due to having other commits since then which git can't >> easily >> > > > > figure out, but if you revert all of these commits too, then you >> may >> > > > > have to redo some of the changes they made which are related to >> general >> > > > > things and are not p4 specific. Commits you'll want to look at >> are: >> > > > > >> > > > > b16192c93834d0a6530169557aa34122e1417bcf bitbake: fetch2: don't >> use >> > > > > deprecated bb.data APIs >> > > > > 6c611d697f9c03867c938cba1b481f38eebed8bf bitbake: fetch2: Rename >> > > > > "setup_revisons" to "setup_revisions" >> > > > > 38438b6cf42fb7ad45b9a901f57913af7e7591a3 bitbake: fetch2: obey >> > > > > BB_ALLOWED_NETWORKS when checking network access >> > > > > 1fce7ecbbb004a5ad82da3eef79cfd52b276708d bitbake: bitbake: >> remove True >> > > > > option to getVar calls >> > > > > ab09541d5517da9b1a23923ea8f5c26ddf745084 bitbake: fetch2: >> preserve >> > > > > current working directory >> > > > > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: >> fetch2/perforce: >> > > Rework >> > > > > to support SRCREV and P4CONFIG >> > > > > >> > > > > I haven't actually reviewed any of these commits after the >> "preserve >> > > > > current working directory" one as I'm still doing my day-to-day >> work on >> > > > > the morty branch of oe-core and bitbake 1.32. But possibly one of >> > > these >> > > > > commits are unintentionally impacting your ability to specify >> SRCREV >> > > and >> > > > > that's worth checking into. >> > > > > >> > > > > Thanks, >> > > > > Andrew >> > > > > >> > > >> > > --001a11c01802d085f80559789497 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello again,

Just to= say that I applied your patch with

git apply --ignore-space-change= --ignore-whitespace 0001-Revert-w-modifications-bitbake-fetch2-perforce-Re= wor.patch

and the old Perforce fetcher seems to be working as = it used to. I haven't tested it much yet but I will report if I see any= problems.

Thanks for your help in this issue.

Re= gards,
Katu

2017-09-18 11:32 GMT+01:00 Katu Txakur <<= a href=3D"mailto:katutxakurra@gmail.com" target=3D"_blank">katutxakurra@gma= il.com>:
<= div>
Hi Andrew,

I'm getting an error when I apply= your patch. Do you now how can I work around this?

user@pc:~/yocto/= pyro/poky$ git fetch
user@pc:~/yocto/pyro/poky$ git reset
user@pc:~/y= octo/pyro/poky$ git clean -fd
user@pc:~/yocto/pyro/poky$ git reset --har= d
HEAD is now at b51b4f5 gawk: Enable native building
user@pc:~/yocto= /pyro/poky$ git branch
* pyro
user@pc:~/yocto/pyro/poky$ git apply ..= /0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.patch<= br>../0001-Revert-w-modifications-bitbake-fetch2-perforce-Rewor.p= atch:28: trailing whitespace.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 'DBUS_SESS= ION_BUS_ADDRESS']
../0001-Revert-w-modifications-bitbake-fetch2= -perforce-Rewor.patch:41: trailing whitespace.
BitBake 'Fetch&#= 39; implementations
../0001-Revert-w-modifications-bitbake-fetch2-<= wbr>perforce-Rewor.patch:42: trailing whitespace.

../0001-Revert-w-<= wbr>modifications-bitbake-fetch2-perforce-Rewor.patch:43: trailing whi= tespace.
Classes for obtaining upstream sources for the
../0001-Rever= t-w-modifications-bitbake-fetch2-perforce-Rewor.patch:44: trailin= g whitespace.
BitBake build tools.
error: patch failed: bitbake/lib/b= b/fetch2/__init__.py:817
error: bitbake/lib/bb/fetch2/__init__= .py: patch does not apply
error: patch failed: bitbake/lib/bb/fetch2/perforce.py:1
error: bitbake/lib/bb/fetch2/perforce.py: patch doe= s not apply

Cheers,
Katu

2017-09-14 17:17 GMT+01:00 Andrew Bradford <andrew@bradforde= mbedded.com>:
Hi Katu,

Sorry for my delay in responding.

On 09/14 11:13, Katu Txakur wrote:
> Have you had time to look at this? I tried to go back to the old perfo= rce
> fetcher but I got a license error and I couldn't work around it.
On the pyro branch of poky, if you revert these two commits, in this=
order:

35081f55185a8a9804c21b16cd4600ba70646a10 "bitbake: bitbake-user-m= anual: Addeds support for the Perforce Fetcher"
40f3099e8ec5c6f3a8b7f3d0e90f1c65c388ee77 "base.bbclass: p4 fetche= r supports srcrev"

And then apply the patch attached, this should bring the pyro branch= of
poky back to how the old perforce fetcher worked.=C2=A0 I have tested havin= g
two different depot paths in a recipe SRC_URI and they both are fetched
how I believe you want them to be, next to each other in WORKDIR.

Just be aware, I believe this loses the ability to use
SRC_REV=3D"${AUTOREV}" to automatically find the newest changelis= t which
my previous changes introduced.

The format for SRC_URI that I tested was like:

SRC_URI =3D "p4://username:password:server.hostname.example.com:1666@depot/path/to/directory/...;cset=3D178374"

You should also be able to specify P4PORT to set the host and port
number but you may then lose the ability to have the username and
password (although I haven't tested this and don't remember how it = used
to work).=C2=A0 You should also be able to specify the P4DATE variable in your recipe to apply to all p4 fetches instead of using the "cset=3D&q= uot;
parameter, and instead of using the "cset=3D" param you should al= so be
able to specify a "revision=3D" for a single file or a "labe= l=3D" for a
label.=C2=A0 I've only tested using the "cset=3D" way as the = others don't
easily apply to how my team internally uses perforce, sorry.

I think I understand how I could make the current perforce fetcher
(without the above reverts or attached patch) do the multi-directory
fetching that you want, but I don't personally want to do that in my workflow so I'm not going to spend a bunch of time implementing it now.=
But if you do implement it, I'd be happy to test patches for you.

Thanks,
Andrew

> 2017-08-31 18:54 GMT+01:00 Andrew Bradford <andrew@bradfordembedded.com&g= t;:
>
> > Hi Katu,
> >
> > On 08/28 17:43, Katu Txakur wrote:
> > > Thanks for looking at this. See my recipe below. I've le= ft the bits
> > related
> > > to systemd service but I don't think they matter for thi= s. I'm using an
> > old
> > > implementation of Perforce (2010) in case this matters. I= 9;ve tried going
> > > back to the old perforce.py fetcher but I get a license erro= r... do you
> > > know if it would be easy to revert to the old version in my = bitbake
> > folder
> > > until we make this work?
> >
> > Sorry, I've been swamped this week and haven't been able = to look into
> > this yet.=C2=A0 This coming weekend is a holiday weekend in the U= SA, too.
> > But I plan to look at this early next week, hope that's OK. > >
> > We should be able to create a patch series to revert my changes s= o you
> > can go back to the old perforce fetcher.=C2=A0 It might also be w= orth
> > investigating how to take the current perforce fetcher and enable= some
> > of the use cases that you have (but we can do this after we succe= ssfully
> > revert).
> >
> > I'll try to send a patch to you next week for the reverting.<= br> > > Thanks,
> > Andrew
> >
> > > DESCRIPTION =3D "Dummy recipe to fetch from Perforce&qu= ot;
> > > SECTION =3D "PerforceRecipes"
> > > LICENSE =3D "CLOSED"
> > > PR =3D "r0"
> > >
> > > inherit cmake
> > > DEPENDS =3D "boost alsa-tools"
> > >
> > > P4USER =3D "myuser"
> > > P4PASSWD =3D ""
> > > P4PORT =3D "192.168.1.55:1666"
> > > FETCHCMD_p4 =3D "/usr/local/bin/p4"
> > >
> > > SRC_URI =3D " \
> > > p4://${P4USER}:${P4PASSWD}@myrepo/myfolder/...; \
> > > file://perforce-recipe.service \
> > > "
> > > SRCREV =3D "${AUTOREV}"
> > > S =3D "${WORKDIR}/p4"
> > >
> > > do_install_append () {
> > >=C2=A0 =C2=A0 =C2=A0install -d ${D}${systemd_unitdir}/system<= br> > > >=C2=A0 =C2=A0 =C2=A0install -m 0644 ../perforce-recipe.servic= e
> > ${D}${systemd_unitdir}/system
> > > }
> > >
> > > NATIVE_SYSTEMD_SUPPORT =3D "1"
> > > SYSTEMD_PACKAGES =3D "perforce-recipe"
> > > SYSTEMD_SERVICE_${PN} =3D "perforce-recipe.service"= ;
> > >
> > > SYSTEMD_AUTO_ENABLE =3D "disable"
> > >
> > > FILES_${PN} =3D "${systemd_unitdir} ${bindir}"
> > >
> > >
> > > 2017-08-28 15:27 GMT+01:00 Andrew Bradford <andrew@bradfordembedded= .com
> > >:
> > >
> > > > Hi Katu,
> > > >
> > > > On 08/28 08:56, Katu Txakur wrote:
> > > > > 2017-08-11 13:52 GMT+01:00 Andrew Bradford < > > = andrew@bradfordembedded.com
> > > > >:
> > > > > On 08/11 11:34, Katu Txakur wrote:
> > > > > > > 2017-08-07 12:17 GMT+01:00 Andrew Bradfo= rd <
> > > > andrew@bradfordembedded.com
> > > > > > >:
> > > > > > > > On 08/02 10:28, Paul Eggleton wrote= :
> > > > > > > > > On Wednesday, 2 August 2017 9:= 40:10 AM CEST Katu Txakur
> > wrote:
> > > > > > > > > > I'm still having prob= lems fetching from Perforce. Is there
> > any
> > > > > > > > > > documentation based on th= e latest implementation of the
> > > > > > > > > > poky/bitbake/lib/bb/fetch= 2/perforce.py file?
> > > > > > > >
> > > > > > > > There's some documentation now = in the official Yocto Project
> > > > > > > > documentation location for bitbake = fetchers [1].=C2=A0 Prior to my
> > > > changes
> > > > > > > > there was very little if any docume= ntation, afaik.
> > > > > > > >
> > > > > > > > [1]: htt= p://www.yoctoproject.org/docs/2.3.1/bitbake-user-
> > > > > > > > manual/bitbake-user-manual.htm= l#perforce-fetcher
> > > > > > > >
> > > > > > > > > > 2017-07-25 10:05 GMT+01:0= 0 Katu Txakur <
> > katut= xakurra@gmail.com
> > > > >:
> > > > > > > > > > > I'm upgrading a = recipe that fetches the source code from
> > > > > > Perforce.
> > > > > > > > > > >
> > > > > > > > > > > The old recipe was:<= br> > > > > > > > > > > >
> > > > > > > > > > > SRC_URI =3D " \=
> > > > > > > > > > >=C2=A0 =C2=A0p4://${P= 4USER}:${P4PASSWD}:${P4HOST}:${P4PORT}@Depot
> > /path/
> > > > pe
> > > > > > > > > > > rforce/...;module=3D= local/path/relativeto/p4;label=3D${
> > > > P4CHANGELIST}
> > > > > > \
> > > > > > > > > > >=C2=A0 =C2=A0" > > > > > > > > > > >
> > > > > > > > > > > With the new version= of /lib/bb/fetch2/perforce.py, I
> > had to
> > > > set
> > > > > > > > P4PORT
> > > > > > > > > > > outside SRC_URI, lea= ving the recipe with:
> > > > > > > > > > >
> > > > > > > > > > > SRC_URI =3D " \=
> > > > > > > > > > >=C2=A0 =C2=A0p4://${P= 4USER}:${P4PASSWD}@Depot/path/perforce/...;
> > module=3D
> > > > > > > > > > > local/path/relativet= o/p4;label=3D${P4CHANGELIST} \
> > > > > > > > > > >=C2=A0 =C2=A0" > > > > > > > > > > >
> > > > > > > > > > > The Perforce fetcher= kind of works, but it seems to be
> > > > ignoring
> > > > > > the
> > > > > > > > > > > "module" a= nd the "label" attributes. After reading the
> > Python
> > > > > > script
> > > > > > > > I can
> > > > > > > > > > > see that after the &= quot;@", only the substring before the
> > first
> > > > ";"
> > > > > > is
> > > > > > > > used.
> > > > > > > >
> > > > > > > > Sorry for not making the label chan= ge more clear, but you
> > should be
> > > > > > able
> > > > > > > > to simply set SRCREV=3D"labeln= ame" and have the functionality you
> > > > desire.
> > > > > > > > However, we don't use labels ve= ry often internally, so the use
> > of
> > > > > > labels
> > > > > > > > in SRCREV isn't tested that oft= en by me.
> > > > > > > >
> > > > > > >
> > > > > > > I don't use labels that much either,= however, I used to be able
> > to
> > > > write
> > > > > > > the changeslist number as a label and it= would
> > > > > > > fetch the code for that changelist. This= doesn't work using
> > SRCREV =3D
> > > > > > > "changelistnumber"
> > > > > >
> > > > > > This does work for me using oe-core morty bra= nch and bitbake
> > > > > > 1.32 which is the versions where these change= s were first applied
> > and
> > > > > > where I did my testing.
> > > > > >
> > > > > > Like I have a recipe which I put:
> > > > > >
> > > > > > SRCREV =3D "184127"
> > > > > >
> > > > > > The p4 fetcher will go fetch changelist 18412= 7 of the SRC_URI path
> > in
> > > > > > perforce.
> > > > > >
> > > > > > Or if I put:
> > > > > >
> > > > > > SRCREV =3D "${AUTOREV}"
> > > > > >
> > > > > > Then it'll fetch the HEAD/tip of the SRC_= URI path in perforce.
> > > > > >
> > > > > > Is this not working the same way with poky no= w?
> > > > > >
> > > > >
> > > > > I'm using yocto pyro with bitbake 1.34.0. When= I first fetch the
> > code for
> > > > > my recipe, it gets the latest changelist.
> > > > > I submit a change and the next time, the recipe th= inks there nothing
> > new
> > > > to
> > > > > fetch. Also, using SRCREV =3D "184127" > > > > > doesn't work for me, it always gets the latest= changelist.
> > > >
> > > > Can you post the recipe where this is failing for you?<= br> > > > >
> > > > I could then try to recreate this issue in a sandbox de= pot we have in
> > > > our perforce server and see what's going on.
> > > >
> > > > > >
> > > > > > > > This change to using labels, change= list numbers, and allowing
> > the
> > > > use
> > > > > > of
> > > > > > > > AUTOREV was made so that the perfor= ce fetcher would act more
> > like
> > > > the
> > > > > > > > other source code control system fe= tchers.=C2=A0 Prior to my
> > changes to
> > > > the
> > > > > > > > p4 fetcher, the way that recipes us= ing p4 had to be written
> > seemed
> > > > > > > > rather awkward to me.=C2=A0 The way= it is now isn't perfect, but
> > it's
> > > > closer
> > > > > > > > to how the other fetchers act, I th= ink (and hope).
> > > > > > > >
> > > > > > >
> > > > > > > > > > Is there a way to set mod= ule and label/changelist? I have
> > > > several
> > > > > > > > folders
> > > > > > > > > > > per recipe that I ne= ed to map to different subfolders and
> > > > with
> > > > > > the
> > > > > > > > current
> > > > > > > > > > > script some of the f= olders don't get fetched.
> > > > > > > >
> > > > > > > > I'm not sure I understand enoug= h about how you use perforce and
> > > > what
> > > > > > you
> > > > > > > > mean by mapping different subfolder= s.=C2=A0 Can you explain this in
> > a
> > > > more
> > > > > > > > expanded way to me?=C2=A0 Maybe wit= h some examples?
> > > > > > > >
> > > > > > >
> > > > > > > What I mean by this is taking folders in= different locations in
> > > > Perforce
> > > > > > > and fetch them to a custom folder struct= ure inside the
> > {WORKDIR}/p4
> > > > > > folder.
> > > > > > > This is an example of how I was doing th= is with the old version
> > of
> > > > the
> > > > > > > fetcher:
> > > > > > >
> > > > > > > #leave blank, "", for latest r= evision
> > > > > > > P4CHANGELIST =3D "${PR}"
> > > > > > > S =3D "${WORKDIR}"
> > > > > > > SRC_URI =3D " \
> > > > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST= }:${P4PORT}@Depot/project
> > > > > > 1/folder1/...;module=3Dsubfolderunderp4/= src/;label=3D${P4CHANGELIST}
> > > > > > > \
> > > > > > > p4://${P4USER}:${P4PASSWD}:${P4HOST= }:${P4PORT}@Depot/project
> > > > > > 2/folder3/...;module=3Dsubfolderunderp4/= pkg/;label=3D${P4CHANGELIST}
> > > > > > > \
> > > > > > > "
> > > > > > >
> > > > > > > Where folder1 contains the src files and= folder3 contains the pkg
> > > > > > > information. These need to be in an spec= ific folder structure
> > for the
> > > > > > > recipe to work.
> > > > > > > I had some issue when adding more than o= ne line because it was
> > > > trying to
> > > > > > > copy everything to the same p4 folder. > > > > > >
> > > > > > Ah, yeah... Sorry, that is a bit of a shortco= ming of my changes.
> > The
> > > > > > reason I made this change is because putting = the given SRC_URI's
> > > > > > repository directly into a directory like ${W= ORKDIR}/p4/ is how
> > some of
> > > > > > the other fetchers operate and it makes the a= utomatic tasks just
> > work
> > > > > > correctly.=C2=A0 This is how the git fetcher = does things so I tried to
> > > > mimmic
> > > > > > that but maybe I missed some nuance in how th= e git fetcher deals
> > with
> > > > > > more than one SRC_URI line being a git repo.= =C2=A0 I hadn't realized
> > that
> > > > > > there was users of the p4 fetcher who did thi= ngs like you are
> > doing,
> > > > > > sorry.
> > > > > >
> > > > > > I think it should be reasonably easy to make = a patch to the p4
> > fetcher
> > > > > > which will put each SRC_URI line which uses t= he p4 fetcher into
> > its own
> > > > > > directory within the ${WORKDIR}/p4/ directory= such that it supports
> > > > your
> > > > > > style of fetching.=C2=A0 This then will likel= y require your recipe to
> > know
> > > > > > that this is the directory structure and prop= erly change into
> > > > > > directories as needed, but it sounds like you= already comprehend
> > that.
> > > > > > And for users of the p4 fetcher who only use = one SRC_URI line, I
> > > > believe
> > > > > > they would just need to set the S variable co= rrectly to the
> > directory
> > > > > > name within ${WORKDIR}/p4/ to get the current= operation.
> > > > >
> > > > > I have been looking at this and it wasn't stra= ight forward for me...
> > I
> > > > > don't have much experience with this. I will c= ontinue trying but I
> > might
> > > > > need some help.
> > > > > It's not only not being able to fetch more tha= n one SRC_URI. We are
> > also
> > > > > loosing the flexibility of fetching different chan= gelists for each
> > line.
> > > > > I'm not sure if this can be solved in this new= fetcher whilst
> > keeping git
> > > > > and p4 "similar" as git doesn't work= this way afaik.
> > > >
> > > > It sounds like you really just want to revert to how th= e p4 fetcher
> > used
> > > > to work.=C2=A0 My changes were because I wanted the p4 = fetcher to work more
> > > > like the git and svn fetchers.=C2=A0 But that doesn'= ;t sound like what you
> > > > want.
> > > >
> > > > My p4 fetcher changes which are impacting you on poky&#= 39;s pyro are these:
> > > >
> > > > 35081f55185a8a9804c21b16cd4600ba70646a10 bitbake: = bitbake-user-manual:
> > > > Addeds support for the Perforce Fetcher
> > > > 40f3099e8ec5c6f3a8b7f3d0e90f1c65c388ee77 base.bbcl= ass: p4 fetcher
> > > > supports srcrev
> > > > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: = fetch2/perforce:
> > Rework
> > > > to support SRCREV and P4CONFIG
> > > >
> > > > But that last one will run into conflicts if you just t= ry to revert it
> > > > directly due to having other commits since then which g= it can't easily
> > > > figure out, but if you revert all of these commits too,= then you may
> > > > have to redo some of the changes they made which are re= lated to general
> > > > things and are not p4 specific.=C2=A0 Commits you'l= l want to look at are:
> > > >
> > > > b16192c93834d0a6530169557aa34122e1417bcf bitbake: = fetch2: don't use
> > > > deprecated bb.data APIs
> > > > 6c611d697f9c03867c938cba1b481f38eebed8bf bitbake: = fetch2: Rename
> > > > "setup_revisons" to "setup_revisions&quo= t;
> > > > 38438b6cf42fb7ad45b9a901f57913af7e7591a3 bitbake: = fetch2: obey
> > > > BB_ALLOWED_NETWORKS when checking network access
> > > > 1fce7ecbbb004a5ad82da3eef79cfd52b276708d bitbake: = bitbake: remove True
> > > > option to getVar calls
> > > > ab09541d5517da9b1a23923ea8f5c26ddf745084 bitbake: = fetch2: preserve
> > > > current working directory
> > > > 26447a0d0bf5ceaac382a78c1dbd00807dbab706 bitbake: = fetch2/perforce:
> > Rework
> > > > to support SRCREV and P4CONFIG
> > > >
> > > > I haven't actually reviewed any of these commits af= ter the "preserve
> > > > current working directory" one as I'm still do= ing my day-to-day work on
> > > > the morty branch of oe-core and bitbake 1.32.=C2=A0 But= possibly one of
> > these
> > > > commits are unintentionally impacting your ability to s= pecify SRCREV
> > and
> > > > that's worth checking into.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> >


--001a11c01802d085f80559789497--