* [PATCH 0/1] Add a new task and fix some bugs in distro_check.py
@ 2011-05-16 13:19 Mei Lei
2011-05-16 13:19 ` [PATCH 1/1] Add a new task checklicense " Mei Lei
2011-05-17 21:06 ` [PATCH 0/1] Add a new task " Saul Wold
0 siblings, 2 replies; 4+ messages in thread
From: Mei Lei @ 2011-05-16 13:19 UTC (permalink / raw)
To: poky
Hi Saul,
As Josh's comments, I have revised this patch, please review it.
For the date issue, I can't find the same issue in my and other PRC team members' mailbox. For ruling out other potential causes, I cloned a new tree to rebase my patch, then generate the pull request mail again, hope this issue could disappear.
Thanks
Lei
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: lmei3/checklicense
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lmei3/checklicense
Thanks,
Mei Lei <lei.mei@intel.com>
---
Mei Lei (1):
Add a new task checklicense and fix some bugs in distro_check.py
meta/classes/distrodata.bbclass | 177 ++++++++++++++++++++++-----------------
meta/lib/oe/distro_check.py | 27 ++++--
2 files changed, 119 insertions(+), 85 deletions(-)
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/1] Add a new task checklicense and fix some bugs in distro_check.py
2011-05-16 13:19 [PATCH 0/1] Add a new task and fix some bugs in distro_check.py Mei Lei
@ 2011-05-16 13:19 ` Mei Lei
2011-05-16 17:58 ` Joshua Lock
2011-05-17 21:06 ` [PATCH 0/1] Add a new task " Saul Wold
1 sibling, 1 reply; 4+ messages in thread
From: Mei Lei @ 2011-05-16 13:19 UTC (permalink / raw)
To: poky
distro_check.py: Create a new function called create_log_file to reduce a lot of repeat code in distrodata.bbclass.
We needn't to create log file in function save_distro_check_result, because the log file has been generated in check_eventhandler.
Another bug is that we maybe access the /tmp/Meego-1.0 before we create this file.
Add a judge statement to decide whether we need to create this file firstly.
distrodata.bbclass: Add a new task checklicense to collect missing text license information.
This can help package-report system to know how many recipes are missing license text.
Signed-off-by: Mei Lei <lei.mei@intel.com>
---
meta/classes/distrodata.bbclass | 177 ++++++++++++++++++++++-----------------
meta/lib/oe/distro_check.py | 27 ++++--
2 files changed, 119 insertions(+), 85 deletions(-)
diff --git a/meta/classes/distrodata.bbclass b/meta/classes/distrodata.bbclass
index 905dad7..f24cff8 100644
--- a/meta/classes/distrodata.bbclass
+++ b/meta/classes/distrodata.bbclass
@@ -4,19 +4,9 @@ addhandler distro_eventhandler
python distro_eventhandler() {
if bb.event.getName(e) == "BuildStarted":
- """initialize log files."""
- logpath = bb.data.getVar('LOG_DIR', e.data, 1)
- bb.utils.mkdirhier(logpath)
- logfile = os.path.join(logpath, "distrodata.%s.csv" % bb.data.getVar('DATETIME', e.data, 1))
- if not os.path.exists(logfile):
- slogfile = os.path.join(logpath, "distrodata.csv")
- if os.path.exists(slogfile):
- os.remove(slogfile)
- os.system("touch %s" % logfile)
- os.symlink(logfile, slogfile)
- bb.data.setVar('LOG_FILE', logfile, e.data)
-
- lf = bb.utils.lockfile(logfile + ".lock")
+ import oe.distro_check as dc
+ logfile = dc.create_log_file(e.data, "distrodata.csv")
+ lf = bb.utils.lockfile("%s.lock" % logfile)
f = open(logfile, "a")
f.write("Package,Description,Owner,License,ChkSum,Status,VerMatch,Version,Upsteam,Non-Update,Reason,Recipe Status\n")
f.close()
@@ -33,9 +23,9 @@ python do_distrodata_np() {
bb.note("Package Name: %s" % pn)
import oe.distro_check as dist_check
- tmpdir = bb.data.getVar('TMPDIR', d, 1)
+ tmpdir = bb.data.getVar('TMPDIR', d, True)
distro_check_dir = os.path.join(tmpdir, "distro_check")
- datetime = bb.data.getVar('DATETIME', localdata, 1)
+ datetime = bb.data.getVar('DATETIME', localdata, True)
dist_check.update_distro_data(distro_check_dir, datetime)
if pn.find("-native") != -1:
@@ -111,15 +101,15 @@ python do_distrodata_np() {
addtask distrodata
do_distrodata[nostamp] = "1"
python do_distrodata() {
- logpath = bb.data.getVar('LOG_DIR', d, 1)
+ logpath = bb.data.getVar('LOG_DIR', d, True)
bb.utils.mkdirhier(logpath)
logfile = os.path.join(logpath, "distrodata.csv")
import oe.distro_check as dist_check
localdata = bb.data.createCopy(d)
- tmpdir = bb.data.getVar('TMPDIR', d, 1)
+ tmpdir = bb.data.getVar('TMPDIR', d, True)
distro_check_dir = os.path.join(tmpdir, "distro_check")
- datetime = bb.data.getVar('DATETIME', localdata, 1)
+ datetime = bb.data.getVar('DATETIME', localdata, True)
dist_check.update_distro_data(distro_check_dir, datetime)
pn = bb.data.getVar("PN", d, True)
@@ -189,7 +179,7 @@ python do_distrodata() {
# do the comparison
result = dist_check.compare_in_distro_packages_list(distro_check_dir, localdata)
- lf = bb.utils.lockfile(logfile + ".lock")
+ lf = bb.utils.lockfile("%s.lock" % logfile)
f = open(logfile, "a")
f.write("%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s," % \
(pname, pdesc, maintainer, plicense, pchksum, hasrstatus, vermatch, pcurver, pupver, noupdate, noupdate_reason, rstatus))
@@ -211,19 +201,10 @@ do_distrodataall() {
addhandler checkpkg_eventhandler
python checkpkg_eventhandler() {
if bb.event.getName(e) == "BuildStarted":
- """initialize log files."""
- logpath = bb.data.getVar('LOG_DIR', e.data, 1)
- bb.utils.mkdirhier(logpath)
- logfile = os.path.join(logpath, "checkpkg.%s.csv" % bb.data.getVar('DATETIME', e.data, 1))
- if not os.path.exists(logfile):
- slogfile = os.path.join(logpath, "checkpkg.csv")
- if os.path.exists(slogfile):
- os.remove(slogfile)
- os.system("touch %s" % logfile)
- os.symlink(logfile, slogfile)
- bb.data.setVar('LOG_FILE', logfile, e.data)
-
- lf = bb.utils.lockfile(logfile + ".lock")
+ import oe.distro_check as dc
+ logfile = dc.create_log_file(e.data, "checkpkg.csv")
+
+ lf = bb.utils.lockfile("%s.lock" % logfile)
f = open(logfile, "a")
f.write("Package\tVersion\tUpver\tLicense\tSection\tHome\tRelease\tPriority\tDepends\tBugTracker\tPE\tDescription\tStatus\tTracking\tURI\tMAINTAINER\n")
f.close()
@@ -304,7 +285,7 @@ python do_checkpkg() {
Clear internal url cache as it's a temporary check. Not doing so will have
bitbake check url multiple times when looping through a single url
"""
- fn = bb.data.getVar('FILE', d, 1)
+ fn = bb.data.getVar('FILE', d, True)
bb.fetch2.urldata_cache[fn] = {}
"""
@@ -335,7 +316,7 @@ python do_checkpkg() {
Return new version if success, or else error in "Errxxxx" style
"""
def check_new_dir(url, curver, d):
- pn = bb.data.getVar('PN', d, 1)
+ pn = bb.data.getVar('PN', d, True)
f = tempfile.NamedTemporaryFile(delete=False, prefix="%s-1-" % pn)
status = internal_fetch_wget(url, d, f)
fhtml = f.read()
@@ -394,7 +375,7 @@ python do_checkpkg() {
"""possible to have no version in pkg name, such as spectrum-fw"""
if not re.search("\d+", curname):
return pcurver
- pn = bb.data.getVar('PN', d, 1)
+ pn = bb.data.getVar('PN', d, True)
f = tempfile.NamedTemporaryFile(delete=False, prefix="%s-2-" % pn)
status = internal_fetch_wget(url, d, f)
fhtml = f.read()
@@ -437,35 +418,35 @@ python do_checkpkg() {
f.close()
"""if host hasn't directory information, no need to save tmp file"""
if status != "ErrHostNoDir" and re.match("Err", status):
- logpath = bb.data.getVar('LOG_DIR', d, 1)
+ logpath = bb.data.getVar('LOG_DIR', d, True)
os.system("cp %s %s/" % (f.name, logpath))
os.unlink(f.name)
return status
"""first check whether a uri is provided"""
- src_uri = bb.data.getVar('SRC_URI', d, 1)
+ src_uri = bb.data.getVar('SRC_URI', d, True)
if not src_uri:
return
"""initialize log files."""
- logpath = bb.data.getVar('LOG_DIR', d, 1)
+ logpath = bb.data.getVar('LOG_DIR', d, True)
bb.utils.mkdirhier(logpath)
logfile = os.path.join(logpath, "checkpkg.csv")
"""generate package information from .bb file"""
- pname = bb.data.getVar('PN', d, 1)
- pdesc = bb.data.getVar('DESCRIPTION', d, 1)
- pgrp = bb.data.getVar('SECTION', d, 1)
- pversion = bb.data.getVar('PV', d, 1)
- plicense = bb.data.getVar('LICENSE',d,1)
- psection = bb.data.getVar('SECTION',d,1)
- phome = bb.data.getVar('HOMEPAGE', d, 1)
- prelease = bb.data.getVar('PR',d,1)
- ppriority = bb.data.getVar('PRIORITY',d,1)
- pdepends = bb.data.getVar('DEPENDS',d,1)
- pbugtracker = bb.data.getVar('BUGTRACKER',d,1)
- ppe = bb.data.getVar('PE',d,1)
- psrcuri = bb.data.getVar('SRC_URI',d,1)
+ pname = bb.data.getVar('PN', d, True)
+ pdesc = bb.data.getVar('DESCRIPTION', d, True)
+ pgrp = bb.data.getVar('SECTION', d, True)
+ pversion = bb.data.getVar('PV', d, True)
+ plicense = bb.data.getVar('LICENSE', d, True)
+ psection = bb.data.getVar('SECTION', d, True)
+ phome = bb.data.getVar('HOMEPAGE', d, True)
+ prelease = bb.data.getVar('PR', d, True)
+ ppriority = bb.data.getVar('PRIORITY', d, True)
+ pdepends = bb.data.getVar('DEPENDS', d, True)
+ pbugtracker = bb.data.getVar('BUGTRACKER', d, True)
+ ppe = bb.data.getVar('PE', d, True)
+ psrcuri = bb.data.getVar('SRC_URI', d, True)
found = 0
for uri in src_uri.split():
@@ -483,9 +464,9 @@ python do_checkpkg() {
(type, host, path, user, pswd, parm) = bb.decodeurl(uri)
if type in ['http', 'https', 'ftp']:
- pcurver = bb.data.getVar('PV', d, 1)
+ pcurver = bb.data.getVar('PV', d, True)
else:
- pcurver = bb.data.getVar("SRCREV", d, 1)
+ pcurver = bb.data.getVar("SRCREV", d, True)
if type in ['http', 'https', 'ftp']:
newver = pcurver
@@ -509,7 +490,7 @@ python do_checkpkg() {
newver = check_new_dir(alturi, dirver, d)
altpath = path
if not re.match("Err", newver) and dirver != newver:
- altpath = altpath.replace(dirver, newver, 1)
+ altpath = altpath.replace(dirver, newver, True)
"""Now try to acquire all remote files in current directory"""
if not re.match("Err", newver):
@@ -625,7 +606,7 @@ python do_checkpkg() {
pstatus += ":%s%s" % (host, path)
"""Read from manual distro tracking fields as alternative"""
- pmver = bb.data.getVar("RECIPE_LATEST_VERSION", d, 1)
+ pmver = bb.data.getVar("RECIPE_LATEST_VERSION", d, True)
if not pmver:
pmver = "N/A"
pmstatus = "ErrNoRecipeData"
@@ -639,7 +620,7 @@ python do_checkpkg() {
psrcuri = psrcuri.split()[0]
pdepends = "".join(pdepends.split("\t"))
pdesc = "".join(pdesc.split("\t"))
- lf = bb.utils.lockfile(logfile + ".lock")
+ lf = bb.utils.lockfile("%s.lock" % logfile)
f = open(logfile, "a")
f.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n" % \
(pname,pversion,pupver,plicense,psection, phome,prelease, ppriority,pdepends,pbugtracker,ppe,pdesc,pstatus,pmver,psrcuri,maintainer))
@@ -654,25 +635,12 @@ do_checkpkgall() {
:
}
-#addhandler check_eventhandler
-python check_eventhandler() {
+addhandler distro_check_eventhandler
+python distro_check_eventhandler() {
if bb.event.getName(e) == "BuildStarted":
- import oe.distro_check as dc
- tmpdir = bb.data.getVar('TMPDIR', e.data, 1)
- distro_check_dir = os.path.join(tmpdir, "distro_check")
- datetime = bb.data.getVar('DATETIME', e.data, 1)
"""initialize log files."""
- logpath = bb.data.getVar('LOG_DIR', e.data, 1)
- bb.utils.mkdirhier(logpath)
- logfile = os.path.join(logpath, "distrocheck.%s.csv" % bb.data.getVar('DATETIME', e.data, 1))
- if not os.path.exists(logfile):
- slogfile = os.path.join(logpath, "distrocheck.csv")
- if os.path.exists(slogfile):
- os.remove(slogfile)
- os.system("touch %s" % logfile)
- os.symlink(logfile, slogfile)
- bb.data.setVar('LOG_FILE', logfile, e.data)
-
+ import oe.distro_check as dc
+ result_file = dc.create_log_file(e.data, "distrocheck.csv")
return
}
@@ -681,18 +649,23 @@ do_distro_check[nostamp] = "1"
python do_distro_check() {
"""checks if the package is present in other public Linux distros"""
import oe.distro_check as dc
+ import bb
+ import shutil
localdata = bb.data.createCopy(d)
bb.data.update_data(localdata)
- tmpdir = bb.data.getVar('TMPDIR', d, 1)
+ tmpdir = bb.data.getVar('TMPDIR', d, True)
distro_check_dir = os.path.join(tmpdir, "distro_check")
- datetime = bb.data.getVar('DATETIME', localdata, 1)
+ logpath = bb.data.getVar('LOG_DIR', d, True)
+ bb.utils.mkdirhier(logpath)
+ result_file = os.path.join(logpath, "distrocheck.csv")
+ datetime = bb.data.getVar('DATETIME', localdata, True)
dc.update_distro_data(distro_check_dir, datetime)
# do the comparison
result = dc.compare_in_distro_packages_list(distro_check_dir, d)
# save the results
- dc.save_distro_check_result(result, datetime, d)
+ dc.save_distro_check_result(result, datetime, result_file, d)
}
addtask distro_checkall after do_distro_check
@@ -701,3 +674,55 @@ do_distro_checkall[nostamp] = "1"
do_distro_checkall() {
:
}
+#
+#Check Missing License Text.
+#Use this task to generate the missing license text data for pkg-report system,
+#then we can search those recipes which license text isn't exsit in common-licenses directory
+#
+addhandler checklicense_eventhandler
+python checklicense_eventhandler() {
+ if bb.event.getName(e) == "BuildStarted":
+ """initialize log files."""
+ import oe.distro_check as dc
+ logfile = dc.create_log_file(e.data, "missinglicense.csv")
+ lf = bb.utils.lockfile("%s.lock" % logfile)
+ f = open(logfile, "a")
+ f.write("Package\tLicense\tMissingLicense\n")
+ f.close()
+ bb.utils.unlockfile(lf)
+ return
+}
+
+addtask checklicense
+do_checklicense[nostamp] = "1"
+python do_checklicense() {
+ import os
+ import bb
+ import shutil
+ logpath = bb.data.getVar('LOG_DIR', d, True)
+ bb.utils.mkdirhier(logpath)
+ pn = bb.data.getVar('PN', d, True)
+ logfile = os.path.join(logpath, "missinglicense.csv")
+ generic_directory = bb.data.getVar('COMMON_LICENSE_DIR', d, True)
+ license_types = bb.data.getVar('LICENSE', d, True)
+ for license_type in ((license_types.replace('+', '').replace('|', '&')
+ .replace('(', '').replace(')', '').replace(';', '')
+ .replace(',', '').replace(" ", "").split("&"))):
+ if not os.path.isfile(os.path.join(generic_directory, license_type)):
+ lf = bb.utils.lockfile("%s.lock" % logfile)
+ f = open(logfile, "a")
+ f.write("%s\t%s\t%s\n" % \
+ (pn,license_types,license_type))
+ f.close()
+ bb.utils.unlockfile(lf)
+ return
+}
+
+addtask checklicenseall after do_checklicense
+do_checklicenseall[recrdeptask] = "do_checklicense"
+do_checklicenseall[nostamp] = "1"
+do_checklicenseall() {
+ :
+}
+
+
diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
index c85d4fb..55cdcad 100644
--- a/meta/lib/oe/distro_check.py
+++ b/meta/lib/oe/distro_check.py
@@ -73,7 +73,8 @@ def clean_package_list(package_list):
def get_latest_released_meego_source_package_list():
"Returns list of all the name os packages in the latest meego distro"
-
+ if not os.path.isfile("/tmp/Meego-1.0"):
+ os.mknod("/tmp/Meego-1.0")
f = open("/tmp/Meego-1.0", "r")
package_names = []
for line in f:
@@ -341,7 +342,22 @@ def compare_in_distro_packages_list(distro_check_dir, d):
bb.note("Matching: %s" % matching_distros)
return matching_distros
-def save_distro_check_result(result, datetime, d):
+def create_log_file(d, logname):
+ logpath = bb.data.getVar('LOG_DIR', d, True)
+ bb.utils.mkdirhier(logpath)
+ logfn, logsuffix = os.path.splitext(logname)
+ logfile = os.path.join(logpath, "%s.%s%s" % (logfn, bb.data.getVar('DATETIME', d, True), logsuffix))
+ if not os.path.exists(logfile):
+ slogfile = os.path.join(logpath, logname)
+ if os.path.exists(slogfile):
+ os.remove(slogfile)
+ os.system("touch %s" % logfile)
+ os.symlink(logfile, slogfile)
+ bb.data.setVar('LOG_FILE', logfile, d)
+ return logfile
+
+
+def save_distro_check_result(result, datetime, result_file, d):
pn = bb.data.getVar('PN', d, True)
logdir = bb.data.getVar('LOG_DIR', d, True)
if not logdir:
@@ -349,16 +365,9 @@ def save_distro_check_result(result, datetime, d):
return
if not os.path.isdir(logdir):
os.makedirs(logdir)
- result_file = os.path.join(logdir, "distrocheck.%s.csv" % datetime)
line = pn
for i in result:
line = line + "," + i
- if not os.path.exists(result_file):
- sresult_file = os.path.join(logdir, "distrocheck.csv")
- if os.path.exists(sresult_file):
- os.remove(sresult_file)
- os.system("touch %s" % result_file)
- os.symlink(result_file, sresult_file)
f = open(result_file, "a")
import fcntl
fcntl.lockf(f, fcntl.LOCK_EX)
--
1.6.3.3
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 1/1] Add a new task checklicense and fix some bugs in distro_check.py
2011-05-16 13:19 ` [PATCH 1/1] Add a new task checklicense " Mei Lei
@ 2011-05-16 17:58 ` Joshua Lock
0 siblings, 0 replies; 4+ messages in thread
From: Joshua Lock @ 2011-05-16 17:58 UTC (permalink / raw)
To: poky
On Mon, 2011-05-16 at 21:19 +0800, Mei Lei wrote:
> distro_check.py: Create a new function called create_log_file to reduce a lot of repeat code in distrodata.bbclass.
> We needn't to create log file in function save_distro_check_result, because the log file has been generated in check_eventhandler.
> Another bug is that we maybe access the /tmp/Meego-1.0 before we create this file.
> Add a judge statement to decide whether we need to create this file firstly.
> distrodata.bbclass: Add a new task checklicense to collect missing text license information.
> This can help package-report system to know how many recipes are missing license text.
>
> Signed-off-by: Mei Lei <lei.mei@intel.com>
Arguably this should be more than one patch but I'm much happier with
this iteration, thanks.
Acked-by: Joshua Lock <josh@linux.intel.com>
> ---
> meta/classes/distrodata.bbclass | 177 ++++++++++++++++++++++-----------------
> meta/lib/oe/distro_check.py | 27 ++++--
> 2 files changed, 119 insertions(+), 85 deletions(-)
>
> diff --git a/meta/classes/distrodata.bbclass b/meta/classes/distrodata.bbclass
> index 905dad7..f24cff8 100644
> --- a/meta/classes/distrodata.bbclass
> +++ b/meta/classes/distrodata.bbclass
> @@ -4,19 +4,9 @@ addhandler distro_eventhandler
> python distro_eventhandler() {
>
> if bb.event.getName(e) == "BuildStarted":
> - """initialize log files."""
> - logpath = bb.data.getVar('LOG_DIR', e.data, 1)
> - bb.utils.mkdirhier(logpath)
> - logfile = os.path.join(logpath, "distrodata.%s.csv" % bb.data.getVar('DATETIME', e.data, 1))
> - if not os.path.exists(logfile):
> - slogfile = os.path.join(logpath, "distrodata.csv")
> - if os.path.exists(slogfile):
> - os.remove(slogfile)
> - os.system("touch %s" % logfile)
> - os.symlink(logfile, slogfile)
> - bb.data.setVar('LOG_FILE', logfile, e.data)
> -
> - lf = bb.utils.lockfile(logfile + ".lock")
> + import oe.distro_check as dc
> + logfile = dc.create_log_file(e.data, "distrodata.csv")
> + lf = bb.utils.lockfile("%s.lock" % logfile)
> f = open(logfile, "a")
> f.write("Package,Description,Owner,License,ChkSum,Status,VerMatch,Version,Upsteam,Non-Update,Reason,Recipe Status\n")
> f.close()
> @@ -33,9 +23,9 @@ python do_distrodata_np() {
> bb.note("Package Name: %s" % pn)
>
> import oe.distro_check as dist_check
> - tmpdir = bb.data.getVar('TMPDIR', d, 1)
> + tmpdir = bb.data.getVar('TMPDIR', d, True)
> distro_check_dir = os.path.join(tmpdir, "distro_check")
> - datetime = bb.data.getVar('DATETIME', localdata, 1)
> + datetime = bb.data.getVar('DATETIME', localdata, True)
> dist_check.update_distro_data(distro_check_dir, datetime)
>
> if pn.find("-native") != -1:
> @@ -111,15 +101,15 @@ python do_distrodata_np() {
> addtask distrodata
> do_distrodata[nostamp] = "1"
> python do_distrodata() {
> - logpath = bb.data.getVar('LOG_DIR', d, 1)
> + logpath = bb.data.getVar('LOG_DIR', d, True)
> bb.utils.mkdirhier(logpath)
> logfile = os.path.join(logpath, "distrodata.csv")
>
> import oe.distro_check as dist_check
> localdata = bb.data.createCopy(d)
> - tmpdir = bb.data.getVar('TMPDIR', d, 1)
> + tmpdir = bb.data.getVar('TMPDIR', d, True)
> distro_check_dir = os.path.join(tmpdir, "distro_check")
> - datetime = bb.data.getVar('DATETIME', localdata, 1)
> + datetime = bb.data.getVar('DATETIME', localdata, True)
> dist_check.update_distro_data(distro_check_dir, datetime)
>
> pn = bb.data.getVar("PN", d, True)
> @@ -189,7 +179,7 @@ python do_distrodata() {
> # do the comparison
> result = dist_check.compare_in_distro_packages_list(distro_check_dir, localdata)
>
> - lf = bb.utils.lockfile(logfile + ".lock")
> + lf = bb.utils.lockfile("%s.lock" % logfile)
> f = open(logfile, "a")
> f.write("%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s," % \
> (pname, pdesc, maintainer, plicense, pchksum, hasrstatus, vermatch, pcurver, pupver, noupdate, noupdate_reason, rstatus))
> @@ -211,19 +201,10 @@ do_distrodataall() {
> addhandler checkpkg_eventhandler
> python checkpkg_eventhandler() {
> if bb.event.getName(e) == "BuildStarted":
> - """initialize log files."""
> - logpath = bb.data.getVar('LOG_DIR', e.data, 1)
> - bb.utils.mkdirhier(logpath)
> - logfile = os.path.join(logpath, "checkpkg.%s.csv" % bb.data.getVar('DATETIME', e.data, 1))
> - if not os.path.exists(logfile):
> - slogfile = os.path.join(logpath, "checkpkg.csv")
> - if os.path.exists(slogfile):
> - os.remove(slogfile)
> - os.system("touch %s" % logfile)
> - os.symlink(logfile, slogfile)
> - bb.data.setVar('LOG_FILE', logfile, e.data)
> -
> - lf = bb.utils.lockfile(logfile + ".lock")
> + import oe.distro_check as dc
> + logfile = dc.create_log_file(e.data, "checkpkg.csv")
> +
> + lf = bb.utils.lockfile("%s.lock" % logfile)
> f = open(logfile, "a")
> f.write("Package\tVersion\tUpver\tLicense\tSection\tHome\tRelease\tPriority\tDepends\tBugTracker\tPE\tDescription\tStatus\tTracking\tURI\tMAINTAINER\n")
> f.close()
> @@ -304,7 +285,7 @@ python do_checkpkg() {
> Clear internal url cache as it's a temporary check. Not doing so will have
> bitbake check url multiple times when looping through a single url
> """
> - fn = bb.data.getVar('FILE', d, 1)
> + fn = bb.data.getVar('FILE', d, True)
> bb.fetch2.urldata_cache[fn] = {}
>
> """
> @@ -335,7 +316,7 @@ python do_checkpkg() {
> Return new version if success, or else error in "Errxxxx" style
> """
> def check_new_dir(url, curver, d):
> - pn = bb.data.getVar('PN', d, 1)
> + pn = bb.data.getVar('PN', d, True)
> f = tempfile.NamedTemporaryFile(delete=False, prefix="%s-1-" % pn)
> status = internal_fetch_wget(url, d, f)
> fhtml = f.read()
> @@ -394,7 +375,7 @@ python do_checkpkg() {
> """possible to have no version in pkg name, such as spectrum-fw"""
> if not re.search("\d+", curname):
> return pcurver
> - pn = bb.data.getVar('PN', d, 1)
> + pn = bb.data.getVar('PN', d, True)
> f = tempfile.NamedTemporaryFile(delete=False, prefix="%s-2-" % pn)
> status = internal_fetch_wget(url, d, f)
> fhtml = f.read()
> @@ -437,35 +418,35 @@ python do_checkpkg() {
> f.close()
> """if host hasn't directory information, no need to save tmp file"""
> if status != "ErrHostNoDir" and re.match("Err", status):
> - logpath = bb.data.getVar('LOG_DIR', d, 1)
> + logpath = bb.data.getVar('LOG_DIR', d, True)
> os.system("cp %s %s/" % (f.name, logpath))
> os.unlink(f.name)
> return status
>
> """first check whether a uri is provided"""
> - src_uri = bb.data.getVar('SRC_URI', d, 1)
> + src_uri = bb.data.getVar('SRC_URI', d, True)
> if not src_uri:
> return
>
> """initialize log files."""
> - logpath = bb.data.getVar('LOG_DIR', d, 1)
> + logpath = bb.data.getVar('LOG_DIR', d, True)
> bb.utils.mkdirhier(logpath)
> logfile = os.path.join(logpath, "checkpkg.csv")
>
> """generate package information from .bb file"""
> - pname = bb.data.getVar('PN', d, 1)
> - pdesc = bb.data.getVar('DESCRIPTION', d, 1)
> - pgrp = bb.data.getVar('SECTION', d, 1)
> - pversion = bb.data.getVar('PV', d, 1)
> - plicense = bb.data.getVar('LICENSE',d,1)
> - psection = bb.data.getVar('SECTION',d,1)
> - phome = bb.data.getVar('HOMEPAGE', d, 1)
> - prelease = bb.data.getVar('PR',d,1)
> - ppriority = bb.data.getVar('PRIORITY',d,1)
> - pdepends = bb.data.getVar('DEPENDS',d,1)
> - pbugtracker = bb.data.getVar('BUGTRACKER',d,1)
> - ppe = bb.data.getVar('PE',d,1)
> - psrcuri = bb.data.getVar('SRC_URI',d,1)
> + pname = bb.data.getVar('PN', d, True)
> + pdesc = bb.data.getVar('DESCRIPTION', d, True)
> + pgrp = bb.data.getVar('SECTION', d, True)
> + pversion = bb.data.getVar('PV', d, True)
> + plicense = bb.data.getVar('LICENSE', d, True)
> + psection = bb.data.getVar('SECTION', d, True)
> + phome = bb.data.getVar('HOMEPAGE', d, True)
> + prelease = bb.data.getVar('PR', d, True)
> + ppriority = bb.data.getVar('PRIORITY', d, True)
> + pdepends = bb.data.getVar('DEPENDS', d, True)
> + pbugtracker = bb.data.getVar('BUGTRACKER', d, True)
> + ppe = bb.data.getVar('PE', d, True)
> + psrcuri = bb.data.getVar('SRC_URI', d, True)
>
> found = 0
> for uri in src_uri.split():
> @@ -483,9 +464,9 @@ python do_checkpkg() {
>
> (type, host, path, user, pswd, parm) = bb.decodeurl(uri)
> if type in ['http', 'https', 'ftp']:
> - pcurver = bb.data.getVar('PV', d, 1)
> + pcurver = bb.data.getVar('PV', d, True)
> else:
> - pcurver = bb.data.getVar("SRCREV", d, 1)
> + pcurver = bb.data.getVar("SRCREV", d, True)
>
> if type in ['http', 'https', 'ftp']:
> newver = pcurver
> @@ -509,7 +490,7 @@ python do_checkpkg() {
> newver = check_new_dir(alturi, dirver, d)
> altpath = path
> if not re.match("Err", newver) and dirver != newver:
> - altpath = altpath.replace(dirver, newver, 1)
> + altpath = altpath.replace(dirver, newver, True)
>
> """Now try to acquire all remote files in current directory"""
> if not re.match("Err", newver):
> @@ -625,7 +606,7 @@ python do_checkpkg() {
> pstatus += ":%s%s" % (host, path)
>
> """Read from manual distro tracking fields as alternative"""
> - pmver = bb.data.getVar("RECIPE_LATEST_VERSION", d, 1)
> + pmver = bb.data.getVar("RECIPE_LATEST_VERSION", d, True)
> if not pmver:
> pmver = "N/A"
> pmstatus = "ErrNoRecipeData"
> @@ -639,7 +620,7 @@ python do_checkpkg() {
> psrcuri = psrcuri.split()[0]
> pdepends = "".join(pdepends.split("\t"))
> pdesc = "".join(pdesc.split("\t"))
> - lf = bb.utils.lockfile(logfile + ".lock")
> + lf = bb.utils.lockfile("%s.lock" % logfile)
> f = open(logfile, "a")
> f.write("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n" % \
> (pname,pversion,pupver,plicense,psection, phome,prelease, ppriority,pdepends,pbugtracker,ppe,pdesc,pstatus,pmver,psrcuri,maintainer))
> @@ -654,25 +635,12 @@ do_checkpkgall() {
> :
> }
>
> -#addhandler check_eventhandler
> -python check_eventhandler() {
> +addhandler distro_check_eventhandler
> +python distro_check_eventhandler() {
> if bb.event.getName(e) == "BuildStarted":
> - import oe.distro_check as dc
> - tmpdir = bb.data.getVar('TMPDIR', e.data, 1)
> - distro_check_dir = os.path.join(tmpdir, "distro_check")
> - datetime = bb.data.getVar('DATETIME', e.data, 1)
> """initialize log files."""
> - logpath = bb.data.getVar('LOG_DIR', e.data, 1)
> - bb.utils.mkdirhier(logpath)
> - logfile = os.path.join(logpath, "distrocheck.%s.csv" % bb.data.getVar('DATETIME', e.data, 1))
> - if not os.path.exists(logfile):
> - slogfile = os.path.join(logpath, "distrocheck.csv")
> - if os.path.exists(slogfile):
> - os.remove(slogfile)
> - os.system("touch %s" % logfile)
> - os.symlink(logfile, slogfile)
> - bb.data.setVar('LOG_FILE', logfile, e.data)
> -
> + import oe.distro_check as dc
> + result_file = dc.create_log_file(e.data, "distrocheck.csv")
> return
> }
>
> @@ -681,18 +649,23 @@ do_distro_check[nostamp] = "1"
> python do_distro_check() {
> """checks if the package is present in other public Linux distros"""
> import oe.distro_check as dc
> + import bb
> + import shutil
> localdata = bb.data.createCopy(d)
> bb.data.update_data(localdata)
> - tmpdir = bb.data.getVar('TMPDIR', d, 1)
> + tmpdir = bb.data.getVar('TMPDIR', d, True)
> distro_check_dir = os.path.join(tmpdir, "distro_check")
> - datetime = bb.data.getVar('DATETIME', localdata, 1)
> + logpath = bb.data.getVar('LOG_DIR', d, True)
> + bb.utils.mkdirhier(logpath)
> + result_file = os.path.join(logpath, "distrocheck.csv")
> + datetime = bb.data.getVar('DATETIME', localdata, True)
> dc.update_distro_data(distro_check_dir, datetime)
>
> # do the comparison
> result = dc.compare_in_distro_packages_list(distro_check_dir, d)
>
> # save the results
> - dc.save_distro_check_result(result, datetime, d)
> + dc.save_distro_check_result(result, datetime, result_file, d)
> }
>
> addtask distro_checkall after do_distro_check
> @@ -701,3 +674,55 @@ do_distro_checkall[nostamp] = "1"
> do_distro_checkall() {
> :
> }
> +#
> +#Check Missing License Text.
> +#Use this task to generate the missing license text data for pkg-report system,
> +#then we can search those recipes which license text isn't exsit in common-licenses directory
> +#
> +addhandler checklicense_eventhandler
> +python checklicense_eventhandler() {
> + if bb.event.getName(e) == "BuildStarted":
> + """initialize log files."""
> + import oe.distro_check as dc
> + logfile = dc.create_log_file(e.data, "missinglicense.csv")
> + lf = bb.utils.lockfile("%s.lock" % logfile)
> + f = open(logfile, "a")
> + f.write("Package\tLicense\tMissingLicense\n")
> + f.close()
> + bb.utils.unlockfile(lf)
> + return
> +}
> +
> +addtask checklicense
> +do_checklicense[nostamp] = "1"
> +python do_checklicense() {
> + import os
> + import bb
> + import shutil
> + logpath = bb.data.getVar('LOG_DIR', d, True)
> + bb.utils.mkdirhier(logpath)
> + pn = bb.data.getVar('PN', d, True)
> + logfile = os.path.join(logpath, "missinglicense.csv")
> + generic_directory = bb.data.getVar('COMMON_LICENSE_DIR', d, True)
> + license_types = bb.data.getVar('LICENSE', d, True)
> + for license_type in ((license_types.replace('+', '').replace('|', '&')
> + .replace('(', '').replace(')', '').replace(';', '')
> + .replace(',', '').replace(" ", "").split("&"))):
> + if not os.path.isfile(os.path.join(generic_directory, license_type)):
> + lf = bb.utils.lockfile("%s.lock" % logfile)
> + f = open(logfile, "a")
> + f.write("%s\t%s\t%s\n" % \
> + (pn,license_types,license_type))
> + f.close()
> + bb.utils.unlockfile(lf)
> + return
> +}
> +
> +addtask checklicenseall after do_checklicense
> +do_checklicenseall[recrdeptask] = "do_checklicense"
> +do_checklicenseall[nostamp] = "1"
> +do_checklicenseall() {
> + :
> +}
> +
> +
> diff --git a/meta/lib/oe/distro_check.py b/meta/lib/oe/distro_check.py
> index c85d4fb..55cdcad 100644
> --- a/meta/lib/oe/distro_check.py
> +++ b/meta/lib/oe/distro_check.py
> @@ -73,7 +73,8 @@ def clean_package_list(package_list):
> def get_latest_released_meego_source_package_list():
> "Returns list of all the name os packages in the latest meego distro"
>
> -
> + if not os.path.isfile("/tmp/Meego-1.0"):
> + os.mknod("/tmp/Meego-1.0")
> f = open("/tmp/Meego-1.0", "r")
> package_names = []
> for line in f:
> @@ -341,7 +342,22 @@ def compare_in_distro_packages_list(distro_check_dir, d):
> bb.note("Matching: %s" % matching_distros)
> return matching_distros
>
> -def save_distro_check_result(result, datetime, d):
> +def create_log_file(d, logname):
> + logpath = bb.data.getVar('LOG_DIR', d, True)
> + bb.utils.mkdirhier(logpath)
> + logfn, logsuffix = os.path.splitext(logname)
> + logfile = os.path.join(logpath, "%s.%s%s" % (logfn, bb.data.getVar('DATETIME', d, True), logsuffix))
> + if not os.path.exists(logfile):
> + slogfile = os.path.join(logpath, logname)
> + if os.path.exists(slogfile):
> + os.remove(slogfile)
> + os.system("touch %s" % logfile)
> + os.symlink(logfile, slogfile)
> + bb.data.setVar('LOG_FILE', logfile, d)
> + return logfile
> +
> +
> +def save_distro_check_result(result, datetime, result_file, d):
> pn = bb.data.getVar('PN', d, True)
> logdir = bb.data.getVar('LOG_DIR', d, True)
> if not logdir:
> @@ -349,16 +365,9 @@ def save_distro_check_result(result, datetime, d):
> return
> if not os.path.isdir(logdir):
> os.makedirs(logdir)
> - result_file = os.path.join(logdir, "distrocheck.%s.csv" % datetime)
> line = pn
> for i in result:
> line = line + "," + i
> - if not os.path.exists(result_file):
> - sresult_file = os.path.join(logdir, "distrocheck.csv")
> - if os.path.exists(sresult_file):
> - os.remove(sresult_file)
> - os.system("touch %s" % result_file)
> - os.symlink(result_file, sresult_file)
> f = open(result_file, "a")
> import fcntl
> fcntl.lockf(f, fcntl.LOCK_EX)
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH 0/1] Add a new task and fix some bugs in distro_check.py
2011-05-16 13:19 [PATCH 0/1] Add a new task and fix some bugs in distro_check.py Mei Lei
2011-05-16 13:19 ` [PATCH 1/1] Add a new task checklicense " Mei Lei
@ 2011-05-17 21:06 ` Saul Wold
1 sibling, 0 replies; 4+ messages in thread
From: Saul Wold @ 2011-05-17 21:06 UTC (permalink / raw)
To: Mei Lei; +Cc: poky
On 05/16/2011 06:19 AM, Mei Lei wrote:
> Hi Saul,
> As Josh's comments, I have revised this patch, please review it.
> For the date issue, I can't find the same issue in my and other PRC team members' mailbox. For ruling out other potential causes, I cloned a new tree to rebase my patch, then generate the pull request mail again, hope this issue could disappear.
>
> Thanks
> Lei
>
>
> Pull URL: git://git.pokylinux.org/poky-contrib.git
> Branch: lmei3/checklicense
> Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lmei3/checklicense
>
> Thanks,
> Mei Lei<lei.mei@intel.com>
> ---
>
>
> Mei Lei (1):
> Add a new task checklicense and fix some bugs in distro_check.py
>
> meta/classes/distrodata.bbclass | 177 ++++++++++++++++++++++-----------------
> meta/lib/oe/distro_check.py | 27 ++++--
> 2 files changed, 119 insertions(+), 85 deletions(-)
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>
Pulled into oe-core and poky
Thanks
Sau!
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2011-05-17 21:06 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-05-16 13:19 [PATCH 0/1] Add a new task and fix some bugs in distro_check.py Mei Lei
2011-05-16 13:19 ` [PATCH 1/1] Add a new task checklicense " Mei Lei
2011-05-16 17:58 ` Joshua Lock
2011-05-17 21:06 ` [PATCH 0/1] Add a new task " Saul Wold
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.