From mboxrd@z Thu Jan 1 00:00:00 1970 From: Priya Subject: Re: Regarding Outreachy project on Improving CR Dashboard Date: Wed, 13 Apr 2016 22:03:38 +0530 Message-ID: References: <1459898587.7498.95.camel@bitergia.com> <1459979990.7498.156.camel@bitergia.com> <1460051830.7498.210.camel@bitergia.com> <1460361216.7498.242.camel@bitergia.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7948344168117339973==" Return-path: In-Reply-To: <1460361216.7498.242.camel@bitergia.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: "Jesus M. Gonzalez-Barahona" Cc: Lars Kurth , Xen-devel , Daniel Izquierdo List-Id: xen-devel@lists.xenproject.org --===============7948344168117339973== Content-Type: multipart/alternative; boundary=001a11c25fd4ffe11905306056a3 --001a11c25fd4ffe11905306056a3 Content-Type: text/plain; charset=UTF-8 Hello, Forgot to CC the Xen-devel mailing list. I have changed the code to add the tests, and now it is like you need to provide the whole link of the mbox file rather than just the file name. So, $python3 createjson.py --mbox http://lists.xenproject.org/archives/html/mbox/advisory-board-2013-05-2014-02--output new.json should work instead of $ python3 createjson.py --mbox advisory-board-2014-02 --output new.json I forgot to update the README file, and I have done that now. I will try k['data']['Message-ID'] instead, and I would update soon. I'm working on the testing part and I'll complete it soon. *Priya V* Amrita University LinkedIn | GitHub | Bitbucket On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona < jgb@bitergia.com> wrote: > On Fri, 2016-04-08 at 19:33 +0530, Priya wrote: > > Hello, > > > > I tried running the same command in new version of perceval. I found > > the following missing message id errors in perceval_mbox_parse.log > > file. I am working on the testing part and I will be able to finish > > it in one or two days. > > > > You can see the errors here [1] > > > > [1]:http://imgur.com/yVsIoCT > > Hi, Priya. I'm not sure about what exactly is causing your messages, > since I cannot reproduce them (see below). But I still suspect that > they may happen because in current versions of Perceval the data parsed > from an mbox is no longer stored as first level key/data in the > dictionary returned by Perceval for each message, but in data for key > "data", which is itself a dictionary. > > In particular, in the code: > > ------------- > for k in msg_json: > try: > if key == k['Message-ID'].strip('<>'): > k['property'] = key > ------------- > > probably you should be checking for k['data']['Message-ID'] instead of > just k['Message-ID']. > > Please, have a look at how recent versions of Perceval produce the > dictionaries for each message... > > But as I said, I cannot reproduce your error. When running your most > recent code right now (9a5abc47bbab3b06550) with the most recent > Perceval/master code (53efc14001c806f0452) I get: > > -------------------- > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 > createjson.py --mbox advisory-board-2014-02 --output new.json > Traceback (most recent call last): > File "createjson.py", line 96, in > main() > File "createjson.py", line 92, in main > mparser.create_json(args.mbox,args.output) > File "createjson.py", line 59, in create_json > messages = th.message_details(mbox_files) > File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py", > line 338, in message_details > urllib.request.urlretrieve(filename, 'mbox') > File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve > with contextlib.closing(urlopen(url, data)) as fp: > File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen > return opener.open(url, data, timeout) > File "/usr/lib/python3.4/urllib/request.py", line 449, in open > req = Request(fullurl, data) > File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__ > self.full_url = url > File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url > self._parse() > File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse > raise ValueError("unknown url type: %r" % self.full_url) > ValueError: unknown url type: 'advisory-board-2014-02' > ----------------------------- > > Could you please try to checkout and install exactly the same version > of Perceval I'm using, and see if you get the same error? And if the > above problem with the format returned by Perceval persists, maybe you > can fix that too. > > Saludos, > > Jesus. > > -- > Bitergia: http://bitergia.com > /me at Twitter: https://twitter.com/jgbarah > > --001a11c25fd4ffe11905306056a3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello,

Forgot to CC the Xen-devel mailin= g list.

I have changed the code to add the tests, and now= it is=20 like you need to provide the whole link of the mbox file rather than=20 just the file name.=C2=A0

So, =C2=A0$python3 createjson.py --mboxhttp://lists.xenproject.org/archives/html/mbox/advisory-board-2= 013-05-2014-02--output new.json should work instead of

=C2=A0$=C2=A0python3=C2=A0createjson.py --mbox advisory-boar= d-2014-02 --output new.json

I forgot to update the README file, and I have done that=20 now. I will try=C2=A0k['data']['Message-ID'] instead, and I= would update=20 soon.=C2=A0I'm working on the testing part and I'll complete it soo= n.

=


Priya V
Amrita University


On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Go= nzalez-Barahona <jgb@bitergia.com> wrote:
On Fri, 2016-04-08 at 19:33 +0530, Priya wr= ote:
> Hello,
>
> I tried running the same command in new version of perceval.=C2=A0 I f= ound
> the following missing message id errors in perceval_mbox_parse.log
> file. I am working on the testing part and I will be able to finish > it in one or two days.
>
> You can see the errors here [1]
>
> [1]:http://imgur.com/yVsIoCT

Hi, Priya. I'm not sure about what exactly is causing your messa= ges,
since I cannot reproduce them (see below). But I still suspect that
they may happen because in current versions of Perceval the data parsed
from an mbox is no longer stored as first level key/data in the
dictionary returned by Perceval for each message, but in data for key
"data", which is itself a dictionary.

In particular, in the code:

-------------
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 for k in msg_json:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0try:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0if key =3D=3D k['Message-ID'].strip('<>')= :
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0k['property'] =3D key
-------------

probably you should be checking for k['data']['Message-ID']= instead of
just k['Message-ID'].

Please, have a look at how recent versions of Perceval produce the
dictionaries for each message...

But as I said, I cannot reproduce your error. When running your most
recent code right now (9a5abc47bbab3b06550) with the most recent
Perceval/master code (53efc14001c806f0452) I get:

--------------------
(perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
createjson.py --mbox advisory-board-2014-02 --output new.json
Traceback (most recent call last):
=C2=A0 File "createjson.py", line 96, in <module> =C2=A0=C2=A0=C2=A0=C2=A0main()
=C2=A0 File "createjson.py", line 92, in main
=C2=A0=C2=A0=C2=A0=C2=A0mparser.create_json(args.mbox,args.output)
=C2=A0 File "createjson.py", line 59, in create_json
=C2=A0=C2=A0=C2=A0=C2=A0messages =3D th.message_details(mbox_files)
=C2=A0 File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_= r.py",
line 338, in message_details
=C2=A0=C2=A0=C2=A0=C2=A0urllib.request.urlretrieve(filename, 'mbox'= )
=C2=A0 File "/usr/lib/python3.4/urllib/request.py", line 186, in = urlretrieve
=C2=A0=C2=A0=C2=A0=C2=A0with contextlib.closing(urlopen(url, data)) as fp:<= br> =C2=A0 File "/usr/lib/python3.4/urllib/request.py", line 161, in = urlopen
=C2=A0=C2=A0=C2=A0=C2=A0return opener.open(url, data, timeout)
=C2=A0 File "/usr/lib/python3.4/urllib/request.py", line 449, in = open
=C2=A0=C2=A0=C2=A0=C2=A0req =3D Request(fullurl, data)
=C2=A0 File "/usr/lib/python3.4/urllib/request.py", line 267, in = __init__
=C2=A0=C2=A0=C2=A0=C2=A0self.full_url =3D url
=C2=A0 File "/usr/lib/python3.4/urllib/request.py", line 293, in = full_url
=C2=A0=C2=A0=C2=A0=C2=A0self._parse()
=C2=A0 File "/usr/lib/python3.4/urllib/request.py", line 322, in = _parse
=C2=A0=C2=A0=C2=A0=C2=A0raise ValueError("unknown url type: %r" %= self.full_url)
ValueError: unknown url type: 'advisory-board-2014-02'
-----------------------------

Could you please try to checkout and install exactly the same version
of Perceval I'm using, and see if you get the same error? And if the above problem with the format returned by Perceval persists, maybe you
can fix that too.

Saludos,

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Jesus.

--
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


--001a11c25fd4ffe11905306056a3-- --===============7948344168117339973== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwOi8vbGlzdHMueGVuLm9y Zy94ZW4tZGV2ZWwK --===============7948344168117339973==--