Regarding Outreachy project on Improving CR Dashboard

Regarding Outreachy project on Improving CR Dashboard

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 1277 bytes --]

Hello all,

I have completed coding the initial task of grouping the email thread using
the Zawinski algorithms and then adding property entity to the json for the
messages that belong to the same email thread.

You can see my git repo [1]. The new.json is the output of my script and
out.json is the output of Perceval.

Also, I have updated the README.md file regarding the execution procedures
in github.

Instructions
============

git clone https://github.com/priya299/Dashboard.git

cd Dashboard

python createjson.py 'Perceval Ouputfile' 'mbox file' 'output_file'

eg: python createjson.py out.json xen-devel-2016-03 new.json

"new.json" json file will be created with each message belong to a single
thread having an additional attribute "property". The property attribute
will have message id of the first message in the thread.

Now, I will be pushing the new.json into the elastic search db[2]. Please
give me your valuable feedback about my progress.

[1]:https://github.com/priya299/Dashboard
[2]:https://www.elastic.co/guide/en/kibana/3.0/import-some-data.html


*Priya V*
Amrita University
LinkedIn
<https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile>
| GitHub <https://github.com/priya299> | Bitbucket
<https://bitbucket.org/priya299/>

[-- Attachment #1.2: Type: text/html, Size: 2657 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Tue, 2016-04-05 at 22:05 +0530, Priya wrote:
> Hello all, 
> 
> I have completed coding the initial task of grouping the email thread
> using the Zawinski algorithms and then adding property entity to the
> json for the messages that belong to the same email thread. 
> 
> You can see my git repo [1]. The new.json is the output of my script
> and out.json is the output of Perceval. 
> 
> Also, I have updated the README.md file regarding the execution
> procedures in github.
> 
> Instructions
> ============
> 
> git clone https://github.com/priya299/Dashboard.git
> 
> cd Dashboard
> 
> python createjson.py 'Perceval Ouputfile' 'mbox file' 'output_file'
> 
> eg: python createjson.py out.json xen-devel-2016-03 new.json
> 
> "new.json" json file will be created with each message belong to a
> single thread having an additional attribute "property". The property
> attribute will have message id of the first message in the thread.
> 
> Now, I will be pushing the new.json into the elastic search db[2].
> Please give me your valuable feedback about my progress. 
> 
> [1]:https://github.com/priya299/Dashboard
> [2]:https://www.elastic.co/guide/en/kibana/3.0/import-some-data.html

Hi, Priya. To begin with, could you please integrate your code with the
Perceval iterator? In other words, you can run Perceval on the mailing
list archive directly from your code, which will render the use of
"out.json" void. That way, the invocation of the script would be more
like:

python createjson.py xen-devel-2016-03 new.json

In other words, create.json would use Perceval to parse the mailing
list archive. For this end, the Perceval mbox backend is a class, which
once instantiated, provides an iterator function, fetch(), that you can
run inside a loop. For each iteration of the loop, you get the
equivalent to a JSON element in out.json.

The code would be similar to:

-------------------------------
import perceval

mbox_parser = perceval.backends.mbox.MBox(
  origin=mbox_url,
  dirpath=mbox_file_name
)
for item in mbox_parser.fetch():
  thread_id = find_thread(item)
  ...
---------------------------------

Some details about the Perceval mbox class:

http://perceval.readthedocs.org/en/master/perceval.backends.html#module
-perceval.backends.mbox

If you have trouble running the Perceval backend as an iterator, please
let me know.

In addition, you can use argparse for reading the arguments in the
command line. It is easy and convenient.

Saludos,

	Jesus.

> 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 499 bytes --]

Hello,

Thanks for your suggestions.
I have made the appropriate changes as you had mentioned.
It took a little time to change from python3 to python3.4 as perceval
supports python3.4. I have updated the changes in my github. You can see my
git repo [1]

[1]:https://github.com/priya299/Dashboard


*Priya V*
Amrita University
LinkedIn
<https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile>
| GitHub <https://github.com/priya299> | Bitbucket
<https://bitbucket.org/priya299/>

[-- Attachment #1.2: Type: text/html, Size: 1486 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Wed, 2016-04-06 at 17:30 +0530, Priya wrote:
> Hello,
> 
> Thanks for your suggestions.
> I have made the appropriate changes as you had mentioned.
> It took a little time to change from python3 to python3.4 as perceval
> supports python3.4. I have updated the changes in my github. You can
> see my git repo [1]
> 
> [1]:https://github.com/priya299/Dashboard

Thanks a lot, Priya. Good work. Some preliminary comments, below.

* When runing the script on the xen-devel-2016-03 mbox, I seen an
exception raised:

------------
(perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
createjson.py --mbox xen-devel-2016-03 --output new.json
Traceback (most recent call last):
  File "createjson.py", line 61, in <module>
    create_json(args.mbox,args.output)
  File "createjson.py", line 43, in create_json
    if key == k['Message-ID'].strip('<>'):
KeyError: 'Message-ID'
------------

Maybe some message is not having a Message-ID field? I suggest that you
capture this exception, print out the offending message, and go on with
the next one. You can use the Python logging package for printing out
this kind of information (you can see how to use it in the Perceval
package itself). But see below.

* Minor typo in the README:

Instead of 

eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json

it should be

eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output new.json

* The files have no licensing info. If you agree, it could be GPLv3, as
is Perceval itself. For that, it would be enough that you mimic the
header in Perceval files in your Python files (of course, indicating
your authorship information).

* Which version of Perceval are you using? Some weeks ago, the format
of the dictionary produced by Perceval for each message changed. Now
the  actual fields of the message are in a data subdictionary. Please,
check that: the above exception with respect to the Message-ID key
could be because of this... Please, try to make it work with master
HEAD for Perceval (I don't expect any new major change in the next
days/weeks, and I'll try to warn you in case some happens).

* Could you please write at least one unit test for your code? You can
see examples of the testing schema we use in the tests directory in
Perceval, but we use vanilla unittest (the Python package for tests).
At this stage I don't need that you produce a whole set of tests, only
one or two to show that you know how to write unit tests, please.

Saludos,

	Jesus.

> Priya V
> Amrita University
> LinkedIn | GitHub | Bitbucket
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 3743 bytes --]

Hello all,

Thanks for the suggestions. I have updated the changes as u had mentioned.
I am sorry, but I could not find any errors while running

$ python3 createjson.py --mbox xen-devel-2016-03 --output new.json

command. I am wondering what is wrong with this and might be a problem with
python3 or my perceval version. I have added licensing and python logging.
You can see it in my github repo [1]. I will try upgrading perceval and
adding in the tests in the coming days, and will update.

[1]: https://github.com/priya299/Dashboard


*Priya V*
Amrita University
LinkedIn
<https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile>
| GitHub <https://github.com/priya299> | Bitbucket
<https://bitbucket.org/priya299/>


On Thu, Apr 7, 2016 at 3:29 AM, Jesus M. Gonzalez-Barahona <jgb@bitergia.com
> wrote:

> On Wed, 2016-04-06 at 17:30 +0530, Priya wrote:
> > Hello,
> >
> > Thanks for your suggestions.
> > I have made the appropriate changes as you had mentioned.
> > It took a little time to change from python3 to python3.4 as perceval
> > supports python3.4. I have updated the changes in my github. You can
> > see my git repo [1]
> >
> > [1]:https://github.com/priya299/Dashboard
>
> Thanks a lot, Priya. Good work. Some preliminary comments, below.
>
> * When runing the script on the xen-devel-2016-03 mbox, I seen an
> exception raised:
>
> ------------
> (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
> createjson.py --mbox xen-devel-2016-03 --output new.json
> Traceback (most recent call last):
>   File "createjson.py", line 61, in <module>
>     create_json(args.mbox,args.output)
>   File "createjson.py", line 43, in create_json
>     if key == k['Message-ID'].strip('<>'):
> KeyError: 'Message-ID'
> ------------
>
> Maybe some message is not having a Message-ID field? I suggest that you
> capture this exception, print out the offending message, and go on with
> the next one. You can use the Python logging package for printing out
> this kind of information (you can see how to use it in the Perceval
> package itself). But see below.
>
> * Minor typo in the README:
>
> Instead of
>
> eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json
>
> it should be
>
> eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output new.json
>
> * The files have no licensing info. If you agree, it could be GPLv3, as
> is Perceval itself. For that, it would be enough that you mimic the
> header in Perceval files in your Python files (of course, indicating
> your authorship information).
>
> * Which version of Perceval are you using? Some weeks ago, the format
> of the dictionary produced by Perceval for each message changed. Now
> the  actual fields of the message are in a data subdictionary. Please,
> check that: the above exception with respect to the Message-ID key
> could be because of this... Please, try to make it work with master
> HEAD for Perceval (I don't expect any new major change in the next
> days/weeks, and I'll try to warn you in case some happens).
>
> * Could you please write at least one unit test for your code? You can
> see examples of the testing schema we use in the tests directory in
> Perceval, but we use vanilla unittest (the Python package for tests).
> At this stage I don't need that you produce a whole set of tests, only
> one or two to show that you know how to write unit tests, please.
>
> Saludos,
>
>         Jesus.
>
> > Priya V
> > Amrita University
> > LinkedIn | GitHub | Bitbucket
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> --
> Bitergia: http://bitergia.com
> /me at Twitter: https://twitter.com/jgbarah
>
>

[-- Attachment #1.2: Type: text/html, Size: 5752 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Thu, 2016-04-07 at 17:57 +0530, Priya wrote:
> Hello all,
> 
> Thanks for the suggestions. I have updated the changes as u had
> mentioned. I am sorry, but I could not find any errors while running 
> 
> $ python3 createjson.py --mbox xen-devel-2016-03 --output new.json 
> 
> command. I am wondering what is wrong with this and might be a
> problem with python3 or my perceval version. I have added licensing
> and python logging. You can see it in my github repo [1]. I will try
> upgrading perceval and adding in the tests in the coming days, and
> will update.

Priya, I guess I didn't explain the source for the error clearly
enough, sorry about that. It is very likely that you're not running the
latest version of Perceval, which changed the format for the items it
produces some days ago. That's why, if you're running a version of
Perceval of more than some days ago, it works, but if you run the
latest one, it doesn't work (at least for me). I'm not completely sure,
but that's why I asked you to upgrade to the latest version (either
master HEAD or version 0.1.0). If you don't see the error once you do
this, please let me know.

Thanks,

	Jesus.

> [1]: https://github.com/priya299/Dashboard
> 
> Priya V
> Amrita University
> LinkedIn | GitHub | Bitbucket
> 
> 
> On Thu, Apr 7, 2016 at 3:29 AM, Jesus M. Gonzalez-Barahona <jgb@biter
> gia.com> wrote:
> > On Wed, 2016-04-06 at 17:30 +0530, Priya wrote:
> > > Hello,
> > >
> > > Thanks for your suggestions.
> > > I have made the appropriate changes as you had mentioned.
> > > It took a little time to change from python3 to python3.4 as
> > perceval
> > > supports python3.4. I have updated the changes in my github. You
> > can
> > > see my git repo [1]
> > >
> > > [1]:https://github.com/priya299/Dashboard
> > 
> > Thanks a lot, Priya. Good work. Some preliminary comments, below.
> > 
> > * When runing the script on the xen-devel-2016-03 mbox, I seen an
> > exception raised:
> > 
> > ------------
> > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
> > createjson.py --mbox xen-devel-2016-03 --output new.json
> > Traceback (most recent call last):
> >   File "createjson.py", line 61, in <module>
> >     create_json(args.mbox,args.output)
> >   File "createjson.py", line 43, in create_json
> >     if key == k['Message-ID'].strip('<>'):
> > KeyError: 'Message-ID'
> > ------------
> > 
> > Maybe some message is not having a Message-ID field? I suggest that
> > you
> > capture this exception, print out the offending message, and go on
> > with
> > the next one. You can use the Python logging package for printing
> > out
> > this kind of information (you can see how to use it in the Perceval
> > package itself). But see below.
> > 
> > * Minor typo in the README:
> > 
> > Instead of 
> > 
> > eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json
> > 
> > it should be
> > 
> > eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output
> > new.json
> > 
> > * The files have no licensing info. If you agree, it could be
> > GPLv3, as
> > is Perceval itself. For that, it would be enough that you mimic the
> > header in Perceval files in your Python files (of course,
> > indicating
> > your authorship information).
> > 
> > * Which version of Perceval are you using? Some weeks ago, the
> > format
> > of the dictionary produced by Perceval for each message changed.
> > Now
> > the  actual fields of the message are in a data subdictionary.
> > Please,
> > check that: the above exception with respect to the Message-ID key
> > could be because of this... Please, try to make it work with master
> > HEAD for Perceval (I don't expect any new major change in the next
> > days/weeks, and I'll try to warn you in case some happens).
> > 
> > * Could you please write at least one unit test for your code? You
> > can
> > see examples of the testing schema we use in the tests directory in
> > Perceval, but we use vanilla unittest (the Python package for
> > tests).
> > At this stage I don't need that you produce a whole set of tests,
> > only
> > one or two to show that you know how to write unit tests, please.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > > Priya V
> > > Amrita University
> > > LinkedIn | GitHub | Bitbucket
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 503 bytes --]

Hello,

I tried running the same command in new version of perceval.  I found the
following missing message id errors in perceval_mbox_parse.log file. I am
working on the testing part and I will be able to finish it in one or two
days.

You can see the errors here [1]

[1]:http://imgur.com/yVsIoCT




*Priya V*
Amrita University
LinkedIn
<https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile>
| GitHub <https://github.com/priya299> | Bitbucket
<https://bitbucket.org/priya299/>

[-- Attachment #1.2: Type: text/html, Size: 1455 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

On Fri, 2016-04-08 at 19:33 +0530, Priya wrote:
> Hello,
> 
> I tried running the same command in new version of perceval.  I found
> the following missing message id errors in perceval_mbox_parse.log
> file. I am working on the testing part and I will be able to finish
> it in one or two days.
> 
> You can see the errors here [1]
> 
> [1]:http://imgur.com/yVsIoCT

Hi, Priya. I'm not sure about what exactly is causing your messages,
since I cannot reproduce them (see below). But I still suspect that
they may happen because in current versions of Perceval the data parsed
from an mbox is no longer stored as first level key/data in the
dictionary returned by Perceval for each message, but in data for key
"data", which is itself a dictionary.

In particular, in the code:

-------------
              for k in msg_json:
                    try:
                        if key == k['Message-ID'].strip('<>'):
                            k['property'] = key
-------------

probably you should be checking for k['data']['Message-ID'] instead of
just k['Message-ID'].

Please, have a look at how recent versions of Perceval produce the
dictionaries for each message...

But as I said, I cannot reproduce your error. When running your most
recent code right now (9a5abc47bbab3b06550) with the most recent
Perceval/master code (53efc14001c806f0452) I get:

--------------------
(perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
createjson.py --mbox advisory-board-2014-02 --output new.json
Traceback (most recent call last):
  File "createjson.py", line 96, in <module>
    main()
  File "createjson.py", line 92, in main
    mparser.create_json(args.mbox,args.output)
  File "createjson.py", line 59, in create_json
    messages = th.message_details(mbox_files)
  File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py",
line 338, in message_details
    urllib.request.urlretrieve(filename, 'mbox')
  File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 449, in open
    req = Request(fullurl, data)
  File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__
    self.full_url = url
  File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url
    self._parse()
  File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'advisory-board-2014-02'
-----------------------------

Could you please try to checkout and install exactly the same version
of Perceval I'm using, and see if you get the same error? And if the
above problem with the format returned by Perceval persists, maybe you
can fix that too.

Saludos,

	Jesus.

-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 4046 bytes --]

Hello,

Forgot to CC the Xen-devel mailing list.

I have changed the code to add the tests, and now it is like you need to
provide the whole link of the mbox file rather than just the file name.

So,  $python3 createjson.py --mbox
http://lists.xenproject.org/archives/html/mbox/advisory-board-2013-05-2014-02--output
new.json should work instead of

 $ python3 createjson.py --mbox advisory-board-2014-02 --output new.json

I forgot to update the README file, and I have done that now. I will
try k['data']['Message-ID'] instead, and I would update soon. I'm working
on the testing part and I'll complete it soon.



*Priya V*
Amrita University
LinkedIn
<https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile>
| GitHub <https://github.com/priya299> | Bitbucket
<https://bitbucket.org/priya299/>


On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona <
jgb@bitergia.com> wrote:

> On Fri, 2016-04-08 at 19:33 +0530, Priya wrote:
> > Hello,
> >
> > I tried running the same command in new version of perceval.  I found
> > the following missing message id errors in perceval_mbox_parse.log
> > file. I am working on the testing part and I will be able to finish
> > it in one or two days.
> >
> > You can see the errors here [1]
> >
> > [1]:http://imgur.com/yVsIoCT
>
> Hi, Priya. I'm not sure about what exactly is causing your messages,
> since I cannot reproduce them (see below). But I still suspect that
> they may happen because in current versions of Perceval the data parsed
> from an mbox is no longer stored as first level key/data in the
> dictionary returned by Perceval for each message, but in data for key
> "data", which is itself a dictionary.
>
> In particular, in the code:
>
> -------------
>               for k in msg_json:
>                     try:
>                         if key == k['Message-ID'].strip('<>'):
>                             k['property'] = key
> -------------
>
> probably you should be checking for k['data']['Message-ID'] instead of
> just k['Message-ID'].
>
> Please, have a look at how recent versions of Perceval produce the
> dictionaries for each message...
>
> But as I said, I cannot reproduce your error. When running your most
> recent code right now (9a5abc47bbab3b06550) with the most recent
> Perceval/master code (53efc14001c806f0452) I get:
>
> --------------------
> (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
> createjson.py --mbox advisory-board-2014-02 --output new.json
> Traceback (most recent call last):
>   File "createjson.py", line 96, in <module>
>     main()
>   File "createjson.py", line 92, in main
>     mparser.create_json(args.mbox,args.output)
>   File "createjson.py", line 59, in create_json
>     messages = th.message_details(mbox_files)
>   File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py",
> line 338, in message_details
>     urllib.request.urlretrieve(filename, 'mbox')
>   File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
>     with contextlib.closing(urlopen(url, data)) as fp:
>   File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
>     return opener.open(url, data, timeout)
>   File "/usr/lib/python3.4/urllib/request.py", line 449, in open
>     req = Request(fullurl, data)
>   File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__
>     self.full_url = url
>   File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url
>     self._parse()
>   File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse
>     raise ValueError("unknown url type: %r" % self.full_url)
> ValueError: unknown url type: 'advisory-board-2014-02'
> -----------------------------
>
> Could you please try to checkout and install exactly the same version
> of Perceval I'm using, and see if you get the same error? And if the
> above problem with the format returned by Perceval persists, maybe you
> can fix that too.
>
> Saludos,
>
>         Jesus.
>
> --
> Bitergia: http://bitergia.com
> /me at Twitter: https://twitter.com/jgbarah
>
>

[-- Attachment #1.2: Type: text/html, Size: 6453 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 3789 bytes --]

Hello Jesus,

I had made changes to my code to work with the latest version of Perceval,
you can see my latest commit [1]. Let me know if come across issues?

I am working on the testing part now, stuck with few issues. Hoping to
complete by tomorrow or day after.


[1]:
https://github.com/priya299/Dashboard/commit/150e259c22b36b359f79ea711ba4e294d0b0c9ab


*Priya V*
Amrita University
LinkedIn
<https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile>
| GitHub <https://github.com/priya299> | Bitbucket
<https://bitbucket.org/priya299/>


On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona <
jgb@bitergia.com> wrote:

> On Fri, 2016-04-08 at 19:33 +0530, Priya wrote:
> > Hello,
> >
> > I tried running the same command in new version of perceval.  I found
> > the following missing message id errors in perceval_mbox_parse.log
> > file. I am working on the testing part and I will be able to finish
> > it in one or two days.
> >
> > You can see the errors here [1]
> >
> > [1]:http://imgur.com/yVsIoCT
>
> Hi, Priya. I'm not sure about what exactly is causing your messages,
> since I cannot reproduce them (see below). But I still suspect that
> they may happen because in current versions of Perceval the data parsed
> from an mbox is no longer stored as first level key/data in the
> dictionary returned by Perceval for each message, but in data for key
> "data", which is itself a dictionary.
>
> In particular, in the code:
>
> -------------
>               for k in msg_json:
>                     try:
>                         if key == k['Message-ID'].strip('<>'):
>                             k['property'] = key
> -------------
>
> probably you should be checking for k['data']['Message-ID'] instead of
> just k['Message-ID'].
>
> Please, have a look at how recent versions of Perceval produce the
> dictionaries for each message...
>
> But as I said, I cannot reproduce your error. When running your most
> recent code right now (9a5abc47bbab3b06550) with the most recent
> Perceval/master code (53efc14001c806f0452) I get:
>
> --------------------
> (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
> createjson.py --mbox advisory-board-2014-02 --output new.json
> Traceback (most recent call last):
>   File "createjson.py", line 96, in <module>
>     main()
>   File "createjson.py", line 92, in main
>     mparser.create_json(args.mbox,args.output)
>   File "createjson.py", line 59, in create_json
>     messages = th.message_details(mbox_files)
>   File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py",
> line 338, in message_details
>     urllib.request.urlretrieve(filename, 'mbox')
>   File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
>     with contextlib.closing(urlopen(url, data)) as fp:
>   File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
>     return opener.open(url, data, timeout)
>   File "/usr/lib/python3.4/urllib/request.py", line 449, in open
>     req = Request(fullurl, data)
>   File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__
>     self.full_url = url
>   File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url
>     self._parse()
>   File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse
>     raise ValueError("unknown url type: %r" % self.full_url)
> ValueError: unknown url type: 'advisory-board-2014-02'
> -----------------------------
>
> Could you please try to checkout and install exactly the same version
> of Perceval I'm using, and see if you get the same error? And if the
> above problem with the format returned by Perceval persists, maybe you
> can fix that too.
>
> Saludos,
>
>         Jesus.
>
> --
> Bitergia: http://bitergia.com
> /me at Twitter: https://twitter.com/jgbarah
>
>

[-- Attachment #1.2: Type: text/html, Size: 6063 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Regarding Outreachy project on Improving CR Dashboard

[Jesus M. Gonzalez-Barahona]

Thanks a lot. I can run it now, with the latest version of Perceval,
and at first glance seems to work. Some improvements could be done,
but it seems to work. Since we're quite close to the evaluation for the
microtask, let's stop here, except for the tests. Please, produce some,
to validate at least a bit the code...

Saludos,

	Jesus.

On Thu, 2016-04-14 at 22:41 +0530, Priya wrote:
> Hello Jesus, 
> 
> I had made changes to my code to work with the latest version of
> Perceval, you can see my latest commit [1]. Let me know if come
> across issues?
> 
> I am working on the testing part now, stuck with few issues. Hoping
> to complete by tomorrow or day after. 
> 
> 
> [1]:https://github.com/priya299/Dashboard/commit/150e259c22b36b359f79
> ea711ba4e294d0b0c9ab
> 
> 
> Priya V
> Amrita University
> LinkedIn | GitHub | Bitbucket
> 
> 
> On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona <jgb@bite
> rgia.com> wrote:
> > On Fri, 2016-04-08 at 19:33 +0530, Priya wrote:
> > > Hello,
> > >
> > > I tried running the same command in new version of perceval.  I
> > found
> > > the following missing message id errors in
> > perceval_mbox_parse.log
> > > file. I am working on the testing part and I will be able to
> > finish
> > > it in one or two days.
> > >
> > > You can see the errors here [1]
> > >
> > > [1]:http://imgur.com/yVsIoCT
> > 
> > Hi, Priya. I'm not sure about what exactly is causing your
> > messages,
> > since I cannot reproduce them (see below). But I still suspect that
> > they may happen because in current versions of Perceval the data
> > parsed
> > from an mbox is no longer stored as first level key/data in the
> > dictionary returned by Perceval for each message, but in data for
> > key
> > "data", which is itself a dictionary.
> > 
> > In particular, in the code:
> > 
> > -------------
> >               for k in msg_json:
> >                     try:
> >                         if key == k['Message-ID'].strip('<>'):
> >                             k['property'] = key
> > -------------
> > 
> > probably you should be checking for k['data']['Message-ID'] instead
> > of
> > just k['Message-ID'].
> > 
> > Please, have a look at how recent versions of Perceval produce the
> > dictionaries for each message...
> > 
> > But as I said, I cannot reproduce your error. When running your
> > most
> > recent code right now (9a5abc47bbab3b06550) with the most recent
> > Perceval/master code (53efc14001c806f0452) I get:
> > 
> > --------------------
> > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
> > createjson.py --mbox advisory-board-2014-02 --output new.json
> > Traceback (most recent call last):
> >   File "createjson.py", line 96, in <module>
> >     main()
> >   File "createjson.py", line 92, in main
> >     mparser.create_json(args.mbox,args.output)
> >   File "createjson.py", line 59, in create_json
> >     messages = th.message_details(mbox_files)
> >   File
> > "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py",
> > line 338, in message_details
> >     urllib.request.urlretrieve(filename, 'mbox')
> >   File "/usr/lib/python3.4/urllib/request.py", line 186, in
> > urlretrieve
> >     with contextlib.closing(urlopen(url, data)) as fp:
> >   File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
> >     return opener.open(url, data, timeout)
> >   File "/usr/lib/python3.4/urllib/request.py", line 449, in open
> >     req = Request(fullurl, data)
> >   File "/usr/lib/python3.4/urllib/request.py", line 267, in
> > __init__
> >     self.full_url = url
> >   File "/usr/lib/python3.4/urllib/request.py", line 293, in
> > full_url
> >     self._parse()
> >   File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse
> >     raise ValueError("unknown url type: %r" % self.full_url)
> > ValueError: unknown url type: 'advisory-board-2014-02'
> > -----------------------------
> > 
> > Could you please try to checkout and install exactly the same
> > version
> > of Perceval I'm using, and see if you get the same error? And if
> > the
> > above problem with the format returned by Perceval persists, maybe
> > you
> > can fix that too.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Regarding Outreachy project on Improving CR Dashboard

[Priya]

[-- Attachment #1.1: Type: text/plain, Size: 1375 bytes --]

Hello Lars Kurth,

I'm Priya V, a final year Computer Science student from Amrita University,
India. I came across this project on 'Improving Code Review Dashboard' in
Xen project ideas list, and thought about applying for the same in this
round of Outreachy.

>From a quick look, I understand that the CR data from repositories are
obtained using Metrics Grimoire tools and visualized on a Kibana dashboard.
I see the screenshots within the description, and I was wondering if there is
a way to take a look at any of these ( or can I reproduce it in my local ? )

>From the description of the project, I see that we need to produce Perceval
based scripts to analyze code review messages in Xen, instead of
MLStats/CVSAanlY-based scripts which is currently used. I was looking for
the Xen specific CVS scripts in http://xenbits.xen.org/ and couldnt find it
yet. It would be great if you can point me to the original scripts, so that
I can setup Perceval and try figuring out how to bring about the conversion.

I find my skill-sets matching with the project requirements, and
considering that the deadline is drawing near, please guide me with
pointers to start with.

Thanks,
Priya V
Amrita University
LinkedIn
<https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile>
| GitHub <https://github.com/priya299> | Bitbucket
<https://bitbucket.org/priya299/>

[-- Attachment #1.2: Type: text/html, Size: 3154 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel