* Regarding Outreachy project on Improving CR Dashboard @ 2016-03-19 9:54 Priya 2016-03-21 9:43 ` Regarding Outreachy project on Improving CR Dashboard (Urgent) Lars Kurth 0 siblings, 1 reply; 16+ messages in thread From: Priya @ 2016-03-19 9:54 UTC (permalink / raw) To: lars.kurth; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1375 bytes --] Hello Lars Kurth, I'm Priya V, a final year Computer Science student from Amrita University, India. I came across this project on 'Improving Code Review Dashboard' in Xen project ideas list, and thought about applying for the same in this round of Outreachy. >From a quick look, I understand that the CR data from repositories are obtained using Metrics Grimoire tools and visualized on a Kibana dashboard. I see the screenshots within the description, and I was wondering if there is a way to take a look at any of these ( or can I reproduce it in my local ? ) >From the description of the project, I see that we need to produce Perceval based scripts to analyze code review messages in Xen, instead of MLStats/CVSAanlY-based scripts which is currently used. I was looking for the Xen specific CVS scripts in http://xenbits.xen.org/ and couldnt find it yet. It would be great if you can point me to the original scripts, so that I can setup Perceval and try figuring out how to bring about the conversion. I find my skill-sets matching with the project requirements, and considering that the deadline is drawing near, please guide me with pointers to start with. Thanks, Priya V Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> [-- Attachment #1.2: Type: text/html, Size: 3154 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard (Urgent) 2016-03-19 9:54 Regarding Outreachy project on Improving CR Dashboard Priya @ 2016-03-21 9:43 ` Lars Kurth 2016-03-22 9:43 ` Priya 0 siblings, 1 reply; 16+ messages in thread From: Lars Kurth @ 2016-03-21 9:43 UTC (permalink / raw) To: Priya; +Cc: Lars Kurth, Jesus M. Gonzalez-Barahona, Daniel Izquierdo, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 2174 bytes --] Priya, so for the delay: wasn't checking my mail on the weekend. Also adding Jesus and Daniel > On 19 Mar 2016, at 09:54, Priya <vppriya9@gmail.com> wrote: > > Hello Lars Kurth, > > I'm Priya V, a final year Computer Science student from Amrita University, India. I came across this project on 'Improving Code Review Dashboard' in Xen project ideas list, and thought about applying for the same in this round of Outreachy. > > From a quick look, I understand that the CR data from repositories are obtained using Metrics Grimoire tools and visualized on a Kibana dashboard. I see the screenshots within the description, and I was wondering if there is a way to take a look at any of these ( or can I reproduce it in my local ? ) That is correct: there is a set of scripts that create a database. We have a test data sets at http://projects.bitergia.com/xen-project-dashboard/browser/data/db/ <http://projects.bitergia.com/xen-project-dashboard/browser/data/db/> And all the xen specific code and scripts are at https://github.com/dicortazar/xen-code-review-analysis/tree/master <https://github.com/dicortazar/xen-code-review-analysis/tree/master> > From the description of the project, I see that we need to produce Perceval based scripts to analyze code review messages in Xen, instead of MLStats/CVSAanlY-based scripts which is currently used. I was looking for the Xen specific CVS scripts in http://xenbits.xen.org/ <http://xenbits.xen.org/> and couldnt find it yet. It would be great if you can point me to the original scripts, so that I can setup Perceval and try figuring out how to bring about the conversion. There are no Perceval based scripts at this stage yet. But maybe we can find some improvements. @Priya: What is your IRC handle. I think we ought to set up a quick chat, if possible. I am usually lars_kurth on #xendevel on freenode > I find my skill-sets matching with the project requirements, and considering that the deadline is drawing near, please guide me with pointers to start with. @Dani, @Jesus: could you find a suitable task. We can also consider extending the deadline by a couple of days. Regards Lars [-- Attachment #1.2: Type: text/html, Size: 3879 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard (Urgent) 2016-03-21 9:43 ` Regarding Outreachy project on Improving CR Dashboard (Urgent) Lars Kurth @ 2016-03-22 9:43 ` Priya 2016-03-22 12:05 ` Lars Kurth [not found] ` <1458647225.4380.391.camel@bitergia.com> 0 siblings, 2 replies; 16+ messages in thread From: Priya @ 2016-03-22 9:43 UTC (permalink / raw) To: Lars Kurth Cc: Lars Kurth, Jesus M. Gonzalez-Barahona, Daniel Izquierdo, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1058 bytes --] Hello all, Thank you for the introduction Lars. On Mon, Mar 21, 2016 at 3:13 PM, Lars Kurth <lars.kurth.xen@gmail.com> wrote: > > There are no Perceval based scripts at this stage yet. But maybe we can > find some improvements. > @Priya: What is your IRC handle. I think we ought to set up a quick chat, > if possible. I am usually lars_kurth on #xendevel on freenode > I went through some of the existing scripts used, and in my understanding - the project is to shift completely from existing MLStats like scripts to Perceval. I have started working on my proposal focusing around this idea, and I will be sharing it with you shortly. btw, I am priya_ on freenode. @Dani, @Jesus: could you find a suitable task. We can also consider > extending the deadline by a couple of days. > This would be really great, as we are approaching the deadline soon. Priya V Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> [-- Attachment #1.2: Type: text/html, Size: 2586 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard (Urgent) 2016-03-22 9:43 ` Priya @ 2016-03-22 12:05 ` Lars Kurth [not found] ` <1458647225.4380.391.camel@bitergia.com> 1 sibling, 0 replies; 16+ messages in thread From: Lars Kurth @ 2016-03-22 12:05 UTC (permalink / raw) To: Priya; +Cc: Lars Kurth, Jesus M. Gonzalez-Barahona, Daniel Izquierdo, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1009 bytes --] > On 22 Mar 2016, at 09:43, Priya <vppriya9@gmail.com> wrote: > > Hello all, > > Thank you for the introduction Lars. > > On Mon, Mar 21, 2016 at 3:13 PM, Lars Kurth <lars.kurth.xen@gmail.com <mailto:lars.kurth.xen@gmail.com>> wrote: > > There are no Perceval based scripts at this stage yet. But maybe we can find some improvements. > @Priya: What is your IRC handle. I think we ought to set up a quick chat, if possible. I am usually lars_kurth on #xendevel on freenode > > I went through some of the existing scripts used, and in my understanding - the project is to shift completely from existing MLStats like scripts to Perceval. I have started working on my proposal focusing around this idea, and I will be sharing it with you shortly. > > btw, I am priya_ on freenode. I am in meetings all day today. See http://lists.xenproject.org/archives/html/xen-devel/2016-03/msg02641.html <http://lists.xenproject.org/archives/html/xen-devel/2016-03/msg02641.html> though Lars [-- Attachment #1.2: Type: text/html, Size: 2179 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <1458647225.4380.391.camel@bitergia.com>]
* Re: Code-Review-Dashboard: Regarding Outreachy project on Improving CR Dashboard [not found] ` <1458647225.4380.391.camel@bitergia.com> @ 2016-03-24 17:33 ` Lars Kurth 0 siblings, 0 replies; 16+ messages in thread From: Lars Kurth @ 2016-03-24 17:33 UTC (permalink / raw) To: Jesus M. Gonzalez-Barahona, Xen-devel; +Cc: Priya, Daniel Izquierdo, Lars Kurth Removed urgent, added xen-devel back in > On 22 Mar 2016, at 11:47, Jesus M. Gonzalez-Barahona <jgb@bitergia.com> wrote: > > On Tue, 2016-03-22 at 15:13 +0530, Priya wrote: >> Hello all, >> >> Thank you for the introduction Lars. >> >> On Mon, Mar 21, 2016 at 3:13 PM, Lars Kurth <lars.kurth.xen@gmail.com >>> wrote: >>> >>> There are no Perceval based scripts at this stage yet. But maybe we >>> can find some improvements. >>> @Priya: What is your IRC handle. I think we ought to set up a quick >>> chat, if possible. I am usually lars_kurth on #xendevel on >>> freenode >>> >> I went through some of the existing scripts used, and in my >> understanding - the project is to shift completely from existing >> MLStats like scripts to Perceval. I have started working on my >> proposal focusing around this idea, and I will be sharing it with you >> shortly. > > Indeed, the idea is to switch from MetricsGrimoire (MLStats and > CVSAnalY in this case) to Perceval backends. That means moving from SQL > databases to ElasticSearch too. > >> btw, I am priya_ on freenode. > > I guess we were talking some days ago on IRC, right? (I'm jgbarah > there). > >>> @Dani, @Jesus: could you find a suitable task. We can also consider >>> extending the deadline by a couple of days. >>> >> This would be really great, as we are approaching the deadline soon. > > I guess a task to evaluate your proposal could be writting an script to > use the Perceval email backend to feed data from the xen-devel mailing > list to an ElasticSearch database, and annotating in it messages in the > same thread. > > For identifying threads, you can use the Zawinski algorithm, https://ww > w.jwz.org/doc/threading.html > > The result of the script would be an ElasticSearch search index, with > one JSON document per message (it could be the same document produced > by Perceval), with one extra field (property), with the same value for > messages in in the same thread. The value could be the message-id of > the first message in the thread. > > Priya, if you need, I could refine the definition of the problem. > > Lars, I removed xen-devel from the CC, just to avoid making noise. We > can summarize later (but if you prefer all of this copied to xen-devel, > just let me know). As part of the objective of Outreachy is for participants to work in the open and conduct open code reviews and discussions, please ensure that xen-devel is added in future. I think we should mark emails related to this project with "Code-Review-Dashboard:" in the subject line, which means people who don't care can ignore the mails. Also, reviewers, maintainers and committers expect to be CC'ed and will either ignore or just scan mails that only go to xen-devel@ Regards Lars _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Regarding Outreachy project on Improving CR Dashboard @ 2016-04-05 16:35 Priya 2016-04-05 23:23 ` Jesus M. Gonzalez-Barahona 0 siblings, 1 reply; 16+ messages in thread From: Priya @ 2016-04-05 16:35 UTC (permalink / raw) To: Xen-devel; +Cc: Lars Kurth, Jesus M. Gonzalez-Barahona, Daniel Izquierdo [-- Attachment #1.1: Type: text/plain, Size: 1277 bytes --] Hello all, I have completed coding the initial task of grouping the email thread using the Zawinski algorithms and then adding property entity to the json for the messages that belong to the same email thread. You can see my git repo [1]. The new.json is the output of my script and out.json is the output of Perceval. Also, I have updated the README.md file regarding the execution procedures in github. Instructions ============ git clone https://github.com/priya299/Dashboard.git cd Dashboard python createjson.py 'Perceval Ouputfile' 'mbox file' 'output_file' eg: python createjson.py out.json xen-devel-2016-03 new.json "new.json" json file will be created with each message belong to a single thread having an additional attribute "property". The property attribute will have message id of the first message in the thread. Now, I will be pushing the new.json into the elastic search db[2]. Please give me your valuable feedback about my progress. [1]:https://github.com/priya299/Dashboard [2]:https://www.elastic.co/guide/en/kibana/3.0/import-some-data.html *Priya V* Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> [-- Attachment #1.2: Type: text/html, Size: 2657 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-05 16:35 Priya @ 2016-04-05 23:23 ` Jesus M. Gonzalez-Barahona 2016-04-06 12:00 ` Priya 0 siblings, 1 reply; 16+ messages in thread From: Jesus M. Gonzalez-Barahona @ 2016-04-05 23:23 UTC (permalink / raw) To: Priya, Xen-devel; +Cc: Lars Kurth, Daniel Izquierdo On Tue, 2016-04-05 at 22:05 +0530, Priya wrote: > Hello all, > > I have completed coding the initial task of grouping the email thread > using the Zawinski algorithms and then adding property entity to the > json for the messages that belong to the same email thread. > > You can see my git repo [1]. The new.json is the output of my script > and out.json is the output of Perceval. > > Also, I have updated the README.md file regarding the execution > procedures in github. > > Instructions > ============ > > git clone https://github.com/priya299/Dashboard.git > > cd Dashboard > > python createjson.py 'Perceval Ouputfile' 'mbox file' 'output_file' > > eg: python createjson.py out.json xen-devel-2016-03 new.json > > "new.json" json file will be created with each message belong to a > single thread having an additional attribute "property". The property > attribute will have message id of the first message in the thread. > > Now, I will be pushing the new.json into the elastic search db[2]. > Please give me your valuable feedback about my progress. > > [1]:https://github.com/priya299/Dashboard > [2]:https://www.elastic.co/guide/en/kibana/3.0/import-some-data.html Hi, Priya. To begin with, could you please integrate your code with the Perceval iterator? In other words, you can run Perceval on the mailing list archive directly from your code, which will render the use of "out.json" void. That way, the invocation of the script would be more like: python createjson.py xen-devel-2016-03 new.json In other words, create.json would use Perceval to parse the mailing list archive. For this end, the Perceval mbox backend is a class, which once instantiated, provides an iterator function, fetch(), that you can run inside a loop. For each iteration of the loop, you get the equivalent to a JSON element in out.json. The code would be similar to: ------------------------------- import perceval mbox_parser = perceval.backends.mbox.MBox( origin=mbox_url, dirpath=mbox_file_name ) for item in mbox_parser.fetch(): thread_id = find_thread(item) ... --------------------------------- Some details about the Perceval mbox class: http://perceval.readthedocs.org/en/master/perceval.backends.html#module -perceval.backends.mbox If you have trouble running the Perceval backend as an iterator, please let me know. In addition, you can use argparse for reading the arguments in the command line. It is easy and convenient. Saludos, Jesus. > -- Bitergia: http://bitergia.com /me at Twitter: https://twitter.com/jgbarah _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-05 23:23 ` Jesus M. Gonzalez-Barahona @ 2016-04-06 12:00 ` Priya 2016-04-06 21:59 ` Jesus M. Gonzalez-Barahona 0 siblings, 1 reply; 16+ messages in thread From: Priya @ 2016-04-06 12:00 UTC (permalink / raw) To: Jesus M. Gonzalez-Barahona; +Cc: Lars Kurth, Daniel Izquierdo, Xen-devel [-- Attachment #1.1: Type: text/plain, Size: 499 bytes --] Hello, Thanks for your suggestions. I have made the appropriate changes as you had mentioned. It took a little time to change from python3 to python3.4 as perceval supports python3.4. I have updated the changes in my github. You can see my git repo [1] [1]:https://github.com/priya299/Dashboard *Priya V* Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> [-- Attachment #1.2: Type: text/html, Size: 1486 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-06 12:00 ` Priya @ 2016-04-06 21:59 ` Jesus M. Gonzalez-Barahona 2016-04-07 12:27 ` Priya 0 siblings, 1 reply; 16+ messages in thread From: Jesus M. Gonzalez-Barahona @ 2016-04-06 21:59 UTC (permalink / raw) To: Priya; +Cc: Lars Kurth, Xen-devel, Daniel Izquierdo On Wed, 2016-04-06 at 17:30 +0530, Priya wrote: > Hello, > > Thanks for your suggestions. > I have made the appropriate changes as you had mentioned. > It took a little time to change from python3 to python3.4 as perceval > supports python3.4. I have updated the changes in my github. You can > see my git repo [1] > > [1]:https://github.com/priya299/Dashboard Thanks a lot, Priya. Good work. Some preliminary comments, below. * When runing the script on the xen-devel-2016-03 mbox, I seen an exception raised: ------------ (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 createjson.py --mbox xen-devel-2016-03 --output new.json Traceback (most recent call last): File "createjson.py", line 61, in <module> create_json(args.mbox,args.output) File "createjson.py", line 43, in create_json if key == k['Message-ID'].strip('<>'): KeyError: 'Message-ID' ------------ Maybe some message is not having a Message-ID field? I suggest that you capture this exception, print out the offending message, and go on with the next one. You can use the Python logging package for printing out this kind of information (you can see how to use it in the Perceval package itself). But see below. * Minor typo in the README: Instead of eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json it should be eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output new.json * The files have no licensing info. If you agree, it could be GPLv3, as is Perceval itself. For that, it would be enough that you mimic the header in Perceval files in your Python files (of course, indicating your authorship information). * Which version of Perceval are you using? Some weeks ago, the format of the dictionary produced by Perceval for each message changed. Now the actual fields of the message are in a data subdictionary. Please, check that: the above exception with respect to the Message-ID key could be because of this... Please, try to make it work with master HEAD for Perceval (I don't expect any new major change in the next days/weeks, and I'll try to warn you in case some happens). * Could you please write at least one unit test for your code? You can see examples of the testing schema we use in the tests directory in Perceval, but we use vanilla unittest (the Python package for tests). At this stage I don't need that you produce a whole set of tests, only one or two to show that you know how to write unit tests, please. Saludos, Jesus. > Priya V > Amrita University > LinkedIn | GitHub | Bitbucket > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel -- Bitergia: http://bitergia.com /me at Twitter: https://twitter.com/jgbarah _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-06 21:59 ` Jesus M. Gonzalez-Barahona @ 2016-04-07 12:27 ` Priya 2016-04-07 17:57 ` Jesus M. Gonzalez-Barahona 0 siblings, 1 reply; 16+ messages in thread From: Priya @ 2016-04-07 12:27 UTC (permalink / raw) To: Jesus M. Gonzalez-Barahona; +Cc: Lars Kurth, Xen-devel, Daniel Izquierdo [-- Attachment #1.1: Type: text/plain, Size: 3743 bytes --] Hello all, Thanks for the suggestions. I have updated the changes as u had mentioned. I am sorry, but I could not find any errors while running $ python3 createjson.py --mbox xen-devel-2016-03 --output new.json command. I am wondering what is wrong with this and might be a problem with python3 or my perceval version. I have added licensing and python logging. You can see it in my github repo [1]. I will try upgrading perceval and adding in the tests in the coming days, and will update. [1]: https://github.com/priya299/Dashboard *Priya V* Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> On Thu, Apr 7, 2016 at 3:29 AM, Jesus M. Gonzalez-Barahona <jgb@bitergia.com > wrote: > On Wed, 2016-04-06 at 17:30 +0530, Priya wrote: > > Hello, > > > > Thanks for your suggestions. > > I have made the appropriate changes as you had mentioned. > > It took a little time to change from python3 to python3.4 as perceval > > supports python3.4. I have updated the changes in my github. You can > > see my git repo [1] > > > > [1]:https://github.com/priya299/Dashboard > > Thanks a lot, Priya. Good work. Some preliminary comments, below. > > * When runing the script on the xen-devel-2016-03 mbox, I seen an > exception raised: > > ------------ > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 > createjson.py --mbox xen-devel-2016-03 --output new.json > Traceback (most recent call last): > File "createjson.py", line 61, in <module> > create_json(args.mbox,args.output) > File "createjson.py", line 43, in create_json > if key == k['Message-ID'].strip('<>'): > KeyError: 'Message-ID' > ------------ > > Maybe some message is not having a Message-ID field? I suggest that you > capture this exception, print out the offending message, and go on with > the next one. You can use the Python logging package for printing out > this kind of information (you can see how to use it in the Perceval > package itself). But see below. > > * Minor typo in the README: > > Instead of > > eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json > > it should be > > eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output new.json > > * The files have no licensing info. If you agree, it could be GPLv3, as > is Perceval itself. For that, it would be enough that you mimic the > header in Perceval files in your Python files (of course, indicating > your authorship information). > > * Which version of Perceval are you using? Some weeks ago, the format > of the dictionary produced by Perceval for each message changed. Now > the actual fields of the message are in a data subdictionary. Please, > check that: the above exception with respect to the Message-ID key > could be because of this... Please, try to make it work with master > HEAD for Perceval (I don't expect any new major change in the next > days/weeks, and I'll try to warn you in case some happens). > > * Could you please write at least one unit test for your code? You can > see examples of the testing schema we use in the tests directory in > Perceval, but we use vanilla unittest (the Python package for tests). > At this stage I don't need that you produce a whole set of tests, only > one or two to show that you know how to write unit tests, please. > > Saludos, > > Jesus. > > > Priya V > > Amrita University > > LinkedIn | GitHub | Bitbucket > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > -- > Bitergia: http://bitergia.com > /me at Twitter: https://twitter.com/jgbarah > > [-- Attachment #1.2: Type: text/html, Size: 5752 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-07 12:27 ` Priya @ 2016-04-07 17:57 ` Jesus M. Gonzalez-Barahona 2016-04-08 14:03 ` Priya 0 siblings, 1 reply; 16+ messages in thread From: Jesus M. Gonzalez-Barahona @ 2016-04-07 17:57 UTC (permalink / raw) To: Priya; +Cc: Lars Kurth, Xen-devel, Daniel Izquierdo On Thu, 2016-04-07 at 17:57 +0530, Priya wrote: > Hello all, > > Thanks for the suggestions. I have updated the changes as u had > mentioned. I am sorry, but I could not find any errors while running > > $ python3 createjson.py --mbox xen-devel-2016-03 --output new.json > > command. I am wondering what is wrong with this and might be a > problem with python3 or my perceval version. I have added licensing > and python logging. You can see it in my github repo [1]. I will try > upgrading perceval and adding in the tests in the coming days, and > will update. Priya, I guess I didn't explain the source for the error clearly enough, sorry about that. It is very likely that you're not running the latest version of Perceval, which changed the format for the items it produces some days ago. That's why, if you're running a version of Perceval of more than some days ago, it works, but if you run the latest one, it doesn't work (at least for me). I'm not completely sure, but that's why I asked you to upgrade to the latest version (either master HEAD or version 0.1.0). If you don't see the error once you do this, please let me know. Thanks, Jesus. > [1]: https://github.com/priya299/Dashboard > > Priya V > Amrita University > LinkedIn | GitHub | Bitbucket > > > On Thu, Apr 7, 2016 at 3:29 AM, Jesus M. Gonzalez-Barahona <jgb@biter > gia.com> wrote: > > On Wed, 2016-04-06 at 17:30 +0530, Priya wrote: > > > Hello, > > > > > > Thanks for your suggestions. > > > I have made the appropriate changes as you had mentioned. > > > It took a little time to change from python3 to python3.4 as > > perceval > > > supports python3.4. I have updated the changes in my github. You > > can > > > see my git repo [1] > > > > > > [1]:https://github.com/priya299/Dashboard > > > > Thanks a lot, Priya. Good work. Some preliminary comments, below. > > > > * When runing the script on the xen-devel-2016-03 mbox, I seen an > > exception raised: > > > > ------------ > > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 > > createjson.py --mbox xen-devel-2016-03 --output new.json > > Traceback (most recent call last): > > File "createjson.py", line 61, in <module> > > create_json(args.mbox,args.output) > > File "createjson.py", line 43, in create_json > > if key == k['Message-ID'].strip('<>'): > > KeyError: 'Message-ID' > > ------------ > > > > Maybe some message is not having a Message-ID field? I suggest that > > you > > capture this exception, print out the offending message, and go on > > with > > the next one. You can use the Python logging package for printing > > out > > this kind of information (you can see how to use it in the Perceval > > package itself). But see below. > > > > * Minor typo in the README: > > > > Instead of > > > > eg: python3.4 createjson --mbox xen-devel-2016-03 --output new.json > > > > it should be > > > > eg: python3.4 createjson.py --mbox xen-devel-2016-03 --output > > new.json > > > > * The files have no licensing info. If you agree, it could be > > GPLv3, as > > is Perceval itself. For that, it would be enough that you mimic the > > header in Perceval files in your Python files (of course, > > indicating > > your authorship information). > > > > * Which version of Perceval are you using? Some weeks ago, the > > format > > of the dictionary produced by Perceval for each message changed. > > Now > > the actual fields of the message are in a data subdictionary. > > Please, > > check that: the above exception with respect to the Message-ID key > > could be because of this... Please, try to make it work with master > > HEAD for Perceval (I don't expect any new major change in the next > > days/weeks, and I'll try to warn you in case some happens). > > > > * Could you please write at least one unit test for your code? You > > can > > see examples of the testing schema we use in the tests directory in > > Perceval, but we use vanilla unittest (the Python package for > > tests). > > At this stage I don't need that you produce a whole set of tests, > > only > > one or two to show that you know how to write unit tests, please. > > > > Saludos, > > > > Jesus. > > > > > Priya V > > > Amrita University > > > LinkedIn | GitHub | Bitbucket > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen.org > > > http://lists.xen.org/xen-devel > > -- > > Bitergia: http://bitergia.com > > /me at Twitter: https://twitter.com/jgbarah > > > > -- Bitergia: http://bitergia.com /me at Twitter: https://twitter.com/jgbarah _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-07 17:57 ` Jesus M. Gonzalez-Barahona @ 2016-04-08 14:03 ` Priya 2016-04-11 7:53 ` Jesus M. Gonzalez-Barahona 0 siblings, 1 reply; 16+ messages in thread From: Priya @ 2016-04-08 14:03 UTC (permalink / raw) To: Jesus M. Gonzalez-Barahona; +Cc: Lars Kurth, Xen-devel, Daniel Izquierdo [-- Attachment #1.1: Type: text/plain, Size: 503 bytes --] Hello, I tried running the same command in new version of perceval. I found the following missing message id errors in perceval_mbox_parse.log file. I am working on the testing part and I will be able to finish it in one or two days. You can see the errors here [1] [1]:http://imgur.com/yVsIoCT *Priya V* Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> [-- Attachment #1.2: Type: text/html, Size: 1455 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-08 14:03 ` Priya @ 2016-04-11 7:53 ` Jesus M. Gonzalez-Barahona 2016-04-13 16:33 ` Priya 2016-04-14 17:11 ` Priya 0 siblings, 2 replies; 16+ messages in thread From: Jesus M. Gonzalez-Barahona @ 2016-04-11 7:53 UTC (permalink / raw) To: Priya; +Cc: Lars Kurth, Xen-devel, Daniel Izquierdo On Fri, 2016-04-08 at 19:33 +0530, Priya wrote: > Hello, > > I tried running the same command in new version of perceval. I found > the following missing message id errors in perceval_mbox_parse.log > file. I am working on the testing part and I will be able to finish > it in one or two days. > > You can see the errors here [1] > > [1]:http://imgur.com/yVsIoCT Hi, Priya. I'm not sure about what exactly is causing your messages, since I cannot reproduce them (see below). But I still suspect that they may happen because in current versions of Perceval the data parsed from an mbox is no longer stored as first level key/data in the dictionary returned by Perceval for each message, but in data for key "data", which is itself a dictionary. In particular, in the code: ------------- for k in msg_json: try: if key == k['Message-ID'].strip('<>'): k['property'] = key ------------- probably you should be checking for k['data']['Message-ID'] instead of just k['Message-ID']. Please, have a look at how recent versions of Perceval produce the dictionaries for each message... But as I said, I cannot reproduce your error. When running your most recent code right now (9a5abc47bbab3b06550) with the most recent Perceval/master code (53efc14001c806f0452) I get: -------------------- (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 createjson.py --mbox advisory-board-2014-02 --output new.json Traceback (most recent call last): File "createjson.py", line 96, in <module> main() File "createjson.py", line 92, in main mparser.create_json(args.mbox,args.output) File "createjson.py", line 59, in create_json messages = th.message_details(mbox_files) File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py", line 338, in message_details urllib.request.urlretrieve(filename, 'mbox') File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python3.4/urllib/request.py", line 449, in open req = Request(fullurl, data) File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__ self.full_url = url File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url self._parse() File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse raise ValueError("unknown url type: %r" % self.full_url) ValueError: unknown url type: 'advisory-board-2014-02' ----------------------------- Could you please try to checkout and install exactly the same version of Perceval I'm using, and see if you get the same error? And if the above problem with the format returned by Perceval persists, maybe you can fix that too. Saludos, Jesus. -- Bitergia: http://bitergia.com /me at Twitter: https://twitter.com/jgbarah _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-11 7:53 ` Jesus M. Gonzalez-Barahona @ 2016-04-13 16:33 ` Priya 2016-04-14 17:11 ` Priya 1 sibling, 0 replies; 16+ messages in thread From: Priya @ 2016-04-13 16:33 UTC (permalink / raw) To: Jesus M. Gonzalez-Barahona; +Cc: Lars Kurth, Xen-devel, Daniel Izquierdo [-- Attachment #1.1: Type: text/plain, Size: 4046 bytes --] Hello, Forgot to CC the Xen-devel mailing list. I have changed the code to add the tests, and now it is like you need to provide the whole link of the mbox file rather than just the file name. So, $python3 createjson.py --mbox http://lists.xenproject.org/archives/html/mbox/advisory-board-2013-05-2014-02--output new.json should work instead of $ python3 createjson.py --mbox advisory-board-2014-02 --output new.json I forgot to update the README file, and I have done that now. I will try k['data']['Message-ID'] instead, and I would update soon. I'm working on the testing part and I'll complete it soon. *Priya V* Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona < jgb@bitergia.com> wrote: > On Fri, 2016-04-08 at 19:33 +0530, Priya wrote: > > Hello, > > > > I tried running the same command in new version of perceval. I found > > the following missing message id errors in perceval_mbox_parse.log > > file. I am working on the testing part and I will be able to finish > > it in one or two days. > > > > You can see the errors here [1] > > > > [1]:http://imgur.com/yVsIoCT > > Hi, Priya. I'm not sure about what exactly is causing your messages, > since I cannot reproduce them (see below). But I still suspect that > they may happen because in current versions of Perceval the data parsed > from an mbox is no longer stored as first level key/data in the > dictionary returned by Perceval for each message, but in data for key > "data", which is itself a dictionary. > > In particular, in the code: > > ------------- > for k in msg_json: > try: > if key == k['Message-ID'].strip('<>'): > k['property'] = key > ------------- > > probably you should be checking for k['data']['Message-ID'] instead of > just k['Message-ID']. > > Please, have a look at how recent versions of Perceval produce the > dictionaries for each message... > > But as I said, I cannot reproduce your error. When running your most > recent code right now (9a5abc47bbab3b06550) with the most recent > Perceval/master code (53efc14001c806f0452) I get: > > -------------------- > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 > createjson.py --mbox advisory-board-2014-02 --output new.json > Traceback (most recent call last): > File "createjson.py", line 96, in <module> > main() > File "createjson.py", line 92, in main > mparser.create_json(args.mbox,args.output) > File "createjson.py", line 59, in create_json > messages = th.message_details(mbox_files) > File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py", > line 338, in message_details > urllib.request.urlretrieve(filename, 'mbox') > File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve > with contextlib.closing(urlopen(url, data)) as fp: > File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen > return opener.open(url, data, timeout) > File "/usr/lib/python3.4/urllib/request.py", line 449, in open > req = Request(fullurl, data) > File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__ > self.full_url = url > File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url > self._parse() > File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse > raise ValueError("unknown url type: %r" % self.full_url) > ValueError: unknown url type: 'advisory-board-2014-02' > ----------------------------- > > Could you please try to checkout and install exactly the same version > of Perceval I'm using, and see if you get the same error? And if the > above problem with the format returned by Perceval persists, maybe you > can fix that too. > > Saludos, > > Jesus. > > -- > Bitergia: http://bitergia.com > /me at Twitter: https://twitter.com/jgbarah > > [-- Attachment #1.2: Type: text/html, Size: 6453 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-11 7:53 ` Jesus M. Gonzalez-Barahona 2016-04-13 16:33 ` Priya @ 2016-04-14 17:11 ` Priya 2016-04-14 22:41 ` Jesus M. Gonzalez-Barahona 1 sibling, 1 reply; 16+ messages in thread From: Priya @ 2016-04-14 17:11 UTC (permalink / raw) To: Jesus M. Gonzalez-Barahona; +Cc: Lars Kurth, Xen-devel, Daniel Izquierdo [-- Attachment #1.1: Type: text/plain, Size: 3789 bytes --] Hello Jesus, I had made changes to my code to work with the latest version of Perceval, you can see my latest commit [1]. Let me know if come across issues? I am working on the testing part now, stuck with few issues. Hoping to complete by tomorrow or day after. [1]: https://github.com/priya299/Dashboard/commit/150e259c22b36b359f79ea711ba4e294d0b0c9ab *Priya V* Amrita University LinkedIn <https://www.linkedin.com/in/priya-v-195560b6?trk=nav_responsive_tab_profile> | GitHub <https://github.com/priya299> | Bitbucket <https://bitbucket.org/priya299/> On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona < jgb@bitergia.com> wrote: > On Fri, 2016-04-08 at 19:33 +0530, Priya wrote: > > Hello, > > > > I tried running the same command in new version of perceval. I found > > the following missing message id errors in perceval_mbox_parse.log > > file. I am working on the testing part and I will be able to finish > > it in one or two days. > > > > You can see the errors here [1] > > > > [1]:http://imgur.com/yVsIoCT > > Hi, Priya. I'm not sure about what exactly is causing your messages, > since I cannot reproduce them (see below). But I still suspect that > they may happen because in current versions of Perceval the data parsed > from an mbox is no longer stored as first level key/data in the > dictionary returned by Perceval for each message, but in data for key > "data", which is itself a dictionary. > > In particular, in the code: > > ------------- > for k in msg_json: > try: > if key == k['Message-ID'].strip('<>'): > k['property'] = key > ------------- > > probably you should be checking for k['data']['Message-ID'] instead of > just k['Message-ID']. > > Please, have a look at how recent versions of Perceval produce the > dictionaries for each message... > > But as I said, I cannot reproduce your error. When running your most > recent code right now (9a5abc47bbab3b06550) with the most recent > Perceval/master code (53efc14001c806f0452) I get: > > -------------------- > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 > createjson.py --mbox advisory-board-2014-02 --output new.json > Traceback (most recent call last): > File "createjson.py", line 96, in <module> > main() > File "createjson.py", line 92, in main > mparser.create_json(args.mbox,args.output) > File "createjson.py", line 59, in create_json > messages = th.message_details(mbox_files) > File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py", > line 338, in message_details > urllib.request.urlretrieve(filename, 'mbox') > File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve > with contextlib.closing(urlopen(url, data)) as fp: > File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen > return opener.open(url, data, timeout) > File "/usr/lib/python3.4/urllib/request.py", line 449, in open > req = Request(fullurl, data) > File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__ > self.full_url = url > File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url > self._parse() > File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse > raise ValueError("unknown url type: %r" % self.full_url) > ValueError: unknown url type: 'advisory-board-2014-02' > ----------------------------- > > Could you please try to checkout and install exactly the same version > of Perceval I'm using, and see if you get the same error? And if the > above problem with the format returned by Perceval persists, maybe you > can fix that too. > > Saludos, > > Jesus. > > -- > Bitergia: http://bitergia.com > /me at Twitter: https://twitter.com/jgbarah > > [-- Attachment #1.2: Type: text/html, Size: 6063 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Regarding Outreachy project on Improving CR Dashboard 2016-04-14 17:11 ` Priya @ 2016-04-14 22:41 ` Jesus M. Gonzalez-Barahona 0 siblings, 0 replies; 16+ messages in thread From: Jesus M. Gonzalez-Barahona @ 2016-04-14 22:41 UTC (permalink / raw) To: Priya; +Cc: Lars Kurth, Daniel Izquierdo, Xen-devel Thanks a lot. I can run it now, with the latest version of Perceval, and at first glance seems to work. Some improvements could be done, but it seems to work. Since we're quite close to the evaluation for the microtask, let's stop here, except for the tests. Please, produce some, to validate at least a bit the code... Saludos, Jesus. On Thu, 2016-04-14 at 22:41 +0530, Priya wrote: > Hello Jesus, > > I had made changes to my code to work with the latest version of > Perceval, you can see my latest commit [1]. Let me know if come > across issues? > > I am working on the testing part now, stuck with few issues. Hoping > to complete by tomorrow or day after. > > > [1]:https://github.com/priya299/Dashboard/commit/150e259c22b36b359f79 > ea711ba4e294d0b0c9ab > > > Priya V > Amrita University > LinkedIn | GitHub | Bitbucket > > > On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona <jgb@bite > rgia.com> wrote: > > On Fri, 2016-04-08 at 19:33 +0530, Priya wrote: > > > Hello, > > > > > > I tried running the same command in new version of perceval. I > > found > > > the following missing message id errors in > > perceval_mbox_parse.log > > > file. I am working on the testing part and I will be able to > > finish > > > it in one or two days. > > > > > > You can see the errors here [1] > > > > > > [1]:http://imgur.com/yVsIoCT > > > > Hi, Priya. I'm not sure about what exactly is causing your > > messages, > > since I cannot reproduce them (see below). But I still suspect that > > they may happen because in current versions of Perceval the data > > parsed > > from an mbox is no longer stored as first level key/data in the > > dictionary returned by Perceval for each message, but in data for > > key > > "data", which is itself a dictionary. > > > > In particular, in the code: > > > > ------------- > > for k in msg_json: > > try: > > if key == k['Message-ID'].strip('<>'): > > k['property'] = key > > ------------- > > > > probably you should be checking for k['data']['Message-ID'] instead > > of > > just k['Message-ID']. > > > > Please, have a look at how recent versions of Perceval produce the > > dictionaries for each message... > > > > But as I said, I cannot reproduce your error. When running your > > most > > recent code right now (9a5abc47bbab3b06550) with the most recent > > Perceval/master code (53efc14001c806f0452) I get: > > > > -------------------- > > (perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3 > > createjson.py --mbox advisory-board-2014-02 --output new.json > > Traceback (most recent call last): > > File "createjson.py", line 96, in <module> > > main() > > File "createjson.py", line 92, in main > > mparser.create_json(args.mbox,args.output) > > File "createjson.py", line 59, in create_json > > messages = th.message_details(mbox_files) > > File > > "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py", > > line 338, in message_details > > urllib.request.urlretrieve(filename, 'mbox') > > File "/usr/lib/python3.4/urllib/request.py", line 186, in > > urlretrieve > > with contextlib.closing(urlopen(url, data)) as fp: > > File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen > > return opener.open(url, data, timeout) > > File "/usr/lib/python3.4/urllib/request.py", line 449, in open > > req = Request(fullurl, data) > > File "/usr/lib/python3.4/urllib/request.py", line 267, in > > __init__ > > self.full_url = url > > File "/usr/lib/python3.4/urllib/request.py", line 293, in > > full_url > > self._parse() > > File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse > > raise ValueError("unknown url type: %r" % self.full_url) > > ValueError: unknown url type: 'advisory-board-2014-02' > > ----------------------------- > > > > Could you please try to checkout and install exactly the same > > version > > of Perceval I'm using, and see if you get the same error? And if > > the > > above problem with the format returned by Perceval persists, maybe > > you > > can fix that too. > > > > Saludos, > > > > Jesus. > > > > -- > > Bitergia: http://bitergia.com > > /me at Twitter: https://twitter.com/jgbarah > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel -- Bitergia: http://bitergia.com /me at Twitter: https://twitter.com/jgbarah _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2016-04-14 22:41 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-03-19 9:54 Regarding Outreachy project on Improving CR Dashboard Priya 2016-03-21 9:43 ` Regarding Outreachy project on Improving CR Dashboard (Urgent) Lars Kurth 2016-03-22 9:43 ` Priya 2016-03-22 12:05 ` Lars Kurth [not found] ` <1458647225.4380.391.camel@bitergia.com> 2016-03-24 17:33 ` Code-Review-Dashboard: Regarding Outreachy project on Improving CR Dashboard Lars Kurth 2016-04-05 16:35 Priya 2016-04-05 23:23 ` Jesus M. Gonzalez-Barahona 2016-04-06 12:00 ` Priya 2016-04-06 21:59 ` Jesus M. Gonzalez-Barahona 2016-04-07 12:27 ` Priya 2016-04-07 17:57 ` Jesus M. Gonzalez-Barahona 2016-04-08 14:03 ` Priya 2016-04-11 7:53 ` Jesus M. Gonzalez-Barahona 2016-04-13 16:33 ` Priya 2016-04-14 17:11 ` Priya 2016-04-14 22:41 ` Jesus M. Gonzalez-Barahona
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.