[Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Ishani]

Hello Sir,

I am Ishani Chugh, a fourth year undergraduate student of Electronics and Communications Engineering at International Institute of Information Technology, Hyderabad. I am passionate about Open Source Development. I am interested to contribute to your organization Xen Project as an Outreachy intern . I am interested to work in the project: Code Standards Checking using clang-format. I am proficient in C++ and have slight experience of working in clang-format.
Please guide me for initial contributions to prove my candidature.

Regards
Ishani

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Lars Kurth]

Hi Ishani, (+Doug, +Andy)

not that the deadline as been extended to April 13. I am also CC'ing Andy Cooper as Doug has not responded. 

The challenge that we will have is to satisfy the small task requirement while also making the task sensible in the context of the project. Now we do have some flexibility there: in other words, we can declare for example an RFC that everyone agrees with as such a contribution.

But as a basic idea, the first step of the project would be to try and come up with a .clang-format file that matches one of the coding styles in the git tree (either http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CODING_STYLE or http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxl/CODING_STYLE) and prove that it works. I don't know whether this is doable as a small task: the number of options in clang-format do not seem to be huge. And there are tools such as https://clangformat.com/ to experiment with.

If it is possible to do the whole thing in a small task, the output would be a .clang-format file that is submitted for code review to a suitable location in the xen tree. I don't know what the right location should be, but xen.git:tools/clang-format would be sensible. 

If it is not possible to do this for one of the complete coding standards, I would develop for one or two subsets: 
1: a .clang-format settings snippet for a reasonably complex coding style pattern in the coding standard
2: a piece of test code that adheres to the coding standard 
3: a variant of the same code snippet that does not adhere to the coding standard 
4: a small script that checks that .clang-format works correctly on these pieces of code 

For 4, you probably want to run clang-format with -output-replacements-xml on the files in 1 - 3 and then do some grep magic to see whether it does the right thing.

Again this could go into xen.git:tools/clang-format - maybe under tools/clang-format/tests for now

@Andy, @Doug: does this sound sensible?

Regards
Lars


> On 31 Mar 2017, at 10:37, Ishani <chugh.ishani@research.iiit.ac.in> wrote:
> 
> Hello Sir,
> 
> I am Ishani Chugh, a fourth year undergraduate student of Electronics and Communications Engineering at International Institute of Information Technology, Hyderabad. I am passionate about Open Source Development. I am interested to contribute to your organization Xen Project as an Outreachy intern . I am interested to work in the project: Code Standards Checking using clang-format. I am proficient in C++ and have slight experience of working in clang-format.
> Please guide me for initial contributions to prove my candidature.
> 
> Regards
> Ishani
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Ishani]

Hello,

Thanks for the reply. I started with looking into format of Xen Hypervisor files and creating a clang format file for the same. However, given the present implementation of clang-format, it is not possible to incorporate fine-grained custom coding format requirements. For example, although clang-format provides a custom brace wrapping format specification, it is still not sufficient to incorporate the do while loop exception for braces in Xen Hypervisor format. The implementations is presently on higher layers of abstraction and needs to be fine grained to allow for different types of loops as well. I figured that there are two ways of doing it, First , by creating a wrapper over clang-format and using regexes to find appropriate partitions of code to follow different format styles. However, this will be rather cumbersome with multiple heuristics and we will lose the advantage of clang-format tool using the inbuilt parser.
Second way is to extend the functionality of clang-format. I have experience working with compilers. Going through the code of clang it seems possible to include kw_do token for do while loop as a functionality for braces wrapping( although it would require a very thorough reading through the codebase first). 
At present, I am working on creating a clang format file supporting subset of coding formats for Xen Hypervisor.

Can you provide me with some more insight into the project?
clang-format formats the tool to specified format. However, there are many standards which include a particular format for a variable name. It is not feasible for a tool to automatically change the name of variable according to given standards because of possible breaking build of the code. As a part of the project, are we required to create an auto-formatter or something like pylint which suggests the necessary changes and gives an overall score, or a combination of both?

I wish to apply to GSOC as well for the same project. However, since the deadline is today, I will be grateful if you could provide some suggestions for the project.

Regards,
Ishani

----- Original Message -----
From: "Lars Kurth" <lars.kurth.xen@gmail.com>
To: "Ishani" <chugh.ishani@research.iiit.ac.in>
Cc: "cardoe" <cardoe@cardoe.com>, "xen-devel" <xen-devel@lists.xenproject.org>, "Andrew Cooper" <andrew.cooper3@citrix.com>
Sent: Monday, April 3, 2017 4:30:11 PM
Subject: Re: [Xen-devel] [Outreachy] Interested in contribution: Code Standards Checking using clang-format

Hi Ishani, (+Doug, +Andy)

not that the deadline as been extended to April 13. I am also CC'ing Andy Cooper as Doug has not responded. 

The challenge that we will have is to satisfy the small task requirement while also making the task sensible in the context of the project. Now we do have some flexibility there: in other words, we can declare for example an RFC that everyone agrees with as such a contribution.

But as a basic idea, the first step of the project would be to try and come up with a .clang-format file that matches one of the coding styles in the git tree (either http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CODING_STYLE or http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/libxl/CODING_STYLE) and prove that it works. I don't know whether this is doable as a small task: the number of options in clang-format do not seem to be huge. And there are tools such as https://clangformat.com/ to experiment with.

If it is possible to do the whole thing in a small task, the output would be a .clang-format file that is submitted for code review to a suitable location in the xen tree. I don't know what the right location should be, but xen.git:tools/clang-format would be sensible. 

If it is not possible to do this for one of the complete coding standards, I would develop for one or two subsets: 
1: a .clang-format settings snippet for a reasonably complex coding style pattern in the coding standard
2: a piece of test code that adheres to the coding standard 
3: a variant of the same code snippet that does not adhere to the coding standard 
4: a small script that checks that .clang-format works correctly on these pieces of code 

For 4, you probably want to run clang-format with -output-replacements-xml on the files in 1 - 3 and then do some grep magic to see whether it does the right thing.

Again this could go into xen.git:tools/clang-format - maybe under tools/clang-format/tests for now

@Andy, @Doug: does this sound sensible?

Regards
Lars


> On 31 Mar 2017, at 10:37, Ishani <chugh.ishani@research.iiit.ac.in> wrote:
> 
> Hello Sir,
> 
> I am Ishani Chugh, a fourth year undergraduate student of Electronics and Communications Engineering at International Institute of Information Technology, Hyderabad. I am passionate about Open Source Development. I am interested to contribute to your organization Xen Project as an Outreachy intern . I am interested to work in the project: Code Standards Checking using clang-format. I am proficient in C++ and have slight experience of working in clang-format.
> Please guide me for initial contributions to prove my candidature.
> 
> Regards
> Ishani
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Lars Kurth]

Hi Ishani,

> On 3 Apr 2017, at 13:01, Ishani <chugh.ishani@research.iiit.ac.in> wrote:
> 
> Thanks for the reply. I started with looking into format of Xen Hypervisor files and creating a clang format file for the same. However, given the present implementation of clang-format, it is not possible to incorporate fine-grained custom coding format requirements. For example, although clang-format provides a custom brace wrapping format specification, it is still not sufficient to incorporate the do while loop exception for braces in Xen Hypervisor format.

OK, this makes this project a lot harder. I don't think we are particularly tied to clang-format. 

> The implementations is presently on higher layers of abstraction and needs to be fine grained to allow for different types of loops as well. I figured that there are two ways of doing it, First , by creating a wrapper over clang-format and using regexes to find appropriate partitions of code to follow different format styles.
> However, this will be rather cumbersome with multiple heuristics and we will lose the advantage of clang-format tool using the inbuilt parser.

Indeed.

> Second way is to extend the functionality of clang-format. I have experience working with compilers. Going through the code of clang it seems possible to include kw_do token for do while loop as a functionality for braces wrapping( although it would require a very thorough reading through the codebase first). 

One of the objectives behind using clang was that we wanted to be able to use a compiler front-end that has a plug-in model or a library which allows access to the AST. Clang-format appeared to fit the bill. 

> At present, I am working on creating a clang format file supporting subset of coding formats for Xen Hypervisor.

I think that would fit with what I suggested.

> Can you provide me with some more insight into the project?

There are really two objectives:

A) A tool which can be run over submissions by new contributors. Right now, adherence to coding standards has to be manually checked. Having a tool which lists coding standard violations in a git-patch or in the files a patch touches would allow contributors to run the tool before submitting the code. Ideally, it could be run over a patch, but that may be harder than running it over the files the patch touches. In any case, it would improve the work-flow.

B) Be able to run the tool on the entire code base.

We also want to make this as easy to maintain in future as possible, which is why we originally suggested clang-format. It might in fact be easier or cleaner to use LibTooling rather than clang-format for this as base-line and use clang-format as an inspiration (looking at the source of clang-format, there does not seem to be that much).  Keeping as close as possibly to the clang-format config file. The downside is that we would have to maintain such a tool in the long-run (but then we would have to do the same if we used a stand-alone tool using script hackery). I don't think either Doug, Andy, or me could give you concrete advice on what the best way forward is here without further discussion though.

> clang-format formats the tool to specified format. However, there are many standards which include a particular format for a variable name. It is not feasible for a tool to automatically change the name of variable according to given standards because of possible breaking build of the code.

That is not the intention: it is definitely not desirable to do that kind of refactoring automatically. 

> As a part of the project, are we required to create an auto-formatter or something like pylint which suggests the necessary changes and gives an overall score, or a combination of both?

We don't need an auto-formatter. Just a tool - such a pylint - which runs on our codebase and makes suggestions on how the code should look like. This should make things easier. 

> I wish to apply to GSOC as well for the same project. However, since the deadline is today, I will be grateful if you could provide some suggestions for the project.

I see you submitted already

Regards
Lars
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Ishani]

Hello,

I have created a clang-format file for specifications of format for Xen hypervisor and incorporated all of the support which present clang-format could provide. I have seen all the options provided by clang-format and have some discrepancies regarding some of format specifications which are not provided in the doc but is followed. I will commit the code and provide you with full report by today.

>4: a small script that checks that .clang-format works correctly on these pieces of code

>For 4, you probably want to run clang-format with -output-replacements-xml on the files in 1 >- 3 and then do some grep magic to see whether it does the right thing.

Can you elaborate on this a bit. What is the expected end product?

Meanwhile, I have resumed on writing the proposal. I somehow missed previous comments. Please have a look at it.
https://docs.google.com/document/d/10NJn-QvO1TvyJJJGE2PD6FtElYCT3neBAffIqeWHdiE/edit?usp=sharing

Regards,
Ishani

----- Original Message -----
From: "Lars Kurth" <lars.kurth.xen@gmail.com>
To: "Ishani" <chugh.ishani@research.iiit.ac.in>
Cc: "cardoe" <cardoe@cardoe.com>, "xen-devel" <xen-devel@lists.xenproject.org>, "Andrew Cooper" <andrew.cooper3@citrix.com>
Sent: Monday, April 3, 2017 9:52:38 PM
Subject: Re: [Xen-devel] [Outreachy] Interested in contribution: Code Standards Checking using clang-format

Hi Ishani,

> On 3 Apr 2017, at 13:01, Ishani <chugh.ishani@research.iiit.ac.in> wrote:
> 
> Thanks for the reply. I started with looking into format of Xen Hypervisor files and creating a clang format file for the same. However, given the present implementation of clang-format, it is not possible to incorporate fine-grained custom coding format requirements. For example, although clang-format provides a custom brace wrapping format specification, it is still not sufficient to incorporate the do while loop exception for braces in Xen Hypervisor format.

OK, this makes this project a lot harder. I don't think we are particularly tied to clang-format. 

> The implementations is presently on higher layers of abstraction and needs to be fine grained to allow for different types of loops as well. I figured that there are two ways of doing it, First , by creating a wrapper over clang-format and using regexes to find appropriate partitions of code to follow different format styles.
> However, this will be rather cumbersome with multiple heuristics and we will lose the advantage of clang-format tool using the inbuilt parser.

Indeed.

> Second way is to extend the functionality of clang-format. I have experience working with compilers. Going through the code of clang it seems possible to include kw_do token for do while loop as a functionality for braces wrapping( although it would require a very thorough reading through the codebase first). 

One of the objectives behind using clang was that we wanted to be able to use a compiler front-end that has a plug-in model or a library which allows access to the AST. Clang-format appeared to fit the bill. 

> At present, I am working on creating a clang format file supporting subset of coding formats for Xen Hypervisor.

I think that would fit with what I suggested.

> Can you provide me with some more insight into the project?

There are really two objectives:

A) A tool which can be run over submissions by new contributors. Right now, adherence to coding standards has to be manually checked. Having a tool which lists coding standard violations in a git-patch or in the files a patch touches would allow contributors to run the tool before submitting the code. Ideally, it could be run over a patch, but that may be harder than running it over the files the patch touches. In any case, it would improve the work-flow.

B) Be able to run the tool on the entire code base.

We also want to make this as easy to maintain in future as possible, which is why we originally suggested clang-format. It might in fact be easier or cleaner to use LibTooling rather than clang-format for this as base-line and use clang-format as an inspiration (looking at the source of clang-format, there does not seem to be that much).  Keeping as close as possibly to the clang-format config file. The downside is that we would have to maintain such a tool in the long-run (but then we would have to do the same if we used a stand-alone tool using script hackery). I don't think either Doug, Andy, or me could give you concrete advice on what the best way forward is here without further discussion though.

> clang-format formats the tool to specified format. However, there are many standards which include a particular format for a variable name. It is not feasible for a tool to automatically change the name of variable according to given standards because of possible breaking build of the code.

That is not the intention: it is definitely not desirable to do that kind of refactoring automatically. 

> As a part of the project, are we required to create an auto-formatter or something like pylint which suggests the necessary changes and gives an overall score, or a combination of both?

We don't need an auto-formatter. Just a tool - such a pylint - which runs on our codebase and makes suggestions on how the code should look like. This should make things easier. 

> I wish to apply to GSOC as well for the same project. However, since the deadline is today, I will be grateful if you could provide some suggestions for the project.

I see you submitted already

Regards
Lars

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Lars Kurth]

Ishani,

> On 11 Apr 2017, at 12:21, Ishani <chugh.ishani@research.iiit.ac.in> wrote:
> 
> Hello,
> 
> I have created a clang-format file for specifications of format for Xen hypervisor and incorporated all of the support which present clang-format could provide. I have seen all the options provided by clang-format and have some discrepancies regarding some of format specifications which are not provided in the doc but is followed. I will commit the code and provide you with full report by today.

Can you post this on any public repo for now (e.g. github gitlab bitbucket, whatever) and post an e-mail on this list with title "[RFC] Code Standards Checking using clang-format" or something like it with link to the repo. I think this is enough for the small task required for this project.

Ideally, because at some point you will need to do this, you would follow https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches - but we could waive this in this case. But it will make reviewing the output easier.

>> 4: a small script that checks that .clang-format works correctly on these pieces of code
> 
>> For 4, you probably want to run clang-format with -output-replacements-xml on the files in 1 >- 3 and then do some grep magic to see whether it does the right thing.
> 
> Can you elaborate on this a bit. What is the expected end product?

I think this is a stretch goal, given the time-frame. The idea here was to be able to verify that the clang-format file works as expected on some code snippets that we know adhere to the coding standards. This would in essence produce a stand-alone test for the .clang-format file on the coding style patterns that we know works. 

> Meanwhile, I have resumed on writing the proposal. I somehow missed previous comments. Please have a look at it.
> https://docs.google.com/document/d/10NJn-QvO1TvyJJJGE2PD6FtElYCT3neBAffIqeWHdiE/edit?usp=sharing

I saw it. Thank you!

Best Regards
Lars
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Outreachy] Interested in contribution: Code Standards Checking using clang-format

[Ishani]

Hello, 

Please review the following file for Xen Hypervisor format specification given in http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CODING_STYLE 

The link to the file is:
https://github.com/sam5125/Xen-Clang-format/blob/master/Supported/correct.c

Regards,
Ishani

----- Original Message -----
From: "Lars Kurth" <lars.kurth.xen@gmail.com>
To: "Ishani" <chugh.ishani@research.iiit.ac.in>
Cc: "cardoe" <cardoe@cardoe.com>, "xen-devel" <xen-devel@lists.xenproject.org>, "Andrew Cooper" <andrew.cooper3@citrix.com>, "Wei Liu" <wei.liu2@citrix.com>
Sent: Tuesday, April 11, 2017 5:22:02 PM
Subject: Re: [Xen-devel] [Outreachy] Interested in contribution: Code Standards Checking using clang-format

Ishani,

> On 11 Apr 2017, at 12:21, Ishani <chugh.ishani@research.iiit.ac.in> wrote:
> 
> Hello,
> 
> I have created a clang-format file for specifications of format for Xen hypervisor and incorporated all of the support which present clang-format could provide. I have seen all the options provided by clang-format and have some discrepancies regarding some of format specifications which are not provided in the doc but is followed. I will commit the code and provide you with full report by today.

Can you post this on any public repo for now (e.g. github gitlab bitbucket, whatever) and post an e-mail on this list with title "[RFC] Code Standards Checking using clang-format" or something like it with link to the repo. I think this is enough for the small task required for this project.

Ideally, because at some point you will need to do this, you would follow https://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches - but we could waive this in this case. But it will make reviewing the output easier.

>> 4: a small script that checks that .clang-format works correctly on these pieces of code
> 
>> For 4, you probably want to run clang-format with -output-replacements-xml on the files in 1 >- 3 and then do some grep magic to see whether it does the right thing.
> 
> Can you elaborate on this a bit. What is the expected end product?

I think this is a stretch goal, given the time-frame. The idea here was to be able to verify that the clang-format file works as expected on some code snippets that we know adhere to the coding standards. This would in essence produce a stand-alone test for the .clang-format file on the coding style patterns that we know works. 

> Meanwhile, I have resumed on writing the proposal. I somehow missed previous comments. Please have a look at it.
> https://docs.google.com/document/d/10NJn-QvO1TvyJJJGE2PD6FtElYCT3neBAffIqeWHdiE/edit?usp=sharing

I saw it. Thank you!

Best Regards
Lars

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel