[SPDK] Re: Preparation for SPDK 19.10 release

[SPDK] Re: Preparation for SPDK 19.10 release

[Zawadzki, Tomasz]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

Hi all,

Merge window for SPDK 19.10 release is now closed.
Branch 'v19.10.x' is now created and patch on it tagged as 'v19.10-rc1'.

SPDK 19.10 release is planned for October 31st.

Until then process used for Maintenance releases will be used
to apply any critical fixes on the branch.
First patch has to be submitted/reviewed/merged on master branch.
At any point please add hashtag '19.10' to the patch. 
Then such patch can be cherry-picked to 'v19.10.x' branch.
Current list of patches can always be seen here:
https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)

PS. There is topic submitted for next Community meeting
regarding NVMe-oF validation in preparation for release.
Please feel free to join and discuss, topics related to SPDK releases or any others at:
https://spdk.io/community/

Thanks,
Tomek

> -----Original Message-----
> From: Zawadzki, Tomasz [mailto:tomasz.zawadzki(a)intel.com]
> Sent: Wednesday, October 9, 2019 4:41 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Preparation for SPDK 19.10 release
> 
> Hello all,
> 
> The merge window for SPDK 19.10 release will close by October 24th.
> Please ensure all patches you believe should be included in the release are
> merged to master branch by this date.
> You can do it by adding a hashtag '19.10' in GerritHub on those patches.
> 
> The current set of patches that are tagged and need to be reviewed can be
> seen here:
> https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
> 
> Starting with this release, process used so far for Maintenance releases (ex.
> 19.04.1) will be applied to main releases. On October 24th new branch
> 'v19.10.x' will be created, and a patch on it will be tagged as release
> candidate.
> Then, on October 31th, a formal release will take place tagging the last patch
> on the branch as SPDK 19.10.
> 
> Between release candidate and formal release, only critical fixes shall be
> backported to the 'v19.10.x' branch.
> 
> Development can continue without disruptions on 'master' branch.
> 
> Thanks,
> Tomek
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: Preparation for SPDK 19.10 release

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 10234 bytes --]

I think discussion on a call is a great idea, I've got lots of lots of input ☺ I put it on the agenda already, we can do something sooner if you'd like Sasha, we can always fire up a freeconference call at any time as long as get those who have the most interest in participating to agree on a time....

Thx
Paul

On 10/16/19, 10:13 AM, "Harris, James R" <james.r.harris(a)intel.com> wrote:

    Hi Sasha,
    
    See inline [Jim] below.
    
    -Jim
    
    
    On 10/16/19, 10:01 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:
    
        Hi Jim,
        
        PSB [SK]
        
        -----Original Message-----
        From: Harris, James R <james.r.harris(a)intel.com> 
        Sent: Wednesday, October 16, 2019 6:22 PM
        To: Storage Performance Development Kit <spdk(a)lists.01.org>
        Subject: [SPDK] Re: Preparation for SPDK 19.10 release
        
        Hi Sasha,
        
        The CHANGELOG is built during the months leading up to the release, and Tomek typically does some cleanups just before the release.  If you'd like to help with finalizing the CHANGELOG before the release, please push some patches!  Documentation is always an area where the project could use more help.
        
        [SK] Changelog is good. I'm talking about "Release notes"
        https://github.com/spdk/spdk/releases/tag/v19.07
        I think, that's first point to see that's new in the release
    
    [Jim]  OK.  It sounded like there were also some concerns that some things were getting missed in the CHANGELOG.  So any help there would still be appreciated.
    
    [Jim]  Regarding copying CHANGELOG contents to the GitHub release notes:  looking at CHANGELOG.md, the v19.07 section was almost 300 lines of text.  My personal preference would be to not copy that verbatim into the GitHub release notes.  If we could keep the Release Notes high level, and provide a link in the release notes to the relevant section of the CHANGELOG, would that be sufficient?
        
        For the RDMA testing, I have some of the same questions as Paul.  If there are gaps in upstream per-patch or nightly tests, please help us get them closed.  Can you describe the tests you're running internally, how often they take, and how often you are running them?
        
        [SK] Correct me if I'm wrong, but most of existing tests in SPDK use SoftRoce, or single host without actual transferring data over network.  Runtime behavior on real HW can be different. 
        Our verification system today tests solutions which include SPDK as building block. We work with stable version of SPDK and don't update it frequently. It mostly focus on Mellanox's parts and less on SPDK.
        From functional perspective, we test complex scenarios like OS install/boot to/from distributed storage.  We test reboots and reconnects. In addition,  we test on non-x86 platform too.
        For upstream SPDK (reacting on each patch) we have limited number of sanity tests.
        From my point of view, for network part the verification system should test:
        - Matrix of HW, SW stacks (OS,  OFED)
        - Matrix of protocols: RDMA, TCP 
        - Corner cases like link failure, network congestion , SW/HW failures during real traffic
        - Stress testing
        - Long tests 
    
    [Jim]  Thanks for this level of detail Sasha.  I think this would be a great agenda topic for next community meeting.   Could we plan on discussing this further at the October 29th meeting?
    
    -Jim
    
        
        
        Thanks,
        
        -Jim
        
        On 10/16/19, 7:47 AM, "Luse, Paul E" <paul.e.luse(a)intel.com> wrote:
        
            Hi Sasha,
            
            I'm sure you'll get some other opinions as well but as the changelog now pretty much only contains interface and parameter changes sorted by module, are you just suggesting re-organizing it to make it clearer per module where interface changes have occurred vs parameter changes?
            
            Wrt testing, I love that you guys are testing this so thoroughly! I have to ask though, what additional testing are you proposing that isn't already in either the main CI or your own CI? Is there a possibility to get these tests unstreamed? We really want to continue with a model of continuous integration and test without any additional testing nearing a release.  The freeze window is to provide time to prepare the release, make sure we've got everything in it that was intended, go through some Intel security checklist stuff, etc.  Ben said it best once at a summit, our releases are really just a bookkeeping activity.
            
            You can most definitely use that freeze window to do testing on your own and submit a bug that could potentially halt the release, but I'd wouldn't support making that part of the process.
            
            Thx
            Paul
            
            On 10/16/19, 4:55 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:
            
                Hi Tomek,
                
                That's great news. 
                
                I have couple questions:
                
                - Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
                serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
                wait with formal release till our group in Mellanox will verify release
                candidate in internal verification system? I think, couple of days will be
                enough for us provide a feedback to the community.
                -  Can we add couple special sections to SPDK release notes
                (https://github.com/spdk/spdk/releases/tag/v19.07) ? 
                 	- Changes in interfaces
                	- Changes in configuration parameters (including defaults)
                	- Known bugs/issues
                For customers building solution on SPDK and considering upgrade, it's
                important to see changes in interfaces (including configuration parameters).
                
                
                Example of changes in an interface in coming 19.10 is 
                https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
                #diff-2c205ae4170cbd2bbf219969f03c9c8d
                This commit adds a new  event to bdev interface.
                
                One of commits changed defaults was
                https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
                It changes default IOUnit size from 4K to 8K and affected existing
                configurations.
                
                From my experience, having a table/list in release notes with such changes
                (interfaces, parameters) can help to users to understand side-effects of the
                upgrade.
                I would suggest to maintain the table in Changelog and ask for contributors
                to update it is a part of patch. At release, this table can be just copied
                to release notes.  
                
                Best regards
                Sasha
                
                -----Original Message-----
                From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
                Sent: Wednesday, October 9, 2019 5:41 PM
                To: Storage Performance Development Kit <spdk(a)lists.01.org>
                Subject: [SPDK] Preparation for SPDK 19.10 release
                
                Hello all,
                
                The merge window for SPDK 19.10 release will close by October 24th.
                Please ensure all patches you believe should be included in the release are
                merged to master branch by this date.
                You can do it by adding a hashtag '19.10' in GerritHub on those patches.
                
                The current set of patches that are tagged and need to be reviewed can be
                seen here:
                https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
                
                Starting with this release, process used so far for Maintenance releases
                (ex. 19.04.1) will be applied to main releases. On October 24th new branch
                'v19.10.x' will be created, and a patch on it will be tagged as release
                candidate.
                Then, on October 31th, a formal release will take place tagging the last
                patch on the branch as SPDK 19.10.
                
                Between release candidate and formal release, only critical fixes shall be
                backported to the 'v19.10.x' branch.
                
                Development can continue without disruptions on 'master' branch.
                
                Thanks,
                Tomek
                _______________________________________________
                SPDK mailing list -- spdk(a)lists.01.org
                To unsubscribe send an email to spdk-leave(a)lists.01.org
                _______________________________________________
                SPDK mailing list -- spdk(a)lists.01.org
                To unsubscribe send an email to spdk-leave(a)lists.01.org
                
            
            _______________________________________________
            SPDK mailing list -- spdk(a)lists.01.org
            To unsubscribe send an email to spdk-leave(a)lists.01.org
            
        
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        
    
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

[SPDK] Re: Preparation for SPDK 19.10 release

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 9019 bytes --]

Hi Sasha,

See inline [Jim] below.

-Jim


On 10/16/19, 10:01 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:

    Hi Jim,
    
    PSB [SK]
    
    -----Original Message-----
    From: Harris, James R <james.r.harris(a)intel.com> 
    Sent: Wednesday, October 16, 2019 6:22 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: [SPDK] Re: Preparation for SPDK 19.10 release
    
    Hi Sasha,
    
    The CHANGELOG is built during the months leading up to the release, and Tomek typically does some cleanups just before the release.  If you'd like to help with finalizing the CHANGELOG before the release, please push some patches!  Documentation is always an area where the project could use more help.
    
    [SK] Changelog is good. I'm talking about "Release notes"
    https://github.com/spdk/spdk/releases/tag/v19.07
    I think, that's first point to see that's new in the release

[Jim]  OK.  It sounded like there were also some concerns that some things were getting missed in the CHANGELOG.  So any help there would still be appreciated.

[Jim]  Regarding copying CHANGELOG contents to the GitHub release notes:  looking at CHANGELOG.md, the v19.07 section was almost 300 lines of text.  My personal preference would be to not copy that verbatim into the GitHub release notes.  If we could keep the Release Notes high level, and provide a link in the release notes to the relevant section of the CHANGELOG, would that be sufficient?
    
    For the RDMA testing, I have some of the same questions as Paul.  If there are gaps in upstream per-patch or nightly tests, please help us get them closed.  Can you describe the tests you're running internally, how often they take, and how often you are running them?
    
    [SK] Correct me if I'm wrong, but most of existing tests in SPDK use SoftRoce, or single host without actual transferring data over network.  Runtime behavior on real HW can be different. 
    Our verification system today tests solutions which include SPDK as building block. We work with stable version of SPDK and don't update it frequently. It mostly focus on Mellanox's parts and less on SPDK.
    From functional perspective, we test complex scenarios like OS install/boot to/from distributed storage.  We test reboots and reconnects. In addition,  we test on non-x86 platform too.
    For upstream SPDK (reacting on each patch) we have limited number of sanity tests.
    From my point of view, for network part the verification system should test:
    - Matrix of HW, SW stacks (OS,  OFED)
    - Matrix of protocols: RDMA, TCP 
    - Corner cases like link failure, network congestion , SW/HW failures during real traffic
    - Stress testing
    - Long tests 

[Jim]  Thanks for this level of detail Sasha.  I think this would be a great agenda topic for next community meeting.   Could we plan on discussing this further at the October 29th meeting?

-Jim

    
    
    Thanks,
    
    -Jim
    
    On 10/16/19, 7:47 AM, "Luse, Paul E" <paul.e.luse(a)intel.com> wrote:
    
        Hi Sasha,
        
        I'm sure you'll get some other opinions as well but as the changelog now pretty much only contains interface and parameter changes sorted by module, are you just suggesting re-organizing it to make it clearer per module where interface changes have occurred vs parameter changes?
        
        Wrt testing, I love that you guys are testing this so thoroughly! I have to ask though, what additional testing are you proposing that isn't already in either the main CI or your own CI? Is there a possibility to get these tests unstreamed? We really want to continue with a model of continuous integration and test without any additional testing nearing a release.  The freeze window is to provide time to prepare the release, make sure we've got everything in it that was intended, go through some Intel security checklist stuff, etc.  Ben said it best once at a summit, our releases are really just a bookkeeping activity.
        
        You can most definitely use that freeze window to do testing on your own and submit a bug that could potentially halt the release, but I'd wouldn't support making that part of the process.
        
        Thx
        Paul
        
        On 10/16/19, 4:55 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:
        
            Hi Tomek,
            
            That's great news. 
            
            I have couple questions:
            
            - Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
            serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
            wait with formal release till our group in Mellanox will verify release
            candidate in internal verification system? I think, couple of days will be
            enough for us provide a feedback to the community.
            -  Can we add couple special sections to SPDK release notes
            (https://github.com/spdk/spdk/releases/tag/v19.07) ? 
             	- Changes in interfaces
            	- Changes in configuration parameters (including defaults)
            	- Known bugs/issues
            For customers building solution on SPDK and considering upgrade, it's
            important to see changes in interfaces (including configuration parameters).
            
            
            Example of changes in an interface in coming 19.10 is 
            https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
            #diff-2c205ae4170cbd2bbf219969f03c9c8d
            This commit adds a new  event to bdev interface.
            
            One of commits changed defaults was
            https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
            It changes default IOUnit size from 4K to 8K and affected existing
            configurations.
            
            From my experience, having a table/list in release notes with such changes
            (interfaces, parameters) can help to users to understand side-effects of the
            upgrade.
            I would suggest to maintain the table in Changelog and ask for contributors
            to update it is a part of patch. At release, this table can be just copied
            to release notes.  
            
            Best regards
            Sasha
            
            -----Original Message-----
            From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
            Sent: Wednesday, October 9, 2019 5:41 PM
            To: Storage Performance Development Kit <spdk(a)lists.01.org>
            Subject: [SPDK] Preparation for SPDK 19.10 release
            
            Hello all,
            
            The merge window for SPDK 19.10 release will close by October 24th.
            Please ensure all patches you believe should be included in the release are
            merged to master branch by this date.
            You can do it by adding a hashtag '19.10' in GerritHub on those patches.
            
            The current set of patches that are tagged and need to be reviewed can be
            seen here:
            https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
            
            Starting with this release, process used so far for Maintenance releases
            (ex. 19.04.1) will be applied to main releases. On October 24th new branch
            'v19.10.x' will be created, and a patch on it will be tagged as release
            candidate.
            Then, on October 31th, a formal release will take place tagging the last
            patch on the branch as SPDK 19.10.
            
            Between release candidate and formal release, only critical fixes shall be
            backported to the 'v19.10.x' branch.
            
            Development can continue without disruptions on 'master' branch.
            
            Thanks,
            Tomek
            _______________________________________________
            SPDK mailing list -- spdk(a)lists.01.org
            To unsubscribe send an email to spdk-leave(a)lists.01.org
            _______________________________________________
            SPDK mailing list -- spdk(a)lists.01.org
            To unsubscribe send an email to spdk-leave(a)lists.01.org
            
        
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        
    
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

[SPDK] Re: Preparation for SPDK 19.10 release

[Sasha Kotchubievsky]

[-- Attachment #1: Type: text/plain, Size: 7399 bytes --]

Hi Jim,

PSB [SK]

-----Original Message-----
From: Harris, James R <james.r.harris(a)intel.com> 
Sent: Wednesday, October 16, 2019 6:22 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: Preparation for SPDK 19.10 release

Hi Sasha,

The CHANGELOG is built during the months leading up to the release, and Tomek typically does some cleanups just before the release.  If you'd like to help with finalizing the CHANGELOG before the release, please push some patches!  Documentation is always an area where the project could use more help.

[SK] Changelog is good. I'm talking about "Release notes"
https://github.com/spdk/spdk/releases/tag/v19.07
I think, that's first point to see that's new in the release

For the RDMA testing, I have some of the same questions as Paul.  If there are gaps in upstream per-patch or nightly tests, please help us get them closed.  Can you describe the tests you're running internally, how often they take, and how often you are running them?

[SK] Correct me if I'm wrong, but most of existing tests in SPDK use SoftRoce, or single host without actual transferring data over network.  Runtime behavior on real HW can be different. 
Our verification system today tests solutions which include SPDK as building block. We work with stable version of SPDK and don't update it frequently. It mostly focus on Mellanox's parts and less on SPDK.
From functional perspective, we test complex scenarios like OS install/boot to/from distributed storage.  We test reboots and reconnects. In addition,  we test on non-x86 platform too.
For upstream SPDK (reacting on each patch) we have limited number of sanity tests.
From my point of view, for network part the verification system should test:
- Matrix of HW, SW stacks (OS,  OFED)
- Matrix of protocols: RDMA, TCP 
- Corner cases like link failure, network congestion , SW/HW failures during real traffic
- Stress testing
- Long tests 



Thanks,

-Jim

On 10/16/19, 7:47 AM, "Luse, Paul E" <paul.e.luse(a)intel.com> wrote:

    Hi Sasha,
    
    I'm sure you'll get some other opinions as well but as the changelog now pretty much only contains interface and parameter changes sorted by module, are you just suggesting re-organizing it to make it clearer per module where interface changes have occurred vs parameter changes?
    
    Wrt testing, I love that you guys are testing this so thoroughly! I have to ask though, what additional testing are you proposing that isn't already in either the main CI or your own CI? Is there a possibility to get these tests unstreamed? We really want to continue with a model of continuous integration and test without any additional testing nearing a release.  The freeze window is to provide time to prepare the release, make sure we've got everything in it that was intended, go through some Intel security checklist stuff, etc.  Ben said it best once at a summit, our releases are really just a bookkeeping activity.
    
    You can most definitely use that freeze window to do testing on your own and submit a bug that could potentially halt the release, but I'd wouldn't support making that part of the process.
    
    Thx
    Paul
    
    On 10/16/19, 4:55 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:
    
        Hi Tomek,
        
        That's great news. 
        
        I have couple questions:
        
        - Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
        serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
        wait with formal release till our group in Mellanox will verify release
        candidate in internal verification system? I think, couple of days will be
        enough for us provide a feedback to the community.
        -  Can we add couple special sections to SPDK release notes
        (https://github.com/spdk/spdk/releases/tag/v19.07) ? 
         	- Changes in interfaces
        	- Changes in configuration parameters (including defaults)
        	- Known bugs/issues
        For customers building solution on SPDK and considering upgrade, it's
        important to see changes in interfaces (including configuration parameters).
        
        
        Example of changes in an interface in coming 19.10 is 
        https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
        #diff-2c205ae4170cbd2bbf219969f03c9c8d
        This commit adds a new  event to bdev interface.
        
        One of commits changed defaults was
        https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
        It changes default IOUnit size from 4K to 8K and affected existing
        configurations.
        
        From my experience, having a table/list in release notes with such changes
        (interfaces, parameters) can help to users to understand side-effects of the
        upgrade.
        I would suggest to maintain the table in Changelog and ask for contributors
        to update it is a part of patch. At release, this table can be just copied
        to release notes.  
        
        Best regards
        Sasha
        
        -----Original Message-----
        From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
        Sent: Wednesday, October 9, 2019 5:41 PM
        To: Storage Performance Development Kit <spdk(a)lists.01.org>
        Subject: [SPDK] Preparation for SPDK 19.10 release
        
        Hello all,
        
        The merge window for SPDK 19.10 release will close by October 24th.
        Please ensure all patches you believe should be included in the release are
        merged to master branch by this date.
        You can do it by adding a hashtag '19.10' in GerritHub on those patches.
        
        The current set of patches that are tagged and need to be reviewed can be
        seen here:
        https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
        
        Starting with this release, process used so far for Maintenance releases
        (ex. 19.04.1) will be applied to main releases. On October 24th new branch
        'v19.10.x' will be created, and a patch on it will be tagged as release
        candidate.
        Then, on October 31th, a formal release will take place tagging the last
        patch on the branch as SPDK 19.10.
        
        Between release candidate and formal release, only critical fixes shall be
        backported to the 'v19.10.x' branch.
        
        Development can continue without disruptions on 'master' branch.
        
        Thanks,
        Tomek
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        
    
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: Preparation for SPDK 19.10 release

[Sasha Kotchubievsky]

[-- Attachment #1: Type: text/plain, Size: 5848 bytes --]

Hi Paul,

PSB[SK]

-----Original Message-----
From: Luse, Paul E <paul.e.luse(a)intel.com> 
Sent: Wednesday, October 16, 2019 5:47 PM
To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
Subject: [SPDK] Re: Preparation for SPDK 19.10 release

Hi Sasha,

I'm sure you'll get some other opinions as well but as the changelog now pretty much only contains interface and parameter changes sorted by module, are you just suggesting re-organizing it to make it clearer per module where interface changes have occurred vs parameter changes?

[SK] 
I agree, changes in interfaces, mostly, are documented in changelog. I just suggest to add compact table/list with per module changes in "public" interfaces.
Changes parameters are less documented.
It would be nice to see the same information with focus on changes in interfaces in addition to list of feature descriptions.


Wrt testing, I love that you guys are testing this so thoroughly! I have to ask though, what additional testing are you proposing that isn't already in either the main CI or your own CI?

[SK] Our own CI

 Is there a possibility to get these tests unstreamed? We really want to continue with a model of continuous integration and test without any additional testing nearing a release.

[SK] Unfortunately, it's quite complicated. We focus on network part of SPDK and we test on real HW. I don't see how we can upstream such kind of testing. We're working on continuous testing for upstream SPDK, but at this stage it's not ready. On the other hand, our test suite complements existing SPDK in-house testing and we can provide valuable feedback at least for release version. More frequent testing will come later.


The freeze window is to provide time to prepare the release, make sure we've got everything in it that was intended, go through some Intel security checklist stuff, etc.  Ben said it best once at a summit, our releases are really just a bookkeeping activity.

You can most definitely use that freeze window to do testing on your own and submit a bug that could potentially halt the release, but I'd wouldn't support making that part of the process.

Thx
Paul

On 10/16/19, 4:55 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:

    Hi Tomek,
    
    That's great news. 
    
    I have couple questions:
    
    - Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
    serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
    wait with formal release till our group in Mellanox will verify release
    candidate in internal verification system? I think, couple of days will be
    enough for us provide a feedback to the community.
    -  Can we add couple special sections to SPDK release notes
    (https://github.com/spdk/spdk/releases/tag/v19.07) ? 
     	- Changes in interfaces
    	- Changes in configuration parameters (including defaults)
    	- Known bugs/issues
    For customers building solution on SPDK and considering upgrade, it's
    important to see changes in interfaces (including configuration parameters).
    
    
    Example of changes in an interface in coming 19.10 is 
    https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
    #diff-2c205ae4170cbd2bbf219969f03c9c8d
    This commit adds a new  event to bdev interface.
    
    One of commits changed defaults was
    https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
    It changes default IOUnit size from 4K to 8K and affected existing
    configurations.
    
    From my experience, having a table/list in release notes with such changes
    (interfaces, parameters) can help to users to understand side-effects of the
    upgrade.
    I would suggest to maintain the table in Changelog and ask for contributors
    to update it is a part of patch. At release, this table can be just copied
    to release notes.  
    
    Best regards
    Sasha
    
    -----Original Message-----
    From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
    Sent: Wednesday, October 9, 2019 5:41 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: [SPDK] Preparation for SPDK 19.10 release
    
    Hello all,
    
    The merge window for SPDK 19.10 release will close by October 24th.
    Please ensure all patches you believe should be included in the release are
    merged to master branch by this date.
    You can do it by adding a hashtag '19.10' in GerritHub on those patches.
    
    The current set of patches that are tagged and need to be reviewed can be
    seen here:
    https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
    
    Starting with this release, process used so far for Maintenance releases
    (ex. 19.04.1) will be applied to main releases. On October 24th new branch
    'v19.10.x' will be created, and a patch on it will be tagged as release
    candidate.
    Then, on October 31th, a formal release will take place tagging the last
    patch on the branch as SPDK 19.10.
    
    Between release candidate and formal release, only critical fixes shall be
    backported to the 'v19.10.x' branch.
    
    Development can continue without disruptions on 'master' branch.
    
    Thanks,
    Tomek
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: Preparation for SPDK 19.10 release

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 5859 bytes --]

Hi Sasha,

The CHANGELOG is built during the months leading up to the release, and Tomek typically does some cleanups just before the release.  If you'd like to help with finalizing the CHANGELOG before the release, please push some patches!  Documentation is always an area where the project could use more help.

For the RDMA testing, I have some of the same questions as Paul.  If there are gaps in upstream per-patch or nightly tests, please help us get them closed.  Can you describe the tests you're running internally, how often they take, and how often you are running them?

Thanks,

-Jim

On 10/16/19, 7:47 AM, "Luse, Paul E" <paul.e.luse(a)intel.com> wrote:

    Hi Sasha,
    
    I'm sure you'll get some other opinions as well but as the changelog now pretty much only contains interface and parameter changes sorted by module, are you just suggesting re-organizing it to make it clearer per module where interface changes have occurred vs parameter changes?
    
    Wrt testing, I love that you guys are testing this so thoroughly! I have to ask though, what additional testing are you proposing that isn't already in either the main CI or your own CI? Is there a possibility to get these tests unstreamed? We really want to continue with a model of continuous integration and test without any additional testing nearing a release.  The freeze window is to provide time to prepare the release, make sure we've got everything in it that was intended, go through some Intel security checklist stuff, etc.  Ben said it best once at a summit, our releases are really just a bookkeeping activity.
    
    You can most definitely use that freeze window to do testing on your own and submit a bug that could potentially halt the release, but I'd wouldn't support making that part of the process.
    
    Thx
    Paul
    
    On 10/16/19, 4:55 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:
    
        Hi Tomek,
        
        That's great news. 
        
        I have couple questions:
        
        - Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
        serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
        wait with formal release till our group in Mellanox will verify release
        candidate in internal verification system? I think, couple of days will be
        enough for us provide a feedback to the community.
        -  Can we add couple special sections to SPDK release notes
        (https://github.com/spdk/spdk/releases/tag/v19.07) ? 
         	- Changes in interfaces
        	- Changes in configuration parameters (including defaults)
        	- Known bugs/issues
        For customers building solution on SPDK and considering upgrade, it's
        important to see changes in interfaces (including configuration parameters).
        
        
        Example of changes in an interface in coming 19.10 is 
        https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
        #diff-2c205ae4170cbd2bbf219969f03c9c8d
        This commit adds a new  event to bdev interface.
        
        One of commits changed defaults was
        https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
        It changes default IOUnit size from 4K to 8K and affected existing
        configurations.
        
        From my experience, having a table/list in release notes with such changes
        (interfaces, parameters) can help to users to understand side-effects of the
        upgrade.
        I would suggest to maintain the table in Changelog and ask for contributors
        to update it is a part of patch. At release, this table can be just copied
        to release notes.  
        
        Best regards
        Sasha
        
        -----Original Message-----
        From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
        Sent: Wednesday, October 9, 2019 5:41 PM
        To: Storage Performance Development Kit <spdk(a)lists.01.org>
        Subject: [SPDK] Preparation for SPDK 19.10 release
        
        Hello all,
        
        The merge window for SPDK 19.10 release will close by October 24th.
        Please ensure all patches you believe should be included in the release are
        merged to master branch by this date.
        You can do it by adding a hashtag '19.10' in GerritHub on those patches.
        
        The current set of patches that are tagged and need to be reviewed can be
        seen here:
        https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
        
        Starting with this release, process used so far for Maintenance releases
        (ex. 19.04.1) will be applied to main releases. On October 24th new branch
        'v19.10.x' will be created, and a patch on it will be tagged as release
        candidate.
        Then, on October 31th, a formal release will take place tagging the last
        patch on the branch as SPDK 19.10.
        
        Between release candidate and formal release, only critical fixes shall be
        backported to the 'v19.10.x' branch.
        
        Development can continue without disruptions on 'master' branch.
        
        Thanks,
        Tomek
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org
        
    
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

[SPDK] Re: Preparation for SPDK 19.10 release

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 4639 bytes --]

Hi Sasha,

I'm sure you'll get some other opinions as well but as the changelog now pretty much only contains interface and parameter changes sorted by module, are you just suggesting re-organizing it to make it clearer per module where interface changes have occurred vs parameter changes?

Wrt testing, I love that you guys are testing this so thoroughly! I have to ask though, what additional testing are you proposing that isn't already in either the main CI or your own CI? Is there a possibility to get these tests unstreamed? We really want to continue with a model of continuous integration and test without any additional testing nearing a release.  The freeze window is to provide time to prepare the release, make sure we've got everything in it that was intended, go through some Intel security checklist stuff, etc.  Ben said it best once at a summit, our releases are really just a bookkeeping activity.

You can most definitely use that freeze window to do testing on your own and submit a bug that could potentially halt the release, but I'd wouldn't support making that part of the process.

Thx
Paul

On 10/16/19, 4:55 AM, "Sasha Kotchubievsky" <sashakot(a)dev.mellanox.co.il> wrote:

    Hi Tomek,
    
    That's great news. 
    
    I have couple questions:
    
    - Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
    serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
    wait with formal release till our group in Mellanox will verify release
    candidate in internal verification system? I think, couple of days will be
    enough for us provide a feedback to the community.
    -  Can we add couple special sections to SPDK release notes
    (https://github.com/spdk/spdk/releases/tag/v19.07) ? 
     	- Changes in interfaces
    	- Changes in configuration parameters (including defaults)
    	- Known bugs/issues
    For customers building solution on SPDK and considering upgrade, it's
    important to see changes in interfaces (including configuration parameters).
    
    
    Example of changes in an interface in coming 19.10 is 
    https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
    #diff-2c205ae4170cbd2bbf219969f03c9c8d
    This commit adds a new  event to bdev interface.
    
    One of commits changed defaults was
    https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
    It changes default IOUnit size from 4K to 8K and affected existing
    configurations.
    
    From my experience, having a table/list in release notes with such changes
    (interfaces, parameters) can help to users to understand side-effects of the
    upgrade.
    I would suggest to maintain the table in Changelog and ask for contributors
    to update it is a part of patch. At release, this table can be just copied
    to release notes.  
    
    Best regards
    Sasha
    
    -----Original Message-----
    From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
    Sent: Wednesday, October 9, 2019 5:41 PM
    To: Storage Performance Development Kit <spdk(a)lists.01.org>
    Subject: [SPDK] Preparation for SPDK 19.10 release
    
    Hello all,
    
    The merge window for SPDK 19.10 release will close by October 24th.
    Please ensure all patches you believe should be included in the release are
    merged to master branch by this date.
    You can do it by adding a hashtag '19.10' in GerritHub on those patches.
    
    The current set of patches that are tagged and need to be reviewed can be
    seen here:
    https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
    
    Starting with this release, process used so far for Maintenance releases
    (ex. 19.04.1) will be applied to main releases. On October 24th new branch
    'v19.10.x' will be created, and a patch on it will be tagged as release
    candidate.
    Then, on October 31th, a formal release will take place tagging the last
    patch on the branch as SPDK 19.10.
    
    Between release candidate and formal release, only critical fixes shall be
    backported to the 'v19.10.x' branch.
    
    Development can continue without disruptions on 'master' branch.
    
    Thanks,
    Tomek
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

[SPDK] Re: Preparation for SPDK 19.10 release

[Sasha Kotchubievsky]

[-- Attachment #1: Type: text/plain, Size: 2954 bytes --]

Hi Tomek,

That's great news. 

I have couple questions:

- Some of latest SPDK releases (18.10, 19.04, 19.07) were released with
serious bugs in NVME-OF RDMA. Especially latest 19.07. Is it possible to
wait with formal release till our group in Mellanox will verify release
candidate in internal verification system? I think, couple of days will be
enough for us provide a feedback to the community.
-  Can we add couple special sections to SPDK release notes
(https://github.com/spdk/spdk/releases/tag/v19.07) ? 
 	- Changes in interfaces
	- Changes in configuration parameters (including defaults)
	- Known bugs/issues
For customers building solution on SPDK and considering upgrade, it's
important to see changes in interfaces (including configuration parameters).


Example of changes in an interface in coming 19.10 is 
https://github.com/spdk/spdk/commit/9522ed36f8474c9314b76b2618e8d2d9de56afee
#diff-2c205ae4170cbd2bbf219969f03c9c8d
This commit adds a new  event to bdev interface.

One of commits changed defaults was
https://github.com/spdk/spdk/commit/b6b0a0ba59a21faf1a320dd3ea7079b282f9fc31
It changes default IOUnit size from 4K to 8K and affected existing
configurations.

From my experience, having a table/list in release notes with such changes
(interfaces, parameters) can help to users to understand side-effects of the
upgrade.
I would suggest to maintain the table in Changelog and ask for contributors
to update it is a part of patch. At release, this table can be just copied
to release notes.  

Best regards
Sasha

-----Original Message-----
From: Zawadzki, Tomasz <tomasz.zawadzki(a)intel.com> 
Sent: Wednesday, October 9, 2019 5:41 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Preparation for SPDK 19.10 release

Hello all,

The merge window for SPDK 19.10 release will close by October 24th.
Please ensure all patches you believe should be included in the release are
merged to master branch by this date.
You can do it by adding a hashtag '19.10' in GerritHub on those patches.

The current set of patches that are tagged and need to be reviewed can be
seen here:
https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)

Starting with this release, process used so far for Maintenance releases
(ex. 19.04.1) will be applied to main releases. On October 24th new branch
'v19.10.x' will be created, and a patch on it will be tagged as release
candidate.
Then, on October 31th, a formal release will take place tagging the last
patch on the branch as SPDK 19.10.

Between release candidate and formal release, only critical fixes shall be
backported to the 'v19.10.x' branch.

Development can continue without disruptions on 'master' branch.

Thanks,
Tomek
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org