Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them

From: Keith Busch <kbusch@kernel.org>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Daniel Wagner <dwagner@suse.de>,
	linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org,
	James Smart <james.smart@broadcom.com>,
	Ming Lei <ming.lei@redhat.com>, Hannes Reinecke <hare@suse.de>,
	Wen Xiong <wenxiong@us.ibm.com>
Subject: Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them
Date: Tue, 10 Aug 2021 18:07:18 -0700	[thread overview]
Message-ID: <20210811010718.GA3135947@dhcp-10-100-145-180.wdc.com> (raw)
In-Reply-To: <01d7878c-e396-1d6b-c383-8739ca0b3d11@grimberg.me>

On Tue, Aug 10, 2021 at 06:00:37PM -0700, Sagi Grimberg wrote:
> 
> 
> On 8/9/21 1:52 AM, Daniel Wagner wrote:
> > Hi Sagi,
> > 
> > On Fri, Aug 06, 2021 at 12:57:17PM -0700, Sagi Grimberg wrote:
> > > > -	ret = nvme_tcp_start_io_queues(ctrl);
> > > > -	if (ret)
> > > > -		goto out_cleanup_connect_q;
> > > > -
> > > > -	if (!new) {
> > > > -		nvme_start_queues(ctrl);
> > > > +	} else if (prior_q_cnt != ctrl->queue_count) {
> > > 
> > > So if the queue count did not change we don't wait to make sure
> > > the queue g_usage_counter ref made it to zero? What guarantees that it
> > > did?
> > 
> > Hmm, good point. we should always call nvme_wait_freeze_timeout()
> > for !new queues. Is this what you are implying?
> 
> I think we should always wait for the freeze to complete.

Don't the queues need to be started in order for the freeze to complete?
Any enqueued requests on the quiesced queues will never complete this
way, so the wait_freeze() will be stuck, right? If so, I think the
nvme_start_queues() was in the correct place already.