Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Jeaho Hwang]

Hi.

We found an infinite loop inside the function hw_ep_set_halt
(drivers/usb/chipidea/udc.c) if a cablle is repeatedly
connnected/disconnected while ping through RNDIS with chipidea USB
device.

Using ftrace tracing, we found that hw_ep_set_halt is called due to
error return of hw_ep_prime(drivers/usb/chipidea/udc.c:202) which is
called from isr_tr_complete_handler -> isr_setup_status_phase ->
_ep_queue.

The comment of function hw_ep_prime says (execute without
interruption) but timer interrupt is occurred while hw_ep_prime is
executing. We believe that the interrupt causes an error return of
hw_ep_prime. We tried to protect hw_ep_prime from irqs and then no
case of the infinite loop is occurred.

I want ask if it is appropriate way that turning off irq inside
(threaded) irq handlers. And should we explicitly turn off irqs before
calling hw_ep_prime?

Thanks.

-- 
황재호, Jay Hwang, linux team manager of RTst
010-7242-1593

Re: Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Peter Chen]

On 21-08-09 10:45:16, Jeaho Hwang wrote:
> 2021년 8월 9일 (월) 오전 10:27, Peter Chen <peter.chen@kernel.org>님이 작성:
> >
> > On 21-08-04 11:03:44, Jeaho Hwang wrote:
> > > Hi. linux-usb and linux-rt experts.
> > > >
> > > > On 21-08-02 17:35:01, Jeaho Hwang wrote:
> > > > > Hi.
> > > > >
> > > > > We found an infinite loop inside the function hw_ep_set_halt
> > > > > (drivers/usb/chipidea/udc.c) if a cablle is repeatedly
> > > > > connnected/disconnected while ping through RNDIS with chipidea USB device.
> > > > >
> > > > > Using ftrace tracing, we found that hw_ep_set_halt is called due to error
> > > > > return of hw_ep_prime(drivers/usb/chipidea/udc.c:202) which is called from
> > > > > isr_tr_complete_handler -> isr_setup_status_phase -> _ep_queue.
> > > > >
> > > > > The comment of function hw_ep_prime says (execute without interruption) but
> > > > > timer interrupt is occurred while hw_ep_prime is executing. We believe that
> > > > > the interrupt causes an error return of hw_ep_prime. We tried to protect
> > > > > hw_ep_prime from irqs and then no case of the infinite loop is occurred.
> > > > >
> > > > > I want ask if it is appropriate way that turning off irq inside (threaded)
> > > > > irq handlers. And should we explicitly turn off irqs before calling
> > > > > hw_ep_prime?
> > > > >
> > > >
> > > > Jeaho, do you use RT-Linux or standard Linux? The function hw_ep_prime is
> > > > only called at udc_irq which is registered as top-half irq handlers.
> > > > Why the timer interrupt is occurred when hw_ep_prime is executing?
> > >
> > > We use preempt_RT so timer interrupt could be occurred. Now I found
> > > out that forced threaded irq handler disables local irq on standard
> > > linux so It is a linux-rt issue. Then should I make patch which
> > > disables local irqs during hw_ep_prime for RT kernel and suggest it to
> > > linux-rt maintainers?
> > >
> > > Thanks for a kind answer Peter.
> > >
> >
> 
> Thanks Peter.
> 
> > No, that will lead to deadlock since the normal request queue API function
> > ep_queue disables irq which also calls into hw_ep_prime.
> 
> ep_queue disables irq by calling spin_lock_irqsave but it actually
> does not disable irq for RT kernel.
> Therefore local_irq_save at hw_ep_prime would not cause deadlock if
> ep_queue calls it, i think.
> 
> Is it anything wrong?

Oh, I was wrong that I thought you would use spin_lock_irqsave to
disable interrupt. I am not familiar with spin_lock_irqsave behaviours
at RT kernel.

Yes, if you use local_irq_save at below sequence, that's ok.

spin_lock_irqsave(lock, flag_1);
...
func hw_ep_prime
{
	local_irq_save(flag_2);
	...
	local_irq_restore(flag_2);
}
...
spin_unlock_irqrestore(lock, flag_1);

Peter

> 
> >
> > For RT kernel, you may try to disable local irq at isr_setup_status_phase.
> >
> > --
> >
> > Thanks,
> > Peter Chen
> >
> 
> 
> -- 
> 황재호, Jay Hwang, linux team manager of RTst
> 010-7242-1593

-- 

Thanks,
Peter Chen

Re: Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Jeaho Hwang]

2021년 8월 9일 (월) 오전 10:27, Peter Chen <peter.chen@kernel.org>님이 작성:
>
> On 21-08-04 11:03:44, Jeaho Hwang wrote:
> > Hi. linux-usb and linux-rt experts.
> > >
> > > On 21-08-02 17:35:01, Jeaho Hwang wrote:
> > > > Hi.
> > > >
> > > > We found an infinite loop inside the function hw_ep_set_halt
> > > > (drivers/usb/chipidea/udc.c) if a cablle is repeatedly
> > > > connnected/disconnected while ping through RNDIS with chipidea USB device.
> > > >
> > > > Using ftrace tracing, we found that hw_ep_set_halt is called due to error
> > > > return of hw_ep_prime(drivers/usb/chipidea/udc.c:202) which is called from
> > > > isr_tr_complete_handler -> isr_setup_status_phase -> _ep_queue.
> > > >
> > > > The comment of function hw_ep_prime says (execute without interruption) but
> > > > timer interrupt is occurred while hw_ep_prime is executing. We believe that
> > > > the interrupt causes an error return of hw_ep_prime. We tried to protect
> > > > hw_ep_prime from irqs and then no case of the infinite loop is occurred.
> > > >
> > > > I want ask if it is appropriate way that turning off irq inside (threaded)
> > > > irq handlers. And should we explicitly turn off irqs before calling
> > > > hw_ep_prime?
> > > >
> > >
> > > Jeaho, do you use RT-Linux or standard Linux? The function hw_ep_prime is
> > > only called at udc_irq which is registered as top-half irq handlers.
> > > Why the timer interrupt is occurred when hw_ep_prime is executing?
> >
> > We use preempt_RT so timer interrupt could be occurred. Now I found
> > out that forced threaded irq handler disables local irq on standard
> > linux so It is a linux-rt issue. Then should I make patch which
> > disables local irqs during hw_ep_prime for RT kernel and suggest it to
> > linux-rt maintainers?
> >
> > Thanks for a kind answer Peter.
> >
>

Thanks Peter.

> No, that will lead to deadlock since the normal request queue API function
> ep_queue disables irq which also calls into hw_ep_prime.

ep_queue disables irq by calling spin_lock_irqsave but it actually
does not disable irq for RT kernel.
Therefore local_irq_save at hw_ep_prime would not cause deadlock if
ep_queue calls it, i think.

Is it anything wrong?

>
> For RT kernel, you may try to disable local irq at isr_setup_status_phase.
>
> --
>
> Thanks,
> Peter Chen
>


-- 
황재호, Jay Hwang, linux team manager of RTst
010-7242-1593

Re: Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Peter Chen]

On 21-08-04 12:11:31, Jeaho Hwang wrote:
> Thanks for the answer Peter. I still have two questions.
> 
> 1) There's a busy loop in hw_ep_prime to wait for endpoint priming. Is
> it safe without timeout?
> 
>  192 static int hw_ep_prime(struct ci_hdrc *ci, int num, int dir, int is_ctrl)
>  193 {
>  194     int n = hw_ep_bit(num, dir);
>  195
>  196     /* Synchronize before ep prime */
>  197     wmb();
>  198
>  199     if (is_ctrl && dir == RX && hw_read(ci, OP_ENDPTSETUPSTAT, BIT(num)))
>  200         return -EAGAIN;
>  201
>  202     hw_write(ci, OP_ENDPTPRIME, ~0, BIT(n));
>  203
>  204     while (hw_read(ci, OP_ENDPTPRIME, BIT(n)))
>  205         cpu_relax();
>  206     if (is_ctrl && dir == RX && hw_read(ci, OP_ENDPTSETUPSTAT, BIT(num)))
>  207         return -EAGAIN;
>  208
>  209     /* status shoult be tested according with manual but it doesn't work */
>  210     return 0;
>  211 }

You could add loop-timeout for it. Without timeout, it may lead issue,
I am not so sure.

> 
> 2) We experienced an infinite loop in hw_ep_set_halt, which is called
> at isr_tr_complete_handler due to an error encountered that we
> reported in the previous mail. It seems that hw_write to set halt
> fails. Is it related to the interruption while hw_ep_prime is running?
> If we make a timeout for the loop and error return, are there any
> considerations for the caller functions?
> 
>  223 static int hw_ep_set_halt(struct ci_hdrc *ci, int num, int dir, int value)
>  224 {
>  225     if (value != 0 && value != 1)
>  226         return -EINVAL;
>  227
>  228     do {
>  229         enum ci_hw_regs reg = OP_ENDPTCTRL + num;
>  230         u32 mask_xs = (dir == TX) ? ENDPTCTRL_TXS : ENDPTCTRL_RXS;
>  231         u32 mask_xr = (dir == TX) ? ENDPTCTRL_TXR : ENDPTCTRL_RXR;
>  232
>  233         /* data toggle - reserved for EP0 but it's in ESS */
>  234         hw_write(ci, reg, mask_xs|mask_xr,
>  235               value ? mask_xs : mask_xr);
>  236     } while (value != hw_ep_get_halt(ci, num, dir));
>  237
>  238     return 0;
>  239 }

If the error occurs for set halt, I think the transfer may stop, and it may
can't be recovered unless re-plug the cable. To not affect the whole system,
it is better to add loop-time mechanism.

-- 

Thanks,
Peter Chen

Re: Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Peter Chen]

On 21-08-04 11:03:44, Jeaho Hwang wrote:
> Hi. linux-usb and linux-rt experts.
> >
> > On 21-08-02 17:35:01, Jeaho Hwang wrote:
> > > Hi.
> > >
> > > We found an infinite loop inside the function hw_ep_set_halt
> > > (drivers/usb/chipidea/udc.c) if a cablle is repeatedly
> > > connnected/disconnected while ping through RNDIS with chipidea USB device.
> > >
> > > Using ftrace tracing, we found that hw_ep_set_halt is called due to error
> > > return of hw_ep_prime(drivers/usb/chipidea/udc.c:202) which is called from
> > > isr_tr_complete_handler -> isr_setup_status_phase -> _ep_queue.
> > >
> > > The comment of function hw_ep_prime says (execute without interruption) but
> > > timer interrupt is occurred while hw_ep_prime is executing. We believe that
> > > the interrupt causes an error return of hw_ep_prime. We tried to protect
> > > hw_ep_prime from irqs and then no case of the infinite loop is occurred.
> > >
> > > I want ask if it is appropriate way that turning off irq inside (threaded)
> > > irq handlers. And should we explicitly turn off irqs before calling
> > > hw_ep_prime?
> > >
> >
> > Jeaho, do you use RT-Linux or standard Linux? The function hw_ep_prime is
> > only called at udc_irq which is registered as top-half irq handlers.
> > Why the timer interrupt is occurred when hw_ep_prime is executing?
> 
> We use preempt_RT so timer interrupt could be occurred. Now I found
> out that forced threaded irq handler disables local irq on standard
> linux so It is a linux-rt issue. Then should I make patch which
> disables local irqs during hw_ep_prime for RT kernel and suggest it to
> linux-rt maintainers?
> 
> Thanks for a kind answer Peter.
> 

No, that will lead to deadlock since the normal request queue API function
ep_queue disables irq which also calls into hw_ep_prime.

For RT kernel, you may try to disable local irq at isr_setup_status_phase.

-- 

Thanks,
Peter Chen

Re: Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Jeaho Hwang]

Thanks for the answer Peter. I still have two questions.

1) There's a busy loop in hw_ep_prime to wait for endpoint priming. Is
it safe without timeout?

 192 static int hw_ep_prime(struct ci_hdrc *ci, int num, int dir, int is_ctrl)
 193 {
 194     int n = hw_ep_bit(num, dir);
 195
 196     /* Synchronize before ep prime */
 197     wmb();
 198
 199     if (is_ctrl && dir == RX && hw_read(ci, OP_ENDPTSETUPSTAT, BIT(num)))
 200         return -EAGAIN;
 201
 202     hw_write(ci, OP_ENDPTPRIME, ~0, BIT(n));
 203
 204     while (hw_read(ci, OP_ENDPTPRIME, BIT(n)))
 205         cpu_relax();
 206     if (is_ctrl && dir == RX && hw_read(ci, OP_ENDPTSETUPSTAT, BIT(num)))
 207         return -EAGAIN;
 208
 209     /* status shoult be tested according with manual but it doesn't work */
 210     return 0;
 211 }

2) We experienced an infinite loop in hw_ep_set_halt, which is called
at isr_tr_complete_handler due to an error encountered that we
reported in the previous mail. It seems that hw_write to set halt
fails. Is it related to the interruption while hw_ep_prime is running?
If we make a timeout for the loop and error return, are there any
considerations for the caller functions?

 223 static int hw_ep_set_halt(struct ci_hdrc *ci, int num, int dir, int value)
 224 {
 225     if (value != 0 && value != 1)
 226         return -EINVAL;
 227
 228     do {
 229         enum ci_hw_regs reg = OP_ENDPTCTRL + num;
 230         u32 mask_xs = (dir == TX) ? ENDPTCTRL_TXS : ENDPTCTRL_RXS;
 231         u32 mask_xr = (dir == TX) ? ENDPTCTRL_TXR : ENDPTCTRL_RXR;
 232
 233         /* data toggle - reserved for EP0 but it's in ESS */
 234         hw_write(ci, reg, mask_xs|mask_xr,
 235               value ? mask_xs : mask_xr);
 236     } while (value != hw_ep_get_halt(ci, num, dir));
 237
 238     return 0;
 239 }

Thanks.

>
> On 21-08-02 17:35:01, Jeaho Hwang wrote:
> > Hi.
> >
> > We found an infinite loop inside the function hw_ep_set_halt
> > (drivers/usb/chipidea/udc.c) if a cablle is repeatedly
> > connnected/disconnected while ping through RNDIS with chipidea USB device.
> >
> > Using ftrace tracing, we found that hw_ep_set_halt is called due to error
> > return of hw_ep_prime(drivers/usb/chipidea/udc.c:202) which is called from
> > isr_tr_complete_handler -> isr_setup_status_phase -> _ep_queue.
> >
> > The comment of function hw_ep_prime says (execute without interruption) but
> > timer interrupt is occurred while hw_ep_prime is executing. We believe that
> > the interrupt causes an error return of hw_ep_prime. We tried to protect
> > hw_ep_prime from irqs and then no case of the infinite loop is occurred.
> >
> > I want ask if it is appropriate way that turning off irq inside (threaded)
> > irq handlers. And should we explicitly turn off irqs before calling
> > hw_ep_prime?
> >
>
> Jeaho, do you use RT-Linux or standard Linux? The function hw_ep_prime is
> only called at udc_irq which is registered as top-half irq handlers.
> Why the timer interrupt is occurred when hw_ep_prime is executing?
>
> --
>
> Thanks,
> Peter Chen
>


-- 
황재호, Jay Hwang, linux team manager of RTst
010-7242-1593

Re: Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Jeaho Hwang]

Hi. linux-usb and linux-rt experts.
>
> On 21-08-02 17:35:01, Jeaho Hwang wrote:
> > Hi.
> >
> > We found an infinite loop inside the function hw_ep_set_halt
> > (drivers/usb/chipidea/udc.c) if a cablle is repeatedly
> > connnected/disconnected while ping through RNDIS with chipidea USB device.
> >
> > Using ftrace tracing, we found that hw_ep_set_halt is called due to error
> > return of hw_ep_prime(drivers/usb/chipidea/udc.c:202) which is called from
> > isr_tr_complete_handler -> isr_setup_status_phase -> _ep_queue.
> >
> > The comment of function hw_ep_prime says (execute without interruption) but
> > timer interrupt is occurred while hw_ep_prime is executing. We believe that
> > the interrupt causes an error return of hw_ep_prime. We tried to protect
> > hw_ep_prime from irqs and then no case of the infinite loop is occurred.
> >
> > I want ask if it is appropriate way that turning off irq inside (threaded)
> > irq handlers. And should we explicitly turn off irqs before calling
> > hw_ep_prime?
> >
>
> Jeaho, do you use RT-Linux or standard Linux? The function hw_ep_prime is
> only called at udc_irq which is registered as top-half irq handlers.
> Why the timer interrupt is occurred when hw_ep_prime is executing?

We use preempt_RT so timer interrupt could be occurred. Now I found
out that forced threaded irq handler disables local irq on standard
linux so It is a linux-rt issue. Then should I make patch which
disables local irqs during hw_ep_prime for RT kernel and suggest it to
linux-rt maintainers?

Thanks for a kind answer Peter.

>
> --
>
> Thanks,
> Peter Chen
>


-- 
황재호, Jay Hwang, linux team manager of RTst
010-7242-1593

Re: Chipidea USB device goes infinite loop due to interrupt while hw_ep_prime

[Peter Chen]

On 21-08-02 17:35:01, Jeaho Hwang wrote:
> Hi.
> 
> We found an infinite loop inside the function hw_ep_set_halt
> (drivers/usb/chipidea/udc.c) if a cablle is repeatedly
> connnected/disconnected while ping through RNDIS with chipidea USB device.
> 
> Using ftrace tracing, we found that hw_ep_set_halt is called due to error
> return of hw_ep_prime(drivers/usb/chipidea/udc.c:202) which is called from
> isr_tr_complete_handler -> isr_setup_status_phase -> _ep_queue.
> 
> The comment of function hw_ep_prime says (execute without interruption) but
> timer interrupt is occurred while hw_ep_prime is executing. We believe that
> the interrupt causes an error return of hw_ep_prime. We tried to protect
> hw_ep_prime from irqs and then no case of the infinite loop is occurred.
> 
> I want ask if it is appropriate way that turning off irq inside (threaded)
> irq handlers. And should we explicitly turn off irqs before calling
> hw_ep_prime?
> 

Jeaho, do you use RT-Linux or standard Linux? The function hw_ep_prime is
only called at udc_irq which is registered as top-half irq handlers.
Why the timer interrupt is occurred when hw_ep_prime is executing?

-- 

Thanks,
Peter Chen