Kernel Newbies archive on lore.kernel.org
 help / color / Atom feed
From: Primoz Beltram <primoz.beltram@kate.si>
Cc: kernelnewbies <kernelnewbies@kernelnewbies.org>
Subject: Re: I2C bus driver TIMEDOUT because of PM autosuspend
Date: Tue, 3 Dec 2019 10:26:49 +0100
Message-ID: <2a0c43b9-7b5f-66f5-614c-8cae5b9ad965@kate.si> (raw)
In-Reply-To: <CAK7N6vqafbgeo+bocur1g=Yf-OPyPyfhHR06SVBWzXRXOJt=0w@mail.gmail.com>

On 3. 12. 19 02:30, anish singh wrote:
> On Fri, Nov 29, 2019 at 12:53 PM Primoz Beltram
> <primoz.beltram@gmail.com> wrote:
>> I am analysing a problem with I2C bus driver where the problem shows up
>> as I2C bus completely blocked. The LX driver in question is
>> /drivers/i2c/busses/i2c-xiic.c.
>> Problem is difficult to reproduce, it happens very rarely. So far I saw
>> that the main precondition is to have very heavy I2C traffic on bus.
>> In my case this is achieved/reproduced via netdev driving SFP LEDs via
>> /sys/class/leds/ (via gpio-pca953x). I generate traffic with iperf3.
>> Network traffic is on 10Gbps EMAC. LX kernel is 4.14.0.
>> What I saw from debugging this problem is that I2C bus get blocked when
>> wait_event_timeout() completes because of timeout. The timeout handling
>> in this driver is probably not robust enough (bus should not remain
>> blocked), but at this moment this are just my speculations (don't know
>> enough details).
> Check with salea logic analyzer what happens to the i2c bus.
>
>> Looking the driver code and data on oscilloscope, I saw that SCL in
>> single I2C data transfer sequence can be interrupted for very long
>> delays, e.g up to hundredths of usec (SCL is 100kHz). I started to
>> suspect that PM autosuspend delay could play some role here. There are
>> only two delays in driver code, first in wait_event_timeout and second
>> in set autosuspend delay. Case is a bit strange because in very busy I2C
>> traffic, PM autosuspend should not be triggered at all. Additionally, if
>> I lower PM timeout, e.g. from 1000 (default) to 100, I hit the problem
>> sooner (waits for problem hit are in order of n*10minutes).
>>
>> It looks to me that PM autosupend is playing some role here.
>>
>> Power management options in my .config:
>> # CONFIG_SUSPEND is not set
>> # CONFIG_PM is not set
>> CONFIG_ARCH_SUSPEND_POSSIBLE=y
>>
>> I intentionally did not put all detail descriptions of embedded system
>> and test setup here (long list), because the main reason of this post is:
>>
>> The workaround that works for me/customer (at the moment) is to disable
>> PM autosuspend in the driver code, either by incerementing PM delay from
>> 1000 to 10000 or by disabling autosuspend (comment out call to
>> pm_runtime_put_autosuspend() in xiic_xfer()).
>>
>> But, I would like to expose/discuss this issue (maintainer of the code,
>> or others).
>> The reason/source of the problem can be much more complex and in some
>> other place.
>>
>> So my question is who should I contact, is this the M: in the
>> MAINTAINERS list, the MODULE_AUTHOR, ...?
> You can certainly add the author in loop but I am afraid
> you won't get any help as this would be specific to your board. So,
> best is to check soc vendor who has written your i2c
> bus driver or it could be a issue with your i2c client in that
> case show them your salea logic analyzer logs to see
> if they can figure out anything wrong.

Thanks for reply and suggestions.

My first suspicion was signal integrity on PCB, but if I add some debug 
prints in i2c-xiic driver (e.g. build with DEBUG define), the problem is 
no longer reproducible (not a single timeout completion in 
wait_event_timeout()).

Signal integrity problem does not look credible to me.

For my system I fixed the problem in i2c-xiic driver (in handIing 
timeout, not leave bus blocked).

Found also a contact and fill report for SoC vendor.

WBR Primoz
>> How to proceed.
>>
>> WBR Primoz
>>
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies@kernelnewbies.org
>> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies



_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

  reply index

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-21 16:59 Primoz Beltram
2019-12-03  1:30 ` anish singh
2019-12-03  9:26   ` Primoz Beltram [this message]
2019-11-27 14:49 Primoz Beltram

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2a0c43b9-7b5f-66f5-614c-8cae5b9ad965@kate.si \
    --to=primoz.beltram@kate.si \
    --cc=kernelnewbies@kernelnewbies.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Kernel Newbies archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kernelnewbies/0 kernelnewbies/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kernelnewbies kernelnewbies/ https://lore.kernel.org/kernelnewbies \
		kernelnewbies@kernelnewbies.org
	public-inbox-index kernelnewbies

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernelnewbies.kernelnewbies


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git