Linux-Fsdevel Archive on lore.kernel.org
 help / color / Atom feed
From: Arul Jeniston <arul.jeniston@gmail.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org,
	linux-kernel@vger.kernel.org, arul_mc@dell.com
Subject: Re: [PATCH] FS: timerfd: Fix unexpected return value of timerfd_read function.
Date: Mon, 19 Aug 2019 19:55:40 +0530
Message-ID: <CACAVd4izozzXNF9qwNcXC+EUx5n1sfsNeb9JNXNJF56LdZkkYg@mail.gmail.com> (raw)
In-Reply-To: <alpine.DEB.2.21.1908190947290.1923@nanos.tec.linutronix.de>

hi Tglx,

...
> > > Is the timer expiry and the timerfd_read() on the same CPU or on different
> > > ones?
> >

We have 10000+ units running in production. We see this problem ~20
switches in a span of one year time.
The problem is not seen more than once in the same unit. The
occurrence is random and unpredictable.
We tried to recreate this problem in LAB by loading the units similar
to the production unit.
so far we are not able to recreate it. It is very difficult to predict
what triggered this in the production units.
So, to find root cause, we studied and instrumented kernel code.

>   1) TSCs are out of sync or affected otherwise

If the TSC clock is unstable and not synchronized, Linux kernel throws
dmesg logs and changes the current clock source to next best timer
(hpet). But we didn't see these logs in any of the 10000 units.

>   2) Timekeeping has a bug.

As per our analysis,

After the timer expiry, after tsc is read in hrtimer_forward_now()
-->ktime_get()-->timekeeping_get_ns(), if the current thread (t1) is
interrupted and/or some other thread running in different CPU (t2)
updates timekeeper cycle_last value with a latest tsc than t1,
clocksource_delta() and timekeeping_get_delta() would return 0.
Eventually   timekeeping_delta_to_ns() would return a smaller value
based on the other two parameters (mult, xtime_nsec). If
base(timekeeper.tkr_mono.base) is not updated all this time, then
ktime_get() could return a value lesser than expiry time.
Note: CONFIG_DEBUG_TIMEKEEPING is not configured in our system.

> I was asking for a full boot log for a reason. Is it impossible to stick
> that into a mail?

Give me a day time to sync-up internally and update.

Regards,
Arul

  reply index

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-16  8:32 [PATCH] FS: timerfd: Fix unexpected return value of timerfd_read function. 'hrtimer_forward_now()' returns zero due to bigger backward time drift. This causes timerfd_read to return 0. As per man page, read on timerfd is not expected to return 0. This patch fixes this problem. Signed-off-by: Arul Jeniston <arul.jeniston@gmail.com> arul.jeniston
2019-08-16  9:05 ` [PATCH] FS: timerfd: [Trimmed unreadable long subject line ] Thomas Gleixner
2019-08-16 10:22 ` [PATCH] FS: timerfd: Fix unexpected return value of timerfd_read function Arul Jeniston
2019-08-16 10:45   ` Thomas Gleixner
2019-08-16 16:55     ` Arul Jeniston
2019-08-16 17:00       ` Arul Jeniston
2019-08-16 21:17         ` Thomas Gleixner
     [not found]           ` <CACAVd4hT6QYtgtDsBcgy7c_s9WVBAH+1m0r5geBe7BUWJWYhbA@mail.gmail.com>
2019-08-17 19:23             ` Thomas Gleixner
2019-08-19  6:07               ` Arul Jeniston
2019-08-19  8:04                 ` Thomas Gleixner
2019-08-19 14:25                   ` Arul Jeniston [this message]
2019-08-19 14:52                     ` Thomas Gleixner
2019-08-19 15:26                       ` Arul Jeniston
2019-08-19 15:59                         ` Thomas Gleixner
     [not found]                           ` <CACAVd4iRN7=eq_B1+Yb-xcspU-Sg1dmMo_=VtLXXVPkjN1hY5Q@mail.gmail.com>
2019-08-19 18:29                             ` Thomas Gleixner
     [not found]                               ` <CACAVd4jAJ5QcOH=q=Q9kAz20X4_nAc7=vVU_gPWTS1UuiGK-fg@mail.gmail.com>
2019-08-20  8:49                                 ` Thomas Gleixner
2019-08-20  9:42                                   ` Arul Jeniston
2019-09-05  8:48                                     ` Arul Jeniston
2019-09-05 15:34                                       ` Thomas Gleixner
2019-09-06 16:36                                         ` Arul Jeniston
2019-09-07 14:38                                           ` Thomas Gleixner
2019-11-05  5:01                                             ` Arul Jeniston
2019-11-05 10:01                                               ` Thomas Gleixner
2019-11-06  3:38                                                 ` Arul Jeniston

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CACAVd4izozzXNF9qwNcXC+EUx5n1sfsNeb9JNXNJF56LdZkkYg@mail.gmail.com \
    --to=arul.jeniston@gmail.com \
    --cc=arul_mc@dell.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=tglx@linutronix.de \
    --cc=viro@zeniv.linux.org.uk \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Fsdevel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-fsdevel/0 linux-fsdevel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-fsdevel linux-fsdevel/ https://lore.kernel.org/linux-fsdevel \
		linux-fsdevel@vger.kernel.org
	public-inbox-index linux-fsdevel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-fsdevel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git