[PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[Miroslav Lichvar]

RFC->v1:
- added new patches
- separated PHC timestamp from ptp_system_timestamp
- fixed memory leak in PTP_SYS_OFFSET_EXTENDED
- changed PTP_SYS_OFFSET_EXTENDED to work with array of arrays
- fixed PTP_SYS_OFFSET_EXTENDED to break correctly from loop
- fixed timecounter updates in drivers
- split gettimex in igb driver
- fixed ptp_read_* functions to be available without
  CONFIG_PTP_1588_CLOCK

This series enables a more accurate synchronization between PTP hardware
clocks and the system clock.

The first two patches are minor cleanup/bug fixes.

The third patch adds an extended version of the PTP_SYS_OFFSET ioctl,
which returns three timestamps for each measurement. The idea is to
shorten the interval between the system timestamps to contain just the
reading of the lowest register of the PHC in order to reduce the error
in the measured offset and get a smaller upper bound on the maximum
error.

The fourth patch deprecates the original gettime function.

The remaining patches update the gettime function in order to support
the new ioctl in the e1000e, igb, ixgbe, and tg3 drivers.

Tests with few different NICs in different machines show that:
- with an I219 (e1000e) the measured delay was reduced from 2500 to 1300
  ns and the error in the measured offset, when compared to the cross
  timestamping supported by the driver, was reduced by a factor of 5
- with an I210 (igb) the delay was reduced from 5100 to 1700 ns
- with an I350 (igb) the delay was reduced from 2300 to 750 ns
- with an X550 (ixgbe) the delay was reduced from 1950 to 650 ns
- with a BCM5720 (tg3) the delay was reduced from 2400 to 1200 ns


Miroslav Lichvar (8):
  ptp: reorder declarations in ptp_ioctl()
  ptp: check gettime64 return code in PTP_SYS_OFFSET ioctl
  ptp: add PTP_SYS_OFFSET_EXTENDED ioctl
  ptp: deprecate gettime64() in favor of gettimex64()
  e1000e: extend PTP gettime function to read system clock
  igb: extend PTP gettime function to read system clock
  ixgbe: extend PTP gettime function to read system clock
  tg3: extend PTP gettime function to read system clock

 drivers/net/ethernet/broadcom/tg3.c          | 19 ++++--
 drivers/net/ethernet/intel/e1000e/e1000.h    |  3 +
 drivers/net/ethernet/intel/e1000e/netdev.c   | 42 ++++++++++---
 drivers/net/ethernet/intel/e1000e/ptp.c      | 16 +++--
 drivers/net/ethernet/intel/igb/igb_ptp.c     | 65 +++++++++++++++++---
 drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 54 +++++++++++++---
 drivers/ptp/ptp_chardev.c                    | 55 ++++++++++++++---
 drivers/ptp/ptp_clock.c                      |  5 +-
 include/linux/ptp_clock_kernel.h             | 33 ++++++++++
 include/uapi/linux/ptp_clock.h               | 12 ++++
 10 files changed, 253 insertions(+), 51 deletions(-)

-- 
2.17.2

[PATCH net-next 1/8] ptp: reorder declarations in ptp_ioctl()

[Miroslav Lichvar]

Reorder declarations of variables as reversed Christmas tree.

Cc: Richard Cochran <richardcochran@gmail.com>
Suggested-by: Richard Cochran <richardcochran@gmail.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/ptp/ptp_chardev.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c
index 2012551d93e0..b54b8158ff8a 100644
--- a/drivers/ptp/ptp_chardev.c
+++ b/drivers/ptp/ptp_chardev.c
@@ -121,18 +121,18 @@ int ptp_open(struct posix_clock *pc, fmode_t fmode)
 
 long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 {
-	struct ptp_clock_caps caps;
-	struct ptp_clock_request req;
-	struct ptp_sys_offset *sysoff = NULL;
-	struct ptp_sys_offset_precise precise_offset;
-	struct ptp_pin_desc pd;
 	struct ptp_clock *ptp = container_of(pc, struct ptp_clock, clock);
+	struct ptp_sys_offset_precise precise_offset;
+	struct system_device_crosststamp xtstamp;
 	struct ptp_clock_info *ops = ptp->info;
+	struct ptp_sys_offset *sysoff = NULL;
+	struct ptp_clock_request req;
+	struct ptp_clock_caps caps;
 	struct ptp_clock_time *pct;
+	unsigned int i, pin_index;
+	struct ptp_pin_desc pd;
 	struct timespec64 ts;
-	struct system_device_crosststamp xtstamp;
 	int enable, err = 0;
-	unsigned int i, pin_index;
 
 	switch (cmd) {
 
-- 
2.17.2

[PATCH net-next 2/8] ptp: check gettime64 return code in PTP_SYS_OFFSET ioctl

[Miroslav Lichvar]

If a gettime64 call fails, return the error and avoid copying data back
to user.

Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Jacob Keller <jacob.e.keller@intel.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/ptp/ptp_chardev.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c
index b54b8158ff8a..3c681bed5703 100644
--- a/drivers/ptp/ptp_chardev.c
+++ b/drivers/ptp/ptp_chardev.c
@@ -228,7 +228,9 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 			pct->sec = ts.tv_sec;
 			pct->nsec = ts.tv_nsec;
 			pct++;
-			ptp->info->gettime64(ptp->info, &ts);
+			err = ptp->info->gettime64(ptp->info, &ts);
+			if (err)
+				goto out;
 			pct->sec = ts.tv_sec;
 			pct->nsec = ts.tv_nsec;
 			pct++;
@@ -281,6 +283,7 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 		break;
 	}
 
+out:
 	kfree(sysoff);
 	return err;
 }
-- 
2.17.2

[PATCH net-next 3/8] ptp: add PTP_SYS_OFFSET_EXTENDED ioctl

[Miroslav Lichvar]

The PTP_SYS_OFFSET ioctl, which can be used to measure the offset
between a PHC and the system clock, includes the total time that the
driver needs to read the PHC timestamp.

This typically involves reading of multiple PCI registers (sometimes in
multiple iterations) and the register that contains the lowest bits of
the timestamp is not read in the middle between the two readings of the
system clock. This asymmetry causes the measured offset to have a
significant error.

Introduce a new ioctl, driver function, and helper functions, which
allow the reading of the lowest register to be isolated from the other
readings in order to reduce the asymmetry. The ioctl returns three
timestamps for each measurement:
- system time right before reading the lowest bits of the PHC timestamp
- PHC time
- system time immediately after reading the lowest bits of the PHC
  timestamp

Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Jacob Keller <jacob.e.keller@intel.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/ptp/ptp_chardev.c        | 33 ++++++++++++++++++++++++++++++++
 include/linux/ptp_clock_kernel.h | 31 ++++++++++++++++++++++++++++++
 include/uapi/linux/ptp_clock.h   | 12 ++++++++++++
 3 files changed, 76 insertions(+)

diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c
index 3c681bed5703..aad0d36cf5c0 100644
--- a/drivers/ptp/ptp_chardev.c
+++ b/drivers/ptp/ptp_chardev.c
@@ -122,10 +122,12 @@ int ptp_open(struct posix_clock *pc, fmode_t fmode)
 long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 {
 	struct ptp_clock *ptp = container_of(pc, struct ptp_clock, clock);
+	struct ptp_sys_offset_extended *extoff = NULL;
 	struct ptp_sys_offset_precise precise_offset;
 	struct system_device_crosststamp xtstamp;
 	struct ptp_clock_info *ops = ptp->info;
 	struct ptp_sys_offset *sysoff = NULL;
+	struct ptp_system_timestamp sts;
 	struct ptp_clock_request req;
 	struct ptp_clock_caps caps;
 	struct ptp_clock_time *pct;
@@ -211,6 +213,36 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 			err = -EFAULT;
 		break;
 
+	case PTP_SYS_OFFSET_EXTENDED:
+		if (!ptp->info->gettimex64) {
+			err = -EOPNOTSUPP;
+			break;
+		}
+		extoff = memdup_user((void __user *)arg, sizeof(*extoff));
+		if (IS_ERR(extoff)) {
+			err = PTR_ERR(extoff);
+			extoff = NULL;
+			break;
+		}
+		if (extoff->n_samples > PTP_MAX_SAMPLES) {
+			err = -EINVAL;
+			break;
+		}
+		for (i = 0; i < extoff->n_samples; i++) {
+			err = ptp->info->gettimex64(ptp->info, &ts, &sts);
+			if (err)
+				goto out;
+			extoff->ts[i][0].sec = sts.pre_ts.tv_sec;
+			extoff->ts[i][0].nsec = sts.pre_ts.tv_nsec;
+			extoff->ts[i][1].sec = ts.tv_sec;
+			extoff->ts[i][1].nsec = ts.tv_nsec;
+			extoff->ts[i][2].sec = sts.post_ts.tv_sec;
+			extoff->ts[i][2].nsec = sts.post_ts.tv_nsec;
+		}
+		if (copy_to_user((void __user *)arg, extoff, sizeof(*extoff)))
+			err = -EFAULT;
+		break;
+
 	case PTP_SYS_OFFSET:
 		sysoff = memdup_user((void __user *)arg, sizeof(*sysoff));
 		if (IS_ERR(sysoff)) {
@@ -284,6 +316,7 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 	}
 
 out:
+	kfree(extoff);
 	kfree(sysoff);
 	return err;
 }
diff --git a/include/linux/ptp_clock_kernel.h b/include/linux/ptp_clock_kernel.h
index 51349d124ee5..a1ec0448e341 100644
--- a/include/linux/ptp_clock_kernel.h
+++ b/include/linux/ptp_clock_kernel.h
@@ -39,6 +39,15 @@ struct ptp_clock_request {
 };
 
 struct system_device_crosststamp;
+
+/**
+ * struct ptp_system_timestamp - system time corresponding to a PHC timestamp
+ */
+struct ptp_system_timestamp {
+	struct timespec64 pre_ts;
+	struct timespec64 post_ts;
+};
+
 /**
  * struct ptp_clock_info - decribes a PTP hardware clock
  *
@@ -75,6 +84,14 @@ struct system_device_crosststamp;
  * @gettime64:  Reads the current time from the hardware clock.
  *              parameter ts: Holds the result.
  *
+ * @gettimex64:  Reads the current time from the hardware clock and optionally
+ *               also the system clock.
+ *               parameter ts: Holds the PHC timestamp.
+ *               parameter sts: If not NULL, it holds a pair of timestamps from
+ *               the system clock. The first reading is made right before
+ *               reading the lowest bits of the PHC timestamp and the second
+ *               reading immediately follows that.
+ *
  * @getcrosststamp:  Reads the current time from the hardware clock and
  *                   system clock simultaneously.
  *                   parameter cts: Contains timestamp (device,system) pair,
@@ -124,6 +141,8 @@ struct ptp_clock_info {
 	int (*adjfreq)(struct ptp_clock_info *ptp, s32 delta);
 	int (*adjtime)(struct ptp_clock_info *ptp, s64 delta);
 	int (*gettime64)(struct ptp_clock_info *ptp, struct timespec64 *ts);
+	int (*gettimex64)(struct ptp_clock_info *ptp, struct timespec64 *ts,
+			  struct ptp_system_timestamp *sts);
 	int (*getcrosststamp)(struct ptp_clock_info *ptp,
 			      struct system_device_crosststamp *cts);
 	int (*settime64)(struct ptp_clock_info *p, const struct timespec64 *ts);
@@ -247,4 +266,16 @@ static inline int ptp_schedule_worker(struct ptp_clock *ptp,
 
 #endif
 
+static inline void ptp_read_system_prets(struct ptp_system_timestamp *sts)
+{
+	if (sts)
+		ktime_get_real_ts64(&sts->pre_ts);
+}
+
+static inline void ptp_read_system_postts(struct ptp_system_timestamp *sts)
+{
+	if (sts)
+		ktime_get_real_ts64(&sts->post_ts);
+}
+
 #endif
diff --git a/include/uapi/linux/ptp_clock.h b/include/uapi/linux/ptp_clock.h
index 3039bf6a742e..d73d83950265 100644
--- a/include/uapi/linux/ptp_clock.h
+++ b/include/uapi/linux/ptp_clock.h
@@ -84,6 +84,16 @@ struct ptp_sys_offset {
 	struct ptp_clock_time ts[2 * PTP_MAX_SAMPLES + 1];
 };
 
+struct ptp_sys_offset_extended {
+	unsigned int n_samples; /* Desired number of measurements. */
+	unsigned int rsv[3];    /* Reserved for future use. */
+	/*
+	 * Array of [system, phc, system] time stamps. The kernel will provide
+	 * 3*n_samples time stamps.
+	 */
+	struct ptp_clock_time ts[PTP_MAX_SAMPLES][3];
+};
+
 struct ptp_sys_offset_precise {
 	struct ptp_clock_time device;
 	struct ptp_clock_time sys_realtime;
@@ -136,6 +146,8 @@ struct ptp_pin_desc {
 #define PTP_PIN_SETFUNC    _IOW(PTP_CLK_MAGIC, 7, struct ptp_pin_desc)
 #define PTP_SYS_OFFSET_PRECISE \
 	_IOWR(PTP_CLK_MAGIC, 8, struct ptp_sys_offset_precise)
+#define PTP_SYS_OFFSET_EXTENDED \
+	_IOW(PTP_CLK_MAGIC, 9, struct ptp_sys_offset_extended)
 
 struct ptp_extts_event {
 	struct ptp_clock_time t; /* Time event occured. */
-- 
2.17.2

[PATCH net-next 4/8] ptp: deprecate gettime64() in favor of gettimex64()

[Miroslav Lichvar]

When a driver provides gettimex64(), use it in the PTP_SYS_OFFSET ioctl
and POSIX clock's gettime() instead of gettime64(). Drivers should
provide only one of the functions.

Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Jacob Keller <jacob.e.keller@intel.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/ptp/ptp_chardev.c        | 5 ++++-
 drivers/ptp/ptp_clock.c          | 5 ++++-
 include/linux/ptp_clock_kernel.h | 2 ++
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c
index aad0d36cf5c0..797fab33bb98 100644
--- a/drivers/ptp/ptp_chardev.c
+++ b/drivers/ptp/ptp_chardev.c
@@ -260,7 +260,10 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 			pct->sec = ts.tv_sec;
 			pct->nsec = ts.tv_nsec;
 			pct++;
-			err = ptp->info->gettime64(ptp->info, &ts);
+			if (ops->gettimex64)
+				err = ops->gettimex64(ops, &ts, NULL);
+			else
+				err = ops->gettime64(ops, &ts);
 			if (err)
 				goto out;
 			pct->sec = ts.tv_sec;
diff --git a/drivers/ptp/ptp_clock.c b/drivers/ptp/ptp_clock.c
index 5419a89d300e..40fda23e4b05 100644
--- a/drivers/ptp/ptp_clock.c
+++ b/drivers/ptp/ptp_clock.c
@@ -117,7 +117,10 @@ static int ptp_clock_gettime(struct posix_clock *pc, struct timespec64 *tp)
 	struct ptp_clock *ptp = container_of(pc, struct ptp_clock, clock);
 	int err;
 
-	err = ptp->info->gettime64(ptp->info, tp);
+	if (ptp->info->gettimex64)
+		err = ptp->info->gettimex64(ptp->info, tp, NULL);
+	else
+		err = ptp->info->gettime64(ptp->info, tp);
 	return err;
 }
 
diff --git a/include/linux/ptp_clock_kernel.h b/include/linux/ptp_clock_kernel.h
index a1ec0448e341..7121bbe76979 100644
--- a/include/linux/ptp_clock_kernel.h
+++ b/include/linux/ptp_clock_kernel.h
@@ -82,6 +82,8 @@ struct ptp_system_timestamp {
  *            parameter delta: Desired change in nanoseconds.
  *
  * @gettime64:  Reads the current time from the hardware clock.
+ *              This method is deprecated.  New drivers should implement
+ *              the @gettimex64 method instead.
  *              parameter ts: Holds the result.
  *
  * @gettimex64:  Reads the current time from the hardware clock and optionally
-- 
2.17.2

[PATCH net-next 5/8] e1000e: extend PTP gettime function to read system clock

[Miroslav Lichvar]

This adds support for the PTP_SYS_OFFSET_EXTENDED ioctl.

Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Jacob Keller <jacob.e.keller@intel.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/net/ethernet/intel/e1000e/e1000.h  |  3 ++
 drivers/net/ethernet/intel/e1000e/netdev.c | 42 ++++++++++++++++------
 drivers/net/ethernet/intel/e1000e/ptp.c    | 16 +++++----
 3 files changed, 45 insertions(+), 16 deletions(-)

diff --git a/drivers/net/ethernet/intel/e1000e/e1000.h b/drivers/net/ethernet/intel/e1000e/e1000.h
index c760dc72c520..be13227f1697 100644
--- a/drivers/net/ethernet/intel/e1000e/e1000.h
+++ b/drivers/net/ethernet/intel/e1000e/e1000.h
@@ -505,6 +505,9 @@ extern const struct e1000_info e1000_es2_info;
 void e1000e_ptp_init(struct e1000_adapter *adapter);
 void e1000e_ptp_remove(struct e1000_adapter *adapter);
 
+u64 e1000e_read_systim(struct e1000_adapter *adapter,
+		       struct ptp_system_timestamp *sts);
+
 static inline s32 e1000_phy_hw_reset(struct e1000_hw *hw)
 {
 	return hw->phy.ops.reset(hw);
diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c
index 16a73bd9f4cb..59bd587d809d 100644
--- a/drivers/net/ethernet/intel/e1000e/netdev.c
+++ b/drivers/net/ethernet/intel/e1000e/netdev.c
@@ -4319,13 +4319,16 @@ void e1000e_reinit_locked(struct e1000_adapter *adapter)
 /**
  * e1000e_sanitize_systim - sanitize raw cycle counter reads
  * @hw: pointer to the HW structure
- * @systim: time value read, sanitized and returned
+ * @systim: PHC time value read, sanitized and returned
+ * @sts: structure to hold system time before and after reading SYSTIML,
+ * may be NULL
  *
  * Errata for 82574/82583 possible bad bits read from SYSTIMH/L:
  * check to see that the time is incrementing at a reasonable
  * rate and is a multiple of incvalue.
  **/
-static u64 e1000e_sanitize_systim(struct e1000_hw *hw, u64 systim)
+static u64 e1000e_sanitize_systim(struct e1000_hw *hw, u64 systim,
+				  struct ptp_system_timestamp *sts)
 {
 	u64 time_delta, rem, temp;
 	u64 systim_next;
@@ -4335,7 +4338,9 @@ static u64 e1000e_sanitize_systim(struct e1000_hw *hw, u64 systim)
 	incvalue = er32(TIMINCA) & E1000_TIMINCA_INCVALUE_MASK;
 	for (i = 0; i < E1000_MAX_82574_SYSTIM_REREADS; i++) {
 		/* latch SYSTIMH on read of SYSTIML */
+		ptp_read_system_prets(sts);
 		systim_next = (u64)er32(SYSTIML);
+		ptp_read_system_postts(sts);
 		systim_next |= (u64)er32(SYSTIMH) << 32;
 
 		time_delta = systim_next - systim;
@@ -4353,15 +4358,16 @@ static u64 e1000e_sanitize_systim(struct e1000_hw *hw, u64 systim)
 }
 
 /**
- * e1000e_cyclecounter_read - read raw cycle counter (used by time counter)
- * @cc: cyclecounter structure
+ * e1000e_read_systim - read SYSTIM register
+ * @adapter: board private structure
+ * @sts: structure which will contain system time before and after reading
+ * SYSTIML, may be NULL
  **/
-static u64 e1000e_cyclecounter_read(const struct cyclecounter *cc)
+u64 e1000e_read_systim(struct e1000_adapter *adapter,
+		       struct ptp_system_timestamp *sts)
 {
-	struct e1000_adapter *adapter = container_of(cc, struct e1000_adapter,
-						     cc);
 	struct e1000_hw *hw = &adapter->hw;
-	u32 systimel, systimeh;
+	u32 systimel, systimel_2, systimeh;
 	u64 systim;
 	/* SYSTIMH latching upon SYSTIML read does not work well.
 	 * This means that if SYSTIML overflows after we read it but before
@@ -4369,11 +4375,15 @@ static u64 e1000e_cyclecounter_read(const struct cyclecounter *cc)
 	 * will experience a huge non linear increment in the systime value
 	 * to fix that we test for overflow and if true, we re-read systime.
 	 */
+	ptp_read_system_prets(sts);
 	systimel = er32(SYSTIML);
+	ptp_read_system_postts(sts);
 	systimeh = er32(SYSTIMH);
 	/* Is systimel is so large that overflow is possible? */
 	if (systimel >= (u32)0xffffffff - E1000_TIMINCA_INCVALUE_MASK) {
-		u32 systimel_2 = er32(SYSTIML);
+		ptp_read_system_prets(sts);
+		systimel_2 = er32(SYSTIML);
+		ptp_read_system_postts(sts);
 		if (systimel > systimel_2) {
 			/* There was an overflow, read again SYSTIMH, and use
 			 * systimel_2
@@ -4386,11 +4396,23 @@ static u64 e1000e_cyclecounter_read(const struct cyclecounter *cc)
 	systim |= (u64)systimeh << 32;
 
 	if (adapter->flags2 & FLAG2_CHECK_SYSTIM_OVERFLOW)
-		systim = e1000e_sanitize_systim(hw, systim);
+		systim = e1000e_sanitize_systim(hw, systim, sts);
 
 	return systim;
 }
 
+/**
+ * e1000e_cyclecounter_read - read raw cycle counter (used by time counter)
+ * @cc: cyclecounter structure
+ **/
+static u64 e1000e_cyclecounter_read(const struct cyclecounter *cc)
+{
+	struct e1000_adapter *adapter = container_of(cc, struct e1000_adapter,
+						     cc);
+
+	return e1000e_read_systim(adapter, NULL);
+}
+
 /**
  * e1000_sw_init - Initialize general software structures (struct e1000_adapter)
  * @adapter: board private structure to initialize
diff --git a/drivers/net/ethernet/intel/e1000e/ptp.c b/drivers/net/ethernet/intel/e1000e/ptp.c
index e1f821edbc21..1a4c65d9feb4 100644
--- a/drivers/net/ethernet/intel/e1000e/ptp.c
+++ b/drivers/net/ethernet/intel/e1000e/ptp.c
@@ -161,14 +161,18 @@ static int e1000e_phc_getcrosststamp(struct ptp_clock_info *ptp,
 #endif/*CONFIG_E1000E_HWTS*/
 
 /**
- * e1000e_phc_gettime - Reads the current time from the hardware clock
+ * e1000e_phc_gettimex - Reads the current time from the hardware clock and
+ *                       system clock
  * @ptp: ptp clock structure
- * @ts: timespec structure to hold the current time value
+ * @ts: timespec structure to hold the current PHC time
+ * @sts: structure to hold the current system time
  *
  * Read the timecounter and return the correct value in ns after converting
  * it into a struct timespec.
  **/
-static int e1000e_phc_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts)
+static int e1000e_phc_gettimex(struct ptp_clock_info *ptp,
+			       struct timespec64 *ts,
+			       struct ptp_system_timestamp *sts)
 {
 	struct e1000_adapter *adapter = container_of(ptp, struct e1000_adapter,
 						     ptp_clock_info);
@@ -177,8 +181,8 @@ static int e1000e_phc_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts)
 
 	spin_lock_irqsave(&adapter->systim_lock, flags);
 
-	/* Use timecounter_cyc2time() to allow non-monotonic SYSTIM readings */
-	cycles = adapter->cc.read(&adapter->cc);
+	/* NOTE: Non-monotonic SYSTIM readings may be returned */
+	cycles = e1000e_read_systim(adapter, sts);
 	ns = timecounter_cyc2time(&adapter->tc, cycles);
 
 	spin_unlock_irqrestore(&adapter->systim_lock, flags);
@@ -258,7 +262,7 @@ static const struct ptp_clock_info e1000e_ptp_clock_info = {
 	.pps		= 0,
 	.adjfreq	= e1000e_phc_adjfreq,
 	.adjtime	= e1000e_phc_adjtime,
-	.gettime64	= e1000e_phc_gettime,
+	.gettimex64	= e1000e_phc_gettimex,
 	.settime64	= e1000e_phc_settime,
 	.enable		= e1000e_phc_enable,
 };
-- 
2.17.2

[PATCH net-next 6/8] igb: extend PTP gettime function to read system clock

[Miroslav Lichvar]

This adds support for the PTP_SYS_OFFSET_EXTENDED ioctl.

Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Jacob Keller <jacob.e.keller@intel.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/net/ethernet/intel/igb/igb_ptp.c | 65 ++++++++++++++++++++----
 1 file changed, 55 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/intel/igb/igb_ptp.c b/drivers/net/ethernet/intel/igb/igb_ptp.c
index 29ced6b74d36..8c1833a157d3 100644
--- a/drivers/net/ethernet/intel/igb/igb_ptp.c
+++ b/drivers/net/ethernet/intel/igb/igb_ptp.c
@@ -275,17 +275,53 @@ static int igb_ptp_adjtime_i210(struct ptp_clock_info *ptp, s64 delta)
 	return 0;
 }
 
-static int igb_ptp_gettime_82576(struct ptp_clock_info *ptp,
-				 struct timespec64 *ts)
+static int igb_ptp_gettimex_82576(struct ptp_clock_info *ptp,
+				  struct timespec64 *ts,
+				  struct ptp_system_timestamp *sts)
 {
 	struct igb_adapter *igb = container_of(ptp, struct igb_adapter,
 					       ptp_caps);
+	struct e1000_hw *hw = &igb->hw;
 	unsigned long flags;
+	u32 lo, hi;
 	u64 ns;
 
 	spin_lock_irqsave(&igb->tmreg_lock, flags);
 
-	ns = timecounter_read(&igb->tc);
+	ptp_read_system_prets(sts);
+	lo = rd32(E1000_SYSTIML);
+	ptp_read_system_postts(sts);
+	hi = rd32(E1000_SYSTIMH);
+
+	ns = timecounter_cyc2time(&igb->tc, ((u64)hi << 32) | lo);
+
+	spin_unlock_irqrestore(&igb->tmreg_lock, flags);
+
+	*ts = ns_to_timespec64(ns);
+
+	return 0;
+}
+
+static int igb_ptp_gettimex_82580(struct ptp_clock_info *ptp,
+				  struct timespec64 *ts,
+				  struct ptp_system_timestamp *sts)
+{
+	struct igb_adapter *igb = container_of(ptp, struct igb_adapter,
+					       ptp_caps);
+	struct e1000_hw *hw = &igb->hw;
+	unsigned long flags;
+	u32 lo, hi;
+	u64 ns;
+
+	spin_lock_irqsave(&igb->tmreg_lock, flags);
+
+	ptp_read_system_prets(sts);
+	rd32(E1000_SYSTIMR);
+	ptp_read_system_postts(sts);
+	lo = rd32(E1000_SYSTIML);
+	hi = rd32(E1000_SYSTIMH);
+
+	ns = timecounter_cyc2time(&igb->tc, ((u64)hi << 32) | lo);
 
 	spin_unlock_irqrestore(&igb->tmreg_lock, flags);
 
@@ -294,16 +330,22 @@ static int igb_ptp_gettime_82576(struct ptp_clock_info *ptp,
 	return 0;
 }
 
-static int igb_ptp_gettime_i210(struct ptp_clock_info *ptp,
-				struct timespec64 *ts)
+static int igb_ptp_gettimex_i210(struct ptp_clock_info *ptp,
+				 struct timespec64 *ts,
+				 struct ptp_system_timestamp *sts)
 {
 	struct igb_adapter *igb = container_of(ptp, struct igb_adapter,
 					       ptp_caps);
+	struct e1000_hw *hw = &igb->hw;
 	unsigned long flags;
 
 	spin_lock_irqsave(&igb->tmreg_lock, flags);
 
-	igb_ptp_read_i210(igb, ts);
+	ptp_read_system_prets(sts);
+	rd32(E1000_SYSTIMR);
+	ptp_read_system_postts(sts);
+	ts->tv_nsec = rd32(E1000_SYSTIML);
+	ts->tv_sec = rd32(E1000_SYSTIMH);
 
 	spin_unlock_irqrestore(&igb->tmreg_lock, flags);
 
@@ -656,9 +698,12 @@ static void igb_ptp_overflow_check(struct work_struct *work)
 	struct igb_adapter *igb =
 		container_of(work, struct igb_adapter, ptp_overflow_work.work);
 	struct timespec64 ts;
+	u64 ns;
 
-	igb->ptp_caps.gettime64(&igb->ptp_caps, &ts);
+	/* Update the timecounter */
+	ns = timecounter_read(&igb->tc);
 
+	ts = ns_to_timespec64(ns);
 	pr_debug("igb overflow check at %lld.%09lu\n",
 		 (long long) ts.tv_sec, ts.tv_nsec);
 
@@ -1124,7 +1169,7 @@ void igb_ptp_init(struct igb_adapter *adapter)
 		adapter->ptp_caps.pps = 0;
 		adapter->ptp_caps.adjfreq = igb_ptp_adjfreq_82576;
 		adapter->ptp_caps.adjtime = igb_ptp_adjtime_82576;
-		adapter->ptp_caps.gettime64 = igb_ptp_gettime_82576;
+		adapter->ptp_caps.gettimex64 = igb_ptp_gettimex_82576;
 		adapter->ptp_caps.settime64 = igb_ptp_settime_82576;
 		adapter->ptp_caps.enable = igb_ptp_feature_enable;
 		adapter->cc.read = igb_ptp_read_82576;
@@ -1143,7 +1188,7 @@ void igb_ptp_init(struct igb_adapter *adapter)
 		adapter->ptp_caps.pps = 0;
 		adapter->ptp_caps.adjfine = igb_ptp_adjfine_82580;
 		adapter->ptp_caps.adjtime = igb_ptp_adjtime_82576;
-		adapter->ptp_caps.gettime64 = igb_ptp_gettime_82576;
+		adapter->ptp_caps.gettimex64 = igb_ptp_gettimex_82580;
 		adapter->ptp_caps.settime64 = igb_ptp_settime_82576;
 		adapter->ptp_caps.enable = igb_ptp_feature_enable;
 		adapter->cc.read = igb_ptp_read_82580;
@@ -1171,7 +1216,7 @@ void igb_ptp_init(struct igb_adapter *adapter)
 		adapter->ptp_caps.pin_config = adapter->sdp_config;
 		adapter->ptp_caps.adjfine = igb_ptp_adjfine_82580;
 		adapter->ptp_caps.adjtime = igb_ptp_adjtime_i210;
-		adapter->ptp_caps.gettime64 = igb_ptp_gettime_i210;
+		adapter->ptp_caps.gettimex64 = igb_ptp_gettimex_i210;
 		adapter->ptp_caps.settime64 = igb_ptp_settime_i210;
 		adapter->ptp_caps.enable = igb_ptp_feature_enable_i210;
 		adapter->ptp_caps.verify = igb_ptp_verify_pin;
-- 
2.17.2

[PATCH net-next 7/8] ixgbe: extend PTP gettime function to read system clock

[Miroslav Lichvar]

This adds support for the PTP_SYS_OFFSET_EXTENDED ioctl.

Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Jacob Keller <jacob.e.keller@intel.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 54 ++++++++++++++++----
 1 file changed, 44 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c
index b3e0d8bb5cbd..d81a50dc9535 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c
@@ -443,22 +443,52 @@ static int ixgbe_ptp_adjtime(struct ptp_clock_info *ptp, s64 delta)
 }
 
 /**
- * ixgbe_ptp_gettime
+ * ixgbe_ptp_gettimex
  * @ptp: the ptp clock structure
- * @ts: timespec structure to hold the current time value
+ * @ts: timespec to hold the PHC timestamp
+ * @sts: structure to hold the system time before and after reading the PHC
  *
  * read the timecounter and return the correct value on ns,
  * after converting it into a struct timespec.
  */
-static int ixgbe_ptp_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts)
+static int ixgbe_ptp_gettimex(struct ptp_clock_info *ptp,
+			      struct timespec64 *ts,
+			      struct ptp_system_timestamp *sts)
 {
 	struct ixgbe_adapter *adapter =
 		container_of(ptp, struct ixgbe_adapter, ptp_caps);
+	struct ixgbe_hw *hw = &adapter->hw;
 	unsigned long flags;
-	u64 ns;
+	u64 ns, stamp;
 
 	spin_lock_irqsave(&adapter->tmreg_lock, flags);
-	ns = timecounter_read(&adapter->hw_tc);
+
+	switch (adapter->hw.mac.type) {
+	case ixgbe_mac_X550:
+	case ixgbe_mac_X550EM_x:
+	case ixgbe_mac_x550em_a:
+		/* Upper 32 bits represent billions of cycles, lower 32 bits
+		 * represent cycles. However, we use timespec64_to_ns for the
+		 * correct math even though the units haven't been corrected
+		 * yet.
+		 */
+		ptp_read_system_prets(sts);
+		IXGBE_READ_REG(hw, IXGBE_SYSTIMR);
+		ptp_read_system_postts(sts);
+		ts->tv_nsec = IXGBE_READ_REG(hw, IXGBE_SYSTIML);
+		ts->tv_sec = IXGBE_READ_REG(hw, IXGBE_SYSTIMH);
+		stamp = timespec64_to_ns(ts);
+		break;
+	default:
+		ptp_read_system_prets(sts);
+		stamp = IXGBE_READ_REG(hw, IXGBE_SYSTIML);
+		ptp_read_system_postts(sts);
+		stamp |= (u64)IXGBE_READ_REG(hw, IXGBE_SYSTIMH) << 32;
+		break;
+	}
+
+	ns = timecounter_cyc2time(&adapter->hw_tc, stamp);
+
 	spin_unlock_irqrestore(&adapter->tmreg_lock, flags);
 
 	*ts = ns_to_timespec64(ns);
@@ -567,10 +597,14 @@ void ixgbe_ptp_overflow_check(struct ixgbe_adapter *adapter)
 {
 	bool timeout = time_is_before_jiffies(adapter->last_overflow_check +
 					     IXGBE_OVERFLOW_PERIOD);
-	struct timespec64 ts;
+	unsigned long flags;
 
 	if (timeout) {
-		ixgbe_ptp_gettime(&adapter->ptp_caps, &ts);
+		/* Update the timecounter */
+		spin_lock_irqsave(&adapter->tmreg_lock, flags);
+		timecounter_read(&adapter->hw_tc);
+		spin_unlock_irqrestore(&adapter->tmreg_lock, flags);
+
 		adapter->last_overflow_check = jiffies;
 	}
 }
@@ -1216,7 +1250,7 @@ static long ixgbe_ptp_create_clock(struct ixgbe_adapter *adapter)
 		adapter->ptp_caps.pps = 1;
 		adapter->ptp_caps.adjfreq = ixgbe_ptp_adjfreq_82599;
 		adapter->ptp_caps.adjtime = ixgbe_ptp_adjtime;
-		adapter->ptp_caps.gettime64 = ixgbe_ptp_gettime;
+		adapter->ptp_caps.gettimex64 = ixgbe_ptp_gettimex;
 		adapter->ptp_caps.settime64 = ixgbe_ptp_settime;
 		adapter->ptp_caps.enable = ixgbe_ptp_feature_enable;
 		adapter->ptp_setup_sdp = ixgbe_ptp_setup_sdp_x540;
@@ -1233,7 +1267,7 @@ static long ixgbe_ptp_create_clock(struct ixgbe_adapter *adapter)
 		adapter->ptp_caps.pps = 0;
 		adapter->ptp_caps.adjfreq = ixgbe_ptp_adjfreq_82599;
 		adapter->ptp_caps.adjtime = ixgbe_ptp_adjtime;
-		adapter->ptp_caps.gettime64 = ixgbe_ptp_gettime;
+		adapter->ptp_caps.gettimex64 = ixgbe_ptp_gettimex;
 		adapter->ptp_caps.settime64 = ixgbe_ptp_settime;
 		adapter->ptp_caps.enable = ixgbe_ptp_feature_enable;
 		break;
@@ -1249,7 +1283,7 @@ static long ixgbe_ptp_create_clock(struct ixgbe_adapter *adapter)
 		adapter->ptp_caps.pps = 0;
 		adapter->ptp_caps.adjfreq = ixgbe_ptp_adjfreq_X550;
 		adapter->ptp_caps.adjtime = ixgbe_ptp_adjtime;
-		adapter->ptp_caps.gettime64 = ixgbe_ptp_gettime;
+		adapter->ptp_caps.gettimex64 = ixgbe_ptp_gettimex;
 		adapter->ptp_caps.settime64 = ixgbe_ptp_settime;
 		adapter->ptp_caps.enable = ixgbe_ptp_feature_enable;
 		adapter->ptp_setup_sdp = NULL;
-- 
2.17.2

[PATCH net-next 8/8] tg3: extend PTP gettime function to read system clock

[Miroslav Lichvar]

This adds support for the PTP_SYS_OFFSET_EXTENDED ioctl.

Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
---
 drivers/net/ethernet/broadcom/tg3.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
index 89295306f161..ce44d208e137 100644
--- a/drivers/net/ethernet/broadcom/tg3.c
+++ b/drivers/net/ethernet/broadcom/tg3.c
@@ -6135,10 +6135,16 @@ static int tg3_setup_phy(struct tg3 *tp, bool force_reset)
 }
 
 /* tp->lock must be held */
-static u64 tg3_refclk_read(struct tg3 *tp)
+static u64 tg3_refclk_read(struct tg3 *tp, struct ptp_system_timestamp *sts)
 {
-	u64 stamp = tr32(TG3_EAV_REF_CLCK_LSB);
-	return stamp | (u64)tr32(TG3_EAV_REF_CLCK_MSB) << 32;
+	u64 stamp;
+
+	ptp_read_system_prets(sts);
+	stamp = tr32(TG3_EAV_REF_CLCK_LSB);
+	ptp_read_system_postts(sts);
+	stamp |= (u64)tr32(TG3_EAV_REF_CLCK_MSB) << 32;
+
+	return stamp;
 }
 
 /* tp->lock must be held */
@@ -6229,13 +6235,14 @@ static int tg3_ptp_adjtime(struct ptp_clock_info *ptp, s64 delta)
 	return 0;
 }
 
-static int tg3_ptp_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts)
+static int tg3_ptp_gettimex(struct ptp_clock_info *ptp, struct timespec64 *ts,
+			    struct ptp_system_timestamp *sts)
 {
 	u64 ns;
 	struct tg3 *tp = container_of(ptp, struct tg3, ptp_info);
 
 	tg3_full_lock(tp, 0);
-	ns = tg3_refclk_read(tp);
+	ns = tg3_refclk_read(tp, sts);
 	ns += tp->ptp_adjust;
 	tg3_full_unlock(tp);
 
@@ -6330,7 +6337,7 @@ static const struct ptp_clock_info tg3_ptp_caps = {
 	.pps		= 0,
 	.adjfreq	= tg3_ptp_adjfreq,
 	.adjtime	= tg3_ptp_adjtime,
-	.gettime64	= tg3_ptp_gettime,
+	.gettimex64	= tg3_ptp_gettimex,
 	.settime64	= tg3_ptp_settime,
 	.enable		= tg3_ptp_enable,
 };
-- 
2.17.2

RE: [PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[Keller, Jacob E]

> -----Original Message-----
> From: Miroslav Lichvar [mailto:mlichvar@redhat.com]
> Sent: Friday, November 09, 2018 2:15 AM
> To: netdev@vger.kernel.org
> Cc: Richard Cochran <richardcochran@gmail.com>; Keller, Jacob E
> <jacob.e.keller@intel.com>; Miroslav Lichvar <mlichvar@redhat.com>; Marcelo
> Tosatti <mtosatti@redhat.com>; Kirsher, Jeffrey T <jeffrey.t.kirsher@intel.com>;
> Michael Chan <michael.chan@broadcom.com>
> Subject: [PATCH net-next 0/8] More accurate PHC<->system clock synchronization
> 
> RFC->v1:
> - added new patches
> - separated PHC timestamp from ptp_system_timestamp
> - fixed memory leak in PTP_SYS_OFFSET_EXTENDED
> - changed PTP_SYS_OFFSET_EXTENDED to work with array of arrays
> - fixed PTP_SYS_OFFSET_EXTENDED to break correctly from loop
> - fixed timecounter updates in drivers
> - split gettimex in igb driver
> - fixed ptp_read_* functions to be available without
>   CONFIG_PTP_1588_CLOCK
> 
> This series enables a more accurate synchronization between PTP hardware
> clocks and the system clock.

Thanks for doing this, Miroslav!

> 
> The first two patches are minor cleanup/bug fixes.
> 
> The third patch adds an extended version of the PTP_SYS_OFFSET ioctl,
> which returns three timestamps for each measurement. The idea is to
> shorten the interval between the system timestamps to contain just the
> reading of the lowest register of the PHC in order to reduce the error
> in the measured offset and get a smaller upper bound on the maximum
> error.
> 
> The fourth patch deprecates the original gettime function.
> 
> The remaining patches update the gettime function in order to support
> the new ioctl in the e1000e, igb, ixgbe, and tg3 drivers.
> 
> Tests with few different NICs in different machines show that:
> - with an I219 (e1000e) the measured delay was reduced from 2500 to 1300
>   ns and the error in the measured offset, when compared to the cross
>   timestamping supported by the driver, was reduced by a factor of 5
> - with an I210 (igb) the delay was reduced from 5100 to 1700 ns
> - with an I350 (igb) the delay was reduced from 2300 to 750 ns
> - with an X550 (ixgbe) the delay was reduced from 1950 to 650 ns
> - with a BCM5720 (tg3) the delay was reduced from 2400 to 1200 ns
> 

Impressive results!

For the main portions and the Intel driver changes this is

Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>

Regards,
Jake

> 
> Miroslav Lichvar (8):
>   ptp: reorder declarations in ptp_ioctl()
>   ptp: check gettime64 return code in PTP_SYS_OFFSET ioctl
>   ptp: add PTP_SYS_OFFSET_EXTENDED ioctl
>   ptp: deprecate gettime64() in favor of gettimex64()
>   e1000e: extend PTP gettime function to read system clock
>   igb: extend PTP gettime function to read system clock
>   ixgbe: extend PTP gettime function to read system clock
>   tg3: extend PTP gettime function to read system clock
> 
>  drivers/net/ethernet/broadcom/tg3.c          | 19 ++++--
>  drivers/net/ethernet/intel/e1000e/e1000.h    |  3 +
>  drivers/net/ethernet/intel/e1000e/netdev.c   | 42 ++++++++++---
>  drivers/net/ethernet/intel/e1000e/ptp.c      | 16 +++--
>  drivers/net/ethernet/intel/igb/igb_ptp.c     | 65 +++++++++++++++++---
>  drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 54 +++++++++++++---
>  drivers/ptp/ptp_chardev.c                    | 55 ++++++++++++++---
>  drivers/ptp/ptp_clock.c                      |  5 +-
>  include/linux/ptp_clock_kernel.h             | 33 ++++++++++
>  include/uapi/linux/ptp_clock.h               | 12 ++++
>  10 files changed, 253 insertions(+), 51 deletions(-)
> 
> --
> 2.17.2

RE: [PATCH net-next 7/8] ixgbe: extend PTP gettime function to read system clock

[Keller, Jacob E]

> -----Original Message-----
> From: Miroslav Lichvar [mailto:mlichvar@redhat.com]
> Sent: Friday, November 09, 2018 2:15 AM
> To: netdev@vger.kernel.org
> Cc: Richard Cochran <richardcochran@gmail.com>; Keller, Jacob E
> <jacob.e.keller@intel.com>; Miroslav Lichvar <mlichvar@redhat.com>; Kirsher,
> Jeffrey T <jeffrey.t.kirsher@intel.com>
> Subject: [PATCH net-next 7/8] ixgbe: extend PTP gettime function to read system
> clock
> 
> -static int ixgbe_ptp_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts)
> +static int ixgbe_ptp_gettimex(struct ptp_clock_info *ptp,
> +			      struct timespec64 *ts,
> +			      struct ptp_system_timestamp *sts)
>  {
>  	struct ixgbe_adapter *adapter =
>  		container_of(ptp, struct ixgbe_adapter, ptp_caps);
> +	struct ixgbe_hw *hw = &adapter->hw;
>  	unsigned long flags;
> -	u64 ns;
> +	u64 ns, stamp;
> 
>  	spin_lock_irqsave(&adapter->tmreg_lock, flags);
> -	ns = timecounter_read(&adapter->hw_tc);
> +
> +	switch (adapter->hw.mac.type) {
> +	case ixgbe_mac_X550:
> +	case ixgbe_mac_X550EM_x:
> +	case ixgbe_mac_x550em_a:
> +		/* Upper 32 bits represent billions of cycles, lower 32 bits
> +		 * represent cycles. However, we use timespec64_to_ns for the
> +		 * correct math even though the units haven't been corrected
> +		 * yet.
> +		 */
> +		ptp_read_system_prets(sts);
> +		IXGBE_READ_REG(hw, IXGBE_SYSTIMR);
> +		ptp_read_system_postts(sts);
> +		ts->tv_nsec = IXGBE_READ_REG(hw, IXGBE_SYSTIML);
> +		ts->tv_sec = IXGBE_READ_REG(hw, IXGBE_SYSTIMH);
> +		stamp = timespec64_to_ns(ts);
> +		break;
> +	default:
> +		ptp_read_system_prets(sts);
> +		stamp = IXGBE_READ_REG(hw, IXGBE_SYSTIML);
> +		ptp_read_system_postts(sts);
> +		stamp |= (u64)IXGBE_READ_REG(hw, IXGBE_SYSTIMH) << 32;
> +		break;
> +	}
> +
> +	ns = timecounter_cyc2time(&adapter->hw_tc, stamp);
> +

At first, I was confused by this entire block of code, but then realized that we can't update the timecounter_read method, so we instead have to break this out so that our calls to ptp_read_system_prets() and ptp_read_system_postts() can be added between the register reads.

Ok, that makes sense.

>  	spin_unlock_irqrestore(&adapter->tmreg_lock, flags);
> 
>  	*ts = ns_to_timespec64(ns);
> @@ -567,10 +597,14 @@ void ixgbe_ptp_overflow_check(struct ixgbe_adapter
> *adapter)
>  {
>  	bool timeout = time_is_before_jiffies(adapter->last_overflow_check +
>  					     IXGBE_OVERFLOW_PERIOD);
> -	struct timespec64 ts;
> +	unsigned long flags;
> 
>  	if (timeout) {
> -		ixgbe_ptp_gettime(&adapter->ptp_caps, &ts);
> +		/* Update the timecounter */
> +		spin_lock_irqsave(&adapter->tmreg_lock, flags);
> +		timecounter_read(&adapter->hw_tc);
> +		spin_unlock_irqrestore(&adapter->tmreg_lock, flags);
> +

This also explains this change where we now have to update the timecounter during the overflow check.

Ok, this makes sense to me.

Thanks,
Jake

Re: [PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[David Miller]

From: Miroslav Lichvar <mlichvar@redhat.com>
Date: Fri,  9 Nov 2018 11:14:41 +0100

> RFC->v1:
> - added new patches
> - separated PHC timestamp from ptp_system_timestamp
> - fixed memory leak in PTP_SYS_OFFSET_EXTENDED
> - changed PTP_SYS_OFFSET_EXTENDED to work with array of arrays
> - fixed PTP_SYS_OFFSET_EXTENDED to break correctly from loop
> - fixed timecounter updates in drivers
> - split gettimex in igb driver
> - fixed ptp_read_* functions to be available without
>   CONFIG_PTP_1588_CLOCK
> 
> This series enables a more accurate synchronization between PTP hardware
> clocks and the system clock.
 ...

This series looks good to me but I want to give Richard an opportunity to
review it first.

Re: [PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

On Fri, 2018-11-09 at 15:28 -0800, David Miller wrote:
> From: Miroslav Lichvar <mlichvar@redhat.com>
> Date: Fri,  9 Nov 2018 11:14:41 +0100
> 
> > RFC->v1:
> > - added new patches
> > - separated PHC timestamp from ptp_system_timestamp
> > - fixed memory leak in PTP_SYS_OFFSET_EXTENDED
> > - changed PTP_SYS_OFFSET_EXTENDED to work with array of arrays
> > - fixed PTP_SYS_OFFSET_EXTENDED to break correctly from loop
> > - fixed timecounter updates in drivers
> > - split gettimex in igb driver
> > - fixed ptp_read_* functions to be available without
> >   CONFIG_PTP_1588_CLOCK
> > 
> > This series enables a more accurate synchronization between PTP
> > hardware
> > clocks and the system clock.
>  ...
> 
> This series looks good to me but I want to give Richard an opportunity to
> review it first.

Dave, I also do not want to hold this series up by picking up patches 5, 6
and 7 (Intel drivers) so please apply the entire series after Richard
provides his review.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[David Miller]

From: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Date: Fri, 09 Nov 2018 15:33:10 -0800

> On Fri, 2018-11-09 at 15:28 -0800, David Miller wrote:
>> From: Miroslav Lichvar <mlichvar@redhat.com>
>> Date: Fri,  9 Nov 2018 11:14:41 +0100
>> 
>> > RFC->v1:
>> > - added new patches
>> > - separated PHC timestamp from ptp_system_timestamp
>> > - fixed memory leak in PTP_SYS_OFFSET_EXTENDED
>> > - changed PTP_SYS_OFFSET_EXTENDED to work with array of arrays
>> > - fixed PTP_SYS_OFFSET_EXTENDED to break correctly from loop
>> > - fixed timecounter updates in drivers
>> > - split gettimex in igb driver
>> > - fixed ptp_read_* functions to be available without
>> >   CONFIG_PTP_1588_CLOCK
>> > 
>> > This series enables a more accurate synchronization between PTP
>> > hardware
>> > clocks and the system clock.
>>  ...
>> 
>> This series looks good to me but I want to give Richard an opportunity to
>> review it first.
> 
> Dave, I also do not want to hold this series up by picking up patches 5, 6
> and 7 (Intel drivers) so please apply the entire series after Richard
> provides his review.

Ok, will do.

Re: [PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[Richard Cochran]

On Fri, Nov 09, 2018 at 03:28:46PM -0800, David Miller wrote:
> This series looks good to me but I want to give Richard an opportunity to
> review it first.

The series is good to go.

Acked-by: Richard Cochran <richardcochran@gmail.com>

Re: [PATCH net-next 7/8] ixgbe: extend PTP gettime function to read system clock

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]

On Fri, 2018-11-09 at 11:14 +0100, Miroslav Lichvar wrote:
> This adds support for the PTP_SYS_OFFSET_EXTENDED ioctl.
> 
> Cc: Richard Cochran <richardcochran@gmail.com>
> Cc: Jacob Keller <jacob.e.keller@intel.com>
> Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
> ---
>  drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 54 ++++++++++++++++----
>  1 file changed, 44 insertions(+), 10 deletions(-)

Acked-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH net-next 6/8] igb: extend PTP gettime function to read system clock

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]

On Fri, 2018-11-09 at 11:14 +0100, Miroslav Lichvar wrote:
> This adds support for the PTP_SYS_OFFSET_EXTENDED ioctl.
> 
> Cc: Richard Cochran <richardcochran@gmail.com>
> Cc: Jacob Keller <jacob.e.keller@intel.com>
> Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
> ---
>  drivers/net/ethernet/intel/igb/igb_ptp.c | 65 ++++++++++++++++++++----
>  1 file changed, 55 insertions(+), 10 deletions(-)

Acked-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH net-next 5/8] e1000e: extend PTP gettime function to read system clock

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Fri, 2018-11-09 at 11:14 +0100, Miroslav Lichvar wrote:
> This adds support for the PTP_SYS_OFFSET_EXTENDED ioctl.
> 
> Cc: Richard Cochran <richardcochran@gmail.com>
> Cc: Jacob Keller <jacob.e.keller@intel.com>
> Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> Signed-off-by: Miroslav Lichvar <mlichvar@redhat.com>
> ---
>  drivers/net/ethernet/intel/e1000e/e1000.h  |  3 ++
>  drivers/net/ethernet/intel/e1000e/netdev.c | 42 ++++++++++++++++------
>  drivers/net/ethernet/intel/e1000e/ptp.c    | 16 +++++----
>  3 files changed, 45 insertions(+), 16 deletions(-)

Acked-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH net-next 0/8] More accurate PHC<->system clock synchronization

[David Miller]

From: Richard Cochran <richardcochran@gmail.com>
Date: Fri, 9 Nov 2018 17:44:31 -0800

> On Fri, Nov 09, 2018 at 03:28:46PM -0800, David Miller wrote:
>> This series looks good to me but I want to give Richard an opportunity to
>> review it first.
> 
> The series is good to go.
> 
> Acked-by: Richard Cochran <richardcochran@gmail.com>

Great, series applied to net-next, thanks everyone.

[PATCH net 1/2] ptp: check that rsv field is zero in struct ptp_sys_offset_extended

[Eugene Syromiatnikov]

Otherwise it is impossible to use it for something else, as it will break
userspace that puts garbage there.

The same check should be done in other structures, but the fact that
data in reserved fields is ignored is already part of the kernel ABI.

Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
---
 drivers/ptp/ptp_chardev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c
index 797fab3..7cbea79 100644
--- a/drivers/ptp/ptp_chardev.c
+++ b/drivers/ptp/ptp_chardev.c
@@ -224,7 +224,8 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
 			extoff = NULL;
 			break;
 		}
-		if (extoff->n_samples > PTP_MAX_SAMPLES) {
+		if (extoff->n_samples > PTP_MAX_SAMPLES
+		    || extoff->rsv[0] || extoff->rsv[1] || extoff->rsv[2]) {
 			err = -EINVAL;
 			break;
 		}
-- 
2.1.4

[PATCH net 2/2] ptp: uapi: change _IOW to IOWR in PTP_SYS_OFFSET_EXTENDED definition

[Eugene Syromiatnikov]

The ioctl command is read/write (or just read, if the fact that user space
writes n_samples field is ignored).

Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
---
 include/uapi/linux/ptp_clock.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/uapi/linux/ptp_clock.h b/include/uapi/linux/ptp_clock.h
index d73d839..1bc794a 100644
--- a/include/uapi/linux/ptp_clock.h
+++ b/include/uapi/linux/ptp_clock.h
@@ -147,7 +147,7 @@ struct ptp_pin_desc {
 #define PTP_SYS_OFFSET_PRECISE \
 	_IOWR(PTP_CLK_MAGIC, 8, struct ptp_sys_offset_precise)
 #define PTP_SYS_OFFSET_EXTENDED \
-	_IOW(PTP_CLK_MAGIC, 9, struct ptp_sys_offset_extended)
+	_IOWR(PTP_CLK_MAGIC, 9, struct ptp_sys_offset_extended)
 
 struct ptp_extts_event {
 	struct ptp_clock_time t; /* Time event occured. */
-- 
2.1.4

Re: [PATCH net 1/2] ptp: check that rsv field is zero in struct ptp_sys_offset_extended

[David Miller]

From: Eugene Syromiatnikov <esyr@redhat.com>
Date: Mon, 7 Jan 2019 16:22:29 +0100

> Otherwise it is impossible to use it for something else, as it will break
> userspace that puts garbage there.
> 
> The same check should be done in other structures, but the fact that
> data in reserved fields is ignored is already part of the kernel ABI.
> 
> Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>

I think the opportunity to enforce this has passed and you will break
userspace by doing this.

Re: [PATCH net 2/2] ptp: uapi: change _IOW to IOWR in PTP_SYS_OFFSET_EXTENDED definition

[David Miller]

From: Eugene Syromiatnikov <esyr@redhat.com>
Date: Mon, 7 Jan 2019 16:22:38 +0100

> --- a/include/uapi/linux/ptp_clock.h
> +++ b/include/uapi/linux/ptp_clock.h
> @@ -147,7 +147,7 @@ struct ptp_pin_desc {
>  #define PTP_SYS_OFFSET_PRECISE \
>  	_IOWR(PTP_CLK_MAGIC, 8, struct ptp_sys_offset_precise)
>  #define PTP_SYS_OFFSET_EXTENDED \
> -	_IOW(PTP_CLK_MAGIC, 9, struct ptp_sys_offset_extended)
> +	_IOWR(PTP_CLK_MAGIC, 9, struct ptp_sys_offset_extended)

Again, this changes the ioctl cmd value and will break userspace.

Re: [PATCH net 1/2] ptp: check that rsv field is zero in struct ptp_sys_offset_extended

[Miroslav Lichvar]

On Mon, Jan 07, 2019 at 08:29:38AM -0800, David Miller wrote:
> From: Eugene Syromiatnikov <esyr@redhat.com>
> Date: Mon, 7 Jan 2019 16:22:29 +0100
> 
> > Otherwise it is impossible to use it for something else, as it will break
> > userspace that puts garbage there.
> > 
> > The same check should be done in other structures, but the fact that
> > data in reserved fields is ignored is already part of the kernel ABI.
> > 
> > Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
> 
> I think the opportunity to enforce this has passed and you will break
> userspace by doing this.

FWIW, this ioctl wasn't present in a stable Linux release yet and the
two applications that I know that support it both zero the rsv field,
so I think at least this patch is very unlikely to break anything.

Anyway, thanks to Eugene for catching the issues.

-- 
Miroslav Lichvar

Re: [PATCH net 1/2] ptp: check that rsv field is zero in struct ptp_sys_offset_extended

[David Miller]

From: Miroslav Lichvar <mlichvar@redhat.com>
Date: Mon, 7 Jan 2019 17:57:41 +0100

> On Mon, Jan 07, 2019 at 08:29:38AM -0800, David Miller wrote:
>> From: Eugene Syromiatnikov <esyr@redhat.com>
>> Date: Mon, 7 Jan 2019 16:22:29 +0100
>> 
>> > Otherwise it is impossible to use it for something else, as it will break
>> > userspace that puts garbage there.
>> > 
>> > The same check should be done in other structures, but the fact that
>> > data in reserved fields is ignored is already part of the kernel ABI.
>> > 
>> > Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
>> 
>> I think the opportunity to enforce this has passed and you will break
>> userspace by doing this.
> 
> FWIW, this ioctl wasn't present in a stable Linux release yet and the
> two applications that I know that support it both zero the rsv field,
> so I think at least this patch is very unlikely to break anything.
> 
> Anyway, thanks to Eugene for catching the issues.

Oh, this ioctl went into this merge window didn't it.

Ok.  I'll keep reviewing this then.

Re: [PATCH net 1/2] ptp: check that rsv field is zero in struct ptp_sys_offset_extended

[Richard Cochran]

On Mon, Jan 07, 2019 at 08:29:38AM -0800, David Miller wrote:
> From: Eugene Syromiatnikov <esyr@redhat.com>
> Date: Mon, 7 Jan 2019 16:22:29 +0100
> 
> > Otherwise it is impossible to use it for something else, as it will break
> > userspace that puts garbage there.
> > 
> > The same check should be done in other structures, but the fact that
> > data in reserved fields is ignored is already part of the kernel ABI.
> > 
> > Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
> 
> I think the opportunity to enforce this has passed and you will break
> userspace by doing this.

Does this seriously mean that the 'rsv' field in

	struct ptp_extts_request {
		unsigned int index;  /* Which channel to configure. */
		unsigned int flags;  /* Bit field for PTP_xxx flags. */
		unsigned int rsv[2]; /* Reserved for future use. */
	};

can never be extended with some semantics?

Thanks,
Richard

Re: [PATCH net 1/2] ptp: check that rsv field is zero in struct ptp_sys_offset_extended

[Eugene Syromiatnikov]

On Mon, Jan 07, 2019 at 09:19:23PM -0800, Richard Cochran wrote:
> On Mon, Jan 07, 2019 at 08:29:38AM -0800, David Miller wrote:
> > From: Eugene Syromiatnikov <esyr@redhat.com>
> > Date: Mon, 7 Jan 2019 16:22:29 +0100
> > 
> > > Otherwise it is impossible to use it for something else, as it will break
> > > userspace that puts garbage there.
> > > 
> > > The same check should be done in other structures, but the fact that
> > > data in reserved fields is ignored is already part of the kernel ABI.
> > > 
> > > Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
> > 
> > I think the opportunity to enforce this has passed and you will break
> > userspace by doing this.
> 
> Does this seriously mean that the 'rsv' field in
> 
> 	struct ptp_extts_request {
> 		unsigned int index;  /* Which channel to configure. */
> 		unsigned int flags;  /* Bit field for PTP_xxx flags. */
> 		unsigned int rsv[2]; /* Reserved for future use. */
> 	};
> 
> can never be extended with some semantics?

Yes[*], since there's no check for garbage in both unused flags bits and rsv
values. The same for ptp_perout_request, ptp_sys_offset, ptp_pin_desc in
PTP_PIN_SETFUNC, and to some extent for ptp_sys_offset_precise, ptp_clock_caps,
and ptp_pin_desc in PTP_PIN_GETFUNC (all newly added data has to be
non-zero there). See also [1][2].

It can be worked around by adding new ioctl commands that operate on the
same structures, but also perform proper checks, though.

[*] Well, it could be extended with some data that is written from kernel
    to user space, but, again, it is not possible due to the fact that no
    new flags can be added there and it is an _IOW and not _IOWR command.
[1] https://www.kernel.org/doc/Documentation/process/adding-syscalls.rsti
    "Designing the API: Planning for Extension"
[2] http://man7.org/conf/lcna2015/designing_linux_kernel_APIs-LCNA_2015-Kerrisk.pdf

> Thanks,
> Richard

RE: [PATCH net 1/2] ptp: check that rsv field is zero in struct ptp_sys_offset_extended

[Keller, Jacob E]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
> Behalf Of Eugene Syromiatnikov
> Sent: Tuesday, January 08, 2019 6:38 AM
> To: Richard Cochran <richardcochran@gmail.com>
> Cc: David Miller <davem@davemloft.net>; mlichvar@redhat.com;
> netdev@vger.kernel.org; Keller, Jacob E <jacob.e.keller@intel.com>;
> mtosatti@redhat.com
> Subject: Re: [PATCH net 1/2] ptp: check that rsv field is zero in struct
> ptp_sys_offset_extended
> 
> On Mon, Jan 07, 2019 at 09:19:23PM -0800, Richard Cochran wrote:
> > On Mon, Jan 07, 2019 at 08:29:38AM -0800, David Miller wrote:
> > > From: Eugene Syromiatnikov <esyr@redhat.com>
> > > Date: Mon, 7 Jan 2019 16:22:29 +0100
> > >
> > > > Otherwise it is impossible to use it for something else, as it will break
> > > > userspace that puts garbage there.
> > > >
> > > > The same check should be done in other structures, but the fact that
> > > > data in reserved fields is ignored is already part of the kernel ABI.
> > > >
> > > > Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
> > >
> > > I think the opportunity to enforce this has passed and you will break
> > > userspace by doing this.
> >
> > Does this seriously mean that the 'rsv' field in
> >
> > 	struct ptp_extts_request {
> > 		unsigned int index;  /* Which channel to configure. */
> > 		unsigned int flags;  /* Bit field for PTP_xxx flags. */
> > 		unsigned int rsv[2]; /* Reserved for future use. */
> > 	};
> >
> > can never be extended with some semantics?
> 
> Yes[*], since there's no check for garbage in both unused flags bits and rsv
> values. The same for ptp_perout_request, ptp_sys_offset, ptp_pin_desc in
> PTP_PIN_SETFUNC, and to some extent for ptp_sys_offset_precise, ptp_clock_caps,
> and ptp_pin_desc in PTP_PIN_GETFUNC (all newly added data has to be
> non-zero there). See also [1][2].
> 
> It can be worked around by adding new ioctl commands that operate on the
> same structures, but also perform proper checks, though.
> 

So if/when we want to use these fields, this is what we'd have to do...

Is it worth doing that sooner rather than later? i.e. so that the new ioctl command is available for userspace to update to?

Thanks,
Jake