All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: [SPDK] Issue running CIT's
@ 2018-06-14 20:39 Luse, Paul E
  0 siblings, 0 replies; 5+ messages in thread
From: Luse, Paul E @ 2018-06-14 20:39 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 1989 bytes --]

Hi Ed,

I’ve got a system here I can try this on, not sure I will have time before I head out for vacation though but I will try.  Seth is back next week so if nobody else can jump in maybe send this out again Tue or Wed

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Rodriguez, Edwin
Sent: Thursday, June 14, 2018 12:09 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Issue running CIT's

One of our testbeds had to be rebuilt which resulted in the system getting updated to the most recent Fedora 26 packages.

[edwinr(a)ssan-rx2560-03:14]/home/edwinr> uname -r
4.16.11-100.fc26.x86_64

But I’m having trouble getting the tests to complete successfully, the error from running test/nvmf/filesystem/filesystem.sh is:


14:58:39    # timing_enter parted

 14:58:39   # set +x

 14:58:39   # parted -s /dev/nvme0n1 mklabel msdos mkpart primary 0% 100%

Warning: The resulting partition is not properly aligned for best performance: 1s % 2048s != 0s

 14:58:39   # timing_exit parted

 14:58:39   # set +x

 14:58:39   # sleep 1

 14:58:40   # for fstype in "ext4" "btrfs" "xfs"

 14:58:40   # timing_enter ext4

 14:58:40   # set +x

 14:58:40   # '[' ext4 = ext4 ']'

 14:58:40   # force=-F

 14:58:40   # mkfs.ext4 -F /dev/nvme0n1p1

mke2fs 1.43.4 (31-Jan-2017)

Discarding device blocks: done

Creating filesystem with 65532 1k blocks and 16384 inodes

Filesystem UUID: 9eba159d-8d85-4c40-915e-041844d4a785

Superblock backups stored on blocks:

      8193, 24577, 40961, 57345



Allocating group tables: done

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information:

Warning, had trouble writing out superblocks.

  14:58:40  # trap - ERR

  14:58:40  # print_backtrace

  14:58:40  # local shell_options=ehxBE

  14:58:40  # set +x

What am I missing something?

Ed R



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 10582 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [SPDK] Issue running CIT's
@ 2018-06-20 17:38 Meneghini, John
  0 siblings, 0 replies; 5+ messages in thread
From: Meneghini, John @ 2018-06-20 17:38 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 4596 bytes --]

> I found that some of the tests were failing on kernel version 4.15.x and reverted those machines back to 4.13.x.

Yes, we found that same problem.  ☹

> I will post any new information I gather or updates to this thread

Thanks Seth!

From: SPDK <spdk-bounces(a)lists.01.org> on behalf of "Howell, Seth" <seth.howell(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, June 20, 2018 at 12:01 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Issue running CIT's

Hi John,

Thank you for providing this information. I have had a few issues myself with updating the machines in the SPDK test pool to the latest kernel. I found that some of the tests were failing on kernel version 4.15.x and reverted those machines back to 4.13.x. That was before I left for vacation after graduation.
Now that I am back full time, I will be building a new image using fedora 28 (now that fedora 26 is approaching end of service). With the new kernel version on that machine, I expect to run into the same problems that you and Ed are facing below and will be working on the solution. I will post any new information I gather or updates to this thread.

Thanks,

Seth

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Meneghini, John
Sent: Tuesday, June 19, 2018 8:36 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Issue running CIT's

Here’s the latest:

We have 3 CIT testbeds.  All running Fedora 26.  The first two are down because of the problem Ed documents below.  Still no solution.  This is taking a lot of our time.  Ed and I are starting to feel like full time system administrators.

The third CIT testbed was still working until a recent patch/testbed change cause the master branch to fail.

Here was the fix:


When you reboot the CIT testbed make sure to run the following command prior to running CIT tests,

echo eth | sudo tee /sys/bus/pci/devices/0000\:81\:00.0/mlx4_port1
sudo ifup ens7

After doing this I see that the master branch passes cit.

/John


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Luse, Paul E" <paul.e.luse(a)intel.com<mailto:paul.e.luse(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, June 14, 2018 at 4:39 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Issue running CIT's

Hi Ed,

I’ve got a system here I can try this on, not sure I will have time before I head out for vacation though but I will try.  Seth is back next week so if nobody else can jump in maybe send this out again Tue or Wed

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Rodriguez, Edwin
Sent: Thursday, June 14, 2018 12:09 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Issue running CIT's

One of our testbeds had to be rebuilt which resulted in the system getting updated to the most recent Fedora 26 packages.

[edwinr(a)ssan-rx2560-03:14]/home/edwinr> uname -r
4.16.11-100.fc26.x86_64

But I’m having trouble getting the tests to complete successfully, the error from running test/nvmf/filesystem/filesystem.sh is:


14:58:39    # timing_enter parted

 14:58:39   # set +x

 14:58:39   # parted -s /dev/nvme0n1 mklabel msdos mkpart primary 0% 100%

Warning: The resulting partition is not properly aligned for best performance: 1s % 2048s != 0s

 14:58:39   # timing_exit parted

 14:58:39   # set +x

 14:58:39   # sleep 1

 14:58:40   # for fstype in "ext4" "btrfs" "xfs"

 14:58:40   # timing_enter ext4

 14:58:40   # set +x

 14:58:40   # '[' ext4 = ext4 ']'

 14:58:40   # force=-F

 14:58:40   # mkfs.ext4 -F /dev/nvme0n1p1

mke2fs 1.43.4 (31-Jan-2017)

Discarding device blocks: done

Creating filesystem with 65532 1k blocks and 16384 inodes

Filesystem UUID: 9eba159d-8d85-4c40-915e-041844d4a785

Superblock backups stored on blocks:

      8193, 24577, 40961, 57345



Allocating group tables: done

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information:

Warning, had trouble writing out superblocks.

  14:58:40  # trap - ERR

  14:58:40  # print_backtrace

  14:58:40  # local shell_options=ehxBE

  14:58:40  # set +x

What am I missing something?

Ed R



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 18263 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [SPDK] Issue running CIT's
@ 2018-06-20 16:00 Howell, Seth
  0 siblings, 0 replies; 5+ messages in thread
From: Howell, Seth @ 2018-06-20 16:00 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 4034 bytes --]

Hi John,

Thank you for providing this information. I have had a few issues myself with updating the machines in the SPDK test pool to the latest kernel. I found that some of the tests were failing on kernel version 4.15.x and reverted those machines back to 4.13.x. That was before I left for vacation after graduation.
Now that I am back full time, I will be building a new image using fedora 28 (now that fedora 26 is approaching end of service). With the new kernel version on that machine, I expect to run into the same problems that you and Ed are facing below and will be working on the solution. I will post any new information I gather or updates to this thread.

Thanks,

Seth

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Meneghini, John
Sent: Tuesday, June 19, 2018 8:36 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Issue running CIT's

Here’s the latest:

We have 3 CIT testbeds.  All running Fedora 26.  The first two are down because of the problem Ed documents below.  Still no solution.  This is taking a lot of our time.  Ed and I are starting to feel like full time system administrators.

The third CIT testbed was still working until a recent patch/testbed change cause the master branch to fail.

Here was the fix:


When you reboot the CIT testbed make sure to run the following command prior to running CIT tests,

echo eth | sudo tee /sys/bus/pci/devices/0000\:81\:00.0/mlx4_port1
sudo ifup ens7

After doing this I see that the master branch passes cit.

/John


From: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> on behalf of "Luse, Paul E" <paul.e.luse(a)intel.com<mailto:paul.e.luse(a)intel.com>>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Date: Thursday, June 14, 2018 at 4:39 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Issue running CIT's

Hi Ed,

I’ve got a system here I can try this on, not sure I will have time before I head out for vacation though but I will try.  Seth is back next week so if nobody else can jump in maybe send this out again Tue or Wed

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Rodriguez, Edwin
Sent: Thursday, June 14, 2018 12:09 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Issue running CIT's

One of our testbeds had to be rebuilt which resulted in the system getting updated to the most recent Fedora 26 packages.

[edwinr(a)ssan-rx2560-03:14]/home/edwinr> uname -r
4.16.11-100.fc26.x86_64

But I’m having trouble getting the tests to complete successfully, the error from running test/nvmf/filesystem/filesystem.sh is:


14:58:39    # timing_enter parted

 14:58:39   # set +x

 14:58:39   # parted -s /dev/nvme0n1 mklabel msdos mkpart primary 0% 100%

Warning: The resulting partition is not properly aligned for best performance: 1s % 2048s != 0s

 14:58:39   # timing_exit parted

 14:58:39   # set +x

 14:58:39   # sleep 1

 14:58:40   # for fstype in "ext4" "btrfs" "xfs"

 14:58:40   # timing_enter ext4

 14:58:40   # set +x

 14:58:40   # '[' ext4 = ext4 ']'

 14:58:40   # force=-F

 14:58:40   # mkfs.ext4 -F /dev/nvme0n1p1

mke2fs 1.43.4 (31-Jan-2017)

Discarding device blocks: done

Creating filesystem with 65532 1k blocks and 16384 inodes

Filesystem UUID: 9eba159d-8d85-4c40-915e-041844d4a785

Superblock backups stored on blocks:

      8193, 24577, 40961, 57345



Allocating group tables: done

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information:

Warning, had trouble writing out superblocks.

  14:58:40  # trap - ERR

  14:58:40  # print_backtrace

  14:58:40  # local shell_options=ehxBE

  14:58:40  # set +x

What am I missing something?

Ed R



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 16297 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [SPDK] Issue running CIT's
@ 2018-06-20  3:35 Meneghini, John
  0 siblings, 0 replies; 5+ messages in thread
From: Meneghini, John @ 2018-06-20  3:35 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 2959 bytes --]

Here’s the latest:

We have 3 CIT testbeds.  All running Fedora 26.  The first two are down because of the problem Ed documents below.  Still no solution.  This is taking a lot of our time.  Ed and I are starting to feel like full time system administrators.

The third CIT testbed was still working until a recent patch/testbed change cause the master branch to fail.

Here was the fix:


When you reboot the CIT testbed make sure to run the following command prior to running CIT tests,
echo eth | sudo tee /sys/bus/pci/devices/0000\:81\:00.0/mlx4_port1
sudo ifup ens7
After doing this I see that the master branch passes cit.

/John


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of "Luse, Paul E" <paul.e.luse(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Thursday, June 14, 2018 at 4:39 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Issue running CIT's

Hi Ed,

I’ve got a system here I can try this on, not sure I will have time before I head out for vacation though but I will try.  Seth is back next week so if nobody else can jump in maybe send this out again Tue or Wed

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Rodriguez, Edwin
Sent: Thursday, June 14, 2018 12:09 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Issue running CIT's

One of our testbeds had to be rebuilt which resulted in the system getting updated to the most recent Fedora 26 packages.

[edwinr(a)ssan-rx2560-03:14]/home/edwinr> uname -r
4.16.11-100.fc26.x86_64

But I’m having trouble getting the tests to complete successfully, the error from running test/nvmf/filesystem/filesystem.sh is:


14:58:39    # timing_enter parted

 14:58:39   # set +x

 14:58:39   # parted -s /dev/nvme0n1 mklabel msdos mkpart primary 0% 100%

Warning: The resulting partition is not properly aligned for best performance: 1s % 2048s != 0s

 14:58:39   # timing_exit parted

 14:58:39   # set +x

 14:58:39   # sleep 1

 14:58:40   # for fstype in "ext4" "btrfs" "xfs"

 14:58:40   # timing_enter ext4

 14:58:40   # set +x

 14:58:40   # '[' ext4 = ext4 ']'

 14:58:40   # force=-F

 14:58:40   # mkfs.ext4 -F /dev/nvme0n1p1

mke2fs 1.43.4 (31-Jan-2017)

Discarding device blocks: done

Creating filesystem with 65532 1k blocks and 16384 inodes

Filesystem UUID: 9eba159d-8d85-4c40-915e-041844d4a785

Superblock backups stored on blocks:

      8193, 24577, 40961, 57345



Allocating group tables: done

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information:

Warning, had trouble writing out superblocks.

  14:58:40  # trap - ERR

  14:58:40  # print_backtrace

  14:58:40  # local shell_options=ehxBE

  14:58:40  # set +x

What am I missing something?

Ed R



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13969 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread
* [SPDK] Issue running CIT's
@ 2018-06-14 19:08 Rodriguez, Edwin
  0 siblings, 0 replies; 5+ messages in thread
From: Rodriguez, Edwin @ 2018-06-14 19:08 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

One of our testbeds had to be rebuilt which resulted in the system getting updated to the most recent Fedora 26 packages.

[edwinr(a)ssan-rx2560-03:14]/home/edwinr> uname -r
4.16.11-100.fc26.x86_64

But I’m having trouble getting the tests to complete successfully, the error from running test/nvmf/filesystem/filesystem.sh is:


14:58:39    # timing_enter parted

 14:58:39   # set +x

 14:58:39   # parted -s /dev/nvme0n1 mklabel msdos mkpart primary 0% 100%

Warning: The resulting partition is not properly aligned for best performance: 1s % 2048s != 0s

 14:58:39   # timing_exit parted

 14:58:39   # set +x

 14:58:39   # sleep 1

 14:58:40   # for fstype in "ext4" "btrfs" "xfs"

 14:58:40   # timing_enter ext4

 14:58:40   # set +x

 14:58:40   # '[' ext4 = ext4 ']'

 14:58:40   # force=-F

 14:58:40   # mkfs.ext4 -F /dev/nvme0n1p1

mke2fs 1.43.4 (31-Jan-2017)

Discarding device blocks: done

Creating filesystem with 65532 1k blocks and 16384 inodes

Filesystem UUID: 9eba159d-8d85-4c40-915e-041844d4a785

Superblock backups stored on blocks:

      8193, 24577, 40961, 57345



Allocating group tables: done

Writing inode tables: done

Creating journal (4096 blocks): done

Writing superblocks and filesystem accounting information:

Warning, had trouble writing out superblocks.

  14:58:40  # trap - ERR

  14:58:40  # print_backtrace

  14:58:40  # local shell_options=ehxBE

  14:58:40  # set +x

What am I missing something?

Ed R



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 8830 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-06-20 17:38 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-14 20:39 [SPDK] Issue running CIT's Luse, Paul E
  -- strict thread matches above, loose matches on Subject: below --
2018-06-20 17:38 Meneghini, John
2018-06-20 16:00 Howell, Seth
2018-06-20  3:35 Meneghini, John
2018-06-14 19:08 Rodriguez, Edwin

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.