[PATCH RFC 0/4] pipesz: a utility for examining and resizing pipe buffers

* [PATCH RFC 0/4] pipesz: a utility for examining and resizing pipe buffers
@ 2022-04-12  4:59 Nathan Sharp
  2022-04-13  9:22 ` Karel Zak
  0 siblings, 1 reply; 2+ messages in thread
From: Nathan Sharp @ 2022-04-12  4:59 UTC (permalink / raw)
  To: util-linux; +Cc: Nathan Sharp

Way back in Linux 2.6.35 a mechanism was added to get and set the size of the backing buffers used by a pipe. As dataset sizes have ballooned and context switches have become more costly, I've found a few cases where increasing the capacity of a pipe has improved performance. Examining disk images and tee'ing compressed data where the usual culprits for me. It is probably useful in other cases as well.

I believe that util-linux is the right place for this functionality to be given to the programmers, system administrators, and data scientists who could benefit from it but don't have the time, know-how, or permission to fctnl their way to better performance from C. As far as I am aware, this functionality is unique to Linux and hasn't previously been available outside of C/ffi until now.

While I first stumbled across the usefulness of F_SETPIPE_SZ while working at Amazon Web Services, the code here and the method of interfacing with it is entirely new. Unfortunately, with my departure from AWS several years ago, the benchmark data I had gathered is beyond my reach. I'm not in a position to provide additional benchmarks on "production-scale" workloads.

I understand that adding a new utility to util-linux represents an ongoing maintenance burden for everyone. This is an RFC in part to determine if some mechanism in util-linux is warranted at all, given the relative obscurity of F_SETPIPE_SZ.

This changeset is broken into 4 patches. The first will add the pipesz binary, the second its tests, the third its documentation, and the forth its bash-completion. Care has been taken throughout to match the util-linux coding and documentation styles and to ease the burden of translation into different languages.

Thank you for taking time to read this cover letter, and thank you in advance for the additional time you might spend reviewing these patches.

Collaboratively,
  - Nathan Sharp

Nathan Sharp (4):
  misc-utils: add the pipesz utility
  pipesz: add tests
  pipesz: add manpage
  pipesz: add bash-completion script

 .gitignore                                  |   1 +
 bash-completion/Makemodule.am               |   3 +
 bash-completion/pipesz                      | 102 ++++++
 configure.ac                                |   8 +
 include/pathnames.h                         |   4 +
 meson.build                                 |  14 +
 meson_options.txt                           |   2 +
 misc-utils/Makemodule.am                    |   9 +
 misc-utils/meson.build                      |   4 +
 misc-utils/pipesz.1.adoc                    | 109 ++++++
 misc-utils/pipesz.c                         | 347 ++++++++++++++++++++
 tests/commands.sh                           |   1 +
 tests/expected/misc/pipesz-exec             |   1 +
 tests/expected/misc/pipesz-get-fd           |   1 +
 tests/expected/misc/pipesz-get-fd-bad.err   |   1 +
 tests/expected/misc/pipesz-get-file         |   1 +
 tests/expected/misc/pipesz-get-file-bad.err |   1 +
 tests/expected/misc/pipesz-set-fd-bad.err   |   1 +
 tests/expected/misc/pipesz-set-file-bad.err |   1 +
 tests/ts/misc/pipesz                        |  73 ++++
 20 files changed, 684 insertions(+)
 create mode 100644 bash-completion/pipesz
 create mode 100644 misc-utils/pipesz.1.adoc
 create mode 100644 misc-utils/pipesz.c
 create mode 100644 tests/expected/misc/pipesz-exec
 create mode 100644 tests/expected/misc/pipesz-get-fd
 create mode 100644 tests/expected/misc/pipesz-get-fd-bad.err
 create mode 100644 tests/expected/misc/pipesz-get-file
 create mode 100644 tests/expected/misc/pipesz-get-file-bad.err
 create mode 100644 tests/expected/misc/pipesz-set-fd-bad.err
 create mode 100644 tests/expected/misc/pipesz-set-file-bad.err
 create mode 100755 tests/ts/misc/pipesz

-- 
2.35.1

^ permalink raw reply	[flat|nested] 2+ messages in thread