Question about sendfile

Question about sendfile

[Hao Xu]

Hi Pavel,
I found this mail about sendfile in the maillist, may I ask why it's not
good to have one pipe each for a io-wq thread.
https://lore.kernel.org/io-uring/94dbbb15-4751-d03c-01fd-d25a0fe98e25@gmail.com/

Thanks,
Hao

Re: Question about sendfile

[Pavel Begunkov]

On 7/3/21 11:47 AM, Hao Xu wrote:
> Hi Pavel,
> I found this mail about sendfile in the maillist, may I ask why it's not
> good to have one pipe each for a io-wq thread.
> https://lore.kernel.org/io-uring/94dbbb15-4751-d03c-01fd-d25a0fe98e25@gmail.com/

IIRC, it's one page allocated for each such task, which is bearable but
don't like yet another chunk of uncontrollable implicit state. If there
not a bunch of active workers, IFAIK there is no way to force them to
drop their pipes.

I also don't remember the restrictions on the sendfile and what's with
the eternal question of "what to do if the write part of sendfile has
failed".   

Though, workers are now much more alike to user threads, so there
should be less of concern. And even though my gut feeling don't like
them, it may actually be useful. Do you have a good use case where
explicit pipes don't work well? 

-- 
Pavel Begunkov

Re: Question about sendfile

[Hao Xu]

在 2021/7/7 下午10:16, Pavel Begunkov 写道:
> On 7/3/21 11:47 AM, Hao Xu wrote:
>> Hi Pavel,
>> I found this mail about sendfile in the maillist, may I ask why it's not
>> good to have one pipe each for a io-wq thread.
>> https://lore.kernel.org/io-uring/94dbbb15-4751-d03c-01fd-d25a0fe98e25@gmail.com/
> 
> IIRC, it's one page allocated for each such task, which is bearable but
> don't like yet another chunk of uncontrollable implicit state. If there
> not a bunch of active workers, IFAIK there is no way to force them to
> drop their pipes.
> 
> I also don't remember the restrictions on the sendfile and what's with
> the eternal question of "what to do if the write part of sendfile has
> failed".
I haven't dig into it deeply, will do some investigation.
> 
> Though, workers are now much more alike to user threads, so there
> should be less of concern. And even though my gut feeling don't like
> them, it may actually be useful. Do you have a good use case where
> explicit pipes don't work well?
The thing is two linked splice sqes may be cut off in shared sqthread
case.
> 

Re: Question about sendfile

[Hao Xu]

在 2021/7/7 下午10:16, Pavel Begunkov 写道:
> On 7/3/21 11:47 AM, Hao Xu wrote:
>> Hi Pavel,
>> I found this mail about sendfile in the maillist, may I ask why it's not
>> good to have one pipe each for a io-wq thread.
>> https://lore.kernel.org/io-uring/94dbbb15-4751-d03c-01fd-d25a0fe98e25@gmail.com/
> 
> IIRC, it's one page allocated for each such task, which is bearable but
> don't like yet another chunk of uncontrollable implicit state. If there
> not a bunch of active workers, IFAIK there is no way to force them to
> drop their pipes.
> 
> I also don't remember the restrictions on the sendfile and what's with
> the eternal question of "what to do if the write part of sendfile has
> failed".
Hi Pavel,
Could you explain this question a little bit.., is there any special
concern? What I thought is sendfile does what it does,when it fails,
it will return -1 and errno is set appropriately.
> 
> Though, workers are now much more alike to user threads, so there
> should be less of concern. And even though my gut feeling don't like
> them, it may actually be useful. Do you have a good use case where
> explicit pipes don't work well?
> 

Re: Question about sendfile

[Pavel Begunkov]

On 11/26/21 08:50, Hao Xu wrote:
> 在 2021/7/7 下午10:16, Pavel Begunkov 写道:
>> On 7/3/21 11:47 AM, Hao Xu wrote:
>>> Hi Pavel,
>>> I found this mail about sendfile in the maillist, may I ask why it's not
>>> good to have one pipe each for a io-wq thread.
>>> https://lore.kernel.org/io-uring/94dbbb15-4751-d03c-01fd-d25a0fe98e25@gmail.com/
>>
>> IIRC, it's one page allocated for each such task, which is bearable but
>> don't like yet another chunk of uncontrollable implicit state. If there
>> not a bunch of active workers, IFAIK there is no way to force them to
>> drop their pipes.
>>
>> I also don't remember the restrictions on the sendfile and what's with
>> the eternal question of "what to do if the write part of sendfile has
>> failed".
> Hi Pavel,
> Could you explain this question a little bit.., is there any special
> concern? What I thought is sendfile does what it does,when it fails,
> it will return -1 and errno is set appropriately.

I don't have much concern about this one, though interesting how
it was solved and whether you need to know the issuing task to
handle errors.

I didn't like more having uncontrollable memory, i.e. a pipe per
worker that used sendfile (IIRC it keeps 1 page), and no way to
reuse the memory or release it. In other words, a sendfile request
chooses to which worker it goes randomly. E.g. First sendfile may go
to worker 1 leaving 1 page allocated. The second sendfile goes to
worker 2, so after we have 2 pages allocated, an so on. At some
point you have N pages, where any particular one may likely be
rarely used.

Please correct me if I forgot how it works and wrong here.

>> Though, workers are now much more alike to user threads, so there
>> should be less of concern. And even though my gut feeling don't like
>> them, it may actually be useful. Do you have a good use case where
>> explicit pipes don't work well?

-- 
Pavel Begunkov

Re: Question about sendfile

[Hao Xu]

在 2021/12/4 上午12:03, Pavel Begunkov 写道:
> On 11/26/21 08:50, Hao Xu wrote:
>> 在 2021/7/7 下午10:16, Pavel Begunkov 写道:
>>> On 7/3/21 11:47 AM, Hao Xu wrote:
>>>> Hi Pavel,
>>>> I found this mail about sendfile in the maillist, may I ask why it's 
>>>> not
>>>> good to have one pipe each for a io-wq thread.
>>>> https://lore.kernel.org/io-uring/94dbbb15-4751-d03c-01fd-d25a0fe98e25@gmail.com/ 
>>>>
>>>
>>> IIRC, it's one page allocated for each such task, which is bearable but
>>> don't like yet another chunk of uncontrollable implicit state. If there
>>> not a bunch of active workers, IFAIK there is no way to force them to
>>> drop their pipes.
>>>
>>> I also don't remember the restrictions on the sendfile and what's with
>>> the eternal question of "what to do if the write part of sendfile has
>>> failed".
>> Hi Pavel,
>> Could you explain this question a little bit.., is there any special
>> concern? What I thought is sendfile does what it does,when it fails,
>> it will return -1 and errno is set appropriately.
> 
> I don't have much concern about this one, though interesting how
> it was solved and whether you need to know the issuing task to
> handle errors.
> 
> I didn't like more having uncontrollable memory, i.e. a pipe per
> worker that used sendfile (IIRC it keeps 1 page), and no way to
> reuse the memory or release it. In other words, a sendfile request
> chooses to which worker it goes randomly. E.g. First sendfile may go
> to worker 1 leaving 1 page allocated. The second sendfile goes to
> worker 2, so after we have 2 pages allocated, an so on. At some
> point you have N pages, where any particular one may likely be
> rarely used.
I'm not sure when the pipe is freed(seems it won't be freed after
sendfile call and it is reused). If it won't be freed automatically
we can manually free it when a worker completes a sendfile work. I think
in normal cases, a user cannot and shouldn't visit the internal pipe
after senfile is done no matter it succeeds or fails, which means we
can free the pipe at that time. Not 100% sure but probably..
> 
> Please correct me if I forgot how it works and wrong here.
> 
>>> Though, workers are now much more alike to user threads, so there
>>> should be less of concern. And even though my gut feeling don't like
>>> them, it may actually be useful. Do you have a good use case where
>>> explicit pipes don't work well?
> 

Question about sendfile

[Xiuduan Fang]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]


Hi,

I am trying to beat the I/O bottleneck so as to speed up bulk data transfers 
in high speed network. It seems that the system call sendfile() can help to 
reduce CPU utilization and speedup data transfers. But I have one question 
about the system call,

First,  Linux sendfile requires that the input file descriptor cannot be a 
network socket. What are the reasons for such a restriction? Sending a 
socket to a file via zero copy is definitely useful.  Actually this is one 
approach I am trying to do to improve performance.  Some discussions on 
Linux zero copy said this is because it is harder. Sending a socket to a 
file via zero copy needs the support of NICs. I cannot understand this 
explanation. It seems that FreeBSD has implemented bidirectional zero 
copy(http://people.freebsd.org/~ken/zero_copy/#Download). So why Linux does 
not support it? What shall I do to release the restriction that Linux 
enforces on sendfile?

Any hints will be highly appreciated. Thanks.

Xiuduan Fang 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Xiuduan Fang.vcf --]
[-- Type: text/x-vcard; name="Xiuduan Fang.vcf", Size: 2105 bytes --]

BEGIN:VCARD
VERSION:2.1
N:Fang;Xiuduan
FN:Xiuduan Fang
ORG:University of Virginia;Computer Science Dept
TITLE:2nd Year Graduate
TEL;WORK;VOICE:1-434-982-2296
ADR;WORK:;;151 Engineer's Way, P.O. Box 400740;Charlottesville;VA;22904-4743;USA
LABEL;WORK;ENCODING=QUOTED-PRINTABLE:151 Engineer's Way, P.O. Box 400740=0D=0ACharlottesville, VA 22904-4743=0D=
=0AUSA
KEY;X509;ENCODING=BASE64:
    MIIEYzCCA8ygAwIBAgIQJav9Aj366wHb4hpgZ1JRKDANBgkqhkiG9w0BAQQFADCBzDEXMBUG
    A1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsx
    RjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVwb3NpdG9yeS9SUEEgSW5jb3JwLiBCeSBS
    ZWYuLExJQUIuTFREKGMpOTgxSDBGBgNVBAMTP1ZlcmlTaWduIENsYXNzIDEgQ0EgSW5kaXZp
    ZHVhbCBTdWJzY3JpYmVyLVBlcnNvbmEgTm90IFZhbGlkYXRlZDAeFw0wNDEwMDQwMDAwMDBa
    Fw0wNDEyMDMyMzU5NTlaMIIBBzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsT
    FlZlcmlTaWduIFRydXN0IE5ldHdvcmsxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVw
    b3NpdG9yeS9SUEEgSW5jb3JwLiBieSBSZWYuLExJQUIuTFREKGMpOTgxHjAcBgNVBAsTFVBl
    cnNvbmEgTm90IFZhbGlkYXRlZDEnMCUGA1UECxMeRGlnaXRhbCBJRCBDbGFzcyAxIC0gTWlj
    cm9zb2Z0MRUwEwYDVQQDFAxYaXVkdWFuIEZhbmcxIzAhBgkqhkiG9w0BCQEWFHhmNGNAY3Mu
    dmlyZ2luaWEuZWR1MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDRn6bRIKJguTHWwMQB
    aKdf9VOH3758Ba6owaoGy5ME/fds2ZPTWvuW+IyFskupZ0stK7f9OtzKAi+EFkFlD1umHItr
    XM74PapnYI/8TR/svKbZJLodGNAto9sJjvLQkNK6hwvTp5eBwQ1YgC7GmZHmtshPH8N+8Ast
    xOxoflE6dwIDAQABo4IBBjCCAQIwCQYDVR0TBAIwADCBrAYDVR0gBIGkMIGhMIGeBgtghkgB
    hvhFAQcBATCBjjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL0NQUzBi
    BggrBgEFBQcCAjBWMBUWDlZlcmlTaWduLCBJbmMuMAMCAQEaPVZlcmlTaWduJ3MgQ1BTIGlu
    Y29ycC4gYnkgcmVmZXJlbmNlIGxpYWIuIGx0ZC4gKGMpOTcgVmVyaVNpZ24wEQYJYIZIAYb4
    QgEBBAQDAgeAMDMGA1UdHwQsMCowKKAmoCSGImh0dHA6Ly9jcmwudmVyaXNpZ24uY29tL2Ns
    YXNzMS5jcmwwDQYJKoZIhvcNAQEEBQADgYEASTrowJeKxyNUZbF+AwGXfqXBrOyN3b+3aRDN
    CgSQVp0zaLHwLReTa+3mEnwtrMN6QSM02gPbiuzVkdmGyxmlHAmrHQ2l61fyotoMH47RJbe+
    qzClrcMr2Y9AAyTNeVrvfSZRdKMZ9HFduUu1tn5/FTZFCK8Xoaq3BIo81b8nHGs=


EMAIL;PREF;INTERNET:xf4c@cs.virginia.edu
REV:20050208T032639Z
END:VCARD

Re: Question about sendfile

[Gianni Tedesco]

On Mon, 2005-02-07 at 22:26 -0500, Xiuduan Fang wrote:
> Hi,
> 
> I am trying to beat the I/O bottleneck so as to speed up bulk data transfers 
> in high speed network. It seems that the system call sendfile() can help to 
> reduce CPU utilization and speedup data transfers. But I have one question 
> about the system call,
> 
> First,  Linux sendfile requires that the input file descriptor cannot be a 
> network socket. What are the reasons for such a restriction? Sending a 
> socket to a file via zero copy is definitely useful.  Actually this is one 
> approach I am trying to do to improve performance.  Some discussions on 
> Linux zero copy said this is because it is harder. Sending a socket to a 
> file via zero copy needs the support of NICs. I cannot understand this 
> explanation. It seems that FreeBSD has implemented bidirectional zero 
> copy(http://people.freebsd.org/~ken/zero_copy/#Download). So why Linux does 
> not support it? What shall I do to release the restriction that Linux 
> enforces on sendfile?

>From the URL you posted:

"[zero-copy receive] generally requires some sort of intelligence on the
NIC to make sure that the payload starts in its own buffer.  This is
called "header splitting".  Currently the only NICs with support for
header splitting are Alteon Tigon 2 based boards running slightly
modified firmware."

Perhaps that explains it.

Not to mention the other complications that are involved if you scroll
down the page and read the FAQ.

Have you done any profiling work to see where your CPU cycles are being
spent?

-- 
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import