Question about SOCK_SEQPACKET

Question about SOCK_SEQPACKET

[Sam Kumar]

Hello,
I have recently had occasion to use SOCK_SEQPACKET sockets on Linux,
and noticed some odd behavior. When using sendmsg and recvmsg with
these sockets, it seems that the "end-of-record" flag (MSG_EOR) is not
being propagated correctly.

The man page for recvmsg(2) states:
> The  msg_flags  field  in the msghdr is set on return of recvmsg().  It
>        can contain several flags:
>
>        MSG_EOR
>               indicates end-of-record; the data returned  completed  a  record
>               (generally used with sockets of type SOCK_SEQPACKET).
>

The man page for recvmsg(3) states:
> For
>       message-based  sockets,  such as SOCK_DGRAM and SOCK_SEQPACKET, the entire
>       message shall be read in a single operation.



This leads me to believe that MSG_EOR should be set in the msghdr
struct whenever recvmsg() returns data. However, I am not observing
this flag ever being set, whether or not I set the MSG_EOR when
sending the messages.

If it helps you can take a look at the code I'm using. It is at
https://github.com/samkumar/seqpacket-test/, commit
2a7dbc1f94bafce6950ee726bdd54da96945d083 (HEAD of master at the time
of writing). Look at server.c and client.c (don't bother with
goclient.go).

The reason that I need to check MSG_EOR is that I need to distinguish
between EOF and messages of length 0. For SOCK_STREAM sockets, a
return value of 0 unambiguously means EOF, and for SOCK_DGRAM sockets
a return value of 0 unambiguously means that a datagram of length 0
was received.

Because SOCK_SEQPACKET is both connection-based and message-oriented,
a return value of 0 is ambiguous. Based on my reading of the man
pages, reading the MSG_EOR bit would let me disambiguate between EOF
and a zero-length datagram, because MSG_EOR would be set for a
zero-length datagram, but would not be set for EOF.

If someone could please help me understand MSG_EOR, and how to
distinguish between EOF and zero-length messages in a SOCK_SEQPACKET
connection, I would definitely appreciate it!

Thanks,
Sam Kumar

Re: Question about SOCK_SEQPACKET

[Steven Whitehouse]

Hi,


On 05/05/17 06:18, Sam Kumar wrote:
> Hello,
> I have recently had occasion to use SOCK_SEQPACKET sockets on Linux,
> and noticed some odd behavior. When using sendmsg and recvmsg with
> these sockets, it seems that the "end-of-record" flag (MSG_EOR) is not
> being propagated correctly.
It depends which protocol you are using as to whether that is true. 
SOCK_SEQPACKET is supposed to be identical to SOCK_STREAM except for the 
record separators. That is true for DECnet (but whether DECnet is still 
functional is another thing!) and not true for ax.25 which uses 
SOCK_SEQPACKET incorrectly. For AF_UNIX that you are using I'm not quite 
sure what would be expected.

> The man page for recvmsg(2) states:
>> The  msg_flags  field  in the msghdr is set on return of recvmsg().  It
>>         can contain several flags:
>>
>>         MSG_EOR
>>                indicates end-of-record; the data returned  completed  a  record
>>                (generally used with sockets of type SOCK_SEQPACKET).
>>
> The man page for recvmsg(3) states:
>> For
>>        message-based  sockets,  such as SOCK_DGRAM and SOCK_SEQPACKET, the entire
>>        message shall be read in a single operation.
>
>
> This leads me to believe that MSG_EOR should be set in the msghdr
> struct whenever recvmsg() returns data. However, I am not observing
> this flag ever being set, whether or not I set the MSG_EOR when
> sending the messages.
>
> If it helps you can take a look at the code I'm using. It is at
> https://github.com/samkumar/seqpacket-test/, commit
> 2a7dbc1f94bafce6950ee726bdd54da96945d083 (HEAD of master at the time
> of writing). Look at server.c and client.c (don't bother with
> goclient.go).
>
> The reason that I need to check MSG_EOR is that I need to distinguish
> between EOF and messages of length 0. For SOCK_STREAM sockets, a
> return value of 0 unambiguously means EOF, and for SOCK_DGRAM sockets
> a return value of 0 unambiguously means that a datagram of length 0
> was received.
>
> Because SOCK_SEQPACKET is both connection-based and message-oriented,
> a return value of 0 is ambiguous. Based on my reading of the man
> pages, reading the MSG_EOR bit would let me disambiguate between EOF
> and a zero-length datagram, because MSG_EOR would be set for a
> zero-length datagram, but would not be set for EOF.
>
> If someone could please help me understand MSG_EOR, and how to
> distinguish between EOF and zero-length messages in a SOCK_SEQPACKET
> connection, I would definitely appreciate it!
>
> Thanks,
> Sam Kumar
That would be my expectation of how it should work - if you ignore 
MSG_EOR on recvmsg then what you get is identical to a SOCK_STREAM, and 
that every call to recvmsg will return data from (at most) a single 
message, with MSG_EOR set if the end of that message has been reached. 
That is what POSIX says should happen anyway.

I do wonder if the man page for recvmsg is wrong, or at least a bit 
confusing. SOCK_SEQPACKET is stream based not message based - it just 
happens to have EOR markers in the stream. There is no reason that the 
whole message needs to be returned in a single read, and in fact that 
would be impossible if the sender didn't insert any EOR markers but kept 
sending data beyond the size that the socket could buffer.

I notice that man 7 socket says SOCK_SEQPACKET is for datagrams of fixed 
maximum length which is definitely wrong, as is the statement that a 
consumer has to read an entire packet with each system call.

Also it is probably better to ask this question on netdev where it is 
likely to get more attention from the net developers, so I'm copying my 
reply there too,

Steve.

Re: Question about SOCK_SEQPACKET

[Sowmini Varadhan]

On (05/05/17 10:45), Steven Whitehouse wrote:
> 
> I do wonder if the man page for recvmsg is wrong, or at least a bit
> confusing. SOCK_SEQPACKET is stream based not message based - it just
> happens to have EOR markers in the stream. There is no reason that the whole
> message needs to be returned in a single read, and in fact that would be
> impossible if the sender didn't insert any EOR markers but kept sending data
> beyond the size that the socket could buffer.
> 
> I notice that man 7 socket says SOCK_SEQPACKET is for datagrams of fixed
> maximum length which is definitely wrong, as is the statement that a
> consumer has to read an entire packet with each system call.

Which man page do you think is wrong here? The POSIX definition is here

http://pubs.opengroup.org/onlinepubs/009695399/functions/recvmsg.html

The description in

http://pubs.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_10.html

says, "It is protocol-specific whether a maximum record size is imposed."
In my machine (Ubuntu 4.4.0-72, and it is in socket(2), not socket(7), btw)
doesnt have any references to max length, but I'm not sure I'd boldly assert
"definitely wrong" about the requirement of having to read entire
packet in a system call (see POSIX man page)

--Sowmini

Re: Question about SOCK_SEQPACKET

[Steven Whitehouse]

Hi,


On 05/05/17 11:09, Sowmini Varadhan wrote:
> On (05/05/17 10:45), Steven Whitehouse wrote:
>> I do wonder if the man page for recvmsg is wrong, or at least a bit
>> confusing. SOCK_SEQPACKET is stream based not message based - it just
>> happens to have EOR markers in the stream. There is no reason that the whole
>> message needs to be returned in a single read, and in fact that would be
>> impossible if the sender didn't insert any EOR markers but kept sending data
>> beyond the size that the socket could buffer.
>>
>> I notice that man 7 socket says SOCK_SEQPACKET is for datagrams of fixed
>> maximum length which is definitely wrong, as is the statement that a
>> consumer has to read an entire packet with each system call.
> Which man page do you think is wrong here? The POSIX definition is here
>
> http://pubs.opengroup.org/onlinepubs/009695399/functions/recvmsg.html
>
> The description in
>
> http://pubs.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_10.html
>
> says, "It is protocol-specific whether a maximum record size is imposed."
> In my machine (Ubuntu 4.4.0-72, and it is in socket(2), not socket(7), btw)
> doesnt have any references to max length, but I'm not sure I'd boldly assert
> "definitely wrong" about the requirement of having to read entire
> packet in a system call (see POSIX man page)
>
> --Sowmini
>
Just before the part that you've quoted, the description for 
SOCK_SEQPACKET says:
"The SOCK_SEQPACKET socket type is similar to the SOCK_STREAM type, and 
is also connection-oriented. The only difference between these types is 
that record boundaries are maintained using the SOCK_SEQPACKET type. A 
record can be sent using one or more output operations and received 
using one or more input operations, but a single operation never 
transfers parts of more than one record."

The man page for socket says SOCK_SEQPACKET "Provides  a sequenced,  
reliable,  two-way connection-based data transmission path  for  
datagrams  of  fixed maximum  length" which is not true, because while 
there may be a length restriction, it is quite possible that there is 
not a length restriction (as per DECnet). It also says "a  consumer  is  
required  to read an entire packet with each input system call" which is 
also contradicted by POSIX which says that a record can be "received 
using one or more input operations". So both statements in the man page 
are wrong, I think.

I have to say that I'd not spotted the POSIX recvmsg wording before, 
which says "For message-based sockets, such as SOCK_DGRAM and 
SOCK_SEQPACKET, the entire message shall be read in a single operation" 
however that does contradict the earlier wording, where it explicitly 
says that multiple receive operations per record are ok for 
SOCK_SEQPACKET - at least if we assume that record == message in this 
case. Also, if this restriction was true (one message per recvmsg call) 
then MSG_EOR would never be needed on receive, since every recvmsg would 
be a single message/record only, and that same document does say that 
MSG_EOR can be set on receive for protocols which support it,

Steve.

RE: Question about SOCK_SEQPACKET

[David Laight]

From: Steven Whitehouse
> Sent: 05 May 2017 11:34
...
> Just before the part that you've quoted, the description for
> SOCK_SEQPACKET says:
> "The SOCK_SEQPACKET socket type is similar to the SOCK_STREAM type, and
> is also connection-oriented. The only difference between these types is
> that record boundaries are maintained using the SOCK_SEQPACKET type. A
> record can be sent using one or more output operations and received
> using one or more input operations, but a single operation never
> transfers parts of more than one record."

Right SOCK_SEQPACKET is for protocols like ISO transport.
There is no limit on the length of a 'record'.
I've written file transfer programs that put the entire file
data into a single 'record'. The receiver disconnected on
receipt of the 'end of record'.

The socket man pages could easily be wrong - they are very IP-centric.
Remember ISO transport use declined when Unix became more popular
back in the mid 1980s. Unix sockets have never really been used for
it - the address information needed just doesn't match that IP
(especially IPv4).

	David