Add a flag to use write(2) on the connected socket instead of send(2).

[moritzbuhl]

No description provided.

[moritzbuhl]

the -m usage does not add (2) after the syscall names.

[bluhm]

Is there any indication that write(2) does something different than send(2)? In OpenBSD the implementation is the same. Why do you want to test this?

[moritzbuhl]

My patch for udpbench aimed to investigate the performance differences between iperf3 and udpbench on Linux. The tests revealed that the write system call results in performance closer to that of iperf3. With the patch, I found that the difference in the number of UDP datagrams sent was about 3% with write(2) and about 6% with send(2).
With strace, the difference to iperf3 for write(2) is smaller and send(2) is slower with strace.
The results suggest that the VFS indirection of the write(2) system call has a significant impact on performance.

https://mbuhl.me/t0.iperf-b0.svg
https://mbuhl.me/t0.udpbench.svg

Here are my notes on the experiments I did, I would appreciate it if you could approach the data critically and provide honest feedback.

nstat

iperf3 3.17.1

root@t0:/home/mbuhl/src/iperf# nstat -r; ./src/iperf3 -c 10.0.1.11 -u -b 0 -l 1472; nstat

...

warning: UDP block size 1472 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 10.0.1.11, port 5201
[ 5] local 10.0.1.10 port 49544 connected to 10.0.1.11 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 422 MBytes 3.54 Gbits/sec 300899
[ 5] 1.00-2.00 sec 425 MBytes 3.56 Gbits/sec 302448
[ 5] 2.00-3.00 sec 424 MBytes 3.56 Gbits/sec 301941
[ 5] 3.00-4.00 sec 424 MBytes 3.56 Gbits/sec 302043
[ 5] 4.00-5.00 sec 424 MBytes 3.56 Gbits/sec 302175
[ 5] 5.00-6.00 sec 429 MBytes 3.60 Gbits/sec 305797
[ 5] 6.00-7.00 sec 430 MBytes 3.61 Gbits/sec 306344
[ 5] 7.00-8.00 sec 432 MBytes 3.62 Gbits/sec 307674
[ 5] 8.00-9.00 sec 432 MBytes 3.62 Gbits/sec 307675
[ 5] 9.00-10.00 sec 432 MBytes 3.62 Gbits/sec 307784

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 4.17 GBytes 3.59 Gbits/sec 0.000 ms 0/3044780 (0%) sender
[ 5] 0.00-10.00 sec 4.17 GBytes 3.58 Gbits/sec 0.001 ms 3144/3044780 (0.1%) receiver
iperf Done.
#kernel
IpInReceives 25 0.0
IpInDelivers 25 0.0
IpOutRequests 3044809 0.0
TcpActiveOpens 1 0.0
TcpInSegs 24 0.0
TcpOutSegs 30 0.0
UdpInDatagrams 1 0.0
UdpOutDatagrams 3044781 0.0
TcpExtTW 1 0.0
TcpExtDelayedACKs 2 0.0
TcpExtTCPHPHits 2 0.0
TcpExtTCPPureAcks 11 0.0
TcpExtTCPHPAcks 1 0.0
TcpExtTCPRcvCoalesce 2 0.0
TcpExtTCPAutoCorking 1 0.0
TcpExtTCPOrigDataSent 20 0.0
TcpExtTCPDelivered 22 0.0
IpExtInOctets 1783 0.0
IpExtOutOctets 4567179563 0.0
IpExtInNoECTPkts 25 0.0

udpbench

root@t0:/home/mbuhl/src/iperf# nstat -r; ../udpbench/udpbench -l 1472 -t 10 send 10.0.1.11; nstat

...

sockname: 10.0.1.10 51374
send: syscalls 3216505, packets 3216505, frame 1, payload 1472, ip 1500, ether 1538, begin 1733918719.718117, end 1733918729.718102, duration 9.999985, bit/s 3.957594e+09, start 1733918719.718075, stop 1733918729.718117
#kernel
IpInReceives 7 0.0
IpInDelivers 7 0.0
IpOutRequests 3216513 0.0
TcpInSegs 7 0.0
TcpOutSegs 8 0.0
UdpOutDatagrams 3216505 0.0
TcpExtDelayedACKs 1 0.0
TcpExtTCPPureAcks 3 0.0
TcpExtTCPHPAcks 1 0.0
TcpExtTCPOrigDataSent 6 0.0
TcpExtTCPDelivered 7 0.0
IpExtInOctets 504 0.0
IpExtOutOctets 4824762900 0.0
IpExtInNoECTPkts 7 0.0

udpbench -w

root@t0:/home/mbuhl/src/iperf# nstat -r; ../udpbench/udpbench -w -l 1472 -t 10 send 10.0.1.11; nstat

...

sockname: 10.0.1.10 48918
send: syscalls 3117311, packets 3117311, frame 1, payload 1472, ip 1500, ether 1538, begin 1733998994.151298, end 1733999004.151235, duration 9.999937, bit/s 3.835564e+09, start 1733998994.151115, stop 1733999004.151247
#kernel
IpInReceives 12 0.0
IpInDelivers 12 0.0
IpOutRequests 3117324 0.0
TcpInSegs 12 0.0
TcpOutSegs 15 0.0
UdpOutDatagrams 3117311 0.0
TcpExtDelayedACKs 1 0.0
TcpExtTCPPureAcks 8 0.0
TcpExtTCPHPAcks 1 0.0
TcpExtTCPAutoCorking 1 0.0
TcpExtTCPOrigDataSent 13 0.0
TcpExtTCPDelivered 14 0.0
IpExtInOctets 772 0.0
IpExtOutOctets 4675982048 0.0
IpExtInNoECTPkts 12 0.0

udpbench -m 1024

root@t0:/home/mbuhl/src/iperf# nstat -r; ../udpbench/udpbench -m 1024 -l 1472 -t 10 send 10.0.1.11; nstat

...

sockname: 10.0.1.10 45221
send: syscalls 3793, packets 3884032, frame 1, payload 1472, ip 1500, ether 1538, begin 1733919184.256505, end 1733919194.254556, duration 9.998051, bit/s 4.779845e+09, start 1733919184.252463, stop 1733919194.254672
#kernel
IpInReceives 4 0.0
IpInDelivers 4 0.0
IpOutRequests 3884038 0.0
TcpInSegs 4 0.0
TcpOutSegs 6 0.0
UdpOutDatagrams 3884032 0.0
TcpExtTCPPureAcks 3 0.0
TcpExtTCPHPAcks 1 0.0
TcpExtTCPAutoCorking 1 0.0
TcpExtTCPOrigDataSent 6 0.0
TcpExtTCPDelivered 5 0.0
IpExtInOctets 208 0.0
IpExtOutOctets 5826053188 0.0
IpExtInNoECTPkts 4 0.0

Calculations

IpOutRequests
iperf3 3.17.1: 3044809
udpbench: 3216513
udpbench -m 1024: 3884038 
            3044809 / 3216513 (iperf3 / upbench)
            0.9466179679671743
            3216513 / 3884038 (udpbench / udpbench -m 1024)
            0.8281363364622076

            3044781 / 3117311 (iperf3 / udpbench -w)
            0.9767331523867846

Throughput
            3.58 / 3.957594 (iperf3 / udpbench)
            0.9045900110016338
            3.957594 / 4.779845 (udpbench / udpbench -m 1024)
            0.8279753841390254

            3.58 / 3.835564 (iperf3 / udpbench -w)
            0.9333699033571073

strace

udpbench -m 1024

mbuhl@t0:~/src/udpbench$ strace -ff -o /tmp/udpbench.strace ./udpbench -m 1024 -l 1472 -t 10 send 10.0.1.11
sockname: 10.0.1.10 39933
send: syscalls 3086, packets 3160064, frame 1, payload 1472, ip 1500, ether 1538, begin 1733920141.798554, end 1733920151.803985, duration 10.005431, bit/s 3.886032e+09, start 1733920141.790214, stop 1733920151.804340
udpbench -w

mbuhl@t0:~/src/udpbench$ strace -ff -o /tmp/udpbench-w.strace ./udpbench -w -l 1472 -t 10 send 10.0.1.11
sockname: 10.0.1.10 36017
send: syscalls 378819, packets 378819, frame 1, payload 1472, ip 1500, ether 1538, begin 1733998574.138913, end 1733998584.138341, duration 9.999428, bit/s 4.661256e+08, start 1733998574.137871, stop 1733998584.138354

udpbench

mbuhl@t0:~/src/udpbench$ strace -ff -o /tmp/udpbench.strace ./udpbench -l 1472 -t 10 send 10.0.1.11
sockname: 10.0.1.10 35880
send: syscalls 326474, packets 326474, frame 1, payload 1472, ip 1500, ether 1538, begin 1733920191.316505, end 1733920201.325738, duration 10.009233, bit/s 4.013231e+08, start 1733920191.315841, stop 1733920201.325750

iperf3

root@t0:/home/mbuhl/src/iperf# strace -ff -o /tmp/iperf3.strace ./src/iperf3 -c 10.0.1.11 -u -b 0 -l 1472
warning: UDP block size 1472 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 10.0.1.11, port 5201
[ 5] local 10.0.1.10 port 35018 connected to 10.0.1.11 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 52.0 MBytes 435 Mbits/sec 37011
[ 5] 1.00-2.00 sec 53.1 MBytes 446 Mbits/sec 37838
[ 5] 2.00-3.00 sec 53.0 MBytes 444 Mbits/sec 37734
[ 5] 3.00-4.00 sec 52.8 MBytes 443 Mbits/sec 37629
[ 5] 4.00-5.00 sec 52.6 MBytes 441 Mbits/sec 37460
[ 5] 5.00-6.00 sec 51.8 MBytes 434 Mbits/sec 36884
[ 5] 6.00-7.00 sec 52.5 MBytes 440 Mbits/sec 37376
[ 5] 7.00-8.00 sec 53.2 MBytes 446 Mbits/sec 37871
[ 5] 8.00-9.00 sec 52.9 MBytes 444 Mbits/sec 37688
[ 5] 9.00-10.00 sec 50.5 MBytes 424 Mbits/sec 35999

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 524 MBytes 440 Mbits/sec 0.000 ms 0/373490 (0%) sender
[ 5] 0.00-10.00 sec 524 MBytes 440 Mbits/sec 0.024 ms 0/373490 (0%) receiver

iperf Done.

Evaluation

mbuhl@t0:/tmp$ grep ^sendto udpbench.strace.94551 | wc -l
326474 # 326474 * 1472 = 480569728, 480569728 / 10 * 8 = 384 Mbps
mbuhl@t0:/tmp$ grep ^write iperf3.strace.94514 | wc -l
373490 # 373490 * 1472 = 549777280, 549777280 / 10 * 8 = 440 Mbps
mbuhl@t0:/tmp$ grep ^sendmmsg udpbench-m1024.strace.94528 | wc -l
3086 # 3086 * 1024 * 1472 = 4651614208, 4651614208 / 10 * 8 = 3721 Mbps
mbuhl@t0:/tmp$ grep ^write udpbench-w.strace.102460 | wc -l
378821 # 378821 * 1472 = 557624512, 557624512 / 10 * 8 = 446 Mbps

            326474 / 373490 (udpbench / iperf3)
            0.8741171115692522

            373490 / 378821 (iperf3 / udpbench -w)
            0.9859273905089739
            => write(2) gives similar performance

            4.013231 / 4.4 (udpbench throughput / iperf3 throughput? not sure)
            0.9120979545454545

Conclusion

using write(2) instead of send(2) causes the throughput values of
udpbench and iperf3 to be very similar.
While using strace(1) on the throughput tests, we see that send(2)
gets a lot slower than write(2) and that both write(2) using runs
yield similar results.

[bluhm]

This has been resolved and merged in pull request #66.