Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[inetstack] Enhancement: Add receiver to arch #1460

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

[inetstack] Enhancement: Add receiver to arch #1460

wants to merge 1 commit into from
+885 −890

Conversation

iyzhang
Copy link
Contributor

@iyzhang iyzhang commented Nov 15, 2024

This PR adds a receiver module for the receive path on established sockets. It also removes two SharedAsyncQueues to reduce latency.

@github-actions GitHub Actions
Copy link

libos = catpowder
commit id = 7c1411a

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 18989859.53 687.76 299.35 84.90
bgc::inetstack::poll_recv 9494929.77 1633.04 710.50 54.11
bgc::inetstack::tcp::established::background 12720.57 1538.42 683.54 0.19
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 12720.57 431.21 195.16 22.27
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 12712.39 362.65 159.63 28.99
bgc::inetstack::tcp::established::background;tcp::established::background::sender 12670.48 853.52 373.87 24.69
ioc::network::libos::pop 9831.51 396.61 173.74 0.04
demikernel::sgafree 8861.32 103.07 44.86 0.02
ioc::network::libos::push;inetstack::push 8762.80 5941.64 2691.28 2.23
ioc::network::libos::push 8073.74 33426348.04 21882229.38 40.53
demikernel::push 8073.74 554.73 249.66 0.05
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 6918.47 2450.94 1067.03 1.79
demikernel::sgaalloc 6662.33 130.73 60.50 0.03
demikernel::pop 6077.65 548.14 240.28 0.04
ioc::network::libos::pushto 2671.33 4177.36 1781.52 7.92
ioc::network::libos::pushto;inetstack::push 2671.33 3734.01 1592.43 89.38
demikernel::pushto 2671.33 468.51 199.39 0.84
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 953.67 185.56 78.71 60.24
ioc::network::libos::close 47.30 6317.09 2829.60 0.10
ioc::network::libos::connect 41.86 38496.02 17467.20 0.47
bgc::inetstack::tcp::passiveopen::background 41.86 32741.05 14036.07 0.01
ioc::network::libos::accept 38.29 2205.36 947.31 0.00
demikernel::async_close 32.50 798.92 345.27 0.01
demikernel::accept 21.86 1507.61 640.47 0.00
demikernel::connect 20.93 4085.22 1931.13 0.02
demikernel::socket 10.30 2486.75 1083.30 0.00
bgc::inetstack::arp::background 6.38 2002.81 868.12 0.00
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 6668.17 2921.49 84.60
bgc::inetstack::icmp::background 1.03 3123.83 1359.14 0.00
demikernel::new 1.00 94279259.33 41565167.94 19.13
demikernel::bind 1.00 25741.81 10947.48 0.00
bgc::inetstack::poll_recv;inetstack::poll 1.00 15676.83 6865.99 0.03
demikernel::listen 1.00 4379.36 1850.26 0.00
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnip
commit id = 7c1411a

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 29568771.50 337.07 144.80 76.07
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 29568771.50 197.16 84.71 58.58
bgc::inetstack::poll_recv 14784385.75 892.77 383.77 35.15
bgc::inetstack::tcp::established::background 11397.64 1104.66 465.72 0.11
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11397.64 311.04 129.14 25.15
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11387.30 347.12 149.12 32.62
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11345.18 313.17 126.68 15.31
ioc::network::libos::pop 9234.51 387.26 165.88 0.02
demikernel::sgafree 8784.23 81.75 34.98 0.01
ioc::network::libos::push;inetstack::push 8762.80 1993.68 885.57 0.68
ioc::network::libos::push;inetstack::push;catnip::runtime::transmit 8762.80 327.91 144.34 15.64
ioc::network::libos::push 8073.74 33386261.46 13186929.74 25.63
demikernel::push 8073.74 703.49 302.80 0.04
demikernel::sgaalloc 6540.46 125.81 53.74 0.01
demikernel::pop 6054.59 564.38 239.88 0.02
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4791.80 2402.61 1004.63 2.14
ioc::network::libos::pushto 1309.33 1324.93 567.08 0.05
ioc::network::libos::pushto;inetstack::push 1309.33 705.32 302.10 53.43
demikernel::pushto 1309.33 483.70 207.32 0.02
ioc::network::libos::pushto;inetstack::push;catnip::runtime::transmit 1309.33 260.55 111.48 36.71
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 569.67 263.89 110.57 71.67
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;catnip::runtime::transmit 361.88 636.70 274.18 3.68
ioc::network::libos::close 49.43 6075.75 2599.79 0.01
bgc::inetstack::tcp::passiveopen::background 41.86 34128.19 14928.70 0.01
ioc::network::libos::connect 41.86 31108.53 13447.66 0.04
ioc::network::libos::connect;catnip::runtime::transmit 41.86 6706.65 2992.49 22.76
ioc::network::libos::accept 38.18 2235.82 975.44 0.00
demikernel::async_close 32.50 690.24 274.34 0.00
demikernel::accept 21.86 1180.15 514.90 0.00
bgc::inetstack::tcp::passiveopen::background;catnip::runtime::transmit 20.96 15641.37 6828.02 16.65
demikernel::connect 20.93 2679.78 1180.57 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::sender;catnip::runtime::transmit 20.93 606.31 262.16 11.38
demikernel::socket 10.30 2042.28 895.23 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter;catnip::runtime::transmit 10.00 279.93 124.95 0.01
bgc::inetstack::arp::background 6.43 2235.30 963.79 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger;catnip::runtime::transmit 5.00 2266.56 845.82 20.49
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 21431.18 9197.90 95.67
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 2.00 19239.08 8254.60 89.48
bgc::inetstack::icmp::background 1.05 3213.00 1383.21 0.00
demikernel::new 1.00 1760397564.53 756324865.95 48.40
bgc::inetstack::poll_recv;inetstack::poll 1.00 44757.40 19209.65 0.16
demikernel::bind 1.00 10475.38 4536.44 0.00
demikernel::listen 1.00 6128.71 2744.62 0.00
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnap
commit id = 7c1411a

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::catnap::transport::epoll 13196020.95 2024.32 866.64 33.20
demikernel::sgafree 11608.02 73.61 31.37 0.05
demikernel::sgaalloc 11416.38 129.62 55.11 0.11
ioc::network::libos::push;catnap::linux::transport::push 9066.10 665.74 292.07 4.21
ioc::network::libos::push 8800.48 16475.20 7230.43 6.95
demikernel::push 8800.48 543.87 236.98 0.16
ioc::network::libos::pop;catnap::linux::transport::pop 8051.95 390.41 168.40 2.70
ioc::network::libos::pop 7842.74 3187542.61 1357689.40 24.41
demikernel::pop 7842.74 574.80 246.03 0.12
ioc::network::libos::pushto 3413.33 10126.32 4180.13 24.64
ioc::network::libos::pushto;catnap::linux::transport::push 3413.33 719.65 298.73 6.68
demikernel::pushto 3413.33 658.65 272.91 1.81
demikernel::async_close 32.50 1068.42 423.62 0.01
ioc::network::libos::connect 24.95 6584331.49 2766479.87 27.76
ioc::network::libos::connect;catnap::linux::transport::connect 24.95 79597.19 33580.70 2.42
demikernel::connect 24.95 2200.16 935.29 0.01
ioc::network::libos::close 18.35 28797.41 12219.88 0.06
ioc::network::libos::close;catnap::linux::transport::close 18.35 27629.45 11716.86 95.28
demikernel::accept 15.42 1856.81 797.19 0.00
ioc::network::libos::accept 15.39 880915459.12 374816867.60 44.50
ioc::network::libos::accept;catnap::linux::transport::accept 15.39 7251.32 3207.48 0.00
demikernel::socket 9.50 160443.46 68997.17 0.22
demikernel::socket;catnap::linux::transport::socket 9.50 157940.67 67917.19 96.28
demikernel::new 1.00 264691.40 113521.02 0.06
demikernel::bind 1.00 30611.48 13219.58 0.00
demikernel::bind;catnap::linux::transport::bind 1.00 26065.15 11252.57 84.69
demikernel::listen 1.00 15821.82 6854.78 0.00
demikernel::listen;catnap::linux::transport::listen 1.00 14665.21 6354.25 92.64
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnip
commit id = 6554338

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 29310911.53 337.88 146.54 76.08
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 29310911.53 197.54 85.66 58.55
bgc::inetstack::poll_recv 14655455.77 893.81 388.11 34.73
bgc::inetstack::tcp::established::background 11055.02 1076.48 463.36 0.11
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11055.02 309.09 133.90 25.11
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11044.62 338.48 147.33 32.63
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11002.55 301.35 122.41 15.31
ioc::network::libos::pop 9046.41 382.19 166.41 0.02
demikernel::sgafree 8781.75 83.82 36.69 0.01
ioc::network::libos::push;inetstack::push 8762.80 1959.55 880.27 0.68
ioc::network::libos::push;inetstack::push;catnip::runtime::transmit 8762.80 308.86 138.56 15.20
ioc::network::libos::push 8073.74 33352098.83 14805821.11 25.51
demikernel::push 8073.74 660.74 292.30 0.04
demikernel::sgaalloc 6537.38 117.70 51.38 0.01
demikernel::pop 6052.88 535.39 231.60 0.02
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4565.98 2329.43 986.24 2.04
ioc::network::libos::pushto 1265.67 1322.88 614.77 0.05
ioc::network::libos::pushto;inetstack::push 1265.67 707.90 329.27 53.67
demikernel::pushto 1265.67 484.90 226.03 0.02
ioc::network::libos::pushto;inetstack::push;catnip::runtime::transmit 1265.67 260.78 121.05 36.64
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 557.33 231.25 104.84 69.60
ioc::network::libos::close 49.53 6019.78 2612.51 0.01
bgc::inetstack::tcp::passiveopen::background 41.86 33038.77 14747.70 0.00
ioc::network::libos::connect 41.86 32819.50 14465.75 0.04
ioc::network::libos::connect;catnip::runtime::transmit 41.86 6679.72 2938.15 22.02
ioc::network::libos::accept 38.25 2230.75 983.40 0.00
demikernel::async_close 32.50 622.95 251.83 0.00
demikernel::accept 21.86 1086.93 475.49 0.00
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;catnip::runtime::transmit 21.12 613.98 265.99 3.28
bgc::inetstack::tcp::passiveopen::background;catnip::runtime::transmit 20.93 16253.08 7248.29 17.74
demikernel::connect 20.93 2639.30 1156.27 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::sender;catnip::runtime::transmit 20.93 642.20 278.35 11.86
demikernel::socket 10.30 1957.02 859.63 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger;catnip::runtime::transmit 9.00 3750.67 1658.13 35.87
bgc::inetstack::arp::background 6.43 2321.45 1006.25 0.00
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 20846.52 9072.75 95.41
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once;catnip::runtime::receive 2.00 18614.73 8102.43 89.06
bgc::inetstack::icmp::background 1.05 3081.23 1335.83 0.00
demikernel::new 1.00 1762922056.28 763591650.88 48.89
bgc::inetstack::poll_recv;inetstack::poll 1.00 43646.93 18990.29 0.29
demikernel::bind 1.00 10222.62 4446.92 0.00
demikernel::listen 1.00 3142.50 1356.24 0.00
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter;catnip::runtime::transmit 1.00 318.00 145.30 0.00
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catpowder
commit id = 6554338

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::inetstack::poll_recv;inetstack::layer4_endpoint::poll_once 18978938.10 695.04 306.09 85.03
bgc::inetstack::poll_recv 9489469.05 1648.27 725.78 53.46
bgc::inetstack::tcp::established::background 11950.54 1595.48 705.65 0.20
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 11950.54 473.96 216.76 21.84
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 11941.68 366.67 161.78 29.26
bgc::inetstack::tcp::established::background;tcp::established::background::sender 11899.73 881.58 376.55 25.13
ioc::network::libos::pop 9267.53 430.33 190.73 0.04
demikernel::sgafree 8764.89 108.77 48.88 0.01
ioc::network::libos::push;inetstack::push 8762.80 5776.10 2567.46 2.23
ioc::network::libos::push 8073.74 33462834.36 14924605.84 40.45
demikernel::push 8073.74 563.44 249.45 0.06
demikernel::sgaalloc 6537.00 137.40 60.52 0.02
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 6139.08 2473.20 1072.56 1.70
demikernel::pop 6036.57 560.44 248.24 0.04
ioc::network::libos::pushto 967.67 4335.33 1923.48 2.68
ioc::network::libos::pushto;inetstack::push 967.67 3865.61 1715.12 89.17
demikernel::pushto 967.67 469.46 208.39 0.28
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 215.00 278.33 124.70 66.15
ioc::network::libos::close 47.98 5845.77 2570.81 0.09
bgc::inetstack::tcp::passiveopen::background 41.86 35692.58 15684.29 0.01
ioc::network::libos::connect 41.86 34933.02 15381.93 0.42
ioc::network::libos::accept 38.43 2234.95 981.12 0.00
demikernel::async_close 32.50 743.36 315.13 0.01
demikernel::accept 21.86 1520.88 670.93 0.00
demikernel::connect 20.93 3694.53 1644.55 0.02
demikernel::socket 10.30 2164.54 951.69 0.00
bgc::inetstack::arp::background 6.38 2077.15 915.33 0.00
bgc::inetstack::poll_recv;inetstack::poll;inetstack::layer4_endpoint::poll_once 2.00 6473.45 2855.21 83.81
bgc::inetstack::icmp::background 1.03 3016.15 1326.18 0.00
demikernel::new 1.00 97695638.48 42907904.30 20.16
demikernel::bind 1.00 26209.12 11392.59 0.00
bgc::inetstack::poll_recv;inetstack::poll 1.00 15319.60 6753.16 0.05
demikernel::listen 1.00 3820.50 1681.75 0.00
demikernel::close 0.00 nan nan 0.00

@github-actions GitHub Actions
Copy link

libos = catnap
commit id = 6554338

collapsed_stack num_calls cycles_per_call nanoseconds_per_call percent_time
bgc::catnap::transport::epoll 13338154.85 1963.17 850.64 32.19
demikernel::sgafree 11534.95 71.63 30.81 0.04
demikernel::sgaalloc 11281.95 118.33 50.66 0.10
ioc::network::libos::push;catnap::linux::transport::push 9066.06 633.03 278.56 4.02
ioc::network::libos::push 8800.46 16438.42 7223.22 5.89
demikernel::push 8800.46 523.25 228.84 0.14
ioc::network::libos::pop;catnap::linux::transport::pop 8024.12 391.28 170.24 1.38
ioc::network::libos::pop 7815.63 20850433.62 11025240.69 26.29
demikernel::pop 7815.63 561.58 242.20 0.11
ioc::network::libos::pushto 564.33 8606.18 4071.13 23.43
demikernel::pushto 564.33 468.11 218.72 1.41
ioc::network::libos::pushto;catnap::linux::transport::push 564.33 462.16 215.94 5.52
demikernel::async_close 32.50 948.62 371.21 0.01
ioc::network::libos::connect 24.95 7210094.44 3046286.54 27.16
ioc::network::libos::connect;catnap::linux::transport::connect 24.95 77580.65 33054.87 2.63
demikernel::connect 24.95 1936.00 835.21 0.01
ioc::network::libos::close 18.35 24507.89 10699.76 0.06
ioc::network::libos::close;catnap::linux::transport::close 18.35 23194.99 10118.60 94.64
demikernel::accept 15.42 1593.19 701.09 0.00
ioc::network::libos::accept 15.39 877764244.19 387525007.79 44.87
ioc::network::libos::accept;catnap::linux::transport::accept 15.39 4766.29 2154.72 0.00
demikernel::socket 9.50 155290.38 68290.43 0.33
demikernel::socket;catnap::linux::transport::socket 9.50 152801.73 67201.50 96.30
demikernel::new 1.00 265248.09 114859.91 0.13
demikernel::bind 1.00 32411.08 14246.41 0.02
demikernel::bind;catnap::linux::transport::bind 1.00 27436.55 12063.07 84.34
demikernel::listen 1.00 16851.74 7367.82 0.00
demikernel::listen;catnap::linux::transport::listen 1.00 15726.89 6874.86 93.24
demikernel::close 0.00 nan nan 0.00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anandbonde anandbonde Awaiting requested review from anandbonde

At least 1 approving review is required to merge this pull request.

Assignees

@iyzhang iyzhang

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@iyzhang