\chapter{\label{appendixminus1}Resources and code repositories} The following sections describe how to acquire the resources to reproduce the test results. All compilations were made on Ubuntu 16.04 with kernels \begin{itemize} \item 4.15.0-54-generic (Supporting Desktop), \item 4.4.0-143-generic (BMV2 test VM) \item 4.15.0-55-generic (Desktop with NetFPGA \end{itemize} \section{\label{chapterminus1:thesis}Master Thesis} The master thesis including all self developed source code is available by git via \begin{verbatim} git clone git@gitlab.ethz.ch:nicosc/master-thesis.git \end{verbatim} It can be browsed online on \url{https://gitlab.ethz.ch/nicosc/master-thesis}. % ---------------------------------------------------------------------- \section{\label{chapterminus1:thesis}Xilinx Toolchain} A prerequisite for building the NetFGPA source code is the installation of \begin{itemize} \item \verb=Xilinx_SDNet_2018.2_1005_9= \item \verb=Xilinx_Vivado_SDK_2018.2_0614_1954= \end{itemize} Both tools need to be installed to /opt/Xilinx/, as paths are hardcoded in various places. % ---------------------------------------------------------------------- \section{\label{chapterminus1:thesis}NetFGPA support scripts} To be able to compile P4 source code to the NetFPGA the collection of scripts, Makefiles and sample code of P4-NetFGPA is required. The repository \url{git@github.com:NetFPGA/P4-NetFPGA-live.git} needs to be cloned to ``projects'' subdirectory as ``P4-NetPFGA'' of the user that wants to compile the source code. Access to the repository is granted after applying for access as described on \url{https://github.com/NetFPGA/P4-NetFPGA-public/wiki}. After that the variable \verb=P4_PROJECT_NAME= in \texttt{~/projects/P4-NetFPGA/tools/settings.sh} needs to be modified to read \verb|export P4_PROJECT_NAME=minip4| instead of \verb|export P4_PROJECT_NAME=switch_calc|. Sample code for installation: \begin{verbatim} mkdir -p ~/projects git clone git@github.com:NetFPGA/P4-NetFPGA-live.git P4-NetFPGA sed -i 's/\(P4_PROJECT_NAME=\).*/\1minip4/' ~/projects/P4-NetFPGA/tools/settings.sh \end{verbatim} % ---------------------------------------------------------------------- \chapter{\label{appendix0}BMV2 environment and tests} All BMV2 based compilations were made with the following compiler: \begin{verbatim} p4@ubuntu:~$ p4c --version p4c 0.5 (SHA: 5ae30ee) \end{verbatim} The installation is based on the vagrant files that were provided in the ``Advanced Topics in Communication Networks Fall 2018'' course of ETHZ (\url{https://adv-net.ethz.ch/2018/}) and contains p4tools as well as all utilities that came with the vagrant installation. \section{\label{chapter0:bmv2-diff}Diff based checksumming} For running the diff based checksum code, the following steps are necessary: Compiling the p4 code and starting the switch: \begin{verbatim} cd ~/master-thesis/p4app sudo p4run --config nat64-diff.json \end{verbatim} Starting the controller which sets up the required table entries: \begin{verbatim} cd ~/master-thesis/p4app sudo python ./controller.py --mode range_router \end{verbatim} % ---------------------------------------------------------------------- \chapter{\label{appendixA}NetFPGA environment and tests} \section{\label{chapterA:netpfga-setup}NetFPGA Setup} Description of installation, commit of netpfga-live \section{\label{chapterA:section1}NetFPGA NAT64 Test cases} todo: add graphic of nsg <-> esprimo cabling \begin{verbatim} ip addr add 10.0.0.42/24 dev enp2s0f0 # Adding necessary ARP entries: for the virtual IPv4 address(es) ip neigh add 10.0.0.6 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 ip neigh add 10.0.0.42 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 \end{verbatim} For all test cases the following network settings on esprimo: \begin{verbatim} 12: enp2s0f0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f8:f2:1e:09:62:d0 brd ff:ff:ff:ff:ff:ff inet 10.0.0.42/24 scope global enp2s0f0 valid_lft forever preferred_lft forever inet6 fe80::faf2:1eff:fe09:62d0/64 scope link valid_lft forever preferred_lft forever 13: enp2s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff inet6 2001:db8:42::42/64 scope global valid_lft forever preferred_lft forever inet6 fe80::faf2:1eff:fe09:62d1/64 scope link valid_lft forever preferred_lft forever \end{verbatim} \subsection{Test 1: IPv4 egress settings work} Scenario: simple egress port setting for the IPv4 addresses Step 1: getting correct values for table entries from python: \begin{verbatim} >>> int(ipaddress.IPv4Address(u"10.0.0.42")) 167772202 >>> int(ipaddress.IPv4Address(u"10.0.0.4")) 167772164 >>> \end{verbatim} Step 2: setting table netries \begin{verbatim} >> table_cam_add_entry realmain_v4_networks_0 realmain.set_egress_port 167772202 => 16 0 0 0 0 fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)] action_name = TopPipe.realmain.set_egress_port field_vals = [1, '16', '0', '0', '0', '0'] CAM_Init_ValidateContext() - done WROTE 0x44020250 = 0xa00002a WROTE 0x44020280 = 0x0000 WROTE 0x44020284 = 0x0000 WROTE 0x44020288 = 0x10000000 WROTE 0x4402028c = 0x0001 READ 0x44020244 = 0x0001 WROTE 0x44020240 = 0x0001 READ 0x44020244 = 0x0001 READ 0x44020244 = 0x0001 success >> table_cam_add_entry realmain_v4_networks_0 realmain.set_egress_port 167772164 => 16 0 0 0 0 fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)] action_name = TopPipe.realmain.set_egress_port field_vals = [1, '16', '0', '0', '0', '0'] CAM_Init_ValidateContext() - done WROTE 0x44020250 = 0xa000004 WROTE 0x44020280 = 0x0000 WROTE 0x44020284 = 0x0000 WROTE 0x44020288 = 0x10000000 WROTE 0x4402028c = 0x0001 READ 0x44020244 = 0x0001 WROTE 0x44020240 = 0x0001 READ 0x44020244 = 0x0001 READ 0x44020244 = 0x0001 success >> \end{verbatim} Step 3: setting arp entries \begin{verbatim} root@ESPRIMO-P956:~# ip neigh add 10.0.0.6 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 root@ESPRIMO-P956:~# ip neigh add 10.0.0.4 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 \end{verbatim} Step 3: generating test packets, expecting 4 packets to show up on enp2s0f0: %\tvb \begin{verbatim} nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp2s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:49:28.200407 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 1, length 64 10:49:28.200445 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 1, length 64 10:49:29.222340 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64 10:49:29.222418 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64 \end{verbatim} %\tve Result: sucess \subsection{Test 2: IPv6 egress} Similar to the IPv4 setting before, just for IPv6. Step 1: getting IP address values \begin{verbatim} >>> int(ipaddress.IPv6Address(u"2001:db8:42::4")) 42540766411362381960998550477184434180L >>> int(ipaddress.IPv6Address(u"2001:db8:42::6")) 42540766411362381960998550477184434182L >>> int(ipaddress.IPv6Address(u"2001:db8:42::42")) 42540766411362381960998550477184434242L \end{verbatim} Step 2: setting table entries \begin{verbatim} >> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434182 => 64 0 0 0 0 fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)] action_name = TopPipe.realmain.set_egress_port field_vals = [1, '64', '0', '0', '0', '0'] CAM_Init_ValidateContext() - done WROTE 0x44020350 = 0x0006 WROTE 0x44020354 = 0x0000 WROTE 0x44020358 = 0x420000 WROTE 0x4402035c = 0x20010db8 WROTE 0x44020380 = 0x0000 WROTE 0x44020384 = 0x0000 WROTE 0x44020388 = 0x40000000 WROTE 0x4402038c = 0x0001 READ 0x44020344 = 0x0001 WROTE 0x44020340 = 0x0001 READ 0x44020344 = 0x0001 READ 0x44020344 = 0x0001 success >> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434242 => 64 0 0 0 0 fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)] action_name = TopPipe.realmain.set_egress_port field_vals = [1, '64', '0', '0', '0', '0'] CAM_Init_ValidateContext() - done WROTE 0x44020350 = 0x0042 WROTE 0x44020354 = 0x0000 WROTE 0x44020358 = 0x420000 WROTE 0x4402035c = 0x20010db8 WROTE 0x44020380 = 0x0000 WROTE 0x44020384 = 0x0000 WROTE 0x44020388 = 0x40000000 WROTE 0x4402038c = 0x0001 READ 0x44020344 = 0x0001 WROTE 0x44020340 = 0x0001 READ 0x44020344 = 0x0001 READ 0x44020344 = 0x0001 success >> \end{verbatim} Step 3: setting neighbor entries \begin{verbatim} nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::6 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::4 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 \end{verbatim} Step 4: generating test packets \begin{verbatim} nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::6 PING 2001:db8:42::6(2001:db8:42::6) 56 data bytes nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes 11:30:17.287577 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 1, length 64 11:30:17.287599 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 1, length 64 11:30:18.310178 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64 11:30:18.310258 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64 \end{verbatim} Result: success, packet is seen twice. % ------------------------------------------------------------------------------ \subsection{Test 3: NAT64} Additionally to the preparations done in test 1 and 2, the following steps were taken: Step 1: getting IP address values via Python \begin{verbatim} >>> int(ipaddress.IPv6Address(u"2001:db8:42::2a")) 42540766411362381960998550477184434218L >>> int(ipaddress.IPv6Address(u"2001:db8:42::")) 42540766411362381960998550477184434176L >>> int(ipaddress.IPv6Address(u"2001:db8:42::a00:2a")) 42540766411362381960998550477352206378 >>> int(ipaddress.IPv4Address(u"10.0.0.0")) 167772160 >>> int(ipaddress.IPv4Address(u"10.0.0.66")) 167772226 \end{verbatim} Add table entry for 2001:db8:42:2a to be translated to 10.0.0.42: \begin{verbatim} >> table_cam_add_entry realmain_nat64_0 realmain.nat64_static 42540766411362381960998550477184434218 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0 fields = [(u'hit', 1), (u'action_run', 3), (u'v6_src', 128), (u'v4_dst', 32), (u'nat64_prefix', 128), (u'table_id', 16)] action_name = TopPipe.realmain.nat64_static field_vals = [2, '42540766411362381960998550477184434176', '167772160', '42540766411362381960998550477184434176', '0'] CAM_Init_ValidateContext() - done WROTE 0x44020050 = 0x002a WROTE 0x44020054 = 0x0000 WROTE 0x44020058 = 0x420000 WROTE 0x4402005c = 0x20010db8 WROTE 0x44020080 = 0x0000 WROTE 0x44020084 = 0x0000 WROTE 0x44020088 = 0x0000 WROTE 0x4402008c = 0xdb80042 WROTE 0x44020090 = 0x2001 WROTE 0x44020094 = 0x0a00 WROTE 0x44020098 = 0x0000 WROTE 0x4402009c = 0x0000 WROTE 0x440200a0 = 0xdb80042 WROTE 0x440200a4 = 0x22001 READ 0x44020044 = 0x0001 WROTE 0x44020040 = 0x0001 READ 0x44020044 = 0x0001 READ 0x44020044 = 0x0001 success >> \end{verbatim} Add table entry for 2001:db8:42::a00:2a to be translated to 10.0.0.66: \begin{verbatim} table_cam_add_entry realmain_nat64_0 realmain.nat64_static 42540766411362381960998550477352206378 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0 \end{verbatim} Add table entry for 10.0.0.66 to be translated to 2001:db8:42:42: \begin{verbatim} >> table_cam_add_entry realmain_nat46_0 realmain.nat46_static 167772226 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0 fields = [(u'hit', 1), (u'action_run', 3), (u'v6_src', 128), (u'v4_dst', 32), (u'nat64_prefix', 128), (u'table_id', 16)] action_name = TopPipe.realmain.nat46_static field_vals = [2, '42540766411362381960998550477184434176', '167772160', '42540766411362381960998550477184434176', '0'] CAM_Init_ValidateContext() - done WROTE 0x44020150 = 0xa000042 WROTE 0x44020180 = 0x0000 WROTE 0x44020184 = 0x0000 WROTE 0x44020188 = 0x0000 WROTE 0x4402018c = 0xdb80042 WROTE 0x44020190 = 0x2001 WROTE 0x44020194 = 0x0a00 WROTE 0x44020198 = 0x0000 WROTE 0x4402019c = 0x0000 WROTE 0x440201a0 = 0xdb80042 WROTE 0x440201a4 = 0x22001 READ 0x44020144 = 0x0001 WROTE 0x44020140 = 0x0001 READ 0x44020144 = 0x0001 READ 0x44020144 = 0x0001 success >> \end{verbatim} Step 3: setting neighbor entries \begin{verbatim} sudo ip neigh add 10.0.0.66 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 sudo ip -6 neigh add 2001:db8:42::2a lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 sudo ip -6 neighbor add 2001:db8:42::a00:2a lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 \end{verbatim} Step 4: ping test should translate, but fail with wrong checksum: \begin{verbatim} \end{verbatim} \begin{verbatim} \end{verbatim} \chapter{\label{appendixB}NetFPGA Logs} Majority of the log files are stored inside the source code directory stored at ``netpfga/logs''. It follows a selection of log files \section{\label{chapterB:netpfga-flasherror}NetFPGA Flash Errors} Sometimes flashing bitfiles to the NetFPGA will fail. A random amount of reboots (1 to 3) and a random amount of reflashing will fix this problem. Below can be found the log output from the flashing process. \begin{verbatim} nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh" ++ which vivado + xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado + bitimage=minip4.bit + configWrites=config_writes.sh + '[' -z minip4.bit ']' + '[' -z config_writes.sh ']' + '[' /opt/Xilinx/Vivado/2018.2/bin/vivado == '' ']' + rmmod sume_riffa + xsct /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl -tclargs minip4.bit rlwrap: warning: your $TERM is 'screen' but rlwrap couldn't find it in the terminfo database. Expect some problems. RUN loading image file. minip4.bit 100% 19MB 1.7MB/s 00:11 fpga configuration failed. DONE PIN is not HIGH invoked from within "::tcf::eval -progress ::xsdb::print_progress {::tcf::cache_enter tcfchan#0 {tcf_cache_eval {process_tcf_actions_cache_client ::tcfclient#0::arg}}}" (procedure "::tcf::cache_eval_with_progress" line 2) invoked from within "::tcf::cache_eval_with_progress [dict get $arg chan] [list process_tcf_actions_cache_client $argvar] $progress" (procedure "process_tcf_actions" line 1) invoked from within "process_tcf_actions $arg ::xsdb::print_progress" (procedure "fpga" line 430) invoked from within "fpga -f $bitimage" (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl" line 33) + bash /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/pci_rescan_run.sh Check programming FPGA or Reboot machine ! + rmmod sume_riffa rmmod: ERROR: Module sume_riffa is not currently loaded + modprobe sume_riffa + ifconfig nf0 up nf0: ERROR while getting interface flags: No such device + ifconfig nf1 up nf1: ERROR while getting interface flags: No such device + ifconfig nf2 up nf2: ERROR while getting interface flags: No such device + ifconfig nf3 up nf3: ERROR while getting interface flags: No such device + bash config_writes.sh \end{verbatim} \section{\label{chapterB:netpfga-flashok}NetFPGA Flash Success} A successful flashing process also emits a couple of errors, however the message ``fpga configuration failed. DONE PIN is not HIGH'' and its succeeding lines are missing, as seen below. After that in all cases a reboot is required; the PCI rescan in no tested case showed the nf devices. \begin{verbatim} nico@nsg-System:~$ cd $NF_DESIGN_DIR/bitfiles/ nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh" ++ which vivado + xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado + bitimage=minip4.bit + configWrites=config_writes.sh + '[' -z minip4.bit ']' + '[' -z config_writes.sh ']' + '[' /opt/Xilinx/Vivado/2018.2/bin/vivado == '' ']' + rmmod sume_riffa + xsct /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl -tclargs minip4.bit rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems. RUN loading image file. minip4.bit attempting to launch hw_server ****** Xilinx hw_server v2018.2 **** Build date : Jun 14 2018-20:18:37 ** Copyright 1986-2018 Xilinx, Inc. All Rights Reserved. INFO: hw_server application started INFO: Use Ctrl-C to exit hw_server application INFO: To connect to this hw_server instance use url: TCP:127.0.0.1:3121 100% 19MB 1.7MB/s 00:11 + bash /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/pci_rescan_run.sh Check programming FPGA or Reboot machine ! + rmmod sume_riffa rmmod: ERROR: Module sume_riffa is not currently loaded + modprobe sume_riffa + ifconfig nf0 up nf0: ERROR while getting interface flags: No such device + ifconfig nf1 up nf1: ERROR while getting interface flags: No such device + ifconfig nf2 up nf2: ERROR while getting interface flags: No such device + ifconfig nf3 up nf3: ERROR while getting interface flags: No such device + bash config_writes.sh nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ \end{verbatim} \section{\label{chapterB:netpfga-kernelmodule}NetFPGA Kernel module} After a successful flash, loading the kernel module will enable nf devices to appear in the operating system. \begin{verbatim} nico@nsg-System:~$ ip l 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 74:d0:2b:98:38:f6 brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether f8:f2:1e:41:44:9c brd ff:ff:ff:ff:ff:ff 4: eth2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether f8:f2:1e:41:44:9d brd ff:ff:ff:ff:ff:ff 5: wg0: mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/none nico@nsg-System:~$ ~/master-thesis/bin/build-load-drivers.sh + cd /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 + sudo modprobe -r sume_riffa + make clean make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 clean make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic' CLEAN /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/.tmp_versions CLEAN /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/Module.symvers make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic' + make all make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic' CC [M] /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.o Building modules, stage 2. MODPOST 1 modules CC /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.mod.o LD [M] /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.ko make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic' + sudo make install make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic' Building modules, stage 2. MODPOST 1 modules make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic' install -o root -g root -m 0755 -d /lib/modules/4.15.0-55-generic/extra/sume_riffa/ install -o root -g root -m 0755 sume_riffa.ko /lib/modules/4.15.0-55-generic/extra/sume_riffa/ depmod -a 4.15.0-55-generic + sudo modprobe sume_riffa + grep sume_riffa + lsmod sume_riffa 28672 0 nico@nsg-System:~$ nico@nsg-System:~$ ip l 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 74:d0:2b:98:38:f6 brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether f8:f2:1e:41:44:9c brd ff:ff:ff:ff:ff:ff 4: eth2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether f8:f2:1e:41:44:9d brd ff:ff:ff:ff:ff:ff 5: wg0: mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/none 6: nf0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 02:53:55:4d:45:00 brd ff:ff:ff:ff:ff:ff 7: nf1: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 02:53:55:4d:45:01 brd ff:ff:ff:ff:ff:ff 8: nf2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 02:53:55:4d:45:02 brd ff:ff:ff:ff:ff:ff 9: nf3: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 02:53:55:4d:45:03 brd ff:ff:ff:ff:ff:ff nico@nsg-System:~$ \end{verbatim} \section{\label{chapterB:netpfga-nftraffic}NetFPGA misses packets on nf*} While the nf devices appear in the operating system, packets emitted by the netpfga cannot be sniffed on the nf interfaces directly. Instead one has to sniff packets on a physical network card that is connected to the specific output port. %--------------------------------------------------------------------------------------------------------- \chapter{\label{benchmark}Benchmark Logs} \section{\label{benchmark:iperf}iperf} Omitting startup time \section{\label{benchmark:general}General} MTU setting to 1500, as netpfga doesn't support jumbo frames iperf3, iperf 3.0.11 50 parallel = 2x 100% cpu usage 40 parallel = 100%, 70% cpu usage 30 parallel = 70%-100, 70% cpu usage Turning back on checksum offloading (see below) 30 parallel = 70%, 30% cpu usage \begin{verbatim} root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: on tx-checksum-ip-generic: on tx-checksum-sctp: on tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp6-segmentation: on root@ESPRIMO-P956:~# root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: on tx-checksum-ip-generic: on tx-checksum-sctp: on tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp6-segmentation: on root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# \end{verbatim} Results into \begin{verbatim} root@ESPRIMO-P956:~# ethtool -k enp2s0f0 Features for enp2s0f0: Cannot get device udp-fragmentation-offload settings: Operation not supported rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: on [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: on highdma: on [fixed] rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: on [fixed] tx-gre-segmentation: on tx-gre-csum-segmentation: on tx-ipxip4-segmentation: on tx-ipxip6-segmentation: on tx-udp_tnl-segmentation: on tx-udp_tnl-csum-segmentation: on tx-gso-partial: on tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off hw-tc-offload: off esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: off root@ESPRIMO-P956:~# ethtool -k enp2s0f1 Features for enp2s0f1: Cannot get device udp-fragmentation-offload settings: Operation not supported rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: on [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: on highdma: on [fixed] rx-vlan-filter: on vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: on [fixed] tx-gre-segmentation: on tx-gre-csum-segmentation: on tx-ipxip4-segmentation: on tx-ipxip6-segmentation: on tx-udp_tnl-segmentation: on tx-udp_tnl-csum-segmentation: on tx-gso-partial: on tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off hw-tc-offload: off esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: off root@ESPRIMO-P956:~# \end{verbatim} \section{\label{benchmark:netpfga}NetFPGA} iperf3-tcp-listening-v4 connected by v6 \begin{verbatim} nico@ESPRIMO-P956:~$ iperf3 -p 2345 -4 -B 10.0.0.42 -s ----------------------------------------------------------- Server listening on 2345 ----------------------------------------------------------- Accepted connection from 10.0.0.66, port 50900 [ 5] local 10.0.0.42 port 2345 connected to 10.0.0.66 port 50902 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 693 MBytes 5.81 Gbits/sec [ 5] 1.00-2.00 sec 645 MBytes 5.41 Gbits/sec [ 5] 2.00-3.00 sec 644 MBytes 5.40 Gbits/sec [ 5] 3.00-4.00 sec 868 MBytes 7.28 Gbits/sec [ 5] 4.00-5.00 sec 853 MBytes 7.16 Gbits/sec [ 5] 5.00-6.00 sec 913 MBytes 7.66 Gbits/sec [ 5] 6.00-7.00 sec 774 MBytes 6.49 Gbits/sec [ 5] 7.00-8.00 sec 641 MBytes 5.38 Gbits/sec [ 5] 8.00-9.00 sec 911 MBytes 7.64 Gbits/sec [ 5] 9.00-10.00 sec 733 MBytes 6.15 Gbits/sec [ 5] 10.00-10.04 sec 25.8 MBytes 5.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-10.04 sec 7.52 GBytes 6.43 Gbits/sec 14 sender [ 5] 0.00-10.04 sec 7.52 GBytes 6.43 Gbits/sec receiver ----------------------------------------------------------- Server listening on 2345 ----------------------------------------------------------- nico@ESPRIMO-P956:~$ iperf3 -6 -p 2345 -c 2001:db8:42::a00:2a Connecting to host 2001:db8:42::a00:2a, port 2345 [ 4] local 2001:db8:42::42 port 50902 connected to 2001:db8:42::a00:2a port 2345 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 719 MBytes 6.03 Gbits/sec 10 449 KBytes [ 4] 1.00-2.00 sec 645 MBytes 5.41 Gbits/sec 0 449 KBytes [ 4] 2.00-3.00 sec 644 MBytes 5.40 Gbits/sec 0 449 KBytes [ 4] 3.00-4.00 sec 878 MBytes 7.36 Gbits/sec 0 449 KBytes [ 4] 4.00-5.00 sec 859 MBytes 7.20 Gbits/sec 0 449 KBytes [ 4] 5.00-6.00 sec 910 MBytes 7.64 Gbits/sec 0 449 KBytes [ 4] 6.00-7.00 sec 758 MBytes 6.36 Gbits/sec 0 449 KBytes [ 4] 7.00-8.00 sec 658 MBytes 5.52 Gbits/sec 0 449 KBytes [ 4] 8.00-9.00 sec 906 MBytes 7.60 Gbits/sec 4 449 KBytes [ 4] 9.00-10.00 sec 724 MBytes 6.07 Gbits/sec 0 449 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 7.52 GBytes 6.46 Gbits/sec 14 sender [ 4] 0.00-10.00 sec 7.52 GBytes 6.46 Gbits/sec receiver iperf Done. nico@ESPRIMO-P956:~$ \end{verbatim} listening on v6, connecting from v4: \begin{verbatim} nico@ESPRIMO-P956:~$ iperf3 -p 2345 -6 -B 2001:db8:42::42 -s ----------------------------------------------------------- Server listening on 2345 ----------------------------------------------------------- Accepted connection from 2001:db8:42::a00:2a, port 47520 [ 5] local 2001:db8:42::42 port 2345 connected to 2001:db8:42::a00:2a port 47522 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.02 GBytes 8.73 Gbits/sec [ 5] 1.00-2.00 sec 879 MBytes 7.38 Gbits/sec [ 5] 2.00-3.00 sec 859 MBytes 7.20 Gbits/sec [ 5] 3.00-4.00 sec 1.02 GBytes 8.78 Gbits/sec [ 5] 4.00-5.00 sec 1.04 GBytes 8.89 Gbits/sec [ 5] 5.00-6.00 sec 1.05 GBytes 9.00 Gbits/sec [ 5] 6.00-7.00 sec 1.03 GBytes 8.89 Gbits/sec [ 5] 7.00-8.00 sec 1.04 GBytes 8.91 Gbits/sec [ 5] 8.00-9.00 sec 1.03 GBytes 8.84 Gbits/sec [ 5] 9.00-10.00 sec 953 MBytes 7.99 Gbits/sec [ 5] 10.00-10.04 sec 38.6 MBytes 7.81 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-10.04 sec 9.89 GBytes 8.46 Gbits/sec 151 sender [ 5] 0.00-10.04 sec 9.89 GBytes 8.46 Gbits/sec receiver ----------------------------------------------------------- Server listening on 2345 ----------------------------------------------------------- nico@ESPRIMO-P956:~$ iperf3 -4 -p 2345 -c 10.0.0.66 Connecting to host 10.0.0.66, port 2345 [ 4] local 10.0.0.42 port 47522 connected to 10.0.0.66 port 2345 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.06 GBytes 9.10 Gbits/sec 53 208 KBytes [ 4] 1.00-2.00 sec 867 MBytes 7.27 Gbits/sec 6 379 KBytes [ 4] 2.00-3.00 sec 870 MBytes 7.29 Gbits/sec 0 423 KBytes [ 4] 3.00-4.00 sec 1.02 GBytes 8.77 Gbits/sec 37 364 KBytes [ 4] 4.00-5.00 sec 1.04 GBytes 8.91 Gbits/sec 1 450 KBytes [ 4] 5.00-6.00 sec 1.05 GBytes 8.98 Gbits/sec 0 462 KBytes [ 4] 6.00-7.00 sec 1.04 GBytes 8.92 Gbits/sec 30 324 KBytes [ 4] 7.00-8.00 sec 1.04 GBytes 8.88 Gbits/sec 0 471 KBytes [ 4] 8.00-9.00 sec 1.03 GBytes 8.86 Gbits/sec 10 452 KBytes [ 4] 9.00-10.00 sec 947 MBytes 7.94 Gbits/sec 14 409 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 9.89 GBytes 8.49 Gbits/sec 151 sender [ 4] 0.00-10.00 sec 9.89 GBytes 8.49 Gbits/sec receiver iperf Done. nico@ESPRIMO-P956:~$ \end{verbatim} % \section{\label{benchmark:bmv2}BMV2} %--------------------------------------------------------------------------------------------------------- \section{\label{benchmark:tayga}Tayga} \begin{verbatim} ii tayga 0.9.2-6 amd64 userspace stateless NAT64 \end{verbatim} Setting up IPv4 networking \begin{verbatim} [15:12] nsg-System:~# ip addr add 10.0.0.77/24 dev eth1 [15:12] nsg-System:~# ip l s eth1 up nico@ESPRIMO-P956:~$ ~/master-thesis/bin/init_ipv4_esprimo.sh nico@ESPRIMO-P956:~$ cat ~/master-thesis/bin/init_ipv4_esprimo.sh #!/bin/sh sudo ip addr add 10.0.0.42/24 dev enp2s0f0 sudo ip link set enp2s0f0 up nico@ESPRIMO-P956:~$ sudo ip route add 10.0.1.0/24 via 10.0.0.77 \end{verbatim} Verify networking works: \begin{verbatim} [15:12] nsg-System:~# ping 10.0.0.42 PING 10.0.0.42 (10.0.0.42) 56(84) bytes of data. 64 bytes from 10.0.0.42: icmp_seq=1 ttl=64 time=0.304 ms 64 bytes from 10.0.0.42: icmp_seq=2 ttl=64 time=0.097 ms ^C --- 10.0.0.42 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1011ms rtt min/avg/max/mdev = 0.097/0.200/0.304/0.104 ms [15:12] nsg-System:~# \end{verbatim} Setting up IPv6 networking \begin{verbatim} nico@ESPRIMO-P956:~$ ip addr show dev enp2s0f1 13: enp2s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff inet6 2001:db8:42::42/64 scope global valid_lft forever preferred_lft forever inet6 fe80::faf2:1eff:fe09:62d1/64 scope link valid_lft forever preferred_lft forever nico@ESPRIMO-P956:~$ sudo ip route add 2001:db8:23::/96 via 2001:db8:42::77 [15:12] nsg-System:~# ip addr add 2001:db8:42::77/64 dev eth2 [15:15] nsg-System:~# ip link set eth2 up \end{verbatim} Verify IPv6 networking works: \begin{verbatim} nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::77 PING 2001:db8:42::77(2001:db8:42::77) 56 data bytes 64 bytes from 2001:db8:42::77: icmp_seq=1 ttl=64 time=0.169 ms 64 bytes from 2001:db8:42::77: icmp_seq=2 ttl=64 time=0.153 ms --- 2001:db8:42::77 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1010ms rtt min/avg/max/mdev = 0.153/0.161/0.169/0.008 ms nico@ESPRIMO-P956:~$ \end{verbatim} Enabling IPv6 and IPv4 forwarding: \begin{verbatim} [15:16] nsg-System:~# sysctl -w net.ipv6.conf.all.forwarding=1 net.ipv6.conf.all.forwarding = 1 [15:20] nsg-System:~# sysctl -w net.ipv4.ip_forward=1 net.ipv4.ip_forward = 1 \end{verbatim} Testing NAT64 in tayga \begin{verbatim} nico@ESPRIMO-P956:~$ ping -c2 10.0.1.42 PING 10.0.1.42 (10.0.1.42) 56(84) bytes of data. 64 bytes from 10.0.1.42: icmp_seq=1 ttl=61 time=0.356 ms 64 bytes from 10.0.1.42: icmp_seq=2 ttl=61 time=0.410 ms --- 10.0.1.42 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1019ms rtt min/avg/max/mdev = 0.356/0.383/0.410/0.027 ms nico@ESPRIMO-P956:~$ nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes 15:21:39.851057 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 1, length 64 15:21:39.851124 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 1, length 64 15:21:40.870448 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 2, length 64 15:21:40.870507 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 2, length 64 ^C 4 packets captured 4 packets received by filter 0 packets dropped by kernel nico@ESPRIMO-P956:~$ \end{verbatim} Testing NAT64 (v6 to v4) \begin{verbatim} nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:23::a00:2a PING 2001:db8:23::a00:2a(2001:db8:23::a00:2a) 56 data bytes 64 bytes from 2001:db8:23::a00:2a: icmp_seq=1 ttl=61 time=0.240 ms 64 bytes from 2001:db8:23::a00:2a: icmp_seq=2 ttl=61 time=0.400 ms --- 2001:db8:23::a00:2a ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1003ms rtt min/avg/max/mdev = 0.240/0.320/0.400/0.080 ms nico@ESPRIMO-P956:~$ \end{verbatim} \subsection{\label{benchmark:tayga:tcp}Tayga/TCP} Tayga running at 100% cpu load, v4->v6 tcp delivering 3.36 gbit/s at P1 3.30 Gbit/s at P20 3.11 gbit/s at P50 v6->v4 tcp P1: 3.02 Gbit/s P20: 3.28 gbit/s P50: 2.85 gbit/s Commands: \begin{verbatim} Server always: iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-tayga-v4tov6server-P50 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -c 10.0.1.42 -T taygav4tov6tcpP1 | tee iperf-tayga-v4tov6server-client nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -c 10.0.1.42 -T taygav4tov6tcpP20 | tee iperf-tayga-v4tov6server-client-P20 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -c 10.0.1.42 -T taygav4tov6tcpP50 | tee iperf-tayga-v4tov6server-client-P50 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-P1 \end{verbatim} Testing v6->v4 \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-P20 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -c 2001:db8:23::a00:2a -T taygav6tov4tcpP1 | tee iperf-tayga-v6tov4-client-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P20 -c 2001:db8:23::a00:2a -T taygav6tov4tcpP20 | tee iperf-tayga-v6tov4-client-P20 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -c 2001:db8:23::a00:2a -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-P50 \end{verbatim} UDP v6->v4, again 100% cpu P1: 5.81 gbit/s P20: 9.40 gbit/s P50: 19.6 gbits/sec On the line only ca. 3600 mbit/s seen \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-udp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -u -b10000m -c 2001:db8:23::a00:2a -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-udp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -u -b10000m -c 2001:db8:23::a00:2a -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-udp-P50 \end{verbatim} Messsages from server: \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-udp-P1 iperf3: OUT OF ORDER - incoming packet = 198902 and received packet = 198904 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 441615 and received packet = 441617 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 441616 and received packet = 441618 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 567495 and received packet = 567501 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 567496 and received packet = 567501 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 567497 and received packet = 567501 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 567499 and received packet = 567503 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 567500 and received packet = 567503 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 567502 and received packet = 567503 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 631160 and received packet = 631164 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 631161 and received packet = 631164 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 631162 and received packet = 631165 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 631163 and received packet = 631165 AND SP = 5 \end{verbatim} UDP v4->v6, again 100% cpu P1: 8.26 gbit/s [atop: ~2500 Mbit/s per direction] P20: 9.92 Gbits/sec [atop: ~2500 Mbit/s per direction] P50: 19.3 gbit/s [atop: ~2500 Mbit/s per direction] \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-tayga-v4tov6-server-udp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP1 | tee iperf-tayga-v4tov6server-client-udp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP20 | tee iperf-tayga-v4tov6server-client-udp-P20 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP50 | tee iperf-tayga-v4tov6server-client-udp-P50 \end{verbatim} %--------------------------------------------------------------------------------------------------------- \section{\label{benchmark:jool}Jool} \subsection{\label{benchmark:jool:setup}Jool Setup} Installation of 4.0.1 from \url{https://www.jool.mx/en/download.html}. \begin{verbatim} nico@nsg-System:~$ wget https://github.com/NICMx/Jool/releases/download/v4.0.1/jool_4.0.1.tar.gz nico@nsg-System:~$ tar xvfz jool_4.0.1.tar.gz nico@nsg-System:~$ cd jool-4.0.1/ nico@nsg-System:~/jool-4.0.1$ sudo apt install linux-headers-$(uname -r) nico@nsg-System:~/jool-4.0.1$ sudo apt install libnl-genl-3-dev \end{verbatim} xtables cannot be found: \begin{verbatim} nico@nsg-System:~/jool-4.0.1$ sudo apt install libxtables-dev Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package libxtables-dev nico@nsg-System:~/jool-4.0.1$ \end{verbatim} Does not compile without: \begin{verbatim} checking for library containing argp_parse... none required checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for LIBNLGENL3... yes checking for XTABLES... no configure: error: Package requirements (xtables) were not met: No package 'xtables' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables XTABLES_CFLAGS and XTABLES_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. nico@nsg-System:~/jool-4.0.1$ \end{verbatim} Trying different package: \begin{verbatim} nico@nsg-System:~/jool-4.0.1$ sudo apt install iptables-dev \end{verbatim} Compiles! \begin{verbatim} nico@nsg-System:~/jool-4.0.1$ sudo make install \end{verbatim} \subsection{\label{benchmark:jool:config}Jool Configuration} Loading module: \begin{verbatim} nico@nsg-System:~/jool-4.0.1$ sudo modprobe jool_siit \end{verbatim} enabling forwarding: \begin{verbatim} sysctl -w net.ipv4.conf.all.forwarding=1 sysctl -w net.ipv6.conf.all.forwarding=1 \end{verbatim} Mapping configuration: \begin{verbatim} nico@nsg-System:~/jool-4.0.1$ sudo jool_siit instance add example --iptables --pool6 2001:db8:23::/96 nico@nsg-System:~/jool-4.0.1$ sudo ip6tables -t mangle -A PREROUTING \ -s 2001:db8:42::/64 -d 2001:db8:23::/96 -j JOOL_SIIT --instance example nico@nsg-System:~/jool-4.0.1$ sudo iptables -t mangle -A PREROUTING \ -s 10.0.0.0/24 -j JOOL_SIIT --instance example \end{verbatim} Debugging: \begin{verbatim} [16:39] nsg-System:~# lsmod| grep jool jool_siit 147456 2 x_tables 40960 5 jool_siit,ip6_tables,ip_tables,ip6table_mangle,iptable_mangle [16:39] nsg-System:~# [16:41] nsg-System:~# jool_siit -i example stats display --explain JSTAT64_DST: 276 Translations cancelled: IPv6 packet's destination address did not match pool6 nor any EAMT entries, or the resulting address was blacklist4ed. \end{verbatim} Try 2 w/ eamt: \begin{verbatim} [16:53] nsg-System:~# modprobe jool_siit [16:54] nsg-System:~# jool_siit instance add "example" --iptables [16:54] nsg-System:~# jool_siit -i example eamt add 2001:db8:42::/120 10.0.1.0/24 [16:55] nsg-System:~# jool_siit -i example eamt add 2001:db8:23::/120 10.0.0.0/24 [16:57] nsg-System:~# ip6tables -t mangle -A PREROUTING -s 2001:db8:42::/120 -d 2001:db8:23::/120 -j JOOL_SIIT --instance example [16:57] nsg-System:~# iptables -t mangle -A PREROUTING -s 10.0.0.0/24 -d 10.0.1.0/24 -j JOOL_SIIT --instance example [16:57] nsg-System:~# \end{verbatim} Testing NAT64: \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ ping6 2001:db8:23::2a PING 2001:db8:23::2a(2001:db8:23::2a) 56 data bytes 64 bytes from 2001:db8:23::2a: icmp_seq=1 ttl=63 time=0.199 ms 64 bytes from 2001:db8:23::2a: icmp_seq=2 ttl=63 time=0.282 ms 64 bytes from 2001:db8:23::2a: icmp_seq=3 ttl=63 time=0.186 ms ^C --- 2001:db8:23::2a ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2040ms rtt min/avg/max/mdev = 0.186/0.222/0.282/0.044 ms nico@ESPRIMO-P956:~/master-thesis/iperf$ ping 10.0.1.66 PING 10.0.1.66 (10.0.1.66) 56(84) bytes of data. 64 bytes from 10.0.1.66: icmp_seq=1 ttl=63 time=0.218 ms 64 bytes from 10.0.1.66: icmp_seq=2 ttl=63 time=0.281 ms 64 bytes from 10.0.1.66: icmp_seq=3 ttl=63 time=0.280 ms ^C --- 10.0.1.66 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2051ms rtt min/avg/max/mdev = 0.218/0.259/0.281/0.034 ms nico@ESPRIMO-P956:~/master-thesis/iperf$ \end{verbatim} \subsection{\label{benchmark:jool:benchmark}Jool Benchmarks} v4->v6 tcp P1: 8.24 gbit/s no cpu load visible P20: 8.26 gbit/s iperf 42 + 10% cpu no cpu load visible P50: 8.29 gbit/s v6->v4 tcp P1: 8.22 P20: 8.22 15/60% iperf P50: 8.23 iperf: 73/16% Commands: \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-jool-v4tov6-server-tcp-P50 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P20 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P50 Other way: nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-jool-v6tov4-server-tcp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P1 ... nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -b0 -u -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P1 \end{verbatim} v4->v6 udp P1: 4.46 iperf 30% cpu nat64: 100% cpu P20: 18.8 iperf 100% nat64: 100% P50: 22.8 iperf 100% nat64: 100% atop: ca. 5gbit/s \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-jool-v4tov6-server-udp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -c 10.0.1.66 -u -b0 | tee iperf-jool-v4tov6-client-udp-P1 nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -c 10.0.1.66 -u -b0 | tee iperf-jool-v4tov6-client-udp-P20 \end{verbatim} v6->v4 udp P1: 6.67 gbit/s iperf 50/50%, nat64: 100% cpu! P20: 16.8 nat64: iperf: ? 100% cpu P50: 20.5 Gbits/sec nat64: 100% (1 core) iperf: 100%/50% atop: 7000mbit/s in / out \begin{verbatim} \end{verbatim} Turning off offloading, redoing tcp: \begin{verbatim} root@ESPRIMO-P956:~# ethtool -K enp2s0f0 gso off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# ethtool -K enp2s0f0 rx off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: off tx-checksum-ip-generic: off tx-checksum-sctp: off tcp-segmentation-offload: off tx-tcp-segmentation: off [requested on] tx-tcp6-segmentation: off [requested on] root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: off tx-checksum-ip-generic: off tx-checksum-sctp: off tcp-segmentation-offload: off tx-tcp-segmentation: off [requested on] tx-tcp6-segmentation: off [requested on] root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# ethtool -K enp2s0f1 gso off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# [17:26] nsg-System:~# ethtool -K eth1 tx off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: off tx-checksum-ip-generic: off tx-checksum-sctp: off tcp-segmentation-offload: off tx-tcp-segmentation: off [requested on] tx-tcp6-segmentation: off [requested on] [17:26] nsg-System:~# ethtool -K eth1 gso off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported [17:26] nsg-System:~# ethtool -K eth2 gso off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported [17:26] nsg-System:~# ethtool -K eth2 rx off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported [17:26] nsg-System:~# ethtool -K eth2 tx off Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: off tx-checksum-ip-generic: off tx-checksum-sctp: off tcp-segmentation-offload: off tx-tcp-segmentation: off [requested on] tx-tcp6-segmentation: off [requested on] [17:26] nsg-System:~# \end{verbatim} Retesting using -P50: Still no cpu load with tcp, 100% cpu load iperf result: 7.96 gbit/s \begin{verbatim} nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-jool-v6tov4-server-tcp-P50-no-offload nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P50-no-offload \end{verbatim} \subsection{\label{benchmark:netfpga:benchmark}NetPFGA Benchmarks} Only 1 test did have offloading on esprimo off, was redone v4->v6 tcp P1: 7.41 gbit/s iperf 50% / 20% P1-offload-on-esprimo: 8.43 gbit/s P20: 9.29 gbit/s iperf: 66/20% P50: v4->v6 udp P1: P20: P50: v6->v4 tcp P1: iperf 50% / 20% P20: P50: v6->v4 udp P1: P20: P50: After first netpfga, tcp v4->v6 p1 turned offloading on again \begin{verbatim} root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx-checksum-ipv6 on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Could not change any device features root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: on tx-checksum-ip-generic: on tx-checksum-sctp: on tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp6-segmentation: on root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# ethtool -K enp2s0f1 gso on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# ethtool -K enp2s0f0 gso on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported Actual changes: tx-checksumming: on tx-checksum-ip-generic: on tx-checksum-sctp: on tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp6-segmentation: on root@ESPRIMO-P956:~# ethtool -K enp2s0f0 rx on Cannot get device udp-fragmentation-offload settings: Operation not supported Cannot get device udp-fragmentation-offload settings: Operation not supported root@ESPRIMO-P956:~# \end{verbatim} %Entries for the list of abbrevations: % %To generate the list of abbrevations, execute: %makeindex Thesis.nlo -s nomencl.ist -o Thesis.nls % %If you are using TeXniCenter, specify: %"%bm.nlo" -s nomencl.ist -o "%bm.nls" %as beeing the argument list for makeindex. %--------------------------------------------------------------------------------------------------------- %For old nomencl package uncomment this: %\printglossary %For new nomencl package uncomment this: \printnomenclature \abbrev{XCA}{\markup{X}tremely \markup{C}ool \markup{A}bbrevations}