diff --git a/doc/Conclusion.tex b/doc/Conclusion.tex index 996ff43..47e89cb 100644 --- a/doc/Conclusion.tex +++ b/doc/Conclusion.tex @@ -66,3 +66,6 @@ Long term supporting python3 would be helpful. P4OS. - react on FIN/RST (?) -- could be an addition + P4os - reusable code + + Future work: session handling diff --git a/doc/Results.tex b/doc/Results.tex index d0c8996..6a8b6fd 100644 --- a/doc/Results.tex +++ b/doc/Results.tex @@ -14,12 +14,26 @@ P4 software implementation. % ok % ---------------------------------------------------------------------- \section{\label{results:p4}P4 based implementations} +We successfully implemented P4 code to realise +NAT64~\cite{schottelius:thesisrepo}. It contains parsers +for all related protocols (ipv6, ipv4, udp, tcp, icmp, icmp6, ndp, +arp), supports EAMT as defined by RFC7757 ~\cite{rfc7757} and is +feature equivalent to the two compared software solutions +tayga~\cite{lutchansky:_tayga_simpl_nat64_linux} and +jool~\cite{mexico:_jool_open_sourc_siit_nat64_linux}. +Due to limitations in the P4 environment of the +NetFPGA~\cite{conclusion:netfpga} environment, the BMV2 implementation +is more feature rich. Table \ref{tab:benchmark} summarises the +achieved bandwidths of the NAT64 solutions. + +BEFORE OR AFTER MARKER - FIXME + All planned features could be realised with P4 and a controller. For this thesis the parsing capabilities of P4 were adequate. However P4, at the time of writing, cannot parse ICMP6 options in general, as the upper level protocol does not specify the number of options that follow and parsing of an undefined number -of 64 bit blocks is required. +of 64 bit blocks is required, which P4 does not support. The language has some limitations on where the placement of conditional statements (\texttt{if/switch}).\footnote{In general, @@ -61,34 +75,15 @@ The supporting scripts in the P4 toolchain are usually written in python2. However python2 ``is legacy''~\cite{various:_shoul_i_python_python}. During development errors with unicode string handling in python2 caused -changes to IPv6 addresses.~\ref{appendix:p4:python2unicode} - -P4os - reusable code - -% idomatic problem: Security issue: not checking checksums before - - -****** TODO IPv6 udp -> IPv4 -- Got 4-5 tuple ([proto], src ip, src port, dst ip, dst port) -- Does not / never signal end -- Needs timeout for cleaning up - - P4/BMV2 thus -allows us to closest resemble any other translation implementation. - -Only supporting /96, not other embeddings as described in -section \ref{background:transition:prefixnat}. - +changes to IPv6 addresses.\footnote{Compare section ~\ref{appendix:p4:python2unicode}.} +% ok % ---------------------------------------------------------------------- -\subsection{\label{Results:BMV2}BMV2} +\section{\label{results:bmv2}P4/BMV2} The software implementation of P4 has most features, which is -mostly due to the capability of checksumming the payload: Acting -as a ``proper'' participant in NDP, requires the host to calculate -checksums over the payload. - - -List of features BMV2 ~\cite{tab:p4bmv2features} - +mostly due to the capability of creating checksums over the payload. +It enables the switch to act as a ``proper'' participant in NDP, as +this requires the host to calculate checksums over the payload. +Table~\ref{tab:p4bmv2features} references all implemented features. \begin{table}[htbp] \begin{center}\begin{minipage}{\textwidth} \begin{tabular}{| c | c | c |} @@ -140,42 +135,38 @@ fully implemented\footnote{Source code: \texttt{checksum\_bmv2.p4}}\\ \hline \end{tabular} \end{minipage} -\caption{P4 / BMV2 feature list} +\caption{P4/BMV2 feature list} \label{tab:p4bmv2features} \end{center} \end{table} +The switch responds to ICMP echo requests, ICMP6 echo requests, +answers NDP and ARP requests. Overall P4/BMV is very easy to use +even without a controller a fully functional network host can be +implemented. -Responds to icmp, icmp6 -ndp ~\cite{rfc4861} -arp - -very easy to use - -Fully functional host -Can compute checksums on its own. - -focus on typical use cases of icmp, icmp6, the software implementation -supports translating echo request and echo reply messages, but does -not support all ICMP/ICMP6 translations that are defined in +This P4/BMV implementation supports translating ICMP/ICMP6 +echo request and echo reply messages, but does not support +all ICMP/ICMP6 translations that are defined in RFC6145~\cite{rfc6145}. - -Stateful : no automatic removal - -Session management not benchmarked, as it is only a matter of creating -table entries. - -Jool and tayga are supported by - - % ---------------------------------------------------------------------- -\subsection{\label{results:netpfga}NetFPGA - FIXME: writing} -The reduced feature set of the NetPFGA implementation is due to two -factors: compile time. Between 2 to 6 hours per compile run. No -payload checksum - -overview - general translation - not advanced features +\section{\label{results:netpfga}P4/NetFPGA} +In the following section we describe the achieved feature set of +P4/NetFPGA in detail and analyse differences to the BMV2 based +implementation. +% ok % ---------------------------------------------------------------------- -\subsubsection{\label{results:netpfga:features}Features} +\subsection{\label{results:netpfga:features}Features} +While the NetFPGA target supports P4, compared to P4/BMV2 +we only implemented a reduced features set on P4/NetPFGA. The first +reason for this is missing +support of the NetFPGA P4 compiler to inspect payload and to compute +checksums over payload. While this can (partially) be compensated +using delta checksums, the compile time of 2 to 6 hours contributed to +a significant slower development cycle compared to BMV2. +Lastly, the focus of this thesis was to implement high speed NAT64 on +P4, which only requires a subset of the features that we realised on +BMV2. Table \ref{tab:p4netpfgafeatures} summarises the implemented +features and reasons about their implementation status. \begin{table}[htbp] \begin{center}\begin{minipage}{\textwidth} \begin{tabular}{| c | c | c |} @@ -239,12 +230,13 @@ unsupported\footnote{To support creating payload checksums, either an \hline \end{tabular} \end{minipage} -\caption{P4 / NetFPGA feature list} +\caption{P4/NetFPGA feature list} \label{tab:p4netpfgafeatures} \end{center} \end{table} +% ok % ---------------------------------------------------------------------- -\subsubsection{\label{results:netpfga:stability}Stability} +\subsection{\label{results:netpfga:stability}Stability} Two different NetPFGA cards were used during the development of the thesis. The first card had consistent ioctl errors (compare section \ref{netpfgaioctlerror}) when writing table entries. The available @@ -266,25 +258,33 @@ on the first NetFPGA card. \label{fig:hwtesthendrik} \end{figure} During the development and benchmarking, the second NetFPGA card stopped to -function properly multiple times. In both cases the card would not -forward packets anymore. Multiple reboots (3 were usually enough) +function properly multiple times. In theses cases the card would not +forward packets anymore. Multiple reboots (up to 3) and multiple times reflashing the bitstream to the NetFPGA usually restored the intended behaviour. However due to this ``crashes'', it -was impossible to complete a full benchmark run that would last for -more than one hour. - -Sometimes it was also required to reboot the host containing the -NetFPGA card 3 times to enable successful flashing.\footnote{Typical -output of the flashing process would be: ``fpga configuration failed. DONE PIN is not HIGH''} +was impossible for us run a benchmark for more than one hour. +Similariy, sometimes flashing the bitstream to the NetFPGA would fail. +It was required to reboot the host containing the +NetFPGA card up to 3 times to enable successful flashing.\footnote{Typical +output of the flashing process would be: ``fpga configuration +failed. DONE PIN is not HIGH''} +% ok % ---------------------------------------------------------------------- \subsubsection{\label{results:netpfga:performance}Performance} -As expected, the NetFGPA card performed at near line speed and offers -NAT64 translations at 9.28 Gbit/s. Single and multiple streams +The NetFGPA card performed at near line speed and offers +NAT64 translations at 9.28 Gbit/s (see section \ref{results:benchmark} +for details). +Single and multiple streams performed almost exactly identical and have been consistent through multiple iterations of the benchmarks. +% ok % ---------------------------------------------------------------------- -\subsubsection{\label{results:netpfga:usability}Usability} -To use the NetFGPA, Vivado and SDNET provided by Xilinx need to be +\subsection{\label{results:netpfga:usability}Usability} +The handling and usability of the NetFPGA card is rather difficult. In +this section we describe our findings and experiences with the card +and its toolchain. + +To use the NetFGPA, the tools Vivado and SDNET provided by Xilinx need to be installed. However a bug in the installer triggers an infinite loop, if a certain shared library\footnote{The required shared library is libncurses5.} is missing on the target operating system. The @@ -388,36 +388,68 @@ techniques are missing or not supported. Renaming variables in the declaration of the parser or deparser lead to compilation errors. Function syntax is not supported. For this reason our implementation uses \texttt{\#define} statements instead of functions. - -FIXME: - -General result: limited NAT64 is working, however - No Payload ; checksumming - requires controller -Hash funktion in Arbeit ; No NDP, no ARP - focused on key factors of NAT64 translation, -other features can be supported by controller -Needed to debug internal parsing errors -debugging generated tcl code to debug impl1 error - +%ok % ---------------------------------------------------------------------- \section{\label{results:softwarenat64}Software based NAT64} -with Tayga and - Jool -Both cpu bound. - -During the benchmark cpu bound, single thread -tayga: Single threaded -easy to use - -Jool kernel module -100\% cpu usage on 1 core for udp -0\% visible cpu usage for tcp, might be tcp offloading -Integration with iptables -Requires routing - - +Both solutions Tayga and Jool worked flawlessly. However as expected, +both solutions have a bottleneck that is CPU bound. Under high load +scenarios both solutions utilise one core fully. Neither Tayga as a +user space program nor Jool as a kernel module implement multi +threading. +%ok % ---------------------------------------------------------------------- -\section{\label{results:benchmark}NAT64 Benchmarks - FIXME: explain - numbers} +\section{\label{results:benchmark}NAT64 Benchmarks} +In this section we summarise the benchmarking results, in the +sub sections we discuss the benchmark design and the individual results. + +FIXME: summary here + +MTU setting to 1500, as netpfga doesn't support jumbo frames + + +iperf3, iperf 3.0.11 + +50 parallel = 2x 100% cpu usage +40 parallel = 100%, 70% cpu usage +30 parallel = 70%-100, 70% cpu usage + +Turning back on checksum offloading (see below) + +30 parallel = 70%, 30% cpu usage + + +\subsection{\label{benchmark:tayga:tcp}Tayga/TCP} + +Tayga running at 100% cpu load, + +v4->v6 tcp +delivering +3.36 gbit/s at P1 +3.30 Gbit/s at P20 +3.11 gbit/s at P50 + +v6->v4 tcp +P1: 3.02 Gbit/s +P20: 3.28 gbit/s +P50: 2.85 gbit/s + +Commands: + + +UDP load generator hitting 100\% cpu at P20. +TCP confirmed. +Over bandwidth results + +Feature comparison +speed - sessions - eamt +can act as host +lpm tables +ping +ping6 support +ndp +controller support + +netpfga consistent % ---------------------------------------------------------------------- \subsection{\label{results:benchmark:design}Benchmark Design} \begin{figure}[h] @@ -449,20 +481,10 @@ warm up phase.\footnote{iperf -O 10 parameter, see section \ref{design:tests}.} \end{figure} % ok % ---------------------------------------------------------------------- - - -We successfully implemented P4 code to realise -NAT64~\cite{schottelius:thesisrepo}. It contains parsers -for all related protocols (ipv6, ipv4, udp, tcp, icmp, icmp6, ndp, -arp), supports EAMT as defined by RFC7757 ~\cite{rfc7757} and is -feature equivalent to the two compared software solutions -tayga~\cite{lutchansky:_tayga_simpl_nat64_linux} and -jool~\cite{mexico:_jool_open_sourc_siit_nat64_linux}. -Due to limitations in the P4 environment of the -NetFPGA~\cite{conclusion:netfpga} environment, the BMV2 implementation -is more feature rich. Table \ref{tab:benchmark} summarises the -achieved bandwidths of the NAT64 solutions. - +\newpage +\subsection{\label{results:benchmark:v6v4tcp}IPv6 to IPv4 TCP + Benchmark Results} +some text \begin{table}[htbp] \begin{center}\begin{minipage}{\textwidth} @@ -487,8 +509,8 @@ Parallel connections & 1 & 10 & 20 & 50 \\ \label{tab:benchmarkv6} \end{center} \end{table} - - +% --------------------------------------------------------------------- +\subsection{\label{results:benchmark:v4v6tcp}IPv4 to IPv6 TCP Benchmark Results} During the benchmarks the client -- CPU usage \begin{table}[htbp] \begin{center}\begin{minipage}{\textwidth} @@ -514,7 +536,11 @@ Parallel connections & 1 & 10 & 20 & 50 \\ \end{center} \end{table} - +% --------------------------------------------------------------------- +\newpage +\subsection{\label{results:benchmark:v6v4udp}IPv6 to IPv4 UDP + Benchmark Results} +other text \begin{table}[htbp] \begin{center}\begin{minipage}{\textwidth} \begin{tabular}{| c | c | c | c | c |} @@ -540,7 +566,9 @@ Parallel connections & 1 & 10 & 20 & 50 \\ \end{center} \end{table} - +% --------------------------------------------------------------------- +\subsection{\label{results:benchmark:v4v6udp}IPv4 to IPv6 UDP Benchmark Results} +last text \begin{table}[htbp] \begin{center}\begin{minipage}{\textwidth} \begin{tabular}{| c | c | c | c | c |} @@ -565,18 +593,3 @@ Parallel connections & 1 & 10 & 20 & 50 \\ \label{tab:benchmarkv4} \end{center} \end{table} - -UDP load generator hitting 100\% cpu at P20. -TCP confirmed. -Over bandwidth results - -Feature comparison -speed - sessions - eamt -can act as host -lpm tables -ping -ping6 support -ndp -controller support - -netpfga consistent diff --git a/doc/Thesis.pdf b/doc/Thesis.pdf index a232e2e..afe03c1 100644 Binary files a/doc/Thesis.pdf and b/doc/Thesis.pdf differ diff --git a/doc/appendix.tex b/doc/appendix.tex index 114a168..3399441 100644 --- a/doc/appendix.tex +++ b/doc/appendix.tex @@ -1,7 +1,10 @@ \chapter{\label{appendixminus1}Resources and code repositories} The following sections describe how to acquire the resources -to reproduce the test results. All compilations were made on Ubuntu -16.04 with kernels +to reproduce the test results. + +% ---------------------------------------------------------------------- +\section{\label{chapterminus1:thesis:os}Operating Systems} +All P4 compilations were made on Ubuntu 16.04 with kernels \begin{itemize} \item 4.15.0-54-generic (Supporting Desktop) \item 4.4.0-143-generic (BMV2 test VM) @@ -12,10 +15,10 @@ to reproduce the test results. All compilations were made on Ubuntu \section{\label{chapterminus1:thesis:general}Master Thesis} The master thesis including all self developed source code is available by git via -\begin{verbatim} -git clone git@gitlab.ethz.ch:nicosc/master-thesis.git -git clone https://gitlab.ethz.ch/nsg/student-projects/ma-2019-19_high_speed_nat64_with_p4 -\end{verbatim} +\begin{itemize} +\item git clone \url{git@gitlab.ethz.ch:nicosc/master-thesis.git} +\item git clone \url{git@gitlab.ethz.ch:nsg/student-projects/ma-2019-19_high_speed_nat64_with_p4} +\end{itemize} It can be browsed online on \url{https://gitlab.ethz.ch/nicosc/master-thesis} and on \url{https://gitlab.ethz.ch/nsg/student-projects/ma-2019-19_high_speed_nat64_with_p4}. @@ -25,29 +28,27 @@ It can be browsed online on A prerequisite for building the NetFGPA source code is the installation of \begin{itemize} -\item \verb=Xilinx_SDNet_2018.2_1005_9= -\item \verb=Xilinx_Vivado_SDK_2018.2_0614_1954= +\item \texttt{Xilinx\_SDNet\_2018.2\_1005\_9} +\item \texttt{Xilinx\_Vivado\_SDK\_2018\.2\_0614\_1954} \end{itemize} -Both tools need to be installed to /opt/Xilinx/, as paths are -hardcoded in various places. +Both tools need to be installed to \texttt{/opt/Xilinx/}, +as paths are hardcoded in various places. % ok % ---------------------------------------------------------------------- -\section{\label{chapterminus1:thesis}NetFGPA support scripts} +\section{\label{chapterminus1:thesis}P4/NetFGPA support scripts} To be able to compile P4 source code to the NetFPGA the collection of scripts, Makefiles and sample code of P4-NetFGPA is required. - The repository \url{git@github.com:NetFPGA/P4-NetFPGA-live.git} needs to be cloned to ``projects'' subdirectory as ``P4-NetPFGA'' of the user that wants to compile the source code. Access to the repository is granted after applying for access as described on \url{https://github.com/NetFPGA/P4-NetFPGA-public/wiki}. -After that the variable \verb=P4_PROJECT_NAME= in +After that the variable \texttt{P4\_PROJECT\_NAME} in \texttt{~/projects/P4-NetFPGA/tools/settings.sh} needs to be modified to -read \verb|export P4_PROJECT_NAME=minip4| instead of -\verb|export P4_PROJECT_NAME=switch_calc|. +read \texttt{export P4\_PROJECT\_NAME=minip4} +instead of \texttt{export P4\_PROJECT\_NAME=switch\_calc}. Sample code for installation: - \begin{verbatim} mkdir -p ~/projects git clone git@github.com:NetFPGA/P4-NetFPGA-live.git P4-NetFPGA @@ -61,95 +62,32 @@ v1.3.1-46-g97d3aaa \end{verbatim} % ok % ---------------------------------------------------------------------- -\chapter{\label{appendix0}BMV2 environment and tests} -All BMV2 based compilations were made with the following compiler: -\begin{verbatim} -p4@ubuntu:~$ p4c --version -p4c 0.5 (SHA: 5ae30ee) -\end{verbatim} - -The installation is based on the vagrant files that were provided in -the ``Advanced Topics in -Communication Networks Fall 2018'' course of -ETHZ (\url{https://adv-net.ethz.ch/2018/}) and contains p4tools as -well as all utilities that came with the vagrant installation. -% ok +\section{\label{appendix:netfpga:compile}P4/NetFGPA compilation process} +After having setup the compile host as described above, the script +\texttt{bin/do-all-steps.sh} that is included in the thesis' git +repository. With a NetFPGA card installed in the host, this script +will compile the P4 source code to PX and in a second step to HDL and +then upload the resulting bitstream to the NetFPGA. The compilation +process will log its output to the directory +\texttt{\~/master-thesis/netpfga/log/}. % ---------------------------------------------------------------------- -%\section{\label{chapter0:bmv2-diff}Diff based checksumming} -For running the diff based checksum code, the following steps are -necessary: - -Compiling the p4 code and starting the switch: -\begin{verbatim} -cd ~/master-thesis/p4app -sudo p4run --config nat64-diff.json -\end{verbatim} - -Starting the controller which sets up the required table entries: - -\begin{verbatim} -cd ~/master-thesis/p4app -sudo python ./controller.py --mode range_router -\end{verbatim} - - +\section{\label{appendix:netfpga:tests}P4/NetFGPA Tests} +In the following sections we describe functionality tests +of our code on the NetFPGA. % ---------------------------------------------------------------------- -\chapter{\label{appendixA}NetFPGA environment and tests} -% ---------------------------------------------------------------------- -\section{\label{chapterA:netpfga-setup}NetFPGA Setup} -Description of installation, commit of netpfga-live -% ---------------------------------------------------------------------- - -\section{\label{chapterA:netpfga:compile}NetFPGA Compile Flow} - -% ---------------------------------------------------------------------- -\section{\label{chapterA:section1}NetFPGA NAT64 Test cases} -todo: add graphic of nsg <-> esprimo cabling -\begin{verbatim} - -ip addr add 10.0.0.42/24 dev enp2s0f0 - -# Adding necessary ARP entries: for the virtual IPv4 address(es) -ip neigh add 10.0.0.6 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 -ip neigh add 10.0.0.42 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 - -\end{verbatim} - -For all test cases the following network settings on esprimo: - -\begin{verbatim} -12: enp2s0f0: mtu 1500 qdisc mq state UP group default qlen 1000 - link/ether f8:f2:1e:09:62:d0 brd ff:ff:ff:ff:ff:ff - inet 10.0.0.42/24 scope global enp2s0f0 - valid_lft forever preferred_lft forever - inet6 fe80::faf2:1eff:fe09:62d0/64 scope link - valid_lft forever preferred_lft forever -13: enp2s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 - link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff - inet6 2001:db8:42::42/64 scope global - valid_lft forever preferred_lft forever - inet6 fe80::faf2:1eff:fe09:62d1/64 scope link - valid_lft forever preferred_lft forever -\end{verbatim} -% ---------------------------------------------------------------------- -\subsection{Test 1: IPv4 egress} -Scenario: simple egress port setting for the IPv4 addresses - -Step 1: getting correct values for table entries from python: - +\subsection{Test 1: IPv4 Egress} +In this test we test whether setting the output port based on the +IPv4 address. +First we get the integer values of the IPv4 addresses in python: \begin{verbatim} >>> int(ipaddress.IPv4Address(u"10.0.0.42")) 167772202 >>> int(ipaddress.IPv4Address(u"10.0.0.4")) 167772164 >>> - \end{verbatim} - -Step 2: setting table netries - +After that we set the table table entries for the NetFPGA. \begin{verbatim} - >> table_cam_add_entry realmain_v4_networks_0 realmain.set_egress_port 167772202 => 16 0 0 0 0 fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)] action_name = TopPipe.realmain.set_egress_port @@ -182,18 +120,15 @@ READ 0x44020244 = 0x0001 success >> \end{verbatim} - -Step 3: setting arp entries - +On the host we setup the ARP entries: \begin{verbatim} root@ESPRIMO-P956:~# ip neigh add 10.0.0.6 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 root@ESPRIMO-P956:~# ip neigh add 10.0.0.4 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 - \end{verbatim} -Step 3: generating test packets, expecting 4 packets to show up on -enp2s0f0: - -%\tvb +And then we generate test packets and expect 4 packets to show up on +enp2s0f0. +The following \texttt{tcpdump} output shows the expected packets +arriving on enp2s0f0: \begin{verbatim} nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode @@ -203,17 +138,12 @@ listening on enp2s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:49:29.222340 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64 10:49:29.222418 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64 \end{verbatim} -%\tve - -Result: sucess - - +% ok % ---------------------------------------------------------------------- \subsection{Test 2: IPv6 egress} -Similar to the IPv4 setting before, just for IPv6. - -Step 1: getting IP address values - +This test shows how setting the egress port based on the IPv6 address +works with the NetPFGA. Similar to the previous test, we first the +the Integer values of the IPv6 addresses: \begin{verbatim} >>> int(ipaddress.IPv6Address(u"2001:db8:42::4")) 42540766411362381960998550477184434180L @@ -221,11 +151,8 @@ Step 1: getting IP address values 42540766411362381960998550477184434182L >>> int(ipaddress.IPv6Address(u"2001:db8:42::42")) 42540766411362381960998550477184434242L - \end{verbatim} - -Step 2: setting table entries - +After that we set the table entries: \begin{verbatim} >> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434182 => 64 0 0 0 0 fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)] @@ -265,16 +192,12 @@ READ 0x44020344 = 0x0001 success >> \end{verbatim} - -Step 3: setting neighbor entries - +On the host we set the IPv6 neighbor entries: \begin{verbatim} nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::6 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::4 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 \end{verbatim} - -Step 4: generating test packets - +And generate the test packets: \begin{verbatim} nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::6 PING 2001:db8:42::6(2001:db8:42::6) 56 data bytes @@ -287,129 +210,49 @@ listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes 11:30:17.287599 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 1, length 64 11:30:18.310178 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64 11:30:18.310258 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64 - \end{verbatim} - -Result: success, packet is seen twice. - -% ------------------------------------------------------------------------------ -\subsection{Test 3: NAT64} - -Additionally to the preparations done in test 1 and 2, the following -steps were taken: - -Step 1: getting IP address values via Python - -\begin{verbatim} ->>> int(ipaddress.IPv6Address(u"2001:db8:42::2a")) -42540766411362381960998550477184434218L - ->>> int(ipaddress.IPv6Address(u"2001:db8:42::")) -42540766411362381960998550477184434176L - ->>> int(ipaddress.IPv6Address(u"2001:db8:42::a00:2a")) -42540766411362381960998550477352206378 - ->>> int(ipaddress.IPv4Address(u"10.0.0.0")) -167772160 - - ->>> int(ipaddress.IPv4Address(u"10.0.0.66")) -167772226 - -\end{verbatim} - -Add table entry for 2001:db8:42:2a to be translated to 10.0.0.42: - -\begin{verbatim} ->> table_cam_add_entry realmain_nat64_0 realmain.nat64_static 42540766411362381960998550477184434218 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0 -fields = [(u'hit', 1), (u'action_run', 3), (u'v6_src', 128), (u'v4_dst', 32), (u'nat64_prefix', 128), (u'table_id', 16)] -action_name = TopPipe.realmain.nat64_static -field_vals = [2, '42540766411362381960998550477184434176', '167772160', '42540766411362381960998550477184434176', '0'] -CAM_Init_ValidateContext() - done -WROTE 0x44020050 = 0x002a -WROTE 0x44020054 = 0x0000 -WROTE 0x44020058 = 0x420000 -WROTE 0x4402005c = 0x20010db8 -WROTE 0x44020080 = 0x0000 -WROTE 0x44020084 = 0x0000 -WROTE 0x44020088 = 0x0000 -WROTE 0x4402008c = 0xdb80042 -WROTE 0x44020090 = 0x2001 -WROTE 0x44020094 = 0x0a00 -WROTE 0x44020098 = 0x0000 -WROTE 0x4402009c = 0x0000 -WROTE 0x440200a0 = 0xdb80042 -WROTE 0x440200a4 = 0x22001 -READ 0x44020044 = 0x0001 -WROTE 0x44020040 = 0x0001 -READ 0x44020044 = 0x0001 -READ 0x44020044 = 0x0001 -success ->> - -\end{verbatim} - -Add table entry for 2001:db8:42::a00:2a to be translated to 10.0.0.66: -\begin{verbatim} -table_cam_add_entry realmain_nat64_0 realmain.nat64_static 42540766411362381960998550477352206378 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0 -\end{verbatim} - -Add table entry for 10.0.0.66 to be translated to 2001:db8:42:42: - -\begin{verbatim} ->> table_cam_add_entry realmain_nat46_0 realmain.nat46_static 167772226 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0 -fields = [(u'hit', 1), (u'action_run', 3), (u'v6_src', 128), (u'v4_dst', 32), (u'nat64_prefix', 128), (u'table_id', 16)] -action_name = TopPipe.realmain.nat46_static -field_vals = [2, '42540766411362381960998550477184434176', '167772160', '42540766411362381960998550477184434176', '0'] -CAM_Init_ValidateContext() - done -WROTE 0x44020150 = 0xa000042 -WROTE 0x44020180 = 0x0000 -WROTE 0x44020184 = 0x0000 -WROTE 0x44020188 = 0x0000 -WROTE 0x4402018c = 0xdb80042 -WROTE 0x44020190 = 0x2001 -WROTE 0x44020194 = 0x0a00 -WROTE 0x44020198 = 0x0000 -WROTE 0x4402019c = 0x0000 -WROTE 0x440201a0 = 0xdb80042 -WROTE 0x440201a4 = 0x22001 -READ 0x44020144 = 0x0001 -WROTE 0x44020140 = 0x0001 -READ 0x44020144 = 0x0001 -READ 0x44020144 = 0x0001 -success ->> -\end{verbatim} - - -Step 3: setting neighbor entries - -\begin{verbatim} -sudo ip neigh add 10.0.0.66 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0 -sudo ip -6 neigh add 2001:db8:42::2a lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 -sudo ip -6 neighbor add 2001:db8:42::a00:2a lladdr f8:f2:1e:09:62:d0 dev enp2s0f1 - -\end{verbatim} - -Step 4: ping test should translate, but fail with wrong checksum: - +The packets are successfully seen by tcpdump. % ---------------------------------------------------------------------- -\chapter{\label{appendixB}NetFPGA Logs} -Majority of the log files are stored inside the source code directory -stored at ``netpfga/logs''. It follows a selection of excerpts -of log files that might be relevant for reproducing the work. +\section{\label{appendix:bmv2}P4/BMV2 environment and tests} +All BMV2 based compilations were made with the following compiler: +\begin{verbatim} +p4@ubuntu:~$ p4c --version +p4c 0.5 (SHA: 5ae30ee) +\end{verbatim} +The installation is based on the vagrant files that were provided in +the ``Advanced Topics in +Communication Networks Fall 2018'' course of +ETHZ (\url{https://adv-net.ethz.ch/2018/}) and contains p4tools as +well as all utilities that came with the vagrant installation. +% ok % ---------------------------------------------------------------------- - -\section{\label{chapterB:netpfga-flasherror}NetFPGA Flash Errors} +For running the diff based checksum code, the following steps are +necessary: First compile the p4 code and then start the switch, both +with \texttt{p4run}. +\begin{verbatim} +cd ~/master-thesis/p4app +sudo p4run --config nat64-diff.json +\end{verbatim} +Then with starting the controller the required table entries will +\begin{verbatim} +cd ~/master-thesis/p4app +sudo python ./controller.py --mode range_router +\end{verbatim} +% ---------------------------------------------------------------------- +\chapter{\label{appendix:netpfgalogs}NetFPGA Logs} +The log files of the NetFPGA compilations are stored inside +the source code directory stored at \texttt{netpfga/logs}. +It follows a selection of excerpts of log files that might +be relevant for reproducing the work. +% ---------------------------------------------------------------------- +\section{\label{appendix:netpfgalogs:flasherror}NetFPGA Flash Errors} Sometimes flashing bitfiles to the NetFPGA will fail. A random amount of reboots (1 to 3) and a random amount of reflashing will fix this problem. - Below can be found the log output from the flashing process. - \begin{verbatim} -nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh" +nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ +sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh" ++ which vivado + xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado + bitimage=minip4.bit @@ -453,20 +296,19 @@ nf3: ERROR while getting interface flags: No such device + bash config_writes.sh \end{verbatim} +% ok % ---------------------------------------------------------------------- - -\section{\label{chapterB:netpfga-flashok}NetFPGA Flash Success} +\section{\label{appendix:netpfgalogs:flashsuccess}NetFPGA Flash Success} A successful flashing process also emits a couple of errors, however the message ``fpga configuration failed. DONE PIN is not HIGH'' and its succeeding lines are missing, as seen below. -After that in all cases a reboot is required; the PCI rescan in no -tested case showed the nf devices. - - +After that in all cases a reboot is required; the PCI rescan in none +of our test cases re enabled the nf devices. \begin{verbatim} nico@nsg-System:~$ cd $NF_DESIGN_DIR/bitfiles/ -nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh" +nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ +sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh" ++ which vivado + xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado + bitimage=minip4.bit @@ -507,11 +349,11 @@ nf3: ERROR while getting interface flags: No such device + bash config_writes.sh nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ \end{verbatim} +% ok % ---------------------------------------------------------------------- -\section{\label{chapterB:netpfga-kernelmodule}NetFPGA Kernel module} +\section{\label{appendix:netfpgalogs:kernelmodule}NetFPGA Kernel module} After a successful flash, loading the kernel module will enable nf devices to appear in the operating system. - \begin{verbatim} nico@nsg-System:~$ ip l 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 @@ -579,929 +421,11 @@ nico@nsg-System:~$ \end{verbatim} % ---------------------------------------------------------------------- -\section{\label{chapterB:netpfga-nftraffic}NetFPGA misses packets on nf*} -While the nf devices appear in the operating system, packets emitted -by the netpfga cannot be sniffed on the nf interfaces -directly. Instead one has to sniff packets on a physical network card -that is connected to the specific output port. - +\section{\label{appendix:netfpgalogs:compilelogs}NetFPGA compile logs} % ---------------------------------------------------------------------- -\section{\label{chapterB:netpfga-kernelmodule}NetFPGA Kernel module} - -%--------------------------------------------------------------------------------------------------------- -\chapter{\label{benchmark}Benchmark Logs} -% ---------------------------------------------------------------------- - -\section{\label{benchmark:iperf}iperf} -Omitting startup time - -\section{\label{benchmark:general}General} - -MTU setting to 1500, as netpfga doesn't support jumbo frames - - -iperf3, iperf 3.0.11 - -50 parallel = 2x 100% cpu usage -40 parallel = 100%, 70% cpu usage -30 parallel = 70%-100, 70% cpu usage - -Turning back on checksum offloading (see below) - -30 parallel = 70%, 30% cpu usage - - -\begin{verbatim} -root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: on - tx-checksum-ip-generic: on - tx-checksum-sctp: on -tcp-segmentation-offload: on - tx-tcp-segmentation: on - tx-tcp6-segmentation: on -root@ESPRIMO-P956:~# -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: on - tx-checksum-ip-generic: on - tx-checksum-sctp: on -tcp-segmentation-offload: on - tx-tcp-segmentation: on - tx-tcp6-segmentation: on -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# - -\end{verbatim} - -Results into - -\begin{verbatim} - -root@ESPRIMO-P956:~# ethtool -k enp2s0f0 -Features for enp2s0f0: -Cannot get device udp-fragmentation-offload settings: Operation not supported -rx-checksumming: on -tx-checksumming: on - tx-checksum-ipv4: off [fixed] - tx-checksum-ip-generic: on - tx-checksum-ipv6: off [fixed] - tx-checksum-fcoe-crc: on [fixed] - tx-checksum-sctp: on -scatter-gather: on - tx-scatter-gather: on - tx-scatter-gather-fraglist: off [fixed] -tcp-segmentation-offload: on - tx-tcp-segmentation: on - tx-tcp-ecn-segmentation: off [fixed] - tx-tcp-mangleid-segmentation: off - tx-tcp6-segmentation: on -udp-fragmentation-offload: off -generic-segmentation-offload: on -generic-receive-offload: on -large-receive-offload: off -rx-vlan-offload: on -tx-vlan-offload: on -ntuple-filters: off -receive-hashing: on -highdma: on [fixed] -rx-vlan-filter: on -vlan-challenged: off [fixed] -tx-lockless: off [fixed] -netns-local: off [fixed] -tx-gso-robust: off [fixed] -tx-fcoe-segmentation: on [fixed] -tx-gre-segmentation: on -tx-gre-csum-segmentation: on -tx-ipxip4-segmentation: on -tx-ipxip6-segmentation: on -tx-udp_tnl-segmentation: on -tx-udp_tnl-csum-segmentation: on -tx-gso-partial: on -tx-sctp-segmentation: off [fixed] -tx-esp-segmentation: off [fixed] -fcoe-mtu: off [fixed] -tx-nocache-copy: off -loopback: off [fixed] -rx-fcs: off [fixed] -rx-all: off -tx-vlan-stag-hw-insert: off [fixed] -rx-vlan-stag-hw-parse: off [fixed] -rx-vlan-stag-filter: off [fixed] -l2-fwd-offload: off -hw-tc-offload: off -esp-hw-offload: off [fixed] -esp-tx-csum-hw-offload: off [fixed] -rx-udp_tunnel-port-offload: off -root@ESPRIMO-P956:~# ethtool -k enp2s0f1 -Features for enp2s0f1: -Cannot get device udp-fragmentation-offload settings: Operation not supported -rx-checksumming: on -tx-checksumming: on - tx-checksum-ipv4: off [fixed] - tx-checksum-ip-generic: on - tx-checksum-ipv6: off [fixed] - tx-checksum-fcoe-crc: on [fixed] - tx-checksum-sctp: on -scatter-gather: on - tx-scatter-gather: on - tx-scatter-gather-fraglist: off [fixed] -tcp-segmentation-offload: on - tx-tcp-segmentation: on - tx-tcp-ecn-segmentation: off [fixed] - tx-tcp-mangleid-segmentation: off - tx-tcp6-segmentation: on -udp-fragmentation-offload: off -generic-segmentation-offload: on -generic-receive-offload: on -large-receive-offload: off -rx-vlan-offload: on -tx-vlan-offload: on -ntuple-filters: off -receive-hashing: on -highdma: on [fixed] -rx-vlan-filter: on -vlan-challenged: off [fixed] -tx-lockless: off [fixed] -netns-local: off [fixed] -tx-gso-robust: off [fixed] -tx-fcoe-segmentation: on [fixed] -tx-gre-segmentation: on -tx-gre-csum-segmentation: on -tx-ipxip4-segmentation: on -tx-ipxip6-segmentation: on -tx-udp_tnl-segmentation: on -tx-udp_tnl-csum-segmentation: on -tx-gso-partial: on -tx-sctp-segmentation: off [fixed] -tx-esp-segmentation: off [fixed] -fcoe-mtu: off [fixed] -tx-nocache-copy: off -loopback: off [fixed] -rx-fcs: off [fixed] -rx-all: off -tx-vlan-stag-hw-insert: off [fixed] -rx-vlan-stag-hw-parse: off [fixed] -rx-vlan-stag-filter: off [fixed] -l2-fwd-offload: off -hw-tc-offload: off -esp-hw-offload: off [fixed] -esp-tx-csum-hw-offload: off [fixed] -rx-udp_tunnel-port-offload: off -root@ESPRIMO-P956:~# - -\end{verbatim} - -\section{\label{benchmark:netpfga}NetFPGA} -iperf3-tcp-listening-v4 connected by v6 -\begin{verbatim} -nico@ESPRIMO-P956:~$ iperf3 -p 2345 -4 -B 10.0.0.42 -s ------------------------------------------------------------ -Server listening on 2345 ------------------------------------------------------------ -Accepted connection from 10.0.0.66, port 50900 -[ 5] local 10.0.0.42 port 2345 connected to 10.0.0.66 port 50902 -[ ID] Interval Transfer Bandwidth -[ 5] 0.00-1.00 sec 693 MBytes 5.81 Gbits/sec -[ 5] 1.00-2.00 sec 645 MBytes 5.41 Gbits/sec -[ 5] 2.00-3.00 sec 644 MBytes 5.40 Gbits/sec -[ 5] 3.00-4.00 sec 868 MBytes 7.28 Gbits/sec -[ 5] 4.00-5.00 sec 853 MBytes 7.16 Gbits/sec -[ 5] 5.00-6.00 sec 913 MBytes 7.66 Gbits/sec -[ 5] 6.00-7.00 sec 774 MBytes 6.49 Gbits/sec -[ 5] 7.00-8.00 sec 641 MBytes 5.38 Gbits/sec -[ 5] 8.00-9.00 sec 911 MBytes 7.64 Gbits/sec -[ 5] 9.00-10.00 sec 733 MBytes 6.15 Gbits/sec -[ 5] 10.00-10.04 sec 25.8 MBytes 5.38 Gbits/sec -- - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bandwidth Retr -[ 5] 0.00-10.04 sec 7.52 GBytes 6.43 Gbits/sec 14 sender -[ 5] 0.00-10.04 sec 7.52 GBytes 6.43 Gbits/sec receiver ------------------------------------------------------------ -Server listening on 2345 ------------------------------------------------------------ - -nico@ESPRIMO-P956:~$ iperf3 -6 -p 2345 -c 2001:db8:42::a00:2a -Connecting to host 2001:db8:42::a00:2a, port 2345 -[ 4] local 2001:db8:42::42 port 50902 connected to 2001:db8:42::a00:2a port 2345 -[ ID] Interval Transfer Bandwidth Retr Cwnd -[ 4] 0.00-1.00 sec 719 MBytes 6.03 Gbits/sec 10 449 KBytes -[ 4] 1.00-2.00 sec 645 MBytes 5.41 Gbits/sec 0 449 KBytes -[ 4] 2.00-3.00 sec 644 MBytes 5.40 Gbits/sec 0 449 KBytes -[ 4] 3.00-4.00 sec 878 MBytes 7.36 Gbits/sec 0 449 KBytes -[ 4] 4.00-5.00 sec 859 MBytes 7.20 Gbits/sec 0 449 KBytes -[ 4] 5.00-6.00 sec 910 MBytes 7.64 Gbits/sec 0 449 KBytes -[ 4] 6.00-7.00 sec 758 MBytes 6.36 Gbits/sec 0 449 KBytes -[ 4] 7.00-8.00 sec 658 MBytes 5.52 Gbits/sec 0 449 KBytes -[ 4] 8.00-9.00 sec 906 MBytes 7.60 Gbits/sec 4 449 KBytes -[ 4] 9.00-10.00 sec 724 MBytes 6.07 Gbits/sec 0 449 KBytes -- - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bandwidth Retr -[ 4] 0.00-10.00 sec 7.52 GBytes 6.46 Gbits/sec 14 sender -[ 4] 0.00-10.00 sec 7.52 GBytes 6.46 Gbits/sec receiver - -iperf Done. -nico@ESPRIMO-P956:~$ - -\end{verbatim} - -listening on v6, connecting from v4: - -\begin{verbatim} -nico@ESPRIMO-P956:~$ iperf3 -p 2345 -6 -B 2001:db8:42::42 -s ------------------------------------------------------------ -Server listening on 2345 ------------------------------------------------------------ -Accepted connection from 2001:db8:42::a00:2a, port 47520 -[ 5] local 2001:db8:42::42 port 2345 connected to 2001:db8:42::a00:2a port 47522 -[ ID] Interval Transfer Bandwidth -[ 5] 0.00-1.00 sec 1.02 GBytes 8.73 Gbits/sec -[ 5] 1.00-2.00 sec 879 MBytes 7.38 Gbits/sec -[ 5] 2.00-3.00 sec 859 MBytes 7.20 Gbits/sec -[ 5] 3.00-4.00 sec 1.02 GBytes 8.78 Gbits/sec -[ 5] 4.00-5.00 sec 1.04 GBytes 8.89 Gbits/sec -[ 5] 5.00-6.00 sec 1.05 GBytes 9.00 Gbits/sec -[ 5] 6.00-7.00 sec 1.03 GBytes 8.89 Gbits/sec -[ 5] 7.00-8.00 sec 1.04 GBytes 8.91 Gbits/sec -[ 5] 8.00-9.00 sec 1.03 GBytes 8.84 Gbits/sec -[ 5] 9.00-10.00 sec 953 MBytes 7.99 Gbits/sec -[ 5] 10.00-10.04 sec 38.6 MBytes 7.81 Gbits/sec -- - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bandwidth Retr -[ 5] 0.00-10.04 sec 9.89 GBytes 8.46 Gbits/sec 151 sender -[ 5] 0.00-10.04 sec 9.89 GBytes 8.46 Gbits/sec receiver ------------------------------------------------------------ -Server listening on 2345 ------------------------------------------------------------ - -nico@ESPRIMO-P956:~$ iperf3 -4 -p 2345 -c 10.0.0.66 -Connecting to host 10.0.0.66, port 2345 -[ 4] local 10.0.0.42 port 47522 connected to 10.0.0.66 port 2345 -[ ID] Interval Transfer Bandwidth Retr Cwnd -[ 4] 0.00-1.00 sec 1.06 GBytes 9.10 Gbits/sec 53 208 KBytes -[ 4] 1.00-2.00 sec 867 MBytes 7.27 Gbits/sec 6 379 KBytes -[ 4] 2.00-3.00 sec 870 MBytes 7.29 Gbits/sec 0 423 KBytes -[ 4] 3.00-4.00 sec 1.02 GBytes 8.77 Gbits/sec 37 364 KBytes -[ 4] 4.00-5.00 sec 1.04 GBytes 8.91 Gbits/sec 1 450 KBytes -[ 4] 5.00-6.00 sec 1.05 GBytes 8.98 Gbits/sec 0 462 KBytes -[ 4] 6.00-7.00 sec 1.04 GBytes 8.92 Gbits/sec 30 324 KBytes -[ 4] 7.00-8.00 sec 1.04 GBytes 8.88 Gbits/sec 0 471 KBytes -[ 4] 8.00-9.00 sec 1.03 GBytes 8.86 Gbits/sec 10 452 KBytes -[ 4] 9.00-10.00 sec 947 MBytes 7.94 Gbits/sec 14 409 KBytes -- - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bandwidth Retr -[ 4] 0.00-10.00 sec 9.89 GBytes 8.49 Gbits/sec 151 sender -[ 4] 0.00-10.00 sec 9.89 GBytes 8.49 Gbits/sec receiver - -iperf Done. -nico@ESPRIMO-P956:~$ - -\end{verbatim} -% \section{\label{benchmark:bmv2}BMV2} -%--------------------------------------------------------------------------------------------------------- -\section{\label{benchmark:tayga}Tayga} - - -\begin{verbatim} -ii tayga 0.9.2-6 amd64 userspace stateless NAT64 -\end{verbatim} - -Setting up IPv4 networking - -\begin{verbatim} -[15:12] nsg-System:~# ip addr add 10.0.0.77/24 dev eth1 -[15:12] nsg-System:~# ip l s eth1 up - -nico@ESPRIMO-P956:~$ ~/master-thesis/bin/init_ipv4_esprimo.sh -nico@ESPRIMO-P956:~$ cat ~/master-thesis/bin/init_ipv4_esprimo.sh -#!/bin/sh - -sudo ip addr add 10.0.0.42/24 dev enp2s0f0 -sudo ip link set enp2s0f0 up - -nico@ESPRIMO-P956:~$ sudo ip route add 10.0.1.0/24 via 10.0.0.77 - -\end{verbatim} - -Verify networking works: - -\begin{verbatim} -[15:12] nsg-System:~# ping 10.0.0.42 -PING 10.0.0.42 (10.0.0.42) 56(84) bytes of data. -64 bytes from 10.0.0.42: icmp_seq=1 ttl=64 time=0.304 ms -64 bytes from 10.0.0.42: icmp_seq=2 ttl=64 time=0.097 ms -^C ---- 10.0.0.42 ping statistics --- -2 packets transmitted, 2 received, 0% packet loss, time 1011ms -rtt min/avg/max/mdev = 0.097/0.200/0.304/0.104 ms -[15:12] nsg-System:~# - -\end{verbatim} - -Setting up IPv6 networking - -\begin{verbatim} -nico@ESPRIMO-P956:~$ ip addr show dev enp2s0f1 -13: enp2s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 - link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff - inet6 2001:db8:42::42/64 scope global - valid_lft forever preferred_lft forever - inet6 fe80::faf2:1eff:fe09:62d1/64 scope link - valid_lft forever preferred_lft forever -nico@ESPRIMO-P956:~$ sudo ip route add 2001:db8:23::/96 via 2001:db8:42::77 - -[15:12] nsg-System:~# ip addr add 2001:db8:42::77/64 dev eth2 -[15:15] nsg-System:~# ip link set eth2 up - -\end{verbatim} - -Verify IPv6 networking works: - -\begin{verbatim} -nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::77 -PING 2001:db8:42::77(2001:db8:42::77) 56 data bytes -64 bytes from 2001:db8:42::77: icmp_seq=1 ttl=64 time=0.169 ms -64 bytes from 2001:db8:42::77: icmp_seq=2 ttl=64 time=0.153 ms - ---- 2001:db8:42::77 ping statistics --- -2 packets transmitted, 2 received, 0% packet loss, time 1010ms -rtt min/avg/max/mdev = 0.153/0.161/0.169/0.008 ms -nico@ESPRIMO-P956:~$ - -\end{verbatim} - -Enabling IPv6 and IPv4 forwarding: - -\begin{verbatim} -[15:16] nsg-System:~# sysctl -w net.ipv6.conf.all.forwarding=1 -net.ipv6.conf.all.forwarding = 1 - -[15:20] nsg-System:~# sysctl -w net.ipv4.ip_forward=1 -net.ipv4.ip_forward = 1 - -\end{verbatim} - - -Testing NAT64 in tayga - -\begin{verbatim} -nico@ESPRIMO-P956:~$ ping -c2 10.0.1.42 -PING 10.0.1.42 (10.0.1.42) 56(84) bytes of data. -64 bytes from 10.0.1.42: icmp_seq=1 ttl=61 time=0.356 ms -64 bytes from 10.0.1.42: icmp_seq=2 ttl=61 time=0.410 ms - ---- 10.0.1.42 ping statistics --- -2 packets transmitted, 2 received, 0% packet loss, time 1019ms -rtt min/avg/max/mdev = 0.356/0.383/0.410/0.027 ms -nico@ESPRIMO-P956:~$ - -nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1 -tcpdump: verbose output suppressed, use -v or -vv for full protocol decode -listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes -15:21:39.851057 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 1, length 64 -15:21:39.851124 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 1, length 64 -15:21:40.870448 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 2, length 64 -15:21:40.870507 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 2, length 64 -^C -4 packets captured -4 packets received by filter -0 packets dropped by kernel -nico@ESPRIMO-P956:~$ - -\end{verbatim} - -Testing NAT64 (v6 to v4) - -\begin{verbatim} -nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:23::a00:2a -PING 2001:db8:23::a00:2a(2001:db8:23::a00:2a) 56 data bytes -64 bytes from 2001:db8:23::a00:2a: icmp_seq=1 ttl=61 time=0.240 ms -64 bytes from 2001:db8:23::a00:2a: icmp_seq=2 ttl=61 time=0.400 ms - ---- 2001:db8:23::a00:2a ping statistics --- -2 packets transmitted, 2 received, 0% packet loss, time 1003ms -rtt min/avg/max/mdev = 0.240/0.320/0.400/0.080 ms -nico@ESPRIMO-P956:~$ - -\end{verbatim} - -\subsection{\label{benchmark:tayga:tcp}Tayga/TCP} - -Tayga running at 100% cpu load, - -v4->v6 tcp -delivering -3.36 gbit/s at P1 -3.30 Gbit/s at P20 -3.11 gbit/s at P50 - -v6->v4 tcp -P1: 3.02 Gbit/s -P20: 3.28 gbit/s -P50: 2.85 gbit/s - -Commands: - -\begin{verbatim} - -Server always: iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-tayga-v4tov6server-P50 - - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -c 10.0.1.42 -T taygav4tov6tcpP1 | tee iperf-tayga-v4tov6server-client - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -c 10.0.1.42 -T taygav4tov6tcpP20 | tee iperf-tayga-v4tov6server-client-P20 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -c 10.0.1.42 -T taygav4tov6tcpP50 | tee iperf-tayga-v4tov6server-client-P50 - - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-P1 - -\end{verbatim} - - -Testing v6->v4 - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-P20 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -c 2001:db8:23::a00:2a -T taygav6tov4tcpP1 | tee iperf-tayga-v6tov4-client-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P20 -c 2001:db8:23::a00:2a -T taygav6tov4tcpP20 | tee iperf-tayga-v6tov4-client-P20 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -c 2001:db8:23::a00:2a -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-P50 - -\end{verbatim} - -UDP v6->v4, again 100% cpu - -P1: 5.81 gbit/s -P20: 9.40 gbit/s -P50: 19.6 gbits/sec - -On the line only ca. 3600 mbit/s seen - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-udp-P1 - - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -u -b10000m -c 2001:db8:23::a00:2a -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-udp-P1 - - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -u -b10000m -c 2001:db8:23::a00:2a -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-udp-P50 - - -\end{verbatim} - -Messsages from server: - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-tayga-v6tov4-server-udp-P1 -iperf3: OUT OF ORDER - incoming packet = 198902 and received packet = 198904 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 441615 and received packet = 441617 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 441616 and received packet = 441618 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 567495 and received packet = 567501 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 567496 and received packet = 567501 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 567497 and received packet = 567501 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 567499 and received packet = 567503 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 567500 and received packet = 567503 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 567502 and received packet = 567503 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 631160 and received packet = 631164 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 631161 and received packet = 631164 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 631162 and received packet = 631165 AND SP = 5 -iperf3: OUT OF ORDER - incoming packet = 631163 and received packet = 631165 AND SP = 5 - -\end{verbatim} - - -UDP v4->v6, again 100% cpu - -P1: 8.26 gbit/s [atop: ~2500 Mbit/s per direction] -P20: 9.92 Gbits/sec [atop: ~2500 Mbit/s per direction] -P50: 19.3 gbit/s [atop: ~2500 Mbit/s per direction] - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-tayga-v4tov6-server-udp-P1 - - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP1 | tee iperf-tayga-v4tov6server-client-udp-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP20 | tee iperf-tayga-v4tov6server-client-udp-P20 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP50 | tee iperf-tayga-v4tov6server-client-udp-P50 - -\end{verbatim} - -%--------------------------------------------------------------------------------------------------------- - -\section{\label{benchmark:jool}Jool} -\subsection{\label{benchmark:jool:setup}Jool Setup} - -Installation of 4.0.1 from \url{https://www.jool.mx/en/download.html}. - -\begin{verbatim} -nico@nsg-System:~$ wget https://github.com/NICMx/Jool/releases/download/v4.0.1/jool_4.0.1.tar.gz -nico@nsg-System:~$ tar xvfz jool_4.0.1.tar.gz -nico@nsg-System:~$ cd jool-4.0.1/ -nico@nsg-System:~/jool-4.0.1$ sudo apt install linux-headers-$(uname -r) -nico@nsg-System:~/jool-4.0.1$ sudo apt install libnl-genl-3-dev - -\end{verbatim} - -xtables cannot be found: - -\begin{verbatim} -nico@nsg-System:~/jool-4.0.1$ sudo apt install libxtables-dev -Reading package lists... Done -Building dependency tree -Reading state information... Done -E: Unable to locate package libxtables-dev -nico@nsg-System:~/jool-4.0.1$ - -\end{verbatim} - -Does not compile without: - -\begin{verbatim} -checking for library containing argp_parse... none required -checking for pkg-config... /usr/bin/pkg-config -checking pkg-config is at least version 0.9.0... yes -checking for LIBNLGENL3... yes -checking for XTABLES... no -configure: error: Package requirements (xtables) were not met: - -No package 'xtables' found - -Consider adjusting the PKG_CONFIG_PATH environment variable if you -installed software in a non-standard prefix. - -Alternatively, you may set the environment variables XTABLES_CFLAGS -and XTABLES_LIBS to avoid the need to call pkg-config. -See the pkg-config man page for more details. -nico@nsg-System:~/jool-4.0.1$ - -\end{verbatim} - -Trying different package: - -\begin{verbatim} -nico@nsg-System:~/jool-4.0.1$ sudo apt install iptables-dev - -\end{verbatim} - -Compiles! - -\begin{verbatim} -nico@nsg-System:~/jool-4.0.1$ sudo make install - -\end{verbatim} - -\subsection{\label{benchmark:jool:config}Jool Configuration} - -Loading module: -\begin{verbatim} -nico@nsg-System:~/jool-4.0.1$ sudo modprobe jool_siit - - -\end{verbatim} - -enabling forwarding: -\begin{verbatim} -sysctl -w net.ipv4.conf.all.forwarding=1 -sysctl -w net.ipv6.conf.all.forwarding=1 -\end{verbatim} - -Mapping configuration: - -\begin{verbatim} -nico@nsg-System:~/jool-4.0.1$ sudo jool_siit instance add example --iptables --pool6 2001:db8:23::/96 - -nico@nsg-System:~/jool-4.0.1$ sudo ip6tables -t mangle -A PREROUTING \ - -s 2001:db8:42::/64 -d 2001:db8:23::/96 -j JOOL_SIIT --instance example - -nico@nsg-System:~/jool-4.0.1$ sudo iptables -t mangle -A PREROUTING \ - -s 10.0.0.0/24 -j JOOL_SIIT --instance example - -\end{verbatim} - - - -Debugging: - -\begin{verbatim} -[16:39] nsg-System:~# lsmod| grep jool -jool_siit 147456 2 -x_tables 40960 5 jool_siit,ip6_tables,ip_tables,ip6table_mangle,iptable_mangle -[16:39] nsg-System:~# - -[16:41] nsg-System:~# jool_siit -i example stats display --explain -JSTAT64_DST: 276 -Translations cancelled: IPv6 packet's destination address did not match pool6 nor any EAMT entries, or the resulting address was blacklist4ed. - - -\end{verbatim} - -Try 2 w/ eamt: - -\begin{verbatim} -[16:53] nsg-System:~# modprobe jool_siit -[16:54] nsg-System:~# jool_siit instance add "example" --iptables -[16:54] nsg-System:~# jool_siit -i example eamt add 2001:db8:42::/120 10.0.1.0/24 -[16:55] nsg-System:~# jool_siit -i example eamt add 2001:db8:23::/120 10.0.0.0/24 -[16:57] nsg-System:~# ip6tables -t mangle -A PREROUTING -s 2001:db8:42::/120 -d 2001:db8:23::/120 -j JOOL_SIIT --instance example -[16:57] nsg-System:~# iptables -t mangle -A PREROUTING -s 10.0.0.0/24 -d 10.0.1.0/24 -j JOOL_SIIT --instance example -[16:57] nsg-System:~# - -\end{verbatim} - -Testing NAT64: - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ ping6 2001:db8:23::2a -PING 2001:db8:23::2a(2001:db8:23::2a) 56 data bytes -64 bytes from 2001:db8:23::2a: icmp_seq=1 ttl=63 time=0.199 ms -64 bytes from 2001:db8:23::2a: icmp_seq=2 ttl=63 time=0.282 ms -64 bytes from 2001:db8:23::2a: icmp_seq=3 ttl=63 time=0.186 ms -^C ---- 2001:db8:23::2a ping statistics --- -3 packets transmitted, 3 received, 0% packet loss, time 2040ms -rtt min/avg/max/mdev = 0.186/0.222/0.282/0.044 ms -nico@ESPRIMO-P956:~/master-thesis/iperf$ ping 10.0.1.66 -PING 10.0.1.66 (10.0.1.66) 56(84) bytes of data. -64 bytes from 10.0.1.66: icmp_seq=1 ttl=63 time=0.218 ms -64 bytes from 10.0.1.66: icmp_seq=2 ttl=63 time=0.281 ms -64 bytes from 10.0.1.66: icmp_seq=3 ttl=63 time=0.280 ms -^C ---- 10.0.1.66 ping statistics --- -3 packets transmitted, 3 received, 0% packet loss, time 2051ms -rtt min/avg/max/mdev = 0.218/0.259/0.281/0.034 ms -nico@ESPRIMO-P956:~/master-thesis/iperf$ - -\end{verbatim} - - -\subsection{\label{benchmark:jool:benchmark}Jool Benchmarks} - -v4->v6 tcp - -P1: 8.24 gbit/s no cpu load visible -P20: 8.26 gbit/s iperf 42 + 10% cpu no cpu load visible -P50: 8.29 gbit/s - -v6->v4 tcp - -P1: 8.22 -P20: 8.22 15/60% iperf -P50: 8.23 iperf: 73/16% - -Commands: - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-jool-v4tov6-server-tcp-P50 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P20 -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P50 - -Other way: - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-jool-v6tov4-server-tcp-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P1 - -... - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -b0 -u -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P1 - - -\end{verbatim} - - -v4->v6 udp - -P1: 4.46 iperf 30% cpu nat64: 100% cpu -P20: 18.8 iperf 100% nat64: 100% -P50: 22.8 iperf 100% nat64: 100% atop: ca. 5gbit/s - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s | tee iperf-jool-v4tov6-server-udp-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -c 10.0.1.66 -u -b0 | tee iperf-jool-v4tov6-client-udp-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -c 10.0.1.66 -u -b0 | tee iperf-jool-v4tov6-client-udp-P20 - - -\end{verbatim} - -v6->v4 udp - -P1: 6.67 gbit/s iperf 50/50%, nat64: 100% cpu! -P20: 16.8 nat64: iperf: ? 100% cpu -P50: 20.5 Gbits/sec nat64: 100% (1 core) iperf: 100%/50% atop: 7000mbit/s in / out - -\begin{verbatim} - -\end{verbatim} - -Turning off offloading, redoing tcp: - -\begin{verbatim} -root@ESPRIMO-P956:~# ethtool -K enp2s0f0 gso off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# ethtool -K enp2s0f0 rx off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: off - tx-checksum-ip-generic: off - tx-checksum-sctp: off -tcp-segmentation-offload: off - tx-tcp-segmentation: off [requested on] - tx-tcp6-segmentation: off [requested on] -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: off - tx-checksum-ip-generic: off - tx-checksum-sctp: off -tcp-segmentation-offload: off - tx-tcp-segmentation: off [requested on] - tx-tcp6-segmentation: off [requested on] -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 gso off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# - -[17:26] nsg-System:~# ethtool -K eth1 tx off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: off - tx-checksum-ip-generic: off - tx-checksum-sctp: off -tcp-segmentation-offload: off - tx-tcp-segmentation: off [requested on] - tx-tcp6-segmentation: off [requested on] -[17:26] nsg-System:~# ethtool -K eth1 gso off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -[17:26] nsg-System:~# ethtool -K eth2 gso off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -[17:26] nsg-System:~# ethtool -K eth2 rx off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -[17:26] nsg-System:~# ethtool -K eth2 tx off -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: off - tx-checksum-ip-generic: off - tx-checksum-sctp: off -tcp-segmentation-offload: off - tx-tcp-segmentation: off [requested on] - tx-tcp6-segmentation: off [requested on] -[17:26] nsg-System:~# - -\end{verbatim} - -Retesting using -P50: - -Still no cpu load with tcp, 100% cpu load iperf - -result: 7.96 gbit/s - -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-jool-v6tov4-server-tcp-P50-no-offload -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P50-no-offload - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -u -b0 -c 10.0.0.66 | tee iperf-netpfga-v4tov6-client-udp-P20 - - - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-netfpga-v6tov4-server-tcp-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -c 2001:db8:42::a00:2a | tee iperf-netfpga-v6tov4-client-tcp-P1 - - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s | tee iperf-netfpga-v6tov4-server-udp-P1 - -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -b0 -u -c 2001:db8:42::a00:2a | tee iperf-netfpga-v6tov4-client-udp-P1 - -\end{verbatim} - -\subsection{\label{benchmark:netfpga:benchmark}NetPFGA Benchmarks} - -Only 1 test did have offloading on esprimo off, was redone - -v4->v6 tcp - -P1: 7.41 gbit/s iperf 50% / 20% -P1-offload-on-esprimo: 8.43 gbit/s -P20: 9.29 gbit/s iperf: 66/20% -P50: 9.29 gbit/s 84/42% iperf - -v4->v6 udp - -P1: 7.4gbit/s 100% iperf -P20: 17.7gbit/s iperf 100% atop: ca. 9800 Mbit/s -P50: 21.5 gbit/s iperf 100% ca. 9800 Mbit/s - - -v6->v4 tcp - -P1: 9.28 gbit/s atop ~9800 mbit/s iperf 44% -P20: 9.29 gbit/s atop ~9800 mbit/s iperf 70% -P50: 9.29 gbit/s atop ~9800 mbit/s iperf 90% - -v6->v4 udp - -P1: 7.96 gbit/s atop ~8200mbit/s iperf 70% -P20: 13.4 gbit/s atop 9800 mbit/s iperf 100% -P50: 19.0 gbit/s atop 9800 mbit/s iperf 100% - - - -Commands: -\begin{verbatim} -nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -u -b0 -c 10.0.0.66 | tee iperf-netpfga-v4tov6-client-udp-P1 - -\end{verbatim} - -After first netpfga, tcp v4->v6 p1 turned offloading on again - -\begin{verbatim} -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx-checksum-ipv6 on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Could not change any device features -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: on - tx-checksum-ip-generic: on - tx-checksum-sctp: on -tcp-segmentation-offload: on - tx-tcp-segmentation: on - tx-tcp6-segmentation: on -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# ethtool -K enp2s0f1 gso on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# ethtool -K enp2s0f0 gso on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -Actual changes: -tx-checksumming: on - tx-checksum-ip-generic: on - tx-checksum-sctp: on -tcp-segmentation-offload: on - tx-tcp-segmentation: on - tx-tcp6-segmentation: on -root@ESPRIMO-P956:~# ethtool -K enp2s0f0 rx on -Cannot get device udp-fragmentation-offload settings: Operation not supported -Cannot get device udp-fragmentation-offload settings: Operation not supported -root@ESPRIMO-P956:~# - -\end{verbatim} - -%Entries for the list of abbrevations: -% -%To generate the list of abbrevations, execute: -%makeindex Thesis.nlo -s nomencl.ist -o Thesis.nls -% -%If you are using TeXniCenter, specify: -%"%bm.nlo" -s nomencl.ist -o "%bm.nls" -%as beeing the argument list for makeindex. - -%--------------------------------------------------------------------------------------------------------- - -\chapter{\label{bufferssssss}Buffer} - -\section{NetFPGA compile errors} - -- infinite loop in installer +This section shows a compilation of of NetFPGA compile output and errors. +Unfound tbl files that are not correctly generated fail the compilation: \begin{verbatim} # Fix introduced for SDNet 2017.4 sed -i 's/xsim\.dir\/xsc\/dpi\.so/dpi\.so/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim.bash @@ -1513,14 +437,13 @@ cp src/*.tbl nf_sume_sdnet_ip/SimpleSumeSwitch/ cp: cannot stat 'src/*.tbl': No such file or directory make: *** [Makefile:23: cpp_test] Error 1 [23:12] loch:minip4% - \end{verbatim} - +Failure to generate an intermediate file: \begin{verbatim} ERROR: [XSIM 43-3409] Failed to compile generated C file xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.c. ERROR: [XSIM 43-3915] Encountered a fatal error. Cannot continue. Exiting... \end{verbatim} - +Failure to compile because libncurses.so.5 is missing: \begin{verbatim} /opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/clang -fPIC -c -std=gnu89 -nobuiltininc -nostdinc++ -w -Wl,--unres olved-symbols=ignore-in-object-files -fbracket-depth=1048576 -I/opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/../li @@ -1531,9 +454,9 @@ j/xsim_3.c" -O0 -sim -o "xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3. ERROR: [XSIM 43-3409] Failed to compile generated C file xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.c. ERROR: [XSIM 43-3915] Encountered a fatal error. Cannot continue. Exiting... [20:00] rainbow:SimpleSumeSwitch% - \end{verbatim} - +Failure to access txt files that were not correctly generated in a +different compilation step: \begin{verbatim} # Fix introduced for SDNet 2018.2 sed -i 's/glbl_sim/glbl/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash @@ -1543,9 +466,8 @@ cp testdata/*.txt nf_sume_sdnet_ip/SimpleSumeSwitch/ cp: cannot stat 'testdata/*.txt': No such file or directory make: *** [Makefile:17: all] Error 1 [15:46] rainbow:minip4% - \end{verbatim} - +Missing pcap files of non generated testdata causing compile abortion: \begin{verbatim} make -C testdata/ make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata' @@ -1567,9 +489,8 @@ make[1]: *** [Makefile:5: all] Error 1 make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata' make: *** [Makefile:32: frontend] Error 2 [15:47] rainbow:minip4% - \end{verbatim} - +Syntax errors due to incorrect generation of a python script: \begin{verbatim} update_compile_order: Time (s): cpu = 00:00:17 ; elapsed = 00:00:09 . Memory (MB): peak = 1995.594 ; gain = 0.016 ; free physic al = 21975 ; free virtual = 33161 @@ -1590,38 +511,40 @@ IndentationError: expected an indented block (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/tcl/simple_s ume_switch_sim.tcl" line 177) INFO: [Common 17-206] Exiting Vivado at Sat May 18 15:21:21 2019... - - \end{verbatim} - +Missing axi files don't abort the compilation process: (shortened for formatting) \begin{verbatim} -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.log': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_expect.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.axi': No such file or directory -=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ... using cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim'] +cp: cannot stat 'simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_0_stim.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_0_expected.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_1_stim.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_1_expected.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_2_stim.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_2_expected.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_3_stim.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_3_expected.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/dma_0_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/dma_0_expected.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/reg_stim.log': No such file or directory +cp: cannot stat 'simple_sume_switch/test/reg_expect.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/reg_stim.axi': No such file or directory +=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default + ... using cmd + ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', +'--sim', 'xsim'] \end{verbatim} - +Add Wave error during compilation: (shortened for formatting) \begin{verbatim} # add_wave $nf_sume_sdnet_ip/out_src_port # add_wave $nf_sume_sdnet_ip/out_dst_port # set const_reg_ip /top_tb/top_sim/nf_datapath_0/nf_sume_sdnet_wrapper_1/inst/SimpleSumeSwitch_inst/const_reg_rw_0/ # add_wave_divider {const reg extern signals} # add_wave $const_reg_ip -ERROR: [Wavedata 42-471] Note: Nothing was found for the following items: /top_tb/top_sim/nf_datapath_0/nf_sume_sdnet_wrapper_1/inst/SimpleSumeSwitch_inst/const_reg_rw_0/ +ERROR: [Wavedata 42-471] Note: Nothing was found for the following items: + /top_tb/top_sim/nf_datapath_0/nf_sume_sdnet_wrapper_1/inst/SimpleSumeSwitch_inst/const_reg_rw_0/ ERROR: [Common 17-39] 'add_wave' failed due to earlier errors. while executing @@ -1633,15 +556,16 @@ make: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sd 512 === Work directory is /tmp/nico/test/simple_sume_switch === Setting up test in /tmp/nico/test/simple_sume_switch/sim_switch_default -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_log.axi': No such file or directory -=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ... using cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim'] +cp: cannot stat 'simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory +cp: cannot stat 'simple_sume_switch/test/dma_0_log.axi': No such file or directory +=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ... +using cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim'] [15:31] rainbow:P4-NetFPGA% \end{verbatim} - +Compilation error failing to run ``connect\_bd\_intf\_net.'' \begin{verbatim} ERROR: [BD 41-171] The modes of the interface pins 'cfg_interrupt'(Slave) and 'pcie3_cfg_interrupt'(Slave) are incompatible. They cannot be connected. ERROR: [BD 5-3] Error: running connect_bd_intf_net. @@ -1660,45 +584,23 @@ ERROR: [Common 17-39] 'connect_bd_intf_net' failed due to earlier errors. while executing "source ./tcl/control_sub.tcl" (file "tcl/simple_sume_switch.tcl" line 89) - \end{verbatim} - +Compilation aborts due to missing IP: \begin{verbatim} - ### set NF_10G_INTERFACE3_BASEADDR $M07_BASEADDR ### set NF_10G_INTERFACE3_HIGHADDR $M07_HIGHADDR ### set NF_10G_INTERFACE3_SIZEADDR $M07_SIZEADDR ### set NF_RIFFA_DMA_BASEADDR $M08_BASEADDR ### set NF_RIFFA_DMA_HIGHADDR $M08_HIGHADDR ### set NF_RIFFA_DMA_SIZEADDR $M08_SIZEADDR -Wrote : +Wrote : + # create_ip -name nf_sume_sdnet -vendor NetFPGA -library NetFPGA -module_name nf_sume_sdnet_ip ERROR: [Coretcl 2-1134] No IP matching VLNV 'NetFPGA:NetFPGA:nf_sume_sdnet:*' was found. Please check your repository configuration. INFO: [Common 17-206] Exiting Vivado at Sat May 25 11:52:01 2019... - \end{verbatim} -\begin{verbatim} -# set_property compxlib.xsim_compiled_library_dir {} [current_project] [0/1819] -# set_property top_lib xil_defaultlib [get_filesets sim_1] -# update_compile_order -fileset sim_1 -update_compile_order: Time (s): cpu = 00:00:17 ; elapsed = 00:00:09 . Memory (MB): peak = 2003.578 ; gain = 8.004 ; free physical = 27661 ; free virtual = 33990 -loading libsume.. -Traceback (most recent call last): - File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py", line 42, in - import config_writes - File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/config_writes.py", line 7 - - ^ -IndentationError: expected an indented block - while executing -"exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py" - invoked from within -"set output [exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py]" - (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/tcl/simple_sume_switch_sim.tcl" line 177) -INFO: [Common 17-206] Exiting Vivado at Sat May 25 13:45:13 2019... -make: *** [Makefile:121: sim] Error 1 -\end{verbatim} - +Mismatch: a non-critical critical error that does not abort the compilation process \begin{verbatim} [SW] CAM_EnableDevice() - done [2420698] INFO: finished packet stimulus file @@ -1709,15 +611,7 @@ $finish called at time : 2735572 ps : File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/Testbench/Check.v" Line 120 \end{verbatim} - -\begin{verbatim} -[14:54] rainbow:bitfiles% cd $NF_DESIGN_DIR/bitfiles/ && sudo bash ./program_switch.sh -./program_switch.sh: line 34: /tools/program_switch.sh: No such file or directory -[14:56] rainbow:bitfiles% ls -config_writes.sh minip4.bit program_switch.sh README -[14:56] rainbow:bitfiles% -\end{verbatim} - +Missing interface when testing switch\_calc: \begin{verbatim} root@rainbow:~/master-thesis/netpfga/minip4/sw/hw_test_tool# python switch_calc_tester.py SIOCSIFADDR: No such device @@ -1729,8 +623,7 @@ The HW testing tool for the switch_calc design type help to see all commands testing> \end{verbatim} - -\label{netpfgaioctlerror} +Ioctl error when adding table errors on the first NetFPGA card: \begin{verbatim} >> table_cam_add_entry lookup_table send_to_port1 ff:ff:ff:ff:ff:ff => CAM_Init_ValidateContext() - done @@ -1739,9 +632,8 @@ WROTE 0x44020054 = 0xffff WROTE 0x44020080 = 0x0003 python: ioctl: Unknown error 512 [20:27] rainbow:CLI% - \end{verbatim} - +Exec format errors when loading the kernel module due to incompabilities: \begin{verbatim} [7:05] rainbow:netpfga% bash build-load-drivers.sh + cd /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 @@ -1763,12 +655,10 @@ depmod -a 5.0.0-16-generic + sudo modprobe sume_riffa modprobe: ERROR: could not insert 'sume_riffa': Exec format error [7:06] rainbow:netpfga% - \end{verbatim} - +Java traceback when trying to install SDNET: +(reason was a hidden window) \begin{verbatim} -java traceback when trying to install sdnet -#+BEGIN_CENTER Exception in thread "AWT-EventQueue-0" java.lang.IllegalArgumentException: Window must not be zero at java.desktop/sun.awt.X11.XAtom.checkWindow(Unknown Source) at java.desktop/sun.awt.X11.XAtom.getAtomData(Unknown Source) @@ -1828,15 +718,8 @@ Exception in thread "AWT-EventQueue-0" java.lang.IllegalArgumentException: Windo at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.desktop/java.awt.EventDispatchThread.run(Unknown Source) - - -#+END_CENTER - -Reason was a hidden window. - \end{verbatim} - -Testing the card +Failures when testing the first NetFPGA card \begin{verbatim} --------------------------------------------- [ddr3B]: Running Auto Test @@ -1868,9 +751,8 @@ Traceback (most recent call last): File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory - \end{verbatim} - +More failures when testing the first NetFPGA card \begin{verbatim} --------------------------------------------- [pcie]: Running Auto Test @@ -1894,14 +776,12 @@ Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 495, in read raise SerialException('device reports readiness to read but returned no data (device disconnected or multiple access on port?)') SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?) - - \end{verbatim} - -Another generated file problem: +Unexpected EOF during compilation: \begin{verbatim} -nico@nsg-System:~/master-thesis/netpfga$ grep -i error $P4_PROJECT_DIR/nf_sume_sdnet_ip/SimpleSumeSwitch/LOG -ERROR: [VRFC 10-1491] unexpected EOF [/home/nico/master-thesis/netpfga/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/S_CONTROLLERs.HDL/S_CONTROLLER_SimpleSumeSwitch.vp:37] +ERROR: [VRFC 10-1491] unexpected EOF +[/home/nico/master-thesis/netpfga/minip4/nf_sume_sdnet_ip/ +SimpleSumeSwitch/S_CONTROLLERs.HDL/S_CONTROLLER_SimpleSumeSwitch.vp:37] INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_0_ErrorCheck INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_1_ErrorCheck INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_2_ErrorCheck @@ -1909,10 +789,8 @@ INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_3_ErrorCheck INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_4_ErrorCheck INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_5_ErrorCheck INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_6_ErrorCheck - \end{verbatim} - -function syntax not supported +The function syntax is not supported by p4/netfpga: \begin{verbatim} make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src' p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4 @@ -1926,80 +804,71 @@ make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume Makefile:31: recipe for target 'frontend' failed make: *** [frontend] Error 2 nico@nsg-System:~/master-thesis/netpfga$ - - \end{verbatim} - +The config\_writes.py is missing due to a previous, non critical +compilation error: \begin{verbatim} -nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default$ cd $NF_DESIGN_DIR/test/sim_switch_default && make 2>&1 | tee ~/master-thesis/netpfga/log/step8-$(date +%F-%H%M%S) +nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default$ +cd $NF_DESIGN_DIR/test/sim_switch_default && make 2>&1 | tee ~/master-thesis/netpfga/log/step8-$(date +%F-%H%M%S) rm -f config_writes.py* rm -f *.pyc cp /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.py ./ cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.py': No such file or directory Makefile:36: recipe for target 'all' failed make: *** [all] Error 1 - \end{verbatim} - +Failed to synthesizing module errors: \begin{verbatim} -Finished scanning sources -INFO: [IP_Flow 19-234] Refreshing IP repositories -INFO: [IP_Flow 19-1700] Loaded user IP repository '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/ip_repo'. -INFO: [IP_Flow 19-2313] Loaded Vivado IP repository '/opt/Xilinx/Vivado/2018.2/data/ip'. -WARNING: [IP_Flow 19-3664] IP 'bd_7ad4_xpcs_0' generated file not found '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/.Xil/Vivado-26302-nsg-System/coregen/bd_7ad4_xpcs_0_1/elaborate/configure_gt.tcl'. Please regenerate to continue. -WARNING: [IP_Flow 19-3664] IP 'bd_a1aa_xpcs_0' generated file not found '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/.Xil/Vivado-26302-nsg-System/coregen/bd_a1aa_xpcs_0_2/elaborate/configure_gt.tcl'. Please regenerate to continue. -open_project: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 1365.715 ; gain = 188.977 ; free physical = 9396 ; free virtual = 15104 -# puts "\nOpening $design Implementation design\n" - +WARNING: [Synth 8-689] width (12) of port connection 'control_S_AXI_ARADDR' does not match port width (8) of module 'SimpleSumeSwitch' +[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/ +simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:199] +ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_VALID' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch' +[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/ +simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:218] +ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_DATA' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch' +[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/ +simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:219] +ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet' +[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/ +simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:44] +ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet_ip' +[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/ +simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/synth/nf_sume_sdnet_ip.v:57] +ERROR: [Synth 8-6156] failed synthesizing module 'nf_datapath' +[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/ +simple_sume_switch/hw/hdl/nf_datapath.v:44] +ERROR: [Synth 8-6156] failed synthesizing module 'top' +[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/ +simple_sume_switch/hw/hdl/top.v:43] \end{verbatim} - +Missing ``souce'' files abort CLI compilation errors: \begin{verbatim} -WARNING: [Synth 8-689] width (12) of port connection 'control_S_AXI_ARADDR' does not match port width (8) of module 'SimpleSumeSwitch' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:199] -ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_VALID' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:218] -ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_DATA' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:219] -ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:44] -ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet_ip' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/synth/nf_sume_sdnet_ip.v:57] -ERROR: [Synth 8-6156] failed synthesizing module 'nf_datapath' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/hdl/nf_datapath.v:44] -ERROR: [Synth 8-6156] failed synthesizing module 'top' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/hdl/top.v:43] -\end{verbatim} - -Missing ``souce'' files: -\begin{verbatim} -cc -c -fPIC /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API/CAM.c -I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API -cc -std=c99 -Wall -Werror -fPIC -c libcam.c -I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume -I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API -cc -L/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume -shared -o libcam.so libcam.o CAM.o -lsumereg +cc -c -fPIC /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API/CAM.c +-I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API +cc -std=c99 -Wall -Werror -fPIC -c libcam.c +-I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume +-I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API +cc -L/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume + -shared -o libcam.so libcam.o CAM.o -lsumereg /usr/bin/ld: cannot find -lsumereg collect2: error: ld returned 1 exit status Makefile:52: recipe for target 'libcam' failed make[1]: *** [libcam] Error 1 make[1]: Leaving directory '/home/nico/master-thesis/netpfga/minip4/sw/CLI' ERROR: could not compile libcam souce files - - \end{verbatim} - -Generated files not found: +Generated axi files not found at a different stage: \begin{verbatim} -make: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test' -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/Makefile': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_stim.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_log.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_expected.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/Makefile': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.log': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_expect.axi': No such file or directory -cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.axi': No such file or directory +cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/ +sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_expected.axi': No such file or directory +cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/ +sume-sdnet-switch/projects/minip4/simple_sume_switch/test/Makefile': No such file or directory +cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/ +sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.log': No such file or directory +cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/ +sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_expect.axi': No such file or directory +cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/ +sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.axi': No such file or directory NetFPGA environment: Root dir: /home/nico/projects/P4-NetFPGA Project name: simple_sume_switch @@ -2015,10 +884,8 @@ Die Jul 23 13:34:54 CEST 2019 + cd /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch + make make: *** No targets specified and no makefile found. Stop. - \end{verbatim} - -Renaming variables breaks the compile process +Renaming variables as follows breaks the compile process \begin{verbatim} @Xilinx_MaxPacketRegion(1024) control TopDeparser( @@ -2038,9 +905,7 @@ Renaming variables breaks the compile process + } \end{verbatim} - -LPM size must be != 64 - +In NetPFGA the LPM table size must be != 64: \begin{verbatim} minip4_solution.p4(38): [--Wwarn=uninitialized_out_param] warning: out parameter meta may be uninitialized when RealParser terminates out metadata meta, @@ -2066,10 +931,8 @@ make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume Makefile:31: recipe for target 'frontend' failed make: *** [frontend] Error 2 nico@nsg-System:~/master-thesis/netpfga/log$ - \end{verbatim} - -LIMIT table match types are not the same error +Cannot mix the key table types with P4/NetFPGA: \begin{verbatim} make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src' p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4 @@ -2109,12 +972,8 @@ make[1]: *** [all] Error 1 size = ICMP6_TABLE_SIZE; default_action = controller_debug_table_id(TABLE_ARP); } - - \end{verbatim} - -Implicit error saying that LPM tables don't work: - +Implicit error saying that LPM tables don't work in P4/NetFPGA: \begin{verbatim} s/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/realmain_lookup_table_0_t.HDL/xpm_memory.sv [SW] LPM_Init() - start @@ -2125,7 +984,7 @@ FATAL_ERROR: Vivado Simulator kernel has encounted an exception from DPI C funct Time: 2016466 ps Iteration: 0 Process: /SimpleSumeSwitch_tb/LPM_VerifyDataset File: /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/Testbench/SimpleSumeSwitch_tb.sv \end{verbatim} - +The table for exact matches must be at least 64 in P4/NetFPGA: \begin{verbatim} minip4_solution.p4(35) parser RealParser( @@ -2136,10 +995,8 @@ actions_nat64_generic.p4(173): error: table size too small for match_type(EM): 6 actions_nat64_generic.p4(173): error: could not not map table size size size = 63; ^^^^ - \end{verbatim} - -Unsupported default parameters +Unsupported default parameters in P4/NetFPGA: \begin{verbatim} actions_egress.p4(89): error: data-plane arguments in default_actions are currently unsupported: realmain_controller_debug_table_id_0 default_action = controller_debug_table_id(TABLE_V4_NETWORKS); @@ -2149,10 +1006,8 @@ terminate called after throwing an instance of 'Util::CompilerBug' Compiler Bug: actions_egress.p4(89): unhandled expression realmain_controller_debug_table_id/realmain_controller_debug_table_id_0(5); default_action = controller_debug_table_id(TABLE_V4_NETWORKS); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - \end{verbatim} - -Compiler Bug / ifstatement +Causing compiler bug by using an if statement at a wrong place in P4/NetFPGA: \begin{verbatim} minip4_solution.p4(39) parser RealParser( @@ -2165,14 +1020,10 @@ Makefile:34: recipe for target 'all' failed make[1]: *** [all] Error 134 make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src' Makefile:31: recipe for target 'frontend' failed - \end{verbatim} - -Applying table ``twice'' in different branches is impossible (another -compiler bug) - +Applying table ``twice'' in different branches is impossible in +P4/NetFPGA causes a different compiler bug: \begin{verbatim} - make -C src/ make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src' p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4 @@ -2187,24 +1038,257 @@ terminate called after throwing an instance of 'Util::CompilerBug' Compiler Bug: overwrite Makefile:34: recipe for target 'all' failed - - \end{verbatim} - -Adding entries requires setting all parameters +Adding table entries requires setting parameters for all possible +actions that are registered in a table: \begin{verbatim} ->> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434178 => 1 -ERROR: not enough fields provided to complete _hexify() - +>> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port +42540766411362381960998550477184434178 => 1 ERROR: not enough fields provided to complete _hexify() \end{verbatim} - -Broken code that cannot convret long to int: +The table handling scripts do not handle conversion for long integers +for P4/NetFPGA: \begin{verbatim} >> table_cam_delete_entry realmain_v6_networks_0 42540766411362381960998550477184434179 ERROR: failed to convert 42540766411362381960998550477184434179 of type to an integer nico@nsg-System:~/master-thesis/netpfga/minip4/sw/CLI$ - \end{verbatim} +%---------------------------------------------------------------------- +\chapter{\label{benchmark}Benchmark Logs} +% ---------------------------------------------------------------------- +\section{\label{benchmark:offset}Enabling hardware offloading} +The following commands enable hardware offloading even though error +messages are printed: +\begin{verbatim} +root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on +Cannot get device udp-fragmentation-offload settings: Operation not supported +Cannot get device udp-fragmentation-offload settings: Operation not supported +Actual changes: +tx-checksumming: on + tx-checksum-ip-generic: on + tx-checksum-sctp: on +tcp-segmentation-offload: on + tx-tcp-segmentation: on + tx-tcp6-segmentation: on +root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on +Cannot get device udp-fragmentation-offload settings: Operation not supported +Cannot get device udp-fragmentation-offload settings: Operation not supported +root@ESPRIMO-P956:~# +\end{verbatim} +This results into the following: +\begin{verbatim} +root@ESPRIMO-P956:~# ethtool -k enp2s0f1 +Features for enp2s0f1: +Cannot get device udp-fragmentation-offload settings: Operation not supported +rx-checksumming: on +tx-checksumming: on + tx-checksum-ipv4: off [fixed] + tx-checksum-ip-generic: on + tx-checksum-ipv6: off [fixed] + tx-checksum-fcoe-crc: on [fixed] + tx-checksum-sctp: on +scatter-gather: on + tx-scatter-gather: on + tx-scatter-gather-fraglist: off [fixed] +tcp-segmentation-offload: on + tx-tcp-segmentation: on + tx-tcp-ecn-segmentation: off [fixed] + tx-tcp-mangleid-segmentation: off + tx-tcp6-segmentation: on +udp-fragmentation-offload: off +generic-segmentation-offload: on +generic-receive-offload: on +large-receive-offload: off +rx-vlan-offload: on +tx-vlan-offload: on +ntuple-filters: off +receive-hashing: on +highdma: on [fixed] +rx-vlan-filter: on +vlan-challenged: off [fixed] +tx-lockless: off [fixed] +netns-local: off [fixed] +tx-gso-robust: off [fixed] +tx-fcoe-segmentation: on [fixed] +tx-gre-segmentation: on +tx-gre-csum-segmentation: on +tx-ipxip4-segmentation: on +tx-ipxip6-segmentation: on +tx-udp_tnl-segmentation: on +tx-udp_tnl-csum-segmentation: on +tx-gso-partial: on +tx-sctp-segmentation: off [fixed] +tx-esp-segmentation: off [fixed] +fcoe-mtu: off [fixed] +tx-nocache-copy: off +loopback: off [fixed] +rx-fcs: off [fixed] +rx-all: off +tx-vlan-stag-hw-insert: off [fixed] +rx-vlan-stag-hw-parse: off [fixed] +rx-vlan-stag-filter: off [fixed] +l2-fwd-offload: off +hw-tc-offload: off +esp-hw-offload: off [fixed] +esp-tx-csum-hw-offload: off [fixed] +rx-udp_tunnel-port-offload: off +root@ESPRIMO-P956:~# +\end{verbatim} + +%--------------------------------------------------------------------------------------------------------- +\section{\label{benchmark:tayga}Tayga} +Tayga is installed from the regular package database: +\begin{verbatim} +ii tayga 0.9.2-6 amd64 userspace stateless NAT64 +\end{verbatim} +We prepare the networking as follows: +\begin{verbatim} +[15:12] nsg-System:~# ip addr add 10.0.0.77/24 dev eth1 +[15:12] nsg-System:~# ip l s eth1 up + +nico@ESPRIMO-P956:~$ ~/master-thesis/bin/init_ipv4_esprimo.sh +nico@ESPRIMO-P956:~$ cat ~/master-thesis/bin/init_ipv4_esprimo.sh +#!/bin/sh + +sudo ip addr add 10.0.0.42/24 dev enp2s0f0 +sudo ip link set enp2s0f0 up + +nico@ESPRIMO-P956:~$ sudo ip route add 10.0.1.0/24 via 10.0.0.77 +\end{verbatim} +And verify that networking works: +\begin{verbatim} +[15:12] nsg-System:~# ping 10.0.0.42 +PING 10.0.0.42 (10.0.0.42) 56(84) bytes of data. +64 bytes from 10.0.0.42: icmp_seq=1 ttl=64 time=0.304 ms +64 bytes from 10.0.0.42: icmp_seq=2 ttl=64 time=0.097 ms +^C +--- 10.0.0.42 ping statistics --- +2 packets transmitted, 2 received, 0% packet loss, time 1011ms +rtt min/avg/max/mdev = 0.097/0.200/0.304/0.104 ms +[15:12] nsg-System:~# +\end{verbatim} +We also setup the IPv6 networking: +\begin{verbatim} +nico@ESPRIMO-P956:~$ ip addr show dev enp2s0f1 +13: enp2s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 + link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff + inet6 2001:db8:42::42/64 scope global + valid_lft forever preferred_lft forever + inet6 fe80::faf2:1eff:fe09:62d1/64 scope link + valid_lft forever preferred_lft forever +nico@ESPRIMO-P956:~$ sudo ip route add 2001:db8:23::/96 via 2001:db8:42::77 + +[15:12] nsg-System:~# ip addr add 2001:db8:42::77/64 dev eth2 +[15:15] nsg-System:~# ip link set eth2 up +\end{verbatim} +And verify that IPv6 networking works: +\begin{verbatim} +nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::77 +PING 2001:db8:42::77(2001:db8:42::77) 56 data bytes +64 bytes from 2001:db8:42::77: icmp_seq=1 ttl=64 time=0.169 ms +64 bytes from 2001:db8:42::77: icmp_seq=2 ttl=64 time=0.153 ms + +--- 2001:db8:42::77 ping statistics --- +2 packets transmitted, 2 received, 0% packet loss, time 1010ms +rtt min/avg/max/mdev = 0.153/0.161/0.169/0.008 ms +nico@ESPRIMO-P956:~$ +\end{verbatim} +We enable IPv6 and IPv4 forwarding: +\begin{verbatim} +[15:16] nsg-System:~# sysctl -w net.ipv6.conf.all.forwarding=1 +net.ipv6.conf.all.forwarding = 1 + +[15:20] nsg-System:~# sysctl -w net.ipv4.ip_forward=1 +net.ipv4.ip_forward = 1 +\end{verbatim} +And we test NAT64 with tayga: +\begin{verbatim} +nico@ESPRIMO-P956:~$ ping -c2 10.0.1.42 +PING 10.0.1.42 (10.0.1.42) 56(84) bytes of data. +64 bytes from 10.0.1.42: icmp_seq=1 ttl=61 time=0.356 ms +64 bytes from 10.0.1.42: icmp_seq=2 ttl=61 time=0.410 ms + +--- 10.0.1.42 ping statistics --- +2 packets transmitted, 2 received, 0% packet loss, time 1019ms +rtt min/avg/max/mdev = 0.356/0.383/0.410/0.027 ms +nico@ESPRIMO-P956:~$ + +nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1 +tcpdump: verbose output suppressed, use -v or -vv for full protocol decode +listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes +15:21:39.851057 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 1, length 64 +15:21:39.851124 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 1, length 64 +15:21:40.870448 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 2, length 64 +15:21:40.870507 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 2, length 64 +^C +4 packets captured +4 packets received by filter +0 packets dropped by kernel +nico@ESPRIMO-P956:~$ +\end{verbatim} +And test NAT64 from IPv6 to IPv4: +\begin{verbatim} +nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:23::a00:2a +PING 2001:db8:23::a00:2a(2001:db8:23::a00:2a) 56 data bytes +64 bytes from 2001:db8:23::a00:2a: icmp_seq=1 ttl=61 time=0.240 ms +64 bytes from 2001:db8:23::a00:2a: icmp_seq=2 ttl=61 time=0.400 ms + +--- 2001:db8:23::a00:2a ping statistics --- +2 packets transmitted, 2 received, 0% packet loss, time 1003ms +rtt min/avg/max/mdev = 0.240/0.320/0.400/0.080 ms +nico@ESPRIMO-P956:~$ +\end{verbatim} +%--------------------------------------------------------------------------------------------------------- +\section{\label{benchmark:jool}Jool} +We install Jool 4.0.1 from source from +\url{https://www.jool.mx/en/download.html} as follows: +\begin{verbatim} +nico@nsg-System:~$ wget https://github.com/NICMx/Jool/releases/download/v4.0.1/jool_4.0.1.tar.gz +nico@nsg-System:~$ tar xvfz jool_4.0.1.tar.gz +nico@nsg-System:~$ cd jool-4.0.1/ +nico@nsg-System:~/jool-4.0.1$ sudo apt install linux-headers-$(uname -r) +nico@nsg-System:~/jool-4.0.1$ sudo apt install libnl-genl-3-dev +nico@nsg-System:~/jool-4.0.1$ sudo apt install iptables-dev +nico@nsg-System:~/jool-4.0.1$ sudo make install +\end{verbatim} +We enable forwarding: +\begin{verbatim} +sysctl -w net.ipv4.conf.all.forwarding=1 +sysctl -w net.ipv6.conf.all.forwarding=1 +\end{verbatim} +We configure jool to map the network prefixes and setup iptables to +redirect the traffic into the jool instance: +\begin{verbatim} +[16:53] nsg-System:~# modprobe jool_siit +[16:54] nsg-System:~# jool_siit instance add "example" --iptables +[16:54] nsg-System:~# jool_siit -i example eamt add 2001:db8:42::/120 10.0.1.0/24 +[16:55] nsg-System:~# jool_siit -i example eamt add 2001:db8:23::/120 10.0.0.0/24 +[16:57] nsg-System:~# ip6tables -t mangle -A PREROUTING -s 2001:db8:42::/120 -d 2001:db8:23::/120 -j JOOL_SIIT --instance example +[16:57] nsg-System:~# iptables -t mangle -A PREROUTING -s 10.0.0.0/24 -d 10.0.1.0/24 -j JOOL_SIIT --instance example +\end{verbatim} +Afterwards we test NAT64: +\begin{verbatim} +nico@ESPRIMO-P956:~/master-thesis/iperf$ ping6 2001:db8:23::2a +PING 2001:db8:23::2a(2001:db8:23::2a) 56 data bytes +64 bytes from 2001:db8:23::2a: icmp_seq=1 ttl=63 time=0.199 ms +64 bytes from 2001:db8:23::2a: icmp_seq=2 ttl=63 time=0.282 ms +64 bytes from 2001:db8:23::2a: icmp_seq=3 ttl=63 time=0.186 ms +^C +--- 2001:db8:23::2a ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2040ms +rtt min/avg/max/mdev = 0.186/0.222/0.282/0.044 ms +nico@ESPRIMO-P956:~/master-thesis/iperf$ ping 10.0.1.66 +PING 10.0.1.66 (10.0.1.66) 56(84) bytes of data. +64 bytes from 10.0.1.66: icmp_seq=1 ttl=63 time=0.218 ms +64 bytes from 10.0.1.66: icmp_seq=2 ttl=63 time=0.281 ms +64 bytes from 10.0.1.66: icmp_seq=3 ttl=63 time=0.280 ms +^C +--- 10.0.1.66 ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2051ms +rtt min/avg/max/mdev = 0.218/0.259/0.281/0.034 ms +nico@ESPRIMO-P956:~/master-thesis/iperf$ +\end{verbatim} + + % ---------------------------------------------------------------------- \section{P4 error messages} @@ -2551,8 +1635,7 @@ I could work around this by using if(! .. .hit) { my_action(table_id) \end{verbatim} % ---------------------------------------------------------------------- -\subsection{\label{appendix:p4:python2unicode - }Python2 unicode issue} +\subsection{\label{appendix:p4:python2unicode}Python2 unicode issue} \begin{verbatim} ipaddress.ip_network("2001:db8:61::/64")