\chapter{\label{appendixminus1}Resources and code repositories}
The following sections describe how to acquire the resources
to reproduce the test results.

% ----------------------------------------------------------------------
\section{\label{chapterminus1:thesis:os}Operating Systems}
All P4 compilations were made on Ubuntu 16.04 with kernels
\begin{itemize}
\item 4.15.0-54-generic (Supporting Desktop)
\item 4.4.0-143-generic (BMV2 test VM)
\item 4.15.0-55-generic (Desktop with NetFPGA card)
\end{itemize}
% ok
% ----------------------------------------------------------------------
\section{\label{chapterminus1:thesis:general}Master Thesis}
The master thesis including all self developed source code is
available by git via
\begin{itemize}
\item git clone \url{git@gitlab.ethz.ch:nicosc/master-thesis.git}
\item git clone \url{git@gitlab.ethz.ch:nsg/student-projects/ma-2019-19_high_speed_nat64_with_p4}
\end{itemize}
It can be browsed online on
\url{https://gitlab.ethz.ch/nicosc/master-thesis} and on
\url{https://gitlab.ethz.ch/nsg/student-projects/ma-2019-19_high_speed_nat64_with_p4}.
% ok
% ----------------------------------------------------------------------
\section{\label{chapterminus1:thesis:xilinx}Xilinx Toolchain}
A prerequisite for building the NetFGPA source code is the
installation of
\begin{itemize}
\item \texttt{Xilinx\_SDNet\_2018.2\_1005\_9}
\item \texttt{Xilinx\_Vivado\_SDK\_2018\.2\_0614\_1954}
\end{itemize}
Both tools need to be installed to \texttt{/opt/Xilinx/},
as paths are hardcoded in various places.
% ok
% ----------------------------------------------------------------------
\section{\label{chapterminus1:thesis}P4/NetFGPA support scripts}
To be able to compile P4 source code to the NetFPGA the collection of
scripts, Makefiles and sample code of P4-NetFGPA is required.
The repository \url{git@github.com:NetFPGA/P4-NetFPGA-live.git} needs
to be cloned to ``projects'' subdirectory as ``P4-NetPFGA''
of the user that wants to
compile the source code. Access to the repository is granted after
applying for access as described on
\url{https://github.com/NetFPGA/P4-NetFPGA-public/wiki}.
After that the variable \texttt{P4\_PROJECT\_NAME} in
\texttt{~/projects/P4-NetFPGA/tools/settings.sh} needs to be modified to
read \texttt{export P4\_PROJECT\_NAME=minip4}
instead of \texttt{export P4\_PROJECT\_NAME=switch\_calc}.
Sample code for installation:
\begin{tiny}\begin{verbatim}
  mkdir -p ~/projects
  git clone git@github.com:NetFPGA/P4-NetFPGA-live.git P4-NetFPGA
  sed -i 's/\(P4_PROJECT_NAME=\).*/\1minip4/' ~/projects/P4-NetFPGA/tools/settings.sh
\end{verbatim}
\end{tiny}
Version \textbf{v1.3.1-46-g97d3aaa} of the P4-NetPFGA repository was
used for creating the bitfiles of this project.
\begin{verbatim}
nico@nsg-System:~/projects/P4-NetFPGA$ git describe --always
v1.3.1-46-g97d3aaa
\end{verbatim}
% ok
% ----------------------------------------------------------------------
\section{\label{appendix:netfpga:compile}P4/NetFGPA compilation process}
After having setup the compile host as described above, the script
\texttt{bin/do-all-steps.sh} that is included in the thesis' git
repository. With a NetFPGA card installed in the host, this script
will compile the P4 source code to PX and in a second step to HDL and
then upload the resulting bitstream to the NetFPGA. The compilation
process will log its output to the directory
\texttt{\~/master-thesis/netpfga/log/}.
% ----------------------------------------------------------------------
\section{\label{appendix:netfpga:tests}P4/NetFGPA Tests}
In the following sections we describe functionality tests
of our code on the NetFPGA.
% ----------------------------------------------------------------------
\subsection{Test 1: IPv4 Egress}
In this test we test whether setting the output port based on the
IPv4 address.
First we get the integer values of the IPv4 addresses in python:
\begin{verbatim}
>>> int(ipaddress.IPv4Address(u"10.0.0.42"))
167772202
>>> int(ipaddress.IPv4Address(u"10.0.0.4"))
167772164
>>>
\end{verbatim}
After that we set the table table entries for the NetFPGA.
\begin{tiny}\begin{verbatim}
>> table_cam_add_entry realmain_v4_networks_0 realmain.set_egress_port 167772202 => 16 0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '16', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020250 = 0xa00002a
WROTE 0x44020280 = 0x0000
WROTE 0x44020284 = 0x0000
WROTE 0x44020288 = 0x10000000
WROTE 0x4402028c = 0x0001
READ  0x44020244 = 0x0001
WROTE 0x44020240 = 0x0001
READ  0x44020244 = 0x0001
READ  0x44020244 = 0x0001
success
>> table_cam_add_entry realmain_v4_networks_0 realmain.set_egress_port 167772164 => 16 0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '16', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020250 = 0xa000004
WROTE 0x44020280 = 0x0000
WROTE 0x44020284 = 0x0000
WROTE 0x44020288 = 0x10000000
WROTE 0x4402028c = 0x0001
READ  0x44020244 = 0x0001
WROTE 0x44020240 = 0x0001
READ  0x44020244 = 0x0001
READ  0x44020244 = 0x0001
success
>>
\end{verbatim}
\end{tiny}
On the host we setup the ARP entries:
\begin{tiny}\begin{verbatim}
root@ESPRIMO-P956:~# ip neigh add 10.0.0.6 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0
root@ESPRIMO-P956:~# ip neigh add 10.0.0.4 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0
\end{verbatim}
\end{tiny}
And then we generate test packets and expect 4 packets to show up on
enp2s0f0.
The following \texttt{tcpdump} output shows the expected packets
arriving on enp2s0f0:
\begin{tiny}\begin{verbatim}
nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:49:28.200407 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 1, length 64
10:49:28.200445 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 1, length 64
10:49:29.222340 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64
10:49:29.222418 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64
\end{verbatim}
\end{tiny}
% ok
% ----------------------------------------------------------------------
\subsection{Test 2: IPv6 egress}
This test shows how setting the egress port based on the IPv6 address
works with the NetPFGA. Similar to the previous test, we first the
the Integer values of the IPv6 addresses:
\begin{verbatim}
>>> int(ipaddress.IPv6Address(u"2001:db8:42::4"))
42540766411362381960998550477184434180L
>>> int(ipaddress.IPv6Address(u"2001:db8:42::6"))
42540766411362381960998550477184434182L
>>> int(ipaddress.IPv6Address(u"2001:db8:42::42"))
42540766411362381960998550477184434242L
\end{verbatim}
After that we set the table entries:
\begin{tiny}
\begin{verbatim}
>> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434182 => 64  0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '64', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020350 = 0x0006
WROTE 0x44020354 = 0x0000
WROTE 0x44020358 = 0x420000
WROTE 0x4402035c = 0x20010db8
WROTE 0x44020380 = 0x0000
WROTE 0x44020384 = 0x0000
WROTE 0x44020388 = 0x40000000
WROTE 0x4402038c = 0x0001
READ  0x44020344 = 0x0001
WROTE 0x44020340 = 0x0001
READ  0x44020344 = 0x0001
READ  0x44020344 = 0x0001
success
>> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434242 => 64  0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '64', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020350 = 0x0042
WROTE 0x44020354 = 0x0000
WROTE 0x44020358 = 0x420000
WROTE 0x4402035c = 0x20010db8
WROTE 0x44020380 = 0x0000
WROTE 0x44020384 = 0x0000
WROTE 0x44020388 = 0x40000000
WROTE 0x4402038c = 0x0001
READ  0x44020344 = 0x0001
WROTE 0x44020340 = 0x0001
READ  0x44020344 = 0x0001
READ  0x44020344 = 0x0001
success
>>
\end{verbatim}
\end{tiny}
On the host we set the IPv6 neighbor entries:
\begin{tiny}\begin{verbatim}
nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::6 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1
nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::4 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1
\end{verbatim}
\end{tiny}
And generate the test packets:
\begin{tiny}\begin{verbatim}
nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::6
PING 2001:db8:42::6(2001:db8:42::6) 56 data bytes


nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:30:17.287577 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 1, length 64
11:30:17.287599 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 1, length 64
11:30:18.310178 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64
11:30:18.310258 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64
\end{verbatim}
\end{tiny}
The packets are successfully seen by tcpdump.
% ----------------------------------------------------------------------
\section{\label{appendix:bmv2}P4/BMV2 environment and tests}
All BMV2 based compilations were made with the following compiler:
\begin{verbatim}
p4@ubuntu:~$ p4c --version
p4c 0.5 (SHA: 5ae30ee)
\end{verbatim}
The installation is based on the vagrant files that were provided in
the ``Advanced Topics in
Communication Networks Fall 2018'' course of
ETHZ (\url{https://adv-net.ethz.ch/2018/}) and contains p4tools as
well as all utilities that came with the vagrant installation.
% ok
% ----------------------------------------------------------------------
For running the diff based checksum code, the following steps are
necessary: First compile the p4 code and then start the switch, both
with \texttt{p4run}.
\begin{verbatim}
cd ~/master-thesis/p4app
sudo p4run --config nat64-diff.json
\end{verbatim}
Then with starting the controller the required table entries will
\begin{verbatim}
cd ~/master-thesis/p4app
sudo python ./controller.py --mode range_router
\end{verbatim}
% ----------------------------------------------------------------------
\chapter{\label{appendix:netpfgalogs}NetFPGA Logs}
The log files of the NetFPGA compilations are stored inside
the source code directory stored at \texttt{netpfga/logs}.
It follows a selection of excerpts of log files that might
be relevant for reproducing the work.
% ----------------------------------------------------------------------
\section{\label{appendix:netpfgalogs:flasherror}NetFPGA Flash Errors}
Sometimes flashing bitfiles to the NetFPGA will fail. A random amount
of reboots (1 to 3) and a random amount of reflashing will fix this
problem.
Below can be found the log output from the flashing process.
\begin{tiny}\begin{verbatim}
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$
sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh"
++ which vivado
+ xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado
+ bitimage=minip4.bit
+ configWrites=config_writes.sh
+ '[' -z minip4.bit ']'
+ '[' -z config_writes.sh ']'
+ '[' /opt/Xilinx/Vivado/2018.2/bin/vivado == '' ']'
+ rmmod sume_riffa
+ xsct /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl -tclargs minip4.bit
rlwrap: warning: your $TERM is 'screen' but rlwrap couldn't find it in the terminfo database. Expect some problems.
RUN loading image file.
minip4.bit
100%   19MB   1.7MB/s  00:11
fpga configuration failed. DONE PIN is not HIGH
    invoked from within
"::tcf::eval -progress ::xsdb::print_progress {::tcf::cache_enter tcfchan#0 {tcf_cache_eval {process_tcf_actions_cache_client ::tcfclient#0::arg}}}"
    (procedure "::tcf::cache_eval_with_progress" line 2)
    invoked from within
"::tcf::cache_eval_with_progress [dict get $arg chan]  [list process_tcf_actions_cache_client $argvar] $progress"
    (procedure "process_tcf_actions" line 1)
    invoked from within
"process_tcf_actions $arg ::xsdb::print_progress"
    (procedure "fpga" line 430)
    invoked from within
"fpga -f $bitimage"
    (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl" line 33)

+ bash /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/pci_rescan_run.sh
Check programming FPGA or Reboot machine !
+ rmmod sume_riffa
rmmod: ERROR: Module sume_riffa is not currently loaded
+ modprobe sume_riffa
+ ifconfig nf0 up
nf0: ERROR while getting interface flags: No such device
+ ifconfig nf1 up
nf1: ERROR while getting interface flags: No such device
+ ifconfig nf2 up
nf2: ERROR while getting interface flags: No such device
+ ifconfig nf3 up
nf3: ERROR while getting interface flags: No such device
+ bash config_writes.sh

\end{verbatim}
\end{tiny}
% ok
% ----------------------------------------------------------------------
\section{\label{appendix:netpfgalogs:flashsuccess}NetFPGA Flash Success}
A successful flashing process also emits a couple of errors, however
the message ``fpga configuration failed. DONE PIN is not HIGH'' and
its succeeding lines are missing, as seen below.

After that in all cases a reboot is required; the PCI rescan in none
of our test cases re enabled the nf devices.
\begin{tiny}\begin{verbatim}
nico@nsg-System:~$ cd $NF_DESIGN_DIR/bitfiles/
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$
sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh"
++ which vivado
+ xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado
+ bitimage=minip4.bit
+ configWrites=config_writes.sh
+ '[' -z minip4.bit ']'
+ '[' -z config_writes.sh ']'
+ '[' /opt/Xilinx/Vivado/2018.2/bin/vivado == '' ']'
+ rmmod sume_riffa
+ xsct /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl -tclargs minip4.bit
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
RUN loading image file.
minip4.bit
attempting to launch hw_server

****** Xilinx hw_server v2018.2
  **** Build date : Jun 14 2018-20:18:37
    ** Copyright 1986-2018 Xilinx, Inc. All Rights Reserved.

INFO: hw_server application started
INFO: Use Ctrl-C to exit hw_server application

INFO: To connect to this hw_server instance use url: TCP:127.0.0.1:3121

100%   19MB   1.7MB/s  00:11
+ bash /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/pci_rescan_run.sh
Check programming FPGA or Reboot machine !
+ rmmod sume_riffa
rmmod: ERROR: Module sume_riffa is not currently loaded
+ modprobe sume_riffa
+ ifconfig nf0 up
nf0: ERROR while getting interface flags: No such device
+ ifconfig nf1 up
nf1: ERROR while getting interface flags: No such device
+ ifconfig nf2 up
nf2: ERROR while getting interface flags: No such device
+ ifconfig nf3 up
nf3: ERROR while getting interface flags: No such device
+ bash config_writes.sh
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$
\end{verbatim}
\end{tiny}
% ok
% ----------------------------------------------------------------------
\section{\label{appendix:netfpgalogs:kernelmodule}NetFPGA Kernel module}
After a successful flash, loading the kernel module will enable nf
devices to appear in the operating system.
\begin{tiny}\begin{verbatim}
nico@nsg-System:~$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 74:d0:2b:98:38:f6 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9c brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9d brd ff:ff:ff:ff:ff:ff
5: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none
nico@nsg-System:~$ ~/master-thesis/bin/build-load-drivers.sh
+ cd /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0
+ sudo modprobe -r sume_riffa
+ make clean
make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 clean
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic'
  CLEAN   /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/.tmp_versions
  CLEAN   /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/Module.symvers
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic'
+ make all
make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic'
  CC [M]  /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.mod.o
  LD [M]  /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic'
+ sudo make install
make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic'
  Building modules, stage 2.
  MODPOST 1 modules
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic'
install -o root -g root -m 0755 -d /lib/modules/4.15.0-55-generic/extra/sume_riffa/
install -o root -g root -m 0755 sume_riffa.ko /lib/modules/4.15.0-55-generic/extra/sume_riffa/
depmod -a 4.15.0-55-generic
+ sudo modprobe sume_riffa
+ grep sume_riffa
+ lsmod
sume_riffa             28672  0
nico@nsg-System:~$
nico@nsg-System:~$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 74:d0:2b:98:38:f6 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9c brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9d brd ff:ff:ff:ff:ff:ff
5: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none
6: nf0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:00 brd ff:ff:ff:ff:ff:ff
7: nf1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:01 brd ff:ff:ff:ff:ff:ff
8: nf2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:02 brd ff:ff:ff:ff:ff:ff
9: nf3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:03 brd ff:ff:ff:ff:ff:ff
nico@nsg-System:~$

\end{verbatim}
\end{tiny}
% ----------------------------------------------------------------------
\section{\label{appendix:netfpgalogs:compilelogs}NetFPGA compile logs}
% ----------------------------------------------------------------------
This section shows a compilation of of NetFPGA compile output and errors.

Unfound tbl files that are not correctly generated fail the compilation:
\begin{tiny}\begin{verbatim}
# Fix introduced for SDNet 2017.4
sed -i 's/xsim\.dir\/xsc\/dpi\.so/dpi\.so/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim.bash
sed -i 's/xsim\.dir\/xsc\/dpi\.so/dpi\.so/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
# Fix introduced for SDNet 2018.2
sed -i 's/glbl_sim/glbl/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
sed -i 's/SimpleSumeSwitch_tb_sim#work.glbl/SimpleSumeSwitch_tb/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
cp src/*.tbl nf_sume_sdnet_ip/SimpleSumeSwitch/
cp: cannot stat 'src/*.tbl': No such file or directory
make: *** [Makefile:23: cpp_test] Error 1
[23:12] loch:minip4%
\end{verbatim}
\end{tiny}
Failure to generate an intermediate file:
\begin{tiny}\begin{verbatim}
 ERROR: [XSIM 43-3409] Failed to compile generated C file xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.c.
 ERROR: [XSIM 43-3915] Encountered a fatal error. Cannot continue. Exiting...
\end{verbatim}
\end{tiny}
Failure to compile because libncurses.so.5 is missing:
\begin{tiny}\begin{verbatim}
/opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/clang  -fPIC -c -std=gnu89 -nobuiltininc -nostdinc++ -w  -Wl,--unres
olved-symbols=ignore-in-object-files -fbracket-depth=1048576 -I/opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/../li
b/clang/3.1/include -fPIC -m64  -I"/opt/Xilinx/Vivado/2018.2/data/xsim/include" "xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/ob
j/xsim_3.c" -O0 -sim -o "xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.lnx64.o" -DXILINX_SIMULATOR
/opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/clang: error while loading shared libraries: libncurses.so.5: cannot
 open shared object file: No such file or directory
ERROR: [XSIM 43-3409] Failed to compile generated C file xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.c.
ERROR: [XSIM 43-3915] Encountered a fatal error. Cannot continue. Exiting...
[20:00] rainbow:SimpleSumeSwitch%
\end{verbatim}
\end{tiny}
Failure to access txt files that were not correctly generated in a
different compilation step:
\begin{tiny}\begin{verbatim}
# Fix introduced for SDNet 2018.2
sed -i 's/glbl_sim/glbl/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
sed -i 's/SimpleSumeSwitch_tb_sim#work.glbl/SimpleSumeSwitch_tb/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
cp src/*.tbl nf_sume_sdnet_ip/SimpleSumeSwitch/
cp testdata/*.txt nf_sume_sdnet_ip/SimpleSumeSwitch/
cp: cannot stat 'testdata/*.txt': No such file or directory
make: *** [Makefile:17: all] Error 1
[15:46] rainbow:minip4%
\end{verbatim}
\end{tiny}
Missing pcap files of non generated testdata causing compile abortion:
\begin{tiny}\begin{verbatim}
make -C testdata/
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata'
./gen_testdata.py
/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/bin/pcap2axi --output Packet_in.axi --bus_width 256 src.pcap
Traceback (most recent call last):
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/bin/pcap2axi", line 108, in <module>
    write_to_file(args.file_pcap, args.output)
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/bin/pcap2axi", line 88, in write_to_file
    for pkt in rdpcap(file_in):
  File "/usr/lib/python2.7/dist-packages/scapy/utils.py", line 728, in rdpcap
    with PcapReader(filename) as fdesc:
  File "/usr/lib/python2.7/dist-packages/scapy/utils.py", line 751, in __call__
    filename, fdesc, magic = cls.open(filename)
  File "/usr/lib/python2.7/dist-packages/scapy/utils.py", line 778, in open
    fdesc = open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'src.pcap'
make[1]: *** [Makefile:5: all] Error 1
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata'
make: *** [Makefile:32: frontend] Error 2
[15:47] rainbow:minip4%
\end{verbatim}
\end{tiny}
Syntax errors due to incorrect generation of a python script:
\begin{tiny}
\begin{verbatim}
update_compile_order: Time (s): cpu = 00:00:17 ; elapsed = 00:00:09 . Memory (MB): peak = 1995.594 ; gain = 0.016 ; free physic
al = 21975 ; free virtual = 33161
loading libsume..
Traceback (most recent call last):
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_de
fault/run.py", line 42, in <module>
    import config_writes
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_de
fault/config_writes.py", line 7

                        ^
IndentationError: expected an indented block
    while executing
"exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py"
    invoked from within
"set output [exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py]"
    (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/tcl/simple_s
ume_switch_sim.tcl" line 177)
INFO: [Common 17-206] Exiting Vivado at Sat May 18 15:21:21 2019...
\end{verbatim}
\end{tiny}
Missing axi files don't abort the compilation process: (shortened for formatting)
\begin{tiny}\begin{verbatim}
cp: cannot stat 'simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_0_stim.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_0_expected.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_1_stim.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_1_expected.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_2_stim.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_2_expected.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_3_stim.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_3_expected.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/dma_0_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/dma_0_expected.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/reg_stim.log': No such file or directory
cp: cannot stat 'simple_sume_switch/test/reg_expect.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/reg_stim.axi': No such file or directory
=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default
                ... using cmd
 ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py',
'--sim', 'xsim']
\end{verbatim}
\end{tiny}
Add Wave error during compilation: (shortened for formatting)
\begin{tiny}\begin{verbatim}
# add_wave $nf_sume_sdnet_ip/out_src_port
# add_wave $nf_sume_sdnet_ip/out_dst_port
# set const_reg_ip /top_tb/top_sim/nf_datapath_0/nf_sume_sdnet_wrapper_1/inst/SimpleSumeSwitch_inst/const_reg_rw_0/
# add_wave_divider {const reg extern signals}
# add_wave $const_reg_ip
ERROR: [Wavedata 42-471] Note: Nothing was found for the following items:
 /top_tb/top_sim/nf_datapath_0/nf_sume_sdnet_wrapper_1/inst/SimpleSumeSwitch_inst/const_reg_rw_0/
ERROR: [Common 17-39] 'add_wave' failed due to earlier errors.

    while executing
"add_wave $const_reg_ip "
    (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/tcl/simple_sume_switch_sim.tcl" line 328)
INFO: [Common 17-206] Exiting Vivado at Sat May 18 15:31:59 2019...
make: *** [Makefile:121: sim] Error 1
make: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test'
512
=== Work directory is /tmp/nico/test/simple_sume_switch
=== Setting up test in /tmp/nico/test/simple_sume_switch/sim_switch_default
cp: cannot stat 'simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory
cp: cannot stat 'simple_sume_switch/test/dma_0_log.axi': No such file or directory
=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ...
using cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim']
[15:31] rainbow:P4-NetFPGA%
\end{verbatim}
\end{tiny}
Compilation error failing to run ``connect\_bd\_intf\_net.''
\begin{tiny}\begin{verbatim}
ERROR: [BD 41-171] The modes of the interface pins 'cfg_interrupt'(Slave) and 'pcie3_cfg_interrupt'(Slave) are incompatible. They cannot be connected.
ERROR: [BD 5-3] Error: running connect_bd_intf_net.
ERROR: [Common 17-39] 'connect_bd_intf_net' failed due to earlier errors.

    while executing
"connect_bd_intf_net -intf_net nf_riffa_dma_1_pcie3_cfg_interrupt [get_bd_intf_pins nf_riffa_dma_1/cfg_interrupt] [get_bd_intf_pins pcie3_7x_1/pcie3_cf..."
    (procedure "create_hier_cell_dma_sub" line 141)
    invoked from within
"create_hier_cell_dma_sub [current_bd_instance .] dma_sub"
    (procedure "create_root_design" line 68)
    invoked from within
"create_root_design """
    (file "./tcl/control_sub.tcl" line 729)

    while executing
"source ./tcl/control_sub.tcl"
    (file "tcl/simple_sume_switch.tcl" line 89)
\end{verbatim}
\end{tiny}
Compilation aborts due to missing IP:
\begin{tiny}\begin{verbatim}
### set NF_10G_INTERFACE3_BASEADDR $M07_BASEADDR
### set NF_10G_INTERFACE3_HIGHADDR $M07_HIGHADDR
### set NF_10G_INTERFACE3_SIZEADDR $M07_SIZEADDR
### set NF_RIFFA_DMA_BASEADDR $M08_BASEADDR
### set NF_RIFFA_DMA_HIGHADDR $M08_HIGHADDR
### set NF_RIFFA_DMA_SIZEADDR $M08_SIZEADDR
Wrote  :
</home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/
hw/project/simple_sume_switch.srcs/sources_1/bd/control_sub/control_sub.bd>
# create_ip -name nf_sume_sdnet -vendor NetFPGA -library NetFPGA -module_name nf_sume_sdnet_ip
ERROR: [Coretcl 2-1134] No IP matching VLNV 'NetFPGA:NetFPGA:nf_sume_sdnet:*' was found. Please check your repository configuration.
INFO: [Common 17-206] Exiting Vivado at Sat May 25 11:52:01 2019...
\end{verbatim}
\end{tiny}
Mismatch: a non-critical critical error that does not abort the compilation process
\begin{tiny}\begin{verbatim}
[SW] CAM_EnableDevice() - done
[2420698]  INFO: finished packet stimulus file
[2735572] ERROR: tuple mismatch for packet 1
expected < tuple_out_digest_data, tuple_out_sume_metadata > = < 0000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000001010000 >
actual   < tuple_out_digest_data, tuple_out_sume_metadata > = < 0000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000010000 >
$finish called at time : 2735572 ps : File
"/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/Testbench/Check.v"
Line 120
\end{verbatim}
\end{tiny}
Missing interface when testing switch\_calc:
\begin{tiny}\begin{verbatim}
root@rainbow:~/master-thesis/netpfga/minip4/sw/hw_test_tool# python switch_calc_tester.py
SIOCSIFADDR: No such device
eth1: ERROR while getting interface flags: No such device
SIOCSIFNETMASK: No such device
tcpdump: eth1: No such device exists
(SIOCGIFHWADDR: No such device)
The HW testing tool for the switch_calc design
 type help to see all commands
testing>
\end{verbatim}
\end{tiny}
Ioctl error when adding table errors on the first NetFPGA card:
\begin{verbatim}
>> table_cam_add_entry lookup_table send_to_port1 ff:ff:ff:ff:ff:ff =>
CAM_Init_ValidateContext() - done
WROTE 0x44020050 = 0xffffffff
WROTE 0x44020054 = 0xffff
WROTE 0x44020080 = 0x0003
python: ioctl: Unknown error 512
[20:27] rainbow:CLI%
\end{verbatim}
Exec format errors when loading the kernel module due to incompabilities:
\begin{tiny}\begin{verbatim}
[7:05] rainbow:netpfga% bash  build-load-drivers.sh
+ cd /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0
+ make all
make -C /lib/modules/5.0.0-16-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-5.0.0-16-generic'
  Building modules, stage 2.
  MODPOST 1 modules
make[1]: Leaving directory '/usr/src/linux-headers-5.0.0-16-generic'
+ sudo make install
make -C /lib/modules/5.0.0-16-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-5.0.0-16-generic'
  Building modules, stage 2.
  MODPOST 1 modules
make[1]: Leaving directory '/usr/src/linux-headers-5.0.0-16-generic'
install -o root -g root -m 0755 -d /lib/modules/5.0.0-16-generic/extra/sume_riffa/
install -o root -g root -m 0755 sume_riffa.ko /lib/modules/5.0.0-16-generic/extra/sume_riffa/
depmod -a 5.0.0-16-generic
+ sudo modprobe sume_riffa
modprobe: ERROR: could not insert 'sume_riffa': Exec format error
[7:06] rainbow:netpfga%
\end{verbatim}
\end{tiny}
Java traceback when trying to install SDNET:
(reason was a hidden window)
\begin{tiny}
\begin{verbatim}
Exception in thread "AWT-EventQueue-0" java.lang.IllegalArgumentException: Window must not be zero
	at java.desktop/sun.awt.X11.XAtom.checkWindow(Unknown Source)
	at java.desktop/sun.awt.X11.XAtom.getAtomData(Unknown Source)
	at java.desktop/sun.awt.X11.XToolkit.getWorkArea(Unknown Source)
	at java.desktop/sun.awt.X11.XToolkit.getInsets(Unknown Source)
	at java.desktop/sun.awt.X11.XToolkit.getScreenInsets(Unknown Source)
	at java.desktop/java.awt.Window.init(Unknown Source)
	at java.desktop/java.awt.Window.<init>(Unknown Source)
	at java.desktop/java.awt.Window.<init>(Unknown Source)
	at java.desktop/java.awt.Dialog.<init>(Unknown Source)
	at java.desktop/java.awt.Dialog.<init>(Unknown Source)
	at java.desktop/javax.swing.JDialog.<init>(Unknown Source)
	at java.desktop/javax.swing.JOptionPane.createDialog(Unknown Source)
	at java.desktop/javax.swing.JOptionPane.createDialog(Unknown Source)
	at j.a.c(Unknown Source)
	at j.a.a(Unknown Source)
	at j.a.a(Unknown Source)
	at j.a.c(Unknown Source)
	at com.xilinx.installer.gui.panel.destination.b.a(Unknown Source)
	at com.xilinx.installer.gui.panel.destination.DestinationPanel.z(Unknown Source)
	at com.xilinx.installer.gui.E.a(Unknown Source)
	at com.xilinx.installer.gui.InstallerGUI.l(Unknown Source)
	at com.xilinx.installer.gui.i.actionPerformed(Unknown Source)
	at java.desktop/javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
	at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
	at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
	at java.desktop/javax.swing.DefaultButtonModel.setPressed(Unknown Source)
	at java.desktop/javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
	at java.desktop/java.awt.Component.processMouseEvent(Unknown Source)
	at java.desktop/javax.swing.JComponent.processMouseEvent(Unknown Source)
	at java.desktop/java.awt.Component.processEvent(Unknown Source)
	at java.desktop/java.awt.Container.processEvent(Unknown Source)
	at java.desktop/java.awt.Component.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Container.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Component.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
	at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
	at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.Container.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Window.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Component.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.EventQueue.access$500(Unknown Source)
	at java.desktop/java.awt.EventQueue$3.run(Unknown Source)
	at java.desktop/java.awt.EventQueue$3.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.run(Unknown Source)
\end{verbatim}
\end{tiny}
Failures when testing the first NetFPGA card
\begin{tiny}\begin{verbatim}
---------------------------------------------
[ddr3B]: Running Auto Test
---------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16765, in <lambda>
    lambda event: event.callable(*event.args, **event.kw) )
  File "sw/host/script/NfSumeTest.py", line 848, in UpdateProgress
    self.progressDlg.Update(self.curProgress, str(localLine))
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16710, in __getattr__
    raise PyDeadObjectError(self.attrStr % self._name)
wx._core.PyDeadObjectError: The C++ part of the NfSumeProgress object has been deleted, attribute access no longer allowed.
Exception in thread Thread-18:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "sw/host/script/NfSumeTest.py", line 947, in run
    self.target(*self.data)
  File "sw/host/script/NfSumeTest.py", line 355, in StartAutoTest
    self.TestInterface(testName)
  File "sw/host/script/NfSumeTest.py", line 465, in TestInterface
    self.ProgramFpga('../../../bitfiles/' + self.nfSumeTestConfiguration[testName]['bitstream'])
  File "sw/host/script/NfSumeTest.py", line 586, in ProgramFpga
    self.getFpgaIndex()
  File "sw/host/script/NfSumeTest.py", line 574, in getFpgaIndex
    p = Popen(['djtgcfg', 'init', '-d', 'NetSUME'], stdout=PIPE, bufsize = 1)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
\end{verbatim}
\end{tiny}
More failures when testing the first NetFPGA card
\begin{tiny}
\begin{verbatim}
---------------------------------------------
[pcie]: Running Auto Test
---------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16765, in <lambda>
    lambda event: event.callable(*event.args, **event.kw) )
  File "sw/host/script/NfSumeTest.py", line 848, in UpdateProgress
    self.progressDlg.Update(self.curProgress, str(localLine))
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16710, in __getattr__
    raise PyDeadObjectError(self.attrStr % self._name)
wx._core.PyDeadObjectError: The C++ part of the NfSumeProgress object has been deleted, attribute access no longer allowed.
Exception in thread Thread-21:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "sw/host/script/NfSumeTest.py", line 947, in run
    self.target(*self.data)
  File "sw/host/script/NfSumeTest.py", line 466, in TestInterface
    self.serialCon.readlines()
  File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 495, in read
    raise SerialException('device reports readiness to read but returned no data (device disconnected or multiple access on port?)')
SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)
\end{verbatim}
\end{tiny}
Unexpected EOF during compilation:
\begin{verbatim}
ERROR: [VRFC 10-1491] unexpected EOF
[/home/nico/master-thesis/netpfga/minip4/nf_sume_sdnet_ip/
SimpleSumeSwitch/S_CONTROLLERs.HDL/S_CONTROLLER_SimpleSumeSwitch.vp:37]
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_0_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_1_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_2_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_3_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_4_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_5_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_6_ErrorCheck
\end{verbatim}
The function syntax is not supported by p4/netfpga:
\begin{tiny}
\begin{verbatim}
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4
headers.p4(246):syntax error, unexpected IDENTIFIER, expecting (
bit<16> ones_complement_sum
        ^^^^^^^^^^^^^^^^^^^
error: 1 errors encountered, aborting compilation
Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 1
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
Makefile:31: recipe for target 'frontend' failed
make: *** [frontend] Error 2
nico@nsg-System:~/master-thesis/netpfga$
\end{verbatim}
\end{tiny}
The config\_writes.py is missing due to a previous, non critical
compilation error:
\begin{tiny}\begin{verbatim}
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default$
cd $NF_DESIGN_DIR/test/sim_switch_default && make 2>&1 |  tee ~/master-thesis/netpfga/log/step8-$(date +%F-%H%M%S)
rm -f config_writes.py*
rm -f *.pyc
cp /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.py ./
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.py': No such file or directory
Makefile:36: recipe for target 'all' failed
make: *** [all] Error 1
\end{verbatim}
\end{tiny}
Failed to synthesizing module errors:
\begin{tiny}\begin{verbatim}
WARNING: [Synth 8-689] width (12) of port connection 'control_S_AXI_ARADDR' does not match port width (8) of module 'SimpleSumeSwitch'
[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/
simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:199]
ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_VALID' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch'
[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/
simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:218]
ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_DATA' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch'
[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/
simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:219]
ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet'
[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/
simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:44]
ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet_ip'
[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/
simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/synth/nf_sume_sdnet_ip.v:57]
ERROR: [Synth 8-6156] failed synthesizing module 'nf_datapath'
[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/
simple_sume_switch/hw/hdl/nf_datapath.v:44]
ERROR: [Synth 8-6156] failed synthesizing module 'top'
[/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/
simple_sume_switch/hw/hdl/top.v:43]
\end{verbatim}
\end{tiny}
Missing ``souce'' files abort CLI compilation errors:
\begin{tiny}\begin{verbatim}
cc -c -fPIC /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API/CAM.c
-I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API
cc -std=c99 -Wall -Werror -fPIC -c libcam.c
-I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume
-I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API
cc -L/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume
 -shared -o libcam.so libcam.o CAM.o -lsumereg
/usr/bin/ld: cannot find -lsumereg
collect2: error: ld returned 1 exit status
Makefile:52: recipe for target 'libcam' failed
make[1]: *** [libcam] Error 1
make[1]: Leaving directory '/home/nico/master-thesis/netpfga/minip4/sw/CLI'
ERROR: could not compile libcam souce files
\end{verbatim}
\end{tiny}
Generated axi files not found at a different stage:
\begin{tiny}
\begin{verbatim}
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/
sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/
sume-sdnet-switch/projects/minip4/simple_sume_switch/test/Makefile': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/
sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.log': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/
sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_expect.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/
sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.axi': No such file or directory
NetFPGA environment:
   Root dir:       /home/nico/projects/P4-NetFPGA
   Project name:   simple_sume_switch
   Project dir:    /tmp/nico/test/simple_sume_switch
   Work dir:       /tmp/nico
512
=== Work directory is /tmp/nico/test/simple_sume_switch
=== Setting up test in /tmp/nico/test/simple_sume_switch/sim_switch_default
=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ... using
cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/
minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim']+ date
Die Jul 23 13:34:54 CEST 2019
+ [  = no ]
+ cd /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch
+ make
make: *** No targets specified and no makefile found.  Stop.
\end{verbatim}
\end{tiny}
Renaming variables as follows breaks the compile process
\begin{verbatim}
 @Xilinx_MaxPacketRegion(1024)
 control TopDeparser(
-    packet_out b,
-    in Parsed_packet p,
+    packet_out packet,
+    in Parsed_packet hdr,
     in user_metadata_t user_metadata,
     inout digest_data_t digest_data,
     inout sume_metadata_t sume_metadata) {

     apply {
-        b.emit(p.ethernet);
+        packet.emit(hdr.ethernet);
     }
+
+
 }
\end{verbatim}
In NetPFGA the LPM table size must be != 64:
\begin{tiny}
\begin{verbatim}
minip4_solution.p4(38): [--Wwarn=uninitialized_out_param] warning: out parameter meta may be uninitialized when RealParser terminates
    out metadata meta,
                 ^^^^
minip4_solution.p4(35)
parser RealParser(
       ^^^^^^^^^^
error: LPM table size should be 2^n - 1
actions_nat64_generic.p4(169): error: could not not map table size size
        size = 64;
        ^^^^
error: table match_types are not the same
actions_arp.p4(35): error: could not map table key(s) KeyElement
            hdr.arp.dst_ipv4_addr: lpm;
            ^^^^^^^^^^^^^^^^^^^^^
error: LPM table size should be 2^n - 1
actions_arp.p4(55): error: could not not map table size size
        size = 64;
        ^^^^
Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 1
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
Makefile:31: recipe for target 'frontend' failed
make: *** [frontend] Error 2
nico@nsg-System:~/master-thesis/netpfga/log$
\end{verbatim}
\end{tiny}
Cannot mix the key table types with P4/NetFPGA:
\begin{tiny}\begin{verbatim}
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4
actions_egress.p4(52): warning: Table v6_networks is not used; removing
table v6_networks {
      ^^^^^^^^^^^
actions_egress.p4(69): warning: Table v4_networks is not used; removing
table v4_networks {
      ^^^^^^^^^^^
actions_nat64_generic.p4(174): warning: Table nat46 is not used; removing
    table nat46 {
          ^^^^^
minip4_solution.p4(38): [--Wwarn=uninitialized_out_param] warning: out parameter meta may be uninitialized when RealParser terminates
    out metadata meta,
                 ^^^^
minip4_solution.p4(35)
parser RealParser(
       ^^^^^^^^^^
error: table match_types are not the same
actions_arp.p4(35): error: could not map table key(s) KeyElement
            hdr.arp.dst_ipv4_addr: lpm;
            ^^^^^^^^^^^^^^^^^^^^^
Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 1

    table v4_arp {
        key = {
            hdr.ethernet.dst_addr: exact;
            hdr.arp.opcode: exact;
            hdr.arp.dst_ipv4_addr: lpm;
        }
        actions = {
            controller_debug_table_id;
            arp_reply;
            NoAction;
        }
        size = ICMP6_TABLE_SIZE;
        default_action = controller_debug_table_id(TABLE_ARP);
    }
\end{verbatim}
\end{tiny}
Implicit error saying that LPM tables don't work in P4/NetFPGA:
\begin{tiny}\begin{verbatim}
s/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/realmain_lookup_table_0_t.HDL/xpm_memory.sv
[SW] LPM_Init() - start
[SW] LPM_Init() - done
[SW] LPM_LoadDataset() - start
[SW] LPM_LoadDataset() failed with error code = 12
FATAL_ERROR: Vivado Simulator kernel has encounted an exception from DPI C function: LPM_VerifyDataset(). Please correct.
Time: 2016466 ps  Iteration: 0  Process: /SimpleSumeSwitch_tb/LPM_VerifyDataset
  File: /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/Testbench/SimpleSumeSwitch_tb.sv
\end{verbatim}
\end{tiny}
The table for exact matches must be at least 64 in P4/NetFPGA:
\begin{tiny}\begin{verbatim}
minip4_solution.p4(35)
parser RealParser(
       ^^^^^^^^^^
actions_nat64_generic.p4(173): error: table size too small for match_type(EM): 63 < 64
        size = 63;
               ^^
actions_nat64_generic.p4(173): error: could not not map table size size
        size = 63;
        ^^^^
\end{verbatim}
\end{tiny}
Unsupported default parameters in P4/NetFPGA:
\begin{tiny}\begin{verbatim}
actions_egress.p4(89): error: data-plane arguments in default_actions are currently unsupported: realmain_controller_debug_table_id_0
    default_action = controller_debug_table_id(TABLE_V4_NETWORKS);
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
terminate called after throwing an instance of 'Util::CompilerBug'
  what():  In file: /wrk/hdscratch/staff/mohan/p4c_sdnet/build/p4c/extensions/sdnet/translate/core/lookupEngine.cpp:137
Compiler Bug: actions_egress.p4(89): unhandled expression realmain_controller_debug_table_id/realmain_controller_debug_table_id_0(5);
    default_action = controller_debug_table_id(TABLE_V4_NETWORKS);
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
\end{verbatim}
\end{tiny}
Causing compiler bug by using an if statement at a wrong place in P4/NetFPGA:
\begin{tiny}\begin{verbatim}
minip4_solution.p4(39)
parser RealParser(
       ^^^^^^^^^^
terminate called after throwing an instance of 'Util::CompilerBug'
  what():  In file: /wrk/hdscratch/staff/mohan/p4c_sdnet/build/p4c/extensions/sdnet/writers/pxWriter.h:20
Compiler Bug: unhandled node: <IfStatement>(471564)

Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 134
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
Makefile:31: recipe for target 'frontend' failed
\end{verbatim}
\end{tiny}
Applying table ``twice'' in different branches is impossible in
P4/NetFPGA causes a different compiler bug:
\begin{tiny}
\begin{verbatim}
make -C src/
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4
minip4_solution.p4(19): [--Wwarn=uninitialized_out_param] warning: out parameter meta may be uninitialized when RealParser terminates
    out metadata meta,
                 ^^^^
minip4_solution.p4(16)
parser RealParser(
       ^^^^^^^^^^
terminate called after throwing an instance of 'Util::CompilerBug'
  what():  In file: /wrk/hdscratch/staff/mohan/p4c_sdnet/build/p4c/extensions/sdnet/translate/core/tupleEngine.cpp:324
Compiler Bug: overwrite

Makefile:34: recipe for target 'all' failed
\end{verbatim}
\end{tiny}
Adding table entries requires setting parameters for all possible
actions that are registered in a table:
\begin{tiny}
\begin{verbatim}
>> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port
42540766411362381960998550477184434178  => 1 ERROR: not enough fields provided to complete _hexify()
\end{verbatim}
\end{tiny}
The table handling scripts do not handle conversion for long integers
for P4/NetFPGA:
\begin{tiny}
\begin{verbatim}
>> table_cam_delete_entry realmain_v6_networks_0 42540766411362381960998550477184434179
ERROR: failed to convert 42540766411362381960998550477184434179 of type <type 'long'> to an integer
nico@nsg-System:~/master-thesis/netpfga/minip4/sw/CLI$
\end{verbatim}
\end{tiny}
A P4/BMV2 compiler error:
\begin{tiny}
\begin{verbatim}
Warning: you requested the nanomsg event logger, but bmv2 was compiled without -DBMELOG, and the event logger cannot be activated
Calling target program-options parser
[14:01:44.334] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.icmp6': MyIngress.controller_debug_table_id - 2,
[14:01:44.341] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.nat64': MyIngress.controller_debug_table_id - 1,
[14:01:44.344] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act': act -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_0': act_0 -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_nat64_icmp6_generic': MyIngress.nat64_icmp6_generic -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_1': act_1 -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_2': act_2 -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.v4_networks': MyIngress.controller_debug_table_id - 5,
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.v6_networks': MyIngress.controller_debug_table_id - 3,
[14:01:44.346] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_3': act_3 -
Invalid entry type 'expression' in field list
bad json:
{
   "type" : "expression",
   "value" : {
      "type" : "expression",
      "value" : {
         "left" : null,
         "op" : "d2b",
         "right" : {
            "type" : "field",
            "value" : [ "scalars", "metadata.chk_icmp6_na_ns" ]
         }
      }
   }
}
\end{verbatim}
\end{tiny}

Inability of P4/BMV2 to have multiple LPM keys in a table:
\begin{verbatim}
../p4src/static-mapping.p4(121): error: MyIngress.nat64, Multiple LPM keys in table
    table nat64 {
          ^^^^^
Compilation Error

    table nat64 {
        key = {
            hdr.ipv6.src_addr: lpm;
            hdr.ipv6.dst_addr: lpm;
        }
        actions = {
            controller_debug;
            nat64_static;
            NoAction;
        }
        size = NAT64_TABLE_SIZE;
        default_action = controller_debug;
    }
\end{verbatim}
Switch statements are not allowed in P4/BMV:
\begin{tiny}
\begin{verbatim}
../p4src/static-mapping.p4(60): error: SwitchStatement: switch statements not allowed in actions
        switch(hdr.icmp6.type) {
        ^^^^^^
\end{verbatim}
\end{tiny}
And also no ifs in actions:
\begin{tiny}
\begin{verbatim}
../p4src/static-mapping.p4(57): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp.setValid();
        ^^^^^^^^^^^^^^^^^^^
../p4src/static-mapping.p4(70): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp6.setInvalid();
        ^^^^^^^^^^^^^^^^^^^^^^
../p4src/static-mapping.p4(73): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp6_na_ns.setInvalid();
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../p4src/static-mapping.p4(74): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp6_option_link_layer_addr.setInvalid();
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Compilation Error
p4@ubuntu:~/master-thesis/p4app$

        if(hdr.ipv6.next_header == PROTO_ICMP6) {
            nat64_icmp6();
        }
\end{verbatim}
\end{tiny}
Compiler bug in P4/BMV2:
\begin{tiny}
\begin{verbatim}
p4c --target bmv2 --arch v1model --std p4-16 "../p4src/checksum_diff.p4" -o "/home/p4/master-thesis/p4src"
In file: /home/p4/p4-tools/p4c/backends/bmv2/common/expression.cpp:168
Compiler Bug: ../p4src/actions_delta_checksum.p4(60): ones_complement_sum(hdr.udp.checksum, tmp);: unhandled case
    tmp = ones_complement_sum(hdr.udp.checksum, meta.v6sum);
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Compilation Error```
Using the following code:
```/* copied from
https://p4.org/p4-spec/docs/PSA-v1.1.0.html#appendix-internetchecksum-implementation
*/

bit<16> ones_complement_sum(in bit<16> x, in bit<16> y) {
    bit<17> ret = (bit<17>) x + (bit<17>) y;
    if (ret[16:16] == 1) {
        ret = ret + 1;
    }
    return ret[15:0];
}```
And p4c version:
```p4@ubuntu:~/master-thesis/p4app$ p4c --version
p4c 0.5 (SHA: 5ae30ee)```
\end{verbatim}
\end{tiny}

%----------------------------------------------------------------------
\chapter{\label{benchmark}Benchmark Logs}
% ----------------------------------------------------------------------
\section{\label{benchmark:offset}Enabling hardware offloading}
The following commands enable hardware offloading even though error
messages are printed:
\begin{verbatim}
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: on
        tx-checksum-ip-generic: on
        tx-checksum-sctp: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp6-segmentation: on
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~#
\end{verbatim}
This results into the following:
\begin{verbatim}
root@ESPRIMO-P956:~# ethtool -k enp2s0f1
Features for enp2s0f1:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off
root@ESPRIMO-P956:~#
\end{verbatim}

%---------------------------------------------------------------------------------------------------------
\section{\label{benchmark:tayga}Tayga}
Tayga is installed from the regular package database:
\begin{verbatim}
ii  tayga                                       0.9.2-6                                      amd64        userspace stateless NAT64
\end{verbatim}
We prepare the networking as follows:
\begin{verbatim}
[15:12] nsg-System:~# ip addr add 10.0.0.77/24 dev eth1
[15:12] nsg-System:~# ip l s eth1 up

nico@ESPRIMO-P956:~$ ~/master-thesis/bin/init_ipv4_esprimo.sh
nico@ESPRIMO-P956:~$ cat ~/master-thesis/bin/init_ipv4_esprimo.sh
#!/bin/sh

sudo ip addr add 10.0.0.42/24 dev enp2s0f0
sudo ip link set enp2s0f0 up

nico@ESPRIMO-P956:~$ sudo ip route add 10.0.1.0/24 via 10.0.0.77
\end{verbatim}
And verify that networking works:
\begin{verbatim}
[15:12] nsg-System:~# ping 10.0.0.42
PING 10.0.0.42 (10.0.0.42) 56(84) bytes of data.
64 bytes from 10.0.0.42: icmp_seq=1 ttl=64 time=0.304 ms
64 bytes from 10.0.0.42: icmp_seq=2 ttl=64 time=0.097 ms
^C
--- 10.0.0.42 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1011ms
rtt min/avg/max/mdev = 0.097/0.200/0.304/0.104 ms
[15:12] nsg-System:~#
\end{verbatim}
We also setup the IPv6 networking:
\begin{tiny}\begin{verbatim}
nico@ESPRIMO-P956:~$ ip addr show dev enp2s0f1
13: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8:42::42/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::faf2:1eff:fe09:62d1/64 scope link
       valid_lft forever preferred_lft forever
nico@ESPRIMO-P956:~$ sudo ip route add 2001:db8:23::/96 via 2001:db8:42::77

[15:12] nsg-System:~# ip addr add 2001:db8:42::77/64 dev eth2
[15:15] nsg-System:~# ip link set eth2 up
\end{verbatim}
\end{tiny}
And verify that IPv6 networking works:
\begin{verbatim}
nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::77
PING 2001:db8:42::77(2001:db8:42::77) 56 data bytes
64 bytes from 2001:db8:42::77: icmp_seq=1 ttl=64 time=0.169 ms
64 bytes from 2001:db8:42::77: icmp_seq=2 ttl=64 time=0.153 ms

--- 2001:db8:42::77 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1010ms
rtt min/avg/max/mdev = 0.153/0.161/0.169/0.008 ms
nico@ESPRIMO-P956:~$
\end{verbatim}
We enable IPv6 and IPv4 forwarding:
\begin{verbatim}
[15:16] nsg-System:~# sysctl -w net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.forwarding = 1

[15:20] nsg-System:~# sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
\end{verbatim}
And we test NAT64 with tayga:
\begin{tiny}
\begin{verbatim}
nico@ESPRIMO-P956:~$ ping -c2 10.0.1.42
PING 10.0.1.42 (10.0.1.42) 56(84) bytes of data.
64 bytes from 10.0.1.42: icmp_seq=1 ttl=61 time=0.356 ms
64 bytes from 10.0.1.42: icmp_seq=2 ttl=61 time=0.410 ms

--- 10.0.1.42 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1019ms
rtt min/avg/max/mdev = 0.356/0.383/0.410/0.027 ms
nico@ESPRIMO-P956:~$

nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:21:39.851057 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 1, length 64
15:21:39.851124 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 1, length 64
15:21:40.870448 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 2, length 64
15:21:40.870507 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 2, length 64
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
nico@ESPRIMO-P956:~$
\end{verbatim}
\end{tiny}

And test NAT64 from IPv6 to IPv4:
\begin{verbatim}
nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:23::a00:2a
PING 2001:db8:23::a00:2a(2001:db8:23::a00:2a) 56 data bytes
64 bytes from 2001:db8:23::a00:2a: icmp_seq=1 ttl=61 time=0.240 ms
64 bytes from 2001:db8:23::a00:2a: icmp_seq=2 ttl=61 time=0.400 ms

--- 2001:db8:23::a00:2a ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.240/0.320/0.400/0.080 ms
nico@ESPRIMO-P956:~$
\end{verbatim}
%---------------------------------------------------------------------------------------------------------
\section{\label{benchmark:jool}Jool}
We install Jool 4.0.1 from source from
\url{https://www.jool.mx/en/download.html} as follows:
\begin{tiny}
\begin{verbatim}
nico@nsg-System:~$ wget https://github.com/NICMx/Jool/releases/download/v4.0.1/jool_4.0.1.tar.gz
nico@nsg-System:~$ tar xvfz jool_4.0.1.tar.gz
nico@nsg-System:~$ cd jool-4.0.1/
nico@nsg-System:~/jool-4.0.1$ sudo  apt install linux-headers-$(uname -r)
nico@nsg-System:~/jool-4.0.1$ sudo  apt install libnl-genl-3-dev
nico@nsg-System:~/jool-4.0.1$ sudo apt install iptables-dev
nico@nsg-System:~/jool-4.0.1$ sudo make install
\end{verbatim}
\end{tiny}
We enable forwarding:
\begin{verbatim}
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
\end{verbatim}
We configure jool to map the network prefixes and setup iptables to
redirect the traffic into the jool instance:
\begin{tiny}
\begin{verbatim}
[16:53] nsg-System:~# modprobe jool_siit
[16:54] nsg-System:~# jool_siit instance add "example" --iptables
[16:54] nsg-System:~# jool_siit -i example eamt add 2001:db8:42::/120 10.0.1.0/24
[16:55] nsg-System:~# jool_siit -i example eamt add 2001:db8:23::/120 10.0.0.0/24
[16:57] nsg-System:~# ip6tables -t mangle -A PREROUTING -s 2001:db8:42::/120 -d 2001:db8:23::/120 -j JOOL_SIIT --instance example
[16:57] nsg-System:~# iptables -t mangle -A PREROUTING -s 10.0.0.0/24 -d 10.0.1.0/24 -j JOOL_SIIT --instance example
\end{verbatim}
\end{tiny}
Afterwards we test NAT64:
\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ ping6 2001:db8:23::2a
PING 2001:db8:23::2a(2001:db8:23::2a) 56 data bytes
64 bytes from 2001:db8:23::2a: icmp_seq=1 ttl=63 time=0.199 ms
64 bytes from 2001:db8:23::2a: icmp_seq=2 ttl=63 time=0.282 ms
64 bytes from 2001:db8:23::2a: icmp_seq=3 ttl=63 time=0.186 ms
^C
--- 2001:db8:23::2a ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2040ms
rtt min/avg/max/mdev = 0.186/0.222/0.282/0.044 ms
nico@ESPRIMO-P956:~/master-thesis/iperf$ ping 10.0.1.66
PING 10.0.1.66 (10.0.1.66) 56(84) bytes of data.
64 bytes from 10.0.1.66: icmp_seq=1 ttl=63 time=0.218 ms
64 bytes from 10.0.1.66: icmp_seq=2 ttl=63 time=0.281 ms
64 bytes from 10.0.1.66: icmp_seq=3 ttl=63 time=0.280 ms
^C
--- 10.0.1.66 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2051ms
rtt min/avg/max/mdev = 0.218/0.259/0.281/0.034 ms
nico@ESPRIMO-P956:~/master-thesis/iperf$
\end{verbatim}
%---------------------------------------------------------------------------------------------------------
\printnomenclature
\abbrev{ARP}{Address resolution protocol}
\abbrev{ASIC}{Application-specific integrated circuit}
\abbrev{DAC}{Direct attach cable}
\abbrev{FGPA}{Field-programmable gate array}
\abbrev{LPM}{Longes prefix matching}
\abbrev{MTU}{Maximum transfer unit}
\abbrev{NDP}{Neighbor Discovery Protocol}
\abbrev{NAT}{Network Address Translation}
\abbrev{NAT64}{Network Address Translation from / to IPv6 to / from IPv4}
\abbrev{RIR}{Regional Internet Registry}
\abbrev{RTT}{Round Trip Time}
