\chapter{\label{appendixminus1}Resources and code repositories}
The following sections describe how to acquire the resources
to reproduce the test results. All compilations were made on Ubuntu
16.04 with kernels
\begin{itemize}
\item 4.15.0-54-generic (Supporting Desktop)
\item 4.4.0-143-generic (BMV2 test VM)
\item 4.15.0-55-generic (Desktop with NetFPGA card)
\end{itemize}
% ok
% ----------------------------------------------------------------------
\section{\label{chapterminus1:thesis:general}Master Thesis}
The master thesis including all self developed source code is
available by git via
\begin{verbatim}
git clone git@gitlab.ethz.ch:nicosc/master-thesis.git
git clone https://gitlab.ethz.ch/nsg/student-projects/ma-2019-19_high_speed_nat64_with_p4
\end{verbatim}
It can be browsed online on
\url{https://gitlab.ethz.ch/nicosc/master-thesis} and on
\url{https://gitlab.ethz.ch/nsg/student-projects/ma-2019-19_high_speed_nat64_with_p4}.
% ok
% ----------------------------------------------------------------------
\section{\label{chapterminus1:thesis:xilinx}Xilinx Toolchain}
A prerequisite for building the NetFGPA source code is the
installation of
\begin{itemize}
\item \verb=Xilinx_SDNet_2018.2_1005_9=
\item \verb=Xilinx_Vivado_SDK_2018.2_0614_1954=
\end{itemize}
Both tools need to be installed to /opt/Xilinx/, as paths are
hardcoded in various places.
% ok
% ----------------------------------------------------------------------
\section{\label{chapterminus1:thesis}NetFGPA support scripts}
To be able to compile P4 source code to the NetFPGA the collection of
scripts, Makefiles and sample code of P4-NetFGPA is required.

The repository \url{git@github.com:NetFPGA/P4-NetFPGA-live.git} needs
to be cloned to ``projects'' subdirectory as ``P4-NetPFGA''
of the user that wants to
compile the source code. Access to the repository is granted after
applying for access as described on
\url{https://github.com/NetFPGA/P4-NetFPGA-public/wiki}.
After that the variable \verb=P4_PROJECT_NAME= in
\texttt{~/projects/P4-NetFPGA/tools/settings.sh} needs to be modified to
read \verb|export P4_PROJECT_NAME=minip4| instead of
\verb|export P4_PROJECT_NAME=switch_calc|.
Sample code for installation:

\begin{verbatim}
  mkdir -p ~/projects
  git clone git@github.com:NetFPGA/P4-NetFPGA-live.git P4-NetFPGA
  sed -i 's/\(P4_PROJECT_NAME=\).*/\1minip4/' ~/projects/P4-NetFPGA/tools/settings.sh
\end{verbatim}
Version \textbf{v1.3.1-46-g97d3aaa} of the P4-NetPFGA repository was
used for creating the bitfiles of this project.
\begin{verbatim}
nico@nsg-System:~/projects/P4-NetFPGA$ git describe --always
v1.3.1-46-g97d3aaa
\end{verbatim}
% ok
% ----------------------------------------------------------------------
\chapter{\label{appendix0}BMV2 environment and tests}
All BMV2 based compilations were made with the following compiler:
\begin{verbatim}
p4@ubuntu:~$ p4c --version
p4c 0.5 (SHA: 5ae30ee)
\end{verbatim}

The installation is based on the vagrant files that were provided in
the ``Advanced Topics in
Communication Networks Fall 2018'' course of
ETHZ (\url{https://adv-net.ethz.ch/2018/}) and contains p4tools as
well as all utilities that came with the vagrant installation.
% ok
% ----------------------------------------------------------------------
%\section{\label{chapter0:bmv2-diff}Diff based checksumming}
For running the diff based checksum code, the following steps are
necessary:

Compiling the p4 code and starting the switch:
\begin{verbatim}
cd ~/master-thesis/p4app
sudo p4run --config nat64-diff.json
\end{verbatim}

Starting the controller which sets up the required table entries:

\begin{verbatim}
cd ~/master-thesis/p4app
sudo python ./controller.py --mode range_router
\end{verbatim}


% ----------------------------------------------------------------------
\chapter{\label{appendixA}NetFPGA environment and tests}
% ----------------------------------------------------------------------
\section{\label{chapterA:netpfga-setup}NetFPGA Setup}
Description of installation, commit of netpfga-live
% ----------------------------------------------------------------------

\section{\label{chapterA:netpfga:compile}NetFPGA Compile Flow}

% ----------------------------------------------------------------------
\section{\label{chapterA:section1}NetFPGA NAT64 Test cases}
todo: add graphic of nsg <-> esprimo cabling
\begin{verbatim}

ip addr add 10.0.0.42/24 dev enp2s0f0

# Adding necessary ARP entries: for the virtual IPv4 address(es)
ip neigh add 10.0.0.6  lladdr f8:f2:1e:09:62:d1 dev enp2s0f0
ip neigh add 10.0.0.42 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0

\end{verbatim}

For all test cases the following network settings on esprimo:

\begin{verbatim}
12: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:09:62:d0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.42/24 scope global enp2s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::faf2:1eff:fe09:62d0/64 scope link
       valid_lft forever preferred_lft forever
13: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8:42::42/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::faf2:1eff:fe09:62d1/64 scope link
       valid_lft forever preferred_lft forever
\end{verbatim}
% ----------------------------------------------------------------------
\subsection{Test 1: IPv4 egress}
Scenario: simple egress port setting for the IPv4 addresses

Step 1: getting correct values for table entries from python:

\begin{verbatim}
>>> int(ipaddress.IPv4Address(u"10.0.0.42"))
167772202
>>> int(ipaddress.IPv4Address(u"10.0.0.4"))
167772164
>>>

\end{verbatim}

Step 2: setting table netries

\begin{verbatim}

>> table_cam_add_entry realmain_v4_networks_0 realmain.set_egress_port 167772202 => 16 0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '16', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020250 = 0xa00002a
WROTE 0x44020280 = 0x0000
WROTE 0x44020284 = 0x0000
WROTE 0x44020288 = 0x10000000
WROTE 0x4402028c = 0x0001
READ  0x44020244 = 0x0001
WROTE 0x44020240 = 0x0001
READ  0x44020244 = 0x0001
READ  0x44020244 = 0x0001
success
>> table_cam_add_entry realmain_v4_networks_0 realmain.set_egress_port 167772164 => 16 0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '16', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020250 = 0xa000004
WROTE 0x44020280 = 0x0000
WROTE 0x44020284 = 0x0000
WROTE 0x44020288 = 0x10000000
WROTE 0x4402028c = 0x0001
READ  0x44020244 = 0x0001
WROTE 0x44020240 = 0x0001
READ  0x44020244 = 0x0001
READ  0x44020244 = 0x0001
success
>>
\end{verbatim}

Step 3: setting arp entries

\begin{verbatim}
root@ESPRIMO-P956:~# ip neigh add 10.0.0.6 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0
root@ESPRIMO-P956:~# ip neigh add 10.0.0.4 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0

\end{verbatim}
Step 3: generating test packets, expecting 4 packets to show up on
enp2s0f0:

%\tvb
\begin{verbatim}
nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:49:28.200407 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 1, length 64
10:49:28.200445 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 1, length 64
10:49:29.222340 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64
10:49:29.222418 IP 10.0.0.42 > 10.0.0.4: ICMP echo request, id 4440, seq 2, length 64
\end{verbatim}
%\tve

Result: sucess


% ----------------------------------------------------------------------
\subsection{Test 2: IPv6 egress}
Similar to the IPv4 setting before, just for IPv6.

Step 1: getting IP address values

\begin{verbatim}
>>> int(ipaddress.IPv6Address(u"2001:db8:42::4"))
42540766411362381960998550477184434180L
>>> int(ipaddress.IPv6Address(u"2001:db8:42::6"))
42540766411362381960998550477184434182L
>>> int(ipaddress.IPv6Address(u"2001:db8:42::42"))
42540766411362381960998550477184434242L

\end{verbatim}

Step 2: setting table entries

\begin{verbatim}
>> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434182 => 64  0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '64', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020350 = 0x0006
WROTE 0x44020354 = 0x0000
WROTE 0x44020358 = 0x420000
WROTE 0x4402035c = 0x20010db8
WROTE 0x44020380 = 0x0000
WROTE 0x44020384 = 0x0000
WROTE 0x44020388 = 0x40000000
WROTE 0x4402038c = 0x0001
READ  0x44020344 = 0x0001
WROTE 0x44020340 = 0x0001
READ  0x44020344 = 0x0001
READ  0x44020344 = 0x0001
success
>> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434242 => 64  0 0 0 0
fields = [(u'hit', 1), (u'action_run', 3), (u'out_port', 8), (u'out_port', 8), (u'mac_addr', 48), (u'task', 16), (u'table_id', 16)]
action_name = TopPipe.realmain.set_egress_port
field_vals = [1, '64', '0', '0', '0', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020350 = 0x0042
WROTE 0x44020354 = 0x0000
WROTE 0x44020358 = 0x420000
WROTE 0x4402035c = 0x20010db8
WROTE 0x44020380 = 0x0000
WROTE 0x44020384 = 0x0000
WROTE 0x44020388 = 0x40000000
WROTE 0x4402038c = 0x0001
READ  0x44020344 = 0x0001
WROTE 0x44020340 = 0x0001
READ  0x44020344 = 0x0001
READ  0x44020344 = 0x0001
success
>>
\end{verbatim}

Step 3: setting neighbor entries

\begin{verbatim}
nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::6 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1
nico@ESPRIMO-P956:~$ sudo ip -6 neigh add 2001:db8:42::4 lladdr f8:f2:1e:09:62:d0 dev enp2s0f1
\end{verbatim}

Step 4: generating test packets

\begin{verbatim}
nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::6
PING 2001:db8:42::6(2001:db8:42::6) 56 data bytes


nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes
11:30:17.287577 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 1, length 64
11:30:17.287599 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 1, length 64
11:30:18.310178 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64
11:30:18.310258 IP6 2001:db8:42::42 > 2001:db8:42::6: ICMP6, echo request, seq 2, length 64

\end{verbatim}

Result: success, packet is seen twice.

% ------------------------------------------------------------------------------
\subsection{Test 3: NAT64}

Additionally to the preparations done in test 1 and 2, the following
steps were taken:

Step 1: getting IP address values via Python

\begin{verbatim}
>>> int(ipaddress.IPv6Address(u"2001:db8:42::2a"))
42540766411362381960998550477184434218L

>>> int(ipaddress.IPv6Address(u"2001:db8:42::"))
42540766411362381960998550477184434176L

>>> int(ipaddress.IPv6Address(u"2001:db8:42::a00:2a"))
42540766411362381960998550477352206378

>>> int(ipaddress.IPv4Address(u"10.0.0.0"))
167772160


>>> int(ipaddress.IPv4Address(u"10.0.0.66"))
167772226

\end{verbatim}

Add table entry for 2001:db8:42:2a to be translated to 10.0.0.42:

\begin{verbatim}
>> table_cam_add_entry realmain_nat64_0  realmain.nat64_static 42540766411362381960998550477184434218 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0
fields = [(u'hit', 1), (u'action_run', 3), (u'v6_src', 128), (u'v4_dst', 32), (u'nat64_prefix', 128), (u'table_id', 16)]
action_name = TopPipe.realmain.nat64_static
field_vals = [2, '42540766411362381960998550477184434176', '167772160', '42540766411362381960998550477184434176', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020050 = 0x002a
WROTE 0x44020054 = 0x0000
WROTE 0x44020058 = 0x420000
WROTE 0x4402005c = 0x20010db8
WROTE 0x44020080 = 0x0000
WROTE 0x44020084 = 0x0000
WROTE 0x44020088 = 0x0000
WROTE 0x4402008c = 0xdb80042
WROTE 0x44020090 = 0x2001
WROTE 0x44020094 = 0x0a00
WROTE 0x44020098 = 0x0000
WROTE 0x4402009c = 0x0000
WROTE 0x440200a0 = 0xdb80042
WROTE 0x440200a4 = 0x22001
READ  0x44020044 = 0x0001
WROTE 0x44020040 = 0x0001
READ  0x44020044 = 0x0001
READ  0x44020044 = 0x0001
success
>>

\end{verbatim}

Add table entry for 2001:db8:42::a00:2a to be translated to 10.0.0.66:
\begin{verbatim}
table_cam_add_entry realmain_nat64_0  realmain.nat64_static 42540766411362381960998550477352206378  => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0
\end{verbatim}

Add table entry for 10.0.0.66 to be translated to 2001:db8:42:42:

\begin{verbatim}
>> table_cam_add_entry realmain_nat46_0 realmain.nat46_static 167772226 => 42540766411362381960998550477184434176 167772160 42540766411362381960998550477184434176 0
fields = [(u'hit', 1), (u'action_run', 3), (u'v6_src', 128), (u'v4_dst', 32), (u'nat64_prefix', 128), (u'table_id', 16)]
action_name = TopPipe.realmain.nat46_static
field_vals = [2, '42540766411362381960998550477184434176', '167772160', '42540766411362381960998550477184434176', '0']
CAM_Init_ValidateContext() - done
WROTE 0x44020150 = 0xa000042
WROTE 0x44020180 = 0x0000
WROTE 0x44020184 = 0x0000
WROTE 0x44020188 = 0x0000
WROTE 0x4402018c = 0xdb80042
WROTE 0x44020190 = 0x2001
WROTE 0x44020194 = 0x0a00
WROTE 0x44020198 = 0x0000
WROTE 0x4402019c = 0x0000
WROTE 0x440201a0 = 0xdb80042
WROTE 0x440201a4 = 0x22001
READ  0x44020144 = 0x0001
WROTE 0x44020140 = 0x0001
READ  0x44020144 = 0x0001
READ  0x44020144 = 0x0001
success
>>
\end{verbatim}


Step 3: setting neighbor entries

\begin{verbatim}
sudo ip neigh add 10.0.0.66 lladdr f8:f2:1e:09:62:d1 dev enp2s0f0
sudo ip -6 neigh add 2001:db8:42::2a lladdr f8:f2:1e:09:62:d0 dev enp2s0f1
sudo ip -6 neighbor add 2001:db8:42::a00:2a lladdr f8:f2:1e:09:62:d0 dev enp2s0f1

\end{verbatim}

Step 4: ping test should translate, but fail with wrong checksum:

% ----------------------------------------------------------------------
\chapter{\label{appendixB}NetFPGA Logs}
Majority of the log files are stored inside the source code directory
stored at ``netpfga/logs''. It follows a selection of excerpts
of log files that might be relevant for reproducing the work.
% ----------------------------------------------------------------------

\section{\label{chapterB:netpfga-flasherror}NetFPGA Flash Errors}
Sometimes flashing bitfiles to the NetFPGA will fail. A random amount
of reboots (1 to 3) and a random amount of reflashing will fix this
problem.

Below can be found the log output from the flashing process.

\begin{verbatim}
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh"
++ which vivado
+ xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado
+ bitimage=minip4.bit
+ configWrites=config_writes.sh
+ '[' -z minip4.bit ']'
+ '[' -z config_writes.sh ']'
+ '[' /opt/Xilinx/Vivado/2018.2/bin/vivado == '' ']'
+ rmmod sume_riffa
+ xsct /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl -tclargs minip4.bit
rlwrap: warning: your $TERM is 'screen' but rlwrap couldn't find it in the terminfo database. Expect some problems.
RUN loading image file.
minip4.bit
100%   19MB   1.7MB/s  00:11
fpga configuration failed. DONE PIN is not HIGH
    invoked from within
"::tcf::eval -progress ::xsdb::print_progress {::tcf::cache_enter tcfchan#0 {tcf_cache_eval {process_tcf_actions_cache_client ::tcfclient#0::arg}}}"
    (procedure "::tcf::cache_eval_with_progress" line 2)
    invoked from within
"::tcf::cache_eval_with_progress [dict get $arg chan]  [list process_tcf_actions_cache_client $argvar] $progress"
    (procedure "process_tcf_actions" line 1)
    invoked from within
"process_tcf_actions $arg ::xsdb::print_progress"
    (procedure "fpga" line 430)
    invoked from within
"fpga -f $bitimage"
    (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl" line 33)

+ bash /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/pci_rescan_run.sh
Check programming FPGA or Reboot machine !
+ rmmod sume_riffa
rmmod: ERROR: Module sume_riffa is not currently loaded
+ modprobe sume_riffa
+ ifconfig nf0 up
nf0: ERROR while getting interface flags: No such device
+ ifconfig nf1 up
nf1: ERROR while getting interface flags: No such device
+ ifconfig nf2 up
nf2: ERROR while getting interface flags: No such device
+ ifconfig nf3 up
nf3: ERROR while getting interface flags: No such device
+ bash config_writes.sh

\end{verbatim}
% ----------------------------------------------------------------------

\section{\label{chapterB:netpfga-flashok}NetFPGA Flash Success}
A successful flashing process also emits a couple of errors, however
the message ``fpga configuration failed. DONE PIN is not HIGH'' and
its succeeding lines are missing, as seen below.

After that in all cases a reboot is required; the PCI rescan in no
tested case showed the nf devices.


\begin{verbatim}
nico@nsg-System:~$ cd $NF_DESIGN_DIR/bitfiles/
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh"
++ which vivado
+ xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado
+ bitimage=minip4.bit
+ configWrites=config_writes.sh
+ '[' -z minip4.bit ']'
+ '[' -z config_writes.sh ']'
+ '[' /opt/Xilinx/Vivado/2018.2/bin/vivado == '' ']'
+ rmmod sume_riffa
+ xsct /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl -tclargs minip4.bit
rlwrap: warning: your $TERM is 'xterm-256color' but rlwrap couldn't find it in the terminfo database. Expect some problems.
RUN loading image file.
minip4.bit
attempting to launch hw_server

****** Xilinx hw_server v2018.2
  **** Build date : Jun 14 2018-20:18:37
    ** Copyright 1986-2018 Xilinx, Inc. All Rights Reserved.

INFO: hw_server application started
INFO: Use Ctrl-C to exit hw_server application

INFO: To connect to this hw_server instance use url: TCP:127.0.0.1:3121

100%   19MB   1.7MB/s  00:11
+ bash /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/pci_rescan_run.sh
Check programming FPGA or Reboot machine !
+ rmmod sume_riffa
rmmod: ERROR: Module sume_riffa is not currently loaded
+ modprobe sume_riffa
+ ifconfig nf0 up
nf0: ERROR while getting interface flags: No such device
+ ifconfig nf1 up
nf1: ERROR while getting interface flags: No such device
+ ifconfig nf2 up
nf2: ERROR while getting interface flags: No such device
+ ifconfig nf3 up
nf3: ERROR while getting interface flags: No such device
+ bash config_writes.sh
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$
\end{verbatim}
% ----------------------------------------------------------------------
\section{\label{chapterB:netpfga-kernelmodule}NetFPGA Kernel module}
After a successful flash, loading the kernel module will enable nf
devices to appear in the operating system.

\begin{verbatim}
nico@nsg-System:~$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 74:d0:2b:98:38:f6 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9c brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9d brd ff:ff:ff:ff:ff:ff
5: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none
nico@nsg-System:~$ ~/master-thesis/bin/build-load-drivers.sh
+ cd /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0
+ sudo modprobe -r sume_riffa
+ make clean
make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 clean
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic'
  CLEAN   /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/.tmp_versions
  CLEAN   /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/Module.symvers
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic'
+ make all
make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic'
  CC [M]  /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.mod.o
  LD [M]  /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0/sume_riffa.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic'
+ sudo make install
make -C /lib/modules/4.15.0-55-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-55-generic'
  Building modules, stage 2.
  MODPOST 1 modules
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-55-generic'
install -o root -g root -m 0755 -d /lib/modules/4.15.0-55-generic/extra/sume_riffa/
install -o root -g root -m 0755 sume_riffa.ko /lib/modules/4.15.0-55-generic/extra/sume_riffa/
depmod -a 4.15.0-55-generic
+ sudo modprobe sume_riffa
+ grep sume_riffa
+ lsmod
sume_riffa             28672  0
nico@nsg-System:~$
nico@nsg-System:~$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 74:d0:2b:98:38:f6 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9c brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:41:44:9d brd ff:ff:ff:ff:ff:ff
5: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/none
6: nf0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:00 brd ff:ff:ff:ff:ff:ff
7: nf1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:01 brd ff:ff:ff:ff:ff:ff
8: nf2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:02 brd ff:ff:ff:ff:ff:ff
9: nf3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:53:55:4d:45:03 brd ff:ff:ff:ff:ff:ff
nico@nsg-System:~$

\end{verbatim}
% ----------------------------------------------------------------------
\section{\label{chapterB:netpfga-nftraffic}NetFPGA misses packets on nf*}
While the nf devices appear in the operating system, packets emitted
by the netpfga cannot be sniffed on the nf interfaces
directly. Instead one has to sniff packets on a physical network card
that is connected to the specific output port.

% ----------------------------------------------------------------------
\section{\label{chapterB:netpfga-kernelmodule}NetFPGA Kernel module}

%---------------------------------------------------------------------------------------------------------
\chapter{\label{benchmark}Benchmark Logs}
% ----------------------------------------------------------------------

\section{\label{benchmark:iperf}iperf}
Omitting startup time

\section{\label{benchmark:general}General}

MTU setting to 1500, as netpfga doesn't support jumbo frames


iperf3, iperf 3.0.11

50 parallel = 2x 100% cpu usage
40 parallel = 100%, 70% cpu usage
30 parallel = 70%-100, 70% cpu usage

Turning back on checksum offloading (see below)

30 parallel = 70%, 30% cpu usage


\begin{verbatim}
root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: on
        tx-checksum-ip-generic: on
        tx-checksum-sctp: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp6-segmentation: on
root@ESPRIMO-P956:~#
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: on
        tx-checksum-ip-generic: on
        tx-checksum-sctp: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp6-segmentation: on
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~#

\end{verbatim}

Results into

\begin{verbatim}

root@ESPRIMO-P956:~# ethtool -k enp2s0f0
Features for enp2s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off
root@ESPRIMO-P956:~# ethtool -k enp2s0f1
Features for enp2s0f1:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off
root@ESPRIMO-P956:~#

\end{verbatim}

\section{\label{benchmark:netpfga}NetFPGA}
iperf3-tcp-listening-v4 connected by v6
\begin{verbatim}
nico@ESPRIMO-P956:~$ iperf3 -p 2345 -4 -B 10.0.0.42 -s
-----------------------------------------------------------
Server listening on 2345
-----------------------------------------------------------
Accepted connection from 10.0.0.66, port 50900
[  5] local 10.0.0.42 port 2345 connected to 10.0.0.66 port 50902
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   693 MBytes  5.81 Gbits/sec
[  5]   1.00-2.00   sec   645 MBytes  5.41 Gbits/sec
[  5]   2.00-3.00   sec   644 MBytes  5.40 Gbits/sec
[  5]   3.00-4.00   sec   868 MBytes  7.28 Gbits/sec
[  5]   4.00-5.00   sec   853 MBytes  7.16 Gbits/sec
[  5]   5.00-6.00   sec   913 MBytes  7.66 Gbits/sec
[  5]   6.00-7.00   sec   774 MBytes  6.49 Gbits/sec
[  5]   7.00-8.00   sec   641 MBytes  5.38 Gbits/sec
[  5]   8.00-9.00   sec   911 MBytes  7.64 Gbits/sec
[  5]   9.00-10.00  sec   733 MBytes  6.15 Gbits/sec
[  5]  10.00-10.04  sec  25.8 MBytes  5.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec  7.52 GBytes  6.43 Gbits/sec   14             sender
[  5]   0.00-10.04  sec  7.52 GBytes  6.43 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 2345
-----------------------------------------------------------

nico@ESPRIMO-P956:~$ iperf3 -6 -p 2345 -c 2001:db8:42::a00:2a
Connecting to host 2001:db8:42::a00:2a, port 2345
[  4] local 2001:db8:42::42 port 50902 connected to 2001:db8:42::a00:2a port 2345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   719 MBytes  6.03 Gbits/sec   10    449 KBytes
[  4]   1.00-2.00   sec   645 MBytes  5.41 Gbits/sec    0    449 KBytes
[  4]   2.00-3.00   sec   644 MBytes  5.40 Gbits/sec    0    449 KBytes
[  4]   3.00-4.00   sec   878 MBytes  7.36 Gbits/sec    0    449 KBytes
[  4]   4.00-5.00   sec   859 MBytes  7.20 Gbits/sec    0    449 KBytes
[  4]   5.00-6.00   sec   910 MBytes  7.64 Gbits/sec    0    449 KBytes
[  4]   6.00-7.00   sec   758 MBytes  6.36 Gbits/sec    0    449 KBytes
[  4]   7.00-8.00   sec   658 MBytes  5.52 Gbits/sec    0    449 KBytes
[  4]   8.00-9.00   sec   906 MBytes  7.60 Gbits/sec    4    449 KBytes
[  4]   9.00-10.00  sec   724 MBytes  6.07 Gbits/sec    0    449 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  7.52 GBytes  6.46 Gbits/sec   14             sender
[  4]   0.00-10.00  sec  7.52 GBytes  6.46 Gbits/sec                  receiver

iperf Done.
nico@ESPRIMO-P956:~$

\end{verbatim}

listening on v6, connecting from v4:

\begin{verbatim}
nico@ESPRIMO-P956:~$ iperf3 -p 2345 -6 -B 2001:db8:42::42 -s
-----------------------------------------------------------
Server listening on 2345
-----------------------------------------------------------
Accepted connection from 2001:db8:42::a00:2a, port 47520
[  5] local 2001:db8:42::42 port 2345 connected to 2001:db8:42::a00:2a port 47522
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  1.02 GBytes  8.73 Gbits/sec
[  5]   1.00-2.00   sec   879 MBytes  7.38 Gbits/sec
[  5]   2.00-3.00   sec   859 MBytes  7.20 Gbits/sec
[  5]   3.00-4.00   sec  1.02 GBytes  8.78 Gbits/sec
[  5]   4.00-5.00   sec  1.04 GBytes  8.89 Gbits/sec
[  5]   5.00-6.00   sec  1.05 GBytes  9.00 Gbits/sec
[  5]   6.00-7.00   sec  1.03 GBytes  8.89 Gbits/sec
[  5]   7.00-8.00   sec  1.04 GBytes  8.91 Gbits/sec
[  5]   8.00-9.00   sec  1.03 GBytes  8.84 Gbits/sec
[  5]   9.00-10.00  sec   953 MBytes  7.99 Gbits/sec
[  5]  10.00-10.04  sec  38.6 MBytes  7.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec  9.89 GBytes  8.46 Gbits/sec  151             sender
[  5]   0.00-10.04  sec  9.89 GBytes  8.46 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 2345
-----------------------------------------------------------

nico@ESPRIMO-P956:~$ iperf3 -4 -p 2345 -c 10.0.0.66
Connecting to host 10.0.0.66, port 2345
[  4] local 10.0.0.42 port 47522 connected to 10.0.0.66 port 2345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.06 GBytes  9.10 Gbits/sec   53    208 KBytes
[  4]   1.00-2.00   sec   867 MBytes  7.27 Gbits/sec    6    379 KBytes
[  4]   2.00-3.00   sec   870 MBytes  7.29 Gbits/sec    0    423 KBytes
[  4]   3.00-4.00   sec  1.02 GBytes  8.77 Gbits/sec   37    364 KBytes
[  4]   4.00-5.00   sec  1.04 GBytes  8.91 Gbits/sec    1    450 KBytes
[  4]   5.00-6.00   sec  1.05 GBytes  8.98 Gbits/sec    0    462 KBytes
[  4]   6.00-7.00   sec  1.04 GBytes  8.92 Gbits/sec   30    324 KBytes
[  4]   7.00-8.00   sec  1.04 GBytes  8.88 Gbits/sec    0    471 KBytes
[  4]   8.00-9.00   sec  1.03 GBytes  8.86 Gbits/sec   10    452 KBytes
[  4]   9.00-10.00  sec   947 MBytes  7.94 Gbits/sec   14    409 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.89 GBytes  8.49 Gbits/sec  151             sender
[  4]   0.00-10.00  sec  9.89 GBytes  8.49 Gbits/sec                  receiver

iperf Done.
nico@ESPRIMO-P956:~$

\end{verbatim}
% \section{\label{benchmark:bmv2}BMV2}
%---------------------------------------------------------------------------------------------------------
\section{\label{benchmark:tayga}Tayga}


\begin{verbatim}
ii  tayga                                       0.9.2-6                                      amd64        userspace stateless NAT64
\end{verbatim}

Setting up IPv4 networking

\begin{verbatim}
[15:12] nsg-System:~# ip addr add 10.0.0.77/24 dev eth1
[15:12] nsg-System:~# ip l s eth1 up

nico@ESPRIMO-P956:~$ ~/master-thesis/bin/init_ipv4_esprimo.sh
nico@ESPRIMO-P956:~$ cat ~/master-thesis/bin/init_ipv4_esprimo.sh
#!/bin/sh

sudo ip addr add 10.0.0.42/24 dev enp2s0f0
sudo ip link set enp2s0f0 up

nico@ESPRIMO-P956:~$ sudo ip route add 10.0.1.0/24 via 10.0.0.77

\end{verbatim}

Verify networking works:

\begin{verbatim}
[15:12] nsg-System:~# ping 10.0.0.42
PING 10.0.0.42 (10.0.0.42) 56(84) bytes of data.
64 bytes from 10.0.0.42: icmp_seq=1 ttl=64 time=0.304 ms
64 bytes from 10.0.0.42: icmp_seq=2 ttl=64 time=0.097 ms
^C
--- 10.0.0.42 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1011ms
rtt min/avg/max/mdev = 0.097/0.200/0.304/0.104 ms
[15:12] nsg-System:~#

\end{verbatim}

Setting up IPv6 networking

\begin{verbatim}
nico@ESPRIMO-P956:~$ ip addr show dev enp2s0f1
13: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f8:f2:1e:09:62:d1 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8:42::42/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::faf2:1eff:fe09:62d1/64 scope link
       valid_lft forever preferred_lft forever
nico@ESPRIMO-P956:~$ sudo ip route add 2001:db8:23::/96 via 2001:db8:42::77

[15:12] nsg-System:~# ip addr add 2001:db8:42::77/64 dev eth2
[15:15] nsg-System:~# ip link set eth2 up

\end{verbatim}

Verify IPv6 networking works:

\begin{verbatim}
nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:42::77
PING 2001:db8:42::77(2001:db8:42::77) 56 data bytes
64 bytes from 2001:db8:42::77: icmp_seq=1 ttl=64 time=0.169 ms
64 bytes from 2001:db8:42::77: icmp_seq=2 ttl=64 time=0.153 ms

--- 2001:db8:42::77 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1010ms
rtt min/avg/max/mdev = 0.153/0.161/0.169/0.008 ms
nico@ESPRIMO-P956:~$

\end{verbatim}

Enabling IPv6 and IPv4 forwarding:

\begin{verbatim}
[15:16] nsg-System:~# sysctl -w net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.forwarding = 1

[15:20] nsg-System:~# sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

\end{verbatim}


Testing NAT64 in tayga

\begin{verbatim}
nico@ESPRIMO-P956:~$ ping -c2 10.0.1.42
PING 10.0.1.42 (10.0.1.42) 56(84) bytes of data.
64 bytes from 10.0.1.42: icmp_seq=1 ttl=61 time=0.356 ms
64 bytes from 10.0.1.42: icmp_seq=2 ttl=61 time=0.410 ms

--- 10.0.1.42 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1019ms
rtt min/avg/max/mdev = 0.356/0.383/0.410/0.027 ms
nico@ESPRIMO-P956:~$

nico@ESPRIMO-P956:~$ sudo tcpdump -ni enp2s0f1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:21:39.851057 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 1, length 64
15:21:39.851124 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 1, length 64
15:21:40.870448 IP6 2001:db8:23::a00:2a > 2001:db8:42::42: ICMP6, echo request, seq 2, length 64
15:21:40.870507 IP6 2001:db8:42::42 > 2001:db8:23::a00:2a: ICMP6, echo reply, seq 2, length 64
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
nico@ESPRIMO-P956:~$

\end{verbatim}

Testing NAT64 (v6 to v4)

\begin{verbatim}
nico@ESPRIMO-P956:~$ ping6 -c2 2001:db8:23::a00:2a
PING 2001:db8:23::a00:2a(2001:db8:23::a00:2a) 56 data bytes
64 bytes from 2001:db8:23::a00:2a: icmp_seq=1 ttl=61 time=0.240 ms
64 bytes from 2001:db8:23::a00:2a: icmp_seq=2 ttl=61 time=0.400 ms

--- 2001:db8:23::a00:2a ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.240/0.320/0.400/0.080 ms
nico@ESPRIMO-P956:~$

\end{verbatim}

\subsection{\label{benchmark:tayga:tcp}Tayga/TCP}

Tayga running at 100% cpu load,

v4->v6 tcp
delivering
3.36 gbit/s at P1
3.30 Gbit/s at P20
3.11 gbit/s at P50

v6->v4 tcp
P1:  3.02 Gbit/s
P20: 3.28 gbit/s
P50: 2.85 gbit/s

Commands:

\begin{verbatim}

Server always: iperf3 -6 -p 2345 -B 2001:db8:42::42 -s  | tee iperf-tayga-v4tov6server-P50


nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -c 10.0.1.42 -T taygav4tov6tcpP1 | tee iperf-tayga-v4tov6server-client

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -c 10.0.1.42 -T taygav4tov6tcpP20 | tee iperf-tayga-v4tov6server-client-P20

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -c 10.0.1.42 -T taygav4tov6tcpP50 | tee iperf-tayga-v4tov6server-client-P50


nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-tayga-v6tov4-server-P1

\end{verbatim}


Testing v6->v4

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-tayga-v6tov4-server-P20

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -c 2001:db8:23::a00:2a  -T taygav6tov4tcpP1 | tee iperf-tayga-v6tov4-client-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P20 -c 2001:db8:23::a00:2a  -T taygav6tov4tcpP20 | tee iperf-tayga-v6tov4-client-P20

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -c 2001:db8:23::a00:2a  -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-P50

\end{verbatim}

UDP v6->v4, again 100% cpu

P1:   5.81 gbit/s
P20:  9.40 gbit/s
P50:  19.6 gbits/sec

On the line only ca. 3600 mbit/s seen

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-tayga-v6tov4-server-udp-P1


nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -u -b10000m -c 2001:db8:23::a00:2a  -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-udp-P1


nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50 -u -b10000m -c 2001:db8:23::a00:2a  -T taygav6tov4tcpP50 | tee iperf-tayga-v6tov4-client-udp-P50


\end{verbatim}

Messsages from server:

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-tayga-v6tov4-server-udp-P1
iperf3: OUT OF ORDER - incoming packet = 198902 and received packet = 198904 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 441615 and received packet = 441617 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 441616 and received packet = 441618 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 567495 and received packet = 567501 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 567496 and received packet = 567501 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 567497 and received packet = 567501 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 567499 and received packet = 567503 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 567500 and received packet = 567503 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 567502 and received packet = 567503 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 631160 and received packet = 631164 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 631161 and received packet = 631164 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 631162 and received packet = 631165 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 631163 and received packet = 631165 AND SP = 5

\end{verbatim}


UDP v4->v6, again 100% cpu

P1:  8.26 gbit/s [atop: ~2500 Mbit/s per direction]
P20: 9.92 Gbits/sec [atop: ~2500 Mbit/s per direction]
P50: 19.3 gbit/s  [atop: ~2500 Mbit/s per direction]

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s  | tee iperf-tayga-v4tov6-server-udp-P1


nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP1 | tee iperf-tayga-v4tov6server-client-udp-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP20 | tee iperf-tayga-v4tov6server-client-udp-P20

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50 -u -b0 -c 10.0.1.42 -T taygav4tov6udpP50 | tee iperf-tayga-v4tov6server-client-udp-P50

\end{verbatim}

%---------------------------------------------------------------------------------------------------------

\section{\label{benchmark:jool}Jool}
\subsection{\label{benchmark:jool:setup}Jool Setup}

Installation of 4.0.1 from \url{https://www.jool.mx/en/download.html}.

\begin{verbatim}
nico@nsg-System:~$ wget https://github.com/NICMx/Jool/releases/download/v4.0.1/jool_4.0.1.tar.gz
nico@nsg-System:~$ tar xvfz jool_4.0.1.tar.gz
nico@nsg-System:~$ cd jool-4.0.1/
nico@nsg-System:~/jool-4.0.1$ sudo  apt install linux-headers-$(uname -r)
nico@nsg-System:~/jool-4.0.1$ sudo  apt install libnl-genl-3-dev

\end{verbatim}

xtables cannot be found:

\begin{verbatim}
nico@nsg-System:~/jool-4.0.1$ sudo apt install libxtables-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libxtables-dev
nico@nsg-System:~/jool-4.0.1$

\end{verbatim}

Does not compile without:

\begin{verbatim}
checking for library containing argp_parse... none required
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LIBNLGENL3... yes
checking for XTABLES... no
configure: error: Package requirements (xtables) were not met:

No package 'xtables' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables XTABLES_CFLAGS
and XTABLES_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
nico@nsg-System:~/jool-4.0.1$

\end{verbatim}

Trying different package:

\begin{verbatim}
nico@nsg-System:~/jool-4.0.1$ sudo apt install iptables-dev

\end{verbatim}

Compiles!

\begin{verbatim}
nico@nsg-System:~/jool-4.0.1$ sudo make install

\end{verbatim}

\subsection{\label{benchmark:jool:config}Jool Configuration}

Loading module:
\begin{verbatim}
nico@nsg-System:~/jool-4.0.1$ sudo modprobe jool_siit


\end{verbatim}

enabling forwarding:
\begin{verbatim}
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
\end{verbatim}

Mapping configuration:

\begin{verbatim}
nico@nsg-System:~/jool-4.0.1$ sudo jool_siit instance add example --iptables --pool6 2001:db8:23::/96

nico@nsg-System:~/jool-4.0.1$ sudo ip6tables -t mangle -A PREROUTING \
                              -s 2001:db8:42::/64 -d 2001:db8:23::/96 -j JOOL_SIIT --instance example

nico@nsg-System:~/jool-4.0.1$ sudo iptables -t mangle -A PREROUTING \
                              -s 10.0.0.0/24 -j JOOL_SIIT --instance example

\end{verbatim}



Debugging:

\begin{verbatim}
[16:39] nsg-System:~# lsmod| grep jool
jool_siit             147456  2
x_tables               40960  5 jool_siit,ip6_tables,ip_tables,ip6table_mangle,iptable_mangle
[16:39] nsg-System:~#

[16:41] nsg-System:~#  jool_siit -i  example stats display --explain
JSTAT64_DST: 276
Translations cancelled: IPv6 packet's destination address did not match pool6 nor any EAMT entries, or the resulting address was blacklist4ed.


\end{verbatim}

Try 2 w/ eamt:

\begin{verbatim}
[16:53] nsg-System:~# modprobe jool_siit
[16:54] nsg-System:~# jool_siit instance add "example" --iptables
[16:54] nsg-System:~# jool_siit -i example eamt add 2001:db8:42::/120 10.0.1.0/24
[16:55] nsg-System:~# jool_siit -i example eamt add 2001:db8:23::/120 10.0.0.0/24
[16:57] nsg-System:~# ip6tables -t mangle -A PREROUTING -s 2001:db8:42::/120 -d 2001:db8:23::/120 -j JOOL_SIIT --instance example
[16:57] nsg-System:~# iptables -t mangle -A PREROUTING -s 10.0.0.0/24 -d 10.0.1.0/24 -j JOOL_SIIT --instance example
[16:57] nsg-System:~#

\end{verbatim}

Testing NAT64:

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ ping6 2001:db8:23::2a
PING 2001:db8:23::2a(2001:db8:23::2a) 56 data bytes
64 bytes from 2001:db8:23::2a: icmp_seq=1 ttl=63 time=0.199 ms
64 bytes from 2001:db8:23::2a: icmp_seq=2 ttl=63 time=0.282 ms
64 bytes from 2001:db8:23::2a: icmp_seq=3 ttl=63 time=0.186 ms
^C
--- 2001:db8:23::2a ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2040ms
rtt min/avg/max/mdev = 0.186/0.222/0.282/0.044 ms
nico@ESPRIMO-P956:~/master-thesis/iperf$ ping 10.0.1.66
PING 10.0.1.66 (10.0.1.66) 56(84) bytes of data.
64 bytes from 10.0.1.66: icmp_seq=1 ttl=63 time=0.218 ms
64 bytes from 10.0.1.66: icmp_seq=2 ttl=63 time=0.281 ms
64 bytes from 10.0.1.66: icmp_seq=3 ttl=63 time=0.280 ms
^C
--- 10.0.1.66 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2051ms
rtt min/avg/max/mdev = 0.218/0.259/0.281/0.034 ms
nico@ESPRIMO-P956:~/master-thesis/iperf$

\end{verbatim}


\subsection{\label{benchmark:jool:benchmark}Jool Benchmarks}

v4->v6 tcp

P1:  8.24 gbit/s no cpu load visible
P20: 8.26 gbit/s iperf 42 + 10% cpu no cpu load visible
P50: 8.29 gbit/s

v6->v4 tcp

P1:  8.22
P20: 8.22 15/60% iperf
P50: 8.23 iperf: 73/16%

Commands:

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s  | tee iperf-jool-v4tov6-server-tcp-P50

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1  -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20  -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P20
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P50  -c 10.0.1.66 | tee iperf-jool-v4tov6-client-tcp-P50

Other way:

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-jool-v6tov4-server-tcp-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1  -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P1

...

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -b0 -u  -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P1


\end{verbatim}


v4->v6 udp

P1:   4.46  iperf 30% cpu nat64: 100% cpu
P20: 18.8     iperf 100% nat64: 100%
P50: 22.8  iperf 100% nat64: 100% atop: ca. 5gbit/s

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -B 2001:db8:42::42 -s  | tee iperf-jool-v4tov6-server-udp-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1  -c 10.0.1.66 -u -b0 | tee iperf-jool-v4tov6-client-udp-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20  -c 10.0.1.66 -u -b0 | tee iperf-jool-v4tov6-client-udp-P20


\end{verbatim}

v6->v4 udp

P1: 6.67 gbit/s iperf 50/50%, nat64: 100% cpu!
P20: 16.8 nat64: iperf: ? 100% cpu
P50: 20.5 Gbits/sec nat64: 100% (1 core) iperf: 100%/50% atop: 7000mbit/s in / out

\begin{verbatim}

\end{verbatim}

Turning off offloading, redoing tcp:

\begin{verbatim}
root@ESPRIMO-P956:~# ethtool -K enp2s0f0 gso off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~# ethtool -K enp2s0f0 rx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: off
        tx-checksum-ip-generic: off
        tx-checksum-sctp: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: off
        tx-checksum-ip-generic: off
        tx-checksum-sctp: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 gso off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~#

[17:26] nsg-System:~# ethtool -K eth1 tx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: off
        tx-checksum-ip-generic: off
        tx-checksum-sctp: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
[17:26] nsg-System:~# ethtool -K eth1 gso off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
[17:26] nsg-System:~# ethtool -K eth2 gso off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
[17:26] nsg-System:~# ethtool -K eth2 rx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
[17:26] nsg-System:~# ethtool -K eth2 tx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: off
        tx-checksum-ip-generic: off
        tx-checksum-sctp: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
[17:26] nsg-System:~#

\end{verbatim}

Retesting using -P50:

Still no cpu load with tcp, 100% cpu load iperf

result: 7.96 gbit/s

\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-jool-v6tov4-server-tcp-P50-no-offload
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P50  -c 2001:db8:23::2a | tee iperf-jool-v6tov4-client-tcp-P50-no-offload

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P20 -u -b0  -c 10.0.0.66 | tee iperf-netpfga-v4tov6-client-udp-P20



nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-netfpga-v6tov4-server-tcp-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1  -c 2001:db8:42::a00:2a | tee iperf-netfpga-v6tov4-client-tcp-P1


nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -B 10.0.0.42 -s  | tee iperf-netfpga-v6tov4-server-udp-P1

nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -6 -p 2345 -t 70 -O 10 -P1 -b0 -u  -c 2001:db8:42::a00:2a | tee iperf-netfpga-v6tov4-client-udp-P1

\end{verbatim}

\subsection{\label{benchmark:netfpga:benchmark}NetPFGA Benchmarks}

Only 1 test did have offloading on esprimo off, was redone

v4->v6 tcp

P1:                      7.41 gbit/s iperf 50% / 20%
P1-offload-on-esprimo:   8.43 gbit/s
P20: 9.29 gbit/s iperf: 66/20%
P50: 9.29 gbit/s 84/42% iperf

v4->v6 udp

P1: 7.4gbit/s 100% iperf
P20: 17.7gbit/s    iperf 100% atop: ca. 9800 Mbit/s
P50: 21.5 gbit/s   iperf 100% ca. 9800 Mbit/s


v6->v4 tcp

P1:  9.28 gbit/s  atop ~9800 mbit/s iperf 44%
P20: 9.29 gbit/s atop ~9800 mbit/s iperf 70%
P50: 9.29 gbit/s atop ~9800 mbit/s iperf 90%

v6->v4 udp

P1:  7.96 gbit/s atop ~8200mbit/s iperf 70%
P20: 13.4 gbit/s  atop 9800 mbit/s iperf 100%
P50: 19.0 gbit/s  atop 9800 mbit/s iperf 100%



Commands:
\begin{verbatim}
nico@ESPRIMO-P956:~/master-thesis/iperf$ iperf3 -4 -p 2345 -t 70 -O 10 -P1 -u -b0  -c 10.0.0.66 | tee iperf-netpfga-v4tov6-client-udp-P1

\end{verbatim}

After first netpfga, tcp v4->v6 p1 turned offloading on again

\begin{verbatim}
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx-checksum-ipv6 on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Could not change any device features
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 tx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: on
        tx-checksum-ip-generic: on
        tx-checksum-sctp: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp6-segmentation: on
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 rx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~# ethtool -K enp2s0f1 gso on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~# ethtool -K enp2s0f0 gso on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~# ethtool -K enp2s0f0 tx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
tx-checksumming: on
        tx-checksum-ip-generic: on
        tx-checksum-sctp: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp6-segmentation: on
root@ESPRIMO-P956:~# ethtool -K enp2s0f0 rx on
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
root@ESPRIMO-P956:~#

\end{verbatim}

%Entries for the list of abbrevations:
%
%To generate the list of abbrevations, execute:
%makeindex Thesis.nlo -s nomencl.ist -o Thesis.nls
%
%If you are using TeXniCenter, specify:
%"%bm.nlo" -s nomencl.ist -o "%bm.nls"
%as beeing the argument list for makeindex.

%---------------------------------------------------------------------------------------------------------

\chapter{\label{bufferssssss}Buffer}

\section{NetFPGA compile errors}

- infinite loop in installer

\begin{verbatim}
# Fix introduced for SDNet 2017.4
sed -i 's/xsim\.dir\/xsc\/dpi\.so/dpi\.so/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim.bash
sed -i 's/xsim\.dir\/xsc\/dpi\.so/dpi\.so/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
# Fix introduced for SDNet 2018.2
sed -i 's/glbl_sim/glbl/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
sed -i 's/SimpleSumeSwitch_tb_sim#work.glbl/SimpleSumeSwitch_tb/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
cp src/*.tbl nf_sume_sdnet_ip/SimpleSumeSwitch/
cp: cannot stat 'src/*.tbl': No such file or directory
make: *** [Makefile:23: cpp_test] Error 1
[23:12] loch:minip4%

\end{verbatim}

\begin{verbatim}
 ERROR: [XSIM 43-3409] Failed to compile generated C file xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.c.
 ERROR: [XSIM 43-3915] Encountered a fatal error. Cannot continue. Exiting...
\end{verbatim}

\begin{verbatim}
/opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/clang  -fPIC -c -std=gnu89 -nobuiltininc -nostdinc++ -w  -Wl,--unres
olved-symbols=ignore-in-object-files -fbracket-depth=1048576 -I/opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/../li
b/clang/3.1/include -fPIC -m64  -I"/opt/Xilinx/Vivado/2018.2/data/xsim/include" "xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/ob
j/xsim_3.c" -O0 -sim -o "xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.lnx64.o" -DXILINX_SIMULATOR
/opt/Xilinx/Vivado/2018.2/data/../tps/llvm/3.1/lnx64.o/bin/clang: error while loading shared libraries: libncurses.so.5: cannot
 open shared object file: No such file or directory
ERROR: [XSIM 43-3409] Failed to compile generated C file xsim.dir/work.SimpleSumeSwitch_tb#work.glbl/obj/xsim_3.c.
ERROR: [XSIM 43-3915] Encountered a fatal error. Cannot continue. Exiting...
[20:00] rainbow:SimpleSumeSwitch%

\end{verbatim}

\begin{verbatim}
# Fix introduced for SDNet 2018.2
sed -i 's/glbl_sim/glbl/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
sed -i 's/SimpleSumeSwitch_tb_sim#work.glbl/SimpleSumeSwitch_tb/g' nf_sume_sdnet_ip/SimpleSumeSwitch/vivado_sim_waveform.bash
cp src/*.tbl nf_sume_sdnet_ip/SimpleSumeSwitch/
cp testdata/*.txt nf_sume_sdnet_ip/SimpleSumeSwitch/
cp: cannot stat 'testdata/*.txt': No such file or directory
make: *** [Makefile:17: all] Error 1
[15:46] rainbow:minip4%

\end{verbatim}

\begin{verbatim}
make -C testdata/
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata'
./gen_testdata.py
/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/bin/pcap2axi --output Packet_in.axi --bus_width 256 src.pcap
Traceback (most recent call last):
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/bin/pcap2axi", line 108, in <module>
    write_to_file(args.file_pcap, args.output)
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/bin/pcap2axi", line 88, in write_to_file
    for pkt in rdpcap(file_in):
  File "/usr/lib/python2.7/dist-packages/scapy/utils.py", line 728, in rdpcap
    with PcapReader(filename) as fdesc:
  File "/usr/lib/python2.7/dist-packages/scapy/utils.py", line 751, in __call__
    filename, fdesc, magic = cls.open(filename)
  File "/usr/lib/python2.7/dist-packages/scapy/utils.py", line 778, in open
    fdesc = open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'src.pcap'
make[1]: *** [Makefile:5: all] Error 1
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata'
make: *** [Makefile:32: frontend] Error 2
[15:47] rainbow:minip4%

\end{verbatim}

\begin{verbatim}
update_compile_order: Time (s): cpu = 00:00:17 ; elapsed = 00:00:09 . Memory (MB): peak = 1995.594 ; gain = 0.016 ; free physic
al = 21975 ; free virtual = 33161
loading libsume..
Traceback (most recent call last):
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_de
fault/run.py", line 42, in <module>
    import config_writes
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_de
fault/config_writes.py", line 7

                        ^
IndentationError: expected an indented block
    while executing
"exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py"
    invoked from within
"set output [exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py]"
    (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/tcl/simple_s
ume_switch_sim.tcl" line 177)
INFO: [Common 17-206] Exiting Vivado at Sat May 18 15:21:21 2019...


\end{verbatim}

\begin{verbatim}
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.log': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_expect.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.axi': No such file or directory
=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ... using cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim']
\end{verbatim}

\begin{verbatim}
# add_wave $nf_sume_sdnet_ip/out_src_port
# add_wave $nf_sume_sdnet_ip/out_dst_port
# set const_reg_ip /top_tb/top_sim/nf_datapath_0/nf_sume_sdnet_wrapper_1/inst/SimpleSumeSwitch_inst/const_reg_rw_0/
# add_wave_divider {const reg extern signals}
# add_wave $const_reg_ip
ERROR: [Wavedata 42-471] Note: Nothing was found for the following items: /top_tb/top_sim/nf_datapath_0/nf_sume_sdnet_wrapper_1/inst/SimpleSumeSwitch_inst/const_reg_rw_0/
ERROR: [Common 17-39] 'add_wave' failed due to earlier errors.

    while executing
"add_wave $const_reg_ip "
    (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/tcl/simple_sume_switch_sim.tcl" line 328)
INFO: [Common 17-206] Exiting Vivado at Sat May 18 15:31:59 2019...
make: *** [Makefile:121: sim] Error 1
make: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test'
512
=== Work directory is /tmp/nico/test/simple_sume_switch
=== Setting up test in /tmp/nico/test/simple_sume_switch/sim_switch_default
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_log.axi': No such file or directory
=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ... using cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim']
[15:31] rainbow:P4-NetFPGA%
\end{verbatim}

\begin{verbatim}
ERROR: [BD 41-171] The modes of the interface pins 'cfg_interrupt'(Slave) and 'pcie3_cfg_interrupt'(Slave) are incompatible. They cannot be connected.
ERROR: [BD 5-3] Error: running connect_bd_intf_net.
ERROR: [Common 17-39] 'connect_bd_intf_net' failed due to earlier errors.

    while executing
"connect_bd_intf_net -intf_net nf_riffa_dma_1_pcie3_cfg_interrupt [get_bd_intf_pins nf_riffa_dma_1/cfg_interrupt] [get_bd_intf_pins pcie3_7x_1/pcie3_cf..."
    (procedure "create_hier_cell_dma_sub" line 141)
    invoked from within
"create_hier_cell_dma_sub [current_bd_instance .] dma_sub"
    (procedure "create_root_design" line 68)
    invoked from within
"create_root_design """
    (file "./tcl/control_sub.tcl" line 729)

    while executing
"source ./tcl/control_sub.tcl"
    (file "tcl/simple_sume_switch.tcl" line 89)

\end{verbatim}

\begin{verbatim}

### set NF_10G_INTERFACE3_BASEADDR $M07_BASEADDR
### set NF_10G_INTERFACE3_HIGHADDR $M07_HIGHADDR
### set NF_10G_INTERFACE3_SIZEADDR $M07_SIZEADDR
### set NF_RIFFA_DMA_BASEADDR $M08_BASEADDR
### set NF_RIFFA_DMA_HIGHADDR $M08_HIGHADDR
### set NF_RIFFA_DMA_SIZEADDR $M08_SIZEADDR
Wrote  : </home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/bd/control_sub/control_sub.bd>
# create_ip -name nf_sume_sdnet -vendor NetFPGA -library NetFPGA -module_name nf_sume_sdnet_ip
ERROR: [Coretcl 2-1134] No IP matching VLNV 'NetFPGA:NetFPGA:nf_sume_sdnet:*' was found. Please check your repository configuration.
INFO: [Common 17-206] Exiting Vivado at Sat May 25 11:52:01 2019...

\end{verbatim}
\begin{verbatim}
# set_property compxlib.xsim_compiled_library_dir {} [current_project]                                                 [0/1819]
# set_property top_lib xil_defaultlib [get_filesets sim_1]
# update_compile_order -fileset sim_1
update_compile_order: Time (s): cpu = 00:00:17 ; elapsed = 00:00:09 . Memory (MB): peak = 2003.578 ; gain = 8.004 ; free physical = 27661 ; free virtual = 33990
loading libsume..
Traceback (most recent call last):
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py", line 42, in <module>
    import config_writes
  File "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/config_writes.py", line 7

                        ^
IndentationError: expected an indented block
    while executing
"exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py"
    invoked from within
"set output [exec python $::env(NF_DESIGN_DIR)/test/${test_name}/run.py]"
    (file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/tcl/simple_sume_switch_sim.tcl" line 177)
INFO: [Common 17-206] Exiting Vivado at Sat May 25 13:45:13 2019...
make: *** [Makefile:121: sim] Error 1
\end{verbatim}

\begin{verbatim}
[SW] CAM_EnableDevice() - done
[2420698]  INFO: finished packet stimulus file
[2735572] ERROR: tuple mismatch for packet 1
expected < tuple_out_digest_data, tuple_out_sume_metadata > = < 0000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000001010000 >
actual   < tuple_out_digest_data, tuple_out_sume_metadata > = < 0000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000010000 >
$finish called at time : 2735572 ps : File
"/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/Testbench/Check.v"
Line 120
\end{verbatim}

\begin{verbatim}
[14:54] rainbow:bitfiles% cd $NF_DESIGN_DIR/bitfiles/ && sudo bash ./program_switch.sh
./program_switch.sh: line 34: /tools/program_switch.sh: No such file or directory
[14:56] rainbow:bitfiles% ls
config_writes.sh  minip4.bit  program_switch.sh  README
[14:56] rainbow:bitfiles%
\end{verbatim}

\begin{verbatim}
root@rainbow:~/master-thesis/netpfga/minip4/sw/hw_test_tool# python switch_calc_tester.py
SIOCSIFADDR: No such device
eth1: ERROR while getting interface flags: No such device
SIOCSIFNETMASK: No such device
tcpdump: eth1: No such device exists
(SIOCGIFHWADDR: No such device)
The HW testing tool for the switch_calc design
 type help to see all commands
testing>
\end{verbatim}

\label{netpfgaioctlerror}
\begin{verbatim}
>> table_cam_add_entry lookup_table send_to_port1 ff:ff:ff:ff:ff:ff =>
CAM_Init_ValidateContext() - done
WROTE 0x44020050 = 0xffffffff
WROTE 0x44020054 = 0xffff
WROTE 0x44020080 = 0x0003
python: ioctl: Unknown error 512
[20:27] rainbow:CLI%

\end{verbatim}

\begin{verbatim}
[7:05] rainbow:netpfga% bash  build-load-drivers.sh
+ cd /home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0
+ make all
make -C /lib/modules/5.0.0-16-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-5.0.0-16-generic'
  Building modules, stage 2.
  MODPOST 1 modules
make[1]: Leaving directory '/usr/src/linux-headers-5.0.0-16-generic'
+ sudo make install
make -C /lib/modules/5.0.0-16-generic/build M=/home/nico/projects/P4-NetFPGA/lib/sw/std/driver/sume_riffa_v1_0_0 modules
make[1]: Entering directory '/usr/src/linux-headers-5.0.0-16-generic'
  Building modules, stage 2.
  MODPOST 1 modules
make[1]: Leaving directory '/usr/src/linux-headers-5.0.0-16-generic'
install -o root -g root -m 0755 -d /lib/modules/5.0.0-16-generic/extra/sume_riffa/
install -o root -g root -m 0755 sume_riffa.ko /lib/modules/5.0.0-16-generic/extra/sume_riffa/
depmod -a 5.0.0-16-generic
+ sudo modprobe sume_riffa
modprobe: ERROR: could not insert 'sume_riffa': Exec format error
[7:06] rainbow:netpfga%

\end{verbatim}

\begin{verbatim}
java traceback when trying to install sdnet
#+BEGIN_CENTER
Exception in thread "AWT-EventQueue-0" java.lang.IllegalArgumentException: Window must not be zero
	at java.desktop/sun.awt.X11.XAtom.checkWindow(Unknown Source)
	at java.desktop/sun.awt.X11.XAtom.getAtomData(Unknown Source)
	at java.desktop/sun.awt.X11.XToolkit.getWorkArea(Unknown Source)
	at java.desktop/sun.awt.X11.XToolkit.getInsets(Unknown Source)
	at java.desktop/sun.awt.X11.XToolkit.getScreenInsets(Unknown Source)
	at java.desktop/java.awt.Window.init(Unknown Source)
	at java.desktop/java.awt.Window.<init>(Unknown Source)
	at java.desktop/java.awt.Window.<init>(Unknown Source)
	at java.desktop/java.awt.Dialog.<init>(Unknown Source)
	at java.desktop/java.awt.Dialog.<init>(Unknown Source)
	at java.desktop/javax.swing.JDialog.<init>(Unknown Source)
	at java.desktop/javax.swing.JOptionPane.createDialog(Unknown Source)
	at java.desktop/javax.swing.JOptionPane.createDialog(Unknown Source)
	at j.a.c(Unknown Source)
	at j.a.a(Unknown Source)
	at j.a.a(Unknown Source)
	at j.a.c(Unknown Source)
	at com.xilinx.installer.gui.panel.destination.b.a(Unknown Source)
	at com.xilinx.installer.gui.panel.destination.DestinationPanel.z(Unknown Source)
	at com.xilinx.installer.gui.E.a(Unknown Source)
	at com.xilinx.installer.gui.InstallerGUI.l(Unknown Source)
	at com.xilinx.installer.gui.i.actionPerformed(Unknown Source)
	at java.desktop/javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
	at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
	at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
	at java.desktop/javax.swing.DefaultButtonModel.setPressed(Unknown Source)
	at java.desktop/javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
	at java.desktop/java.awt.Component.processMouseEvent(Unknown Source)
	at java.desktop/javax.swing.JComponent.processMouseEvent(Unknown Source)
	at java.desktop/java.awt.Component.processEvent(Unknown Source)
	at java.desktop/java.awt.Container.processEvent(Unknown Source)
	at java.desktop/java.awt.Component.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Container.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Component.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
	at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
	at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.Container.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Window.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.Component.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(Unknown Source)
	at java.desktop/java.awt.EventQueue.access$500(Unknown Source)
	at java.desktop/java.awt.EventQueue$3.run(Unknown Source)
	at java.desktop/java.awt.EventQueue$3.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
	at java.desktop/java.awt.EventQueue.dispatchEvent(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
	at java.desktop/java.awt.EventDispatchThread.run(Unknown Source)


#+END_CENTER

Reason was a hidden window.

\end{verbatim}

Testing the card
\begin{verbatim}
---------------------------------------------
[ddr3B]: Running Auto Test
---------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16765, in <lambda>
    lambda event: event.callable(*event.args, **event.kw) )
  File "sw/host/script/NfSumeTest.py", line 848, in UpdateProgress
    self.progressDlg.Update(self.curProgress, str(localLine))
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16710, in __getattr__
    raise PyDeadObjectError(self.attrStr % self._name)
wx._core.PyDeadObjectError: The C++ part of the NfSumeProgress object has been deleted, attribute access no longer allowed.
Exception in thread Thread-18:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "sw/host/script/NfSumeTest.py", line 947, in run
    self.target(*self.data)
  File "sw/host/script/NfSumeTest.py", line 355, in StartAutoTest
    self.TestInterface(testName)
  File "sw/host/script/NfSumeTest.py", line 465, in TestInterface
    self.ProgramFpga('../../../bitfiles/' + self.nfSumeTestConfiguration[testName]['bitstream'])
  File "sw/host/script/NfSumeTest.py", line 586, in ProgramFpga
    self.getFpgaIndex()
  File "sw/host/script/NfSumeTest.py", line 574, in getFpgaIndex
    p = Popen(['djtgcfg', 'init', '-d', 'NetSUME'], stdout=PIPE, bufsize = 1)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

\end{verbatim}

\begin{verbatim}
---------------------------------------------
[pcie]: Running Auto Test
---------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16765, in <lambda>
    lambda event: event.callable(*event.args, **event.kw) )
  File "sw/host/script/NfSumeTest.py", line 848, in UpdateProgress
    self.progressDlg.Update(self.curProgress, str(localLine))
  File "/usr/lib/python2.7/dist-packages/wx-3.0-gtk2/wx/_core.py", line 16710, in __getattr__
    raise PyDeadObjectError(self.attrStr % self._name)
wx._core.PyDeadObjectError: The C++ part of the NfSumeProgress object has been deleted, attribute access no longer allowed.
Exception in thread Thread-21:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "sw/host/script/NfSumeTest.py", line 947, in run
    self.target(*self.data)
  File "sw/host/script/NfSumeTest.py", line 466, in TestInterface
    self.serialCon.readlines()
  File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 495, in read
    raise SerialException('device reports readiness to read but returned no data (device disconnected or multiple access on port?)')
SerialException: device reports readiness to read but returned no data (device disconnected or multiple access on port?)


\end{verbatim}

Another generated file problem:
\begin{verbatim}
nico@nsg-System:~/master-thesis/netpfga$ grep -i error $P4_PROJECT_DIR/nf_sume_sdnet_ip/SimpleSumeSwitch/LOG
ERROR: [VRFC 10-1491] unexpected EOF [/home/nico/master-thesis/netpfga/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/S_CONTROLLERs.HDL/S_CONTROLLER_SimpleSumeSwitch.vp:37]
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_0_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_1_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_2_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_3_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_4_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_5_ErrorCheck
INFO: [VRFC 10-311] analyzing module TopDeparser_t_EngineStage_6_ErrorCheck

\end{verbatim}

function syntax not supported
\begin{verbatim}
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4
headers.p4(246):syntax error, unexpected IDENTIFIER, expecting (
bit<16> ones_complement_sum
        ^^^^^^^^^^^^^^^^^^^
error: 1 errors encountered, aborting compilation
Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 1
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
Makefile:31: recipe for target 'frontend' failed
make: *** [frontend] Error 2
nico@nsg-System:~/master-thesis/netpfga$


\end{verbatim}

\begin{verbatim}
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default$ cd $NF_DESIGN_DIR/test/sim_switch_default && make 2>&1 |  tee ~/master-thesis/netpfga/log/step8-$(date +%F-%H%M%S)
rm -f config_writes.py*
rm -f *.pyc
cp /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.py ./
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.py': No such file or directory
Makefile:36: recipe for target 'all' failed
make: *** [all] Error 1

\end{verbatim}

\begin{verbatim}
Finished scanning sources
INFO: [IP_Flow 19-234] Refreshing IP repositories
INFO: [IP_Flow 19-1700] Loaded user IP repository '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/ip_repo'.
INFO: [IP_Flow 19-2313] Loaded Vivado IP repository '/opt/Xilinx/Vivado/2018.2/data/ip'.
WARNING: [IP_Flow 19-3664] IP 'bd_7ad4_xpcs_0' generated file not found '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/.Xil/Vivado-26302-nsg-System/coregen/bd_7ad4_xpcs_0_1/elaborate/configure_gt.tcl'. Please regenerate to continue.
WARNING: [IP_Flow 19-3664] IP 'bd_a1aa_xpcs_0' generated file not found '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/.Xil/Vivado-26302-nsg-System/coregen/bd_a1aa_xpcs_0_2/elaborate/configure_gt.tcl'. Please regenerate to continue.
open_project: Time (s): cpu = 00:00:05 ; elapsed = 00:00:05 . Memory (MB): peak = 1365.715 ; gain = 188.977 ; free physical = 9396 ; free virtual = 15104
# puts "\nOpening $design Implementation design\n"

\end{verbatim}

\begin{verbatim}
WARNING: [Synth 8-689] width (12) of port connection 'control_S_AXI_ARADDR' does not match port width (8) of module 'SimpleSumeSwitch' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:199]
ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_VALID' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:218]
ERROR: [Synth 8-448] named port connection 'tuple_out_sume_metadata_DATA' does not exist for instance 'SimpleSumeSwitch_inst' of module 'SimpleSumeSwitch' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:219]
ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:44]
ERROR: [Synth 8-6156] failed synthesizing module 'nf_sume_sdnet_ip' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/synth/nf_sume_sdnet_ip.v:57]
ERROR: [Synth 8-6156] failed synthesizing module 'nf_datapath' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/hdl/nf_datapath.v:44]
ERROR: [Synth 8-6156] failed synthesizing module 'top' [/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/hdl/top.v:43]
\end{verbatim}

Missing ``souce'' files:
\begin{verbatim}
cc -c -fPIC /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API/CAM.c -I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API
cc -std=c99 -Wall -Werror -fPIC -c libcam.c -I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume -I/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/sw/API
cc -L/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/sw/sume -shared -o libcam.so libcam.o CAM.o -lsumereg
/usr/bin/ld: cannot find -lsumereg
collect2: error: ld returned 1 exit status
Makefile:52: recipe for target 'libcam' failed
make[1]: *** [libcam] Error 1
make[1]: Leaving directory '/home/nico/master-thesis/netpfga/minip4/sw/CLI'
ERROR: could not compile libcam souce files


\end{verbatim}

Generated files not found:
\begin{verbatim}
make: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test'
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/Makefile': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_0_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_1_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_2_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_stim.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/nf_interface_3_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_log.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/dma_0_expected.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/Makefile': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.log': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_expect.axi': No such file or directory
cp: cannot stat '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/reg_stim.axi': No such file or directory
NetFPGA environment:
   Root dir:       /home/nico/projects/P4-NetFPGA
   Project name:   simple_sume_switch
   Project dir:    /tmp/nico/test/simple_sume_switch
   Work dir:       /tmp/nico
512
=== Work directory is /tmp/nico/test/simple_sume_switch
=== Setting up test in /tmp/nico/test/simple_sume_switch/sim_switch_default
=== Running test /tmp/nico/test/simple_sume_switch/sim_switch_default ... using cmd ['/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default/run.py', '--sim', 'xsim']
+ date
Die Jul 23 13:34:54 CEST 2019
+ [  = no ]
+ cd /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch
+ make
make: *** No targets specified and no makefile found.  Stop.

\end{verbatim}

Renaming variables breaks the compile process
\begin{verbatim}
 @Xilinx_MaxPacketRegion(1024)
 control TopDeparser(
-    packet_out b,
-    in Parsed_packet p,
+    packet_out packet,
+    in Parsed_packet hdr,
     in user_metadata_t user_metadata,
     inout digest_data_t digest_data,
     inout sume_metadata_t sume_metadata) {

     apply {
-        b.emit(p.ethernet);
+        packet.emit(hdr.ethernet);
     }
+
+
 }
\end{verbatim}

LPM size must be != 64

\begin{verbatim}
minip4_solution.p4(38): [--Wwarn=uninitialized_out_param] warning: out parameter meta may be uninitialized when RealParser terminates
    out metadata meta,
                 ^^^^
minip4_solution.p4(35)
parser RealParser(
       ^^^^^^^^^^
error: LPM table size should be 2^n - 1
actions_nat64_generic.p4(169): error: could not not map table size size
        size = 64;
        ^^^^
error: table match_types are not the same
actions_arp.p4(35): error: could not map table key(s) KeyElement
            hdr.arp.dst_ipv4_addr: lpm;
            ^^^^^^^^^^^^^^^^^^^^^
error: LPM table size should be 2^n - 1
actions_arp.p4(55): error: could not not map table size size
        size = 64;
        ^^^^
Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 1
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
Makefile:31: recipe for target 'frontend' failed
make: *** [frontend] Error 2
nico@nsg-System:~/master-thesis/netpfga/log$

\end{verbatim}

LIMIT table match types are not the same error
\begin{verbatim}
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4
actions_egress.p4(52): warning: Table v6_networks is not used; removing
table v6_networks {
      ^^^^^^^^^^^
actions_egress.p4(69): warning: Table v4_networks is not used; removing
table v4_networks {
      ^^^^^^^^^^^
actions_nat64_generic.p4(174): warning: Table nat46 is not used; removing
    table nat46 {
          ^^^^^
minip4_solution.p4(38): [--Wwarn=uninitialized_out_param] warning: out parameter meta may be uninitialized when RealParser terminates
    out metadata meta,
                 ^^^^
minip4_solution.p4(35)
parser RealParser(
       ^^^^^^^^^^
error: table match_types are not the same
actions_arp.p4(35): error: could not map table key(s) KeyElement
            hdr.arp.dst_ipv4_addr: lpm;
            ^^^^^^^^^^^^^^^^^^^^^
Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 1

    table v4_arp {
        key = {
            hdr.ethernet.dst_addr: exact;
            hdr.arp.opcode: exact;
            hdr.arp.dst_ipv4_addr: lpm;
        }
        actions = {
            controller_debug_table_id;
            arp_reply;
            NoAction;
        }
        size = ICMP6_TABLE_SIZE;
        default_action = controller_debug_table_id(TABLE_ARP);
    }


\end{verbatim}

Implicit error saying that LPM tables don't work:

\begin{verbatim}
s/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/realmain_lookup_table_0_t.HDL/xpm_memory.sv
[SW] LPM_Init() - start
[SW] LPM_Init() - done
[SW] LPM_LoadDataset() - start
[SW] LPM_LoadDataset() failed with error code = 12
FATAL_ERROR: Vivado Simulator kernel has encounted an exception from DPI C function: LPM_VerifyDataset(). Please correct.
Time: 2016466 ps  Iteration: 0  Process: /SimpleSumeSwitch_tb/LPM_VerifyDataset
  File: /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/Testbench/SimpleSumeSwitch_tb.sv
\end{verbatim}

\begin{verbatim}
minip4_solution.p4(35)
parser RealParser(
       ^^^^^^^^^^
actions_nat64_generic.p4(173): error: table size too small for match_type(EM): 63 < 64
        size = 63;
               ^^
actions_nat64_generic.p4(173): error: could not not map table size size
        size = 63;
        ^^^^

\end{verbatim}

Unsupported default parameters
\begin{verbatim}
actions_egress.p4(89): error: data-plane arguments in default_actions are currently unsupported: realmain_controller_debug_table_id_0
    default_action = controller_debug_table_id(TABLE_V4_NETWORKS);
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
terminate called after throwing an instance of 'Util::CompilerBug'
  what():  In file: /wrk/hdscratch/staff/mohan/p4c_sdnet/build/p4c/extensions/sdnet/translate/core/lookupEngine.cpp:137
Compiler Bug: actions_egress.p4(89): unhandled expression realmain_controller_debug_table_id/realmain_controller_debug_table_id_0(5);
    default_action = controller_debug_table_id(TABLE_V4_NETWORKS);
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

\end{verbatim}

Compiler Bug / ifstatement
\begin{verbatim}
minip4_solution.p4(39)
parser RealParser(
       ^^^^^^^^^^
terminate called after throwing an instance of 'Util::CompilerBug'
  what():  In file: /wrk/hdscratch/staff/mohan/p4c_sdnet/build/p4c/extensions/sdnet/writers/pxWriter.h:20
Compiler Bug: unhandled node: <IfStatement>(471564)

Makefile:34: recipe for target 'all' failed
make[1]: *** [all] Error 134
make[1]: Leaving directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
Makefile:31: recipe for target 'frontend' failed

\end{verbatim}

Applying table ``twice'' in different branches is impossible (another
compiler bug)

\begin{verbatim}

make -C src/
make[1]: Entering directory '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/src'
p4c-sdnet -o minip4.sdnet --sdnet_info .sdnet_switch_info.dat minip4_solution.p4
minip4_solution.p4(19): [--Wwarn=uninitialized_out_param] warning: out parameter meta may be uninitialized when RealParser terminates
    out metadata meta,
                 ^^^^
minip4_solution.p4(16)
parser RealParser(
       ^^^^^^^^^^
terminate called after throwing an instance of 'Util::CompilerBug'
  what():  In file: /wrk/hdscratch/staff/mohan/p4c_sdnet/build/p4c/extensions/sdnet/translate/core/tupleEngine.cpp:324
Compiler Bug: overwrite

Makefile:34: recipe for target 'all' failed


\end{verbatim}

Adding entries requires setting all parameters
\begin{verbatim}
>> table_cam_add_entry realmain_v6_networks_0 realmain.set_egress_port 42540766411362381960998550477184434178 => 1
ERROR: not enough fields provided to complete _hexify()

\end{verbatim}

Broken code that cannot convret long to int:
\begin{verbatim}
>> table_cam_delete_entry realmain_v6_networks_0 42540766411362381960998550477184434179
ERROR: failed to convert 42540766411362381960998550477184434179 of type <type 'long'> to an integer
nico@nsg-System:~/master-thesis/netpfga/minip4/sw/CLI$

\end{verbatim}

% ----------------------------------------------------------------------
\section{P4 error messages}

\begin{verbatim}
Warning: you requested the nanomsg event logger, but bmv2 was compiled without -DBMELOG, and the event logger cannot be activated
Calling target program-options parser
[14:01:44.334] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.icmp6': MyIngress.controller_debug_table_id - 2,
[14:01:44.341] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.nat64': MyIngress.controller_debug_table_id - 1,
[14:01:44.344] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act': act -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_0': act_0 -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_nat64_icmp6_generic': MyIngress.nat64_icmp6_generic -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_1': act_1 -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_2': act_2 -
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.v4_networks': MyIngress.controller_debug_table_id - 5,
[14:01:44.345] [bmv2] [D] [thread 23356] Set default default entry for table 'MyIngress.v6_networks': MyIngress.controller_debug_table_id - 3,
[14:01:44.346] [bmv2] [D] [thread 23356] Set default default entry for table 'tbl_act_3': act_3 -
Invalid entry type 'expression' in field list
bad json:
{
   "type" : "expression",
   "value" : {
      "type" : "expression",
      "value" : {
         "left" : null,
         "op" : "d2b",
         "right" : {
            "type" : "field",
            "value" : [ "scalars", "metadata.chk_icmp6_na_ns" ]
         }
      }
   }
}

\end{verbatim}

\begin{verbatim}
../p4src/static-mapping.p4(121): error: MyIngress.nat64, Multiple LPM keys in table
    table nat64 {
          ^^^^^
Compilation Error

    table nat64 {
        key = {
            hdr.ipv6.src_addr: lpm;
            hdr.ipv6.dst_addr: lpm;
        }
        actions = {
            controller_debug;
            nat64_static;
            NoAction;
        }
        size = NAT64_TABLE_SIZE;
        default_action = controller_debug;
    }

\end{verbatim}

\begin{verbatim}
../p4src/static-mapping.p4(60): error: SwitchStatement: switch statements not allowed in actions
        switch(hdr.icmp6.type) {
        ^^^^^^
\end{verbatim}

No if in actions:

\begin{verbatim}
../p4src/static-mapping.p4(57): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp.setValid();
        ^^^^^^^^^^^^^^^^^^^
../p4src/static-mapping.p4(70): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp6.setInvalid();
        ^^^^^^^^^^^^^^^^^^^^^^
../p4src/static-mapping.p4(73): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp6_na_ns.setInvalid();
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../p4src/static-mapping.p4(74): error: MethodCallStatement: Conditional execution in actions is not supported on this target
        hdr.icmp6_option_link_layer_addr.setInvalid();
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Compilation Error
p4@ubuntu:~/master-thesis/p4app$

        if(hdr.ipv6.next_header == PROTO_ICMP6) {
            nat64_icmp6();
        }
\end{verbatim}

\begin{verbatim}
p4c --target bmv2 --arch v1model --std p4-16 "../p4src/checksum_diff.p4" -o "/home/p4/master-thesis/p4src"
In file: /home/p4/p4-tools/p4c/backends/bmv2/common/expression.cpp:168
Compiler Bug: ../p4src/actions_delta_checksum.p4(60): ones_complement_sum(hdr.udp.checksum, tmp);: unhandled case
    tmp = ones_complement_sum(hdr.udp.checksum, meta.v6sum);
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Compilation Error```
Using the following code:
```/* copied from
https://p4.org/p4-spec/docs/PSA-v1.1.0.html#appendix-internetchecksum-implementation
*/

bit<16> ones_complement_sum(in bit<16> x, in bit<16> y) {
    bit<17> ret = (bit<17>) x + (bit<17>) y;
    if (ret[16:16] == 1) {
        ret = ret + 1;
    }
    return ret[15:0];
}```
And p4c version:
```p4@ubuntu:~/master-thesis/p4app$ p4c --version
p4c 0.5 (SHA: 5ae30ee)```
\end{verbatim}

% ----------------------------------------------------------------------
\section{Traces}

Proof of stuff working, reference for each stage / feature

Stuff that needs to be cleaned up


% ----------------------------------------------------------------------
\subsection{P4/BMV NAT64 Delta based traces}
\begin{verbatim}
*** DONE 2019-07-21: Proof of v6->v4 working delta based
    CLOSED: [2019-07-21 Sun 12:30]
#+BEGIN_CENTER
 pcap/tcp-udp-delta-from-v6-2019-07-21-0853-h1.pcap | Bin 0 -> 4252 bytes
 pcap/tcp-udp-delta-from-v6-2019-07-21-0853-h3.pcap | Bin 0 -> 2544 bytes
#+END_CENTER

\end{verbatim}


\begin{verbatim}
 create mode 100644 pcap/tcp-udp-delta-2019-07-17-1555-h1.pcap
 create mode 100644 pcap/tcp-udp-delta-2019-07-17-1555-h3.pcap
 create mode 100644 pcap/tcp-udp-delta-2019-07-17-1557-h1.pcap
 create mode 100644 pcap/tcp-udp-delta-2019-07-17-1558-h3.pcap
\end{verbatim}


% ----------------------------------------------------------------------
\subsection{P4/NetFPGA NAT64 Delta based traces}
\begin{verbatim}
**** DONE Testing v4->v6 tcp: ok (version 10.0)
     CLOSED: [2019-08-04 Sun 09:15]
#+BEGIN_CENTER
nico@ESPRIMO-P956:~/master-thesis/bin$ ./socat-connect-tcp-v4
+ echo from-v4-ok
+ socat - TCP:10.0.0.66:2345
TCPv6-ok
nico@ESPRIMO-P956:~/master-thesis/bin$ ./socat-listen-tcp-v6
from-v4-ok

#+END_CENTER

trace:
netfpga-nat64-2019-08-04-0907-enp2s0f0.pcap
netfpga-nat64-2019-08-04-0907-enp2s0f1.pcap

**** DONE Testing v4->v6 udp: ok (version 10.1)
trace:
 create mode 100644 pcap/netfpga-nat64-udp-2019-08-04-0913-enp2s0f0.pcap
 create mode 100644 pcap/netfpga-nat64-udp-2019-08-04-0913-enp2s0f1.pcap

\end{verbatim}

Bigger packets
\begin{verbatim}
*** DONE 2019-08-04: version 10.1/10.2: new maxpacketregion: v4->v6 works
    CLOSED: [2019-08-04 Sun 19:42]
#+BEGIN_CENTER
nico@ESPRIMO-P956:~/master-thesis/bin$ ./init_ipv4_esprimo.sh
nico@ESPRIMO-P956:~/master-thesis/bin$ ./set_ipv4_neighbor.sh

#+END_CENTER

Test 20 first:

- Does't work -> missed to add table entries
- Does work after setting table entries
- 300 works
- 1450 works
- 1500 does not work

Proof:

 create mode 100644 pcap/netfpga-10.2-maxpacket-2019-08-04-1931-enp2s0f0.pcap
 create mode 100644 pcap/netfpga-10.2-maxpacket-2019-08-04-1931-enp2s0f1.pcap

\end{verbatim}
\begin{verbatim}
*** DONE 2019-08-04: test v6 -> v4: works for 1420
    CLOSED: [2019-08-04 Sun 20:30]

Proof:
#+BEGIN_CENTER
 create mode 100644 pcap/netfpga-10.2-fromv6tov4-2019-08-04-1943-enp2s0f0.pcap
 create mode 100644 pcap/netfpga-10.2-fromv6tov4-2019-08-04-1943-enp2s0f1.pcap


\end{verbatim}

% ----------------------------------------------------------------------

\section{Introduction}

\subsection{\label{introduction:taskdescription}The Task}
 - Milestone 1: Stateless NAT64/NAT46 translations in P4
 - Milestone 2: Stateful (dynamic) NAT64/NAT46 translations
 - Milestone 3: Hardware adaption

This thesis is
into 3 milestone
P4 environment
a lot of potential
Programming language in the network
Not only faster, but also more convienient.

**** High speed NAT64 with P4
 Currently there are two main open source NAT64 solution available:
 tayga and jool. The former is a single threaded, cpu bound user
 space solution, the latter a custom Linux kernel module.

 This thesis challenges this status quo by developing a P4 based
 solution supporting all features of jool/tayga and comparing the
 performance, security and adaptivity of the solutions.


Describe your task.


\begin{verbatim}

***** Motivation zeigen
***** IPv6, NetPFGA mehr Möglichketien
***** P4 erwähnen
***** Task gut zu zeigen,  alles erreicht
 use cases / sample applications


\end{verbatim}

% ----------------------------------------------------------------------
\section{\label{appendix:p4}P4 notes}
% ----------------------------------------------------------------------
\subsection{\label{appendix:p4:keyretrieval}Key retrieval chat log}
\begin{verbatim}
Key and mask for matching destination is in table. We need this
 information in the action. However this information is not exposed, so
 we need to specify another parameter with the same information as in
 the key(s).

 Log from slack: (2019-03-14)

 nico [1:55 PM]
 If I use LPM for matching, can I easily get the network address from P4 or do I have to use a bitmask myself? In the latter case it is not exactly clear how to get the mask from the table

 Nate Foster [1:58 PM]
 You want to retrieve the address in the packet? In a table?
 And do you want to do the retrieving from the data plane or the control plane? (edited)

 nico [2:00 PM]
 If I have a match in a table that matches on LPM, it can be any IP address in a network
 For calculating the NAT64/NAT46 translation, I will need the base address, i.e. network address to do subtractions/additions
 So it is fully data plane, what I would like to do
 I'll commit sample code to show the use case more clearly
 https://gitlab.ethz.ch/nicosc/master-thesis/blob/master/p4src/static-mapping.p4#L73
 GitLab
 p4src/static-mapping.p4 · master · nicosc / master-thesis
 gitlab.ethz.ch
 So the action nat64_static() is used in the table v6_networks.
 In v6_networks I use a match on `hdr.ipv6.dst_addr: lpm;`
 What I would like to be able is to get the network address ; I can do that manually, if I have the mask
 I can also re-inject this parameter by another action argument, but I'd assume that I can somewhere read this out from the table / match

 Nate Foster [2:15 PM]
 To make sure I understand, in the data plane, you want to retrieve the address in the lpm pattern? (edited)

 nico [2:16 PM]
 I want to retrieve the key

 Nate Foster [2:16 PM]
 Wait. The value `hdr.ipv6.dst_addr` is the thing used in the match.
 So you have that.
 What you don’t have is the IPv6 address and mask put into the table by the control plane.
 I assume you want the latter, right?

 nico [2:17 PM]
 For example, if my matching key is 2001:db8::/32 and the real address is 2001:db8::f00, then I would like to retrieve 2001:db8:: and 32 from the table
 exactly :slightly_smiling_face:
 I can "fix" this by adding another argument, but it feels somewhat wrong to do that
 Because the table already knows this information

 Nate Foster [2:26 PM]
 I can’t think of a way other than the action parameter hack.

 nico [2:26 PM]
 Oh, ok
 Is it because the information is "lost in hardware"?

 Nate Foster [2:31 PM]
 No you’re right that most implementations have the value in memory. And one can imagine a different table API that allowed one to retrieve it in the data plane.
 But unless I am missing something obvious, P4 hides it…

\end{verbatim}

% ----------------------------------------------------------------------
\subsection{\label{appendix:p4:tableretrieval}Table retrieval problem}
\begin{verbatim}
Is there any meta information for "from which table was the action
called" available? My use case is having a debug action that sends
packets to the controller and I use it as a default_action in various
tables; however know I don't know anymore from which table the action
was called. Is there any kind of meta information which table called
me available?

I could work around this by using if(! .. .hit) { my_action(table_id)
}, but it would not work with using default_action = ...

\end{verbatim}

\begin{verbatim}
Is there any meta information for "from which table was the action
called" available? My use case is having a debug action that sends
packets to the controller and I use it as a default_action in various
tables; however know I don't know anymore from which table the action
was called. Is there any kind of meta information which table called
me available?

I could work around this by using if(! .. .hit) { my_action(table_id)
}, but it would not work with using default_action = ...

\end{verbatim}
% ----------------------------------------------------------------------
\subsection{\label{appendix:p4:datadefinition}Data definition
  redundancy}
\begin{verbatim}
*** DONE Synchronisation with the controller
    - Double data type definition -> might differ
    - TYPE_CPU for ethernet
    - Port ingress offset (9 vs. 16 bit)

\end{verbatim}

% ----------------------------------------------------------------------
\subsection{\label{appendix:p4:python2unicode
  }Python2 unicode issue}

\begin{verbatim}
ipaddress.ip_network("2001:db8:61::/64")
IPv6Network(u'3230:3031:3a64:6238:3a36:313a:3a2f:3634/128')

Fix:
from __future__ import unicode_literals

\end{verbatim}
% ----------------------------------------------------------------------
\subsection{\label{appendix:p4:p4os}P4 OS}

\begin{verbatim}
Not addressed so far: how to create re-usable code fragments that can
be plugged in easily. There could be a hypothetical "P4OS" that
manages code fragments. This might include, but not limited to
downloading (signed?) source code, managing dependencies similar to
Linux package management, handling updates, etc.

\end{verbatim}



%---------------------------------------------------------------------------------------------------------
\printnomenclature
\abbrev{ARP}{Address resolution protocol}
\abbrev{ASIC}{Application-specific integrated circuit}
\abbrev{DAC}{Direct attach cable}
\abbrev{FGPA}{Field-programmable gate array}
\abbrev{LPM}{Longes prefix matching}
\abbrev{MTU}{Maximum transfer unit}
\abbrev{NDP}{Neighbor Discovery Protocol}
\abbrev{NAT}{Network Address Translation}
\abbrev{NAT64}{Network Address Translation from / to IPv6 to / from IPv4}
\abbrev{RIR}{Regional Internet Registry}
\abbrev{RTT}{Round Trip Time}
