master-thesis/doc/Conclusion.tex

548 lines
24 KiB
TeX
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

\chapter{\label{summary}Conclusion}
%** Summary.tex: What have you achieved, what have you presented in this
% document. What are the highlights of your work.
% It should conclude by a conclusion.
Sum up what you have done and recapitulate your key findings.
%\section{\label{conclusion:overall}Overall}
\section{\label{conclusion:softwarenat64}Software based NAT64}
\section{\label{conclusion:general}General}
Many misleading
\section{\label{conclusion:bmv2}BMV2}
\section{\label{conclusion:P4}P4}
NDP parsing problem
checksumming a frequent problem and helper
Many possibilities
Protocol independent
Easy architecture
Limitations in
if in action limitations
Limits if in actions
python2 only - unicode errors
IPv6: NDP: not easy to parse, as unknown number of following fields
No support for multiple LPM keys in a table, can be solved with
ternary matching.
switch cannot be used in actions
if things don't work, often a checksum problem.
if frame checksum, then length of packet is broken
\begin{verbatim}
p4c --target bmv2 --arch v1model --std p4-16 "../p4src/static-mapping.p4" -o "/home/p4/master-thesis/p4src"
../p4src/static-mapping.p4(366): error: Program is not supported by this target, because table MyIngress.v6_networks has multiple successors
table v6_networks {
^^^^^^^^^^^
\end{verbatim}
\begin{verbatim}
ipaddress.ip_network("2001:db8:61::/64")
IPv6Network(u'3230:3031:3a64:6238:3a36:313a:3a2f:3634/128')
Fix:
from __future__ import unicode_literals
\end{verbatim}
The tooling around P4 is still fragile, encountered many bugs
in the development.\cite{schottelius:github1675}
or missing features (\cite{schottelius:github745},
\cite{theojepsen:_get})
Hitting expression bug
retrieving information from tables
\begin{verbatim}
Key and mask for matching destination is in table. We need this
information in the action. However this information is not exposed, so
we need to specify another parameter with the same information as in
the key(s).
Log from slack: (2019-03-14)
nico [1:55 PM]
If I use LPM for matching, can I easily get the network address from P4 or do I have to use a bitmask myself? In the latter case it is not exactly clear how to get the mask from the table
Nate Foster [1:58 PM]
You want to retrieve the address in the packet? In a table?
And do you want to do the retrieving from the data plane or the control plane? (edited)
nico [2:00 PM]
If I have a match in a table that matches on LPM, it can be any IP address in a network
For calculating the NAT64/NAT46 translation, I will need the base address, i.e. network address to do subtractions/additions
So it is fully data plane, what I would like to do
I'll commit sample code to show the use case more clearly
https://gitlab.ethz.ch/nicosc/master-thesis/blob/master/p4src/static-mapping.p4#L73
GitLab
p4src/static-mapping.p4 · master · nicosc / master-thesis
gitlab.ethz.ch
So the action nat64_static() is used in the table v6_networks.
In v6_networks I use a match on `hdr.ipv6.dst_addr: lpm;`
What I would like to be able is to get the network address ; I can do that manually, if I have the mask
I can also re-inject this parameter by another action argument, but I'd assume that I can somewhere read this out from the table / match
Nate Foster [2:15 PM]
To make sure I understand, in the data plane, you want to retrieve the address in the lpm pattern? (edited)
nico [2:16 PM]
I want to retrieve the key
Nate Foster [2:16 PM]
Wait. The value `hdr.ipv6.dst_addr` is the thing used in the match.
So you have that.
What you dont have is the IPv6 address and mask put into the table by the control plane.
I assume you want the latter, right?
nico [2:17 PM]
For example, if my matching key is 2001:db8::/32 and the real address is 2001:db8::f00, then I would like to retrieve 2001:db8:: and 32 from the table
exactly :slightly_smiling_face:
I can "fix" this by adding another argument, but it feels somewhat wrong to do that
Because the table already knows this information
Nate Foster [2:26 PM]
I cant think of a way other than the action parameter hack.
nico [2:26 PM]
Oh, ok
Is it because the information is "lost in hardware"?
Nate Foster [2:31 PM]
No youre right that most implementations have the value in memory. And one can imagine a different table API that allowed one to retrieve it in the data plane.
But unless I am missing something obvious, P4 hides it…
\end{verbatim}
no meta information
\begin{verbatim}
Is there any meta information for "from which table was the action
called" available? My use case is having a debug action that sends
packets to the controller and I use it as a default_action in various
tables; however know I don't know anymore from which table the action
was called. Is there any kind of meta information which table called
me available?
I could work around this by using if(! .. .hit) { my_action(table_id)
}, but it would not work with using default_action = ...
\end{verbatim}
type definitions separate
Code sharing (controller, switch)
\begin{verbatim}
*** DONE Synchronisation with the controller
- Double data type definition -> might differ
- TYPE_CPU for ethernet
- Port ingress offset (9 vs. 16 bit)
\end{verbatim}
No switch in actions, No conditional execution in actions
P4os - reusable code
\begin{verbatim}
Not addressed so far: how to create re-usable code fragments that can
be plugged in easily. There could be a hypothetical "P4OS" that
manages code fragments. This might include, but not limited to
downloading (signed?) source code, managing dependencies similar to
Linux package management, handling updates, etc.
\end{verbatim}
idomatic problem: Security issue: not checking checksums before
% ----------------------------------------------------------------------
\section{\label{conclusion:netpfga}NetFGPA - all HERE}
personal note here
tested various kernels for table debugging
MTU limitations: 1500 according to a private mail from Salvator Galea
cambridge / uk
long compile process
error prone compile process
many dependencies
lpm not supported!
Netpfga live,
Vivado
SDNET
xx k lines of supporting code
Vivado installation: silent errors, infinite loop, missing libncurses5
82k lines of code that are interdependent
Many non critical error messages on the way
Zero exit fatal errors
missing / spreaded documentation
tcpdump on local nfX doesn't work -> can only debug on other endpoint
First card: Writing tables fails
hardware debug shows some errors
but hardware debug on correct card also shows some error
Debug ioctl errors when writing table entries
Output all ports -> port mapping documented only in a testdata script
hwtest: Execution fails due to missing djtgcfg
no payload accessq
Many workarounds
Table size 63, table size 64,
Table entries require arguments of all possible actions, not only used
one.
Compile time hours
Silent errors
Unclear errors: broken board
Due to the very fragile nature of the build framework from the
NetFPGA-Live repository,
Renaming VARIABLES in the definition of
Reproducibility:
hours for finding right output ports
packet size / annotation
Needed to debug internal parsing errors
3x rebooting to get card working with bitstream
Variable renaming breaks the compile process
\begin{verbatim}
It seems I was really mistaken for the last weeks
If I am not totally mistaken, the following is happening with the netpfga:
I was testing sending and receiving packets on the same computer; so I sent a packet on nfX and expected an answer on nf0, which is how I wanted to verify that the card works
So I ran tcpdump on nf0, send a packet with ping6 and scapy on nf{0,1,2,3} (edited)
I have never seen the switch emitting ANY packet back with tcpdump
Now with the card connected to another host, sending neighbor solicitation, I see duplicated packets on the other host - so it seems that it might have worked all the time, just that tcpdump on nfX on the host which contains the card does not show the packets
\end{verbatim}
debugging generated tcl code to debug impl1 error
Cable problems:
\begin{verbatim}
[ 488.265148] ixgbe 0000:02:00.0: failed to initialize because an unsupported SFP+ module type was detected.
[ 488.265157] ixgbe 0000:02:00.0: Reload the driver after installing a supported module.
[ 488.265605] ixgbe 0000:02:00.0: removed PHC on enp2s0f0
\end{verbatim}
function syntax not supported, using defines instead
4-6 MB logfiles for a compile process.
confusing messages
\begin{verbatim}
WARNING: command 'get_user_parameter' will be removed in the 2015.3
release, use 'get_user_parameters' instead
\end{verbatim}
critical non critical errors
\begin{verbatim}
CRITICAL WARNING: [BD 41-737] Cannot set the parameter TRANSLATION_MODE on /axi_interconnect_0. It is read-only.
\end{verbatim}
\begin{verbatim}
- step9 (sume simulation, the longest step) in the process calls
"config_writes.py"
- config_writes.py fails with a syntax error, as it is incomplete
python code
- config_writes.py and config_writes.sh are generated by
gen_config_writes.py
- gen_config_writes.py reads config_writes.txt
- config_writes.txt is created in step 5 (sdnet simulation)
- step 5 consists of running xsc, xelab and xsim
- xsim (re-)generates config_writes.txt according to a watch ls -l
on the file: ${XILINX_VIVADO}/bin/xsim --runall
SimpleSumeSwitch_tb#work.glbl
- it seems (by grep -r) that ./Testbench/SimpleSumeSwitch_tb.sv is
responsible for writing config_writes.txt
- It seems that the "task" "SV_write_control" inside that file is
responsible for writing the content, which in turn uses
axi4_lite_master_write_request_control
\end{verbatim}
\begin{verbatim}
- Cannot easily run P4 on notebook - changes to the system very
invasive
- Varous compiler bugs/limitations
- Very very deep rabbithole problems
- Hanging/sleeping issue -- unclear whether it does something or
not
- Open impl_1 error with unclear reason
- logfiles referenced that don't exist
Run output will be captured here: /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.runs/synth/runme.log
nico@nsg-System:~/master-thesis/netpfga/log$ ls -alh /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.runs/synth/runme.log
ls: cannot access '/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.runs/synth/runme.log': No such file or directory
- even "short" compile runs taking 30m+
control_sub_m02_data_fifo_0_synth_1: /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.runs/control_sub_m02_data_fifo_0_synth_1/runme.log
nico@nsg-System:~/master-thesis/netpfga/minip4/testdata$ less /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.runs/control_sub_m02_data_fifo_0_synth_1/runme.log
/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/hw/project/simple_sume_switch.runs/control_sub_m02_data_fifo_0_synth_1/runme.log: No such file or directory
- Wrong warnings: using 2018.2, getting warnings about things
removed in 2015.3
WARNING: command 'get_user_parameter' will be removed in the 2015.3
release, use 'get_user_parameters' instead
- A script/makefile generates a python script that generates a shell
script and later then a python script. If there is a mistake in
generating the first python script (syntax ok, but content is
not correct) then a much later stage of the compile process will
fail due to a syntax error in the third generated
script. However that syntax error is not fatal in the build
process and thus can only be seen with careful analysis of the
logfile, which is around 700 KiB or 10k lines per compile
process and contains 328 lines matching "error" and
"warning".
Most of the error and warning messages seem to be non-critical
(even if saying they are). Then there are a variety of INFO
messages that actually constitute ERROR messages, but are not
flagged as such nor do they cause the build process to abort.
\end{verbatim}
LPM tables don't work
match type exact - table must be at least 64 in size
multiple reboots sometimes required for flashing
Damaged, enlarged packets
\begin{verbatim}
** The NetPFGA saga
Problems encountered:
- The logfile for a compile run is 10k+ lines
- Many logged errors can actually be ignored (?) like:
ERROR: [VRFC 10-1491] unexpected EOF [/home/nico/master-thesis/netpfga/minip4/nf_sume_sdnet_ip/SimpleSumeSwitch/S_CONTROLLERs.HDL/S_CONTROLLER_SimpleSumeSwitch.vp:37]
ERROR: [VRFC 10-426] cannot find port tuple_out_sume_metadata_DATA on this module [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:219]
ERROR: [VRFC 10-426] cannot find port tuple_out_sume_metadata_VALID on this module [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:218]
ERROR: [VRFC 10-426] cannot find port tuple_in_sume_metadata_DATA on this module [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:185]
ERROR: [VRFC 10-426] cannot find port tuple_in_sume_metadata_VALID on this module [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/wrapper/nf_sume_sdnet.v:184]
ERROR: [VRFC 10-2063] Module <S_RESETTER_line> not found while processing module instance <S_RESET_clk_line> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/Simp
leSumeSwitch/SimpleSumeSwitch.v:332]
ERROR: [VRFC 10-2063] Module <S_RESETTER_lookup> not found while processing module instance <S_RESET_clk_lookup> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/
SimpleSumeSwitch/SimpleSumeSwitch.v:343]
ERROR: [VRFC 10-2063] Module <S_RESETTER_control> not found while processing module instance <S_RESET_clk_control> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_i
p/SimpleSumeSwitch/SimpleSumeSwitch.v:354]
ERROR: [VRFC 10-2063] Module <TopParser_t> not found while processing module instance <TopParser> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/SimpleSumeSwitc
h/SimpleSumeSwitch.v:436]
ERROR: [VRFC 10-2063] Module <TopPipe_lvl_t> not found while processing module instance <TopPipe_lvl> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/SimpleSumeS
witch/SimpleSumeSwitch.v:474]
ERROR: [VRFC 10-2063] Module <dummy_table_for_netpfga_t> not found while processing module instance <dummy_table_for_netpfga> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_s
ume_sdnet_ip/SimpleSumeSwitch/SimpleSumeSwitch.v:502]
ERROR: [VRFC 10-2063] Module <TopPipe_lvl_0_t> not found while processing module instance <TopPipe_lvl_0> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/SimpleS
umeSwitch/SimpleSumeSwitch.v:533]
ERROR: [VRFC 10-2063] Module <TopDeparser_t> not found while processing module instance <TopDeparser> [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/nf_sume_sdnet_ip/nf_sume_sdnet_ip/SimpleSumeS
witch/SimpleSumeSwitch.v:561]
# launch_simulation -simset sim_1 -mode behavioral
INFO: [Vivado 12-5698] Checking validity of IPs in the design for the 'XSim' simulator...
CRITICAL WARNING: [BD 41-1356] Address block </M04_AXI/Reg> is not mapped into </S00_AXI>. Please use Address Editor to either map or exclude it.
CRITICAL WARNING: [BD 41-1356] Address block </M05_AXI/Reg> is not mapped into </S00_AXI>. Please use Address Editor to either map or exclude it.
WARNING: [VRFC 10-756] identifier state is used before its declaration [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/axis_sim_record_ip0/hdl/axis_sim_record.v:93]
WARNING: [VRFC 10-756] identifier ready_count is used before its declaration [/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.srcs/sources_1/ip/axis_sim_record_ip0/hdl/axis_sim_record.v:94]
INFO: [#UNDEF] Sorry, too many errors..
ERROR: [XSIM 43-3322] Static elaboration of top level Verilog design unit(s) in library work failed.
INFO: [USF-XSim-69] 'elaborate' step finished in '1' seconds
INFO: [USF-XSim-99] Step results log file:'/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.sim/sim_1/behav/xsim/elaborate.log'
ERROR: [USF-XSim-62] 'elaborate' step failed with error(s). Please check the Tcl console output or '/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.sim/sim_1/behav/xsim/elaborate.log' file for more information.
nico@nsg-System:~/master-thesis$ find . -name elaborate.log
nico@nsg-System:~/master-thesis$ find ~ -name elaborate.log
nico@nsg-System:~/master-thesis$
- Scripts that "fail" (generate wrong data) do exit 0 ->
There is no easy / reliable error detection
- Writing tables resulted in ioctl errors
- Hardware test: unclear if first board was/is broken or not,
BUT: second board in different computer allows writing tables
- Many scripts depend on each other in later stages, without clear
dependencies
- There is basically no documentation for someone who "just wants to
compile from P4 to netpfga" or A LOT of documentation (if vivado,
vhld, sdnet documentation is counted)
- Very high complexity in toolchain, scripts that are generated
+ cd /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/test/sim_switch_default
+ make
rm -f config_writes.py*
rm -f *.pyc
nico@nsg-System:~$ cat /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.py
from NFTest import *
NUM_WRITES = 4
def config_tables():
nftest_regwrite(0x44020050, 0x22222208)
nftest_regwrite(0x44020054, 0x00000822)
nftest_regwrite(0x44020080, 0x00000201)
nftest_regwrite(0x44020040, 0x00000001)
nico@nsg-System:~$ cat /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/testdata/config_writes.sh
#!/bin/bash
${SUME_SDNET}/sw/sume/rwaxi -a 0x44020050 -w 0x22222208
${SUME_SDNET}/sw/sume/rwaxi -a 0x44020054 -w 0x00000822
${SUME_SDNET}/sw/sume/rwaxi -a 0x44020080 -w 0x00000201
${SUME_SDNET}/sw/sume/rwaxi -a 0x44020040 -w 0x00000001
nico@nsg-System:~$
- Misleading errors like
ERROR: [USF-XSim-62] 'elaborate' step failed with error(s). Please check the Tcl console output or '/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.sim/sim_1/behav/xsim/elaborate.log' file for more information.
nico@nsg-System:~/master-thesis/netpfga$ ls /home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.sim/sim_1/behav/xsim/elaborate.log
ls: cannot access '/home/nico/master-thesis/netpfga/minip4/simple_sume_switch/hw/project/simple_sume_switch.sim/sim_1/behav/xsim/elaborate.log': No such file or directory
- not using raise() and hiding source of errors (_hexify)
- sometimes flashing fails:
#+BEGIN_CENTER
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$ sudo bash -c ". $HOME/master-thesis/netpfga/bashinit && $(pwd -P)/program_switch.sh"
++ which vivado
+ xilinx_tool_path=/opt/Xilinx/Vivado/2018.2/bin/vivado
+ bitimage=minip4.bit
+ configWrites=config_writes.sh
+ '[' -z minip4.bit ']'
+ '[' -z config_writes.sh ']'
+ '[' /opt/Xilinx/Vivado/2018.2/bin/vivado == '' ']'
+ rmmod sume_riffa
+ xsct /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl -tclargs minip4.bit
rlwrap: warning: your $TERM is 'screen' but rlwrap couldn't find it in the terminfo database. Expect some problems.
RUN loading image file.
minip4.bit
100% 19MB 1.7MB/s 00:11
fpga configuration failed. DONE PIN is not HIGH
invoked from within
"::tcf::eval -progress ::xsdb::print_progress {::tcf::cache_enter tcfchan#0 {tcf_cache_eval {process_tcf_actions_cache_client ::tcfclient#0::arg}}}"
(procedure "::tcf::cache_eval_with_progress" line 2)
invoked from within
"::tcf::cache_eval_with_progress [dict get $arg chan] [list process_tcf_actions_cache_client $argvar] $progress"
(procedure "process_tcf_actions" line 1)
invoked from within
"process_tcf_actions $arg ::xsdb::print_progress"
(procedure "fpga" line 430)
invoked from within
"fpga -f $bitimage"
(file "/home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/run_xsct.tcl" line 33)
+ bash /home/nico/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/tools/pci_rescan_run.sh
Check programming FPGA or Reboot machine !
+ rmmod sume_riffa
rmmod: ERROR: Module sume_riffa is not currently loaded
+ modprobe sume_riffa
+ ifconfig nf0 up
nf0: ERROR while getting interface flags: No such device
+ ifconfig nf1 up
nf1: ERROR while getting interface flags: No such device
+ ifconfig nf2 up
nf2: ERROR while getting interface flags: No such device
+ ifconfig nf3 up
nf3: ERROR while getting interface flags: No such device
+ bash config_writes.sh
nico@nsg-System:~/projects/P4-NetFPGA/contrib-projects/sume-sdnet-switch/projects/minip4/simple_sume_switch/bitfiles$
#+END_CENTER
\end{verbatim}
\section{\label{conclusion:realworld}Real world applications}
Can be deployed using the netpfga. Or Barefoot or Arista.
\section{\label{conclusion:outlook}Outlook}
%** Outlook.tex: What needs to be done further, what is planed
%
What are the consequences of your work for future work?
Different HW
Speed only limited to line speed. Could be running at 100 Gbit/s
without modifications.
PMTU
handling error cases
Our algorithm uses the IPv4-Compatible IPv6 Address\cite{rfc4291} to
embed IPv4 addresses. However RFC6052\cite{rfc6052} defines different
embeddings depending on the prefix size. A future version should
support these schemes to be compatible to other implementations.
No fragmentation
No address / mac learning
**** No DNS64
has already been solved in a different domain - could even do
transparent / in network modification
**** Incomplete NDP
Very limited option support
No resolution of hardware addresses
\section{\label{conclusion:closing}Closing words (NAME?)}
While the port to NetPFGA was significantly more effort then expected,
the learnings of the different layers were very much appreciated / liked
It was a
\section{todo - FIXME: remove}
\begin{verbatim}
***** Summary eher kurz
***** Outlook als subsection!
\end{verbatim}