forked from uncloud/uncloud
195 lines
5.7 KiB
Text
195 lines
5.7 KiB
Text
The network base - experimental
|
|
|
|
|
|
We want to have 1 "main" network for convience.
|
|
|
|
We want to be able to create networks automatically, once a new
|
|
customer is created -> need hooks!
|
|
|
|
|
|
Mapping:
|
|
|
|
- each network is a "virtual" network. We use vxlan by default, but
|
|
could be any technology!
|
|
- we need a counter for vxlan mappings / network IDs -> cannot use
|
|
|
|
Model in etcd:
|
|
|
|
/v1/networks/
|
|
|
|
|
|
Tests
|
|
see
|
|
https://vincent.bernat.ch/en/blog/2017-vxlan-linux
|
|
|
|
|
|
# local 2001:db8:1::1 \
|
|
|
|
|
|
netid=100
|
|
dev=wlp2s0
|
|
dev=wlp0s20f3
|
|
ip -6 link add vxlan${netid} type vxlan \
|
|
id ${netid} \
|
|
dstport 4789 \
|
|
group ff05::${netid} \
|
|
dev ${dev} \
|
|
ttl 5
|
|
|
|
[root@diamond ~]# ip addr add 2a0a:e5c0:5::1/48 dev vxlan100
|
|
root@manager:~/.ssh# ip addr add 2a0a:e5c0:5::2/48 dev vxlan100
|
|
root@manager:~/.ssh# ping -c3 2a0a:e5c0:5::1
|
|
PING 2a0a:e5c0:5::1(2a0a:e5c0:5::1) 56 data bytes
|
|
64 bytes from 2a0a:e5c0:5::1: icmp_seq=1 ttl=64 time=15.6 ms
|
|
64 bytes from 2a0a:e5c0:5::1: icmp_seq=2 ttl=64 time=30.3 ms
|
|
64 bytes from 2a0a:e5c0:5::1: icmp_seq=3 ttl=64 time=84.4 ms
|
|
|
|
--- 2a0a:e5c0:5::1 ping statistics ---
|
|
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
|
|
rtt min/avg/max/mdev = 15.580/43.437/84.417/29.594 ms
|
|
|
|
--> work even via wifi
|
|
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
Creating a network:
|
|
|
|
1) part of the initialisation / demo data (?)
|
|
|
|
We should probably provide some demo sets that can easily be used.
|
|
|
|
2) manual/hook based request
|
|
|
|
- hosts might have different network interfaces (?)
|
|
-> this will make things very tricky -> don't support it
|
|
- endpoint needs only support
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
IPAM
|
|
|
|
IP address management (IPAM) is related to networks, but needs to be
|
|
decoupled to allow pure L2 networks.
|
|
|
|
From a customer point of view, we probably want to do something like:
|
|
|
|
- ORDERING an IPv6 network can include creating a virtual network and
|
|
an IPAM service
|
|
|
|
Maybe "orders" should always be the first class citizen and ucloud
|
|
internally "hooks" or binds things together.
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
testing / hacking:
|
|
|
|
- starting etcd as storage
|
|
|
|
|
|
[18:07] diamond:~% etcdctl put /v1/network/200 "{ some_network }"
|
|
OK
|
|
[18:08] diamond:~% etcdctl watch -w=json --prefix /v1/network
|
|
{"Header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":6,"raft_term":2},"Events":[{"kv":{"key":"L3YxL25ldHdvcmsvMjAw","create_revision":5,"mod_revision":6,"version":2,"value":"eyBzb21lX25ldHdvcmsgfQ=="}}],"CompactRevision":0,"Canceled":false,"Created":false}
|
|
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
Flow for using and creating networks:
|
|
|
|
- a network is created -> entry in etcd is created
|
|
-> we need to keep a counter/lock so that 2 processes don't create
|
|
the same network [Ahmed]
|
|
-> nothing to be done on the hosts
|
|
- a VM using a network is created
|
|
- a VM using a network is scheduled to some host
|
|
- the local "spawn a VM" process needs to check whether there is a
|
|
vxlan interface existing -> if no, create it before creating the VM.
|
|
-> if no, also create the bridge
|
|
-> possibly adjusting the MTU (??)
|
|
-> both names should be in hexadecimal (i.e. brff01 or vxlanff01)
|
|
--> this way they are consistent with the multicast ipv6 address
|
|
--> attention, ip -6 link ... id XXX expects DECIMAL input
|
|
|
|
--------------------------------------------------------------------------------
|
|
If we also supply IPAM:
|
|
|
|
- ipam needs to be created *after* the network is created
|
|
- ipam is likely to be coupled to netbox (?)
|
|
--> we need a "get next /64 prefix" function
|
|
- when an ipam service is created in etcd, we need to create a new
|
|
radvd instance on all routers (this will be a different service on
|
|
BSDs)
|
|
- we will need to create a new vxlan device on the routers
|
|
- we need to create a new / modify radvd.conf
|
|
- only after all of the routers reloaded radvd the ipam service is
|
|
available!
|
|
|
|
|
|
--------------------------------------------------------------------------------
|
|
If the user requests an IPv4 VM:
|
|
|
|
- we need to get the next free IPv4 address (again, netbox?)
|
|
- we need to create a mapping entry on the routers for NAT64
|
|
--> this requires the VM to be in a network with IPAM
|
|
--> we always assume that the VM embeds itself using EUI64
|
|
|
|
|
|
|
|
--------------------------------------------------------------------------------
|
|
mac address handling!
|
|
|
|
Example
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
TODOs
|
|
|
|
- create-vxlan-on-dev.sh -> the multicast group
|
|
needs to be ff05:: +int(vxlan_id)
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
Python hints:
|
|
|
|
>>> vxlan_id = 3400
|
|
>>> b = ipaddress.IPv6Network("ff05::/16")
|
|
>>> b[vxlan_id]
|
|
IPv6Address('ff05::d48')
|
|
|
|
we need / should assign hex values for vxlan ids in etcd!
|
|
--> easier to read
|
|
|
|
>>> b[0x3400]
|
|
IPv6Address('ff05::3400')
|
|
|
|
|
|
--------------------------------------------------------------------------------
|
|
|
|
Bridge names are limited to 15 characters
|
|
|
|
|
|
Maximum/highest number of vxlan:
|
|
|
|
>>> 2**24
|
|
16777216
|
|
>>> (2**25)-1
|
|
33554431
|
|
|
|
>>> b[33554431]
|
|
IPv6Address('ff05::1ff:ffff')
|
|
|
|
Last interface:
|
|
br1ffffff
|
|
vxlan1ffffff
|
|
|
|
root@manager:~/ucloud/network# ip -6 link add vxlan1ffffff type vxlan id 33554431 dstport 4789 group ff05::1ff:ffff dev wlp2s0 ttl 5
|
|
Error: argument "33554431" is wrong: invalid id
|
|
|
|
root@manager:~/ucloud/network# ip -6 link add vxlanffffff type vxlan id 16777215 dstport 4789 group ff05::ff:ffff dev wlp2s0 ttl 5
|
|
|
|
|
|
# id needs to be decimal
|
|
root@manager:~# ip -6 link add vxlanff01 type vxlan id ff01 dstport 4789 group ff05::ff01 dev ttl 5
|
|
Error: argument "ff01" is wrong: invalid id
|
|
root@manager:~# ip -6 link add vxlanff01 type vxlan id 65281 dstport 4789 group ff05::ff01 dev wlp2s0 ttl 5
|