The network base - experimental We want to have 1 "main" network for convience. We want to be able to create networks automatically, once a new customer is created -> need hooks! Mapping: - each network is a "virtual" network. We use vxlan by default, but could be any technology! - we need a counter for vxlan mappings / network IDs -> cannot use Model in etcd: /v1/networks/ Tests see https://vincent.bernat.ch/en/blog/2017-vxlan-linux # local 2001:db8:1::1 \ netid=100 dev=wlp2s0 dev=wlp0s20f3 ip -6 link add vxlan${netid} type vxlan \ id ${netid} \ dstport 4789 \ group ff05::${netid} \ dev ${dev} \ ttl 5 [root@diamond ~]# ip addr add 2a0a:e5c0:5::1/48 dev vxlan100 root@manager:~/.ssh# ip addr add 2a0a:e5c0:5::2/48 dev vxlan100 root@manager:~/.ssh# ping -c3 2a0a:e5c0:5::1 PING 2a0a:e5c0:5::1(2a0a:e5c0:5::1) 56 data bytes 64 bytes from 2a0a:e5c0:5::1: icmp_seq=1 ttl=64 time=15.6 ms 64 bytes from 2a0a:e5c0:5::1: icmp_seq=2 ttl=64 time=30.3 ms 64 bytes from 2a0a:e5c0:5::1: icmp_seq=3 ttl=64 time=84.4 ms --- 2a0a:e5c0:5::1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 15.580/43.437/84.417/29.594 ms --> work even via wifi -------------------------------------------------------------------------------- Creating a network: 1) part of the initialisation / demo data (?) We should probably provide some demo sets that can easily be used. 2) manual/hook based request - hosts might have different network interfaces (?) -> this will make things very tricky -> don't support it - endpoint needs only support -------------------------------------------------------------------------------- IPAM IP address management (IPAM) is related to networks, but needs to be decoupled to allow pure L2 networks. From a customer point of view, we probably want to do something like: - ORDERING an IPv6 network can include creating a virtual network and an IPAM service Maybe "orders" should always be the first class citizen and ucloud internally "hooks" or binds things together. -------------------------------------------------------------------------------- testing / hacking: - starting etcd as storage [18:07] diamond:~% etcdctl put /v1/network/200 "{ some_network }" OK [18:08] diamond:~% etcdctl watch -w=json --prefix /v1/network {"Header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":6,"raft_term":2},"Events":[{"kv":{"key":"L3YxL25ldHdvcmsvMjAw","create_revision":5,"mod_revision":6,"version":2,"value":"eyBzb21lX25ldHdvcmsgfQ=="}}],"CompactRevision":0,"Canceled":false,"Created":false} -------------------------------------------------------------------------------- Flow for using and creating networks: - a network is created -> entry in etcd is created -> we need to keep a counter/lock so that 2 processes don't create the same network [Ahmed] -> nothing to be done on the hosts - a VM using a network is created - a VM using a network is scheduled to some host - the local "spawn a VM" process needs to check whether there is a vxlan interface existing -> if no, create it before creating the VM. -> if no, also create the bridge -> possibly adjusting the MTU (??) -> both names should be in hexadecimal (i.e. brff01 or vxlanff01) --> this way they are consistent with the multicast ipv6 address --> attention, ip -6 link ... id XXX expects DECIMAL input -------------------------------------------------------------------------------- If we also supply IPAM: - ipam needs to be created *after* the network is created - ipam is likely to be coupled to netbox (?) --> we need a "get next /64 prefix" function - when an ipam service is created in etcd, we need to create a new radvd instance on all routers (this will be a different service on BSDs) - we will need to create a new vxlan device on the routers - we need to create a new / modify radvd.conf - only after all of the routers reloaded radvd the ipam service is available! -------------------------------------------------------------------------------- If the user requests an IPv4 VM: - we need to get the next free IPv4 address (again, netbox?) - we need to create a mapping entry on the routers for NAT64 --> this requires the VM to be in a network with IPAM --> we always assume that the VM embeds itself using EUI64 -------------------------------------------------------------------------------- mac address handling! Example -------------------------------------------------------------------------------- TODOs - create-vxlan-on-dev.sh -> the multicast group needs to be ff05:: +int(vxlan_id) -------------------------------------------------------------------------------- Python hints: >>> vxlan_id = 3400 >>> b = ipaddress.IPv6Network("ff05::/16") >>> b[vxlan_id] IPv6Address('ff05::d48') we need / should assign hex values for vxlan ids in etcd! --> easier to read >>> b[0x3400] IPv6Address('ff05::3400') -------------------------------------------------------------------------------- Bridge names are limited to 15 characters Maximum/highest number of vxlan: >>> 2**24 16777216 >>> (2**25)-1 33554431 >>> b[33554431] IPv6Address('ff05::1ff:ffff') Last interface: br1ffffff vxlan1ffffff root@manager:~/ucloud/network# ip -6 link add vxlan1ffffff type vxlan id 33554431 dstport 4789 group ff05::1ff:ffff dev wlp2s0 ttl 5 Error: argument "33554431" is wrong: invalid id root@manager:~/ucloud/network# ip -6 link add vxlanffffff type vxlan id 16777215 dstport 4789 group ff05::ff:ffff dev wlp2s0 ttl 5 # id needs to be decimal root@manager:~# ip -6 link add vxlanff01 type vxlan id ff01 dstport 4789 group ff05::ff01 dev ttl 5 Error: argument "ff01" is wrong: invalid id root@manager:~# ip -6 link add vxlanff01 type vxlan id 65281 dstport 4789 group ff05::ff01 dev wlp2s0 ttl 5