Merge branch 'master' of code.ungleich.ch:uncloud/uncloud

doc/README-how-to-configure-remote-uncloud-clients.org

@@ -0,0 +1,28 @@
+* What is a remote uncloud client?
+** Systems that configure themselves for the use with uncloud
+** Examples are VMHosts, VPN Servers, cdist control server, etc.
+* Which access do these clients need?
+** They need read / write access to the database
+* Possible methods
+** Overview
+|                          | pros                                                      | cons                                 |
+| SSL based                | Once setup, can access all django parts natively, locally | X.509 infrastructure                 |
+| SSH -L tunnel            | All nodes can use [::1]:5432                              | SSH setup can be fragile             |
+| ssh djangohost manage.py | All DB ops locally                                        | Code is only executed on django host |
+| https + token            | Rest alike / consistent access                            | Code is only executed on django host |
+| from_django              | Everything is on the django host                          | main host can become bottleneck      |
+** remote vs. local Django code execution
+   - If manage.py is executed locally (= on the client), it can
+     check/modify local configs
+   - However local execution requires a pyvenv + packages + db access
+   - Local execution also *could* make use of postgresql notify for
+     triggering actions (which is quite neat)
+   - Remote execution (= on the primary django host) can acess the db
+     via unix socket
+   - However remote execution cannot check local state
+** from_django
+   - might reuse existing methods like celery
+   - reduces the amount of things to be installed on the client to
+     almost zero
+   - follows the opennebula model
+   - has a single point of failurebin

doc/README-identifiers.org

@@ -0,0 +1,29 @@
+* Identifiers
+** Problem description
+   Identifiers can be integers, strings or other objects. They should
+   be unique.
+** Approach 1: integers
+   Integers are somewhat easy to remember, but also include
+   predictable growth, which might allow access to guessed hacking
+   (obivously proper permissions should prevent this).
+** Approach 2: random uuids
+   UUIDs are 128 bit integers. Python supports uuid.uuid4() for random
+   uuids.
+** Approach 3: IPv6 addresses
+   uncloud heavily depends on IPv6 in the first place. uncloud could
+   use a /48 to identify all objects. Objects that have IPv6 addresses
+   on their own, don't need to draw from the system /48.
+*** Possible Subnetworks
+    Assuming uncloud uses a /48 to represent all resources.
+
+    | Network         | Name            | Description                                  |
+    |-----------------+-----------------+----------------------------------------------|
+    | 2001:db8::/48   | uncloud network | All identifiers drawn from here              |
+    | 2001:db8:1::/64 | VM network      | Every VM has an IPv6 address in this network |
+    | 2001:db8:2::/64 | Bill network    | Every bill has an IPv6 address               |
+    | 2001:db8:3::/64 | Order network   | Every order has an IPv6 address              |
+    | 2001:db8:5::/64 | Product network | Every product (?) has an IPv6 address        |
+    | 2001:db8:4::/64 | Disk network    | Every disk is identified                     |
+
+*** Tests
+    [15:47:37] black3.place6:~# rbd create -s 10G ssd/2a0a:e5c0:1::8

doc/README-object-relations.md

@@ -0,0 +1,82 @@
+## Introduction
+
+This article describes how models relate to each other and what the
+design ideas are. It is meant to prevent us from double implementing
+something or changing something that is already solved.
+
+
+## Products
+
+A product is something someone can order. We might have "low level"
+products that need to be composed (= higher degree of flexibility, but
+more amount of details necessary) and "composed products" that present
+some defaults or select other products automatically (f.i. a "dual
+stack VM" can be a VM + a disk + an IPv4 address).
+
+
+## Bills
+
+Bills represent active orders of a month. Bills can be shown during a
+month but only become definitive at the end of the month.
+
+## Orders
+
+When customer X order a (set) of product, it generates an order for billing
+purposes. The ordered products point to that order and register an Order Record
+at creation.
+
+Orders and Order Records are assumed immutable => they are used to generate
+bills and should not be mutated. If a product is updated (e.g. adding RAM to
+VM), a new order should be generated.
+
+The order MUST NOT be deleted when a product is deleted, as it is used for
+billing (including past bills).
+
+### Order record
+
+Used to store billing details of a product at creation: will stay there even if
+the product change (e.g. new pricing, updated) and act as some kind of archive.
+Used to generate bills.
+
+## Payment Methods
+
+Users/customers can register payment methods.
+
+## Sample flows / products
+
+### A VM snapshot
+
+A VM snapshot creates a snapshot of all disks attached to a VM to be
+able to rollback the VM to a previous state.
+
+Creating a VM snapshot (-product) creates a related order. Deleting a
+VMSnapshotproduct sets the order to deleted.
+
+### Object Storage
+
+(tbd by Balazs)
+
+### A "raw" VM
+
+(tbd by Ahmed)
+
+### An IPv6 only VM
+
+(tbd by Ahmed)
+
+### A dual stack VM
+
+(tbd by Ahmed)
+
+### A managed service (e.g. Matrix-as-a-Service)
+
+Customer orders service with:
+	* Service-specific configuration: e.g. domain name for matrix
+	* VM configuration:
+		- CPU
+		- Memory
+		- Disk (soon)
+
+It creates a new Order with two products/records:
+	* Service itself (= management)
+	* Underlying VM

doc/README-postgresql.org

@@ -0,0 +1,8 @@
+* uncloud clients access the data base from a variety of outside hosts
+* So the postgresql data base needs to be remotely accessible
+* Instead of exposing the tcp socket, we make postgresql bind to localhost via IPv6
+** ::1, port 5432
+* Then we remotely connect to the database server with ssh tunneling
+** ssh -L5432:localhost:5432 uncloud-database-host
+* Configuring your database for SSH based remote access
+** host    all             all             ::1/128                 trust

doc/README-products.md

@@ -0,0 +1,34 @@
+## Introduction
+
+This document describes how to create, modify or
+delete a product and use it.
+
+A product (like a VMSnapshotproduct) creates an order when ordered.
+The "order" is used to combine products together.
+
+Sub-products or related products link to the same order.
+Each product has one (?) orderrecord
+
+
+## How to delete a product (logic 1)
+
+If a user want so delete (=cancel) a product, the following steps
+should be taken:
+
+* the associated order is set to cancelled
+* the product itself is deleted
+
+[above steps to be reviewed]
+
+## How to delete a product (rest api)
+
+http -a nicoschottelius:$(pass
+ungleich.ch/nico.schottelius@ungleich.ch)
+http://localhost:8000/net/vpn/43c83088-f4d6-49b9-86c7-40251ac07ada/
+
+-> does not delete the reservation.
+
+
+### Deleting a VPN
+
+When the product is deleted, the network must be marked as free.

doc/README-vpn.org

@@ -0,0 +1,34 @@
+* How to add a new VPN Host
+** Install wireguard to the host
+** Install uncloud to the host
+** Add `python manage.py vpn --hostname fqdn-of-this-host` to the crontab
+** Use the CLI to configure one or more VPN Networks for this host
+* Example of adding a VPN host at ungleich
+** Create a new dual stack alpine VM
+** Add it to DNS as vpn-XXX.ungleich.ch
+** Route a /40 network to its IPv6 address
+** Install wireguard on it
+** TODO Enable wireguard on boot
+** TODO Create a new VPNPool on uncloud with
+*** the network address (selecting from our existing pool)
+*** the network size (/...)
+*** the vpn host that provides the network (selecting the created VM)
+*** the wireguard private key of the vpn host (using wg genkey)
+*** http command
+```
+http -a nicoschottelius:$(pass
+  ungleich.ch/nico.schottelius@ungleich.ch)
+  http://localhost:8000/admin/vpnpool/ network=2a0a:e5c1:200:: \
+  network_size=40 subnetwork_size=48
+  vpn_hostname=vpn-2a0ae5c1200.ungleich.ch
+  wireguard_private_key=...
+```
+* Example http commands / REST calls
+** creating a new vpn pool
+ http -a nicoschottelius:$(pass
+ ungleich.ch/nico.schottelius@ungleich.ch)
+ http://localhost:8000/admin/vpnpool/ network_size=40
+ subnetwork_size=48 network=2a0a:e5c1:200::
+ vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg
+ genkey)
+** Creating a new vpn network

doc/README.md

@@ -0,0 +1,135 @@
+## Install
+
+### OS package requirements
+
+Alpine:
+
+```
+apk add openldap-dev postgresql-dev
+```
+
+Debian/Devuan:
+
+```
+apt install postgresql-server-dev-all
+```
+
+
+### Python requirements
+
+If you prefer using a venv, use:
+
+```
+python -m venv venv
+. ./venv/bin/activate
+```
+
+Then install the requirements
+
+```
+pip install -r requirements.txt
+```
+
+### Database requirements
+
+Due to the use of the JSONField, postgresql is required.
+
+First create a role to be used:
+
+```
+postgres=# create role nico login;
+```
+
+Then create the database owner by the new role:
+
+```
+postgres=# create database uncloud owner nico;
+```
+
+Installing the postgresql service is os dependent, but some hints:
+
+* Alpine: `apk add postgresql-server && rc-update add postgresql && rc-service  postgresql start`
+* Debian/Devuan: `apt install postgresql`
+
+After postresql is started, apply the migrations:
+
+```
+python manage.py migrate
+```
+
+## Flows / Orders
+
+
+### Creating a VMHost
+
+### Creating a VM
+
+* Create a VMHost
+* Create a VM on a VMHost
+
+
+### Creating a VM Snapshot
+
+
+## Working Beta APIs
+
+These APIs can be used for internal testing.
+
+### URL Overview
+
+```
+http -a nicoschottelius:$(pass ungleich.ch/nico.schottelius@ungleich.ch) http://localhost:8000
+```
+
+### Snapshotting
+
+```
+http -a nicoschottelius:$(pass ungleich.ch/nico.schottelius@ungleich.ch) http://localhost:8000/vm/snapshot/  vm_uuid=$(uuidgen)
+```
+
+## VPNs
+
+VPNs consist of VPNPools ("networks of networks") which are handled by
+VPNHosts. Users can requests VPNs with specific sizes.
+
+VPNs support both IPv6 and IPv4. However only IPv6 support has not been
+
+### Managing VPNPools
+
+```
+http -a nico:$(pass ldap/nico) https://uncloud.place7.ungleich.ch/v1/admin/vpnpool/ network=2a0a:e5c1:200:: network_size=40 subnetwork_size=48 vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg genkey)
+```
+
+This will create the VPNPool 2a0a:e5c1:200::/40 from which /48
+networks will be used for clients.
+
+VPNPools can only be managed by staff.
+
+### Managing VPNNetworks
+
+
+To request a network as a client, use the following call:
+
+```
+http -a nicoschottelius:$(pass ungleich.ch/nico.schottelius@ungleich.ch)https://uncloud.place7.ungleich.ch/v1/net/vpn/ network_size=48 wireguard_public_key=$(wg genkey | tee privatekey | wg pubkey)
+```
+
+VPNNetworks can be managed by all authenticated users.
+
+
+## Proposed (uncoded) flows
+
+### Changing the disk size of a VM
+
+* GET on ../vm/vm/<uuid> should list uuids of disks
+* UPDATE on ../vm/disk/<uuid> with size=newsize
+  * Newsize > oldsize!
+* Triggers shutdown of VM
+* Resizes disk
+* Starts VM
+* Maybe confirm flag?
+
+
+### Adding a disk to a VM
+
+(TBD)