* Bootstrap / Installation ** Pre-requisites by operating system *** Alpine #+BEGIN_SRC sh apk add openldap-dev postgresql-dev libxml2-dev libxslt-dev #+END_SRC *** Debian/Devuan: #+BEGIN_SRC sh apt install postgresql-server-dev-all #+END_SRC ** Creating a virtual environment / installing python requirements *** Virtual env To separate uncloud requirements, you can use a python virtual env as follows: #+BEGIN_SRC sh python3 -m venv venv . ./venv/bin/activate #+END_SRC Then install the requirements #+BEGIN_SRC sh pip install -r requirements.txt #+END_SRC ** Setting up the the database *** Install the database service The database can run on the same host as uncloud, but can also run a different server. Consult the usual postgresql documentation for a secure configuration. **** Alpine #+BEGIN_SRC sh apk add postgresql-server rc-update add postgresql rc-service postgresql start` #+END_SRC **** Debian/Devuan: #+BEGIN_SRC sh apt install postgresql #+END_SRC *** Create the database Due to the use of the JSONField, postgresql is required. To get started, create a database and have it owned by the user that runs uncloud (usually "uncloud"): #+BEGIN_SRC sh bridge:~# su - postgres bridge:~$ psql postgres=# create role uncloud login; postgres=# create database uncloud owner nico; #+END_SRC *** Creating the schema #+BEGIN_SRC sh python manage.py migrate #+END_SRC ** Bootstrap - Login via a user so that the user object gets created - Run the following (replace nicocustomer with the username) #+BEGIN_SRC sh python manage.py bootstrap-user --username nicocustomer #+END_SRC * Testing / CLI Access Access via the commandline (CLI) can be done using curl or httpie. In our examples we will use httpie. ** Checkout out the API #+BEGIN_SRC sh http localhost:8000/api/ #+END_SRC ** Authenticate via ldap user in password store #+BEGIN_SRC sh http --auth nicocustomer:$(pass ldap/nicocustomer) localhost:8000/api/ #+END_SRC * Database ** uncloud clients access the data base from a variety of outside hosts ** So the postgresql data base needs to be remotely accessible ** Instead of exposing the tcp socket, we make postgresql bind to localhost via IPv6 *** ::1, port 5432 ** Then we remotely connect to the database server with ssh tunneling *** ssh -L5432:localhost:5432 uncloud-database-host ** Configuring your database for SSH based remote access *** host all all ::1/128 trust * URLs - api/ - the rest API * uncloud Products ** Product features - Dependencies on other products - Minimum parameters (min cpu, min ram, etc). - Can also realise the dcl vm - dualstack vm = VM + IPv4 + SSD - Need to have a non-misguiding name for the "bare VM" - Should support network boot (?) ** VPN *** How to add a new VPN Host **** Install wireguard to the host **** Install uncloud to the host **** Add `python manage.py vpn --hostname fqdn-of-this-host` to the crontab **** Use the CLI to configure one or more VPN Networks for this host *** Example of adding a VPN host at ungleich **** Create a new dual stack alpine VM **** Add it to DNS as vpn-XXX.ungleich.ch **** Route a /40 network to its IPv6 address **** Install wireguard on it **** TODO [#C] Enable wireguard on boot **** TODO [#C] Create a new VPNPool on uncloud with ***** the network address (selecting from our existing pool) ***** the network size (/...) ***** the vpn host that provides the network (selecting the created VM) ***** the wireguard private key of the vpn host (using wg genkey) ***** http command ``` http -a nicoschottelius:$(pass ungleich.ch/nico.schottelius@ungleich.ch) http://localhost:8000/admin/vpnpool/ network=2a0a:e5c1:200:: \ network_size=40 subnetwork_size=48 vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=... ``` *** Example http commands / REST calls **** creating a new vpn pool http -a nicoschottelius:$(pass ungleich.ch/nico.schottelius@ungleich.ch) http://localhost:8000/admin/vpnpool/ network_size=40 subnetwork_size=48 network=2a0a:e5c1:200:: vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg genkey) **** Creating a new vpn network *** Creating a VPN pool #+BEGIN_SRC sh http -a uncloudadmin:$(pass uncloudadmin) https://localhost:8000/v1/admin/vpnpool/ \ network=2a0a:e5c1:200:: network_size=40 subnetwork_size=48 \ vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg genkey) #+END_SRC This will create the VPNPool 2a0a:e5c1:200::/40 from which /48 networks will be used for clients. VPNPools can only be managed by staff. *** Managing VPNNetworks To request a network as a client, use the following call: #+BEGIN_SRC sh http -a user:$(pass user) https://localhost:8000/v1/net/vpn/ \ network_size=48 \ wireguard_public_key=$(wg genkey | tee privatekey | wg pubkey) ``` VPNNetworks can be managed by all authenticated users. * Developer Handbook The following section describe decisions / architecture of uncloud. These chapters are intended to be read by developers. ** Documentation This documentation is written in org-mode. To compile it to html/pdf, just open emacs and press *C-c C-e l p*. ** Models *** Bill Bills are summarising usage in a specific timeframe. Bills usually spawn one month. *** BillRecord Bill records are used to model the usage of one order during the timeframe. *** Order Orders register the intent of a user to buy something. They might refer to a product. (???) Order register the one time price and the recurring price. These fields should be treated as immutable. If they need to be modified, a new order that replaces the current order should be created. **** Replacing orders If an order is updated, a new order is created and points to the old order. The old order stops one second before the new order starts. If a order has been replaced can be seen by its replaced_by count: #+BEGIN_SRC sh >>> Order.objects.get(id=1).replaced_by.count() 1 #+END_SRC *** Product and Product Children - A product describes something a user can buy - A product inherits from the uncloud_pay.models.Product model to get basic attributes ** Identifiers *** Problem description Identifiers can be integers, strings or other objects. They should be unique. *** Approach 1: integers Integers are somewhat easy to remember, but also include predictable growth, which might allow access to guessed hacking (obivously proper permissions should prevent this). *** Approach 2: random uuids UUIDs are 128 bit integers. Python supports uuid.uuid4() for random uuids. *** Approach 3: IPv6 addresses uncloud heavily depends on IPv6 in the first place. uncloud could use a /48 to identify all objects. Objects that have IPv6 addresses on their own, don't need to draw from the system /48. **** Possible Subnetworks Assuming uncloud uses a /48 to represent all resources. | Network | Name | Description | |-----------------+-----------------+----------------------------------------------| | 2001:db8::/48 | uncloud network | All identifiers drawn from here | | 2001:db8:1::/64 | VM network | Every VM has an IPv6 address in this network | | 2001:db8:2::/64 | Bill network | Every bill has an IPv6 address | | 2001:db8:3::/64 | Order network | Every order has an IPv6 address | | 2001:db8:5::/64 | Product network | Every product (?) has an IPv6 address | | 2001:db8:4::/64 | Disk network | Every disk is identified | **** Tests [15:47:37] black3.place6:~# rbd create -s 10G ssd/2a0a:e5c0:1::8 *** Decision We use integers, because they are easy. ** Milestones :uncloud: *** 1.1 (cleanup 1) **** *** 1.0 (initial release) **** TODO Initial Generic product support - Product ***** TODO Recurring product support ****** TODO Support replacing orders for updates ****** DONE [#A] Finish split of bill creation CLOSED: [2020-09-11 Fri 23:19] ****** TODO Test the new functions in the Order class ****** Define the correct order replacement logic Assumption: - recurringperiods are 30days ******* Case 1: downgrading - User commits to 10 CHF for 30 days - Wants to downgrade after 15 days to 5 CHF product - Expected result: - order 1: 10 CHF until +30days - order 2: 5 CHF starting 30days + 1s - Sum of the two orders is 15 CHF - Question is - when is the VM shutdown? - a) instantly - b) at the end of the cycle - best solution - user can choose between a ... b any time ******* Duration - You cannot cancel the duration - You can upgrade and with that cancel the duration - The idea of a duration is that you commit for it - If you want to commit lower (daily basis for instance) you have higher per period prices ******* Case X - User has VM with 2 Core / 2 GB RAM - User modifies with to 1 core / 3 GB RAM - We treat it as down/upgrade independent of the modifications ******* Case 2: upgrading after 1 day - committed for 30 days - upgrade after 1 day - so first order will be charged for 1/30ths ******* Case 2: upgrading - User commits to 10 CHF for 30 days - Wants to upgrade after 15 days to 20 CHF product - Order 1 : 1 VM with 2 Core / 2 GB / 10 SSD -- 10 CHF - 30days period, stopped after 15, so quantity is 0.5 = 5 CHF - Order 2 : 1 VM with 2 Core / 6 GB / 10 SSD -- 20 CHF - after 15 days - VM is upgraded instantly - Expected result: - order 1: 10 CHF until +15days = 0.5 units = 5 CHF - order 2: 20 CHF starting 15days + 1s ... +30 days after the 15 days -> 45 days = 1 unit = 20 CHF - Total on bill: 25 CHF ******* Case 2: upgrading - User commits to 10 CHF for 30 days - Wants to upgrade after 15 days to 20 CHF product - Expected result: - order 1: 10 CHF until +30days = 1 units = 10 CHF - order 2: 20 CHF starting 15days + 1s = 1 unit = 20 CHF - Total on bill: 30 CHF ****** TODO Note: ending date not set if replaced by default (implicit!) - Should the new order modify the old order on save()? ****** DONE Fix totally wrong bill dates in our test case CLOSED: [2020-09-09 Wed 01:00] - 2020 used instead of 2019 - Was due to existing test data ... ***** TODO Bill logic is still wrong - Bill starting_date is the date of the first order - However first encountered order does not have to be the earliest in the bill! - Bills should not have a duration - Bills should only have a (unique) issue date - We charge based on bill_records - Last time charged issue date of the bill OR earliest date after that - Every bill generation checks all (relevant) orders - add a flag "not_for_billing" or "closed" - query on that flag - verify it every time ***** TODO Generating bill for admins/staff -