mravi
/
uncloud
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Compare commits

base: mravi:master
Branches Tags
mravi:master
mravi:test
mravi:user-billing-api-revamp
mravi:update-vm-ordering
mravi:uncloud-api-refactoring
mravi:0.8
mravi:0.7
mravi:0.6
mravi:0.5
mravi:0.4
mravi:0.3
mravi:0.2
mravi:0.1
mravi:0.1.0
mravi:0.0.8
mravi:0.0.7
mravi:0.0.6
mravi:0.0.5
mravi:0.0.4
mravi:0.0.3
mravi:0.0.2
mravi:0.0.1
..
compare: mravi:update-vm-ordering
Branches Tags
mravi:test
mravi:master
mravi:user-billing-api-revamp
mravi:update-vm-ordering
mravi:uncloud-api-refactoring
mravi:0.8
mravi:0.7
mravi:0.6
mravi:0.5
mravi:0.4
mravi:0.3
mravi:0.2
mravi:0.1
mravi:0.1.0
mravi:0.0.8
mravi:0.0.7
mravi:0.0.6
mravi:0.0.5
mravi:0.0.4
mravi:0.0.3
mravi:0.0.2
mravi:0.0.1

1 Commits
master ... update-vm-

Author SHA1 Message Date
fnux 9410b7c56b Make VM order-able again 2020-04-18 13:51:31 +02:00
325 changed files with 3400 additions and 5640 deletions
Show Stats Expand all files Collapse all files

1
.gitignore vendored
View File

@ -24,4 +24,3 @@ venv/
dist/
*.iso
*.sqlite3

6
.gitlab-ci.yml
View File

@ -4,7 +4,7 @@ stages:
run-tests:
stage: test
image: code.ungleich.ch:5050/uncloud/uncloud/uncloud-ci:latest
image: fedora:latest
services:
- postgres:latest
variables:
@ -12,7 +12,11 @@ run-tests:
DATABASE_USER: postgres
POSTGRES_HOST_AUTH_METHOD: trust
coverage: /^TOTAL.+?(\d+\%)$/
before_script:
- dnf install -y python3-devel python3-pip python3-coverage libpq-devel openldap-devel gcc chromium
script:
- cd uncloud_django_based/uncloud
- pip install -r requirements.txt
- cp uncloud/secrets_sample.py uncloud/secrets.py
- coverage run --source='.' ./manage.py test
- coverage report

63
README.md
View File

@ -1,62 +1,3 @@
# Uncloud
# ucloud
Cloud management platform, the ungleich way.
[![pipeline status](https://code.ungleich.ch/uncloud/uncloud/badges/master/pipeline.svg)](https://code.ungleich.ch/uncloud/uncloud/commits/master)
[![coverage report](https://code.ungleich.ch/uncloud/uncloud/badges/master/coverage.svg)](https://code.ungleich.ch/uncloud/uncloud/commits/master)
## Useful commands
* `./manage.py import-vat-rates path/to/csv`
* `./manage.py make-admin username`
## Development setup
Install system dependencies:
* On Fedora, you will need the following packages: `python3-virtualenv python3-devel openldap-devel gcc chromium`
NOTE: you will need to configure a LDAP server and credentials for authentication. See `uncloud/settings.py`.
```
# Initialize virtualenv.
» virtualenv .venv
Using base prefix '/usr'
New python executable in /home/fnux/Workspace/ungleich/uncloud/uncloud/.venv/bin/python3
Also creating executable in /home/fnux/Workspace/ungleich/uncloud/uncloud/.venv/bin/python
Installing setuptools, pip, wheel...
done.
# Enter virtualenv.
» source .venv/bin/activate
# Install dependencies.
» pip install -r requirements.txt
[...]
# Run migrations.
» ./manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, opennebula, sessions, uncloud_auth, uncloud_net, uncloud_pay, uncloud_service, uncloud_vm
Running migrations:
[...]
# Run webserver.
» ./manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
May 07, 2020 - 10:17:08
Django version 3.0.6, using settings 'uncloud.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
```
### Note on PGSQL
If you want to use Postgres:
* Install on configure PGSQL on your base system.
* OR use a container! `podman run --rm -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust -it postgres:latest`
Checkout https://ungleich.ch/ucloud/ for the documentation of ucloud.

6
archive/issues.org
View File

@ -1,6 +0,0 @@
* Intro
This file lists issues that should be handled, are small and likely
not yet high prio.
* Issues
** TODO Register prefered address in User model
** TODO Allow to specify different recurring periods

18
archive/uncloud_django_based/hacks/command-wrapper.sh
View File

@ -1,18 +0,0 @@
#!/bin/sh
dbhost=$1; shift
ssh -L5432:localhost:5432 "$dbhost" &
python manage.py "$@"
# command only needs to be active while manage command is running
# -T no pseudo terminal
# alternatively: commands output shell code
# ssh uncloud@dbhost "python manage.py --hostname xxx ..."

64
archive/uncloud_etcd_based/uncloud/hack/hackcloud/nftrules-v2
View File

@ -1,64 +0,0 @@
flush ruleset
table bridge filter {
chain prerouting {
type filter hook prerouting priority 0;
policy accept;
ibrname br100 jump netpublic
}
chain netpublic {
iifname vxlan100 jump from_uncloud
# Default blocks: router advertisements, dhcpv6, dhcpv4
icmpv6 type nd-router-advert drop
ip6 version 6 udp sport 547 drop
ip version 4 udp sport 67 drop
# Individual blocks
# iifname tap1 jump vm1
}
chain vm1 {
ether saddr != 02:00:f0:a9:c4:4e drop
ip6 saddr != 2a0a:e5c1:111:888:0:f0ff:fea9:c44e drop
}
chain from_uncloud {
accept
}
}
# table ip6 filter {
# chain forward {
# type filter hook forward priority 0;
# # policy drop;
# ct state established,related accept;
# }
# }
# table ip filter {
# chain input {
# type filter hook input priority filter; policy drop;
# iif "lo" accept
# icmp type { echo-reply, destination-unreachable, source-quench, redirect, echo-request, router-advertisement, router-solicitation, time-exceeded, parameter-problem, timestamp-request, timestamp-reply, info-request, info-reply, address-mask-request, address-mask-reply } accept
# ct state established,related accept
# tcp dport { 22 } accept
# log prefix "firewall-ipv4: "
# udp sport 67 drop
# }
# chain forward {
# type filter hook forward priority filter; policy drop;
# log prefix "firewall-ipv4: "
# }
# chain output {
# type filter hook output priority filter; policy accept;
# }
# }

24
archive/uncloud_etcd_based/uncloud/hack/hackcloud/vm-2.sh
View File

@ -1,24 +0,0 @@
#!/bin/sh
vmid=$1; shift
qemu=/usr/bin/qemu-system-x86_64
accel=kvm
#accel=tcg
memory=1024
cores=2
uuid=732e08c7-84f8-4d43-9571-263db4f80080
export bridge=br100
$qemu -name uc${vmid} \
-machine pc,accel=${accel} \
-m ${memory} \
-smp ${cores} \
-uuid ${uuid} \
-drive file=alpine-virt-3.11.2-x86_64.iso,media=cdrom \
-drive file=alpine-virt-3.11.2-x86_64.iso,media=cdrom \
-netdev tap,id=netmain,script=./ifup.sh \
-device virtio-net-pci,netdev=netmain,id=net0,mac=02:00:f0:a9:c4:4e

7
bin/make-migrations-from-scratch.sh
View File

@ -1,7 +0,0 @@
#!/bin/sh
# For undoing/redoing everything
# Needed in special cases and needs to be avoided as soon as
# uncloud.version >= 1
for a in */migrations; do rm ${a}/*.py; done
for a in */migrations; do python manage.py makemigrations ${a%%/migrations}; done

2
doc/.gitignore vendored
View File

@ -1,2 +0,0 @@
*.pdf
*.tex

85
doc/README-billing.org
View File

@ -1,85 +0,0 @@
* How to handle billing in general
** Manual test flow / setting up bills
- Needs orders
-
** Orders
- Orders are the heart of uncloud billing
- Have a starting date
- Have an ending date
- Orders are immutable
- Can usually not be cancelled / cancellation is not a refund
- Customer/user commits on a certain period -> gets discount
based on it
- Can be upgraded
- Create a new order
- We link the new order to the old order and say this one
replaces it
- If the price of the new order is HIGHER than the OLD order,
then we charge the difference until the end of the order period
- In the next billing run we set the OLD order to not to bill anymore
- And only the NEW order will be billed afterwards
- Can be downgraded in the next period (but not for this period)
- We create a new order, same as for upgrade
- The new order starts directly after the OLD order
- As the amount is LOWER than the OLD order, no additional charge is done
during this order period
- We might need to have an activate datetime
- When to implement this
- Order periods can be
*** Statuses
- CREATING/PREPARING
- INACTIVE (?)
- TO_BILL
- NOT_TO_BILL: we use this to accelerate queries to the DB
*** Updating status of orders
- If has succeeding order and billing date is last month -> set inactive
** Bills
- Are always for a month
- Can be preliminary
*** Which orders to include
- Not the cancelled ones / not active ones
** Flows / Approach
*** Finding all orders for a bill
- Get all orders, state != NOT_TO_BILL; for each order do:
- is it a one time order?
- has it a bill assigned?
- yes: set to NOT_TO_BILL
- no:
- get_or_create_bill_for_this_month
- assign bill to this order
- set to NOT_TO_BILL
- is it a recurring order?
- if it has a REPLACING order:
-
- First of month
- Last of month
*** Handling replacement of orders
- The OLD order will appear in the month that it was cancelled on
the bill
- The OLD order needs to be set to NOT_TO_BILL after it was billed
the last time
- The NEW order will be added pro rata if the amount is higher in
the same month
- The NEW order will be used next month
**** Disabling the old order
- On billing run
- If order.replacement_order (naming!) is set
- if the order.replacement_order starts during THIS_MONTH
- add order to bill
- if NOT:
- the order was already replaced in a previous billing period
- set the order to NOT_TO_BILL
**** Billing the new order
- If order.previous_order
*** Handling multiple times a recurring order
- For each recurring order check the order.period
- Find out when it was billed last
- lookup latest bill
- Calculate how many times it has been used until 2359, last day
of month
- For preliminary bill: until datetime.now()
- Call the bill_end_datetime
- Getting duration: bill_end_datetime - order.last_billed
- Amount in seconds; duration_in_seconds
- Divide duration_in_seconds by order.period; amount_used:
- If >= 1: add amount_used * order.recurring_amount to bill

382
doc/uncloud-manual-2020-08-01.org
View File

@ -1,382 +0,0 @@
* Bootstrap / Installation
** Pre-requisites by operating system
*** General
To run uncloud you need:
- ldap development libraries
- libxml2-dev libxslt-dev
- gcc / libc headers: for compiling things
- python3-dev
- wireguard: wg (for checking keys)
*** Alpine
#+BEGIN_SRC sh
apk add openldap-dev postgresql-dev libxml2-dev libxslt-dev gcc python3-dev musl-dev wireguard-tools-wg
#+END_SRC
*** Debian/Devuan:
#+BEGIN_SRC sh
apt install postgresql-server-dev-all
#+END_SRC
** Creating a virtual environment / installing python requirements
*** Virtual env
To separate uncloud requirements, you can use a python virtual
env as follows:
#+BEGIN_SRC sh
python3 -m venv venv
. ./venv/bin/activate
#+END_SRC
Then install the requirements
#+BEGIN_SRC sh
pip install -r requirements.txt
#+END_SRC
** Setting up the the database
*** Install the database service
The database can run on the same host as uncloud, but can also run
a different server. Consult the usual postgresql documentation for
a secure configuration.
**** Alpine
#+BEGIN_SRC sh
apk add postgresql-server
rc-update add postgresql
rc-service postgresql start`
#+END_SRC
**** Debian/Devuan:
#+BEGIN_SRC sh
apt install postgresql
#+END_SRC
*** Create the database
Due to the use of the JSONField, postgresql is required.
To get started,
create a database and have it owned by the user that runs uncloud
(usually "uncloud"):
#+BEGIN_SRC sh
bridge:~# su - postgres
bridge:~$ psql
postgres=# create role uncloud login;
postgres=# create database uncloud owner nico;
#+END_SRC
*** Creating the schema
#+BEGIN_SRC sh
python manage.py migrate
#+END_SRC
** Bootstrap
- Login via a user so that the user object gets created
- Run the following (replace nicocustomer with the username)
#+BEGIN_SRC sh
python manage.py bootstrap-user --username nicocustomer
#+END_SRC
** Initialise the database
While it is not strictly required to add default values to the
database, it might significantly reduce the starting time with
uncloud.
To add the default database values run:
#+BEGIN_SRC shell
# Add local objects
python manage.py db-add-defaults
# Import VAT rates
python manage.py import-vat-rates
#+END_SRC
* Testing / CLI Access
Access via the commandline (CLI) can be done using curl or
httpie. In our examples we will use httpie.
** Checkout out the API
#+BEGIN_SRC sh
http localhost:8000/api/
#+END_SRC
** Authenticate via ldap user in password store
#+BEGIN_SRC sh
http --auth nicocustomer:$(pass ldap/nicocustomer) localhost:8000/api/
#+END_SRC
* Database
** uncloud clients access the data base from a variety of outside hosts
** So the postgresql data base needs to be remotely accessible
** Instead of exposing the tcp socket, we make postgresql bind to localhost via IPv6
*** ::1, port 5432
** Then we remotely connect to the database server with ssh tunneling
*** ssh -L5432:localhost:5432 uncloud-database-host
** Configuring your database for SSH based remote access
*** host all all ::1/128 trust
* URLs
- api/ - the rest API
* uncloud Products
** Product features
- Dependencies on other products
- Minimum parameters (min cpu, min ram, etc).
- Can also realise the dcl vm
- dualstack vm = VM + IPv4 + SSD
- Need to have a non-misguiding name for the "bare VM"
- Should support network boot (?)
** VPN
*** How to add a new VPN Host
**** Install wireguard to the host
**** Install uncloud to the host
**** Add `python manage.py vpn --hostname fqdn-of-this-host` to the crontab
**** Use the CLI to configure one or more VPN Networks for this host
*** Example of adding a VPN host at ungleich
**** Create a new dual stack alpine VM
**** Add it to DNS as vpn-XXX.ungleich.ch
**** Route a /40 network to its IPv6 address
**** Install wireguard on it
**** TODO [#C] Enable wireguard on boot
**** TODO [#C] Create a new VPNPool on uncloud with
***** the network address (selecting from our existing pool)
***** the network size (/...)
***** the vpn host that provides the network (selecting the created VM)
***** the wireguard private key of the vpn host (using wg genkey)
***** http command
```
http -a nicoschottelius:$(pass
ungleich.ch/nico.schottelius@ungleich.ch)
http://localhost:8000/admin/vpnpool/ network=2a0a:e5c1:200:: \
network_size=40 subnetwork_size=48
vpn_hostname=vpn-2a0ae5c1200.ungleich.ch
wireguard_private_key=...
```
*** Example http commands / REST calls
**** creating a new vpn pool
http -a nicoschottelius:$(pass
ungleich.ch/nico.schottelius@ungleich.ch)
http://localhost:8000/admin/vpnpool/ network_size=40
subnetwork_size=48 network=2a0a:e5c1:200::
vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg
genkey)
**** Creating a new vpn network
*** Creating a VPN pool
#+BEGIN_SRC sh
http -a uncloudadmin:$(pass uncloudadmin) https://localhost:8000/v1/admin/vpnpool/ \
network=2a0a:e5c1:200:: network_size=40 subnetwork_size=48 \
vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg genkey)
#+END_SRC
This will create the VPNPool 2a0a:e5c1:200::/40 from which /48
networks will be used for clients.
VPNPools can only be managed by staff.
*** Managing VPNNetworks
To request a network as a client, use the following call:
#+BEGIN_SRC sh
http -a user:$(pass user) https://localhost:8000/v1/net/vpn/ \
network_size=48 \
wireguard_public_key=$(wg genkey | tee privatekey | wg pubkey)
```
VPNNetworks can be managed by all authenticated users.
* Developer Handbook
The following section describe decisions / architecture of
uncloud. These chapters are intended to be read by developers.
** This Documentation
This documentation is written in org-mode. To compile it to
html/pdf, just open emacs and press *C-c C-e l p*.
** Models
*** Bill
Bills are summarising usage in a specific timeframe. Bills usually
spawn one month.
*** BillRecord
Bill records are used to model the usage of one order during the
timeframe.
*** Order
Orders register the intent of a user to buy something. They might
refer to a product. (???)
Order register the one time price and the recurring price. These
fields should be treated as immutable. If they need to be modified,
a new order that replaces the current order should be created.
**** Replacing orders
If an order is updated, a new order is created and points to the
old order. The old order stops one second before the new order
starts.
If a order has been replaced can be seen by its replaced_by count:
#+BEGIN_SRC sh
>>> Order.objects.get(id=1).replaced_by.count()
1
#+END_SRC
*** Product and Product Children
- A product describes something a user can buy
- A product inherits from the uncloud_pay.models.Product model to
get basic attributes
** Identifiers
*** Problem description
Identifiers can be integers, strings or other objects. They should
be unique.
*** Approach 1: integers
Integers are somewhat easy to remember, but also include
predictable growth, which might allow access to guessed hacking
(obivously proper permissions should prevent this).
*** Approach 2: random uuids
UUIDs are 128 bit integers. Python supports uuid.uuid4() for random
uuids.
*** Approach 3: IPv6 addresses
uncloud heavily depends on IPv6 in the first place. uncloud could
use a /48 to identify all objects. Objects that have IPv6 addresses
on their own, don't need to draw from the system /48.
**** Possible Subnetworks
Assuming uncloud uses a /48 to represent all resources.
| Network | Name | Description |
|-----------------+-----------------+----------------------------------------------|
| 2001:db8::/48 | uncloud network | All identifiers drawn from here |
| 2001:db8:1::/64 | VM network | Every VM has an IPv6 address in this network |
| 2001:db8:2::/64 | Bill network | Every bill has an IPv6 address |
| 2001:db8:3::/64 | Order network | Every order has an IPv6 address |
| 2001:db8:5::/64 | Product network | Every product (?) has an IPv6 address |
| 2001:db8:4::/64 | Disk network | Every disk is identified |
**** Tests
[15:47:37] black3.place6:~# rbd create -s 10G ssd/2a0a:e5c0:1::8
*** Decision
We use integers, because they are easy.
** Distributing/Dispatching/Orchestrating
*** Variant 1: using cdist
- The uncloud server can git commit things
- The uncloud server loads cdist and configures the server
- Advantages
- Fully integrated into normal flow
- Disadvantage
- web frontend has access to more data than it needs
- On compromise of the machine, more data leaks
- Some cdist usual delay
*** Variant 2: via celery
- The uncloud server dispatches via celery
- Every decentral node also runs celery/connects to the broker
- Summary brokers:
- If local only celery -> good to use redis - Broker
- If remote: probably better to use rabbitmq
- redis
- simpler
- rabbitmq
- more versatile
- made for remote connections
- quorom queues would be nice, but not clear if supported
- https://github.com/celery/py-amqp/issues/302
- https://github.com/celery/celery/issues/6067
- Cannot be installed on alpine Linux at the moment
- Advantage
- Very python / django integrated
- Rather instant
- Disadvantages
- Every decentral node needs to have the uncloud code available
- Decentral nodes *might* need to access the database
- Tasks can probably be written to work without that
(i.e. only strings/bytes)
**** log/tests
(venv) [19:54] vpn-2a0ae5c1200:~/uncloud$ celery -A uncloud -b redis://bridge.place7.ungleich.ch worker -n worker1@%h --logfile ~/celery.log -
Q vpn-2a0ae5c1200.ungleich.ch
*** Variant 3: dedicated cdist instance via message broker
- A separate VM/machine
- Has Checkout of ~/.cdist
- Has cdist checkout
- Tiny API for management
- Not directly web accessible
- "cdist" queue
** Milestones :uncloud:
*** 1.1 (cleanup 1)
**** TODO [#C] Unify ValidationError, FieldError - define proper Exception
- What do we use for model errors
*** 1.0 (initial release)
**** TODO [#C] Initial Generic product support
- Product
***** TODO [#C] Recurring product support
****** TODO [#C] Support replacing orders for updates
****** DONE [#A] Finish split of bill creation
CLOSED: [2020-09-11 Fri 23:19]
****** TODO [#C] Test the new functions in the Order class
****** Define the correct order replacement logic
Assumption:
- recurringperiods are 30days
******* Case 1: downgrading
- User commits to 10 CHF for 30 days
- Wants to downgrade after 15 days to 5 CHF product
- Expected result:
- order 1: 10 CHF until +30days
- order 2: 5 CHF starting 30days + 1s
- Sum of the two orders is 15 CHF
- Question is
- when is the VM shutdown?
- a) instantly
- b) at the end of the cycle
- best solution
- user can choose between a ... b any time
******* Duration
- You cannot cancel the duration
- You can upgrade and with that cancel the duration
- The idea of a duration is that you commit for it
- If you want to commit lower (daily basis for instance) you
have higher per period prices
******* Case X
- User has VM with 2 Core / 2 GB RAM
- User modifies with to 1 core / 3 GB RAM
- We treat it as down/upgrade independent of the modifications
******* Case 2: upgrading after 1 day
- committed for 30 days
- upgrade after 1 day
- so first order will be charged for 1/30ths
******* Case 2: upgrading
- User commits to 10 CHF for 30 days
- Wants to upgrade after 15 days to 20 CHF product
- Order 1 : 1 VM with 2 Core / 2 GB / 10 SSD -- 10 CHF
- 30days period, stopped after 15, so quantity is 0.5 = 5 CHF
- Order 2 : 1 VM with 2 Core / 6 GB / 10 SSD -- 20 CHF
- after 15 days
- VM is upgraded instantly
- Expected result:
- order 1: 10 CHF until +15days = 0.5 units = 5 CHF
- order 2: 20 CHF starting 15days + 1s ... +30 days after
the 15 days -> 45 days = 1 unit = 20 CHF
- Total on bill: 25 CHF
******* Case 2: upgrading
- User commits to 10 CHF for 30 days
- Wants to upgrade after 15 days to 20 CHF product
- Expected result:
- order 1: 10 CHF until +30days = 1 units = 10 CHF
- order 2: 20 CHF starting 15days + 1s = 1 unit = 20 CHF
- Total on bill: 30 CHF
****** TODO [#C] Note: ending date not set if replaced by default (implicit!)
- Should the new order modify the old order on save()?
****** DONE Fix totally wrong bill dates in our test case
CLOSED: [2020-09-09 Wed 01:00]
- 2020 used instead of 2019
- Was due to existing test data ...
***** DONE Bill logic is still wrong
CLOSED: [2020-11-05 Thu 18:58]
- Bill starting_date is the date of the first order
- However first encountered order does not have to be the
earliest in the bill!
- Bills should not have a duration
- Bills should only have a (unique) issue date
- We charge based on bill_records
- Last time charged issue date of the bill OR earliest date
after that
- Every bill generation checks all (relevant) orders
- add a flag "not_for_billing" or "closed"
- query on that flag
- verify it every time
***** TODO Generating bill for admins/staff
-

21
opennebula/migrations/0001_initial.py
View File

@ -1,21 +0,0 @@
# Generated by Django 3.1 on 2020-12-13 10:38
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='VM',
fields=[
('vmid', models.IntegerField(primary_key=True, serialize=False)),
('data', models.JSONField()),
],
),
]

16
opennebula/views.py
View File

@ -1,16 +0,0 @@
from rest_framework import viewsets, permissions
#from .models import VM
# from .serializers import OpenNebulaVMSerializer
# class VMViewSet(viewsets.ModelViewSet):
# permission_classes = [permissions.IsAuthenticated]
# serializer_class = OpenNebulaVMSerializer
# def get_queryset(self):
# if self.request.user.is_superuser:
# obj = VM.objects.all()
# else:
# obj = VM.objects.filter(owner=self.request.user)
# return obj

3
resources/ci/Dockerfile
View File

@ -1,3 +0,0 @@
FROM fedora:latest
RUN dnf install -y python3-devel python3-pip python3-coverage libpq-devel openldap-devel gcc chromium

2
uncloud/.gitignore vendored
View File

@ -1,2 +0,0 @@
local_settings.py
ldap_max_uid_file

254
uncloud/__init__.py
View File

@ -1,254 +0,0 @@
from django.utils.translation import gettext_lazy as _
import decimal
from .celery import app as celery_app
# Define DecimalField properties, used to represent amounts of money.
AMOUNT_MAX_DIGITS=10
AMOUNT_DECIMALS=2
decimal.getcontext().prec = AMOUNT_DECIMALS
# http://xml.coverpages.org/country3166.html
COUNTRIES = (
('AD', _('Andorra')),
('AE', _('United Arab Emirates')),
('AF', _('Afghanistan')),
('AG', _('Antigua & Barbuda')),
('AI', _('Anguilla')),
('AL', _('Albania')),
('AM', _('Armenia')),
('AN', _('Netherlands Antilles')),
('AO', _('Angola')),
('AQ', _('Antarctica')),
('AR', _('Argentina')),
('AS', _('American Samoa')),
('AT', _('Austria')),
('AU', _('Australia')),
('AW', _('Aruba')),
('AZ', _('Azerbaijan')),
('BA', _('Bosnia and Herzegovina')),
('BB', _('Barbados')),
('BD', _('Bangladesh')),
('BE', _('Belgium')),
('BF', _('Burkina Faso')),
('BG', _('Bulgaria')),
('BH', _('Bahrain')),
('BI', _('Burundi')),
('BJ', _('Benin')),
('BM', _('Bermuda')),
('BN', _('Brunei Darussalam')),
('BO', _('Bolivia')),
('BR', _('Brazil')),
('BS', _('Bahama')),
('BT', _('Bhutan')),
('BV', _('Bouvet Island')),
('BW', _('Botswana')),
('BY', _('Belarus')),
('BZ', _('Belize')),
('CA', _('Canada')),
('CC', _('Cocos (Keeling) Islands')),
('CF', _('Central African Republic')),
('CG', _('Congo')),
('CH', _('Switzerland')),
('CI', _('Ivory Coast')),
('CK', _('Cook Iislands')),
('CL', _('Chile')),
('CM', _('Cameroon')),
('CN', _('China')),
('CO', _('Colombia')),
('CR', _('Costa Rica')),
('CU', _('Cuba')),
('CV', _('Cape Verde')),
('CX', _('Christmas Island')),
('CY', _('Cyprus')),
('CZ', _('Czech Republic')),
('DE', _('Germany')),
('DJ', _('Djibouti')),
('DK', _('Denmark')),
('DM', _('Dominica')),
('DO', _('Dominican Republic')),
('DZ', _('Algeria')),
('EC', _('Ecuador')),
('EE', _('Estonia')),
('EG', _('Egypt')),
('EH', _('Western Sahara')),
('ER', _('Eritrea')),
('ES', _('Spain')),
('ET', _('Ethiopia')),
('FI', _('Finland')),
('FJ', _('Fiji')),
('FK', _('Falkland Islands (Malvinas)')),
('FM', _('Micronesia')),
('FO', _('Faroe Islands')),
('FR', _('France')),
('FX', _('France, Metropolitan')),
('GA', _('Gabon')),
('GB', _('United Kingdom (Great Britain)')),
('GD', _('Grenada')),
('GE', _('Georgia')),
('GF', _('French Guiana')),
('GH', _('Ghana')),
('GI', _('Gibraltar')),
('GL', _('Greenland')),
('GM', _('Gambia')),
('GN', _('Guinea')),
('GP', _('Guadeloupe')),
('GQ', _('Equatorial Guinea')),
('GR', _('Greece')),
('GS', _('South Georgia and the South Sandwich Islands')),
('GT', _('Guatemala')),
('GU', _('Guam')),
('GW', _('Guinea-Bissau')),
('GY', _('Guyana')),
('HK', _('Hong Kong')),
('HM', _('Heard & McDonald Islands')),
('HN', _('Honduras')),
('HR', _('Croatia')),
('HT', _('Haiti')),
('HU', _('Hungary')),
('ID', _('Indonesia')),
('IE', _('Ireland')),
('IL', _('Israel')),
('IN', _('India')),
('IO', _('British Indian Ocean Territory')),
('IQ', _('Iraq')),
('IR', _('Islamic Republic of Iran')),
('IS', _('Iceland')),
('IT', _('Italy')),
('JM', _('Jamaica')),
('JO', _('Jordan')),
('JP', _('Japan')),
('KE', _('Kenya')),
('KG', _('Kyrgyzstan')),
('KH', _('Cambodia')),
('KI', _('Kiribati')),
('KM', _('Comoros')),
('KN', _('St. Kitts and Nevis')),
('KP', _('Korea, Democratic People\'s Republic of')),
('KR', _('Korea, Republic of')),
('KW', _('Kuwait')),
('KY', _('Cayman Islands')),
('KZ', _('Kazakhstan')),
('LA', _('Lao People\'s Democratic Republic')),
('LB', _('Lebanon')),
('LC', _('Saint Lucia')),
('LI', _('Liechtenstein')),
('LK', _('Sri Lanka')),
('LR', _('Liberia')),
('LS', _('Lesotho')),
('LT', _('Lithuania')),
('LU', _('Luxembourg')),
('LV', _('Latvia')),
('LY', _('Libyan Arab Jamahiriya')),
('MA', _('Morocco')),
('MC', _('Monaco')),
('MD', _('Moldova, Republic of')),
('MG', _('Madagascar')),
('MH', _('Marshall Islands')),
('ML', _('Mali')),
('MN', _('Mongolia')),
('MM', _('Myanmar')),
('MO', _('Macau')),
('MP', _('Northern Mariana Islands')),
('MQ', _('Martinique')),
('MR', _('Mauritania')),
('MS', _('Monserrat')),
('MT', _('Malta')),
('MU', _('Mauritius')),
('MV', _('Maldives')),
('MW', _('Malawi')),
('MX', _('Mexico')),
('MY', _('Malaysia')),
('MZ', _('Mozambique')),
('NA', _('Namibia')),
('NC', _('New Caledonia')),
('NE', _('Niger')),
('NF', _('Norfolk Island')),
('NG', _('Nigeria')),
('NI', _('Nicaragua')),
('NL', _('Netherlands')),
('NO', _('Norway')),
('NP', _('Nepal')),
('NR', _('Nauru')),
('NU', _('Niue')),
('NZ', _('New Zealand')),
('OM', _('Oman')),
('PA', _('Panama')),
('PE', _('Peru')),
('PF', _('French Polynesia')),
('PG', _('Papua New Guinea')),
('PH', _('Philippines')),
('PK', _('Pakistan')),
('PL', _('Poland')),
('PM', _('St. Pierre & Miquelon')),
('PN', _('Pitcairn')),
('PR', _('Puerto Rico')),
('PT', _('Portugal')),
('PW', _('Palau')),
('PY', _('Paraguay')),
('QA', _('Qatar')),
('RE', _('Reunion')),
('RO', _('Romania')),
('RU', _('Russian Federation')),
('RW', _('Rwanda')),
('SA', _('Saudi Arabia')),
('SB', _('Solomon Islands')),
('SC', _('Seychelles')),
('SD', _('Sudan')),
('SE', _('Sweden')),
('SG', _('Singapore')),
('SH', _('St. Helena')),
('SI', _('Slovenia')),
('SJ', _('Svalbard & Jan Mayen Islands')),
('SK', _('Slovakia')),
('SL', _('Sierra Leone')),
('SM', _('San Marino')),
('SN', _('Senegal')),
('SO', _('Somalia')),
('SR', _('Suriname')),
('ST', _('Sao Tome & Principe')),
('SV', _('El Salvador')),
('SY', _('Syrian Arab Republic')),
('SZ', _('Swaziland')),
('TC', _('Turks & Caicos Islands')),
('TD', _('Chad')),
('TF', _('French Southern Territories')),
('TG', _('Togo')),
('TH', _('Thailand')),
('TJ', _('Tajikistan')),
('TK', _('Tokelau')),
('TM', _('Turkmenistan')),
('TN', _('Tunisia')),
('TO', _('Tonga')),
('TP', _('East Timor')),
('TR', _('Turkey')),
('TT', _('Trinidad & Tobago')),
('TV', _('Tuvalu')),
('TW', _('Taiwan, Province of China')),
('TZ', _('Tanzania, United Republic of')),
('UA', _('Ukraine')),
('UG', _('Uganda')),
('UM', _('United States Minor Outlying Islands')),
('US', _('United States of America')),
('UY', _('Uruguay')),
('UZ', _('Uzbekistan')),
('VA', _('Vatican City State (Holy See)')),
('VC', _('St. Vincent & the Grenadines')),
('VE', _('Venezuela')),
('VG', _('British Virgin Islands')),
('VI', _('United States Virgin Islands')),
('VN', _('Viet Nam')),
('VU', _('Vanuatu')),
('WF', _('Wallis & Futuna Islands')),
('WS', _('Samoa')),
('YE', _('Yemen')),
('YT', _('Mayotte')),
('YU', _('Yugoslavia')),
('ZA', _('South Africa')),
('ZM', _('Zambia')),
('ZR', _('Zaire')),
('ZW', _('Zimbabwe')),
)
__all__ = ('celery_app',)

6
uncloud/admin.py
View File

@ -1,6 +0,0 @@
from django.contrib import admin
from .models import *
for m in [ UncloudProvider, UncloudNetwork, UncloudTask ]:
admin.site.register(m)

17
uncloud/celery.py
View File

@ -1,17 +0,0 @@
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'uncloud.settings')
app = Celery('uncloud')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

43
uncloud/management/commands/db-add-defaults.py
View File

@ -1,43 +0,0 @@
import random
import string
from django.core.management.base import BaseCommand
from django.core.exceptions import ObjectDoesNotExist
from django.contrib.auth import get_user_model
from django.conf import settings
from uncloud_pay.models import BillingAddress, RecurringPeriod, Product
from uncloud.models import UncloudProvider, UncloudNetwork
class Command(BaseCommand):
help = 'Add standard uncloud values'
def add_arguments(self, parser):
pass
def handle(self, *args, **options):
# Order matters, objects can be dependent on each other
admin_username="uncloud-admin"
pw_length = 32
# Only set password if the user did not exist before
try:
admin_user = get_user_model().objects.get(username=settings.UNCLOUD_ADMIN_NAME)
except ObjectDoesNotExist:
random_password = ''.join(random.SystemRandom().choice(string.ascii_lowercase + string.digits) for _ in range(pw_length))
admin_user = get_user_model().objects.create_user(username=settings.UNCLOUD_ADMIN_NAME, password=random_password)
admin_user.is_superuser=True
admin_user.is_staff=True
admin_user.save()
print(f"Created admin user '{admin_username}' with password '{random_password}'")
BillingAddress.populate_db_defaults()
RecurringPeriod.populate_db_defaults()
Product.populate_db_defaults()
UncloudNetwork.populate_db_defaults()
UncloudProvider.populate_db_defaults()

46
uncloud/migrations/0001_initial.py
View File

File diff suppressed because one or more lines are too long

19
uncloud/migrations/0002_uncloudtasks.py
View File

@ -1,19 +0,0 @@
# Generated by Django 3.1 on 2020-12-20 17:16
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('uncloud', '0001_initial'),
]
operations = [
migrations.CreateModel(
name='UncloudTasks',
fields=[
('task_id', models.UUIDField(primary_key=True, serialize=False)),
],
),
]

17
uncloud/migrations/0003_auto_20201220_1728.py
View File

@ -1,17 +0,0 @@
# Generated by Django 3.1 on 2020-12-20 17:28
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('uncloud', '0002_uncloudtasks'),
]
operations = [
migrations.RenameModel(
old_name='UncloudTasks',
new_name='UncloudTask',
),
]

172
uncloud/models.py
View File

@ -1,172 +0,0 @@
from django.db import models
from django.db.models import JSONField, Q
from django.utils import timezone
from django.utils.translation import gettext_lazy as _
from django.core.validators import MinValueValidator, MaxValueValidator
from django.core.exceptions import FieldError
from uncloud import COUNTRIES
class UncloudModel(models.Model):
"""
This class extends the standard model with an
extra_data field that can be used to include public,
but internal information.
For instance if you migrate from an existing virtualisation
framework to uncloud.
The extra_data attribute should be considered a hack and whenever
data is necessary for running uncloud, it should **not** be stored
in there.
"""
extra_data = JSONField(editable=False, blank=True, null=True)
class Meta:
abstract = True
# See https://docs.djangoproject.com/en/dev/ref/models/fields/#field-choices-enum-types
class UncloudStatus(models.TextChoices):
PENDING = 'PENDING', _('Pending')
AWAITING_PAYMENT = 'AWAITING_PAYMENT', _('Awaiting payment')
BEING_CREATED = 'BEING_CREATED', _('Being created')
SCHEDULED = 'SCHEDULED', _('Scheduled') # resource selected, waiting for dispatching
ACTIVE = 'ACTIVE', _('Active')
MODIFYING = 'MODIFYING', _('Modifying') # Resource is being changed
DELETED = 'DELETED', _('Deleted') # Resource has been deleted
DISABLED = 'DISABLED', _('Disabled') # Is usable, but cannot be used for new things
UNUSABLE = 'UNUSABLE', _('Unusable'), # Has some kind of error
###
# General address handling
class CountryField(models.CharField):
def __init__(self, *args, **kwargs):
kwargs.setdefault('choices', COUNTRIES)
kwargs.setdefault('default', 'CH')
kwargs.setdefault('max_length', 2)
super().__init__(*args, **kwargs)
def get_internal_type(self):
return "CharField"
class UncloudAddress(models.Model):
full_name = models.CharField(max_length=256)
organization = models.CharField(max_length=256, blank=True, null=True)
street = models.CharField(max_length=256)
city = models.CharField(max_length=256)
postal_code = models.CharField(max_length=64)
country = CountryField(blank=True)
class Meta:
abstract = True
###
# UncloudNetworks are used as identifiers - such they are a base of uncloud
class UncloudNetwork(models.Model):
"""
Storing IP networks
"""
network_address = models.GenericIPAddressField(null=False, unique=True)
network_mask = models.IntegerField(null=False,
validators=[MinValueValidator(0),
MaxValueValidator(128)]
)
description = models.CharField(max_length=256)
@classmethod
def populate_db_defaults(cls):
for net, desc in [
( "2a0a:e5c0:11::", "uncloud Billing" ),
( "2a0a:e5c0:11:1::", "uncloud Referral" ),
( "2a0a:e5c0:11:2::", "uncloud Coupon" )
]:
obj, created = cls.objects.get_or_create(network_address=net,
defaults= {
'network_mask': 64,
'description': desc
}
)
def save(self, *args, **kwargs):
if not ':' in self.network_address and self.network_mask > 32:
raise FieldError("Mask cannot exceed 32 for IPv4")
super().save(*args, **kwargs)
def __str__(self):
return f"{self.network_address}/{self.network_mask} {self.description}"
###
# Who is running / providing this instance of uncloud?
class UncloudProvider(UncloudAddress):
"""
A class resembling who is running this uncloud instance.
This might change over time so we allow starting/ending dates
This also defines the taxation rules.
starting/ending date define from when to when this is valid. This way
we can model address changes and have it correct in the bills.
"""
# Meta:
# FIXMe: only allow non overlapping time frames -- how to define this as a constraint?
starting_date = models.DateField()
ending_date = models.DateField(blank=True, null=True)
billing_network = models.ForeignKey(UncloudNetwork, related_name="uncloudproviderbill", on_delete=models.CASCADE)
referral_network = models.ForeignKey(UncloudNetwork, related_name="uncloudproviderreferral", on_delete=models.CASCADE)
coupon_network = models.ForeignKey(UncloudNetwork, related_name="uncloudprovidercoupon", on_delete=models.CASCADE)
@classmethod
def get_provider(cls, when=None):
"""
Find active provide at a certain time - if there was any
"""
if not when:
when = timezone.now()
return cls.objects.get(Q(starting_date__gte=when, ending_date__lte=when) |
Q(starting_date__gte=when, ending_date__isnull=True))
@classmethod
def populate_db_defaults(cls):
obj, created = cls.objects.get_or_create(full_name="ungleich glarus ag",
street="Bahnhofstrasse 1",
postal_code="8783",
city="Linthal",
country="CH",
starting_date=timezone.now(),
billing_network=UncloudNetwork.objects.get(description="uncloud Billing"),
referral_network=UncloudNetwork.objects.get(description="uncloud Referral"),
coupon_network=UncloudNetwork.objects.get(description="uncloud Coupon")
)
def __str__(self):
return f"{self.full_name} {self.country}"
class UncloudTask(models.Model):
"""
Class to store dispatched tasks to be handled
"""
task_id = models.UUIDField(primary_key=True)

19
uncloud/tasks.py
View File

@ -1,19 +0,0 @@
from celery import shared_task
from celery.result import AsyncResult
from .models import UncloudTask
@shared_task(bind=True)
def cleanup_tasks(self):
print(f"Cleanup time from {self}: {self.request.id}")
for task in UncloudTask.objects.all():
print(f"Pruning {task}...")
if str(task.task_id) == str(self.request.id):
print("Skipping myself")
continue
res = AsyncResult(id=str(task.task_id))
if res.ready():
print(res.get())
task.delete()

14
uncloud/templates/uncloud/base.html
View File

@ -1,14 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title>{% block title %}Welcome to uncloud{% endblock %}</title>
{% block header %}{% endblock %}
</head>
<body>
{% block body %}{% endblock %}
</body>
</html>

15
uncloud/templates/uncloud/index.html
View File

@ -1,15 +0,0 @@
{% extends 'uncloud/base.html' %}
{% block title %}{% endblock %}
{% block body %}
<div>
<h1>Welcome to uncloud</h1>
Welcome to uncloud, checkout the following locations:
<ul>
<li><a href="/api/">The API</a>
<li><a href="/cc/reg/">The CC registration</a>
</ul>
</div>
{% endblock %}

94
uncloud/urls.py
View File

@ -1,94 +0,0 @@
"""uncloud URL Configuration
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/3.0/topics/http/urls/
"""
from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static
from rest_framework import routers
from rest_framework.schemas import get_schema_view
#from opennebula import views as oneviews
from uncloud import views as uncloudviews
from uncloud_auth import views as authviews
from uncloud_net import views as netviews
from uncloud_pay import views as payviews
from uncloud_vm import views as vmviews
from uncloud_service import views as serviceviews
router = routers.DefaultRouter()
# Beta endpoints
router.register(r'beta/vm', vmviews.NicoVMProductViewSet, basename='nicovmproduct')
# VM
router.register(r'v1/vm/snapshot', vmviews.VMSnapshotProductViewSet, basename='vmsnapshotproduct')
router.register(r'v1/vm/diskimage', vmviews.VMDiskImageProductViewSet, basename='vmdiskimageproduct')
router.register(r'v1/vm/disk', vmviews.VMDiskProductViewSet, basename='vmdiskproduct')
router.register(r'v1/vm/vm', vmviews.VMProductViewSet, basename='vmproduct')
# creates VM from os image
#router.register(r'vm/ipv6onlyvm', vmviews.VMProductViewSet, basename='vmproduct')
# ... AND adds IPv4 mapping
#router.register(r'vm/dualstackvm', vmviews.VMProductViewSet, basename='vmproduct')
# Services
router.register(r'v1/service/matrix', serviceviews.MatrixServiceProductViewSet, basename='matrixserviceproduct')
router.register(r'v1/service/generic', serviceviews.GenericServiceProductViewSet, basename='genericserviceproduct')
# Pay
router.register(r'v1/my/address', payviews.BillingAddressViewSet, basename='billingaddress')
router.register(r'v1/my/bill', payviews.BillViewSet, basename='bill')
router.register(r'v1/my/order', payviews.OrderViewSet, basename='order')
router.register(r'v1/my/payment', payviews.PaymentViewSet, basename='payment')
router.register(r'v1/my/payment-method', payviews.PaymentMethodViewSet, basename='payment-method')
# admin/staff urls
router.register(r'v1/admin/bill', payviews.AdminBillViewSet, basename='admin/bill')
router.register(r'v1/admin/payment', payviews.AdminPaymentViewSet, basename='admin/payment')
router.register(r'v1/admin/order', payviews.AdminOrderViewSet, basename='admin/order')
router.register(r'v1/admin/vmhost', vmviews.VMHostViewSet)
router.register(r'v1/admin/vmcluster', vmviews.VMClusterViewSet)
#router.register(r'v1/admin/vpnpool', netviews.VPNPoolViewSet)
#router.register(r'v1/admin/opennebula', oneviews.VMViewSet, basename='opennebula')
# User/Account
router.register(r'v1/my/user', authviews.UserViewSet, basename='user')
router.register(r'v1/admin/user', authviews.AdminUserViewSet, basename='useradmin')
router.register(r'v1/user/register', authviews.AccountManagementViewSet, basename='user/register')
################################################################################
# v2
# Net
router.register(r'v2/net/wireguardvpn', netviews.WireGuardVPNViewSet, basename='wireguardvpnnetwork')
router.register(r'v2/net/wireguardvpnsizes', netviews.WireGuardVPNSizes, basename='wireguardvpnnetworksizes')
urlpatterns = [
path(r'api/', include(router.urls)),
path('api-auth/', include('rest_framework.urls', namespace='rest_framework')), # for login to REST API
path('openapi', get_schema_view(
title="uncloud",
description="uncloud API",
version="1.0.0"
), name='openapi-schema'),
# web/ = stuff to view in the browser
# path('web/vpn/create/', netviews.WireGuardVPNCreateView.as_view(), name="vpncreate"),
path('login/', authviews.LoginView.as_view(), name="login"),
path('logout/', authviews.LogoutView.as_view(), name="logout"),
path('admin/', admin.site.urls),
path('cc/reg/', payviews.RegisterCard.as_view(), name="cc_register"),
path('', uncloudviews.UncloudIndex.as_view(), name="uncloudindex"),
]

4
uncloud/views.py
View File

@ -1,4 +0,0 @@
from django.views.generic.base import TemplateView
class UncloudIndex(TemplateView):
template_name = "uncloud/index.html"

21
uncloud_auth/management/commands/make-admin.py
View File

@ -1,21 +0,0 @@
from django.core.management.base import BaseCommand
from django.contrib.auth import get_user_model
import sys
class Command(BaseCommand):
help = 'Give Admin rights to existing user'
def add_arguments(self, parser):
parser.add_argument('username', type=str)
parser.add_argument('--superuser', action='store_true')
def handle(self, *args, **options):
user = get_user_model().objects.get(username=options['username'])
user.is_staff = True
if options['superuser']:
user.is_superuser = True
user.save()
print(f"{user.username} is now admin (superuser={user.is_superuser})")

72
uncloud_auth/serializers.py
View File

@ -1,72 +0,0 @@
from django.contrib.auth import get_user_model
from django.db import transaction
from ldap3.core.exceptions import LDAPEntryAlreadyExistsResult
from rest_framework import serializers
from uncloud import AMOUNT_DECIMALS, AMOUNT_MAX_DIGITS
from uncloud_pay.models import BillingAddress
from .ungleich_ldap import LdapManager
class UserSerializer(serializers.ModelSerializer):
class Meta:
model = get_user_model()
read_only_fields = [ 'username', 'balance', 'maximum_credit' ]
fields = read_only_fields + [ 'email' ] # , 'primary_billing_address' ]
def validate(self, data):
"""
Ensure that the primary billing address belongs to the user
"""
# The following is raising exceptions probably, it is WIP somewhere
# if 'primary_billing_address' in data:
# if not data['primary_billing_address'].owner == self.instance:
# raise serializers.ValidationError('Invalid data')
return data
def update(self, instance, validated_data):
ldap_manager = LdapManager()
return_val, _ = ldap_manager.change_user_details(
instance.username, {'mail': validated_data.get('email')}
)
if not return_val:
raise serializers.ValidationError('Couldn\'t update email')
instance.email = validated_data.get('email')
instance.save()
return instance
class UserRegistrationSerializer(serializers.ModelSerializer):
class Meta:
model = get_user_model()
fields = ['username', 'first_name', 'last_name', 'email', 'password']
extra_kwargs = {
'password': {'style': {'input_type': 'password'}},
'first_name': {'allow_blank': False, 'required': True},
'last_name': {'allow_blank': False, 'required': True},
'email': {'allow_blank': False, 'required': True},
}
def create(self, validated_data):
ldap_manager = LdapManager()
try:
data = {
'user': validated_data['username'],
'password': validated_data['password'],
'email': validated_data['email'],
'firstname': validated_data['first_name'],
'lastname': validated_data['last_name'],
}
ldap_manager.create_user(**data)
except LDAPEntryAlreadyExistsResult:
raise serializers.ValidationError(
{'username': ['A user with that username already exists.']}
)
else:
return get_user_model().objects.create_user(**validated_data)
class ImportUserSerializer(serializers.Serializer):
username = serializers.CharField()

13
uncloud_auth/templates/uncloud_auth/login.html
View File

@ -1,13 +0,0 @@
{% extends 'uncloud/base.html' %}
{% block body %}
<div class="container">
<form method="post">
{% csrf_token %}
{{ form }}
<input type="submit" value="Login">
</form>
</div>
{% endblock %}

42
uncloud_auth/uldap.py
View File

@ -1,42 +0,0 @@
import ldap
# from django.conf import settings
AUTH_LDAP_SERVER_URI = "ldaps://ldap1.ungleich.ch,ldaps://ldap2.ungleich.ch"
AUTH_LDAP_BIND_DN="uid=django-create,ou=system,dc=ungleich,dc=ch"
AUTH_LDAP_BIND_PASSWORD="kS#e+v\zjKn]L!,RIu2}V+DUS"
# AUTH_LDAP_USER_SEARCH = LDAPSearch("dc=ungleich,dc=ch",
# ldap.SCOPE_SUBTREE,
# "(uid=%(user)s)")
ldap_object = ldap.initialize(AUTH_LDAP_SERVER_URI)
cancelid = ldap_object.bind(AUTH_LDAP_BIND_DN, AUTH_LDAP_BIND_PASSWORD)
res = ldap_object.search_s("dc=ungleich,dc=ch", ldap.SCOPE_SUBTREE, "(uid=nico)")
print(res)
# class LDAP(object):
# """
# Managing users in LDAP
# Requires the following settings?
# LDAP_USER_DN: where to create users in the tree
# LDAP_ADMIN_DN: which DN to use for managing users
# LDAP_ADMIN_PASSWORD: which password to used
# This module will reuse information from djagno_auth_ldap, including:
# AUTH_LDAP_SERVER_URI
# """
# def __init__(self):
# pass
# def create_user(self):
# pass
# def change_password(self):
# pass

284
uncloud_auth/ungleich_ldap.py
View File

@ -1,284 +0,0 @@
import base64
import hashlib
import logging
import random
import ldap3
from django.conf import settings
logger = logging.getLogger(__name__)
class LdapManager:
__instance = None
def __new__(cls):
if LdapManager.__instance is None:
LdapManager.__instance = object.__new__(cls)
return LdapManager.__instance
def __init__(self):
"""
Initialize the LDAP subsystem.
"""
self.rng = random.SystemRandom()
self.server = ldap3.Server(settings.AUTH_LDAP_SERVER)
def get_admin_conn(self):
"""
Return a bound :class:`ldap3.Connection` instance which has write
permissions on the dn in which the user accounts reside.
"""
conn = self.get_conn(user=settings.LDAP_ADMIN_DN,
password=settings.LDAP_ADMIN_PASSWORD,
raise_exceptions=True)
conn.bind()
return conn
def get_conn(self, **kwargs):
"""
Return an unbound :class:`ldap3.Connection` which talks to the configured
LDAP server.
The *kwargs* are passed to the constructor of :class:`ldap3.Connection` and
can be used to set *user*, *password* and other useful arguments.
"""
return ldap3.Connection(self.server, **kwargs)
def _ssha_password(self, password):
"""
Apply the SSHA password hashing scheme to the given *password*.
*password* must be a :class:`bytes` object, containing the utf-8
encoded password.
Return a :class:`bytes` object containing ``ascii``-compatible data
which can be used as LDAP value, e.g. after armoring it once more using
base64 or decoding it to unicode from ``ascii``.
"""
SALT_BYTES = 15
sha1 = hashlib.sha1()
salt = self.rng.getrandbits(SALT_BYTES * 8).to_bytes(SALT_BYTES,
"little")
sha1.update(password)
sha1.update(salt)
digest = sha1.digest()
passwd = b"{SSHA}" + base64.b64encode(digest + salt)
return passwd
def create_user(self, user, password, firstname, lastname, email):
conn = self.get_admin_conn()
uidNumber = self._get_max_uid() + 1
logger.debug("uidNumber={uidNumber}".format(uidNumber=uidNumber))
user_exists = True
while user_exists:
user_exists, _ = self.check_user_exists(
"",
'(&(objectClass=inetOrgPerson)(objectClass=posixAccount)'
'(objectClass=top)(uidNumber={uidNumber}))'.format(
uidNumber=uidNumber
)
)
if user_exists:
logger.debug(
"{uid} exists. Trying next.".format(uid=uidNumber)
)
uidNumber += 1
logger.debug("{uid} does not exist. Using it".format(uid=uidNumber))
self._set_max_uid(uidNumber)
try:
uid = user # user.encode("utf-8")
conn.add("uid={uid},{customer_dn}".format(
uid=uid, customer_dn=settings.LDAP_CUSTOMER_DN
),
["inetOrgPerson", "posixAccount", "ldapPublickey"],
{
"uid": [uid],
"sn": [lastname.encode("utf-8")],
"givenName": [firstname.encode("utf-8")],
"cn": [uid],
"displayName": ["{} {}".format(firstname, lastname).encode("utf-8")],
"uidNumber": [str(uidNumber)],
"gidNumber": [str(settings.LDAP_CUSTOMER_GROUP_ID)],
"loginShell": ["/bin/bash"],
"homeDirectory": ["/home/{}".format(user).encode("utf-8")],
"mail": email.encode("utf-8"),
"userPassword": [self._ssha_password(
password.encode("utf-8")
)]
}
)
logger.debug('Created user %s %s' % (user.encode('utf-8'),
uidNumber))
except Exception as ex:
logger.debug('Could not create user %s' % user.encode('utf-8'))
logger.error("Exception: " + str(ex))
raise
finally:
conn.unbind()
def change_password(self, uid, new_password):
"""
Changes the password of the user identified by user_dn
:param uid: str The uid that identifies the user
:param new_password: str The new password string
:return: True if password was changed successfully False otherwise
"""
conn = self.get_admin_conn()
# Make sure the user exists first to change his/her details
user_exists, entries = self.check_user_exists(
uid=uid,
search_base=settings.ENTIRE_SEARCH_BASE
)
return_val = False
if user_exists:
try:
return_val = conn.modify(
entries[0].entry_dn,
{
"userpassword": (
ldap3.MODIFY_REPLACE,
[self._ssha_password(new_password.encode("utf-8"))]
)
}
)
except Exception as ex:
logger.error("Exception: " + str(ex))
else:
logger.error("User {} not found".format(uid))
conn.unbind()
return return_val
def change_user_details(self, uid, details):
"""
Updates the user details as per given values in kwargs of the user
identified by user_dn.
Assumes that all attributes passed in kwargs are valid.
:param uid: str The uid that identifies the user
:param details: dict A dictionary containing the new values
:return: True if user details were updated successfully False otherwise
"""
conn = self.get_admin_conn()
# Make sure the user exists first to change his/her details
user_exists, entries = self.check_user_exists(
uid=uid,
search_base=settings.ENTIRE_SEARCH_BASE
)
return_val = False
if user_exists:
details_dict = {k: (ldap3.MODIFY_REPLACE, [v.encode("utf-8")]) for
k, v in details.items()}
try:
return_val = conn.modify(entries[0].entry_dn, details_dict)
msg = "success"
except Exception as ex:
msg = str(ex)
logger.error("Exception: " + msg)
finally:
conn.unbind()
else:
msg = "User {} not found".format(uid)
logger.error(msg)
conn.unbind()
return return_val, msg
def check_user_exists(self, uid, search_filter="", attributes=None,
search_base=settings.LDAP_CUSTOMER_DN):
"""
Check if the user with the given uid exists in the customer group.
:param uid: str representing the user
:param search_filter: str representing the filter condition to find
users. If its empty, the search finds the user with
the given uid.
:param attributes: list A list of str representing all the attributes
to be obtained in the result entries
:param search_base: str
:return: tuple (bool, [ldap3.abstract.entry.Entry ..])
A bool indicating if the user exists
A list of all entries obtained in the search
"""
conn = self.get_admin_conn()
entries = []
try:
result = conn.search(
search_base=search_base,
search_filter=search_filter if len(search_filter)> 0 else
'(uid={uid})'.format(uid=uid),
attributes=attributes
)
entries = conn.entries
finally:
conn.unbind()
return result, entries
def delete_user(self, uid):
"""
Deletes the user with the given uid from ldap
:param uid: str representing the user
:return: True if the delete was successful False otherwise
"""
conn = self.get_admin_conn()
try:
return_val = conn.delete(
("uid={uid}," + settings.LDAP_CUSTOMER_DN).format(uid=uid),
)
msg = "success"
except Exception as ex:
msg = str(ex)
logger.error("Exception: " + msg)
return_val = False
finally:
conn.unbind()
return return_val, msg
def _set_max_uid(self, max_uid):
"""
a utility function to save max_uid value to a file
:param max_uid: an integer representing the max uid
:return:
"""
with open(settings.LDAP_MAX_UID_FILE_PATH, 'w+') as handler:
handler.write(str(max_uid))
def _get_max_uid(self):
"""
A utility function to read the max uid value that was previously set
:return: An integer representing the max uid value that was previously
set
"""
try:
with open(settings.LDAP_MAX_UID_FILE_PATH, 'r+') as handler:
try:
return_value = int(handler.read())
except ValueError as ve:
logger.error(
"Error reading int value from {}. {}"
"Returning default value {} instead".format(
settings.LDAP_MAX_UID_PATH,
str(ve),
settings.LDAP_DEFAULT_START_UID
)
)
return_value = settings.LDAP_DEFAULT_START_UID
return return_value
except FileNotFoundError as fnfe:
logger.error("File not found : " + str(fnfe))
return_value = settings.LDAP_DEFAULT_START_UID
logger.error("So, returning UID={}".format(return_value))
return return_value

77
uncloud_auth/views.py
View File

@ -1,77 +0,0 @@
from django.contrib.auth import views as auth_views
from django.contrib.auth import logout
from django_auth_ldap.backend import LDAPBackend
from rest_framework import mixins, permissions, status, viewsets
from rest_framework.decorators import action
from rest_framework.response import Response
from .serializers import *
class LoginView(auth_views.LoginView):
template_name = 'uncloud_auth/login.html'
class LogoutView(auth_views.LogoutView):
pass
# template_name = 'uncloud_auth/logo.html'
class UserViewSet(viewsets.GenericViewSet):
permission_classes = [permissions.IsAuthenticated]
serializer_class = UserSerializer
def get_queryset(self):
return self.request.user
def list(self, request, format=None):
# This is a bit stupid: we have a user, we create a queryset by
# matching on the username. But I don't know a "nicer" way.
# Nico, 2020-03-18
user = request.user
serializer = self.get_serializer(user, context = {'request': request})
return Response(serializer.data)
@action(detail=False, methods=['post'])
def change_email(self, request):
serializer = self.get_serializer(
request.user, data=request.data, context={'request': request}
)
serializer.is_valid(raise_exception=True)
serializer.save()
return Response(serializer.data)
class AccountManagementViewSet(mixins.CreateModelMixin, viewsets.GenericViewSet):
serializer_class = UserRegistrationSerializer
def create(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
self.perform_create(serializer)
headers = self.get_success_headers(serializer.data)
return Response(
serializer.data, status=status.HTTP_201_CREATED, headers=headers
)
class AdminUserViewSet(viewsets.ReadOnlyModelViewSet):
permission_classes = [permissions.IsAdminUser]
def get_serializer_class(self):
if self.action == 'import_from_ldap':
return ImportUserSerializer
else:
return UserSerializer
def get_queryset(self):
return get_user_model().objects.all()
@action(detail=False, methods=['post'], url_path='import_from_ldap')
def import_from_ldap(self, request, pk=None):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
ldap_username = serializer.validated_data.pop("username")
user = LDAPBackend().populate_user(ldap_username)
return Response(UserSerializer(user, context = {'request': request}).data)

0
archive/uncloud_django_based/hacks/abk-hacks.py → uncloud_django_based/abk-hacks.py
View File

0
archive/uncloud_django_based/hacks/abkhack/opennebula_hacks.py → uncloud_django_based/abkhack/opennebula_hacks.py
View File

0
archive/uncloud_django_based/meow-payv1/README.md → uncloud_django_based/meow-payv1/README.md
View File

0
archive/uncloud_django_based/meow-payv1/config.py → uncloud_django_based/meow-payv1/config.py
View File

0
archive/uncloud_django_based/meow-payv1/hack-a-vpn.py → uncloud_django_based/meow-payv1/hack-a-vpn.py
View File

0
archive/uncloud_django_based/meow-payv1/helper.py → uncloud_django_based/meow-payv1/helper.py
View File

0
archive/uncloud_django_based/meow-payv1/products/ipv6-only-django.json → uncloud_django_based/meow-payv1/products/ipv6-only-django.json
View File

0
archive/uncloud_django_based/meow-payv1/products/ipv6-only-vm.json → uncloud_django_based/meow-payv1/products/ipv6-only-vm.json
View File

0
archive/uncloud_django_based/meow-payv1/products/ipv6-only-vpn.json → uncloud_django_based/meow-payv1/products/ipv6-only-vpn.json
View File

0
archive/uncloud_django_based/meow-payv1/products/ipv6box.json → uncloud_django_based/meow-payv1/products/ipv6box.json
View File

0
archive/uncloud_django_based/meow-payv1/products/membership.json → uncloud_django_based/meow-payv1/products/membership.json
View File

0
archive/uncloud_django_based/meow-payv1/requirements.txt → uncloud_django_based/meow-payv1/requirements.txt
View File

0
archive/uncloud_django_based/meow-payv1/sample-pay.conf → uncloud_django_based/meow-payv1/sample-pay.conf
View File

0
archive/uncloud_django_based/meow-payv1/schemas.py → uncloud_django_based/meow-payv1/schemas.py
View File

0
archive/uncloud_django_based/meow-payv1/stripe_hack.py → uncloud_django_based/meow-payv1/stripe_hack.py
View File

0
archive/uncloud_django_based/meow-payv1/stripe_utils.py → uncloud_django_based/meow-payv1/stripe_utils.py
View File

0
archive/uncloud_django_based/meow-payv1/ucloud_pay.py → uncloud_django_based/meow-payv1/ucloud_pay.py
View File

0
archive/uncloud_django_based/notes-abk.md → uncloud_django_based/notes-abk.md
View File

0
archive/uncloud_django_based/notes-nico.org → uncloud_django_based/notes-nico.org
View File

0
archive/uncloud_django_based/plan.org → uncloud_django_based/plan.org
View File

0
archive/uncloud_django_based/uncloud/.gitignore → uncloud_django_based/uncloud/.gitignore vendored
View File

7
doc/README-how-to-configure-remote-uncloud-clients.org → uncloud_django_based/uncloud/doc/README-how-to-configure-remote-uncloud-clients.org
View File

@ -10,7 +10,6 @@
| SSH -L tunnel | All nodes can use [::1]:5432 | SSH setup can be fragile |
| ssh djangohost manage.py | All DB ops locally | Code is only executed on django host |
| https + token | Rest alike / consistent access | Code is only executed on django host |
| from_django | Everything is on the django host | main host can become bottleneck |
** remote vs. local Django code execution
- If manage.py is executed locally (= on the client), it can
check/modify local configs
@ -20,9 +19,3 @@
- Remote execution (= on the primary django host) can acess the db
via unix socket
- However remote execution cannot check local state
** from_django
- might reuse existing methods like celery
- reduces the amount of things to be installed on the client to
almost zero
- follows the opennebula model
- has a single point of failurebin

82
uncloud_django_based/uncloud/doc/README-object-relations.md Normal file
View File

@ -0,0 +1,82 @@
## Introduction
This article describes how models relate to each other and what the
design ideas are. It is meant to prevent us from double implementing
something or changing something that is already solved.
## Products
A product is something someone can order. We might have "low level"
products that need to be composed (= higher degree of flexibility, but
more amount of details necessary) and "composed products" that present
some defaults or select other products automatically (f.i. a "dual
stack VM" can be a VM + a disk + an IPv4 address).
## Bills
Bills represent active orders of a month. Bills can be shown during a
month but only become definitive at the end of the month.
## Orders
When customer X order a (set) of product, it generates an order for billing
purposes. The ordered products point to that order and register an Order Record
at creation.
Orders and Order Records are assumed immutable => they are used to generate
bills and should not be mutated. If a product is updated (e.g. adding RAM to
VM), a new order should be generated.
The order MUST NOT be deleted when a product is deleted, as it is used for
billing (including past bills).
### Order record
Used to store billing details of a product at creation: will stay there even if
the product change (e.g. new pricing, updated) and act as some kind of archive.
Used to generate bills.
## Payment Methods
Users/customers can register payment methods.
## Sample flows / products
### A VM snapshot
A VM snapshot creates a snapshot of all disks attached to a VM to be
able to rollback the VM to a previous state.
Creating a VM snapshot (-product) creates a related order. Deleting a
VMSnapshotproduct sets the order to deleted.
### Object Storage
(tbd by Balazs)
### A "raw" VM
(tbd by Ahmed)
### An IPv6 only VM
(tbd by Ahmed)
### A dual stack VM
(tbd by Ahmed)
### A managed service (e.g. Matrix-as-a-Service)
Customer orders service with:
* Service-specific configuration: e.g. domain name for matrix
* VM configuration:
- CPU
- Memory
- Disk (soon)
It creates a new Order with two products/records:
* Service itself (= management)
* Underlying VM

8
uncloud_django_based/uncloud/doc/README-postgresql.org Normal file
View File

@ -0,0 +1,8 @@
* uncloud clients access the data base from a variety of outside hosts
* So the postgresql data base needs to be remotely accessible
* Instead of exposing the tcp socket, we make postgresql bind to localhost via IPv6
** ::1, port 5432
* Then we remotely connect to the database server with ssh tunneling
** ssh -L5432:localhost:5432 uncloud-database-host
* Configuring your database for SSH based remote access
** host all all ::1/128 trust

34
uncloud_django_based/uncloud/doc/README-products.md Normal file
View File

@ -0,0 +1,34 @@
## Introduction
This document describes how to create, modify or
delete a product and use it.
A product (like a VMSnapshotproduct) creates an order when ordered.
The "order" is used to combine products together.
Sub-products or related products link to the same order.
Each product has one (?) orderrecord
## How to delete a product (logic 1)
If a user want so delete (=cancel) a product, the following steps
should be taken:
* the associated order is set to cancelled
* the product itself is deleted
[above steps to be reviewed]
## How to delete a product (rest api)
http -a nicoschottelius:$(pass
ungleich.ch/nico.schottelius@ungleich.ch)
http://localhost:8000/net/vpn/43c83088-f4d6-49b9-86c7-40251ac07ada/
-> does not delete the reservation.
### Deleting a VPN
When the product is deleted, the network must be marked as free.

34
uncloud_django_based/uncloud/doc/README-vpn.org Normal file
View File

@ -0,0 +1,34 @@
* How to add a new VPN Host
** Install wireguard to the host
** Install uncloud to the host
** Add `python manage.py vpn --hostname fqdn-of-this-host` to the crontab
** Use the CLI to configure one or more VPN Networks for this host
* Example of adding a VPN host at ungleich
** Create a new dual stack alpine VM
** Add it to DNS as vpn-XXX.ungleich.ch
** Route a /40 network to its IPv6 address
** Install wireguard on it
** TODO Enable wireguard on boot
** TODO Create a new VPNPool on uncloud with
*** the network address (selecting from our existing pool)
*** the network size (/...)
*** the vpn host that provides the network (selecting the created VM)
*** the wireguard private key of the vpn host (using wg genkey)
*** http command
```
http -a nicoschottelius:$(pass
ungleich.ch/nico.schottelius@ungleich.ch)
http://localhost:8000/admin/vpnpool/ network=2a0a:e5c1:200:: \
network_size=40 subnetwork_size=48
vpn_hostname=vpn-2a0ae5c1200.ungleich.ch
wireguard_private_key=...
```
* Example http commands / REST calls
** creating a new vpn pool
http -a nicoschottelius:$(pass
ungleich.ch/nico.schottelius@ungleich.ch)
http://localhost:8000/admin/vpnpool/ network_size=40
subnetwork_size=48 network=2a0a:e5c1:200::
vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg
genkey)
** Creating a new vpn network

95
uncloud_django_based/uncloud/doc/README.md Normal file
View File

@ -0,0 +1,95 @@
## Install
### OS package requirements
Alpine:
```
apk add openldap-dev postgresql-dev
```
Debian/Devuan:
```
apt install postgresql-server-dev-all
```
### Python requirements
If you prefer using a venv, use:
```
python -m venv venv
. ./venv/bin/activate
```
Then install the requirements
```
pip install -r requirements.txt
```
### Database requirements
Due to the use of the JSONField, postgresql is required.
First create a role to be used:
```
postgres=# create role nico login;
```
Then create the database owner by the new role:
```
postgres=# create database uncloud owner nico;
```
Installing the postgresql service is os dependent, but some hints:
* Alpine: `apk add postgresql-server && rc-update add postgresql && rc-service postgresql start`
* Debian/Devuan: `apt install postgresql`
After postresql is started, apply the migrations:
```
python manage.py migrate
```
### Secrets
cp `uncloud/secrets_sample.py` to `uncloud/secrets.py` and replace the
sample values with real values.
## Flows / Orders
### Creating a VMHost
### Creating a VM
* Create a VMHost
* Create a VM on a VMHost
### Creating a VM Snapshot
## Working Beta APIs
These APIs can be used for internal testing.
### URL Overview
```
http -a nicoschottelius:$(pass ungleich.ch/nico.schottelius@ungleich.ch) http://localhost:8000
```
### Snapshotting
```
http -a nicoschottelius:$(pass ungleich.ch/nico.schottelius@ungleich.ch) http://localhost:8000/vm/snapshot/ vm_uuid=$(uuidgen)
```

0
manage.py → uncloud_django_based/uncloud/manage.py
View File

0
models.dot → uncloud_django_based/uncloud/models.dot
View File

0
models.png → uncloud_django_based/uncloud/models.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 398 KiB

After

Width:  |  Height:  |  Size: 398 KiB

0
archive/uncloud_etcd_based/docs/__init__.py → uncloud_django_based/uncloud/opennebula/__init__.py
View File

0
opennebula/admin.py → uncloud_django_based/uncloud/opennebula/admin.py
View File

0
opennebula/apps.py → uncloud_django_based/uncloud/opennebula/apps.py
View File

0
opennebula/management/commands/opennebula-synchosts.py → uncloud_django_based/uncloud/opennebula/management/commands/opennebula-synchosts.py
View File

9
opennebula/management/commands/opennebula-syncvms.py → uncloud_django_based/uncloud/opennebula/management/commands/opennebula-syncvms.py
View File

@ -1,9 +1,12 @@
import json
import uncloud.secrets as secrets
from xmlrpc.client import ServerProxy as RPCClient
from django_auth_ldap.backend import LDAPBackend
from django.core.management.base import BaseCommand
from django.conf import settings
from xmltodict import parse
from opennebula.models import VM as VMModel
@ -16,9 +19,9 @@ class Command(BaseCommand):
pass
def handle(self, *args, **options):
with RPCClient(settings.OPENNEBULA_URL) as rpc_client:
with RPCClient(secrets.OPENNEBULA_URL) as rpc_client:
success, response, *_ = rpc_client.one.vmpool.infoextended(
settings.OPENNEBULA_USER_PASS, -2, -1, -1, -1
secrets.OPENNEBULA_USER_PASS, -2, -1, -1, -1
)
if success:
vms = json.loads(json.dumps(parse(response)))['VM_POOL']['VM']

0
opennebula/management/commands/opennebula-to-uncloud.py → uncloud_django_based/uncloud/opennebula/management/commands/opennebula-to-uncloud.py
View File

28
uncloud_django_based/uncloud/opennebula/migrations/0001_initial.py Normal file
View File

@ -0,0 +1,28 @@
# Generated by Django 3.0.3 on 2020-02-23 17:12
from django.conf import settings
import django.contrib.postgres.fields.jsonb
from django.db import migrations, models
import django.db.models.deletion
import uuid
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='VM',
fields=[
('vmid', models.IntegerField(primary_key=True, serialize=False)),
('uuid', models.UUIDField(default=uuid.uuid4, editable=False, unique=True)),
('data', django.contrib.postgres.fields.jsonb.JSONField()),
('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
),
]

27
uncloud_django_based/uncloud/opennebula/migrations/0002_auto_20200225_1335.py Normal file
View File

@ -0,0 +1,27 @@
# Generated by Django 3.0.3 on 2020-02-25 13:35
from django.db import migrations, models
import uuid
class Migration(migrations.Migration):
dependencies = [
('opennebula', '0001_initial'),
]
operations = [
migrations.RemoveField(
model_name='vm',
name='uuid',
),
migrations.RemoveField(
model_name='vm',
name='vmid',
),
migrations.AddField(
model_name='vm',
name='id',
field=models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False, unique=True),
),
]

19
uncloud_django_based/uncloud/opennebula/migrations/0003_auto_20200225_1428.py Normal file
View File

@ -0,0 +1,19 @@
# Generated by Django 3.0.3 on 2020-02-25 14:28
from django.db import migrations, models
import uuid
class Migration(migrations.Migration):
dependencies = [
('opennebula', '0002_auto_20200225_1335'),
]
operations = [
migrations.AlterField(
model_name='vm',
name='id',
field=models.CharField(default=uuid.uuid4, max_length=64, primary_key=True, serialize=False, unique=True),
),
]

23
uncloud_django_based/uncloud/opennebula/migrations/0004_auto_20200225_1816.py Normal file
View File

@ -0,0 +1,23 @@
# Generated by Django 3.0.3 on 2020-02-25 18:16
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('opennebula', '0003_auto_20200225_1428'),
]
operations = [
migrations.RemoveField(
model_name='vm',
name='id',
),
migrations.AddField(
model_name='vm',
name='vmid',
field=models.IntegerField(default=42, primary_key=True, serialize=False),
preserve_default=False,
),
]

0
archive/uncloud_etcd_based/docs/source/__init__.py → uncloud_django_based/uncloud/opennebula/migrations/__init__.py
View File

5
opennebula/models.py → uncloud_django_based/uncloud/opennebula/models.py
View File

@ -1,7 +1,7 @@
import uuid
from django.db import models
from django.contrib.auth import get_user_model
from uncloud_pay.models import Product
from django.contrib.postgres.fields import JSONField
# ungleich specific
storage_class_mapping = {
@ -12,7 +12,8 @@ storage_class_mapping = {
class VM(models.Model):
vmid = models.IntegerField(primary_key=True)
data = models.JSONField()
owner = models.ForeignKey(get_user_model(), on_delete=models.CASCADE)
data = JSONField()
@property
def uncloud_name(self):

0
opennebula/serializers.py → uncloud_django_based/uncloud/opennebula/serializers.py
View File

0
opennebula/tests.py → uncloud_django_based/uncloud/opennebula/tests.py
View File

16
uncloud_django_based/uncloud/opennebula/views.py Normal file
View File

@ -0,0 +1,16 @@
from rest_framework import viewsets, permissions
from .models import VM
from .serializers import OpenNebulaVMSerializer
class VMViewSet(viewsets.ModelViewSet):
permission_classes = [permissions.IsAuthenticated]
serializer_class = OpenNebulaVMSerializer
def get_queryset(self):
if self.request.user.is_superuser:
obj = VM.objects.all()
else:
obj = VM.objects.filter(owner=self.request.user)
return obj

16
requirements.txt → uncloud_django_based/uncloud/requirements.txt
View File

@ -1,12 +1,9 @@
# Django basics
django
djangorestframework
django-auth-ldap
psycopg2
ldap3
stripe
xmltodict
psycopg2
parsedatetime
@ -22,11 +19,6 @@ django-hardcopy
pyyaml
uritemplate
# Payment & VAT
# Comprehensive interface to validate VAT numbers, making use of the VIES
# service for European countries.
vat-validator
stripe
# Tasks
celery
redis

1
uncloud_django_based/uncloud/uncloud/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
secrets.py

4
uncloud_django_based/uncloud/uncloud/__init__.py Normal file
View File

@ -0,0 +1,4 @@
# Define DecimalField properties, used to represent amounts of money.
# Used in pay and auth
AMOUNT_MAX_DIGITS=10
AMOUNT_DECIMALS=2

0
uncloud/asgi.py → uncloud_django_based/uncloud/uncloud/asgi.py
View File

0
uncloud/management/commands/uncloud.py → uncloud_django_based/uncloud/uncloud/management/commands/uncloud.py
View File

35
uncloud_django_based/uncloud/uncloud/models.py Normal file
View File

@ -0,0 +1,35 @@
from django.db import models
from django.contrib.postgres.fields import JSONField
from django.utils.translation import gettext_lazy as _
class UncloudModel(models.Model):
"""
This class extends the standard model with an
extra_data field that can be used to include public,
but internal information.
For instance if you migrate from an existing virtualisation
framework to uncloud.
The extra_data attribute should be considered a hack and whenever
data is necessary for running uncloud, it should **not** be stored
in there.
"""
extra_data = JSONField(editable=False, blank=True, null=True)
class Meta:
abstract = True
# See https://docs.djangoproject.com/en/dev/ref/models/fields/#field-choices-enum-types
class UncloudStatus(models.TextChoices):
PENDING = 'PENDING', _('Pending')
AWAITING_PAYMENT = 'AWAITING_PAYMENT', _('Awaiting payment')
BEING_CREATED = 'BEING_CREATED', _('Being created')
SCHEDULED = 'SCHEDULED', _('Scheduled') # resource selected, waiting for dispatching
ACTIVE = 'ACTIVE', _('Active')
MODIFYING = 'MODIFYING', _('Modifying') # Resource is being changed
DELETED = 'DELETED', _('Deleted') # Resource has been deleted
DISABLED = 'DISABLED', _('Disabled') # Is usable, but cannot be used for new things
UNUSABLE = 'UNUSABLE', _('Unusable'), # Has some kind of error

21
uncloud_django_based/uncloud/uncloud/secrets_sample.py Normal file
View File

@ -0,0 +1,21 @@
from django.core.management.utils import get_random_secret_key
# XML-RPC interface of opennebula
OPENNEBULA_URL = 'https://opennebula.ungleich.ch:2634/RPC2'
# user:pass for accessing opennebula
OPENNEBULA_USER_PASS = 'user:password'
POSTGRESQL_DB_NAME="uncloud"
# See https://django-auth-ldap.readthedocs.io/en/latest/authentication.html
LDAP_ADMIN_DN=""
LDAP_ADMIN_PASSWORD=""
LDAP_SERVER_URI = ""
# Stripe (Credit Card payments)
STRIPE_KEY=""
STRIPE_PUBLIC_KEY=""
# The django secret key
SECRET_KEY=get_random_secret_key()

121
uncloud/settings.py → uncloud_django_based/uncloud/uncloud/settings.py
View File

@ -11,33 +11,43 @@ https://docs.djangoproject.com/en/3.0/ref/settings/
"""
import os
import re
import ldap
from django.core.management.utils import get_random_secret_key
# Uncommitted file with secrets
import uncloud.secrets
from django_auth_ldap.config import LDAPSearch, LDAPSearchUnion
# Uncommitted file with local settings i.e logging
try:
from uncloud.local_settings import LOGGING, DATABASES
except ModuleNotFoundError:
LOGGING = {}
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': uncloud.secrets.POSTGRESQL_DB_NAME,
'HOST': os.environ.get('DATABASE_HOST', '::1'),
'USER': os.environ.get('DATABASE_USER', 'postgres'),
}
}
LOGGING = {}
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = uncloud.secrets.SECRET_KEY
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = []
# Application definition
@ -113,12 +123,7 @@ AUTH_PASSWORD_VALIDATORS = [
################################################################################
# AUTH/LDAP
AUTH_LDAP_SERVER_URI = ""
AUTH_LDAP_BIND_DN = ""
AUTH_LDAP_BIND_PASSWORD = ""
AUTH_LDAP_USER_SEARCH = LDAPSearch("dc=example,dc=com",
ldap.SCOPE_SUBTREE,
"(uid=%(user)s)")
AUTH_LDAP_SERVER_URI = uncloud.secrets.LDAP_SERVER_URI
AUTH_LDAP_USER_ATTR_MAP = {
"first_name": "givenName",
@ -126,6 +131,13 @@ AUTH_LDAP_USER_ATTR_MAP = {
"email": "mail"
}
AUTH_LDAP_BIND_DN = uncloud.secrets.LDAP_ADMIN_DN
AUTH_LDAP_BIND_PASSWORD = uncloud.secrets.LDAP_ADMIN_PASSWORD
AUTH_LDAP_USER_SEARCH = LDAPSearch("dc=ungleich,dc=ch", ldap.SCOPE_SUBTREE, "(uid=%(user)s)")
################################################################################
# AUTH/Django
AUTHENTICATION_BACKENDS = [
@ -146,6 +158,7 @@ REST_FRAMEWORK = {
}
# Internationalization
# https://docs.djangoproject.com/en/3.0/topics/i18n/
@ -164,77 +177,3 @@ USE_TZ = True
# https://docs.djangoproject.com/en/3.0/howto/static-files/
STATIC_URL = '/static/'
STATICFILES_DIRS = [ os.path.join(BASE_DIR, "static") ]
# XML-RPC interface of opennebula
OPENNEBULA_URL = 'https://opennebula.example.com:2634/RPC2'
# user:pass for accessing opennebula
OPENNEBULA_USER_PASS = 'user:password'
# Stripe (Credit Card payments)
STRIPE_KEY=""
STRIPE_PUBLIC_KEY=""
# The django secret key
SECRET_KEY=get_random_secret_key()
ALLOWED_HOSTS = []
# required for hardcopy / pdf rendering: https://github.com/loftylabs/django-hardcopy
CHROME_PATH = '/usr/bin/chromium-browser'
# Username that is created by default and owns the configuration objects
UNCLOUD_ADMIN_NAME = "uncloud-admin"
LOGIN_REDIRECT_URL = '/'
LOGOUT_REDIRECT_URL = '/'
# replace these in local_settings.py
AUTH_LDAP_SERVER_URI = "ldaps://ldap1.example.com,ldaps://ldap2.example.com"
AUTH_LDAP_BIND_DN="uid=django,ou=system,dc=example,dc=com"
AUTH_LDAP_BIND_PASSWORD="a very secure ldap password"
AUTH_LDAP_USER_SEARCH = LDAPSearch("dc=example,dc=com",
ldap.SCOPE_SUBTREE,
"(uid=%(user)s)")
# where to create customers
LDAP_CUSTOMER_DN="ou=customer,dc=example,dc=com"
# def route_task(name, args, kwargs, options, task=None, **kw):
# print(f"{name} - {args} - {kwargs}")
# # if name == 'myapp.tasks.compress_video':
# return {'queue': 'vpn1' }
# # 'exchange_type': 'topic',
# # 'routing_key': 'video.compress'}
# CELERY_TASK_ROUTES = (route_task,)
# CELERY_TASK_ROUTES = {
# '*': {
# 'queue': 'vpn1'
# }
# }
CELERY_BROKER_URL = 'redis://:uncloud.example.com:6379/0'
CELERY_RESULT_BACKEND = 'redis://:uncloud.example.com:6379/0'
CELERY_TASK_ROUTES = {
re.compile(r'.*.tasks.cdist.*'): { 'queue': 'cdist' } # cdist tasks go into cdist queue
}
CELERY_BEAT_SCHEDULE = {
'cleanup_tasks': {
'task': 'uncloud.tasks.cleanup_tasks',
'schedule': 10
}
}
# CELERY_TASK_CREATE_MISSING_QUEUES = False
# Overwrite settings with local settings, if existing
try:
from uncloud.local_settings import *
except (ModuleNotFoundError, ImportError):
pass

88
uncloud_django_based/uncloud/uncloud/urls.py Normal file
View File

@ -0,0 +1,88 @@
"""uncloud URL Configuration
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/3.0/topics/http/urls/
Examples:
Function views
1. Add an import: from my_app import views
2. Add a URL to urlpatterns: path('', views.home, name='home')
Class-based views
1. Add an import: from other_app.views import Home
2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')
Including another URLconf
1. Import the include() function: from django.urls import include, path
2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static
from rest_framework import routers
from rest_framework.schemas import get_schema_view
from opennebula import views as oneviews
from uncloud_auth import views as authviews
from uncloud_net import views as netviews
from uncloud_pay import views as payviews
from uncloud_vm import views as vmviews
from uncloud_service import views as serviceviews
router = routers.DefaultRouter()
# VM
router.register(r'vm/snapshot', vmviews.VMSnapshotProductViewSet, basename='vmsnapshotproduct')
router.register(r'vm/diskimage', vmviews.VMDiskImageProductViewSet, basename='vmdiskimageproduct')
router.register(r'vm/disk', vmviews.VMDiskProductViewSet, basename='vmdiskproduct')
router.register(r'vm/vm', vmviews.VMProductViewSet, basename='vmproduct')
# creates VM from os image
#router.register(r'vm/ipv6onlyvm', vmviews.VMProductViewSet, basename='vmproduct')
# ... AND adds IPv4 mapping
#router.register(r'vm/dualstackvm', vmviews.VMProductViewSet, basename='vmproduct')
# Services
router.register(r'service/matrix', serviceviews.MatrixServiceProductViewSet, basename='matrixserviceproduct')
router.register(r'service/generic', serviceviews.GenericServiceProductViewSet, basename='genericserviceproduct')
# Net
router.register(r'net/vpn', netviews.VPNNetworkViewSet, basename='vpnnet')
router.register(r'net/vpnreservation', netviews.VPNNetworkReservationViewSet, basename='vpnnetreservation')
# Pay
router.register(r'address', payviews.BillingAddressViewSet, basename='address')
router.register(r'bill', payviews.BillViewSet, basename='bill')
router.register(r'order', payviews.OrderViewSet, basename='order')
router.register(r'payment', payviews.PaymentViewSet, basename='payment')
router.register(r'payment-method', payviews.PaymentMethodViewSet, basename='payment-method')
# admin/staff urls
router.register(r'admin/bill', payviews.AdminBillViewSet, basename='admin/bill')
router.register(r'admin/payment', payviews.AdminPaymentViewSet, basename='admin/payment')
router.register(r'admin/order', payviews.AdminOrderViewSet, basename='admin/order')
router.register(r'admin/vmhost', vmviews.VMHostViewSet)
router.register(r'admin/vmcluster', vmviews.VMClusterViewSet)
router.register(r'admin/vpnpool', netviews.VPNPoolViewSet)
router.register(r'admin/opennebula', oneviews.VMViewSet, basename='opennebula')
# User/Account
router.register(r'user', authviews.UserViewSet, basename='user')
urlpatterns = [
path('', include(router.urls)),
# web/ = stuff to view in the browser
path('web/pdf/', payviews.MyPDFView.as_view(), name='pdf'),
path('api-auth/', include('rest_framework.urls', namespace='rest_framework')), # for login to REST API
path('openapi', get_schema_view(
title="uncloud",
description="uncloud API",
version="1.0.0"
), name='openapi-schema'),
]

0
uncloud/wsgi.py → uncloud_django_based/uncloud/uncloud/wsgi.py
View File

0
archive/uncloud_etcd_based/test/__init__.py → uncloud_django_based/uncloud/uncloud_auth/__init__.py
View File

0
uncloud_auth/admin.py → uncloud_django_based/uncloud/uncloud_auth/admin.py
View File

0
uncloud_auth/apps.py → uncloud_django_based/uncloud/uncloud_auth/apps.py
View File

8
uncloud_auth/migrations/0001_initial.py → uncloud_django_based/uncloud/uncloud_auth/migrations/0001_initial.py
View File

@ -1,8 +1,7 @@
# Generated by Django 3.1 on 2020-12-13 10:38
# Generated by Django 3.0.3 on 2020-03-03 16:49
import django.contrib.auth.models
import django.contrib.auth.validators
import django.core.validators
from django.db import migrations, models
import django.utils.timezone
@ -12,7 +11,7 @@ class Migration(migrations.Migration):
initial = True
dependencies = [
('auth', '0012_alter_user_first_name_max_length'),
('auth', '0011_update_proxy_permissions'),
]
operations = [
@ -24,13 +23,12 @@ class Migration(migrations.Migration):
('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')),
('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')),
('username', models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.', max_length=150, unique=True, validators=[django.contrib.auth.validators.UnicodeUsernameValidator()], verbose_name='username')),
('first_name', models.CharField(blank=True, max_length=150, verbose_name='first name')),
('first_name', models.CharField(blank=True, max_length=30, verbose_name='first name')),
('last_name', models.CharField(blank=True, max_length=150, verbose_name='last name')),
('email', models.EmailField(blank=True, max_length=254, verbose_name='email address')),
('is_staff', models.BooleanField(default=False, help_text='Designates whether the user can log into this admin site.', verbose_name='staff status')),
('is_active', models.BooleanField(default=True, help_text='Designates whether this user should be treated as active. Unselect this instead of deleting accounts.', verbose_name='active')),
('date_joined', models.DateTimeField(default=django.utils.timezone.now, verbose_name='date joined')),
('maximum_credit', models.DecimalField(decimal_places=2, default=0.0, max_digits=10, validators=[django.core.validators.MinValueValidator(0)])),
('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.Group', verbose_name='groups')),
('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.Permission', verbose_name='user permissions')),
],

25
uncloud_django_based/uncloud/uncloud_auth/migrations/0002_auto_20200318_1343.py Normal file
View File

@ -0,0 +1,25 @@
# Generated by Django 3.0.3 on 2020-03-18 13:43
import django.core.validators
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('uncloud_auth', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='user',
name='amount',
field=models.DecimalField(decimal_places=2, default=0.0, max_digits=10, validators=[django.core.validators.MinValueValidator(0)]),
),
migrations.AddField(
model_name='user',
name='maximum_credit',
field=models.FloatField(default=0),
preserve_default=False,
),
]

23
uncloud_django_based/uncloud/uncloud_auth/migrations/0003_auto_20200318_1345.py Normal file
View File

@ -0,0 +1,23 @@
# Generated by Django 3.0.3 on 2020-03-18 13:45
import django.core.validators
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('uncloud_auth', '0002_auto_20200318_1343'),
]
operations = [
migrations.RemoveField(
model_name='user',
name='amount',
),
migrations.AlterField(
model_name='user',
name='maximum_credit',
field=models.DecimalField(decimal_places=2, default=0.0, max_digits=10, validators=[django.core.validators.MinValueValidator(0)]),
),
]

0
archive/uncloud_etcd_based/uncloud/cli/__init__.py → uncloud_django_based/uncloud/uncloud_auth/migrations/__init__.py
View File

6
uncloud_auth/models.py → uncloud_django_based/uncloud/uncloud_auth/models.py
View File

@ -4,6 +4,8 @@ from django.core.validators import MinValueValidator
from uncloud import AMOUNT_DECIMALS, AMOUNT_MAX_DIGITS
from uncloud_pay.models import get_balance_for_user
class User(AbstractUser):
"""
We use the standard user and add a maximum credit that is allowed
@ -15,3 +17,7 @@ class User(AbstractUser):
max_digits=AMOUNT_MAX_DIGITS,
decimal_places=AMOUNT_DECIMALS,
validators=[MinValueValidator(0)])
@property
def balance(self):
return get_balance_for_user(self)

15
uncloud_django_based/uncloud/uncloud_auth/serializers.py Normal file
View File

@ -0,0 +1,15 @@
from django.contrib.auth import get_user_model
from rest_framework import serializers
from uncloud import AMOUNT_DECIMALS, AMOUNT_MAX_DIGITS
class UserSerializer(serializers.ModelSerializer):
class Meta:
model = get_user_model()
fields = ['username', 'email', 'balance', 'maximum_credit' ]
balance = serializers.DecimalField(max_digits=AMOUNT_MAX_DIGITS,
decimal_places=AMOUNT_DECIMALS)

17
uncloud_django_based/uncloud/uncloud_auth/views.py Normal file
View File

@ -0,0 +1,17 @@
from rest_framework import viewsets, permissions, status
from .serializers import *
class UserViewSet(viewsets.ReadOnlyModelViewSet):
serializer_class = UserSerializer
permission_classes = [permissions.IsAuthenticated]
def get_queryset(self):
if self.request.user.is_superuser:
obj = get_user_model().objects.all()
else:
# This is a bit stupid: we have a user, we create a queryset by
# matching on the username. But I don't know a "nicer" way.
# Nico, 2020-03-18
obj = get_user_model().objects.filter(username=self.request.user.username)
return obj

0
archive/uncloud_etcd_based/uncloud/client/__init__.py → uncloud_django_based/uncloud/uncloud_net/__init__.py
View File

Some files were not shown because too many files have changed in this diff Show More