diff --git a/doc/uncloud-manual-2020-08-01.org b/doc/uncloud-manual-2020-08-01.org
index 6e8e4be..5c6a9f7 100644
--- a/doc/uncloud-manual-2020-08-01.org
+++ b/doc/uncloud-manual-2020-08-01.org
@@ -1,8 +1,15 @@
* Bootstrap / Installation
** Pre-requisites by operating system
+*** General
+ To run uncloud you need:
+ - ldap development libraries
+ - libxml2-dev libxslt-dev
+ - gcc / libc headers: for compiling things
+ - python3-dev
+ - wireguard: wg (for checking keys)
*** Alpine
#+BEGIN_SRC sh
-apk add openldap-dev postgresql-dev libxml2-dev libxslt-dev
+apk add openldap-dev postgresql-dev libxml2-dev libxslt-dev gcc python3-dev musl-dev wireguard-tools-wg
#+END_SRC
*** Debian/Devuan:
#+BEGIN_SRC sh
@@ -60,6 +67,21 @@ python manage.py migrate
python manage.py bootstrap-user --username nicocustomer
#+END_SRC
+** Initialise the database
+ While it is not strictly required to add default values to the
+ database, it might significantly reduce the starting time with
+ uncloud.
+
+ To add the default database values run:
+
+ #+BEGIN_SRC shell
+ # Add local objects
+ python manage.py db-add-defaults
+
+ # Import VAT rates
+ python manage.py import-vat-rates
+ #+END_SRC
+
* Testing / CLI Access
Access via the commandline (CLI) can be done using curl or
httpie. In our examples we will use httpie.
@@ -84,6 +106,14 @@ python manage.py migrate
* URLs
- api/ - the rest API
* uncloud Products
+** Product features
+ - Dependencies on other products
+ - Minimum parameters (min cpu, min ram, etc).
+ - Can also realise the dcl vm
+ - dualstack vm = VM + IPv4 + SSD
+ - Need to have a non-misguiding name for the "bare VM"
+ - Should support network boot (?)
+
** VPN
*** How to add a new VPN Host
**** Install wireguard to the host
@@ -95,8 +125,8 @@ python manage.py migrate
**** Add it to DNS as vpn-XXX.ungleich.ch
**** Route a /40 network to its IPv6 address
**** Install wireguard on it
-**** TODO Enable wireguard on boot
-**** TODO Create a new VPNPool on uncloud with
+**** TODO [#C] Enable wireguard on boot
+**** TODO [#C] Create a new VPNPool on uncloud with
***** the network address (selecting from our existing pool)
***** the network size (/...)
***** the vpn host that provides the network (selecting the created VM)
@@ -119,7 +149,6 @@ python manage.py migrate
vpn_hostname=vpn-2a0ae5c1200.ungleich.ch wireguard_private_key=$(wg
genkey)
**** Creating a new vpn network
-** VPN
*** Creating a VPN pool
#+BEGIN_SRC sh
@@ -148,7 +177,7 @@ VPNNetworks can be managed by all authenticated users.
* Developer Handbook
The following section describe decisions / architecture of
uncloud. These chapters are intended to be read by developers.
-** Documentation
+** This Documentation
This documentation is written in org-mode. To compile it to
html/pdf, just open emacs and press *C-c C-e l p*.
** Models
@@ -211,3 +240,143 @@ VPNNetworks can be managed by all authenticated users.
*** Decision
We use integers, because they are easy.
+
+** Distributing/Dispatching/Orchestrating
+*** Variant 1: using cdist
+ - The uncloud server can git commit things
+ - The uncloud server loads cdist and configures the server
+ - Advantages
+ - Fully integrated into normal flow
+ - Disadvantage
+ - web frontend has access to more data than it needs
+ - On compromise of the machine, more data leaks
+ - Some cdist usual delay
+*** Variant 2: via celery
+ - The uncloud server dispatches via celery
+ - Every decentral node also runs celery/connects to the broker
+ - Summary brokers:
+ - If local only celery -> good to use redis - Broker
+ - If remote: probably better to use rabbitmq
+ - redis
+ - simpler
+ - rabbitmq
+ - more versatile
+ - made for remote connections
+ - quorom queues would be nice, but not clear if supported
+ - https://github.com/celery/py-amqp/issues/302
+ - https://github.com/celery/celery/issues/6067
+ - Cannot be installed on alpine Linux at the moment
+ - Advantage
+ - Very python / django integrated
+ - Rather instant
+ - Disadvantages
+ - Every decentral node needs to have the uncloud code available
+ - Decentral nodes *might* need to access the database
+ - Tasks can probably be written to work without that
+ (i.e. only strings/bytes)
+
+**** log/tests
+ (venv) [19:54] vpn-2a0ae5c1200:~/uncloud$ celery -A uncloud -b redis://bridge.place7.ungleich.ch worker -n worker1@%h --logfile ~/celery.log -
+Q vpn-2a0ae5c1200.ungleich.ch
+
+
+*** Variant 3: dedicated cdist instance via message broker
+ - A separate VM/machine
+ - Has Checkout of ~/.cdist
+ - Has cdist checkout
+ - Tiny API for management
+ - Not directly web accessible
+ - "cdist" queue
+
+** Milestones :uncloud:
+*** 1.1 (cleanup 1)
+**** TODO [#C] Unify ValidationError, FieldError - define proper Exception
+ - What do we use for model errors
+*** 1.0 (initial release)
+**** TODO [#C] Initial Generic product support
+ - Product
+***** TODO [#C] Recurring product support
+****** TODO [#C] Support replacing orders for updates
+****** DONE [#A] Finish split of bill creation
+ CLOSED: [2020-09-11 Fri 23:19]
+****** TODO [#C] Test the new functions in the Order class
+****** Define the correct order replacement logic
+ Assumption:
+ - recurringperiods are 30days
+******* Case 1: downgrading
+ - User commits to 10 CHF for 30 days
+ - Wants to downgrade after 15 days to 5 CHF product
+ - Expected result:
+ - order 1: 10 CHF until +30days
+ - order 2: 5 CHF starting 30days + 1s
+ - Sum of the two orders is 15 CHF
+ - Question is
+ - when is the VM shutdown?
+ - a) instantly
+ - b) at the end of the cycle
+ - best solution
+ - user can choose between a ... b any time
+******* Duration
+ - You cannot cancel the duration
+ - You can upgrade and with that cancel the duration
+ - The idea of a duration is that you commit for it
+ - If you want to commit lower (daily basis for instance) you
+ have higher per period prices
+******* Case X
+ - User has VM with 2 Core / 2 GB RAM
+ - User modifies with to 1 core / 3 GB RAM
+ - We treat it as down/upgrade independent of the modifications
+
+******* Case 2: upgrading after 1 day
+ - committed for 30 days
+ - upgrade after 1 day
+ - so first order will be charged for 1/30ths
+
+******* Case 2: upgrading
+ - User commits to 10 CHF for 30 days
+ - Wants to upgrade after 15 days to 20 CHF product
+ - Order 1 : 1 VM with 2 Core / 2 GB / 10 SSD -- 10 CHF
+ - 30days period, stopped after 15, so quantity is 0.5 = 5 CHF
+ - Order 2 : 1 VM with 2 Core / 6 GB / 10 SSD -- 20 CHF
+ - after 15 days
+ - VM is upgraded instantly
+ - Expected result:
+ - order 1: 10 CHF until +15days = 0.5 units = 5 CHF
+ - order 2: 20 CHF starting 15days + 1s ... +30 days after
+ the 15 days -> 45 days = 1 unit = 20 CHF
+ - Total on bill: 25 CHF
+
+******* Case 2: upgrading
+ - User commits to 10 CHF for 30 days
+ - Wants to upgrade after 15 days to 20 CHF product
+ - Expected result:
+ - order 1: 10 CHF until +30days = 1 units = 10 CHF
+
+ - order 2: 20 CHF starting 15days + 1s = 1 unit = 20 CHF
+ - Total on bill: 30 CHF
+
+
+****** TODO [#C] Note: ending date not set if replaced by default (implicit!)
+ - Should the new order modify the old order on save()?
+****** DONE Fix totally wrong bill dates in our test case
+ CLOSED: [2020-09-09 Wed 01:00]
+ - 2020 used instead of 2019
+ - Was due to existing test data ...
+***** DONE Bill logic is still wrong
+ CLOSED: [2020-11-05 Thu 18:58]
+ - Bill starting_date is the date of the first order
+ - However first encountered order does not have to be the
+ earliest in the bill!
+ - Bills should not have a duration
+ - Bills should only have a (unique) issue date
+ - We charge based on bill_records
+ - Last time charged issue date of the bill OR earliest date
+ after that
+ - Every bill generation checks all (relevant) orders
+ - add a flag "not_for_billing" or "closed"
+ - query on that flag
+ - verify it every time
+
+
+***** TODO Generating bill for admins/staff
+ -
diff --git a/opennebula/migrations/0001_initial.py b/opennebula/migrations/0001_initial.py
index 0852436..9a135c6 100644
--- a/opennebula/migrations/0001_initial.py
+++ b/opennebula/migrations/0001_initial.py
@@ -1,9 +1,6 @@
-# Generated by Django 3.0.6 on 2020-08-01 16:38
+# Generated by Django 3.1 on 2020-12-13 10:38
-from django.conf import settings
-import django.contrib.postgres.fields.jsonb
from django.db import migrations, models
-import django.db.models.deletion
class Migration(migrations.Migration):
@@ -11,23 +8,14 @@ class Migration(migrations.Migration):
initial = True
dependencies = [
- migrations.swappable_dependency(settings.AUTH_USER_MODEL),
- ('uncloud_pay', '0001_initial'),
]
operations = [
migrations.CreateModel(
name='VM',
fields=[
- ('extra_data', django.contrib.postgres.fields.jsonb.JSONField(blank=True, editable=False, null=True)),
- ('status', models.CharField(choices=[('PENDING', 'Pending'), ('AWAITING_PAYMENT', 'Awaiting payment'), ('BEING_CREATED', 'Being created'), ('SCHEDULED', 'Scheduled'), ('ACTIVE', 'Active'), ('MODIFYING', 'Modifying'), ('DELETED', 'Deleted'), ('DISABLED', 'Disabled'), ('UNUSABLE', 'Unusable')], default='AWAITING_PAYMENT', max_length=32)),
('vmid', models.IntegerField(primary_key=True, serialize=False)),
- ('data', django.contrib.postgres.fields.jsonb.JSONField()),
- ('order', models.ForeignKey(editable=False, null=True, on_delete=django.db.models.deletion.CASCADE, to='uncloud_pay.Order')),
- ('owner', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
+ ('data', models.JSONField()),
],
- options={
- 'abstract': False,
- },
),
]
diff --git a/opennebula/migrations/0002_auto_20200801_2332.py b/opennebula/migrations/0002_auto_20200801_2332.py
deleted file mode 100644
index 1aa6d34..0000000
--- a/opennebula/migrations/0002_auto_20200801_2332.py
+++ /dev/null
@@ -1,20 +0,0 @@
-# Generated by Django 3.0.8 on 2020-08-01 23:32
-
-from django.db import migrations, models
-import django.db.models.deletion
-
-
-class Migration(migrations.Migration):
-
- dependencies = [
- ('uncloud_pay', '0003_auto_20200801_2332'),
- ('opennebula', '0001_initial'),
- ]
-
- operations = [
- migrations.AlterField(
- model_name='vm',
- name='order',
- field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, to='uncloud_pay.Order'),
- ),
- ]
diff --git a/opennebula/migrations/0003_auto_20200808_1953.py b/opennebula/migrations/0003_auto_20200808_1953.py
deleted file mode 100644
index 218b9a7..0000000
--- a/opennebula/migrations/0003_auto_20200808_1953.py
+++ /dev/null
@@ -1,23 +0,0 @@
-# Generated by Django 3.1 on 2020-08-08 19:53
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
- dependencies = [
- ('opennebula', '0002_auto_20200801_2332'),
- ]
-
- operations = [
- migrations.AlterField(
- model_name='vm',
- name='data',
- field=models.JSONField(),
- ),
- migrations.AlterField(
- model_name='vm',
- name='extra_data',
- field=models.JSONField(blank=True, editable=False, null=True),
- ),
- ]
diff --git a/opennebula/models.py b/opennebula/models.py
index 6c7dc52..f15b845 100644
--- a/opennebula/models.py
+++ b/opennebula/models.py
@@ -10,7 +10,7 @@ storage_class_mapping = {
'hdd': 'hdd'
}
-class VM(Product):
+class VM(models.Model):
vmid = models.IntegerField(primary_key=True)
data = models.JSONField()
diff --git a/opennebula/views.py b/opennebula/views.py
index 89b1a52..688f0b4 100644
--- a/opennebula/views.py
+++ b/opennebula/views.py
@@ -1,16 +1,16 @@
from rest_framework import viewsets, permissions
-from .models import VM
-from .serializers import OpenNebulaVMSerializer
+#from .models import VM
+# from .serializers import OpenNebulaVMSerializer
-class VMViewSet(viewsets.ModelViewSet):
- permission_classes = [permissions.IsAuthenticated]
- serializer_class = OpenNebulaVMSerializer
+# class VMViewSet(viewsets.ModelViewSet):
+# permission_classes = [permissions.IsAuthenticated]
+# serializer_class = OpenNebulaVMSerializer
- def get_queryset(self):
- if self.request.user.is_superuser:
- obj = VM.objects.all()
- else:
- obj = VM.objects.filter(owner=self.request.user)
+# def get_queryset(self):
+# if self.request.user.is_superuser:
+# obj = VM.objects.all()
+# else:
+# obj = VM.objects.filter(owner=self.request.user)
- return obj
+# return obj
diff --git a/requirements.txt b/requirements.txt
index a7fc9f2..adbda9c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,9 +1,12 @@
+# Django basics
django
djangorestframework
django-auth-ldap
-stripe
-xmltodict
+
psycopg2
+ldap3
+
+xmltodict
parsedatetime
@@ -19,6 +22,11 @@ django-hardcopy
pyyaml
uritemplate
-# Comprehensive interface to validate VAT numbers, making use of the VIES
-# service for European countries.
+# Payment & VAT
vat-validator
+stripe
+
+
+# Tasks
+celery
+redis
diff --git a/uncloud/.gitignore b/uncloud/.gitignore
index 6a07bff..b03e0a5 100644
--- a/uncloud/.gitignore
+++ b/uncloud/.gitignore
@@ -1 +1,2 @@
local_settings.py
+ldap_max_uid_file
\ No newline at end of file
diff --git a/uncloud/__init__.py b/uncloud/__init__.py
index e69de29..e073dd5 100644
--- a/uncloud/__init__.py
+++ b/uncloud/__init__.py
@@ -0,0 +1,254 @@
+from django.utils.translation import gettext_lazy as _
+import decimal
+from .celery import app as celery_app
+
+# Define DecimalField properties, used to represent amounts of money.
+AMOUNT_MAX_DIGITS=10
+AMOUNT_DECIMALS=2
+
+decimal.getcontext().prec = AMOUNT_DECIMALS
+
+# http://xml.coverpages.org/country3166.html
+COUNTRIES = (
+ ('AD', _('Andorra')),
+ ('AE', _('United Arab Emirates')),
+ ('AF', _('Afghanistan')),
+ ('AG', _('Antigua & Barbuda')),
+ ('AI', _('Anguilla')),
+ ('AL', _('Albania')),
+ ('AM', _('Armenia')),
+ ('AN', _('Netherlands Antilles')),
+ ('AO', _('Angola')),
+ ('AQ', _('Antarctica')),
+ ('AR', _('Argentina')),
+ ('AS', _('American Samoa')),
+ ('AT', _('Austria')),
+ ('AU', _('Australia')),
+ ('AW', _('Aruba')),
+ ('AZ', _('Azerbaijan')),
+ ('BA', _('Bosnia and Herzegovina')),
+ ('BB', _('Barbados')),
+ ('BD', _('Bangladesh')),
+ ('BE', _('Belgium')),
+ ('BF', _('Burkina Faso')),
+ ('BG', _('Bulgaria')),
+ ('BH', _('Bahrain')),
+ ('BI', _('Burundi')),
+ ('BJ', _('Benin')),
+ ('BM', _('Bermuda')),
+ ('BN', _('Brunei Darussalam')),
+ ('BO', _('Bolivia')),
+ ('BR', _('Brazil')),
+ ('BS', _('Bahama')),
+ ('BT', _('Bhutan')),
+ ('BV', _('Bouvet Island')),
+ ('BW', _('Botswana')),
+ ('BY', _('Belarus')),
+ ('BZ', _('Belize')),
+ ('CA', _('Canada')),
+ ('CC', _('Cocos (Keeling) Islands')),
+ ('CF', _('Central African Republic')),
+ ('CG', _('Congo')),
+ ('CH', _('Switzerland')),
+ ('CI', _('Ivory Coast')),
+ ('CK', _('Cook Iislands')),
+ ('CL', _('Chile')),
+ ('CM', _('Cameroon')),
+ ('CN', _('China')),
+ ('CO', _('Colombia')),
+ ('CR', _('Costa Rica')),
+ ('CU', _('Cuba')),
+ ('CV', _('Cape Verde')),
+ ('CX', _('Christmas Island')),
+ ('CY', _('Cyprus')),
+ ('CZ', _('Czech Republic')),
+ ('DE', _('Germany')),
+ ('DJ', _('Djibouti')),
+ ('DK', _('Denmark')),
+ ('DM', _('Dominica')),
+ ('DO', _('Dominican Republic')),
+ ('DZ', _('Algeria')),
+ ('EC', _('Ecuador')),
+ ('EE', _('Estonia')),
+ ('EG', _('Egypt')),
+ ('EH', _('Western Sahara')),
+ ('ER', _('Eritrea')),
+ ('ES', _('Spain')),
+ ('ET', _('Ethiopia')),
+ ('FI', _('Finland')),
+ ('FJ', _('Fiji')),
+ ('FK', _('Falkland Islands (Malvinas)')),
+ ('FM', _('Micronesia')),
+ ('FO', _('Faroe Islands')),
+ ('FR', _('France')),
+ ('FX', _('France, Metropolitan')),
+ ('GA', _('Gabon')),
+ ('GB', _('United Kingdom (Great Britain)')),
+ ('GD', _('Grenada')),
+ ('GE', _('Georgia')),
+ ('GF', _('French Guiana')),
+ ('GH', _('Ghana')),
+ ('GI', _('Gibraltar')),
+ ('GL', _('Greenland')),
+ ('GM', _('Gambia')),
+ ('GN', _('Guinea')),
+ ('GP', _('Guadeloupe')),
+ ('GQ', _('Equatorial Guinea')),
+ ('GR', _('Greece')),
+ ('GS', _('South Georgia and the South Sandwich Islands')),
+ ('GT', _('Guatemala')),
+ ('GU', _('Guam')),
+ ('GW', _('Guinea-Bissau')),
+ ('GY', _('Guyana')),
+ ('HK', _('Hong Kong')),
+ ('HM', _('Heard & McDonald Islands')),
+ ('HN', _('Honduras')),
+ ('HR', _('Croatia')),
+ ('HT', _('Haiti')),
+ ('HU', _('Hungary')),
+ ('ID', _('Indonesia')),
+ ('IE', _('Ireland')),
+ ('IL', _('Israel')),
+ ('IN', _('India')),
+ ('IO', _('British Indian Ocean Territory')),
+ ('IQ', _('Iraq')),
+ ('IR', _('Islamic Republic of Iran')),
+ ('IS', _('Iceland')),
+ ('IT', _('Italy')),
+ ('JM', _('Jamaica')),
+ ('JO', _('Jordan')),
+ ('JP', _('Japan')),
+ ('KE', _('Kenya')),
+ ('KG', _('Kyrgyzstan')),
+ ('KH', _('Cambodia')),
+ ('KI', _('Kiribati')),
+ ('KM', _('Comoros')),
+ ('KN', _('St. Kitts and Nevis')),
+ ('KP', _('Korea, Democratic People\'s Republic of')),
+ ('KR', _('Korea, Republic of')),
+ ('KW', _('Kuwait')),
+ ('KY', _('Cayman Islands')),
+ ('KZ', _('Kazakhstan')),
+ ('LA', _('Lao People\'s Democratic Republic')),
+ ('LB', _('Lebanon')),
+ ('LC', _('Saint Lucia')),
+ ('LI', _('Liechtenstein')),
+ ('LK', _('Sri Lanka')),
+ ('LR', _('Liberia')),
+ ('LS', _('Lesotho')),
+ ('LT', _('Lithuania')),
+ ('LU', _('Luxembourg')),
+ ('LV', _('Latvia')),
+ ('LY', _('Libyan Arab Jamahiriya')),
+ ('MA', _('Morocco')),
+ ('MC', _('Monaco')),
+ ('MD', _('Moldova, Republic of')),
+ ('MG', _('Madagascar')),
+ ('MH', _('Marshall Islands')),
+ ('ML', _('Mali')),
+ ('MN', _('Mongolia')),
+ ('MM', _('Myanmar')),
+ ('MO', _('Macau')),
+ ('MP', _('Northern Mariana Islands')),
+ ('MQ', _('Martinique')),
+ ('MR', _('Mauritania')),
+ ('MS', _('Monserrat')),
+ ('MT', _('Malta')),
+ ('MU', _('Mauritius')),
+ ('MV', _('Maldives')),
+ ('MW', _('Malawi')),
+ ('MX', _('Mexico')),
+ ('MY', _('Malaysia')),
+ ('MZ', _('Mozambique')),
+ ('NA', _('Namibia')),
+ ('NC', _('New Caledonia')),
+ ('NE', _('Niger')),
+ ('NF', _('Norfolk Island')),
+ ('NG', _('Nigeria')),
+ ('NI', _('Nicaragua')),
+ ('NL', _('Netherlands')),
+ ('NO', _('Norway')),
+ ('NP', _('Nepal')),
+ ('NR', _('Nauru')),
+ ('NU', _('Niue')),
+ ('NZ', _('New Zealand')),
+ ('OM', _('Oman')),
+ ('PA', _('Panama')),
+ ('PE', _('Peru')),
+ ('PF', _('French Polynesia')),
+ ('PG', _('Papua New Guinea')),
+ ('PH', _('Philippines')),
+ ('PK', _('Pakistan')),
+ ('PL', _('Poland')),
+ ('PM', _('St. Pierre & Miquelon')),
+ ('PN', _('Pitcairn')),
+ ('PR', _('Puerto Rico')),
+ ('PT', _('Portugal')),
+ ('PW', _('Palau')),
+ ('PY', _('Paraguay')),
+ ('QA', _('Qatar')),
+ ('RE', _('Reunion')),
+ ('RO', _('Romania')),
+ ('RU', _('Russian Federation')),
+ ('RW', _('Rwanda')),
+ ('SA', _('Saudi Arabia')),
+ ('SB', _('Solomon Islands')),
+ ('SC', _('Seychelles')),
+ ('SD', _('Sudan')),
+ ('SE', _('Sweden')),
+ ('SG', _('Singapore')),
+ ('SH', _('St. Helena')),
+ ('SI', _('Slovenia')),
+ ('SJ', _('Svalbard & Jan Mayen Islands')),
+ ('SK', _('Slovakia')),
+ ('SL', _('Sierra Leone')),
+ ('SM', _('San Marino')),
+ ('SN', _('Senegal')),
+ ('SO', _('Somalia')),
+ ('SR', _('Suriname')),
+ ('ST', _('Sao Tome & Principe')),
+ ('SV', _('El Salvador')),
+ ('SY', _('Syrian Arab Republic')),
+ ('SZ', _('Swaziland')),
+ ('TC', _('Turks & Caicos Islands')),
+ ('TD', _('Chad')),
+ ('TF', _('French Southern Territories')),
+ ('TG', _('Togo')),
+ ('TH', _('Thailand')),
+ ('TJ', _('Tajikistan')),
+ ('TK', _('Tokelau')),
+ ('TM', _('Turkmenistan')),
+ ('TN', _('Tunisia')),
+ ('TO', _('Tonga')),
+ ('TP', _('East Timor')),
+ ('TR', _('Turkey')),
+ ('TT', _('Trinidad & Tobago')),
+ ('TV', _('Tuvalu')),
+ ('TW', _('Taiwan, Province of China')),
+ ('TZ', _('Tanzania, United Republic of')),
+ ('UA', _('Ukraine')),
+ ('UG', _('Uganda')),
+ ('UM', _('United States Minor Outlying Islands')),
+ ('US', _('United States of America')),
+ ('UY', _('Uruguay')),
+ ('UZ', _('Uzbekistan')),
+ ('VA', _('Vatican City State (Holy See)')),
+ ('VC', _('St. Vincent & the Grenadines')),
+ ('VE', _('Venezuela')),
+ ('VG', _('British Virgin Islands')),
+ ('VI', _('United States Virgin Islands')),
+ ('VN', _('Viet Nam')),
+ ('VU', _('Vanuatu')),
+ ('WF', _('Wallis & Futuna Islands')),
+ ('WS', _('Samoa')),
+ ('YE', _('Yemen')),
+ ('YT', _('Mayotte')),
+ ('YU', _('Yugoslavia')),
+ ('ZA', _('South Africa')),
+ ('ZM', _('Zambia')),
+ ('ZR', _('Zaire')),
+ ('ZW', _('Zimbabwe')),
+)
+
+
+__all__ = ('celery_app',)
diff --git a/uncloud/admin.py b/uncloud/admin.py
new file mode 100644
index 0000000..a89a574
--- /dev/null
+++ b/uncloud/admin.py
@@ -0,0 +1,6 @@
+from django.contrib import admin
+
+from .models import *
+
+for m in [ UncloudProvider, UncloudNetwork, UncloudTask ]:
+ admin.site.register(m)
diff --git a/uncloud/celery.py b/uncloud/celery.py
new file mode 100644
index 0000000..3408634
--- /dev/null
+++ b/uncloud/celery.py
@@ -0,0 +1,17 @@
+import os
+
+from celery import Celery
+
+# set the default Django settings module for the 'celery' program.
+os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'uncloud.settings')
+
+app = Celery('uncloud')
+
+# Using a string here means the worker doesn't have to serialize
+# the configuration object to child processes.
+# - namespace='CELERY' means all celery-related configuration keys
+# should have a `CELERY_` prefix.
+app.config_from_object('django.conf:settings', namespace='CELERY')
+
+# Load task modules from all registered Django app configs.
+app.autodiscover_tasks()
diff --git a/uncloud/management/commands/db-add-defaults.py b/uncloud/management/commands/db-add-defaults.py
new file mode 100644
index 0000000..605c8f5
--- /dev/null
+++ b/uncloud/management/commands/db-add-defaults.py
@@ -0,0 +1,43 @@
+import random
+import string
+
+from django.core.management.base import BaseCommand
+from django.core.exceptions import ObjectDoesNotExist
+from django.contrib.auth import get_user_model
+from django.conf import settings
+
+from uncloud_pay.models import BillingAddress, RecurringPeriod, Product
+from uncloud.models import UncloudProvider, UncloudNetwork
+
+
+class Command(BaseCommand):
+ help = 'Add standard uncloud values'
+
+ def add_arguments(self, parser):
+ pass
+
+ def handle(self, *args, **options):
+ # Order matters, objects can be dependent on each other
+
+ admin_username="uncloud-admin"
+ pw_length = 32
+
+ # Only set password if the user did not exist before
+ try:
+ admin_user = get_user_model().objects.get(username=settings.UNCLOUD_ADMIN_NAME)
+ except ObjectDoesNotExist:
+ random_password = ''.join(random.SystemRandom().choice(string.ascii_lowercase + string.digits) for _ in range(pw_length))
+
+ admin_user = get_user_model().objects.create_user(username=settings.UNCLOUD_ADMIN_NAME, password=random_password)
+ admin_user.is_superuser=True
+ admin_user.is_staff=True
+ admin_user.save()
+
+ print(f"Created admin user '{admin_username}' with password '{random_password}'")
+
+ BillingAddress.populate_db_defaults()
+ RecurringPeriod.populate_db_defaults()
+ Product.populate_db_defaults()
+
+ UncloudNetwork.populate_db_defaults()
+ UncloudProvider.populate_db_defaults()
diff --git a/uncloud/migrations/0001_initial.py b/uncloud/migrations/0001_initial.py
new file mode 100644
index 0000000..10d1144
--- /dev/null
+++ b/uncloud/migrations/0001_initial.py
@@ -0,0 +1,46 @@
+# Generated by Django 3.1 on 2020-12-13 10:38
+
+import django.core.validators
+from django.db import migrations, models
+import django.db.models.deletion
+import uncloud.models
+
+
+class Migration(migrations.Migration):
+
+ initial = True
+
+ dependencies = [
+ ]
+
+ operations = [
+ migrations.CreateModel(
+ name='UncloudNetwork',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('network_address', models.GenericIPAddressField(unique=True)),
+ ('network_mask', models.IntegerField(validators=[django.core.validators.MinValueValidator(0), django.core.validators.MaxValueValidator(128)])),
+ ('description', models.CharField(max_length=256)),
+ ],
+ ),
+ migrations.CreateModel(
+ name='UncloudProvider',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('full_name', models.CharField(max_length=256)),
+ ('organization', models.CharField(blank=True, max_length=256, null=True)),
+ ('street', models.CharField(max_length=256)),
+ ('city', models.CharField(max_length=256)),
+ ('postal_code', models.CharField(max_length=64)),
+ ('country', uncloud.models.CountryField(blank=True, choices=[('AD', 'Andorra'), ('AE', 'United Arab Emirates'), ('AF', 'Afghanistan'), ('AG', 'Antigua & Barbuda'), ('AI', 'Anguilla'), ('AL', 'Albania'), ('AM', 'Armenia'), ('AN', 'Netherlands Antilles'), ('AO', 'Angola'), ('AQ', 'Antarctica'), ('AR', 'Argentina'), ('AS', 'American Samoa'), ('AT', 'Austria'), ('AU', 'Australia'), ('AW', 'Aruba'), ('AZ', 'Azerbaijan'), ('BA', 'Bosnia and Herzegovina'), ('BB', 'Barbados'), ('BD', 'Bangladesh'), ('BE', 'Belgium'), ('BF', 'Burkina Faso'), ('BG', 'Bulgaria'), ('BH', 'Bahrain'), ('BI', 'Burundi'), ('BJ', 'Benin'), ('BM', 'Bermuda'), ('BN', 'Brunei Darussalam'), ('BO', 'Bolivia'), ('BR', 'Brazil'), ('BS', 'Bahama'), ('BT', 'Bhutan'), ('BV', 'Bouvet Island'), ('BW', 'Botswana'), ('BY', 'Belarus'), ('BZ', 'Belize'), ('CA', 'Canada'), ('CC', 'Cocos (Keeling) Islands'), ('CF', 'Central African Republic'), ('CG', 'Congo'), ('CH', 'Switzerland'), ('CI', 'Ivory Coast'), ('CK', 'Cook Iislands'), ('CL', 'Chile'), ('CM', 'Cameroon'), ('CN', 'China'), ('CO', 'Colombia'), ('CR', 'Costa Rica'), ('CU', 'Cuba'), ('CV', 'Cape Verde'), ('CX', 'Christmas Island'), ('CY', 'Cyprus'), ('CZ', 'Czech Republic'), ('DE', 'Germany'), ('DJ', 'Djibouti'), ('DK', 'Denmark'), ('DM', 'Dominica'), ('DO', 'Dominican Republic'), ('DZ', 'Algeria'), ('EC', 'Ecuador'), ('EE', 'Estonia'), ('EG', 'Egypt'), ('EH', 'Western Sahara'), ('ER', 'Eritrea'), ('ES', 'Spain'), ('ET', 'Ethiopia'), ('FI', 'Finland'), ('FJ', 'Fiji'), ('FK', 'Falkland Islands (Malvinas)'), ('FM', 'Micronesia'), ('FO', 'Faroe Islands'), ('FR', 'France'), ('FX', 'France, Metropolitan'), ('GA', 'Gabon'), ('GB', 'United Kingdom (Great Britain)'), ('GD', 'Grenada'), ('GE', 'Georgia'), ('GF', 'French Guiana'), ('GH', 'Ghana'), ('GI', 'Gibraltar'), ('GL', 'Greenland'), ('GM', 'Gambia'), ('GN', 'Guinea'), ('GP', 'Guadeloupe'), ('GQ', 'Equatorial Guinea'), ('GR', 'Greece'), ('GS', 'South Georgia and the South Sandwich Islands'), ('GT', 'Guatemala'), ('GU', 'Guam'), ('GW', 'Guinea-Bissau'), ('GY', 'Guyana'), ('HK', 'Hong Kong'), ('HM', 'Heard & McDonald Islands'), ('HN', 'Honduras'), ('HR', 'Croatia'), ('HT', 'Haiti'), ('HU', 'Hungary'), ('ID', 'Indonesia'), ('IE', 'Ireland'), ('IL', 'Israel'), ('IN', 'India'), ('IO', 'British Indian Ocean Territory'), ('IQ', 'Iraq'), ('IR', 'Islamic Republic of Iran'), ('IS', 'Iceland'), ('IT', 'Italy'), ('JM', 'Jamaica'), ('JO', 'Jordan'), ('JP', 'Japan'), ('KE', 'Kenya'), ('KG', 'Kyrgyzstan'), ('KH', 'Cambodia'), ('KI', 'Kiribati'), ('KM', 'Comoros'), ('KN', 'St. Kitts and Nevis'), ('KP', "Korea, Democratic People's Republic of"), ('KR', 'Korea, Republic of'), ('KW', 'Kuwait'), ('KY', 'Cayman Islands'), ('KZ', 'Kazakhstan'), ('LA', "Lao People's Democratic Republic"), ('LB', 'Lebanon'), ('LC', 'Saint Lucia'), ('LI', 'Liechtenstein'), ('LK', 'Sri Lanka'), ('LR', 'Liberia'), ('LS', 'Lesotho'), ('LT', 'Lithuania'), ('LU', 'Luxembourg'), ('LV', 'Latvia'), ('LY', 'Libyan Arab Jamahiriya'), ('MA', 'Morocco'), ('MC', 'Monaco'), ('MD', 'Moldova, Republic of'), ('MG', 'Madagascar'), ('MH', 'Marshall Islands'), ('ML', 'Mali'), ('MN', 'Mongolia'), ('MM', 'Myanmar'), ('MO', 'Macau'), ('MP', 'Northern Mariana Islands'), ('MQ', 'Martinique'), ('MR', 'Mauritania'), ('MS', 'Monserrat'), ('MT', 'Malta'), ('MU', 'Mauritius'), ('MV', 'Maldives'), ('MW', 'Malawi'), ('MX', 'Mexico'), ('MY', 'Malaysia'), ('MZ', 'Mozambique'), ('NA', 'Namibia'), ('NC', 'New Caledonia'), ('NE', 'Niger'), ('NF', 'Norfolk Island'), ('NG', 'Nigeria'), ('NI', 'Nicaragua'), ('NL', 'Netherlands'), ('NO', 'Norway'), ('NP', 'Nepal'), ('NR', 'Nauru'), ('NU', 'Niue'), ('NZ', 'New Zealand'), ('OM', 'Oman'), ('PA', 'Panama'), ('PE', 'Peru'), ('PF', 'French Polynesia'), ('PG', 'Papua New Guinea'), ('PH', 'Philippines'), ('PK', 'Pakistan'), ('PL', 'Poland'), ('PM', 'St. Pierre & Miquelon'), ('PN', 'Pitcairn'), ('PR', 'Puerto Rico'), ('PT', 'Portugal'), ('PW', 'Palau'), ('PY', 'Paraguay'), ('QA', 'Qatar'), ('RE', 'Reunion'), ('RO', 'Romania'), ('RU', 'Russian Federation'), ('RW', 'Rwanda'), ('SA', 'Saudi Arabia'), ('SB', 'Solomon Islands'), ('SC', 'Seychelles'), ('SD', 'Sudan'), ('SE', 'Sweden'), ('SG', 'Singapore'), ('SH', 'St. Helena'), ('SI', 'Slovenia'), ('SJ', 'Svalbard & Jan Mayen Islands'), ('SK', 'Slovakia'), ('SL', 'Sierra Leone'), ('SM', 'San Marino'), ('SN', 'Senegal'), ('SO', 'Somalia'), ('SR', 'Suriname'), ('ST', 'Sao Tome & Principe'), ('SV', 'El Salvador'), ('SY', 'Syrian Arab Republic'), ('SZ', 'Swaziland'), ('TC', 'Turks & Caicos Islands'), ('TD', 'Chad'), ('TF', 'French Southern Territories'), ('TG', 'Togo'), ('TH', 'Thailand'), ('TJ', 'Tajikistan'), ('TK', 'Tokelau'), ('TM', 'Turkmenistan'), ('TN', 'Tunisia'), ('TO', 'Tonga'), ('TP', 'East Timor'), ('TR', 'Turkey'), ('TT', 'Trinidad & Tobago'), ('TV', 'Tuvalu'), ('TW', 'Taiwan, Province of China'), ('TZ', 'Tanzania, United Republic of'), ('UA', 'Ukraine'), ('UG', 'Uganda'), ('UM', 'United States Minor Outlying Islands'), ('US', 'United States of America'), ('UY', 'Uruguay'), ('UZ', 'Uzbekistan'), ('VA', 'Vatican City State (Holy See)'), ('VC', 'St. Vincent & the Grenadines'), ('VE', 'Venezuela'), ('VG', 'British Virgin Islands'), ('VI', 'United States Virgin Islands'), ('VN', 'Viet Nam'), ('VU', 'Vanuatu'), ('WF', 'Wallis & Futuna Islands'), ('WS', 'Samoa'), ('YE', 'Yemen'), ('YT', 'Mayotte'), ('YU', 'Yugoslavia'), ('ZA', 'South Africa'), ('ZM', 'Zambia'), ('ZR', 'Zaire'), ('ZW', 'Zimbabwe')], default='CH', max_length=2)),
+ ('starting_date', models.DateField()),
+ ('ending_date', models.DateField(blank=True, null=True)),
+ ('billing_network', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='uncloudproviderbill', to='uncloud.uncloudnetwork')),
+ ('coupon_network', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='uncloudprovidercoupon', to='uncloud.uncloudnetwork')),
+ ('referral_network', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='uncloudproviderreferral', to='uncloud.uncloudnetwork')),
+ ],
+ options={
+ 'abstract': False,
+ },
+ ),
+ ]
diff --git a/uncloud/migrations/0002_uncloudtasks.py b/uncloud/migrations/0002_uncloudtasks.py
new file mode 100644
index 0000000..9c69606
--- /dev/null
+++ b/uncloud/migrations/0002_uncloudtasks.py
@@ -0,0 +1,19 @@
+# Generated by Django 3.1 on 2020-12-20 17:16
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('uncloud', '0001_initial'),
+ ]
+
+ operations = [
+ migrations.CreateModel(
+ name='UncloudTasks',
+ fields=[
+ ('task_id', models.UUIDField(primary_key=True, serialize=False)),
+ ],
+ ),
+ ]
diff --git a/uncloud/migrations/0003_auto_20201220_1728.py b/uncloud/migrations/0003_auto_20201220_1728.py
new file mode 100644
index 0000000..2ec0eec
--- /dev/null
+++ b/uncloud/migrations/0003_auto_20201220_1728.py
@@ -0,0 +1,17 @@
+# Generated by Django 3.1 on 2020-12-20 17:28
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('uncloud', '0002_uncloudtasks'),
+ ]
+
+ operations = [
+ migrations.RenameModel(
+ old_name='UncloudTasks',
+ new_name='UncloudTask',
+ ),
+ ]
diff --git a/uncloud_service/migrations/__init__.py b/uncloud/migrations/__init__.py
similarity index 100%
rename from uncloud_service/migrations/__init__.py
rename to uncloud/migrations/__init__.py
diff --git a/uncloud/models.py b/uncloud/models.py
index 212d555..5545303 100644
--- a/uncloud/models.py
+++ b/uncloud/models.py
@@ -1,7 +1,11 @@
from django.db import models
-from django.db.models import JSONField
-
+from django.db.models import JSONField, Q
+from django.utils import timezone
from django.utils.translation import gettext_lazy as _
+from django.core.validators import MinValueValidator, MaxValueValidator
+from django.core.exceptions import FieldError
+
+from uncloud import COUNTRIES
class UncloudModel(models.Model):
"""
@@ -34,3 +38,135 @@ class UncloudStatus(models.TextChoices):
DELETED = 'DELETED', _('Deleted') # Resource has been deleted
DISABLED = 'DISABLED', _('Disabled') # Is usable, but cannot be used for new things
UNUSABLE = 'UNUSABLE', _('Unusable'), # Has some kind of error
+
+
+
+###
+# General address handling
+class CountryField(models.CharField):
+ def __init__(self, *args, **kwargs):
+ kwargs.setdefault('choices', COUNTRIES)
+ kwargs.setdefault('default', 'CH')
+ kwargs.setdefault('max_length', 2)
+
+ super().__init__(*args, **kwargs)
+
+ def get_internal_type(self):
+ return "CharField"
+
+
+class UncloudAddress(models.Model):
+ full_name = models.CharField(max_length=256)
+ organization = models.CharField(max_length=256, blank=True, null=True)
+ street = models.CharField(max_length=256)
+ city = models.CharField(max_length=256)
+ postal_code = models.CharField(max_length=64)
+ country = CountryField(blank=True)
+
+ class Meta:
+ abstract = True
+
+
+###
+# UncloudNetworks are used as identifiers - such they are a base of uncloud
+
+class UncloudNetwork(models.Model):
+ """
+ Storing IP networks
+ """
+
+ network_address = models.GenericIPAddressField(null=False, unique=True)
+ network_mask = models.IntegerField(null=False,
+ validators=[MinValueValidator(0),
+ MaxValueValidator(128)]
+ )
+
+ description = models.CharField(max_length=256)
+
+ @classmethod
+ def populate_db_defaults(cls):
+ for net, desc in [
+ ( "2a0a:e5c0:11::", "uncloud Billing" ),
+ ( "2a0a:e5c0:11:1::", "uncloud Referral" ),
+ ( "2a0a:e5c0:11:2::", "uncloud Coupon" )
+ ]:
+ obj, created = cls.objects.get_or_create(network_address=net,
+ defaults= {
+ 'network_mask': 64,
+ 'description': desc
+ }
+ )
+
+
+ def save(self, *args, **kwargs):
+ if not ':' in self.network_address and self.network_mask > 32:
+ raise FieldError("Mask cannot exceed 32 for IPv4")
+
+ super().save(*args, **kwargs)
+
+
+ def __str__(self):
+ return f"{self.network_address}/{self.network_mask} {self.description}"
+
+###
+# Who is running / providing this instance of uncloud?
+
+class UncloudProvider(UncloudAddress):
+ """
+ A class resembling who is running this uncloud instance.
+ This might change over time so we allow starting/ending dates
+
+ This also defines the taxation rules.
+
+ starting/ending date define from when to when this is valid. This way
+ we can model address changes and have it correct in the bills.
+ """
+
+ # Meta:
+ # FIXMe: only allow non overlapping time frames -- how to define this as a constraint?
+ starting_date = models.DateField()
+ ending_date = models.DateField(blank=True, null=True)
+
+ billing_network = models.ForeignKey(UncloudNetwork, related_name="uncloudproviderbill", on_delete=models.CASCADE)
+ referral_network = models.ForeignKey(UncloudNetwork, related_name="uncloudproviderreferral", on_delete=models.CASCADE)
+ coupon_network = models.ForeignKey(UncloudNetwork, related_name="uncloudprovidercoupon", on_delete=models.CASCADE)
+
+
+ @classmethod
+ def get_provider(cls, when=None):
+ """
+ Find active provide at a certain time - if there was any
+ """
+
+ if not when:
+ when = timezone.now()
+
+
+ return cls.objects.get(Q(starting_date__gte=when, ending_date__lte=when) |
+ Q(starting_date__gte=when, ending_date__isnull=True))
+
+
+ @classmethod
+ def populate_db_defaults(cls):
+ obj, created = cls.objects.get_or_create(full_name="ungleich glarus ag",
+ street="Bahnhofstrasse 1",
+ postal_code="8783",
+ city="Linthal",
+ country="CH",
+ starting_date=timezone.now(),
+ billing_network=UncloudNetwork.objects.get(description="uncloud Billing"),
+ referral_network=UncloudNetwork.objects.get(description="uncloud Referral"),
+ coupon_network=UncloudNetwork.objects.get(description="uncloud Coupon")
+ )
+
+
+ def __str__(self):
+ return f"{self.full_name} {self.country}"
+
+
+class UncloudTask(models.Model):
+ """
+ Class to store dispatched tasks to be handled
+ """
+
+ task_id = models.UUIDField(primary_key=True)
diff --git a/uncloud/settings.py b/uncloud/settings.py
index df3ba17..ae734dc 100644
--- a/uncloud/settings.py
+++ b/uncloud/settings.py
@@ -11,6 +11,7 @@ https://docs.djangoproject.com/en/3.0/ref/settings/
"""
import os
+import re
import ldap
from django.core.management.utils import get_random_secret_key
@@ -19,8 +20,6 @@ from django_auth_ldap.config import LDAPSearch, LDAPSearchUnion
LOGGING = {}
-
-
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -172,7 +171,6 @@ OPENNEBULA_URL = 'https://opennebula.example.com:2634/RPC2'
# user:pass for accessing opennebula
OPENNEBULA_USER_PASS = 'user:password'
-
# Stripe (Credit Card payments)
STRIPE_KEY=""
STRIPE_PUBLIC_KEY=""
@@ -185,6 +183,55 @@ ALLOWED_HOSTS = []
# required for hardcopy / pdf rendering: https://github.com/loftylabs/django-hardcopy
CHROME_PATH = '/usr/bin/chromium-browser'
+# Username that is created by default and owns the configuration objects
+UNCLOUD_ADMIN_NAME = "uncloud-admin"
+
+LOGIN_REDIRECT_URL = '/'
+LOGOUT_REDIRECT_URL = '/'
+
+# replace these in local_settings.py
+AUTH_LDAP_SERVER_URI = "ldaps://ldap1.example.com,ldaps://ldap2.example.com"
+AUTH_LDAP_BIND_DN="uid=django,ou=system,dc=example,dc=com"
+AUTH_LDAP_BIND_PASSWORD="a very secure ldap password"
+AUTH_LDAP_USER_SEARCH = LDAPSearch("dc=example,dc=com",
+ ldap.SCOPE_SUBTREE,
+ "(uid=%(user)s)")
+
+# where to create customers
+LDAP_CUSTOMER_DN="ou=customer,dc=example,dc=com"
+
+# def route_task(name, args, kwargs, options, task=None, **kw):
+# print(f"{name} - {args} - {kwargs}")
+# # if name == 'myapp.tasks.compress_video':
+# return {'queue': 'vpn1' }
+# # 'exchange_type': 'topic',
+# # 'routing_key': 'video.compress'}
+
+
+# CELERY_TASK_ROUTES = (route_task,)
+
+# CELERY_TASK_ROUTES = {
+# '*': {
+# 'queue': 'vpn1'
+# }
+# }
+
+
+CELERY_BROKER_URL = 'redis://:uncloud.example.com:6379/0'
+CELERY_RESULT_BACKEND = 'redis://:uncloud.example.com:6379/0'
+
+CELERY_TASK_ROUTES = {
+ re.compile(r'.*.tasks.cdist.*'): { 'queue': 'cdist' } # cdist tasks go into cdist queue
+}
+
+CELERY_BEAT_SCHEDULE = {
+ 'cleanup_tasks': {
+ 'task': 'uncloud.tasks.cleanup_tasks',
+ 'schedule': 10
+ }
+}
+
+# CELERY_TASK_CREATE_MISSING_QUEUES = False
# Overwrite settings with local settings, if existing
try:
diff --git a/uncloud/tasks.py b/uncloud/tasks.py
new file mode 100644
index 0000000..5a13ec5
--- /dev/null
+++ b/uncloud/tasks.py
@@ -0,0 +1,19 @@
+from celery import shared_task
+from celery.result import AsyncResult
+
+from .models import UncloudTask
+
+@shared_task(bind=True)
+def cleanup_tasks(self):
+ print(f"Cleanup time from {self}: {self.request.id}")
+ for task in UncloudTask.objects.all():
+ print(f"Pruning {task}...")
+
+ if str(task.task_id) == str(self.request.id):
+ print("Skipping myself")
+ continue
+
+ res = AsyncResult(id=str(task.task_id))
+ if res.ready():
+ print(res.get())
+ task.delete()
diff --git a/uncloud/templates/uncloud/index.html b/uncloud/templates/uncloud/index.html
new file mode 100644
index 0000000..b40c3b4
--- /dev/null
+++ b/uncloud/templates/uncloud/index.html
@@ -0,0 +1,15 @@
+{% extends 'uncloud/base.html' %}
+{% block title %}{% endblock %}
+
+{% block body %}
+
+
Welcome to uncloud
+ Welcome to uncloud, checkout the following locations:
+
+
+
+
+
+
+
+
+{% endblock %}
diff --git a/uncloud_auth/uldap.py b/uncloud_auth/uldap.py
new file mode 100644
index 0000000..aa90c77
--- /dev/null
+++ b/uncloud_auth/uldap.py
@@ -0,0 +1,42 @@
+import ldap
+# from django.conf import settings
+
+AUTH_LDAP_SERVER_URI = "ldaps://ldap1.ungleich.ch,ldaps://ldap2.ungleich.ch"
+AUTH_LDAP_BIND_DN="uid=django-create,ou=system,dc=ungleich,dc=ch"
+AUTH_LDAP_BIND_PASSWORD="kS#e+v\zjKn]L!,RIu2}V+DUS"
+# AUTH_LDAP_USER_SEARCH = LDAPSearch("dc=ungleich,dc=ch",
+# ldap.SCOPE_SUBTREE,
+# "(uid=%(user)s)")
+
+
+
+ldap_object = ldap.initialize(AUTH_LDAP_SERVER_URI)
+cancelid = ldap_object.bind(AUTH_LDAP_BIND_DN, AUTH_LDAP_BIND_PASSWORD)
+
+res = ldap_object.search_s("dc=ungleich,dc=ch", ldap.SCOPE_SUBTREE, "(uid=nico)")
+print(res)
+
+# class LDAP(object):
+# """
+# Managing users in LDAP
+
+# Requires the following settings?
+
+# LDAP_USER_DN: where to create users in the tree
+
+# LDAP_ADMIN_DN: which DN to use for managing users
+# LDAP_ADMIN_PASSWORD: which password to used
+
+# This module will reuse information from djagno_auth_ldap, including:
+
+# AUTH_LDAP_SERVER_URI
+
+# """
+# def __init__(self):
+# pass
+
+# def create_user(self):
+# pass
+
+# def change_password(self):
+# pass
diff --git a/uncloud_auth/ungleich_ldap.py b/uncloud_auth/ungleich_ldap.py
new file mode 100644
index 0000000..f22b423
--- /dev/null
+++ b/uncloud_auth/ungleich_ldap.py
@@ -0,0 +1,284 @@
+import base64
+import hashlib
+import logging
+import random
+
+import ldap3
+from django.conf import settings
+
+logger = logging.getLogger(__name__)
+
+
+class LdapManager:
+ __instance = None
+ def __new__(cls):
+ if LdapManager.__instance is None:
+ LdapManager.__instance = object.__new__(cls)
+ return LdapManager.__instance
+
+ def __init__(self):
+ """
+ Initialize the LDAP subsystem.
+ """
+ self.rng = random.SystemRandom()
+ self.server = ldap3.Server(settings.AUTH_LDAP_SERVER)
+
+
+ def get_admin_conn(self):
+ """
+ Return a bound :class:`ldap3.Connection` instance which has write
+ permissions on the dn in which the user accounts reside.
+ """
+ conn = self.get_conn(user=settings.LDAP_ADMIN_DN,
+ password=settings.LDAP_ADMIN_PASSWORD,
+ raise_exceptions=True)
+ conn.bind()
+ return conn
+
+
+ def get_conn(self, **kwargs):
+ """
+ Return an unbound :class:`ldap3.Connection` which talks to the configured
+ LDAP server.
+
+ The *kwargs* are passed to the constructor of :class:`ldap3.Connection` and
+ can be used to set *user*, *password* and other useful arguments.
+ """
+ return ldap3.Connection(self.server, **kwargs)
+
+
+ def _ssha_password(self, password):
+ """
+ Apply the SSHA password hashing scheme to the given *password*.
+ *password* must be a :class:`bytes` object, containing the utf-8
+ encoded password.
+
+ Return a :class:`bytes` object containing ``ascii``-compatible data
+ which can be used as LDAP value, e.g. after armoring it once more using
+ base64 or decoding it to unicode from ``ascii``.
+ """
+ SALT_BYTES = 15
+
+ sha1 = hashlib.sha1()
+ salt = self.rng.getrandbits(SALT_BYTES * 8).to_bytes(SALT_BYTES,
+ "little")
+ sha1.update(password)
+ sha1.update(salt)
+
+ digest = sha1.digest()
+ passwd = b"{SSHA}" + base64.b64encode(digest + salt)
+ return passwd
+
+
+ def create_user(self, user, password, firstname, lastname, email):
+ conn = self.get_admin_conn()
+ uidNumber = self._get_max_uid() + 1
+ logger.debug("uidNumber={uidNumber}".format(uidNumber=uidNumber))
+ user_exists = True
+ while user_exists:
+ user_exists, _ = self.check_user_exists(
+ "",
+ '(&(objectClass=inetOrgPerson)(objectClass=posixAccount)'
+ '(objectClass=top)(uidNumber={uidNumber}))'.format(
+ uidNumber=uidNumber
+ )
+ )
+ if user_exists:
+ logger.debug(
+ "{uid} exists. Trying next.".format(uid=uidNumber)
+ )
+ uidNumber += 1
+ logger.debug("{uid} does not exist. Using it".format(uid=uidNumber))
+ self._set_max_uid(uidNumber)
+ try:
+ uid = user # user.encode("utf-8")
+ conn.add("uid={uid},{customer_dn}".format(
+ uid=uid, customer_dn=settings.LDAP_CUSTOMER_DN
+ ),
+ ["inetOrgPerson", "posixAccount", "ldapPublickey"],
+ {
+ "uid": [uid],
+ "sn": [lastname.encode("utf-8")],
+ "givenName": [firstname.encode("utf-8")],
+ "cn": [uid],
+ "displayName": ["{} {}".format(firstname, lastname).encode("utf-8")],
+ "uidNumber": [str(uidNumber)],
+ "gidNumber": [str(settings.LDAP_CUSTOMER_GROUP_ID)],
+ "loginShell": ["/bin/bash"],
+ "homeDirectory": ["/home/{}".format(user).encode("utf-8")],
+ "mail": email.encode("utf-8"),
+ "userPassword": [self._ssha_password(
+ password.encode("utf-8")
+ )]
+ }
+ )
+ logger.debug('Created user %s %s' % (user.encode('utf-8'),
+ uidNumber))
+ except Exception as ex:
+ logger.debug('Could not create user %s' % user.encode('utf-8'))
+ logger.error("Exception: " + str(ex))
+ raise
+ finally:
+ conn.unbind()
+
+
+ def change_password(self, uid, new_password):
+ """
+ Changes the password of the user identified by user_dn
+
+ :param uid: str The uid that identifies the user
+ :param new_password: str The new password string
+ :return: True if password was changed successfully False otherwise
+ """
+ conn = self.get_admin_conn()
+
+ # Make sure the user exists first to change his/her details
+ user_exists, entries = self.check_user_exists(
+ uid=uid,
+ search_base=settings.ENTIRE_SEARCH_BASE
+ )
+ return_val = False
+ if user_exists:
+ try:
+ return_val = conn.modify(
+ entries[0].entry_dn,
+ {
+ "userpassword": (
+ ldap3.MODIFY_REPLACE,
+ [self._ssha_password(new_password.encode("utf-8"))]
+ )
+ }
+ )
+ except Exception as ex:
+ logger.error("Exception: " + str(ex))
+ else:
+ logger.error("User {} not found".format(uid))
+
+ conn.unbind()
+ return return_val
+
+ def change_user_details(self, uid, details):
+ """
+ Updates the user details as per given values in kwargs of the user
+ identified by user_dn.
+
+ Assumes that all attributes passed in kwargs are valid.
+
+ :param uid: str The uid that identifies the user
+ :param details: dict A dictionary containing the new values
+ :return: True if user details were updated successfully False otherwise
+ """
+ conn = self.get_admin_conn()
+
+ # Make sure the user exists first to change his/her details
+ user_exists, entries = self.check_user_exists(
+ uid=uid,
+ search_base=settings.ENTIRE_SEARCH_BASE
+ )
+
+ return_val = False
+ if user_exists:
+ details_dict = {k: (ldap3.MODIFY_REPLACE, [v.encode("utf-8")]) for
+ k, v in details.items()}
+ try:
+ return_val = conn.modify(entries[0].entry_dn, details_dict)
+ msg = "success"
+ except Exception as ex:
+ msg = str(ex)
+ logger.error("Exception: " + msg)
+ finally:
+ conn.unbind()
+ else:
+ msg = "User {} not found".format(uid)
+ logger.error(msg)
+ conn.unbind()
+ return return_val, msg
+
+ def check_user_exists(self, uid, search_filter="", attributes=None,
+ search_base=settings.LDAP_CUSTOMER_DN):
+ """
+ Check if the user with the given uid exists in the customer group.
+
+ :param uid: str representing the user
+ :param search_filter: str representing the filter condition to find
+ users. If its empty, the search finds the user with
+ the given uid.
+ :param attributes: list A list of str representing all the attributes
+ to be obtained in the result entries
+ :param search_base: str
+ :return: tuple (bool, [ldap3.abstract.entry.Entry ..])
+ A bool indicating if the user exists
+ A list of all entries obtained in the search
+ """
+ conn = self.get_admin_conn()
+ entries = []
+ try:
+ result = conn.search(
+ search_base=search_base,
+ search_filter=search_filter if len(search_filter)> 0 else
+ '(uid={uid})'.format(uid=uid),
+ attributes=attributes
+ )
+ entries = conn.entries
+ finally:
+ conn.unbind()
+ return result, entries
+
+ def delete_user(self, uid):
+ """
+ Deletes the user with the given uid from ldap
+
+ :param uid: str representing the user
+ :return: True if the delete was successful False otherwise
+ """
+ conn = self.get_admin_conn()
+ try:
+ return_val = conn.delete(
+ ("uid={uid}," + settings.LDAP_CUSTOMER_DN).format(uid=uid),
+ )
+ msg = "success"
+ except Exception as ex:
+ msg = str(ex)
+ logger.error("Exception: " + msg)
+ return_val = False
+ finally:
+ conn.unbind()
+ return return_val, msg
+
+ def _set_max_uid(self, max_uid):
+ """
+ a utility function to save max_uid value to a file
+
+ :param max_uid: an integer representing the max uid
+ :return:
+ """
+ with open(settings.LDAP_MAX_UID_FILE_PATH, 'w+') as handler:
+ handler.write(str(max_uid))
+
+ def _get_max_uid(self):
+ """
+ A utility function to read the max uid value that was previously set
+
+ :return: An integer representing the max uid value that was previously
+ set
+ """
+ try:
+ with open(settings.LDAP_MAX_UID_FILE_PATH, 'r+') as handler:
+ try:
+ return_value = int(handler.read())
+ except ValueError as ve:
+ logger.error(
+ "Error reading int value from {}. {}"
+ "Returning default value {} instead".format(
+ settings.LDAP_MAX_UID_PATH,
+ str(ve),
+ settings.LDAP_DEFAULT_START_UID
+ )
+ )
+ return_value = settings.LDAP_DEFAULT_START_UID
+ return return_value
+ except FileNotFoundError as fnfe:
+ logger.error("File not found : " + str(fnfe))
+ return_value = settings.LDAP_DEFAULT_START_UID
+ logger.error("So, returning UID={}".format(return_value))
+ return return_value
diff --git a/uncloud_auth/views.py b/uncloud_auth/views.py
index 77f0a0f..9310a4c 100644
--- a/uncloud_auth/views.py
+++ b/uncloud_auth/views.py
@@ -1,9 +1,22 @@
-from rest_framework import viewsets, permissions, status
-from .serializers import *
+from django.contrib.auth import views as auth_views
+from django.contrib.auth import logout
+
from django_auth_ldap.backend import LDAPBackend
+from rest_framework import mixins, permissions, status, viewsets
from rest_framework.decorators import action
from rest_framework.response import Response
+from .serializers import *
+
+
+class LoginView(auth_views.LoginView):
+ template_name = 'uncloud_auth/login.html'
+
+class LogoutView(auth_views.LogoutView):
+ pass
+# template_name = 'uncloud_auth/logo.html'
+
+
class UserViewSet(viewsets.GenericViewSet):
permission_classes = [permissions.IsAuthenticated]
serializer_class = UserSerializer
@@ -19,19 +32,29 @@ class UserViewSet(viewsets.GenericViewSet):
serializer = self.get_serializer(user, context = {'request': request})
return Response(serializer.data)
- def create(self, request):
- """
- Modify existing user data
- """
-
- user = request.user
- serializer = self.get_serializer(user,
- context = {'request': request},
- data=request.data)
+ @action(detail=False, methods=['post'])
+ def change_email(self, request):
+ serializer = self.get_serializer(
+ request.user, data=request.data, context={'request': request}
+ )
serializer.is_valid(raise_exception=True)
serializer.save()
return Response(serializer.data)
+
+class AccountManagementViewSet(mixins.CreateModelMixin, viewsets.GenericViewSet):
+ serializer_class = UserRegistrationSerializer
+
+ def create(self, request, *args, **kwargs):
+ serializer = self.get_serializer(data=request.data)
+ serializer.is_valid(raise_exception=True)
+ self.perform_create(serializer)
+ headers = self.get_success_headers(serializer.data)
+ return Response(
+ serializer.data, status=status.HTTP_201_CREATED, headers=headers
+ )
+
+
class AdminUserViewSet(viewsets.ReadOnlyModelViewSet):
permission_classes = [permissions.IsAdminUser]
diff --git a/uncloud_net/admin.py b/uncloud_net/admin.py
index 8c38f3f..ca6aaa1 100644
--- a/uncloud_net/admin.py
+++ b/uncloud_net/admin.py
@@ -1,3 +1,7 @@
from django.contrib import admin
-# Register your models here.
+from .models import *
+
+
+for m in [ ReverseDNSEntry, WireGuardVPNPool, WireGuardVPN ]:
+ admin.site.register(m)
diff --git a/uncloud_net/forms.py b/uncloud_net/forms.py
new file mode 100644
index 0000000..ad4e013
--- /dev/null
+++ b/uncloud_net/forms.py
@@ -0,0 +1,11 @@
+from django import forms
+
+from .models import *
+from .selectors import *
+
+class WireGuardVPNForm(forms.ModelForm):
+ network_size = forms.ChoiceField(choices=allowed_vpn_network_reservation_size)
+
+ class Meta:
+ model = WireGuardVPN
+ fields = [ "wireguard_public_key" ]
diff --git a/uncloud_net/migrations/0001_initial.py b/uncloud_net/migrations/0001_initial.py
index 7e018b2..6794156 100644
--- a/uncloud_net/migrations/0001_initial.py
+++ b/uncloud_net/migrations/0001_initial.py
@@ -1,11 +1,9 @@
-# Generated by Django 3.0.6 on 2020-08-01 16:38
+# Generated by Django 3.1 on 2020-12-13 13:42
from django.conf import settings
-import django.contrib.postgres.fields.jsonb
import django.core.validators
from django.db import migrations, models
import django.db.models.deletion
-import uuid
class Migration(migrations.Migration):
@@ -14,7 +12,6 @@ class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
- ('uncloud_pay', '__first__'),
]
operations = [
@@ -25,45 +22,41 @@ class Migration(migrations.Migration):
],
),
migrations.CreateModel(
- name='VPNPool',
- fields=[
- ('extra_data', django.contrib.postgres.fields.jsonb.JSONField(blank=True, editable=False, null=True)),
- ('uuid', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
- ('network', models.GenericIPAddressField(unique=True)),
- ('network_size', models.IntegerField(validators=[django.core.validators.MinValueValidator(0), django.core.validators.MaxValueValidator(128)])),
- ('subnetwork_size', models.IntegerField(validators=[django.core.validators.MinValueValidator(0), django.core.validators.MaxValueValidator(128)])),
- ('vpn_hostname', models.CharField(max_length=256)),
- ('wireguard_private_key', models.CharField(max_length=48)),
- ],
- options={
- 'abstract': False,
- },
- ),
- migrations.CreateModel(
- name='VPNNetworkReservation',
- fields=[
- ('extra_data', django.contrib.postgres.fields.jsonb.JSONField(blank=True, editable=False, null=True)),
- ('address', models.GenericIPAddressField(primary_key=True, serialize=False)),
- ('status', models.CharField(choices=[('used', 'used'), ('free', 'free')], default='used', max_length=256)),
- ('vpnpool', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='uncloud_net.VPNPool')),
- ],
- options={
- 'abstract': False,
- },
- ),
- migrations.CreateModel(
- name='VPNNetwork',
+ name='WireGuardVPNPool',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
- ('extra_data', django.contrib.postgres.fields.jsonb.JSONField(blank=True, editable=False, null=True)),
- ('status', models.CharField(choices=[('PENDING', 'Pending'), ('AWAITING_PAYMENT', 'Awaiting payment'), ('BEING_CREATED', 'Being created'), ('SCHEDULED', 'Scheduled'), ('ACTIVE', 'Active'), ('MODIFYING', 'Modifying'), ('DELETED', 'Deleted'), ('DISABLED', 'Disabled'), ('UNUSABLE', 'Unusable')], default='AWAITING_PAYMENT', max_length=32)),
- ('wireguard_public_key', models.CharField(max_length=48)),
- ('network', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, to='uncloud_net.VPNNetworkReservation')),
- ('order', models.ForeignKey(editable=False, null=True, on_delete=django.db.models.deletion.CASCADE, to='uncloud_pay.Order')),
- ('owner', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
+ ('network', models.GenericIPAddressField(unique=True)),
+ ('network_mask', models.IntegerField(validators=[django.core.validators.MinValueValidator(0), django.core.validators.MaxValueValidator(128)])),
+ ('subnetwork_mask', models.IntegerField(validators=[django.core.validators.MinValueValidator(0), django.core.validators.MaxValueValidator(128)])),
+ ('vpn_server_hostname', models.CharField(max_length=256)),
+ ('wireguard_private_key', models.CharField(max_length=48)),
+ ],
+ ),
+ migrations.CreateModel(
+ name='WireGuardVPNFreeLeases',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('pool_index', models.IntegerField(unique=True)),
+ ('vpnpool', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='uncloud_net.wireguardvpnpool')),
+ ],
+ ),
+ migrations.CreateModel(
+ name='WireGuardVPN',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('pool_index', models.IntegerField(unique=True)),
+ ('wireguard_public_key', models.CharField(max_length=48)),
+ ('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
+ ('vpnpool', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='uncloud_net.wireguardvpnpool')),
+ ],
+ ),
+ migrations.CreateModel(
+ name='ReverseDNSEntry',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('ip_address', models.GenericIPAddressField(unique=True)),
+ ('name', models.CharField(max_length=253)),
+ ('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
- options={
- 'abstract': False,
- },
),
]
diff --git a/uncloud_net/migrations/0002_auto_20200801_2332.py b/uncloud_net/migrations/0002_auto_20200801_2332.py
deleted file mode 100644
index 81d12ab..0000000
--- a/uncloud_net/migrations/0002_auto_20200801_2332.py
+++ /dev/null
@@ -1,20 +0,0 @@
-# Generated by Django 3.0.8 on 2020-08-01 23:32
-
-from django.db import migrations, models
-import django.db.models.deletion
-
-
-class Migration(migrations.Migration):
-
- dependencies = [
- ('uncloud_pay', '0003_auto_20200801_2332'),
- ('uncloud_net', '0001_initial'),
- ]
-
- operations = [
- migrations.AlterField(
- model_name='vpnnetwork',
- name='order',
- field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.CASCADE, to='uncloud_pay.Order'),
- ),
- ]
diff --git a/uncloud_net/migrations/0002_wireguardvpnpool_wireguard_public_key.py b/uncloud_net/migrations/0002_wireguardvpnpool_wireguard_public_key.py
new file mode 100644
index 0000000..479aba1
--- /dev/null
+++ b/uncloud_net/migrations/0002_wireguardvpnpool_wireguard_public_key.py
@@ -0,0 +1,19 @@
+# Generated by Django 3.1 on 2020-12-13 17:04
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('uncloud_net', '0001_initial'),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name='wireguardvpnpool',
+ name='wireguard_public_key',
+ field=models.CharField(default='', max_length=48),
+ preserve_default=False,
+ ),
+ ]
diff --git a/uncloud_net/migrations/0003_auto_20200808_1953.py b/uncloud_net/migrations/0003_auto_20200808_1953.py
deleted file mode 100644
index 211a8a4..0000000
--- a/uncloud_net/migrations/0003_auto_20200808_1953.py
+++ /dev/null
@@ -1,28 +0,0 @@
-# Generated by Django 3.1 on 2020-08-08 19:53
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
- dependencies = [
- ('uncloud_net', '0002_auto_20200801_2332'),
- ]
-
- operations = [
- migrations.AlterField(
- model_name='vpnnetwork',
- name='extra_data',
- field=models.JSONField(blank=True, editable=False, null=True),
- ),
- migrations.AlterField(
- model_name='vpnnetworkreservation',
- name='extra_data',
- field=models.JSONField(blank=True, editable=False, null=True),
- ),
- migrations.AlterField(
- model_name='vpnpool',
- name='extra_data',
- field=models.JSONField(blank=True, editable=False, null=True),
- ),
- ]
diff --git a/uncloud_net/migrations/0003_wireguardvpnpool_wg_name.py b/uncloud_net/migrations/0003_wireguardvpnpool_wg_name.py
new file mode 100644
index 0000000..9ecf52c
--- /dev/null
+++ b/uncloud_net/migrations/0003_wireguardvpnpool_wg_name.py
@@ -0,0 +1,19 @@
+# Generated by Django 3.1 on 2020-12-13 17:31
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('uncloud_net', '0002_wireguardvpnpool_wireguard_public_key'),
+ ]
+
+ operations = [
+ migrations.AddField(
+ model_name='wireguardvpnpool',
+ name='wg_name',
+ field=models.CharField(default='wg0', max_length=15),
+ preserve_default=False,
+ ),
+ ]
diff --git a/uncloud_net/migrations/0004_auto_20201213_1734.py b/uncloud_net/migrations/0004_auto_20201213_1734.py
new file mode 100644
index 0000000..24e46e7
--- /dev/null
+++ b/uncloud_net/migrations/0004_auto_20201213_1734.py
@@ -0,0 +1,17 @@
+# Generated by Django 3.1 on 2020-12-13 17:34
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('uncloud_net', '0003_wireguardvpnpool_wg_name'),
+ ]
+
+ operations = [
+ migrations.AddConstraint(
+ model_name='wireguardvpnpool',
+ constraint=models.UniqueConstraint(fields=('wg_name', 'vpn_server_hostname'), name='unique_interface_name_per_host'),
+ ),
+ ]
diff --git a/uncloud_net/migrations/0005_auto_20201220_1837.py b/uncloud_net/migrations/0005_auto_20201220_1837.py
new file mode 100644
index 0000000..1dbabe6
--- /dev/null
+++ b/uncloud_net/migrations/0005_auto_20201220_1837.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.1 on 2020-12-20 18:37
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('uncloud_net', '0004_auto_20201213_1734'),
+ ]
+
+ operations = [
+ migrations.AlterField(
+ model_name='wireguardvpn',
+ name='wireguard_public_key',
+ field=models.CharField(max_length=48, unique=True),
+ ),
+ ]
diff --git a/uncloud_net/migrations/0006_auto_20201224_1626.py b/uncloud_net/migrations/0006_auto_20201224_1626.py
new file mode 100644
index 0000000..c0dd2ef
--- /dev/null
+++ b/uncloud_net/migrations/0006_auto_20201224_1626.py
@@ -0,0 +1,17 @@
+# Generated by Django 3.1 on 2020-12-24 16:26
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('uncloud_net', '0005_auto_20201220_1837'),
+ ]
+
+ operations = [
+ migrations.AddConstraint(
+ model_name='wireguardvpn',
+ constraint=models.UniqueConstraint(fields=('vpnpool', 'wireguard_public_key'), name='wg_key_unique_per_pool'),
+ ),
+ ]
diff --git a/uncloud_net/models.py b/uncloud_net/models.py
index 4f80246..0c8b02a 100644
--- a/uncloud_net/models.py
+++ b/uncloud_net/models.py
@@ -4,184 +4,189 @@ import ipaddress
from django.db import models
from django.contrib.auth import get_user_model
from django.core.validators import MinValueValidator, MaxValueValidator
+from django.core.exceptions import FieldError, ValidationError
+from uncloud_pay.models import Order
-from uncloud_pay.models import Product, RecurringPeriod
-from uncloud.models import UncloudModel, UncloudStatus
-
-
-class MACAdress(models.Model):
- default_prefix = 0x420000000000
-
-class VPNPool(UncloudModel):
+class WireGuardVPNPool(models.Model):
"""
Network address pools from which VPNs can be created
"""
- uuid = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
+ class Meta:
+ constraints = [
+ models.UniqueConstraint(fields=['wg_name', 'vpn_server_hostname' ],
+ name='unique_interface_name_per_host')
+ ]
+
+
+ # Linux interface naming is restricing to max 15 characters
+ wg_name = models.CharField(max_length=15)
network = models.GenericIPAddressField(unique=True)
- network_size = models.IntegerField(validators=[MinValueValidator(0),
+ network_mask = models.IntegerField(validators=[MinValueValidator(0),
MaxValueValidator(128)])
- subnetwork_size = models.IntegerField(validators=[
- MinValueValidator(0),
- MaxValueValidator(128)
- ])
-
- vpn_hostname = models.CharField(max_length=256)
+ subnetwork_mask = models.IntegerField(validators=[
+ MinValueValidator(0),
+ MaxValueValidator(128)
+ ])
+ vpn_server_hostname = models.CharField(max_length=256)
wireguard_private_key = models.CharField(max_length=48)
+ wireguard_public_key = models.CharField(max_length=48)
@property
- def num_maximum_networks(self):
+ def max_pool_index(self):
"""
- sample:
- network_size = 40
- subnetwork_size = 48
- maximum_networks = 2^(48-40)
-
- 2nd sample:
- network_size = 8
- subnetwork_size = 24
- maximum_networks = 2^(24-8)
+ Return the highest possible network / last network id
"""
- return 2**(self.subnetwork_size - self.network_size)
+ bits = self.subnetwork_mask - self.network_mask
+
+ return (2**bits)-1
@property
- def used_networks(self):
- return self.vpnnetworkreservation_set.filter(vpnpool=self, status='used')
+ def ip_network(self):
+ return ipaddress.ip_network(f"{self.network}/{self.network_mask}")
- @property
- def free_networks(self):
- return self.vpnnetworkreservation_set.filter(vpnpool=self, status='free')
-
- @property
- def num_used_networks(self):
- return len(self.used_networks)
-
- @property
- def num_free_networks(self):
- return self.num_maximum_networks - self.num_used_networks + len(self.free_networks)
-
- @property
- def next_free_network(self):
- if self.num_free_networks == 0:
- # FIXME: use right exception
- raise Exception("No free networks")
-
- if len(self.free_networks) > 0:
- return self.free_networks[0].address
-
- if len(self.used_networks) > 0:
- """
- sample:
-
- pool = 2a0a:e5c1:200::/40
- last_used = 2a0a:e5c1:204::/48
-
- next:
- """
-
- last_net = ipaddress.ip_network(self.used_networks.last().address)
- last_net_ip = last_net[0]
-
- if last_net_ip.version == 6:
- offset_to_next = 2**(128 - self.subnetwork_size)
- elif last_net_ip.version == 4:
- offset_to_next = 2**(32 - self.subnetwork_size)
-
- next_net_ip = last_net_ip + offset_to_next
-
- return str(next_net_ip)
- else:
- # first network to be created
- return self.network
-
- @property
- def wireguard_config_filename(self):
- return '/etc/wireguard/{}.conf'.format(self.network)
+ def __str__(self):
+ return f"{self.ip_network} (subnets: /{self.subnetwork_mask})"
@property
def wireguard_config(self):
- wireguard_config = [
- """
-[Interface]
-ListenPort = 51820
-PrivateKey = {privatekey}
-""".format(privatekey=self.wireguard_private_key) ]
+ wireguard_config = [ f"[Interface]\nListenPort = 51820\nPrivateKey = {self.wireguard_private_key}\n" ]
peers = []
- for reservation in self.vpnnetworkreservation_set.filter(status='used'):
- public_key = reservation.vpnnetwork_set.first().wireguard_public_key
- peer_network = "{}/{}".format(reservation.address, self.subnetwork_size)
- owner = reservation.vpnnetwork_set.first().owner
+ for vpn in self.wireguardvpn_set.all():
+ public_key = vpn.wireguard_public_key
+ peer_network = f"{vpn.address}/{self.subnetwork_mask}"
+ owner = vpn.owner
- peers.append("""
-# Owner: {owner}
-[Peer]
-PublicKey = {public_key}
-AllowedIPs = {peer_network}
-""".format(
- owner=owner,
- public_key=public_key,
- peer_network=peer_network))
+ peers.append(f"# Owner: {owner}\n[Peer]\nPublicKey = {public_key}\nAllowedIPs = {peer_network}\n\n")
wireguard_config.extend(peers)
return "\n".join(wireguard_config)
- def configure_wireguard_vpnserver(self):
+class WireGuardVPN(models.Model):
+ """
+ Created VPNNetworks
+ """
+ owner = models.ForeignKey(get_user_model(),
+ on_delete=models.CASCADE)
+ vpnpool = models.ForeignKey(WireGuardVPNPool,
+ on_delete=models.CASCADE)
+
+ pool_index = models.IntegerField(unique=True)
+
+ wireguard_public_key = models.CharField(max_length=48, unique=True)
+
+ class Meta:
+ constraints = [
+ models.UniqueConstraint(fields=['vpnpool', 'wireguard_public_key'],
+ name='wg_key_unique_per_pool')
+ ]
+
+
+ @property
+ def network_mask(self):
+ return self.vpnpool.subnetwork_mask
+
+ @property
+ def vpn_server(self):
+ return self.vpnpool.vpn_server_hostname
+
+ @property
+ def vpn_server_public_key(self):
+ return self.vpnpool.wireguard_public_key
+
+ @property
+ def address(self):
"""
- This method is designed to run as a celery task and should
- not be called directly from the web
+ Locate the correct subnet in the supernet
+
+ First get the network itself
+
"""
- # subprocess, ssh
+ net = self.vpnpool.ip_network
+ subnet = net[(2**(128-self.vpnpool.subnetwork_mask)) * self.pool_index]
+ return str(subnet)
+
+ def __str__(self):
+ return f"{self.address} ({self.pool_index})"
+
+
+class WireGuardVPNFreeLeases(models.Model):
+ """
+ Previously used VPNNetworks
+ """
+ vpnpool = models.ForeignKey(WireGuardVPNPool,
+ on_delete=models.CASCADE)
+
+ pool_index = models.IntegerField(unique=True)
+
+################################################################################
+
+class MACAdress(models.Model):
+ default_prefix = 0x420000000000
+
+
+class ReverseDNSEntry(models.Model):
+ """
+ A reverse DNS entry
+ """
+ owner = models.ForeignKey(get_user_model(),
+ on_delete=models.CASCADE)
+
+ ip_address = models.GenericIPAddressField(null=False, unique=True)
+
+ name = models.CharField(max_length=253, null=False)
+
+ @property
+ def reverse_pointer(self):
+ return ipaddress.ip_address(self.ip_address).reverse_pointer
+
+ def implement(self):
+ """
+ The implement function implements the change
+ """
+
+ # Get all DNS entries (?) / update this DNS entry
+ # convert to DNS name
+ #
pass
-class VPNNetworkReservation(UncloudModel):
- """
- This class tracks the used VPN networks. It will be deleted, when the product is cancelled.
- """
- vpnpool = models.ForeignKey(VPNPool,
- on_delete=models.CASCADE)
+ def save(self, *args, **kwargs):
+ # Product.objects.filter(config__parameters__contains='reverse_dns_network')
+ # FIXME: check if order is still active / not replaced
- address = models.GenericIPAddressField(primary_key=True)
+ allowed = False
+ product = None
- status = models.CharField(max_length=256,
- default='used',
- choices = (
- ('used', 'used'),
- ('free', 'free')
- )
- )
+ for order in Order.objects.filter(config__parameters__reverse_dns_network__isnull=False,
+ owner=self.owner):
+ network = order.config['parameters']['reverse_dns_network']
+
+ net = ipaddress.ip_network(network)
+ addr = ipaddress.ip_address(self.ip_address)
+
+ if addr in net:
+ allowed = True
+ product = order.product
+ break
-class VPNNetwork(Product):
- """
- A selected network. Used for tracking reservations / used networks
- """
- network = models.ForeignKey(VPNNetworkReservation,
- on_delete=models.CASCADE,
- editable=False)
+ if not allowed:
+ raise ValidationError(f"User {self.owner} does not have the right to create reverse DNS entry for {self.ip_address}")
- wireguard_public_key = models.CharField(max_length=48)
-
- default_recurring_period = RecurringPeriod.PER_365D
-
- @property
- def recurring_price(self):
- return 120
-
-
- def delete(self, *args, **kwargs):
- self.network.status = 'free'
- self.network.save()
super().save(*args, **kwargs)
- print("deleted {}".format(self))
+
+
+ def __str__(self):
+ return f"{self.ip_address} - {self.name}"
diff --git a/uncloud_net/selectors.py b/uncloud_net/selectors.py
new file mode 100644
index 0000000..6e12e8b
--- /dev/null
+++ b/uncloud_net/selectors.py
@@ -0,0 +1,43 @@
+from django.db import transaction
+from django.db.models import Count, F
+from .models import *
+
+def get_suitable_pools(subnetwork_mask):
+ """
+ Find suitable pools for a certain network size.
+
+ First, filter for all pools that offer the requested subnetwork_size.
+
+ Then find those pools that are not fully exhausted:
+
+ The number of available networks in a pool is 2^(subnetwork_size-network_size.
+
+ The number of available networks in a pool is given by the number of VPNNetworkreservations.
+
+ """
+
+ return WireGuardVPNPool.objects.annotate(
+ num_reservations=Count('wireguardvpn'),
+ max_reservations=2**(F('subnetwork_mask')-F('network_mask'))).filter(
+ num_reservations__lt=F('max_reservations'),
+ subnetwork_mask=subnetwork_mask)
+
+
+def allowed_vpn_network_reservation_size():
+ """
+ Find all possible sizes of subnetworks that are available.
+
+ Select all pools with free networks.
+
+ Get their subnetwork sizes, reduce to a set
+
+ """
+
+ pools = WireGuardVPNPool.objects.annotate(num_reservations=Count('wireguardvpn'),
+ max_reservations=2**(F('subnetwork_mask')-F('network_mask'))).filter(
+ num_reservations__lt=F('max_reservations'))
+
+ # Need to return set of tuples, see
+ # https://docs.djangoproject.com/en/3.1/ref/models/fields/#field-choices
+# return set([ (pool.subnetwork_mask, pool.subnetwork_mask) for pool in pools ])
+ return set([pool.subnetwork_mask for pool in pools ])
diff --git a/uncloud_net/serializers.py b/uncloud_net/serializers.py
index dc4866e..09baa59 100644
--- a/uncloud_net/serializers.py
+++ b/uncloud_net/serializers.py
@@ -5,96 +5,53 @@ from django.utils.translation import gettext_lazy as _
from rest_framework import serializers
from .models import *
+from .services import *
+from .selectors import *
+
+
+class WireGuardVPNSerializer(serializers.ModelSerializer):
+ address = serializers.CharField(read_only=True)
+ vpn_server = serializers.CharField(read_only=True)
+ vpn_server_public_key = serializers.CharField(read_only=True)
+ network_mask = serializers.IntegerField()
-class VPNPoolSerializer(serializers.ModelSerializer):
class Meta:
- model = VPNPool
- fields = '__all__'
+ model = WireGuardVPN
+ fields = [ 'wireguard_public_key', 'address', 'network_mask', 'vpn_server',
+ 'vpn_server_public_key' ]
-class VPNNetworkReservationSerializer(serializers.ModelSerializer):
- class Meta:
- model = VPNNetworkReservation
- fields = '__all__'
+ extra_kwargs = {
+ 'network_mask': {'write_only': True }
+ }
-class VPNNetworkSerializer(serializers.ModelSerializer):
- class Meta:
- model = VPNNetwork
- fields = '__all__'
+ def validate_network_mask(self, value):
+ msg = _(f"No pool for network size {value}")
+ sizes = allowed_vpn_network_reservation_size()
- # This is required for finding the VPN pool, but does not
- # exist in the model
- network_size = serializers.IntegerField(min_value=0,
- max_value=128,
- write_only=True)
-
- def validate_wireguard_public_key(self, value):
- msg = _("Supplied key is not a valid wireguard public key")
-
- """ FIXME: verify that this does not create broken wireguard config files,
- i.e. contains \n or similar!
- We might even need to be more strict to not break wireguard...
- """
-
- try:
- base64.standard_b64decode(value)
- except Exception as e:
- raise serializers.ValidationError(msg)
-
- if '\n' in value:
+ if not value in sizes:
raise serializers.ValidationError(msg)
return value
- def validate(self, data):
+ def validate_wireguard_public_key(self, value):
+ msg = _("Supplied key is not a valid wireguard public key")
- # FIXME: filter for status = active or similar
- all_pools = VPNPool.objects.all()
- sizes = [ p.subnetwork_size for p in all_pools ]
+ """
+ Verify wireguard key.
+ See https://lists.zx2c4.com/pipermail/wireguard/2020-December/006221.html
+ """
- pools = VPNPool.objects.filter(subnetwork_size=data['network_size'])
-
- if len(pools) == 0:
- msg = _("No pool available for networks with size = {}. Available are: {}".format(data['network_size'], sizes))
+ try:
+ decoded_key = base64.standard_b64decode(value)
+ except Exception as e:
raise serializers.ValidationError(msg)
- return data
+ if not len(decoded_key) == 32:
+ raise serializers.ValidationError(msg)
- def create(self, validated_data):
- """
- Creating a new vpnnetwork - there are a couple of race conditions,
- especially when run in parallel.
-
- What we should be doing:
-
- - create a reservation race free
- - map the reservation to a network (?)
- """
-
- pools = VPNPool.objects.filter(subnetwork_size=validated_data['network_size'])
-
- vpn_network = None
-
- for pool in pools:
- if pool.num_free_networks > 0:
- next_address = pool.next_free_network
-
- reservation, created = VPNNetworkReservation.objects.update_or_create(
- vpnpool=pool, address=next_address,
- defaults = {
- 'status': 'used'
- })
-
- vpn_network = VPNNetwork.objects.create(
- owner=self.context['request'].user,
- network=reservation,
- wireguard_public_key=validated_data['wireguard_public_key']
- )
-
- break
- if not vpn_network:
- # FIXME: use correct exception
- raise Exception("Did not find any free pool")
+ return value
- return vpn_network
+class WireGuardVPNSizesSerializer(serializers.Serializer):
+ size = serializers.IntegerField(min_value=0, max_value=128)
diff --git a/uncloud_net/services.py b/uncloud_net/services.py
new file mode 100644
index 0000000..4f80c44
--- /dev/null
+++ b/uncloud_net/services.py
@@ -0,0 +1,47 @@
+from django.db import transaction
+
+from .models import *
+from .selectors import *
+from .tasks import *
+
+@transaction.atomic
+def create_wireguard_vpn(owner, public_key, network_mask):
+
+ pool = get_suitable_pools(network_mask)[0]
+ count = pool.wireguardvpn_set.count()
+
+ # Try re-using previously used networks first
+ try:
+ free_lease = WireGuardVPNFreeLeases.objects.get(vpnpool=pool)
+
+ vpn = WireGuardVPN.objects.create(owner=owner,
+ vpnpool=pool,
+ pool_index=free_lease.pool_index,
+ wireguard_public_key=public_key)
+
+ free_lease.delete()
+
+ except WireGuardVPNFreeLeases.DoesNotExist:
+
+ # First object
+ if count == 0:
+ vpn = WireGuardVPN.objects.create(owner=owner,
+ vpnpool=pool,
+ pool_index=0,
+ wireguard_public_key=public_key)
+
+ else: # Select last network and try +1 it
+ last_net = WireGuardVPN.objects.filter(vpnpool=pool).order_by('pool_index').last()
+
+ next_index = last_net.pool_index + 1
+
+ if next_index <= pool.max_pool_index:
+ vpn = WireGuardVPN.objects.create(owner=owner,
+ vpnpool=pool,
+ pool_index=next_index,
+ wireguard_public_key=public_key)
+
+
+
+ configure_wireguard_server(pool)
+ return vpn
diff --git a/uncloud_net/tasks.py b/uncloud_net/tasks.py
new file mode 100644
index 0000000..78ae80c
--- /dev/null
+++ b/uncloud_net/tasks.py
@@ -0,0 +1,60 @@
+from celery import shared_task
+from .models import *
+
+from uncloud.models import UncloudTask
+
+import os
+import subprocess
+import logging
+import uuid
+
+log = logging.getLogger(__name__)
+
+@shared_task
+def whereami():
+ print(os.uname())
+ return os.uname()
+
+def configure_wireguard_server(wireguardvpnpool):
+ """
+ - Create wireguard config (DB query -> string)
+ - Submit config to cdist worker
+ - Change config locally on worker / commit / shared
+
+ """
+
+ config = wireguardvpnpool.wireguard_config
+ server = wireguardvpnpool.vpn_server_hostname
+
+ log.info(f"Configuring VPN server {server} (async)")
+
+ task_id = uuid.UUID(cdist_configure_wireguard_server.apply_async((config, server)).id)
+ UncloudTask.objects.create(task_id=task_id)
+
+
+@shared_task
+def cdist_configure_wireguard_server(config, server):
+ """
+ Create config and configure server.
+
+ To be executed on the cdist workers.
+ """
+
+ dirname= "/home/app/.cdist/type/__ungleich_wireguard/files/"
+ fname = os.path.join(dirname,server)
+
+ log.info(f"Configuring VPN server {server} (on cdist host)")
+ with open(fname, "w") as fd:
+ fd.write(config)
+
+ log.debug("git committing wireguard changes")
+ subprocess.run(f"cd {dirname} && git pull && git add {server} && git commit -m 'Updating config for {server}' && git push",
+ shell=True, check=True)
+
+ log.debug(f"Configuring VPN server {server} with cdist")
+ subprocess.run(f"cdist config {server}", shell=True, check=True)
+
+ # FIXME:
+ # ensure logs are on the server
+ # ensure exit codes are known
+ return True
diff --git a/uncloud_net/templates/uncloud_net/wireguardvpn_form.html b/uncloud_net/templates/uncloud_net/wireguardvpn_form.html
new file mode 100644
index 0000000..1463f41
--- /dev/null
+++ b/uncloud_net/templates/uncloud_net/wireguardvpn_form.html
@@ -0,0 +1,25 @@
+{% extends 'uncloud/base.html' %}
+
+{% block body %}
+
+
+
+
+ Create a VPN Network
+
+ Create a new wireguard based VPN network.
+
+
+
+
+
+
+
+
+
- {% if bill.billing_address.organization != "" %}
- ORG{{ bill.billing_address.organization }}
-
{{ bill.billing_address.name }}
- {% else %}
- {{ bill.billing_address.name }}
- {% endif %}
-
{{ bill.billing_address.street }}
-
{{ bill.billing_address.postal_code }} {{ bill.billing_address.city }}
-
{{ bill.billing_address.country }}
-
+ {{ bill.billing_address.organization }}
+ {{ bill.billing_address.name }}
+ {{ bill.owner.email }}
+ {{ bill.billing_address.street }}
+ {{ bill.billing_address.country }} {{ bill.billing_address.postal_code }} {{ bill.billing_address.city }}
+
- Rechnungsdatum:
-
Rechnungsnummer
-
Zahlbar bis
+ {{ bill.starting_date|date:"c" }} -
+ {{ bill.ending_date|date:"c" }}
+
Bill id: {{ bill }}
+
Due: {{ bill.due_date }}
-
- {{ bill.creation_date.date }}
- {% if bill.billing_address.vat_number != "" %}
- {{ bill.billing_address.vat_number
- }}
- {% else %}
- None
- {% endif %}
- {{ bill.billing_address.vat_number }}
- {{ bill.due_date }}
-
-
RECHNUNG
+ Invoice
| Detail |
- Quantity |
Price/Unit |
- Total |
+ Units |
+ Total price |
{% for record in bill_records %}
| {{ record.starting_date|date:"c" }}
- {% if record.ending_date %}
- {{ record.ending_date|date:"c" }}
- {% endif %}
- {{ record.order.description }}
+ {{ record.order }}
|
+ {{ record.price|floatformat:2 }} |
{{ record.quantity|floatformat:2 }} |
- {{ record.order.price|floatformat:2 }} |
{{ record.sum|floatformat:2 }} |
{% endfor %}
@@ -733,17 +716,17 @@ oAsAAAAAAACGQNAFAAAAAAAAQyDoAgAAAAAAgCEQdAEAAAAAAMAQCLoAAAAAAABgCP83AL6WQ1Y7
- Total
+ Total (excl. VAT)
{{ bill.amount }}
- VAT
+ VAT 7.7%
{{ bill.vat_amount|floatformat:2 }}
- Total
+ Total amount to be paid
{{ bill.sum|floatformat:2 }}
diff --git a/uncloud_pay/templates/uncloud_pay/stripe.html b/uncloud_pay/templates/uncloud_pay/stripe.html
new file mode 100644
index 0000000..3051bf0
--- /dev/null
+++ b/uncloud_pay/templates/uncloud_pay/stripe.html
@@ -0,0 +1,72 @@
+{% extends 'uncloud/base.html' %}
+
+{% block header %}
+
+
+{% endblock %}
+
+{% block body %}
+
+
Registering Stripe Credit Card
+
+
+
+
+
+
+
+