sanghee/uncloud
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Compare commits

base: sanghee:master
Branches Tags
sanghee:master
sanghee:user-billing-api-revamp
sanghee:update-vm-ordering
sanghee:uncloud-api-refactoring
sanghee:0.8
sanghee:0.7
sanghee:0.6
sanghee:0.5
sanghee:0.4
sanghee:0.3
sanghee:0.2
sanghee:0.1
sanghee:0.1.0
sanghee:0.0.8
sanghee:0.0.7
sanghee:0.0.6
sanghee:0.0.5
sanghee:0.0.4
sanghee:0.0.3
sanghee:0.0.2
sanghee:0.0.1
..
compare: sanghee:0.0.2
Branches Tags
sanghee:master
sanghee:user-billing-api-revamp
sanghee:update-vm-ordering
sanghee:uncloud-api-refactoring
sanghee:0.8
sanghee:0.7
sanghee:0.6
sanghee:0.5
sanghee:0.4
sanghee:0.3
sanghee:0.2
sanghee:0.1
sanghee:0.1.0
sanghee:0.0.8
sanghee:0.0.7
sanghee:0.0.6
sanghee:0.0.5
sanghee:0.0.4
sanghee:0.0.3
sanghee:0.0.2
sanghee:0.0.1

No commits in common. "master" and "0.0.2" have entirely different histories.
master ... 0.0.2

345 changed files with 2762 additions and 22373 deletions
Download patch file Download diff file Expand all files Collapse all files

19
.gitignore vendored
View file

@ -1,27 +1,18 @@
.idea/
.vscode/
__pycache__/
.idea
.vscode
pay.conf
log.txt
test.py
STRIPE
venv/
uncloud/docs/build
ucloud/docs/build
logs.txt
uncloud.egg-info
ucloud.egg-info
# run artefacts
default.etcd
__pycache__
# build artefacts
uncloud/version.py
ucloud/version.py
build/
venv/
dist/
.history/
*.iso
*.sqlite3

18
.gitlab-ci.yml
View file

@ -1,18 +0,0 @@
stages:
- lint
- test
run-tests:
stage: test
image: code.ungleich.ch:5050/uncloud/uncloud/uncloud-ci:latest
services:
- postgres:latest
variables:
DATABASE_HOST: postgres
DATABASE_USER: postgres
POSTGRES_HOST_AUTH_METHOD: trust
coverage: /^TOTAL.+?(\d+\%)$/
script:
- pip install -r requirements.txt
- coverage run --source='.' ./manage.py test
- coverage report

71
README.md
View file

@ -1,70 +1,3 @@
# Uncloud
# ucloud
Cloud management platform, the ungleich way.
[![pipeline status](https://code.ungleich.ch/uncloud/uncloud/badges/master/pipeline.svg)](https://code.ungleich.ch/uncloud/uncloud/commits/master)
[![coverage report](https://code.ungleich.ch/uncloud/uncloud/badges/master/coverage.svg)](https://code.ungleich.ch/uncloud/uncloud/commits/master)
## Useful commands
* `./manage.py import-vat-rates path/to/csv`
* `./manage.py createsuperuser`
## Development setup
Install system dependencies:
* On Fedora, you will need the following packages: `python3-virtualenv python3-devel openldap-devel gcc chromium`
* sudo apt-get install libpq-dev python-dev libxml2-dev libxslt1-dev libldap2-dev libsasl2-dev libffi-dev
NOTE: you will need to configure a LDAP server and credentials for authentication. See `uncloud/settings.py`.
```
# Initialize virtualenv.
» virtualenv .venv
Using base prefix '/usr'
New python executable in /home/fnux/Workspace/ungleich/uncloud/uncloud/.venv/bin/python3
Also creating executable in /home/fnux/Workspace/ungleich/uncloud/uncloud/.venv/bin/python
Installing setuptools, pip, wheel...
done.
# Enter virtualenv.
» source .venv/bin/activate
# Install dependencies.
» pip install -r requirements.txt
[...]
# Run migrations.
» ./manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, opennebula, sessions, uncloud_auth, uncloud_net, uncloud_pay, uncloud_service, uncloud_vm
Running migrations:
[...]
# Run webserver.
» ./manage.py runserver
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
May 07, 2020 - 10:17:08
Django version 3.0.6, using settings 'uncloud.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
```
### Run Background Job Queue
We use Django Q to handle the asynchronous code and Background Cron jobs
To start the workers make sure first that Redis or the Django Q broker is working and you can edit it's settings in the settings file.
```
./manage.py qcluster
```
### Note on PGSQL
If you want to use Postgres:
* Install on configure PGSQL on your base system.
* OR use a container! `podman run --rm -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust -it postgres:latest`
Checkout https://ungleich.ch/ucloud/ for the documentation of ucloud.

6
archive/issues.org
View file

@ -1,6 +0,0 @@
* Intro
This file lists issues that should be handled, are small and likely
not yet high prio.
* Issues
** TODO Register prefered address in User model
** TODO Allow to specify different recurring periods

55
archive/uncloud_django_based/hacks/abk-hacks.py
View file

@ -1,55 +0,0 @@
"""
investigate into a simple python function that maps an ldap user to a vat percentage. Basically you need to
lookup the customer address, check if she is a business/registered tax number and if not apply the local
vat
"""
import iso3166
import datetime
from csv import DictReader
def get_vat(street_address, city, postal_code, country, vat_number=None):
vat = {
'Austria': [
{'period': '1984-01-01/', 'rate': 0.2},
{'period': '1976-01-01/1984-01-01', 'rate': 0.18},
{'period': '1973-01-01/1976-01-01', 'rate': 0.16},
]
}
return iso3166.countries.get(country)
# return iso3166.countries_by_name[country]
def main():
# vat = get_vat(
# street_address='82 Nasheman-e-Iqbal near Wapda Town',
# city='Lahore',
# postal_code=53700,
# country='Pakistan',
# )
# print(vat)
vat_rates = {}
with open('vat_rates.csv', newline='') as csvfile:
reader = DictReader(csvfile)
for row in reader:
territory_codes = row['territory_codes'].split('\n')
for code in territory_codes:
if code not in vat_rates:
vat_rates[code] = {}
start_date = row['start_date']
stop_data = row['stop_date']
time_period = f'{start_date}|{stop_data}'
r = row.copy()
del r['start_date']
del r['stop_date']
del r['territory_codes']
vat_rates[code][time_period] = r
print(vat_rates)
if __name__ == '__main__':
main()

46
archive/uncloud_django_based/hacks/abkhack/opennebula_hacks.py
View file

@ -1,46 +0,0 @@
import importlib
import sys
import os
from os.path import join as join_path
from xmlrpc.client import ServerProxy as RPCClient
root = os.path.dirname(os.getcwd())
sys.path.append(join_path(root, 'uncloud'))
secrets = importlib.import_module('uncloud.secrets')
class OpenNebula:
def __init__(self, url, session_string):
self.session_string = session_string
self.client = RPCClient(secrets.OPENNEBULA_URL)
def create_user(self, username, password, authentication_driver='', group_id=None):
# https://docs.opennebula.org/5.10/integration/system_interfaces/api.html#one-user-allocate
if group_id is None:
group_id = []
return self.client.one.user.allocate(
self.session_string,
username,
password,
authentication_driver,
group_id
)
def chmod(self, vm_id, user_id=-1, group_id=-1):
# https://docs.opennebula.org/5.10/integration/system_interfaces/api.html#one-vm-chown
return self.client.one.vm.chown(self.session_string, vm_id, user_id, group_id)
one = OpenNebula(secrets.OPENNEBULA_URL, secrets.OPENNEBULA_USER_PASS)
# Create User in OpenNebula
# success, response, *_ = one.create_user(username='meow12345', password='hello_world')
# print(success, response)
# Change owner of a VM
# success, response, *_ = one.chmod(vm_id=25589, user_id=706)
# print(success, response)

18
archive/uncloud_django_based/hacks/command-wrapper.sh
View file

@ -1,18 +0,0 @@
#!/bin/sh
dbhost=$1; shift
ssh -L5432:localhost:5432 "$dbhost" &
python manage.py "$@"
# command only needs to be active while manage command is running
# -T no pseudo terminal
# alternatively: commands output shell code
# ssh uncloud@dbhost "python manage.py --hostname xxx ..."

51
archive/uncloud_django_based/meow-payv1/README.md
View file

@ -1,51 +0,0 @@
# uncloud-pay
The generic product/payment system.
## Installation
```shell script
pip3 install -r requirements.txt
```
## Getting Started
```shell script
python ucloud_pay.py
```
## Usage
#### 1. Adding of products
```shell script
http --json http://[::]:5000/product/add username=your_username_here password=your_password_here specs:=@ipv6-only-vm.json
```
#### 2. Listing of products
```shell script
http --json http://[::]:5000/product/list
```
#### 3. Registering user's payment method (credit card for now using Stripe)
```shell script
http --json http://[::]:5000/user/register_payment card_number=4111111111111111 cvc=123 expiry_year=2020 expiry_month=8 card_holder_name="The test user" username=your_username_here password=your_password_here line1="your_billing_address" city="your_city" country="your_country"
```
#### 4. Ordering products
First of all, user have to buy the membership first.
```shell script
http --json http://[::]:5000/product/order username=your_username_here password=your_password_here product_id=membership pay=True
```
```shell script
http --json http://[::]:5000/product/order username=your_username_here password=your_password_here product_id=ipv6-only-vm cpu=1 ram=1 os-disk-space=10 os=alpine pay=True
```
#### 5. Listing users orders
```shell script
http --json POST http://[::]:5000/order/list username=your_username_here password=your_password_here
```

21
archive/uncloud_django_based/meow-payv1/config.py
View file

@ -1,21 +0,0 @@
import os
from ungleich_common.ldap.ldap_manager import LdapManager
from ungleich_common.std.configparser import StrictConfigParser
from ungleich_common.etcd.etcd_wrapper import EtcdWrapper
config_file = os.environ.get('meow-pay-config-file', default='pay.conf')
config = StrictConfigParser(allow_no_value=True)
config.read(config_file)
etcd_client = EtcdWrapper(
host=config.get('etcd', 'host'), port=config.get('etcd', 'port'),
ca_cert=config.get('etcd', 'ca_cert'), cert_key=config.get('etcd', 'cert_key'),
cert_cert=config.get('etcd', 'cert_cert')
)
ldap_manager = LdapManager(
server=config.get('ldap', 'server'), admin_dn=config.get('ldap', 'admin_dn'),
admin_password=config.get('ldap', 'admin_password')
)

213
archive/uncloud_django_based/meow-payv1/hack-a-vpn.py
View file

@ -1,213 +0,0 @@
from flask import Flask, request
from flask_restful import Resource, Api
import etcd3
import json
import logging
from functools import wraps
from ldaptest import is_valid_ldap_user
def authenticate(func):
@wraps(func)
def wrapper(*args, **kwargs):
if not getattr(func, 'authenticated', True):
return func(*args, **kwargs)
# pass in username/password !
acct = basic_authentication() # custom account lookup function
if acct:
return func(*args, **kwargs)
flask_restful.abort(401)
return wrapper
def readable_errors(func):
@wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except etcd3.exceptions.ConnectionFailedError as e:
raise UncloudException('Cannot connect to etcd: is etcd running and reachable? {}'.format(e))
except etcd3.exceptions.ConnectionTimeoutError as e:
raise UncloudException('etcd connection timeout. {}'.format(e))
return wrapper
class DB(object):
def __init__(self, config, prefix="/"):
self.config = config
# Root for everything
self.base_prefix= '/nicohack'
# Can be set from outside
self.prefix = prefix
self.connect()
@readable_errors
def connect(self):
self._db_clients = []
for endpoint in self.config.etcd_hosts:
client = etcd3.client(host=endpoint, **self.config.etcd_args)
self._db_clients.append(client)
def realkey(self, key):
return "{}{}/{}".format(self.base_prefix,
self.prefix,
key)
@readable_errors
def get(self, key, as_json=False, **kwargs):
value, _ = self._db_clients[0].get(self.realkey(key), **kwargs)
if as_json:
value = json.loads(value)
return value
@readable_errors
def set(self, key, value, as_json=False, **kwargs):
if as_json:
value = json.dumps(value)
# FIXME: iterate over clients in case of failure ?
return self._db_clients[0].put(self.realkey(key), value, **kwargs)
class Membership(Resource):
def __init__(self, config):
self.config = config
def get(self):
data = request.get_json(silent=True) or {}
print("{} {}".format(data, config))
return {'message': 'Order successful' }, 200
def post(self):
data = request.get_json(silent=True) or {}
print("{} {}".format(data, config))
return {'message': 'Order 2x successful' }, 200
class Order(Resource):
def __init__(self, config):
self.config = config
@staticmethod
def post():
data = request.get_json(silent=True) or {}
print("{} {}".format(data, config))
class Product(Resource):
def __init__(self, config):
self.config = config
self.products = []
self.products.append(
{ "name": "membership-free",
"description": """
This membership gives you access to the API and includes a VPN
with 1 IPv6 address.
See https://redmine.ungleich.ch/issues/7747?
""",
"uuid": "a3883466-0012-4d01-80ff-cbf7469957af",
"recurring": True,
"recurring_time_frame": "per_year",
"features": [
{ "name": "membership",
"price_one_time": 0,
"price_recurring": 0
}
]
}
)
self.products.append(
{ "name": "membership-standard",
"description": """
This membership gives you access to the API and includes an IPv6-VPN with
one IPv6 address ("Road warrior")
See https://redmine.ungleich.ch/issues/7747?
""",
"uuid": "1d85296b-0863-4dd6-a543-a6d5a4fbe4a6",
"recurring": True,
"recurring_time_frame": "per_month",
"features": [
{ "name": "membership",
"price_one_time": 0,
"price_recurring": 5
}
]
}
)
self.products.append(
{ "name": "membership-premium",
"description": """
This membership gives you access to the API and includes an
IPv6-VPN with a /48 IPv6 network.
See https://redmine.ungleich.ch/issues/7747?
""",
"uuid": "bfd63fd2-d227-436f-a8b8-600de74dd6ce",
"recurring": True,
"recurring_time_frame": "per_month",
"features": [
{ "name": "membership",
"price_one_time": 0,
"price_recurring": 5
}
]
}
)
self.products.append(
{ "name": "ipv6-vpn-with-/48",
"description": """
An IPv6 VPN with a /48 network included.
""",
"uuid": "fe5753f8-6fe1-4dc4-9b73-7b803de4c597",
"recurring": True,
"recurring_time_frame": "per_year",
"features": [
{ "name": "vpn",
"price_one_time": 0,
"price_recurring": 120
}
]
}
)
@staticmethod
def post():
data = request.get_json(silent=True) or {}
print("{} {}".format(data, config))
def get(self):
data = request.get_json(silent=True) or {}
print("{} {}".format(data, config))
return self.products
if __name__ == '__main__':
app = Flask(__name__)
config = {}
config['etcd_url']="https://etcd1.ungleich.ch"
config['ldap_url']="ldaps://ldap1.ungleich.ch"
api = Api(app)
api.add_resource(Order, '/orders', resource_class_args=( config, ))
api.add_resource(Product, '/products', resource_class_args=( config, ))
api.add_resource(Membership, '/membership', resource_class_args=( config, ))
app.run(host='::', port=5000, debug=True)

87
archive/uncloud_django_based/meow-payv1/helper.py
View file

@ -1,87 +0,0 @@
import logging
import parsedatetime
from datetime import datetime
from stripe_utils import StripeUtils
def get_plan_id_from_product(product):
plan_id = 'ucloud-v1-'
plan_id += product['name'].strip().replace(' ', '-')
return plan_id
def get_pricing(price_in_chf_cents, product_type, recurring_period):
if product_type == 'recurring':
return 'CHF {}/{}'.format(price_in_chf_cents/100, recurring_period)
elif product_type == 'one-time':
return 'CHF {} (One time charge)'.format(price_in_chf_cents/100)
def get_user_friendly_product(product_dict):
uf_product = {
'name': product_dict['name'],
'description': product_dict['description'],
'product_id': product_dict['usable-id'],
'pricing': get_pricing(
product_dict['price'], product_dict['type'], product_dict['recurring_period']
)
}
if product_dict['type'] == 'recurring':
uf_product['minimum_subscription_period'] = product_dict['minimum_subscription_period']
return uf_product
def get_token(card_number, cvc, exp_month, exp_year):
stripe_utils = StripeUtils()
token_response = stripe_utils.get_token_from_card(
card_number, cvc, exp_month, exp_year
)
if token_response['response_object']:
return token_response['response_object'].id
else:
return None
def resolve_product(usable_id, etcd_client):
products = etcd_client.get_prefix('/v1/products/', value_in_json=True)
for p in products:
if p.value['usable-id'] == usable_id:
return p.value
return None
def calculate_charges(specification, data):
logging.debug('Calculating charges for specs:{} and data:{}'.format(specification, data))
one_time_charge = 0
recurring_charge = 0
for feature_name, feature_detail in specification['features'].items():
if feature_detail['constant']:
data[feature_name] = 1
if feature_detail['unit']['type'] != 'str':
one_time_charge += feature_detail['one_time_fee']
recurring_charge += (
feature_detail['price_per_unit_per_period'] * data[feature_name] /
feature_detail['unit']['value']
)
return one_time_charge, recurring_charge
def is_order_valid(order_timestamp, renewal_period):
"""
Sample Code Usage
>> current_datetime, status = cal.parse('Now')
>> current_datetime = datetime(*current_datetime[:6])
>> print('Is order valid: ', is_order_valid(current_datetime, '1 month'))
>> True
"""
cal = parsedatetime.Calendar()
renewal_datetime, status = cal.parse(renewal_period)
renewal_datetime = datetime(*renewal_datetime[:6])
return order_timestamp <= renewal_datetime

28
archive/uncloud_django_based/meow-payv1/products/ipv6-only-django.json
View file

@ -1,28 +0,0 @@
{
"usable-id": "ipv6-only-django-hosting",
"active": true,
"name": "IPv6 Only Django Hosting",
"description": "Host your Django application on our shiny IPv6 Only VM",
"recurring_period": "month",
"quantity": "inf",
"features": {
"cpu": {
"unit": {"value": 1, "type":"int"},
"price_per_unit_per_period": 3,
"one_time_fee": 0,
"constant": false
},
"ram": {
"unit": {"value": 1, "type":"int"},
"price_per_unit_per_period": 4,
"one_time_fee": 0,
"constant": false
},
"os-disk-space": {
"unit": {"value": 10, "type":"int"},
"one_time_fee": 0,
"price_per_unit_per_period": 3.5,
"constant": false
}
}
}

34
archive/uncloud_django_based/meow-payv1/products/ipv6-only-vm.json
View file

@ -1,34 +0,0 @@
{
"usable-id": "ipv6-only-vm",
"active": true,
"name": "IPv6 Only VM",
"description": "IPv6 Only VM are accessible to only those having IPv6 for themselves",
"recurring_period": "month",
"quantity": "inf",
"features": {
"cpu": {
"unit": {"value": 1, "type":"int"},
"price_per_unit_per_period": 3,
"one_time_fee": 0,
"constant": false
},
"ram": {
"unit": {"value": 1, "type":"int"},
"price_per_unit_per_period": 4,
"one_time_fee": 0,
"constant": false
},
"os-disk-space": {
"unit": {"value": 10, "type":"int"},
"one_time_fee": 0,
"price_per_unit_per_period": 4,
"constant": false
},
"os": {
"unit": {"value": 1, "type":"str"},
"one_time_fee": 0,
"price_per_unit_per_period": 0,
"constant": false
}
}
}

16
archive/uncloud_django_based/meow-payv1/products/ipv6-only-vpn.json
View file

@ -1,16 +0,0 @@
{
"usable-id": "ipv6-only-vpn",
"active": true,
"name": "IPv6 Only VPN",
"description": "IPv6 VPN enable you to access IPv6 only websites and more",
"recurring_period": "month",
"quantity": "inf",
"features": {
"vpn": {
"unit": {"value": 1, "type": "int"},
"price_per_unit_per_period": 10,
"one_time_fee": 0,
"constant": true
}
}
}

16
archive/uncloud_django_based/meow-payv1/products/ipv6box.json
View file

@ -1,16 +0,0 @@
{
"usable-id": "ipv6-box",
"active": true,
"name": "IPv6 Box",
"description": "A ready-to-go IPv6 Box: it creates a VPN to ungleich and distributes IPv6 addresses to all your computers.",
"recurring_period": "eternity",
"quantity": 4,
"features": {
"ipv6-box": {
"unit": {"value": 1, "type":"int"},
"price_per_unit_per_period": 0,
"one_time_fee": 250,
"constant": true
}
}
}

17
archive/uncloud_django_based/meow-payv1/products/membership.json
View file

@ -1,17 +0,0 @@
{
"usable-id": "membership",
"active": true,
"name": "Membership",
"description": "Membership to use uncloud-pay",
"recurring_period": "month",
"quantity": "inf",
"features": {
"membership": {
"unit": {"value": 1, "type":"int"},
"price_per_unit_per_period": 5,
"one_time_fee": 0,
"constant": true
}
},
"max_per_user": "1"
}

7
archive/uncloud_django_based/meow-payv1/requirements.txt
View file

@ -1,7 +0,0 @@
stripe
flask
Flask-RESTful
git+https://code.ungleich.ch/ahmedbilal/ungleich-common/#egg=ungleich-common-etcd&subdirectory=etcd
git+https://code.ungleich.ch/ahmedbilal/ungleich-common/#egg=ungleich-common-ldap&subdirectory=ldap
git+https://code.ungleich.ch/ahmedbilal/ungleich-common/#egg=ungleich-common-std&subdirectory=std
git+https://code.ungleich.ch/ahmedbilal/ungleich-common/#egg=ungleich-common-schemas&subdirectory=schemas

17
archive/uncloud_django_based/meow-payv1/sample-pay.conf
View file

@ -1,17 +0,0 @@
[etcd]
host = 127.0.0.1
port = 2379
ca_cert
cert_cert
cert_key
[stripe]
private_key=stripe_private_key
[app]
port = 5000
[ldap]
server = ldap_server_url
admin_dn = ldap_admin_dn
admin_password = ldap_admin_password

136
archive/uncloud_django_based/meow-payv1/schemas.py
View file

@ -1,136 +0,0 @@
import logging
import config
import json
import math
from config import ldap_manager, etcd_client
from helper import resolve_product
from ungleich_common.schemas.schemas import BaseSchema, Field, ValidationException
class AddProductSchema(BaseSchema):
def __init__(self, data):
super().__init__()
self.add_schema(UserCredentialSchema, data)
self.specs = Field('specs', dict, **self.get(data, 'specs'))
self.update = Field('update', bool, **self.get(data, 'update', return_default=True, default=False))
def validation(self):
user = self.objects['user']
user = json.loads(user.entry_to_json())
uid, ou, *dc = user['dn'].replace('ou=', '').replace('dc=', '').replace('uid=', '').split(',')
if ou != config.config.get('ldap', 'internal_user_ou', fallback='users'):
raise ValidationException('You do not have access to create product.')
product = resolve_product(self.specs.value['usable-id'], etcd_client)
if product:
self.objects['product'] = product
class AddressSchema(BaseSchema):
def __init__(self, data):
super().__init__()
self.line1 = Field('line1', str, **self.get(data, 'line1'))
self.line2 = Field('line2', str, **self.get(data, 'line2', return_default=True))
self.city = Field('city', str, **self.get(data, 'city'))
self.country = Field('country', str, **self.get(data, 'country'))
self.state = Field('state', str, **self.get(data, 'state', return_default=True))
self.postal_code = Field('postal_code', str, **self.get(data, 'postal_code', return_default=True))
class UserRegisterPaymentSchema(BaseSchema):
def __init__(self, data):
super().__init__()
self.add_schema(UserCredentialSchema, data)
self.add_schema(AddressSchema, data, under_field_name='address')
self.card_number = Field('card_number', str, **self.get(data, 'card_number'))
self.cvc = Field('cvc', str, **self.get(data, 'cvc'))
self.expiry_year = Field('expiry_year', int, **self.get(data, 'expiry_year'))
self.expiry_month = Field('expiry_month', int, **self.get(data, 'expiry_month'))
self.card_holder_name = Field('card_holder_name', str, **self.get(data, 'card_holder_name'))
class UserCredentialSchema(BaseSchema):
def __init__(self, data):
super().__init__()
self.username = Field('username', str, **self.get(data, 'username'))
self.password = Field('password', str, **self.get(data, 'password'))
def validation(self):
try:
entry = ldap_manager.is_password_valid(self.username.value, self.password.value, query_key='uid')
except ValueError:
raise ValidationException('No user with \'{}\' username found. You can create account at '
'https://account.ungleich.ch'.format(self.username.value))
except Exception:
raise ValidationException('Invalid username/password.')
else:
self.objects['user'] = entry
class ProductOrderSchema(BaseSchema):
def __init__(self, data):
super().__init__()
self.product_id = Field(
'product_id', str, **self.get(data, 'product_id'), validators=[self.product_id_validation]
)
self.pay_consent = Field('pay', bool, **self.get(data, 'pay', return_default=True, default=False))
self.add_schema(UserCredentialSchema, data)
def product_id_validation(self):
product = resolve_product(self.product_id.value, etcd_client)
if product:
product['quantity'] = float(product['quantity'])
self.product_id.value = product['uuid']
self.objects['product'] = product
logging.debug('Got product {}'.format(product))
if not product['active']:
raise ValidationException('Product is not active at the moment.')
if product['quantity'] <= 0:
raise ValidationException('Out of stock.')
else:
raise ValidationException('No such product exists.')
def validation(self):
username = self.objects['user'].uid
customer_previous_orders = etcd_client.get_prefix('/v1/user/{}'.format(username), value_in_json=True)
customer_previous_orders = [o.value for o in customer_previous_orders]
membership = next(filter(lambda o: o['product'] == 'membership', customer_previous_orders), None)
if membership is None and self.objects['product']['usable-id'] != 'membership':
raise ValidationException('Please buy membership first to use this facility')
max_quantity_user_can_order = float(self.objects['product'].get('max_per_user', math.inf))
previous_order_of_same_product = [
o for o in customer_previous_orders if o['product'] == self.objects['product']['usable-id']
]
if len(previous_order_of_same_product) >= max_quantity_user_can_order:
raise ValidationException(
'You cannot buy {} more than {} times'.format(
self.objects['product']['name'], int(max_quantity_user_can_order)
)
)
class OrderListSchema(BaseSchema):
def __init__(self, data):
super().__init__()
self.add_schema(UserCredentialSchema, data)
def make_return_message(err, status_code=200):
logging.debug('message: {}'.format(str(err)))
return {'message': str(err)}, status_code
def create_schema(specification, data):
fields = {}
for feature_name, feature_detail in specification['features'].items():
if not feature_detail['constant']:
fields[feature_name] = Field(
feature_name, eval(feature_detail['unit']['type']), **BaseSchema.get(data, feature_name)
)
return type('{}Schema'.format(specification['name']), (BaseSchema,), fields)

7
archive/uncloud_django_based/meow-payv1/stripe_hack.py
View file

@ -1,7 +0,0 @@
import stripe_utils
import os
if __name__ == '__main__':
s = stripe_utils.StripeUtils(os.environ['STRIPE_PRIVATE_KEY'])
print(s.get_stripe_customer_from_email('coder.purple+2002@gmail.com'))

491
archive/uncloud_django_based/meow-payv1/stripe_utils.py
View file

@ -1,491 +0,0 @@
import re
import stripe
import stripe.error
import logging
from config import etcd_client as client, config as config
stripe.api_key = config.get('stripe', 'private_key')
def handle_stripe_error(f):
def handle_problems(*args, **kwargs):
response = {
'paid': False,
'response_object': None,
'error': None
}
common_message = "Currently it's not possible to make payments."
try:
response_object = f(*args, **kwargs)
response = {
'response_object': response_object,
'error': None
}
return response
except stripe.error.CardError as e:
# Since it's a decline, stripe.error.CardError will be caught
body = e.json_body
err = body['error']
response.update({'error': err['message']})
logging.error(str(e))
return response
except stripe.error.RateLimitError:
response.update(
{'error': "Too many requests made to the API too quickly"})
return response
except stripe.error.InvalidRequestError as e:
logging.error(str(e))
response.update({'error': "Invalid parameters"})
return response
except stripe.error.AuthenticationError as e:
# Authentication with Stripe's API failed
# (maybe you changed API keys recently)
logging.error(str(e))
response.update({'error': common_message})
return response
except stripe.error.APIConnectionError as e:
logging.error(str(e))
response.update({'error': common_message})
return response
except stripe.error.StripeError as e:
# maybe send email
logging.error(str(e))
response.update({'error': common_message})
return response
except Exception as e:
# maybe send email
logging.error(str(e))
response.update({'error': common_message})
return response
return handle_problems
class StripeUtils(object):
CURRENCY = 'chf'
INTERVAL = 'month'
SUCCEEDED_STATUS = 'succeeded'
STRIPE_PLAN_ALREADY_EXISTS = 'Plan already exists'
STRIPE_NO_SUCH_PLAN = 'No such plan'
PLAN_EXISTS_ERROR_MSG = 'Plan {} exists already.\nCreating a local StripePlan now.'
PLAN_DOES_NOT_EXIST_ERROR_MSG = 'Plan {} does not exist.'
def __init__(self, private_key):
self.stripe = stripe
stripe.api_key = private_key
@handle_stripe_error
def card_exists(self, customer, cc_number, exp_month, exp_year, cvc):
token_obj = stripe.Token.create(
card={
'number': cc_number,
'exp_month': exp_month,
'exp_year': exp_year,
'cvc': cvc,
},
)
cards = stripe.Customer.list_sources(
customer,
limit=20,
object='card'
)
for card in cards.data:
if (card.fingerprint == token_obj.card.fingerprint and
int(card.exp_month) == int(exp_month) and int(card.exp_year) == int(exp_year)):
return True
return False
@staticmethod
def get_stripe_customer_from_email(email):
customer = stripe.Customer.list(limit=1, email=email)
return customer.data[0] if len(customer.data) == 1 else None
@staticmethod
def update_customer_token(customer, token):
customer.source = token
customer.save()
@handle_stripe_error
def get_token_from_card(self, cc_number, cvc, expiry_month, expiry_year):
token_obj = stripe.Token.create(
card={
'number': cc_number,
'exp_month': expiry_month,
'exp_year': expiry_year,
'cvc': cvc,
},
)
return token_obj
@handle_stripe_error
def associate_customer_card(self, stripe_customer_id, token,
set_as_default=False):
customer = stripe.Customer.retrieve(stripe_customer_id)
card = customer.sources.create(source=token)
if set_as_default:
customer.default_source = card.id
customer.save()
return True
@handle_stripe_error
def dissociate_customer_card(self, stripe_customer_id, card_id):
customer = stripe.Customer.retrieve(stripe_customer_id)
card = customer.sources.retrieve(card_id)
card.delete()
@handle_stripe_error
def update_customer_card(self, customer_id, token):
customer = stripe.Customer.retrieve(customer_id)
current_card_token = customer.default_source
customer.sources.retrieve(current_card_token).delete()
customer.source = token
customer.save()
credit_card_raw_data = customer.sources.data.pop()
new_card_data = {
'last4': credit_card_raw_data.last4,
'brand': credit_card_raw_data.brand
}
return new_card_data
@handle_stripe_error
def get_card_details(self, customer_id):
customer = stripe.Customer.retrieve(customer_id)
credit_card_raw_data = customer.sources.data.pop()
card_details = {
'last4': credit_card_raw_data.last4,
'brand': credit_card_raw_data.brand,
'exp_month': credit_card_raw_data.exp_month,
'exp_year': credit_card_raw_data.exp_year,
'fingerprint': credit_card_raw_data.fingerprint,
'card_id': credit_card_raw_data.id
}
return card_details
@handle_stripe_error
def get_all_invoices(self, customer_id, created_gt):
return_list = []
has_more_invoices = True
starting_after = False
while has_more_invoices:
if starting_after:
invoices = stripe.Invoice.list(
limit=10, customer=customer_id, created={'gt': created_gt},
starting_after=starting_after
)
else:
invoices = stripe.Invoice.list(
limit=10, customer=customer_id, created={'gt': created_gt}
)
has_more_invoices = invoices.has_more
for invoice in invoices.data:
sub_ids = []
for line in invoice.lines.data:
if line.type == 'subscription':
sub_ids.append(line.id)
elif line.type == 'invoiceitem':
sub_ids.append(line.subscription)
else:
sub_ids.append('')
invoice_details = {
'created': invoice.created,
'receipt_number': invoice.receipt_number,
'invoice_number': invoice.number,
'paid_at': invoice.status_transitions.paid_at if invoice.paid else 0,
'period_start': invoice.period_start,
'period_end': invoice.period_end,
'billing_reason': invoice.billing_reason,
'discount': invoice.discount.coupon.amount_off if invoice.discount else 0,
'total': invoice.total,
# to see how many line items we have in this invoice and
# then later check if we have more than 1
'lines_data_count': len(invoice.lines.data) if invoice.lines.data is not None else 0,
'invoice_id': invoice.id,
'lines_meta_data_csv': ','.join(
[line.metadata.VM_ID if hasattr(line.metadata, 'VM_ID') else '' for line in invoice.lines.data]
),
'subscription_ids_csv': ','.join(sub_ids),
'line_items': invoice.lines.data
}
starting_after = invoice.id
return_list.append(invoice_details)
return return_list
@handle_stripe_error
def get_cards_details_from_token(self, token):
stripe_token = stripe.Token.retrieve(token)
card_details = {
'last4': stripe_token.card.last4,
'brand': stripe_token.card.brand,
'exp_month': stripe_token.card.exp_month,
'exp_year': stripe_token.card.exp_year,
'fingerprint': stripe_token.card.fingerprint,
'card_id': stripe_token.card.id
}
return card_details
def check_customer(self, stripe_cus_api_id, user, token):
try:
customer = stripe.Customer.retrieve(stripe_cus_api_id)
except stripe.error.InvalidRequestError:
customer = self.create_customer(token, user.email, user.name)
user.stripecustomer.stripe_id = customer.get(
'response_object').get('id')
user.stripecustomer.save()
if type(customer) is dict:
customer = customer['response_object']
return customer
@handle_stripe_error
def get_customer(self, stripe_api_cus_id):
customer = stripe.Customer.retrieve(stripe_api_cus_id)
# data = customer.get('response_object')
return customer
@handle_stripe_error
def create_customer(self, token, email, name=None, address=None):
if name is None or name.strip() == "":
name = email
customer = self.stripe.Customer.create(
source=token,
description=name,
email=email,
address=address
)
return customer
@handle_stripe_error
def make_charge(self, amount=None, customer=None):
_amount = float(amount)
amount = int(_amount * 100) # stripe amount unit, in cents
charge = self.stripe.Charge.create(
amount=amount, # in cents
currency=self.CURRENCY,
customer=customer
)
return charge
@staticmethod
def _get_all_stripe_plans():
all_stripe_plans = client.get("/v1/stripe_plans")
all_stripe_plans_set = set()
if all_stripe_plans:
all_stripe_plans_obj = all_stripe_plans.value
if all_stripe_plans_obj and len(all_stripe_plans_obj['plans']) > 0:
all_stripe_plans_set = set(all_stripe_plans_obj["plans"])
return all_stripe_plans_set
@staticmethod
def _save_all_stripe_plans(stripe_plans):
client.put("/v1/stripe_plans", {"plans": list(stripe_plans)})
@handle_stripe_error
def get_or_create_stripe_plan(self, product_name, amount, stripe_plan_id,
interval=INTERVAL):
"""
This function checks if a StripePlan with the given
stripe_plan_id already exists. If it exists then the function
returns this object otherwise it creates a new StripePlan and
returns the new object.
:param amount: The amount in CHF cents
:param product_name: The name of the Stripe plan (product) to be created.
:param stripe_plan_id: The id of the Stripe plan to be
created. Use get_stripe_plan_id_string function to
obtain the name of the plan to be created
:param interval: The interval for subscription {month, year}. Defaults
to month if not provided
:return: The StripePlan object if it exists else creates a
Plan object in Stripe and a local StripePlan and
returns it. Returns None in case of Stripe error
"""
_amount = float(amount)
amount = int(_amount * 100) # stripe amount unit, in cents
all_stripe_plans = self._get_all_stripe_plans()
if stripe_plan_id in all_stripe_plans:
logging.debug("{} plan exists in db.".format(stripe_plan_id))
else:
logging.debug(("{} plan DOES NOT exist in db. "
"Creating").format(stripe_plan_id))
try:
plan_obj = self.stripe.Plan.retrieve(id=stripe_plan_id)
logging.debug("{} plan exists in Stripe".format(stripe_plan_id))
all_stripe_plans.add(stripe_plan_id)
except stripe.error.InvalidRequestError as e:
if "No such plan" in str(e):
logging.debug("Plan {} does not exist in Stripe, Creating")
plan_obj = self.stripe.Plan.create(
amount=amount,
product={'name': product_name},
interval=interval,
currency=self.CURRENCY,
id=stripe_plan_id)
logging.debug(self.PLAN_EXISTS_ERROR_MSG.format(stripe_plan_id))
all_stripe_plans.add(stripe_plan_id)
self._save_all_stripe_plans(all_stripe_plans)
return stripe_plan_id
@handle_stripe_error
def delete_stripe_plan(self, stripe_plan_id):
"""
Deletes the Plan in Stripe and also deletes the local db copy
of the plan if it exists
:param stripe_plan_id: The stripe plan id that needs to be
deleted
:return: True if the plan was deleted successfully from
Stripe, False otherwise.
"""
return_value = False
try:
plan = self.stripe.Plan.retrieve(stripe_plan_id)
plan.delete()
return_value = True
all_stripe_plans = self._get_all_stripe_plans()
all_stripe_plans.remove(stripe_plan_id)
self._save_all_stripe_plans(all_stripe_plans)
except stripe.error.InvalidRequestError as e:
if self.STRIPE_NO_SUCH_PLAN in str(e):
logging.debug(
self.PLAN_DOES_NOT_EXIST_ERROR_MSG.format(stripe_plan_id))
return return_value
@handle_stripe_error
def subscribe_customer_to_plan(self, customer, plans, trial_end=None):
"""
Subscribes the given customer to the list of given plans
:param customer: The stripe customer identifier
:param plans: A list of stripe plans.
:param trial_end: An integer representing when the Stripe subscription
is supposed to end
Ref: https://stripe.com/docs/api/python#create_subscription-items
e.g.
plans = [
{
"plan": "dcl-v1-cpu-2-ram-5gb-ssd-10gb",
},
]
:return: The subscription StripeObject
"""
subscription_result = self.stripe.Subscription.create(
customer=customer, items=plans, trial_end=trial_end
)
return subscription_result
@handle_stripe_error
def set_subscription_metadata(self, subscription_id, metadata):
subscription = stripe.Subscription.retrieve(subscription_id)
subscription.metadata = metadata
subscription.save()
@handle_stripe_error
def unsubscribe_customer(self, subscription_id):
"""
Cancels a given subscription
:param subscription_id: The Stripe subscription id string
:return:
"""
sub = stripe.Subscription.retrieve(subscription_id)
return sub.delete()
@handle_stripe_error
def make_payment(self, customer, amount, token):
charge = self.stripe.Charge.create(
amount=amount, # in cents
currency=self.CURRENCY,
customer=customer
)
return charge
@staticmethod
def get_stripe_plan_id(cpu, ram, ssd, version, app='dcl', hdd=None,
price=None):
"""
Returns the Stripe plan id string of the form
`dcl-v1-cpu-2-ram-5gb-ssd-10gb` based on the input parameters
:param cpu: The number of cores
:param ram: The size of the RAM in GB
:param ssd: The size of ssd storage in GB
:param hdd: The size of hdd storage in GB
:param version: The version of the Stripe plans
:param app: The application to which the stripe plan belongs
to. By default it is 'dcl'
:param price: The price for this plan
:return: A string of the form `dcl-v1-cpu-2-ram-5gb-ssd-10gb`
"""
dcl_plan_string = 'cpu-{cpu}-ram-{ram}gb-ssd-{ssd}gb'.format(cpu=cpu,
ram=ram,
ssd=ssd)
if hdd is not None:
dcl_plan_string = '{dcl_plan_string}-hdd-{hdd}gb'.format(
dcl_plan_string=dcl_plan_string, hdd=hdd)
stripe_plan_id_string = '{app}-v{version}-{plan}'.format(
app=app,
version=version,
plan=dcl_plan_string
)
if price is not None:
stripe_plan_id_string_with_price = '{}-{}chf'.format(
stripe_plan_id_string,
round(price, 2)
)
return stripe_plan_id_string_with_price
else:
return stripe_plan_id_string
@staticmethod
def get_vm_config_from_stripe_id(stripe_id):
"""
Given a string like "dcl-v1-cpu-2-ram-5gb-ssd-10gb" return different
configuration params as a dict
:param stripe_id|str
:return: dict
"""
pattern = re.compile(r'^dcl-v(\d+)-cpu-(\d+)-ram-(\d+\.?\d*)gb-ssd-(\d+)gb-?(\d*\.?\d*)(chf)?$')
match_res = pattern.match(stripe_id)
if match_res is not None:
price = None
try:
price = match_res.group(5)
except IndexError:
logging.debug("Did not find price in {}".format(stripe_id))
return {
'version': match_res.group(1),
'cores': match_res.group(2),
'ram': match_res.group(3),
'ssd': match_res.group(4),
'price': price
}
@staticmethod
def get_stripe_plan_name(cpu, memory, disk_size, price):
"""
Returns the Stripe plan name
:return:
"""
return "{cpu} Cores, {memory} GB RAM, {disk_size} GB SSD, " \
"{price} CHF".format(
cpu=cpu,
memory=memory,
disk_size=disk_size,
price=round(price, 2)
)
@handle_stripe_error
def set_subscription_meta_data(self, subscription_id, meta_data):
"""
Adds VM metadata to a subscription
:param subscription_id: Stripe identifier for the subscription
:param meta_data: A dict of meta data to be added
:return:
"""
subscription = stripe.Subscription.retrieve(subscription_id)
subscription.metadata = meta_data
subscription.save()

338
archive/uncloud_django_based/meow-payv1/ucloud_pay.py
View file

@ -1,338 +0,0 @@
import logging
from datetime import datetime
from uuid import uuid4
from flask import Flask, request
from flask_restful import Resource, Api
from werkzeug.exceptions import HTTPException
from config import etcd_client as client, config as config
from stripe_utils import StripeUtils
from schemas import (
make_return_message, ValidationException, UserRegisterPaymentSchema,
AddProductSchema, ProductOrderSchema, OrderListSchema, create_schema
)
from helper import get_plan_id_from_product, calculate_charges
class ListProducts(Resource):
@staticmethod
def get():
products = client.get_prefix('/v1/products/')
products = [
product
for product in [p.value for p in products]
if product['active']
]
prod_dict = {}
for p in products:
prod_dict[p['usable-id']] = {
'name': p['name'],
'description': p['description'],
}
logger.debug('Products = {}'.format(prod_dict))
return prod_dict, 200
class AddProduct(Resource):
@staticmethod
def post():
data = request.get_json(silent=True) or {}
try:
logger.debug('Got data: {}'.format(str(data)))
validator = AddProductSchema(data)
validator.is_valid()
except ValidationException as err:
return make_return_message(err, 400)
else:
cleaned_values = validator.get_cleaned_values()
previous_product = cleaned_values.get('product', None)
if previous_product:
if not cleaned_values['update']:
return make_return_message('Product already exists. Pass --update to update the product.')
else:
product_uuid = previous_product.pop('uuid')
else:
product_uuid = uuid4().hex
product_value = cleaned_values['specs']
product_key = '/v1/products/{}'.format(product_uuid)
product_value['uuid'] = product_uuid
logger.debug('Adding product data: {}'.format(str(product_value)))
client.put(product_key, product_value)
if not previous_product:
return make_return_message('Product created.')
else:
return make_return_message('Product updated.')
################################################################################
# Nico-ok-marker
class UserRegisterPayment(Resource):
@staticmethod
def post():
data = request.get_json(silent=True) or {}
try:
logger.debug('Got data: {}'.format(str(data)))
validator = UserRegisterPaymentSchema(data)
validator.is_valid()
except ValidationException as err:
return make_return_message(err, 400)
else:
cleaned_values = validator.get_cleaned_values()
last4 = data['card_number'].strip()[-4:]
stripe_utils = StripeUtils()
# Does customer already exist ?
stripe_customer = stripe_utils.get_stripe_customer_from_email(cleaned_values['user']['mail'])
# Does customer already exist ?
if stripe_customer is not None:
logger.debug('Customer {}-{} exists already'.format(
cleaned_values['username'], cleaned_values['user']['mail'])
)
# Check if the card already exists
ce_response = stripe_utils.card_exists(
stripe_customer.id, cc_number=data['card_number'],
exp_month=int(data['expiry_month']),
exp_year=int(data['expiry_year']),
cvc=data['cvc'])
if ce_response['response_object']:
message = 'The given card ending in {} exists already.'.format(last4)
return make_return_message(message, 400)
elif ce_response['response_object'] is False:
# Associate card with user
logger.debug('Adding card ending in {}'.format(last4))
token_response = stripe_utils.get_token_from_card(
data['card_number'], data['cvc'], data['expiry_month'],
data['expiry_year']
)
if token_response['response_object']:
logger.debug('Token {}'.format(token_response['response_object'].id))
resp = stripe_utils.associate_customer_card(
stripe_customer.id, token_response['response_object'].id
)
if resp['response_object']:
return make_return_message(
'Card ending in {} registered as your payment source'.format(last4)
)
else:
return make_return_message('Error with payment gateway. Contact support', 400)
else:
return make_return_message('Error: {}'.format(ce_response['error']), 400)
else:
# Stripe customer does not exist, create a new one
logger.debug(
'Customer {} does not exist, creating new'.format(cleaned_values['user']['mail'])
)
token_response = stripe_utils.get_token_from_card(
cleaned_values['card_number'], cleaned_values['cvc'],
cleaned_values['expiry_month'], cleaned_values['expiry_year']
)
if token_response['response_object']:
logger.debug('Token {}'.format(token_response['response_object'].id))
# Create stripe customer
stripe_customer_resp = stripe_utils.create_customer(
name=cleaned_values['card_holder_name'],
token=token_response['response_object'].id,
email=cleaned_values['user']['mail'],
address=cleaned_values['address']
)
stripe_customer = stripe_customer_resp['response_object']
if stripe_customer:
logger.debug('Created stripe customer {}'.format(stripe_customer.id))
return make_return_message(
'Card ending in {} registered as your payment source'.format(last4)
)
else:
return make_return_message('Error with card. Contact support', 400)
else:
return make_return_message('Error with payment gateway. Contact support', 400)
class ProductOrder(Resource):
@staticmethod
def post():
data = request.get_json(silent=True) or {}
try:
validator = ProductOrderSchema(data)
validator.is_valid()
except ValidationException as err:
return make_return_message(err, 400)
else:
cleaned_values = validator.get_cleaned_values()
stripe_utils = StripeUtils()
product = cleaned_values['product']
# Check the user has a payment source added
stripe_customer = stripe_utils.get_stripe_customer_from_email(cleaned_values['user']['mail'])
if not stripe_customer or len(stripe_customer.sources) == 0:
return make_return_message('Please register your payment method first.', 400)
try:
product_schema = create_schema(product, data)
product_schema = product_schema()
product_schema.is_valid()
except ValidationException as err:
return make_return_message(err, 400)
else:
transformed_data = product_schema.get_cleaned_values()
logger.debug('Tranformed data: {}'.format(transformed_data))
one_time_charge, recurring_charge = calculate_charges(product, transformed_data)
recurring_charge = int(recurring_charge)
if not cleaned_values['pay']:
return make_return_message(
'You would be charged {} CHF one time and {} CHF every {}. '
'Add --pay to command to order.'.format(
one_time_charge, recurring_charge, product['recurring_period']
)
)
with client.client.lock('product-order') as _:
# Initiate a one-time/subscription based on product type
if recurring_charge > 0:
logger.debug('Product {} is recurring payment'.format(product['name']))
plan_id = get_plan_id_from_product(product)
res = stripe_utils.get_or_create_stripe_plan(
product_name=product['name'],
stripe_plan_id=plan_id, amount=recurring_charge,
interval=product['recurring_period'],
)
if res['response_object']:
logger.debug('Obtained plan {}'.format(plan_id))
subscription_res = stripe_utils.subscribe_customer_to_plan(
stripe_customer.id,
[{'plan': plan_id}]
)
subscription_obj = subscription_res['response_object']
if subscription_obj is None or subscription_obj.status != 'active':
return make_return_message(
'Error subscribing to plan. Detail: {}'.format(subscription_res['error']), 400
)
else:
order_obj = {
'order-id': uuid4().hex,
'ordered-at': datetime.now().isoformat(),
'product': product['usable-id'],
'one-time-price': one_time_charge,
'recurring-price': recurring_charge,
'recurring-period': product['recurring_period']
}
client.put(
'/v1/user/{}/orders/{}'.format(
cleaned_values['username'], order_obj['order-id']
), order_obj
)
product['quantity'] -= 1
client.put('/v1/products/{}'.format(product['uuid']), product)
return {
'message': 'Order Successful.',
**order_obj
}
else:
logger.error('Could not create plan {}'.format(plan_id))
return make_return_message('Something wrong happened. Contact administrator', 400)
elif recurring_charge == 0 and one_time_charge > 0:
logger.debug('Product {} is one-time payment'.format(product['name']))
charge_response = stripe_utils.make_charge(
amount=one_time_charge,
customer=stripe_customer.id
)
stripe_onetime_charge = charge_response.get('response_object')
# Check if the payment was approved
if not stripe_onetime_charge:
msg = charge_response.get('error')
return make_return_message('Error subscribing to plan. Details: {}'.format(msg), 400)
order_obj = {
'order-id': uuid4().hex,
'ordered-at': datetime.now().isoformat(),
'product': product['usable-id'],
'one-time-price': one_time_charge,
}
client.put(
'/v1/user/{}/orders/{}'.format(cleaned_values['username'], order_obj['order-id']),
order_obj
)
product['quantity'] -= 1
client.put('/v1/products/{}'.format(product['uuid']), product)
return {'message': 'Order successful', **order_obj}, 200
class OrderList(Resource):
@staticmethod
def post():
data = request.get_json(silent=True) or {}
try:
validator = OrderListSchema(data)
validator.is_valid()
except ValidationException as err:
return make_return_message(err, 400)
else:
cleaned_values = validator.get_cleaned_values()
orders = client.get_prefix('/v1/user/{}/orders'.format(cleaned_values['username']))
orders_dict = {
order.value['order-id']: {
**order.value
}
for order in orders
}
logger.debug('Orders = {}'.format(orders_dict))
return {'orders': orders_dict}, 200
if __name__ == '__main__':
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
log_formater = logging.Formatter('[%(filename)s:%(lineno)d] %(message)s')
stream_logger = logging.StreamHandler()
stream_logger.setFormatter(log_formater)
# file_logger = logging.FileHandler('log.txt')
# file_logger.setLevel(logging.DEBUG)
# file_logger.setFormatter(log_formater)
logger.addHandler(stream_logger)
# logger.addHandler(file_logger)
app = Flask(__name__)
api = Api(app)
api.add_resource(ListProducts, '/product/list')
api.add_resource(AddProduct, '/product/add')
api.add_resource(ProductOrder, '/product/order')
api.add_resource(UserRegisterPayment, '/user/register_payment')
api.add_resource(OrderList, '/order/list')
app.run(host='::', port=config.get('app', 'port', fallback=5000), debug=True)
@app.errorhandler(Exception)
def handle_exception(e):
app.logger.error(e)
# pass through HTTP errors
if isinstance(e, HTTPException):
return e
# now you're handling non-HTTP exceptions only
return {'message': 'Server Error'}, 500

11
archive/uncloud_django_based/notes-abk.md
View file

@ -1,11 +0,0 @@
## TODO 2020-02-22
* ~~move the current rest api to /opennebula~~
* ~~make the /opennebula api only accessible by an admin account~~
* ~~create a new filtered api on /vm/list that~~
* ~~a) requires authentication~~
* ~~b) only shows the VMs of the current user~~
* ~~the new api should not contain all details, but: cpus (as read by the vcpu field), ram, ips, disks~~
* ~~also make a (random) uuid the primary key for VMs - everything in this uncloud hack will use uuids as the id~~
* ~~still expose the opennebula id as opennebula_id~~
* ~~note put all secrets/configs into uncloud.secrets - I added a sample file into the repo~~

102
archive/uncloud_django_based/notes-nico.org
View file

@ -1,102 +0,0 @@
* snapshot feature
** product: vm-snapshot
** flow
*** list all my VMs
**** get the uuid of the VM I want to take a snapshot of
*** request a snapshot
```
vmuuid=$(http nicocustomer
http -a nicocustomer:xxx http://uncloud.ch/vm/create_snapshot uuid=
password=...
```
** backend realisation
*** list snapshots
- have them in the DB
- create an entry on create
*** creating snapshots
- vm sync / fsync?
- rbd snapshot
- host/cluster mapping?
- need image(s)
* steps
** DONE authenticate via ldap
CLOSED: [2020-02-20 Thu 19:05]
** DONE Make classes / views require authentication
CLOSED: [2020-02-20 Thu 19:05]
** TODO register credit card
*** TODO find out what saving with us
*** Info
**** should not be fully saved in the DB
**** model needs to be a bit different
* Decide where to save sensitive data
** stripe access key, etc.
* python requirements (nicohack202002)
django djangorestframework django-auth-ldap stripe
* os package requirements (alpine)
openldap-dev
* VPN case
** put on /orders with uuid
** register cc
* CC
** TODO check whether we can register or not at stripe
* membership
** required for "smaller" / "shorter" products
* TODO Membership missing
* Flows to be implemented - see https://redmine.ungleich.ch/issues/7609
** Membership
*** 5 CHF
** Django Hosting
*** One time payment 35 CHF
*** Monthly payment depends on VM size
*** Parameters: same as IPv6 only VM
** IPv6 VPN
*** Parameters: none
*** Is for free if the customer has an active VM
** IPv6 only VM
*** Parameters: cores, ram, os_disk_size, OS
* Django rest framework
** viewset: .list and .create
** view: .get .post
* TODO register CC
* DONE list products
CLOSED: [2020-02-24 Mon 20:15]
* An ungleich account - can be registered for free on
https://account.ungleich.ch
* httpie installed (provides the http command)
## Get a membership
## Registering a payment method
To be able to pay for the membership, you will need to register a
credit card or apply for payment on bill (TO BE IMPLEMENTED).
### Register credit card
```
http POST https://api.ungleich.ch/membership \
username=nico password=yourpassword \
cc_number=.. \
cc_
```
### Request payment via bill
## Create the membership
```
http POST https://api.ungleich.ch/membership username=nico password=yourpassword
```
## List available products

6
archive/uncloud_django_based/plan.org
View file

@ -1,6 +0,0 @@
* TODO register CC
* TODO list products
* ahmed
** schemas
*** field: is_valid? - used by schemas
*** definition of a "schema"

4
archive/uncloud_django_based/uncloud/.gitignore vendored
View file

@ -1,4 +0,0 @@
db.sqlite3
uncloud/secrets.py
debug.log
uncloud/local_settings.py

29
archive/uncloud_etcd_based/bin/gen-version
View file

@ -1,29 +0,0 @@
#!/bin/sh
# -*- coding: utf-8 -*-
#
# 2019-2020 Nico Schottelius (nico-uncloud at schottelius.org)
#
# This file is part of uncloud.
#
# uncloud is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# uncloud is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with uncloud. If not, see <http://www.gnu.org/licenses/>.
#
#
# Wrapper for real script to allow execution from checkout
dir=${0%/*}
# Ensure version is present - the bundled/shipped version contains a static version,
# the git version contains a dynamic version
printf "VERSION = \"%s\"\n" "$(git describe --tags --abbrev=0)" > ${dir}/../uncloud/version.py

29
archive/uncloud_etcd_based/bin/uncloud-run-reinstall
View file

@ -1,29 +0,0 @@
#!/bin/sh
# -*- coding: utf-8 -*-
#
# 2012-2019 Nico Schottelius (nico-ucloud at schottelius.org)
#
# This file is part of ucloud.
#
# ucloud is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# ucloud is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with ucloud. If not, see <http://www.gnu.org/licenses/>.
#
#
# Wrapper for real script to allow execution from checkout
dir=${0%/*}
${dir}/gen-version;
pip uninstall -y uncloud >/dev/null
python setup.py install >/dev/null
${dir}/uncloud "$@"

13
archive/uncloud_etcd_based/conf/uncloud.conf
View file

@ -1,13 +0,0 @@
[etcd]
url = localhost
port = 2379
base_prefix = /
ca_cert
cert_cert
cert_key
[client]
name = replace_me
realm = replace_me
seed = replace_me
api_server = http://localhost:5000

12
archive/uncloud_etcd_based/docs/README.md
View file

@ -1,12 +0,0 @@
# uncloud docs
## Requirements
1. Python3
2. Sphinx
## Usage
Run `make build` to build docs.
Run `make clean` to remove build directory.
Run `make publish` to push build dir to https://ungleich.ch/ucloud/

36
archive/uncloud_etcd_based/docs/source/hacking.rst
View file

@ -1,36 +0,0 @@
Hacking
=======
Using uncloud in hacking (aka development) mode.
Get the code
------------
.. code-block:: sh
:linenos:
git clone https://code.ungleich.ch/uncloud/uncloud.git
Install python requirements
---------------------------
You need to have python3 installed.
.. code-block:: sh
:linenos:
cd uncloud!
python -m venv venv
. ./venv/bin/activate
./bin/uncloud-run-reinstall
Install os requirements
-----------------------
Install the following software packages: **dnsmasq**.
If you already have a working IPv6 SLAAC and DNS setup,
this step can be skipped.
Note that you need at least one /64 IPv6 network to run uncloud.

66
archive/uncloud_etcd_based/docs/source/vm-images.rst
View file

@ -1,66 +0,0 @@
VM images
==================================
Overview
---------
ucloud tries to be least invasise towards VMs and only require
strictly necessary changes for running in a virtualised
environment. This includes configurations for:
* Configuring the network
* Managing access via ssh keys
* Resizing the attached disk(s)
Upstream images
---------------
The 'official' uncloud images are defined in the `uncloud/images
<https://code.ungleich.ch/uncloud/images>`_ repository.
How to make you own Uncloud images
----------------------------------
.. note::
It is fairly easy to create your own images for uncloud, as the common
operations (which are detailed below) can be automatically handled by the
`uncloud/uncloud-init <https://code.ungleich.ch/uncloud/uncloud-init>`_ tool.
Network configuration
~~~~~~~~~~~~~~~~~~~~~
All VMs in ucloud are required to support IPv6. The primary network
configuration is always done using SLAAC. A VM thus needs only to be
configured to
* accept router advertisements on all network interfaces
* use the router advertisements to configure the network interfaces
* accept the DNS entries from the router advertisements
Configuring SSH keys
~~~~~~~~~~~~~~~~~~~~
To be able to access the VM, ucloud support provisioning SSH keys.
To accept ssh keys in your VM, request the URL
*http://metadata/ssh_keys*. Add the content to the appropriate user's
**authorized_keys** file. Below you find sample code to accomplish
this task:
.. code-block:: sh
tmp=$(mktemp)
curl -s http://metadata/ssk_keys > "$tmp"
touch ~/.ssh/authorized_keys # ensure it exists
cat ~/.ssh/authorized_keys >> "$tmp"
sort "$tmp" | uniq > ~/.ssh/authorized_keys
Disk resize
~~~~~~~~~~~
In virtualised environments, the disk sizes might grow. The operating
system should detect disks that are bigger than the existing partition
table and resize accordingly. This task is os specific.
ucloud does not support shrinking disks due to the complexity and
intra OS dependencies.

89
archive/uncloud_etcd_based/scripts/uncloud
View file

@ -1,89 +0,0 @@
#!/usr/bin/env python3
import logging
import sys
import importlib
import argparse
import os
from etcd3.exceptions import ConnectionFailedError
from uncloud.common import settings
from uncloud import UncloudException
from uncloud.common.cli import resolve_otp_credentials
# Components that use etcd
ETCD_COMPONENTS = ['api', 'scheduler', 'host', 'filescanner',
'imagescanner', 'metadata', 'configure', 'hack']
ALL_COMPONENTS = ETCD_COMPONENTS.copy()
ALL_COMPONENTS.append('oneshot')
#ALL_COMPONENTS.append('cli')
if __name__ == '__main__':
arg_parser = argparse.ArgumentParser()
subparsers = arg_parser.add_subparsers(dest='command')
parent_parser = argparse.ArgumentParser(add_help=False)
parent_parser.add_argument('--debug', '-d', action='store_true', default=False,
help='More verbose logging')
parent_parser.add_argument('--conf-dir', '-c', help='Configuration directory',
default=os.path.expanduser('~/uncloud'))
etcd_parser = argparse.ArgumentParser(add_help=False)
etcd_parser.add_argument('--etcd-host')
etcd_parser.add_argument('--etcd-port')
etcd_parser.add_argument('--etcd-ca-cert', help='CA that signed the etcd certificate')
etcd_parser.add_argument('--etcd-cert-cert', help='Path to client certificate')
etcd_parser.add_argument('--etcd-cert-key', help='Path to client certificate key')
for component in ALL_COMPONENTS:
mod = importlib.import_module('uncloud.{}.main'.format(component))
parser = getattr(mod, 'arg_parser')
if component in ETCD_COMPONENTS:
subparsers.add_parser(name=parser.prog, parents=[parser, parent_parser, etcd_parser])
else:
subparsers.add_parser(name=parser.prog, parents=[parser, parent_parser])
arguments = vars(arg_parser.parse_args())
etcd_arguments = [key for key, value in arguments.items() if key.startswith('etcd_') and value]
etcd_arguments = {
'etcd': {
key.replace('etcd_', ''): arguments[key]
for key in etcd_arguments
}
}
if not arguments['command']:
arg_parser.print_help()
else:
# Initializing Settings and resolving otp_credentials
# It is neccessary to resolve_otp_credentials after argument parsing is done because
# previously we were reading config file which was fixed to ~/uncloud/uncloud.conf and
# providing the default values for --name, --realm and --seed arguments from the values
# we read from file. But, now we are asking user about where the config file lives. So,
# to providing default value is not possible before parsing arguments. So, we are doing
# it after..
# settings.settings = settings.Settings(arguments['conf_dir'], seed_value=etcd_arguments)
# resolve_otp_credentials(arguments)
name = arguments.pop('command')
mod = importlib.import_module('uncloud.{}.main'.format(name))
main = getattr(mod, 'main')
if arguments['debug']:
logging.basicConfig(level=logging.DEBUG)
else:
logging.basicConfig(level=logging.INFO)
log = logging.getLogger()
try:
main(arguments)
except UncloudException as err:
log.error(err)
sys.exit(1)
# except ConnectionFailedError as err:
# log.error('Cannot connect to etcd: {}'.format(err))
except Exception as err:
log.exception(err)

51
archive/uncloud_etcd_based/setup.py
View file

@ -1,51 +0,0 @@
import os
from setuptools import setup, find_packages
with open("README.md", "r") as fh:
long_description = fh.read()
try:
import uncloud.version
version = uncloud.version.VERSION
except:
import subprocess
c = subprocess.check_output(["git", "describe"])
version = c.decode("utf-8").strip()
setup(
name="uncloud",
version=version,
description="uncloud cloud management",
url="https://code.ungleich.ch/uncloud/uncloud",
long_description=long_description,
long_description_content_type="text/markdown",
classifiers=[
"Development Status :: 3 - Alpha",
"License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
"Programming Language :: Python :: 3",
],
author="ungleich",
author_email="technik@ungleich.ch",
packages=find_packages(),
install_requires=[
"requests",
"Flask>=1.1.1",
"flask-restful",
"bitmath",
"pyotp",
"pynetbox",
"colorama",
"etcd3 @ https://github.com/kragniz/python-etcd3/tarball/master#egg=etcd3",
"marshmallow",
"ldap3"
],
scripts=["scripts/uncloud"],
data_files=[
(os.path.expanduser("~/uncloud/"), ["conf/uncloud.conf"])
],
zip_safe=False,
)

37
archive/uncloud_etcd_based/test/test_mac_local.py
View file

@ -1,37 +0,0 @@
import unittest
from unittest.mock import Mock
from uncloud.hack.mac import MAC
from uncloud import UncloudException
class TestMacLocal(unittest.TestCase):
def setUp(self):
self.config = Mock()
self.config.arguments = {"no_db":True}
self.mac = MAC(self.config)
self.mac.create()
def testMacInt(self):
self.assertEqual(self.mac.__int__(), int("0x420000000001",0), "wrong first MAC index")
def testMacRepr(self):
self.assertEqual(self.mac.__repr__(), '420000000001', "wrong first MAC index")
def testMacStr(self):
self.assertEqual(self.mac.__str__(), '42:00:00:00:00:01', "wrong first MAC index")
def testValidationRaise(self):
with self.assertRaises(UncloudException):
self.mac.validate_mac("2")
def testValidation(self):
self.assertTrue(self.mac.validate_mac("42:00:00:00:00:01"), "Validation of a given MAC not working properly")
def testNextMAC(self):
self.mac.create()
self.assertEqual(self.mac.__repr__(), '420000000001', "wrong second MAC index")
self.assertEqual(self.mac.__int__(), int("0x420000000001",0), "wrong second MAC index")
self.assertEqual(self.mac.__str__(), '42:00:00:00:00:01', "wrong second MAC index")
if __name__ == '__main__':
unittest.main()

2
archive/uncloud_etcd_based/uncloud/__init__.py
View file

@ -1,2 +0,0 @@
class UncloudException(Exception):
pass

19
archive/uncloud_etcd_based/uncloud/api/create_image_store.py
View file

@ -1,19 +0,0 @@
import json
import os
from uuid import uuid4
from uncloud.common.shared import shared
data = {
'is_public': True,
'type': 'ceph',
'name': 'images',
'description': 'first ever public image-store',
'attributes': {'list': [], 'key': [], 'pool': 'images'},
}
shared.etcd_client.put(
os.path.join(shared.settings['etcd']['image_store_prefix'], uuid4().hex),
json.dumps(data),
)

600
archive/uncloud_etcd_based/uncloud/api/main.py
View file

@ -1,600 +0,0 @@
import json
import pynetbox
import logging
import argparse
from uuid import uuid4
from os.path import join as join_path
from flask import Flask, request
from flask_restful import Resource, Api
from werkzeug.exceptions import HTTPException
from uncloud.common.shared import shared
from uncloud.common import counters
from uncloud.common.vm import VMStatus
from uncloud.common.request import RequestEntry, RequestType
from uncloud.api import schemas
from uncloud.api.helper import generate_mac, mac2ipv6
from uncloud import UncloudException
logger = logging.getLogger(__name__)
app = Flask(__name__)
api = Api(app)
app.logger.handlers.clear()
arg_parser = argparse.ArgumentParser('api', add_help=False)
arg_parser.add_argument('--port', '-p')
@app.errorhandler(Exception)
def handle_exception(e):
app.logger.error(e)
# pass through HTTP errors
if isinstance(e, HTTPException):
return e
# now you're handling non-HTTP exceptions only
return {'message': 'Server Error'}, 500
class CreateVM(Resource):
"""API Request to Handle Creation of VM"""
@staticmethod
def post():
data = request.json
validator = schemas.CreateVMSchema(data)
if validator.is_valid():
vm_uuid = uuid4().hex
vm_key = join_path(shared.settings['etcd']['vm_prefix'], vm_uuid)
specs = {
'cpu': validator.specs['cpu'],
'ram': validator.specs['ram'],
'os-ssd': validator.specs['os-ssd'],
'hdd': validator.specs['hdd'],
}
macs = [generate_mac() for _ in range(len(data['network']))]
tap_ids = [
counters.increment_etcd_counter(
shared.etcd_client, shared.settings['etcd']['tap_counter']
)
for _ in range(len(data['network']))
]
vm_entry = {
'name': data['vm_name'],
'owner': data['name'],
'owner_realm': data['realm'],
'specs': specs,
'hostname': '',
'status': VMStatus.stopped,
'image_uuid': validator.image_uuid,
'log': [],
'vnc_socket': '',
'network': list(zip(data['network'], macs, tap_ids)),
'metadata': {'ssh-keys': []},
'in_migration': False,
}
shared.etcd_client.put(vm_key, vm_entry, value_in_json=True)
# Create ScheduleVM Request
r = RequestEntry.from_scratch(
type=RequestType.ScheduleVM,
uuid=vm_uuid,
request_prefix=shared.settings['etcd']['request_prefix'],
)
shared.request_pool.put(r)
return {'message': 'VM Creation Queued'}, 200
return validator.get_errors(), 400
class VmStatus(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.VMStatusSchema(data)
if validator.is_valid():
vm = shared.vm_pool.get(
join_path(shared.settings['etcd']['vm_prefix'], data['uuid'])
)
vm_value = vm.value.copy()
vm_value['ip'] = []
for network_mac_and_tap in vm.network:
network_name, mac, tap = network_mac_and_tap
network = shared.etcd_client.get(
join_path(
shared.settings['etcd']['network_prefix'],
data['name'],
network_name,
),
value_in_json=True,
)
ipv6_addr = (
network.value.get('ipv6').split('::')[0] + '::'
)
vm_value['ip'].append(mac2ipv6(mac, ipv6_addr))
vm.value = vm_value
return vm.value
else:
return validator.get_errors(), 400
class CreateImage(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.CreateImageSchema(data)
if validator.is_valid():
file_entry = shared.etcd_client.get(
join_path(shared.settings['etcd']['file_prefix'], data['uuid'])
)
file_entry_value = json.loads(file_entry.value)
image_entry_json = {
'status': 'TO_BE_CREATED',
'owner': file_entry_value['owner'],
'filename': file_entry_value['filename'],
'name': data['name'],
'store_name': data['image_store'],
'visibility': 'public',
}
shared.etcd_client.put(
join_path(
shared.settings['etcd']['image_prefix'], data['uuid']
),
json.dumps(image_entry_json),
)
return {'message': 'Image queued for creation.'}
return validator.get_errors(), 400
class ListPublicImages(Resource):
@staticmethod
def get():
images = shared.etcd_client.get_prefix(
shared.settings['etcd']['image_prefix'], value_in_json=True
)
r = {'images': []}
for image in images:
image_key = '{}:{}'.format(
image.value['store_name'], image.value['name']
)
r['images'].append(
{'name': image_key, 'status': image.value['status']}
)
return r, 200
class VMAction(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.VmActionSchema(data)
if validator.is_valid():
vm_entry = shared.vm_pool.get(
join_path(shared.settings['etcd']['vm_prefix'], data['uuid'])
)
action = data['action']
if action == 'start':
action = 'schedule'
if action == 'delete' and vm_entry.hostname == '':
if shared.storage_handler.is_vm_image_exists(
vm_entry.uuid
):
r_status = shared.storage_handler.delete_vm_image(
vm_entry.uuid
)
if r_status:
shared.etcd_client.client.delete(vm_entry.key)
return {'message': 'VM successfully deleted'}
else:
logger.error(
'Some Error Occurred while deleting VM'
)
return {'message': 'VM deletion unsuccessfull'}
else:
shared.etcd_client.client.delete(vm_entry.key)
return {'message': 'VM successfully deleted'}
r = RequestEntry.from_scratch(
type='{}VM'.format(action.title()),
uuid=data['uuid'],
hostname=vm_entry.hostname,
request_prefix=shared.settings['etcd']['request_prefix'],
)
shared.request_pool.put(r)
return (
{'message': 'VM {} Queued'.format(action.title())},
200,
)
else:
return validator.get_errors(), 400
class VMMigration(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.VmMigrationSchema(data)
if validator.is_valid():
vm = shared.vm_pool.get(data['uuid'])
r = RequestEntry.from_scratch(
type=RequestType.InitVMMigration,
uuid=vm.uuid,
hostname=join_path(
shared.settings['etcd']['host_prefix'],
validator.destination.value,
),
request_prefix=shared.settings['etcd']['request_prefix'],
)
shared.request_pool.put(r)
return (
{'message': 'VM Migration Initialization Queued'},
200,
)
else:
return validator.get_errors(), 400
class ListUserVM(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.OTPSchema(data)
if validator.is_valid():
vms = shared.etcd_client.get_prefix(
shared.settings['etcd']['vm_prefix'], value_in_json=True
)
return_vms = []
user_vms = filter(
lambda v: v.value['owner'] == data['name'], vms
)
for vm in user_vms:
return_vms.append(
{
'name': vm.value['name'],
'vm_uuid': vm.key.split('/')[-1],
'specs': vm.value['specs'],
'status': vm.value['status'],
'hostname': vm.value['hostname'],
'vnc_socket': vm.value.get('vnc_socket', None),
}
)
if return_vms:
return {'message': return_vms}, 200
return {'message': 'No VM found'}, 404
else:
return validator.get_errors(), 400
class ListUserFiles(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.OTPSchema(data)
if validator.is_valid():
files = shared.etcd_client.get_prefix(
shared.settings['etcd']['file_prefix'], value_in_json=True
)
return_files = []
user_files = [f for f in files if f.value['owner'] == data['name']]
for file in user_files:
file_uuid = file.key.split('/')[-1]
file = file.value
file['uuid'] = file_uuid
file.pop('sha512sum', None)
file.pop('owner', None)
return_files.append(file)
return {'message': return_files}, 200
else:
return validator.get_errors(), 400
class CreateHost(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.CreateHostSchema(data)
if validator.is_valid():
host_key = join_path(
shared.settings['etcd']['host_prefix'], uuid4().hex
)
host_entry = {
'specs': data['specs'],
'hostname': data['hostname'],
'status': 'DEAD',
'last_heartbeat': '',
}
shared.etcd_client.put(
host_key, host_entry, value_in_json=True
)
return {'message': 'Host Created'}, 200
return validator.get_errors(), 400
class ListHost(Resource):
@staticmethod
def get():
hosts = shared.host_pool.hosts
r = {
host.key: {
'status': host.status,
'specs': host.specs,
'hostname': host.hostname,
}
for host in hosts
}
return r, 200
class GetSSHKeys(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.GetSSHSchema(data)
if validator.is_valid():
if not validator.key_name.value:
# {user_prefix}/{realm}/{name}/key/
etcd_key = join_path(
shared.settings['etcd']['user_prefix'],
data['realm'],
data['name'],
'key',
)
etcd_entry = shared.etcd_client.get_prefix(
etcd_key, value_in_json=True
)
keys = {
key.key.split('/')[-1]: key.value
for key in etcd_entry
}
return {'keys': keys}
else:
# {user_prefix}/{realm}/{name}/key/{key_name}
etcd_key = join_path(
shared.settings['etcd']['user_prefix'],
data['realm'],
data['name'],
'key',
data['key_name'],
)
etcd_entry = shared.etcd_client.get(
etcd_key, value_in_json=True
)
if etcd_entry:
return {
'keys': {
etcd_entry.key.split('/')[
-1
]: etcd_entry.value
}
}
else:
return {'keys': {}}
else:
return validator.get_errors(), 400
class AddSSHKey(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.AddSSHSchema(data)
if validator.is_valid():
# {user_prefix}/{realm}/{name}/key/{key_name}
etcd_key = join_path(
shared.settings['etcd']['user_prefix'],
data['realm'],
data['name'],
'key',
data['key_name'],
)
etcd_entry = shared.etcd_client.get(
etcd_key, value_in_json=True
)
if etcd_entry:
return {
'message': 'Key with name "{}" already exists'.format(
data['key_name']
)
}
else:
# Key Not Found. It implies user' haven't added any key yet.
shared.etcd_client.put(
etcd_key, data['key'], value_in_json=True
)
return {'message': 'Key added successfully'}
else:
return validator.get_errors(), 400
class RemoveSSHKey(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.RemoveSSHSchema(data)
if validator.is_valid():
# {user_prefix}/{realm}/{name}/key/{key_name}
etcd_key = join_path(
shared.settings['etcd']['user_prefix'],
data['realm'],
data['name'],
'key',
data['key_name'],
)
etcd_entry = shared.etcd_client.get(
etcd_key, value_in_json=True
)
if etcd_entry:
shared.etcd_client.client.delete(etcd_key)
return {'message': 'Key successfully removed.'}
else:
return {
'message': 'No Key with name "{}" Exists at all.'.format(
data['key_name']
)
}
else:
return validator.get_errors(), 400
class CreateNetwork(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.CreateNetwork(data)
if validator.is_valid():
network_entry = {
'id': counters.increment_etcd_counter(
shared.etcd_client, shared.settings['etcd']['vxlan_counter']
),
'type': data['type'],
}
if validator.user.value:
try:
nb = pynetbox.api(
url=shared.settings['netbox']['url'],
token=shared.settings['netbox']['token'],
)
nb_prefix = nb.ipam.prefixes.get(
prefix=shared.settings['network']['prefix']
)
prefix = nb_prefix.available_prefixes.create(
data={
'prefix_length': int(
shared.settings['network']['prefix_length']
),
'description': '{}\'s network "{}"'.format(
data['name'], data['network_name']
),
'is_pool': True,
}
)
except Exception as err:
app.logger.error(err)
return {
'message': 'Error occured while creating network.'
}
else:
network_entry['ipv6'] = prefix['prefix']
else:
network_entry['ipv6'] = 'fd00::/64'
network_key = join_path(
shared.settings['etcd']['network_prefix'],
data['name'],
data['network_name'],
)
shared.etcd_client.put(
network_key, network_entry, value_in_json=True
)
return {'message': 'Network successfully added.'}
else:
return validator.get_errors(), 400
class ListUserNetwork(Resource):
@staticmethod
def post():
data = request.json
validator = schemas.OTPSchema(data)
if validator.is_valid():
prefix = join_path(
shared.settings['etcd']['network_prefix'], data['name']
)
networks = shared.etcd_client.get_prefix(
prefix, value_in_json=True
)
user_networks = []
for net in networks:
net.value['name'] = net.key.split('/')[-1]
user_networks.append(net.value)
return {'networks': user_networks}, 200
else:
return validator.get_errors(), 400
api.add_resource(CreateVM, '/vm/create')
api.add_resource(VmStatus, '/vm/status')
api.add_resource(VMAction, '/vm/action')
api.add_resource(VMMigration, '/vm/migrate')
api.add_resource(CreateImage, '/image/create')
api.add_resource(ListPublicImages, '/image/list-public')
api.add_resource(ListUserVM, '/user/vms')
api.add_resource(ListUserFiles, '/user/files')
api.add_resource(ListUserNetwork, '/user/networks')
api.add_resource(AddSSHKey, '/user/add-ssh')
api.add_resource(RemoveSSHKey, '/user/remove-ssh')
api.add_resource(GetSSHKeys, '/user/get-ssh')
api.add_resource(CreateHost, '/host/create')
api.add_resource(ListHost, '/host/list')
api.add_resource(CreateNetwork, '/network/create')
def main(arguments):
debug = arguments['debug']
port = arguments['port']
try:
image_stores = list(
shared.etcd_client.get_prefix(
shared.settings['etcd']['image_store_prefix'], value_in_json=True
)
)
except KeyError:
image_stores = False
# Do not inject default values that might be very wrong
# fail when required, not before
#
# if not image_stores:
# data = {
# 'is_public': True,
# 'type': 'ceph',
# 'name': 'images',
# 'description': 'first ever public image-store',
# 'attributes': {'list': [], 'key': [], 'pool': 'images'},
# }
# shared.etcd_client.put(
# join_path(
# shared.settings['etcd']['image_store_prefix'], uuid4().hex
# ),
# json.dumps(data),
# )
try:
app.run(host='::', port=port, debug=debug)
except OSError as e:
raise UncloudException('Failed to start Flask: {}'.format(e))

46
archive/uncloud_etcd_based/uncloud/cli/helper.py
View file

@ -1,46 +0,0 @@
import requests
import json
import argparse
import binascii
from pyotp import TOTP
from os.path import join as join_path
from uncloud.common.shared import shared
def get_otp_parser():
otp_parser = argparse.ArgumentParser('otp')
otp_parser.add_argument('--name')
otp_parser.add_argument('--realm')
otp_parser.add_argument('--seed', type=get_token, dest='token', metavar='SEED')
return otp_parser
def load_dump_pretty(content):
if isinstance(content, bytes):
content = content.decode('utf-8')
parsed = json.loads(content)
return json.dumps(parsed, indent=4, sort_keys=True)
def make_request(*args, data=None, request_method=requests.post):
try:
r = request_method(join_path(shared.settings['client']['api_server'], *args), json=data)
except requests.exceptions.RequestException:
print('Error occurred while connecting to API server.')
else:
try:
print(load_dump_pretty(r.content))
except Exception:
print('Error occurred while getting output from api server.')
def get_token(seed):
if seed is not None:
try:
token = TOTP(seed).now()
except binascii.Error:
raise argparse.ArgumentTypeError('Invalid seed')
else:
return token

45
archive/uncloud_etcd_based/uncloud/cli/host.py
View file

@ -1,45 +0,0 @@
import requests
from uncloud.cli.helper import make_request, get_otp_parser
from uncloud.common.parser import BaseParser
class HostParser(BaseParser):
def __init__(self):
super().__init__('host')
def create(self, **kwargs):
p = self.subparser.add_parser('create', parents=[get_otp_parser()], **kwargs)
p.add_argument('--hostname', required=True)
p.add_argument('--cpu', required=True, type=int)
p.add_argument('--ram', required=True)
p.add_argument('--os-ssd', required=True)
p.add_argument('--hdd', default=list())
def list(self, **kwargs):
self.subparser.add_parser('list', **kwargs)
parser = HostParser()
arg_parser = parser.arg_parser
def main(**kwargs):
subcommand = kwargs.pop('host_subcommand')
if not subcommand:
arg_parser.print_help()
else:
request_method = requests.post
data = None
if subcommand == 'create':
kwargs['specs'] = {
'cpu': kwargs.pop('cpu'),
'ram': kwargs.pop('ram'),
'os-ssd': kwargs.pop('os_ssd'),
'hdd': kwargs.pop('hdd')
}
data = kwargs
elif subcommand == 'list':
request_method = requests.get
make_request('host', subcommand, data=data, request_method=request_method)

38
archive/uncloud_etcd_based/uncloud/cli/image.py
View file

@ -1,38 +0,0 @@
import requests
from uncloud.cli.helper import make_request
from uncloud.common.parser import BaseParser
class ImageParser(BaseParser):
def __init__(self):
super().__init__('image')
def create(self, **kwargs):
p = self.subparser.add_parser('create', **kwargs)
p.add_argument('--name', required=True)
p.add_argument('--uuid', required=True)
p.add_argument('--image-store', required=True, dest='image_store')
def list(self, **kwargs):
self.subparser.add_parser('list', **kwargs)
parser = ImageParser()
arg_parser = parser.arg_parser
def main(**kwargs):
subcommand = kwargs.pop('image_subcommand')
if not subcommand:
arg_parser.print_help()
else:
data = None
request_method = requests.post
if subcommand == 'list':
subcommand = 'list-public'
request_method = requests.get
elif subcommand == 'create':
data = kwargs
make_request('image', subcommand, data=data, request_method=request_method)

23
archive/uncloud_etcd_based/uncloud/cli/main.py
View file

@ -1,23 +0,0 @@
#!/usr/bin/env python3
import argparse
import importlib
arg_parser = argparse.ArgumentParser('cli', add_help=False)
subparser = arg_parser.add_subparsers(dest='subcommand')
for component in ['user', 'host', 'image', 'network', 'vm']:
module = importlib.import_module('uncloud.cli.{}'.format(component))
parser = getattr(module, 'arg_parser')
subparser.add_parser(name=parser.prog, parents=[parser])
def main(arguments):
if not arguments['subcommand']:
arg_parser.print_help()
else:
name = arguments.pop('subcommand')
arguments.pop('debug')
mod = importlib.import_module('uncloud.cli.{}'.format(name))
_main = getattr(mod, 'main')
_main(**arguments)

32
archive/uncloud_etcd_based/uncloud/cli/network.py
View file

@ -1,32 +0,0 @@
import requests
from uncloud.cli.helper import make_request, get_otp_parser
from uncloud.common.parser import BaseParser
class NetworkParser(BaseParser):
def __init__(self):
super().__init__('network')
def create(self, **kwargs):
p = self.subparser.add_parser('create', parents=[get_otp_parser()], **kwargs)
p.add_argument('--network-name', required=True)
p.add_argument('--network-type', required=True, dest='type')
p.add_argument('--user', action='store_true')
parser = NetworkParser()
arg_parser = parser.arg_parser
def main(**kwargs):
subcommand = kwargs.pop('network_subcommand')
if not subcommand:
arg_parser.print_help()
else:
data = None
request_method = requests.post
if subcommand == 'create':
data = kwargs
make_request('network', subcommand, data=data, request_method=request_method)

41
archive/uncloud_etcd_based/uncloud/cli/user.py
View file

@ -1,41 +0,0 @@
from uncloud.cli.helper import make_request, get_otp_parser
from uncloud.common.parser import BaseParser
class UserParser(BaseParser):
def __init__(self):
super().__init__('user')
def files(self, **kwargs):
self.subparser.add_parser('files', parents=[get_otp_parser()], **kwargs)
def vms(self, **kwargs):
self.subparser.add_parser('vms', parents=[get_otp_parser()], **kwargs)
def networks(self, **kwargs):
self.subparser.add_parser('networks', parents=[get_otp_parser()], **kwargs)
def add_ssh(self, **kwargs):
p = self.subparser.add_parser('add-ssh', parents=[get_otp_parser()], **kwargs)
p.add_argument('--key-name', required=True)
p.add_argument('--key', required=True)
def get_ssh(self, **kwargs):
p = self.subparser.add_parser('get-ssh', parents=[get_otp_parser()], **kwargs)
p.add_argument('--key-name', default='')
def remove_ssh(self, **kwargs):
p = self.subparser.add_parser('remove-ssh', parents=[get_otp_parser()], **kwargs)
p.add_argument('--key-name', required=True)
parser = UserParser()
arg_parser = parser.arg_parser
def main(**kwargs):
subcommand = kwargs.pop('user_subcommand')
if not subcommand:
arg_parser.print_help()
else:
make_request('user', subcommand, data=kwargs)

62
archive/uncloud_etcd_based/uncloud/cli/vm.py
View file

@ -1,62 +0,0 @@
from uncloud.common.parser import BaseParser
from uncloud.cli.helper import make_request, get_otp_parser
class VMParser(BaseParser):
def __init__(self):
super().__init__('vm')
def start(self, **args):
p = self.subparser.add_parser('start', parents=[get_otp_parser()], **args)
p.add_argument('--vm-name', required=True)
def stop(self, **args):
p = self.subparser.add_parser('stop', parents=[get_otp_parser()], **args)
p.add_argument('--vm-name', required=True)
def status(self, **args):
p = self.subparser.add_parser('status', parents=[get_otp_parser()], **args)
p.add_argument('--vm-name', required=True)
def delete(self, **args):
p = self.subparser.add_parser('delete', parents=[get_otp_parser()], **args)
p.add_argument('--vm-name', required=True)
def migrate(self, **args):
p = self.subparser.add_parser('migrate', parents=[get_otp_parser()], **args)
p.add_argument('--vm-name', required=True)
p.add_argument('--destination', required=True)
def create(self, **args):
p = self.subparser.add_parser('create', parents=[get_otp_parser()], **args)
p.add_argument('--cpu', required=True)
p.add_argument('--ram', required=True)
p.add_argument('--os-ssd', required=True)
p.add_argument('--hdd', action='append', default=list())
p.add_argument('--image', required=True)
p.add_argument('--network', action='append', default=[])
p.add_argument('--vm-name', required=True)
parser = VMParser()
arg_parser = parser.arg_parser
def main(**kwargs):
subcommand = kwargs.pop('vm_subcommand')
if not subcommand:
arg_parser.print_help()
else:
data = kwargs
endpoint = subcommand
if subcommand in ['start', 'stop', 'delete']:
endpoint = 'action'
data['action'] = subcommand
elif subcommand == 'create':
kwargs['specs'] = {
'cpu': kwargs.pop('cpu'),
'ram': kwargs.pop('ram'),
'os-ssd': kwargs.pop('os_ssd'),
'hdd': kwargs.pop('hdd')
}
make_request('vm', endpoint, data=data)

23
archive/uncloud_etcd_based/uncloud/client/main.py
View file

@ -1,23 +0,0 @@
import argparse
import etcd3
from uncloud.common.etcd_wrapper import Etcd3Wrapper
arg_parser = argparse.ArgumentParser('client', add_help=False)
arg_parser.add_argument('--dump-etcd-contents-prefix', help="Dump contents below the given prefix")
def dump_etcd_contents(prefix):
etcd = Etcd3Wrapper()
for k,v in etcd.get_prefix_raw(prefix):
k = k.decode('utf-8')
v = v.decode('utf-8')
print("{} = {}".format(k,v))
# print("{} = {}".format(k,v))
# for k,v in etcd.get_prefix(prefix):
#
print("done")
def main(arguments):
if 'dump_etcd_contents_prefix' in arguments:
dump_etcd_contents(prefix=arguments['dump_etcd_contents_prefix'])

26
archive/uncloud_etcd_based/uncloud/common/cli.py
View file

@ -1,26 +0,0 @@
from uncloud.common.shared import shared
from pyotp import TOTP
def get_token(seed):
if seed is not None:
try:
token = TOTP(seed).now()
except Exception:
raise Exception('Invalid seed')
else:
return token
def resolve_otp_credentials(kwargs):
d = {
'name': shared.settings['client']['name'],
'realm': shared.settings['client']['realm'],
'token': get_token(shared.settings['client']['seed'])
}
for k, v in d.items():
if k in kwargs and kwargs[k] is None:
kwargs.update({k: v})
return d

75
archive/uncloud_etcd_based/uncloud/common/etcd_wrapper.py
View file

@ -1,75 +0,0 @@
import etcd3
import json
from functools import wraps
from uncloud import UncloudException
from uncloud.common import logger
class EtcdEntry:
def __init__(self, meta_or_key, value, value_in_json=False):
if hasattr(meta_or_key, 'key'):
# if meta has attr 'key' then get it
self.key = meta_or_key.key.decode('utf-8')
else:
# otherwise meta is the 'key'
self.key = meta_or_key
self.value = value.decode('utf-8')
if value_in_json:
self.value = json.loads(self.value)
def readable_errors(func):
@wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except etcd3.exceptions.ConnectionFailedError:
raise UncloudException('Cannot connect to etcd: is etcd running as configured in uncloud.conf?')
except etcd3.exceptions.ConnectionTimeoutError as err:
raise etcd3.exceptions.ConnectionTimeoutError('etcd connection timeout.') from err
except Exception:
logger.exception('Some etcd error occured. See syslog for details.')
return wrapper
class Etcd3Wrapper:
@readable_errors
def __init__(self, *args, **kwargs):
self.client = etcd3.client(*args, **kwargs)
@readable_errors
def get(self, *args, value_in_json=False, **kwargs):
_value, _key = self.client.get(*args, **kwargs)
if _key is None or _value is None:
return None
return EtcdEntry(_key, _value, value_in_json=value_in_json)
@readable_errors
def put(self, *args, value_in_json=False, **kwargs):
_key, _value = args
if value_in_json:
_value = json.dumps(_value)
if not isinstance(_key, str):
_key = _key.decode('utf-8')
return self.client.put(_key, _value, **kwargs)
@readable_errors
def get_prefix(self, *args, value_in_json=False, raise_exception=True, **kwargs):
event_iterator = self.client.get_prefix(*args, **kwargs)
for e in event_iterator:
yield EtcdEntry(*e[::-1], value_in_json=value_in_json)
@readable_errors
def watch_prefix(self, key, raise_exception=True, value_in_json=False):
event_iterator, cancel = self.client.watch_prefix(key)
for e in event_iterator:
if hasattr(e, '_event'):
e = e._event
if e.type == e.PUT:
yield EtcdEntry(e.kv.key, e.kv.value, value_in_json=value_in_json)

70
archive/uncloud_etcd_based/uncloud/common/network.py
View file

@ -1,70 +0,0 @@
import subprocess as sp
import random
import logging
logger = logging.getLogger(__name__)
def random_bytes(num=6):
return [random.randrange(256) for _ in range(num)]
def generate_mac(
uaa=False, multicast=False, oui=None, separator=":", byte_fmt="%02x"
):
mac = random_bytes()
if oui:
if type(oui) == str:
oui = [int(chunk) for chunk in oui.split(separator)]
mac = oui + random_bytes(num=6 - len(oui))
else:
if multicast:
mac[0] |= 1 # set bit 0
else:
mac[0] &= ~1 # clear bit 0
if uaa:
mac[0] &= ~(1 << 1) # clear bit 1
else:
mac[0] |= 1 << 1 # set bit 1
return separator.join(byte_fmt % b for b in mac)
def create_dev(script, _id, dev, ip=None):
command = [
"sudo",
"-p",
"Enter password to create network devices for vm: ",
script,
str(_id),
dev,
]
if ip:
command.append(ip)
try:
output = sp.check_output(command, stderr=sp.PIPE)
except Exception:
logger.exception("Creation of interface %s failed.", dev)
return None
else:
return output.decode("utf-8").strip()
def delete_network_interface(iface):
try:
sp.check_output(
[
"sudo",
"-p",
"Enter password to remove {} network device: ".format(
iface
),
"ip",
"link",
"del",
iface,
],
stderr=sp.PIPE,
)
except Exception:
logger.exception("Interface %s Deletion failed", iface)

13
archive/uncloud_etcd_based/uncloud/common/parser.py
View file

@ -1,13 +0,0 @@
import argparse
class BaseParser:
def __init__(self, command):
self.arg_parser = argparse.ArgumentParser(command, add_help=False)
self.subparser = self.arg_parser.add_subparsers(dest='{}_subcommand'.format(command))
self.common_args = {'add_help': False}
methods = [attr for attr in dir(self) if not attr.startswith('__')
and type(getattr(self, attr)).__name__ == 'method']
for method in methods:
getattr(self, method)(**self.common_args)

41
archive/uncloud_etcd_based/uncloud/common/schemas.py
View file

@ -1,41 +0,0 @@
import bitmath
from marshmallow import fields, Schema
class StorageUnit(fields.Field):
def _serialize(self, value, attr, obj, **kwargs):
return str(value)
def _deserialize(self, value, attr, data, **kwargs):
return bitmath.parse_string_unsafe(value)
class SpecsSchema(Schema):
cpu = fields.Int()
ram = StorageUnit()
os_ssd = StorageUnit(data_key="os-ssd", attribute="os-ssd")
hdd = fields.List(StorageUnit())
class VMSchema(Schema):
name = fields.Str()
owner = fields.Str()
owner_realm = fields.Str()
specs = fields.Nested(SpecsSchema)
status = fields.Str()
log = fields.List(fields.Str())
vnc_socket = fields.Str()
image_uuid = fields.Str()
hostname = fields.Str()
metadata = fields.Dict()
network = fields.List(
fields.Tuple((fields.Str(), fields.Str(), fields.Int()))
)
in_migration = fields.Bool()
class NetworkSchema(Schema):
_id = fields.Int(data_key="id", attribute="id")
_type = fields.Str(data_key="type", attribute="type")
ipv6 = fields.Str()

136
archive/uncloud_etcd_based/uncloud/common/settings.py
View file

@ -1,136 +0,0 @@
import configparser
import logging
import sys
import os
from datetime import datetime
from uncloud.common.etcd_wrapper import Etcd3Wrapper
from os.path import join as join_path
logger = logging.getLogger(__name__)
settings = None
class CustomConfigParser(configparser.RawConfigParser):
def __getitem__(self, key):
try:
result = super().__getitem__(key)
except KeyError as err:
raise KeyError(
'Key \'{}\' not found in configuration. Make sure you configure uncloud.'.format(
key
)
) from err
else:
return result
class Settings(object):
def __init__(self, conf_dir, seed_value=None):
conf_name = 'uncloud.conf'
self.config_file = join_path(conf_dir, conf_name)
# this is used to cache config from etcd for 1 minutes. Without this we
# would make a lot of requests to etcd which slows down everything.
self.last_config_update = datetime.fromtimestamp(0)
self.config_parser = CustomConfigParser(allow_no_value=True)
self.config_parser.add_section('etcd')
self.config_parser.set('etcd', 'base_prefix', '/')
if os.access(self.config_file, os.R_OK):
self.config_parser.read(self.config_file)
else:
raise FileNotFoundError('Config file %s not found!', self.config_file)
self.config_key = join_path(self['etcd']['base_prefix'] + 'uncloud/config/')
self.read_internal_values()
if seed_value is None:
seed_value = dict()
self.config_parser.read_dict(seed_value)
def get_etcd_client(self):
args = tuple()
try:
kwargs = {
'host': self.config_parser.get('etcd', 'url'),
'port': self.config_parser.get('etcd', 'port'),
'ca_cert': self.config_parser.get('etcd', 'ca_cert'),
'cert_cert': self.config_parser.get('etcd', 'cert_cert'),
'cert_key': self.config_parser.get('etcd', 'cert_key'),
}
except configparser.Error as err:
raise configparser.Error(
'{} in config file {}'.format(
err.message, self.config_file
)
) from err
else:
try:
wrapper = Etcd3Wrapper(*args, **kwargs)
except Exception as err:
logger.error(
'etcd connection not successfull. Please check your config file.'
'\nDetails: %s\netcd connection parameters: %s',
err,
kwargs,
)
sys.exit(1)
else:
return wrapper
def read_internal_values(self):
base_prefix = self['etcd']['base_prefix']
self.config_parser.read_dict(
{
'etcd': {
'file_prefix': join_path(base_prefix, 'files/'),
'host_prefix': join_path(base_prefix, 'hosts/'),
'image_prefix': join_path(base_prefix, 'images/'),
'image_store_prefix': join_path(base_prefix, 'imagestore/'),
'network_prefix': join_path(base_prefix, 'networks/'),
'request_prefix': join_path(base_prefix, 'requests/'),
'user_prefix': join_path(base_prefix, 'users/'),
'vm_prefix': join_path(base_prefix, 'vms/'),
'vxlan_counter': join_path(base_prefix, 'counters/vxlan'),
'tap_counter': join_path(base_prefix, 'counters/tap')
}
}
)
def read_config_file_values(self, config_file):
try:
# Trying to read configuration file
with open(config_file) as config_file_handle:
self.config_parser.read_file(config_file_handle)
except FileNotFoundError:
sys.exit('Configuration file {} not found!'.format(config_file))
except Exception as err:
logger.exception(err)
sys.exit('Error occurred while reading configuration file')
def read_values_from_etcd(self):
etcd_client = self.get_etcd_client()
if (datetime.utcnow() - self.last_config_update).total_seconds() > 60:
config_from_etcd = etcd_client.get(self.config_key, value_in_json=True)
if config_from_etcd:
self.config_parser.read_dict(config_from_etcd.value)
self.last_config_update = datetime.utcnow()
else:
raise KeyError('Key \'{}\' not found in etcd. Please configure uncloud.'.format(self.config_key))
def __getitem__(self, key):
# Allow failing to read from etcd if we have
# it locally
if key not in self.config_parser.sections():
try:
self.read_values_from_etcd()
except KeyError:
pass
return self.config_parser[key]
def get_settings():
return settings

34
archive/uncloud_etcd_based/uncloud/common/shared.py
View file

@ -1,34 +0,0 @@
from uncloud.common.settings import get_settings
from uncloud.common.vm import VmPool
from uncloud.common.host import HostPool
from uncloud.common.request import RequestPool
import uncloud.common.storage_handlers as storage_handlers
class Shared:
@property
def settings(self):
return get_settings()
@property
def etcd_client(self):
return self.settings.get_etcd_client()
@property
def host_pool(self):
return HostPool(self.etcd_client, self.settings["etcd"]["host_prefix"])
@property
def vm_pool(self):
return VmPool(self.etcd_client, self.settings["etcd"]["vm_prefix"])
@property
def request_pool(self):
return RequestPool(self.etcd_client, self.settings["etcd"]["request_prefix"])
@property
def storage_handler(self):
return storage_handlers.get_storage_handler()
shared = Shared()

57
archive/uncloud_etcd_based/uncloud/configure/main.py
View file

@ -1,57 +0,0 @@
import os
import argparse
from uncloud.common.shared import shared
arg_parser = argparse.ArgumentParser('configure', add_help=False)
configure_subparsers = arg_parser.add_subparsers(dest='subcommand')
otp_parser = configure_subparsers.add_parser('otp')
otp_parser.add_argument('--verification-controller-url', required=True, metavar='URL')
otp_parser.add_argument('--auth-name', required=True, metavar='OTP-NAME')
otp_parser.add_argument('--auth-realm', required=True, metavar='OTP-REALM')
otp_parser.add_argument('--auth-seed', required=True, metavar='OTP-SEED')
network_parser = configure_subparsers.add_parser('network')
network_parser.add_argument('--prefix-length', required=True, type=int)
network_parser.add_argument('--prefix', required=True)
network_parser.add_argument('--vxlan-phy-dev', required=True)
netbox_parser = configure_subparsers.add_parser('netbox')
netbox_parser.add_argument('--url', required=True)
netbox_parser.add_argument('--token', required=True)
ssh_parser = configure_subparsers.add_parser('ssh')
ssh_parser.add_argument('--username', default='root')
ssh_parser.add_argument('--private-key-path', default=os.path.expanduser('~/.ssh/id_rsa'),)
storage_parser = configure_subparsers.add_parser('storage')
storage_parser.add_argument('--file-dir', required=True)
storage_parser_subparsers = storage_parser.add_subparsers(dest='storage_backend')
filesystem_storage_parser = storage_parser_subparsers.add_parser('filesystem')
filesystem_storage_parser.add_argument('--vm-dir', required=True)
filesystem_storage_parser.add_argument('--image-dir', required=True)
ceph_storage_parser = storage_parser_subparsers.add_parser('ceph')
ceph_storage_parser.add_argument('--ceph-vm-pool', required=True)
ceph_storage_parser.add_argument('--ceph-image-pool', required=True)
def update_config(section, kwargs):
uncloud_config = shared.etcd_client.get(shared.settings.config_key, value_in_json=True)
if not uncloud_config:
uncloud_config = {}
else:
uncloud_config = uncloud_config.value
uncloud_config[section] = kwargs
shared.etcd_client.put(shared.settings.config_key, uncloud_config, value_in_json=True)
def main(arguments):
subcommand = arguments['subcommand']
if not subcommand:
arg_parser.print_help()
else:
update_config(subcommand, arguments)

85
archive/uncloud_etcd_based/uncloud/filescanner/main.py
View file

@ -1,85 +0,0 @@
import glob
import os
import pathlib
import subprocess as sp
import time
import argparse
import bitmath
from uuid import uuid4
from . import logger
from uncloud.common.shared import shared
arg_parser = argparse.ArgumentParser('filescanner', add_help=False)
arg_parser.add_argument('--hostname', required=True)
def sha512sum(file: str):
"""Use sha512sum utility to compute sha512 sum of arg:file
IF arg:file does not exists:
raise FileNotFoundError exception
ELSE IF sum successfully computer:
return computed sha512 sum
ELSE:
return None
"""
if not isinstance(file, str):
raise TypeError
try:
output = sp.check_output(['sha512sum', file], stderr=sp.PIPE)
except sp.CalledProcessError as e:
error = e.stderr.decode('utf-8')
if 'No such file or directory' in error:
raise FileNotFoundError from None
else:
output = output.decode('utf-8').strip()
output = output.split(' ')
return output[0]
return None
def track_file(file, base_dir, host):
file_path = file.relative_to(base_dir)
file_str = str(file)
# Get Username
try:
owner = file_path.parts[0]
except IndexError:
pass
else:
file_path = file_path.relative_to(owner)
creation_date = time.ctime(os.stat(file_str).st_ctime)
entry_key = os.path.join(shared.settings['etcd']['file_prefix'], str(uuid4()))
entry_value = {
'filename': str(file_path),
'owner': owner,
'sha512sum': sha512sum(file_str),
'creation_date': creation_date,
'size': str(bitmath.Byte(os.path.getsize(file_str)).to_MB()),
'host': host
}
logger.info('Tracking %s', file_str)
shared.etcd_client.put(entry_key, entry_value, value_in_json=True)
def main(arguments):
hostname = arguments['hostname']
base_dir = shared.settings['storage']['file_dir']
# Recursively Get All Files and Folder below BASE_DIR
files = glob.glob('{}/**'.format(base_dir), recursive=True)
files = [pathlib.Path(f) for f in files if pathlib.Path(f).is_file()]
# Files that are already tracked
tracked_files = [
pathlib.Path(os.path.join(base_dir, f.value['owner'], f.value['filename']))
for f in shared.etcd_client.get_prefix(shared.settings['etcd']['file_prefix'], value_in_json=True)
if f.value['host'] == hostname
]
untracked_files = set(files) - set(tracked_files)
for file in untracked_files:
track_file(file, base_dir, hostname)

1
archive/uncloud_etcd_based/uncloud/hack/__init__.py
View file

@ -1 +0,0 @@

39
archive/uncloud_etcd_based/uncloud/hack/config.py
View file

@ -1,39 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# 2020 Nico Schottelius (nico.schottelius at ungleich.ch)
#
# This file is part of uncloud.
#
# uncloud is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# uncloud is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with uncloud. If not, see <http://www.gnu.org/licenses/>.
#
#
class Config(object):
def __init__(self, arguments):
""" read arguments dicts as a base """
self.arguments = arguments
# Split them so *etcd_args can be used and we can
# iterate over etcd_hosts
self.etcd_hosts = [ arguments['etcd_host'] ]
self.etcd_args = {
'ca_cert': arguments['etcd_ca_cert'],
'cert_cert': arguments['etcd_cert_cert'],
'cert_key': arguments['etcd_cert_key'],
# 'user': None,
# 'password': None
}
self.etcd_prefix = '/nicohack/'

149
archive/uncloud_etcd_based/uncloud/hack/db.py
View file

@ -1,149 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# 2020 Nico Schottelius (nico.schottelius at ungleich.ch)
#
# This file is part of uncloud.
#
# uncloud is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# uncloud is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with uncloud. If not, see <http://www.gnu.org/licenses/>.
#
#
import etcd3
import json
import logging
import datetime
import re
from functools import wraps
from uncloud import UncloudException
log = logging.getLogger(__name__)
def db_logentry(message):
timestamp = datetime.datetime.now()
return {
"timestamp": str(timestamp),
"message": message
}
def readable_errors(func):
@wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except etcd3.exceptions.ConnectionFailedError as e:
raise UncloudException('Cannot connect to etcd: is etcd running and reachable? {}'.format(e))
except etcd3.exceptions.ConnectionTimeoutError as e:
raise UncloudException('etcd connection timeout. {}'.format(e))
return wrapper
class DB(object):
def __init__(self, config, prefix="/"):
self.config = config
# Root for everything
self.base_prefix= '/nicohack'
# Can be set from outside
self.prefix = prefix
try:
self.connect()
except FileNotFoundError as e:
raise UncloudException("Is the path to the etcd certs correct? {}".format(e))
@readable_errors
def connect(self):
self._db_clients = []
for endpoint in self.config.etcd_hosts:
client = etcd3.client(host=endpoint, **self.config.etcd_args)
self._db_clients.append(client)
def realkey(self, key):
return "{}{}/{}".format(self.base_prefix,
self.prefix,
key)
@readable_errors
def get(self, key, as_json=False, **kwargs):
value, _ = self._db_clients[0].get(self.realkey(key), **kwargs)
if as_json:
value = json.loads(value)
return value
@readable_errors
def get_prefix(self, key, as_json=False, **kwargs):
for value, meta in self._db_clients[0].get_prefix(self.realkey(key), **kwargs):
k = meta.key.decode("utf-8")
value = value.decode("utf-8")
if as_json:
value = json.loads(value)
yield (k, value)
@readable_errors
def set(self, key, value, as_json=False, **kwargs):
if as_json:
value = json.dumps(value)
log.debug("Setting {} = {}".format(self.realkey(key), value))
# FIXME: iterate over clients in case of failure ?
return self._db_clients[0].put(self.realkey(key), value, **kwargs)
@readable_errors
def list_and_filter(self, key, filter_key=None, filter_regexp=None):
for k,v in self.get_prefix(key, as_json=True):
if filter_key and filter_regexp:
if filter_key in v:
if re.match(filter_regexp, v[filter_key]):
yield v
else:
yield v
@readable_errors
def increment(self, key, **kwargs):
print(self.realkey(key))
print("prelock")
lock = self._db_clients[0].lock('/nicohack/foo')
print("prelockacq")
lock.acquire()
print("prelockrelease")
lock.release()
with self._db_clients[0].lock("/nicohack/mac/last_used_index") as lock:
print("in lock")
pass
# with self._db_clients[0].lock(self.realkey(key)) as lock:# value = int(self.get(self.realkey(key), **kwargs))
# self.set(self.realkey(key), str(value + 1), **kwargs)
if __name__ == '__main__':
endpoints = [ "https://etcd1.ungleich.ch:2379",
"https://etcd2.ungleich.ch:2379",
"https://etcd3.ungleich.ch:2379" ]
db = DB(url=endpoints)

3
archive/uncloud_etcd_based/uncloud/hack/hackcloud/.gitignore vendored
View file

@ -1,3 +0,0 @@
*.iso
radvdpid
foo

1
archive/uncloud_etcd_based/uncloud/hack/hackcloud/__init__.py
View file

@ -1 +0,0 @@

6
archive/uncloud_etcd_based/uncloud/hack/hackcloud/etcd-client.sh
View file

@ -1,6 +0,0 @@
#!/bin/sh
etcdctl --cert=$HOME/vcs/ungleich-dot-cdist/files/etcd/nico.pem \
--key=/home/nico/vcs/ungleich-dot-cdist/files/etcd/nico-key.pem \
--cacert=$HOME/vcs/ungleich-dot-cdist/files/etcd/ca.pem \
--endpoints https://etcd1.ungleich.ch:2379,https://etcd2.ungleich.ch:2379,https://etcd3.ungleich.ch:2379 "$@"

3
archive/uncloud_etcd_based/uncloud/hack/hackcloud/ifdown.sh
View file

@ -1,3 +0,0 @@
#!/bin/sh
echo $@

7
archive/uncloud_etcd_based/uncloud/hack/hackcloud/ifup.sh
View file

@ -1,7 +0,0 @@
#!/bin/sh
dev=$1; shift
# bridge is setup from outside
ip link set dev "$dev" master ${bridge}
ip link set dev "$dev" up

1
archive/uncloud_etcd_based/uncloud/hack/hackcloud/mac-last
View file

@ -1 +0,0 @@
000000000252

1
archive/uncloud_etcd_based/uncloud/hack/hackcloud/mac-prefix
View file

@ -1 +0,0 @@
02:00

29
archive/uncloud_etcd_based/uncloud/hack/hackcloud/net.sh
View file

@ -1,29 +0,0 @@
#!/bin/sh
set -x
netid=100
dev=wlp2s0
dev=wlp0s20f3
#dev=wlan0
ip=2a0a:e5c1:111:888::48/64
vxlandev=vxlan${netid}
bridgedev=br${netid}
ip -6 link add ${vxlandev} type vxlan \
id ${netid} \
dstport 4789 \
group ff05::${netid} \
dev ${dev} \
ttl 5
ip link set ${vxlandev} up
ip link add ${bridgedev} type bridge
ip link set ${bridgedev} up
ip link set ${vxlandev} master ${bridgedev} up
ip addr add ${ip} dev ${bridgedev}

31
archive/uncloud_etcd_based/uncloud/hack/hackcloud/nftrules
View file

@ -1,31 +0,0 @@
flush ruleset
table bridge filter {
chain prerouting {
type filter hook prerouting priority 0;
policy accept;
ibrname br100 jump br100
}
chain br100 {
# Allow all incoming traffic from outside
iifname vxlan100 accept
# Default blocks: router advertisements, dhcpv6, dhcpv4
icmpv6 type nd-router-advert drop
ip6 version 6 udp sport 547 drop
ip version 4 udp sport 67 drop
jump br100_vmlist
drop
}
chain br100_vmlist {
# VM1
iifname tap1 ether saddr 02:00:f0:a9:c4:4e ip6 saddr 2a0a:e5c1:111:888:0:f0ff:fea9:c44e accept
# VM2
iifname v343a-0 ether saddr 02:00:f0:a9:c4:4f ip6 saddr 2a0a:e5c1:111:888:0:f0ff:fea9:c44f accept
iifname v343a-0 ether saddr 02:00:f0:a9:c4:4f ip6 saddr 2a0a:e5c1:111:1234::/64 accept
}
}

64
archive/uncloud_etcd_based/uncloud/hack/hackcloud/nftrules-v2
View file

@ -1,64 +0,0 @@
flush ruleset
table bridge filter {
chain prerouting {
type filter hook prerouting priority 0;
policy accept;
ibrname br100 jump netpublic
}
chain netpublic {
iifname vxlan100 jump from_uncloud
# Default blocks: router advertisements, dhcpv6, dhcpv4
icmpv6 type nd-router-advert drop
ip6 version 6 udp sport 547 drop
ip version 4 udp sport 67 drop
# Individual blocks
# iifname tap1 jump vm1
}
chain vm1 {
ether saddr != 02:00:f0:a9:c4:4e drop
ip6 saddr != 2a0a:e5c1:111:888:0:f0ff:fea9:c44e drop
}
chain from_uncloud {
accept
}
}
# table ip6 filter {
# chain forward {
# type filter hook forward priority 0;
# # policy drop;
# ct state established,related accept;
# }
# }
# table ip filter {
# chain input {
# type filter hook input priority filter; policy drop;
# iif "lo" accept
# icmp type { echo-reply, destination-unreachable, source-quench, redirect, echo-request, router-advertisement, router-solicitation, time-exceeded, parameter-problem, timestamp-request, timestamp-reply, info-request, info-reply, address-mask-request, address-mask-reply } accept
# ct state established,related accept
# tcp dport { 22 } accept
# log prefix "firewall-ipv4: "
# udp sport 67 drop
# }
# chain forward {
# type filter hook forward priority filter; policy drop;
# log prefix "firewall-ipv4: "
# }
# chain output {
# type filter hook output priority filter; policy accept;
# }
# }

13
archive/uncloud_etcd_based/uncloud/hack/hackcloud/radvd.conf
View file

@ -1,13 +0,0 @@
interface br100
{
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 5;
AdvDefaultLifetime 3600;
prefix 2a0a:e5c1:111:888::/64 {
};
RDNSS 2a0a:e5c0::3 2a0a:e5c0::4 { AdvRDNSSLifetime 6000; };
DNSSL place7.ungleich.ch { AdvDNSSLLifetime 6000; } ;
};

3
archive/uncloud_etcd_based/uncloud/hack/hackcloud/radvd.sh
View file

@ -1,3 +0,0 @@
#!/bin/sh
radvd -C ./radvd.conf -n -p ./radvdpid

24
archive/uncloud_etcd_based/uncloud/hack/hackcloud/vm-2.sh
View file

@ -1,24 +0,0 @@
#!/bin/sh
vmid=$1; shift
qemu=/usr/bin/qemu-system-x86_64
accel=kvm
#accel=tcg
memory=1024
cores=2
uuid=732e08c7-84f8-4d43-9571-263db4f80080
export bridge=br100
$qemu -name uc${vmid} \
-machine pc,accel=${accel} \
-m ${memory} \
-smp ${cores} \
-uuid ${uuid} \
-drive file=alpine-virt-3.11.2-x86_64.iso,media=cdrom \
-drive file=alpine-virt-3.11.2-x86_64.iso,media=cdrom \
-netdev tap,id=netmain,script=./ifup.sh \
-device virtio-net-pci,netdev=netmain,id=net0,mac=02:00:f0:a9:c4:4e

29
archive/uncloud_etcd_based/uncloud/hack/hackcloud/vm.sh
View file

@ -1,29 +0,0 @@
#!/bin/sh
# if [ $# -ne 1 ]; then
# echo "$0: owner"
# exit 1
# fi
qemu=/usr/bin/qemu-system-x86_64
accel=kvm
#accel=tcg
memory=1024
cores=2
uuid=$(uuidgen)
mac=$(./mac-gen.py)
owner=nico
export bridge=br100
set -x
$qemu -name "uncloud-${uuid}" \
-machine pc,accel=${accel} \
-m ${memory} \
-smp ${cores} \
-uuid ${uuid} \
-drive file=alpine-virt-3.11.2-x86_64.iso,media=cdrom \
-netdev tap,id=netmain,script=./ifup.sh,downscript=./ifdown.sh \
-device virtio-net-pci,netdev=netmain,id=net0,mac=${mac}

75
archive/uncloud_etcd_based/uncloud/hack/host.py
View file

@ -1,75 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# 2020 Nico Schottelius (nico.schottelius at ungleich.ch)
#
# This file is part of uncloud.
#
# uncloud is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# uncloud is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with uncloud. If not, see <http://www.gnu.org/licenses/>.
import uuid
from uncloud.hack.db import DB
from uncloud import UncloudException
class Host(object):
def __init__(self, config, db_entry=None):
self.config = config
self.db = DB(self.config, prefix="/hosts")
if db_entry:
self.db_entry = db_entry
def list_hosts(self, filter_key=None, filter_regexp=None):
""" Return list of all hosts """
for entry in self.db.list_and_filter("", filter_key, filter_regexp):
yield self.__class__(self.config, db_entry=entry)
def cmdline_add_host(self):
""" FIXME: make this a bit smarter and less redundant """
for required_arg in [
'add_vm_host',
'max_cores_per_vm',
'max_cores_total',
'max_memory_in_gb' ]:
if not required_arg in self.config.arguments:
raise UncloudException("Missing argument: {}".format(required_arg))
return self.add_host(
self.config.arguments['add_vm_host'],
self.config.arguments['max_cores_per_vm'],
self.config.arguments['max_cores_total'],
self.config.arguments['max_memory_in_gb'])
def add_host(self,
hostname,
max_cores_per_vm,
max_cores_total,
max_memory_in_gb):
db_entry = {}
db_entry['uuid'] = str(uuid.uuid4())
db_entry['hostname'] = hostname
db_entry['max_cores_per_vm'] = max_cores_per_vm
db_entry['max_cores_total'] = max_cores_total
db_entry['max_memory_in_gb'] = max_memory_in_gb
db_entry["db_version"] = 1
db_entry["log"] = []
self.db.set(db_entry['uuid'], db_entry, as_json=True)
return self.__class__(self.config, db_entry)

104
archive/uncloud_etcd_based/uncloud/hack/mac.py
View file

@ -1,104 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# 2012 Nico Schottelius (nico-cinv at schottelius.org)
#
# This file is part of cinv.
#
# cinv is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# cinv is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with cinv. If not, see <http://www.gnu.org/licenses/>.
#
#
import argparse
import logging
import os.path
import os
import re
import json
from uncloud import UncloudException
from uncloud.hack.db import DB
log = logging.getLogger(__name__)
class MAC(object):
def __init__(self, config):
self.config = config
self.no_db = self.config.arguments['no_db']
if not self.no_db:
self.db = DB(config, prefix="/mac")
self.prefix = 0x420000000000
self._number = 0 # Not set by default
@staticmethod
def validate_mac(mac):
if not re.match(r'([0-9A-F]{2}[-:]){5}[0-9A-F]{2}$', mac, re.I):
raise UncloudException("Not a valid mac address: %s" % mac)
else:
return True
def last_used_index(self):
if not self.no_db:
value = self.db.get("last_used_index")
if not value:
self.db.set("last_used_index", "0")
value = self.db.get("last_used_index")
else:
value = "0"
return int(value)
def last_used_mac(self):
return self.int_to_mac(self.prefix + self.last_used_index())
def to_colon_format(self):
b = self._number.to_bytes(6, byteorder="big")
return ':'.join(format(s, '02x') for s in b)
def to_str_format(self):
b = self._number.to_bytes(6, byteorder="big")
return ''.join(format(s, '02x') for s in b)
def create(self):
last_number = self.last_used_index()
if last_number == int('0xffffffff', 16):
raise UncloudException("Exhausted all possible mac addresses - try to free some")
next_number = last_number + 1
self._number = self.prefix + next_number
#next_number_string = "{:012x}".format(next_number)
#next_mac = self.int_to_mac(next_mac_number)
# db_entry = {}
# db_entry['vm_uuid'] = vmuuid
# db_entry['index'] = next_number
# db_entry['mac_address'] = next_mac
# should be one transaction
# self.db.increment("last_used_index")
# self.db.set("used/{}".format(next_mac),
# db_entry, as_json=True)
def __int__(self):
return self._number
def __repr__(self):
return self.to_str_format()
def __str__(self):
return self.to_colon_format()

186
archive/uncloud_etcd_based/uncloud/hack/main.py
View file

@ -1,186 +0,0 @@
import argparse
import logging
import re
import ldap3
from uncloud.hack.vm import VM
from uncloud.hack.host import Host
from uncloud.hack.config import Config
from uncloud.hack.mac import MAC
from uncloud.hack.net import VXLANBridge, DNSRA
from uncloud import UncloudException
from uncloud.hack.product import ProductOrder
arg_parser = argparse.ArgumentParser('hack', add_help=False)
#description="Commands that are unfinished - use at own risk")
arg_parser.add_argument('--last-used-mac', action='store_true')
arg_parser.add_argument('--get-new-mac', action='store_true')
arg_parser.add_argument('--init-network', help="Initialise networking", action='store_true')
arg_parser.add_argument('--create-vxlan', help="Initialise networking", action='store_true')
arg_parser.add_argument('--network', help="/64 IPv6 network")
arg_parser.add_argument('--vxlan-uplink-device', help="The VXLAN underlay device, i.e. eth0")
arg_parser.add_argument('--vni', help="VXLAN ID (decimal)", type=int)
arg_parser.add_argument('--run-dns-ra', action='store_true',
help="Provide router advertisements and DNS resolution via dnsmasq")
arg_parser.add_argument('--use-sudo', help="Use sudo for command requiring root!", action='store_true')
arg_parser.add_argument('--create-vm', action='store_true')
arg_parser.add_argument('--destroy-vm', action='store_true')
arg_parser.add_argument('--get-vm-status', action='store_true')
arg_parser.add_argument('--get-vm-vnc', action='store_true')
arg_parser.add_argument('--list-vms', action='store_true')
arg_parser.add_argument('--memory', help="Size of memory (GB)", type=int, default=2)
arg_parser.add_argument('--cores', help="Amount of CPU cores", type=int, default=1)
arg_parser.add_argument('--image', help="Path (under hackprefix) to OS image")
arg_parser.add_argument('--image-format', help="Image format: qcow2 or raw", choices=['raw', 'qcow2'])
arg_parser.add_argument('--uuid', help="VM UUID")
arg_parser.add_argument('--no-db', help="Disable connection to etcd. For local testing only!", action='store_true')
arg_parser.add_argument('--hackprefix', help="hackprefix, if you need it you know it (it's where the iso is located and ifup/down.sh")
# order based commands => later to be shifted below "order"
arg_parser.add_argument('--order', action='store_true')
arg_parser.add_argument('--list-orders', help="List all orders", action='store_true')
arg_parser.add_argument('--filter-order-key', help="Which key to filter on")
arg_parser.add_argument('--filter-order-regexp', help="Which regexp the value should match")
arg_parser.add_argument('--process-orders', help="Process all (pending) orders", action='store_true')
arg_parser.add_argument('--product', choices=["dualstack-vm"])
arg_parser.add_argument('--os-image-name', help="Name of OS image (successor to --image)")
arg_parser.add_argument('--os-image-size', help="Size of OS image in GB", type=int, default=10)
arg_parser.add_argument('--username')
arg_parser.add_argument('--password')
arg_parser.add_argument('--api', help="Run the API")
arg_parser.add_argument('--mode',
choices=["direct", "api", "client"],
default="client",
help="Directly manipulate etcd, spawn the API server or behave as a client")
arg_parser.add_argument('--add-vm-host', help="Add a host that can run VMs")
arg_parser.add_argument('--list-vm-hosts', action='store_true')
arg_parser.add_argument('--max-cores-per-vm')
arg_parser.add_argument('--max-cores-total')
arg_parser.add_argument('--max-memory-in-gb')
log = logging.getLogger(__name__)
def authenticate(username, password, totp_token=None):
server = ldap3.Server("ldaps://ldap1.ungleich.ch")
dn = "uid={},ou=customer,dc=ungleich,dc=ch".format(username)
log.debug("LDAP: connecting to {} as {}".format(server, dn))
try:
conn = ldap3.Connection(server, dn, password, auto_bind=True)
except ldap3.core.exceptions.LDAPBindError as e:
raise UncloudException("Credentials not verified by LDAP server: {}".format(e))
def order(config):
for required_arg in [ 'product', 'username', 'password' ]:
if not config.arguments[required_arg]:
raise UncloudException("Missing required argument: {}".format(required_arg))
if config.arguments['product'] == 'dualstack-vm':
for required_arg in [ 'cores', 'memory', 'os_image_name', 'os_image_size' ]:
if not config.arguments[required_arg]:
raise UncloudException("Missing required argument: {}".format(required_arg))
log.debug(config.arguments)
authenticate(config.arguments['username'], config.arguments['password'])
# create DB entry for VM
vm = VM(config)
return vm.product.place_order(owner=config.arguments['username'])
def main(arguments):
config = Config(arguments)
if arguments['add_vm_host']:
h = Host(config)
h.cmdline_add_host()
if arguments['list_vm_hosts']:
h = Host(config)
for host in h.list_hosts(filter_key=arguments['filter_order_key'],
filter_regexp=arguments['filter_order_regexp']):
print("Host {}: {}".format(host.db_entry['uuid'], host.db_entry))
if arguments['order']:
print("Created order: {}".format(order(config)))
if arguments['list_orders']:
p = ProductOrder(config)
for product_order in p.list_orders(filter_key=arguments['filter_order_key'],
filter_regexp=arguments['filter_order_regexp']):
print("Order {}: {}".format(product_order.db_entry['uuid'], product_order.db_entry))
if arguments['process_orders']:
p = ProductOrder(config)
p.process_orders()
if arguments['create_vm']:
vm = VM(config)
vm.create()
if arguments['destroy_vm']:
vm = VM(config)
vm.stop()
if arguments['get_vm_status']:
vm = VM(config)
vm.status()
if arguments['get_vm_vnc']:
vm = VM(config)
vm.vnc_addr()
if arguments['list_vms']:
vm = VM(config)
vm.list()
if arguments['last_used_mac']:
m = MAC(config)
print(m.last_used_mac())
if arguments['get_new_mac']:
print(MAC(config).get_next())
#if arguments['init_network']:
if arguments['create_vxlan']:
if not arguments['network'] or not arguments['vni'] or not arguments['vxlan_uplink_device']:
raise UncloudException("Initialising the network requires an IPv6 network and a VNI. You can use fd00::/64 and vni=1 for testing (non production!)")
vb = VXLANBridge(vni=arguments['vni'],
route=arguments['network'],
uplinkdev=arguments['vxlan_uplink_device'],
use_sudo=arguments['use_sudo'])
vb._setup_vxlan()
vb._setup_bridge()
vb._add_vxlan_to_bridge()
vb._route_network()
if arguments['run_dns_ra']:
if not arguments['network'] or not arguments['vni']:
raise UncloudException("Providing DNS/RAs requires a /64 IPv6 network and a VNI. You can use fd00::/64 and vni=1 for testing (non production!)")
dnsra = DNSRA(route=arguments['network'],
vni=arguments['vni'],
use_sudo=arguments['use_sudo'])
dnsra._setup_dnsmasq()

116
archive/uncloud_etcd_based/uncloud/hack/net.py
View file

@ -1,116 +0,0 @@
import subprocess
import ipaddress
import logging
from uncloud import UncloudException
log = logging.getLogger(__name__)
class VXLANBridge(object):
cmd_create_vxlan = "{sudo}ip -6 link add {vxlandev} type vxlan id {vni_dec} dstport 4789 group {multicast_address} dev {uplinkdev} ttl 5"
cmd_up_dev = "{sudo}ip link set {dev} up"
cmd_create_bridge="{sudo}ip link add {bridgedev} type bridge"
cmd_add_to_bridge="{sudo}ip link set {vxlandev} master {bridgedev} up"
cmd_add_addr="{sudo}ip addr add {ip} dev {bridgedev}"
cmd_add_route_dev="{sudo}ip route add {route} dev {bridgedev}"
# VXLAN ids are at maximum 24 bit - use a /104
multicast_network = ipaddress.IPv6Network("ff05::/104")
max_vni = (2**24)-1
def __init__(self,
vni,
uplinkdev,
route=None,
use_sudo=False):
self.config = {}
if vni > self.max_vni:
raise UncloudException("VNI must be in the range of 0 .. {}".format(self.max_vni))
if use_sudo:
self.config['sudo'] = 'sudo '
else:
self.config['sudo'] = ''
self.config['vni_dec'] = vni
self.config['vni_hex'] = "{:x}".format(vni)
self.config['multicast_address'] = self.multicast_network[vni]
self.config['route_network'] = ipaddress.IPv6Network(route)
self.config['route'] = route
self.config['uplinkdev'] = uplinkdev
self.config['vxlandev'] = "vx{}".format(self.config['vni_hex'])
self.config['bridgedev'] = "br{}".format(self.config['vni_hex'])
def setup_networking(self):
pass
def _setup_vxlan(self):
self._execute_cmd(self.cmd_create_vxlan)
self._execute_cmd(self.cmd_up_dev, dev=self.config['vxlandev'])
def _setup_bridge(self):
self._execute_cmd(self.cmd_create_bridge)
self._execute_cmd(self.cmd_up_dev, dev=self.config['bridgedev'])
def _route_network(self):
self._execute_cmd(self.cmd_add_route_dev)
def _add_vxlan_to_bridge(self):
self._execute_cmd(self.cmd_add_to_bridge)
def _execute_cmd(self, cmd_string, **kwargs):
cmd = cmd_string.format(**self.config, **kwargs)
log.info("Executing: {}".format(cmd))
subprocess.run(cmd.split())
class ManagementBridge(VXLANBridge):
pass
class DNSRA(object):
# VXLAN ids are at maximum 24 bit
max_vni = (2**24)-1
# Command to start dnsmasq
cmd_start_dnsmasq="{sudo}dnsmasq --interface={bridgedev} --bind-interfaces --dhcp-range={route},ra-only,infinite --enable-ra --no-daemon"
def __init__(self,
vni,
route=None,
use_sudo=False):
self.config = {}
if vni > self.max_vni:
raise UncloudException("VNI must be in the range of 0 .. {}".format(self.max_vni))
if use_sudo:
self.config['sudo'] = 'sudo '
else:
self.config['sudo'] = ''
#TODO: remove if not needed
#self.config['vni_dec'] = vni
self.config['vni_hex'] = "{:x}".format(vni)
# dnsmasq only wants the network without the prefix, therefore, cut it off
self.config['route'] = ipaddress.IPv6Network(route).network_address
self.config['bridgedev'] = "br{}".format(self.config['vni_hex'])
def _setup_dnsmasq(self):
self._execute_cmd(self.cmd_start_dnsmasq)
def _execute_cmd(self, cmd_string, **kwargs):
cmd = cmd_string.format(**self.config, **kwargs)
log.info("Executing: {}".format(cmd))
print("Executing: {}".format(cmd))
subprocess.run(cmd.split())
class Firewall(object):
pass

206
archive/uncloud_etcd_based/uncloud/hack/product.py
View file

@ -1,206 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# 2020 Nico Schottelius (nico.schottelius at ungleich.ch)
#
# This file is part of uncloud.
#
# uncloud is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# uncloud is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with uncloud. If not, see <http://www.gnu.org/licenses/>.
import json
import uuid
import logging
import re
import importlib
from uncloud import UncloudException
from uncloud.hack.db import DB, db_logentry
log = logging.getLogger(__name__)
class ProductOrder(object):
def __init__(self, config, product_entry=None, db_entry=None):
self.config = config
self.db = DB(self.config, prefix="/orders")
self.db_entry = {}
self.db_entry["product"] = product_entry
# Overwrite if we are loading an existing product order
if db_entry:
self.db_entry = db_entry
# FIXME: this should return a list of our class!
def list_orders(self, filter_key=None, filter_regexp=None):
for entry in self.db.list_and_filter("", filter_key, filter_regexp):
yield self.__class__(self.config, db_entry=entry)
def set_required_values(self):
"""Set values that are required to make the db entry valid"""
if not "uuid" in self.db_entry:
self.db_entry["uuid"] = str(uuid.uuid4())
if not "status" in self.db_entry:
self.db_entry["status"] = "NEW"
if not "owner" in self.db_entry:
self.db_entry["owner"] = "UNKNOWN"
if not "log" in self.db_entry:
self.db_entry["log"] = []
if not "db_version" in self.db_entry:
self.db_entry["db_version"] = 1
def validate_status(self):
if "status" in self.db_entry:
if self.db_entry["status"] in [ "NEW",
"SCHEDULED",
"CREATED_ACTIVE",
"CANCELLED",
"REJECTED" ]:
return False
return True
def order(self):
self.set_required_values()
if not self.db_entry["status"] == "NEW":
raise UncloudException("Cannot re-order same order. Status: {}".format(self.db_entry["status"]))
self.db.set(self.db_entry["uuid"], self.db_entry, as_json=True)
return self.db_entry["uuid"]
def process_orders(self):
"""processing orders can be done stand alone on server side"""
for order in self.list_orders():
if order.db_entry["status"] == "NEW":
log.info("Handling new order: {}".format(order))
# FIXME: these all should be a transactions! -> fix concurrent access! !
if not "log" in order.db_entry:
order.db_entry['log'] = []
is_valid = True
# Verify the order entry
for must_attribute in [ "owner", "product" ]:
if not must_attribute in order.db_entry:
message = "Missing {} entry in order, rejecting order".format(must_attribute)
log.info("Rejecting order {}: {}".format(order.db_entry["uuid"], message))
order.db_entry['log'].append(db_logentry(message))
order.db_entry['status'] = "REJECTED"
self.db.set(order.db_entry['uuid'], order.db_entry, as_json=True)
is_valid = False
# Rejected the order
if not is_valid:
continue
# Verify the product entry
for must_attribute in [ "python_product_class", "python_product_module" ]:
if not must_attribute in order.db_entry['product']:
message = "Missing {} entry in product of order, rejecting order".format(must_attribute)
log.info("Rejecting order {}: {}".format(order.db_entry["uuid"], message))
order.db_entry['log'].append(db_logentry(message))
order.db_entry['status'] = "REJECTED"
self.db.set(order.db_entry['uuid'], order.db_entry, as_json=True)
is_valid = False
# Rejected the order
if not is_valid:
continue
print(order.db_entry["product"]["python_product_class"])
# Create the product
m = importlib.import_module(order.db_entry["product"]["python_product_module"])
c = getattr(m, order.db_entry["product"]["python_product_class"])
product = c(config, db_entry=order.db_entry["product"])
# STOPPED
product.create_product()
order.db_entry['status'] = "SCHEDULED"
self.db.set(order.db_entry['uuid'], order.db_entry, as_json=True)
def __str__(self):
return str(self.db_entry)
class Product(object):
def __init__(self,
config,
product_name,
product_class,
db_entry=None):
self.config = config
self.db = DB(self.config, prefix="/orders")
self.db_entry = {}
self.db_entry["product_name"] = product_name
self.db_entry["python_product_class"] = product_class.__qualname__
self.db_entry["python_product_module"] = product_class.__module__
self.db_entry["db_version"] = 1
self.db_entry["log"] = []
self.db_entry["features"] = {}
# Existing product? Read in db_entry
if db_entry:
self.db_entry = db_entry
self.valid_periods = [ "per_year", "per_month", "per_week",
"per_day", "per_hour",
"per_minute", "per_second" ]
def define_feature(self,
name,
one_time_price,
recurring_price,
recurring_period,
minimum_period):
self.db_entry['features'][name] = {}
self.db_entry['features'][name]['one_time_price'] = one_time_price
self.db_entry['features'][name]['recurring_price'] = recurring_price
if not recurring_period in self.valid_periods:
raise UncloudException("Invalid recurring period: {}".format(recurring_period))
self.db_entry['features'][name]['recurring_period'] = recurring_period
if not minimum_period in self.valid_periods:
raise UncloudException("Invalid recurring period: {}".format(recurring_period))
recurring_index = self.valid_periods.index(recurring_period)
minimum_index = self.valid_periods.index(minimum_period)
if minimum_index < recurring_index:
raise UncloudException("Minimum period for product '{}' feature '{}' must be shorter or equal than/as recurring period: {} > {}".format(self.db_entry['product_name'], name, minimum_period, recurring_period))
self.db_entry['features'][name]['minimum_period'] = minimum_period
def validate_product(self):
for feature in self.db_entry['features']:
pass
def place_order(self, owner):
""" Schedule creating the product in etcd """
order = ProductOrder(self.config, product_entry=self.db_entry)
order.db_entry["owner"] = owner
return order.order()
def __str__(self):
return json.dumps(self.db_entry)

26
archive/uncloud_etcd_based/uncloud/hack/uncloud-hack-init-host
View file

@ -1,26 +0,0 @@
id=100
rawdev=eth0
# create vxlan
ip -6 link add vxlan${id} type vxlan \
id ${id} \
dstport 4789 \
group ff05::${id} \
dev ${rawdev} \
ttl 5
ip link set vxlan${id} up
# create bridge
ip link set vxlan${id} up
ip link set br${id} up
# Add vxlan into bridge
ip link set vxlan${id} master br${id}
# useradd -m uncloud
# [18:05] tablett.place10:~# id uncloud
# uid=1000(uncloud) gid=1000(uncloud) groups=1000(uncloud),34(kvm),36(qemu)
# apk add qemu-system-x86_64
# also needs group netdev

25
archive/uncloud_etcd_based/uncloud/hack/uncloud-run-vm
View file

@ -1,25 +0,0 @@
#!/bin/sh
if [ $# -ne 1 ]; then
echo $0 vmid
exit 1
fi
id=$1; shift
memory=512
macaddress=02:00:b9:cb:70:${id}
netname=net${id}-1
qemu-system-x86_64 \
-name uncloud-${id} \
-accel kvm \
-m ${memory} \
-smp 2,sockets=2,cores=1,threads=1 \
-device virtio-net-pci,netdev=net0,mac=$macaddress \
-netdev tap,id=net0,ifname=${netname},script=no,downscript=no \
-vnc [::]:0
# To be changed:
# -vnc to unix path
# or -spice

193
archive/uncloud_etcd_based/uncloud/hack/vm.py
View file

@ -1,193 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# 2020 Nico Schottelius (nico.schottelius at ungleich.ch)
#
# This file is part of uncloud.
#
# uncloud is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# uncloud is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with uncloud. If not, see <http://www.gnu.org/licenses/>.
# This module is directly called from the hack module, and can be used as follow:
#
# Create a new VM with default CPU/Memory. The path of the image file is relative to $hackprefix.
# `uncloud hack --hackprefix /tmp/hackcloud --create-vm --image mysuperimage.qcow2`
#
# List running VMs (returns a list of UUIDs).
# `uncloud hack --hackprefix /tmp/hackcloud --list-vms
#
# Get VM status:
# `uncloud hack --hackprefix /tmp/hackcloud --get-vm-status --uuid my-vm-uuid`
#
# Stop a VM:
# `uncloud hack --hackprefix /tmp/hackcloud --destroy-vm --uuid my-vm-uuid`
# ``
import subprocess
import uuid
import os
import logging
from uncloud.hack.db import DB
from uncloud.hack.mac import MAC
from uncloud.vmm import VMM
from uncloud.hack.product import Product
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
class VM(object):
def __init__(self, config, db_entry=None):
self.config = config
#TODO: Enable etcd lookup
self.no_db = self.config.arguments['no_db']
if not self.no_db:
self.db = DB(self.config, prefix="/vm")
if db_entry:
self.db_entry = db_entry
# General CLI arguments.
self.hackprefix = self.config.arguments['hackprefix']
self.uuid = self.config.arguments['uuid']
self.memory = self.config.arguments['memory'] or '1024M'
self.cores = self.config.arguments['cores'] or 1
if self.config.arguments['image']:
self.image = os.path.join(self.hackprefix, self.config.arguments['image'])
else:
self.image = None
if self.config.arguments['image_format']:
self.image_format=self.config.arguments['image_format']
else:
self.image_format='qcow2'
# External components.
# This one is broken:
# TypeError: expected str, bytes or os.PathLike object, not NoneType
# Fix before re-enabling
# self.vmm = VMM(vmm_backend=self.hackprefix)
self.mac = MAC(self.config)
# Harcoded & generated values.
self.owner = 'uncloud'
self.accel = 'kvm'
self.threads = 1
self.ifup = os.path.join(self.hackprefix, "ifup.sh")
self.ifdown = os.path.join(self.hackprefix, "ifdown.sh")
self.ifname = "uc{}".format(self.mac.to_str_format())
self.vm = {}
self.product = Product(config, product_name="dualstack-vm",
product_class=self.__class__)
self.product.define_feature(name="base",
one_time_price=0,
recurring_price=9,
recurring_period="per_month",
minimum_period="per_hour")
self.features = []
def get_qemu_args(self):
command = (
"-name {owner}-{name}"
" -machine pc,accel={accel}"
" -drive file={image},format={image_format},if=virtio"
" -device virtio-rng-pci"
" -m {memory} -smp cores={cores},threads={threads}"
" -netdev tap,id=netmain,script={ifup},downscript={ifdown},ifname={ifname}"
" -device virtio-net-pci,netdev=netmain,id=net0,mac={mac}"
).format(
owner=self.owner, name=self.uuid,
accel=self.accel,
image=self.image, image_format=self.image_format,
memory=self.memory, cores=self.cores, threads=self.threads,
ifup=self.ifup, ifdown=self.ifdown, ifname=self.ifname,
mac=self.mac
)
return command.split(" ")
def create_product(self):
"""Find a VM host and schedule on it"""
pass
def create(self):
# New VM: new UUID, new MAC.
self.uuid = str(uuid.uuid4())
self.mac=MAC(self.config)
self.mac.create()
qemu_args = self.get_qemu_args()
log.debug("QEMU args passed to VMM: {}".format(qemu_args))
self.vmm.start(
uuid=self.uuid,
migration=False,
*qemu_args
)
self.mac.create()
self.vm['mac'] = self.mac
self.vm['ifname'] = "uc{}".format(self.mac.__repr__())
# FIXME: TODO: turn this into a string and THEN
# .split() it later -- easier for using .format()
#self.vm['commandline'] = [ "{}".format(self.sudo),
self.vm['commandline'] = "{sudo}{qemu} -name uncloud-{uuid} -machine pc,accel={accel} -m {memory} -smp {cores} -uuid {uuid} -drive file={os_image},media=cdrom -netdev tap,id=netmain,script={ifup},downscript={ifdown},ifname={ifname} -device virtio-net-pci,netdev=netmain,id=net0,mac={mac}"
# self.vm['commandline'] = [ "{}".format(self.sudo),
# "{}".format(self.qemu),
# "-name", "uncloud-{}".format(self.vm['uuid']),
# "-machine", "pc,accel={}".format(self.accel),
# "-m", "{}".format(self.vm['memory']),
# "-smp", "{}".format(self.vm['cores']),
# "-uuid", "{}".format(self.vm['uuid']),
# "-drive", "file={},media=cdrom".format(self.vm['os_image']),
# "-netdev", "tap,id=netmain,script={},downscript={},ifname={}".format(self.ifup, self.ifdown, self.vm['ifname']),
# "-device", "virtio-net-pci,netdev=netmain,id=net0,mac={}".format(self.vm['mac'])
# ]
def _execute_cmd(self, cmd_string, **kwargs):
cmd = cmd_string.format(**self.vm, **kwargs)
log.info("Executing: {}".format(cmd))
subprocess.run(cmd.split())
def stop(self):
if not self.uuid:
print("Please specific an UUID with the --uuid flag.")
exit(1)
self.vmm.stop(self.uuid)
def status(self):
if not self.uuid:
print("Please specific an UUID with the --uuid flag.")
exit(1)
print(self.vmm.get_status(self.uuid))
def vnc_addr(self):
if not self.uuid:
print("Please specific an UUID with the --uuid flag.")
exit(1)
print(self.vmm.get_vnc(self.uuid))
def list(self):
print(self.vmm.discover())

123
archive/uncloud_etcd_based/uncloud/host/main.py
View file

@ -1,123 +0,0 @@
import argparse
import multiprocessing as mp
import time
from uuid import uuid4
from uncloud.common.request import RequestEntry, RequestType
from uncloud.common.shared import shared
from uncloud.common.vm import VMStatus
from uncloud.vmm import VMM
from os.path import join as join_path
from . import virtualmachine, logger
arg_parser = argparse.ArgumentParser('host', add_help=False)
arg_parser.add_argument('--hostname', required=True)
def update_heartbeat(hostname):
"""Update Last HeartBeat Time for :param hostname: in etcd"""
host_pool = shared.host_pool
this_host = next(
filter(lambda h: h.hostname == hostname, host_pool.hosts), None
)
while True:
this_host.update_heartbeat()
host_pool.put(this_host)
time.sleep(10)
def maintenance(host):
vmm = VMM()
running_vms = vmm.discover()
for vm_uuid in running_vms:
if vmm.is_running(vm_uuid) and vmm.get_status(vm_uuid) == 'running':
logger.debug('VM {} is running on {}'.format(vm_uuid, host))
vm = shared.vm_pool.get(
join_path(shared.settings['etcd']['vm_prefix'], vm_uuid)
)
vm.status = VMStatus.running
vm.vnc_socket = vmm.get_vnc(vm_uuid)
vm.hostname = host
shared.vm_pool.put(vm)
def main(arguments):
hostname = arguments['hostname']
host_pool = shared.host_pool
host = next(filter(lambda h: h.hostname == hostname, host_pool.hosts), None)
# Does not yet exist, create it
if not host:
host_key = join_path(
shared.settings['etcd']['host_prefix'], uuid4().hex
)
host_entry = {
'specs': '',
'hostname': hostname,
'status': 'DEAD',
'last_heartbeat': '',
}
shared.etcd_client.put(
host_key, host_entry, value_in_json=True
)
# update, get ourselves now for sure
host = next(filter(lambda h: h.hostname == hostname, host_pool.hosts), None)
try:
heartbeat_updating_process = mp.Process(target=update_heartbeat, args=(hostname,))
heartbeat_updating_process.start()
except Exception as e:
raise Exception('uncloud-host heartbeat updating mechanism is not working') from e
# The below while True is neccessary for gracefully handling leadership transfer and temporary
# unavailability in etcd. Why does it work? It works because the get_prefix,watch_prefix return
# iter([]) that is iterator of empty list on exception (that occur due to above mentioned reasons)
# which ends the loop immediately. So, having it inside infinite loop we try again and again to
# get prefix until either success or deamon death comes.
while True:
for events_iterator in [
shared.etcd_client.get_prefix(shared.settings['etcd']['request_prefix'], value_in_json=True,
raise_exception=False),
shared.etcd_client.watch_prefix(shared.settings['etcd']['request_prefix'], value_in_json=True,
raise_exception=False)
]:
for request_event in events_iterator:
request_event = RequestEntry(request_event)
maintenance(host.key)
if request_event.hostname == host.key:
logger.debug('VM Request: %s on Host %s', request_event, host.hostname)
shared.request_pool.client.client.delete(request_event.key)
vm_entry = shared.etcd_client.get(
join_path(shared.settings['etcd']['vm_prefix'], request_event.uuid)
)
logger.debug('VM hostname: {}'.format(vm_entry.value))
vm = virtualmachine.VM(vm_entry)
if request_event.type == RequestType.StartVM:
vm.start()
elif request_event.type == RequestType.StopVM:
vm.stop()
elif request_event.type == RequestType.DeleteVM:
vm.delete()
elif request_event.type == RequestType.InitVMMigration:
vm.start(destination_host_key=host.key)
elif request_event.type == RequestType.TransferVM:
destination_host = host_pool.get(request_event.destination_host_key)
if destination_host:
vm.migrate(
destination_host=destination_host.hostname,
destination_sock_path=request_event.destination_sock_path,
)
else:
logger.error('Host %s not found!', request_event.destination_host_key)

303
archive/uncloud_etcd_based/uncloud/host/virtualmachine.py
View file

@ -1,303 +0,0 @@
# QEMU Manual
# https://qemu.weilnetz.de/doc/qemu-doc.html
# For QEMU Monitor Protocol Commands Information, See
# https://qemu.weilnetz.de/doc/qemu-doc.html#pcsys_005fmonitor
import os
import subprocess as sp
import ipaddress
from string import Template
from os.path import join as join_path
from uncloud.common.request import RequestEntry, RequestType
from uncloud.common.vm import VMStatus, declare_stopped
from uncloud.common.network import create_dev, delete_network_interface
from uncloud.common.schemas import VMSchema, NetworkSchema
from uncloud.host import logger
from uncloud.common.shared import shared
from uncloud.vmm import VMM
from marshmallow import ValidationError
class VM:
def __init__(self, vm_entry):
self.schema = VMSchema()
self.vmm = VMM()
self.key = vm_entry.key
try:
self.vm = self.schema.loads(vm_entry.value)
except ValidationError:
logger.exception(
"Couldn't validate VM Entry", vm_entry.value
)
self.vm = None
else:
self.uuid = vm_entry.key.split("/")[-1]
self.host_key = self.vm["hostname"]
logger.debug('VM Hostname {}'.format(self.host_key))
def get_qemu_args(self):
command = (
"-drive file={file},format=raw,if=virtio"
" -device virtio-rng-pci"
" -m {memory} -smp cores={cores},threads={threads}"
" -name {owner}_{name}"
).format(
owner=self.vm["owner"],
name=self.vm["name"],
memory=int(self.vm["specs"]["ram"].to_MB()),
cores=self.vm["specs"]["cpu"],
threads=1,
file=shared.storage_handler.qemu_path_string(self.uuid),
)
return command.split(" ")
def start(self, destination_host_key=None):
migration = False
if destination_host_key:
migration = True
self.create()
try:
network_args = self.create_network_dev()
except Exception as err:
declare_stopped(self.vm)
self.vm["log"].append("Cannot Setup Network Properly")
logger.error("Cannot Setup Network Properly for vm %s", self.uuid, exc_info=err)
else:
self.vmm.start(
uuid=self.uuid,
migration=migration,
*self.get_qemu_args(),
*network_args
)
status = self.vmm.get_status(self.uuid)
logger.debug('VM {} status is {}'.format(self.uuid, status))
if status == "running":
self.vm["status"] = VMStatus.running
self.vm["vnc_socket"] = self.vmm.get_vnc(self.uuid)
elif status == "inmigrate":
r = RequestEntry.from_scratch(
type=RequestType.TransferVM, # Transfer VM
hostname=self.host_key, # Which VM should get this request. It is source host
uuid=self.uuid, # uuid of VM
destination_sock_path=join_path(
self.vmm.socket_dir, self.uuid
),
destination_host_key=destination_host_key, # Where source host transfer VM
request_prefix=shared.settings["etcd"]["request_prefix"],
)
shared.request_pool.put(r)
else:
self.stop()
declare_stopped(self.vm)
logger.debug('VM {} has hostname {}'.format(self.uuid, self.vm['hostname']))
self.sync()
def stop(self):
self.vmm.stop(self.uuid)
self.delete_network_dev()
declare_stopped(self.vm)
self.sync()
def migrate(self, destination_host, destination_sock_path):
self.vmm.transfer(
src_uuid=self.uuid,
destination_sock_path=destination_sock_path,
host=destination_host,
)
def create_network_dev(self):
command = ""
for network_mac_and_tap in self.vm["network"]:
network_name, mac, tap = network_mac_and_tap
_key = os.path.join(
shared.settings["etcd"]["network_prefix"],
self.vm["owner"],
network_name,
)
network = shared.etcd_client.get(_key, value_in_json=True)
network_schema = NetworkSchema()
try:
network = network_schema.load(network.value)
except ValidationError:
continue
if network["type"] == "vxlan":
tap = create_vxlan_br_tap(
_id=network["id"],
_dev=shared.settings["network"]["vxlan_phy_dev"],
tap_id=tap,
ip=network["ipv6"],
)
all_networks = shared.etcd_client.get_prefix(
shared.settings["etcd"]["network_prefix"],
value_in_json=True,
)
if ipaddress.ip_network(network["ipv6"]).is_global:
update_radvd_conf(all_networks)
command += (
"-netdev tap,id=vmnet{net_id},ifname={tap},script=no,downscript=no"
" -device virtio-net-pci,netdev=vmnet{net_id},mac={mac}".format(
tap=tap, net_id=network["id"], mac=mac
)
)
if command:
command = command.split(' ')
return command
def delete_network_dev(self):
try:
for network in self.vm["network"]:
network_name = network[0]
_ = network[1] # tap_mac
tap_id = network[2]
delete_network_interface("tap{}".format(tap_id))
owners_vms = shared.vm_pool.by_owner(self.vm["owner"])
owners_running_vms = shared.vm_pool.by_status(
VMStatus.running, _vms=owners_vms
)
networks = map(
lambda n: n[0],
map(lambda vm: vm.network, owners_running_vms),
)
networks_in_use_by_user_vms = [vm[0] for vm in networks]
if network_name not in networks_in_use_by_user_vms:
network_entry = resolve_network(
network[0], self.vm["owner"]
)
if network_entry:
network_type = network_entry.value["type"]
network_id = network_entry.value["id"]
if network_type == "vxlan":
delete_network_interface(
"br{}".format(network_id)
)
delete_network_interface(
"vxlan{}".format(network_id)
)
except Exception:
logger.exception("Exception in network interface deletion")
def create(self):
if shared.storage_handler.is_vm_image_exists(self.uuid):
# File Already exists. No Problem Continue
logger.debug("Image for vm %s exists", self.uuid)
else:
if shared.storage_handler.make_vm_image(
src=self.vm["image_uuid"], dest=self.uuid
):
if not shared.storage_handler.resize_vm_image(
path=self.uuid,
size=int(self.vm["specs"]["os-ssd"].to_MB()),
):
self.vm["status"] = VMStatus.error
else:
logger.info("New VM Created")
def sync(self):
shared.etcd_client.put(
self.key, self.schema.dump(self.vm), value_in_json=True
)
def delete(self):
self.stop()
if shared.storage_handler.is_vm_image_exists(self.uuid):
r_status = shared.storage_handler.delete_vm_image(self.uuid)
if r_status:
shared.etcd_client.client.delete(self.key)
else:
shared.etcd_client.client.delete(self.key)
def resolve_network(network_name, network_owner):
network = shared.etcd_client.get(
join_path(
shared.settings["etcd"]["network_prefix"],
network_owner,
network_name,
),
value_in_json=True,
)
return network
def create_vxlan_br_tap(_id, _dev, tap_id, ip=None):
network_script_base = os.path.join(
os.path.dirname(os.path.dirname(__file__)), "network"
)
vxlan = create_dev(
script=os.path.join(network_script_base, "create-vxlan.sh"),
_id=_id,
dev=_dev,
)
if vxlan:
bridge = create_dev(
script=os.path.join(
network_script_base, "create-bridge.sh"
),
_id=_id,
dev=vxlan,
ip=ip,
)
if bridge:
tap = create_dev(
script=os.path.join(
network_script_base, "create-tap.sh"
),
_id=str(tap_id),
dev=bridge,
)
if tap:
return tap
def update_radvd_conf(all_networks):
network_script_base = os.path.join(
os.path.dirname(os.path.dirname(__file__)), "network"
)
networks = {
net.value["ipv6"]: net.value["id"]
for net in all_networks
if net.value.get("ipv6")
and ipaddress.ip_network(net.value.get("ipv6")).is_global
}
radvd_template = open(
os.path.join(network_script_base, "radvd-template.conf"), "r"
).read()
radvd_template = Template(radvd_template)
content = [
radvd_template.safe_substitute(
bridge="br{}".format(networks[net]), prefix=net
)
for net in networks
if networks.get(net)
]
with open("/etc/radvd.conf", "w") as radvd_conf:
radvd_conf.writelines(content)
try:
sp.check_output(["systemctl", "restart", "radvd"])
except sp.CalledProcessError:
try:
sp.check_output(["service", "radvd", "restart"])
except sp.CalledProcessError as err:
raise err.__class__(
"Cannot start/restart radvd service", err.cmd
) from err

121
archive/uncloud_etcd_based/uncloud/imagescanner/main.py
View file

@ -1,121 +0,0 @@
import json
import os
import argparse
import subprocess as sp
from os.path import join as join_path
from uncloud.common.shared import shared
from uncloud.imagescanner import logger
arg_parser = argparse.ArgumentParser('imagescanner', add_help=False)
def qemu_img_type(path):
qemu_img_info_command = [
"qemu-img",
"info",
"--output",
"json",
path,
]
try:
qemu_img_info = sp.check_output(qemu_img_info_command)
except Exception as e:
logger.exception(e)
return None
else:
qemu_img_info = json.loads(qemu_img_info.decode("utf-8"))
return qemu_img_info["format"]
def main(arguments):
# We want to get images entries that requests images to be created
images = shared.etcd_client.get_prefix(
shared.settings["etcd"]["image_prefix"], value_in_json=True
)
images_to_be_created = list(
filter(lambda im: im.value["status"] == "TO_BE_CREATED", images)
)
for image in images_to_be_created:
try:
image_uuid = image.key.split("/")[-1]
image_owner = image.value["owner"]
image_filename = image.value["filename"]
image_store_name = image.value["store_name"]
image_full_path = join_path(
shared.settings["storage"]["file_dir"],
image_owner,
image_filename,
)
image_stores = shared.etcd_client.get_prefix(
shared.settings["etcd"]["image_store_prefix"],
value_in_json=True,
)
user_image_store = next(
filter(
lambda s, store_name=image_store_name: s.value[
"name"
]
== store_name,
image_stores,
)
)
image_store_pool = user_image_store.value["attributes"][
"pool"
]
except Exception as e:
logger.exception(e)
else:
# At least our basic data is available
qemu_img_convert_command = [
"qemu-img",
"convert",
"-f",
"qcow2",
"-O",
"raw",
image_full_path,
"image.raw",
]
if qemu_img_type(image_full_path) == "qcow2":
try:
# Convert .qcow2 to .raw
sp.check_output(qemu_img_convert_command,)
except sp.CalledProcessError:
logger.exception(
"Image convertion from .qcow2 to .raw failed."
)
else:
# Import and Protect
r_status = shared.storage_handler.import_image(
src="image.raw", dest=image_uuid, protect=True
)
if r_status:
# Everything is successfully done
image.value["status"] = "CREATED"
shared.etcd_client.put(
image.key, json.dumps(image.value)
)
finally:
try:
os.remove("image.raw")
except Exception:
pass
else:
# The user provided image is either not found or of invalid format
image.value["status"] = "INVALID_IMAGE"
shared.etcd_client.put(
image.key, json.dumps(image.value)
)
if __name__ == "__main__":
main()

3
archive/uncloud_etcd_based/uncloud/metadata/__init__.py
View file

@ -1,3 +0,0 @@
import logging
logger = logging.getLogger(__name__)

95
archive/uncloud_etcd_based/uncloud/metadata/main.py
View file

@ -1,95 +0,0 @@
import os
import argparse
from flask import Flask, request
from flask_restful import Resource, Api
from werkzeug.exceptions import HTTPException
from uncloud.common.shared import shared
app = Flask(__name__)
api = Api(app)
app.logger.handlers.clear()
DEFAULT_PORT=1234
arg_parser = argparse.ArgumentParser('metadata', add_help=False)
arg_parser.add_argument('--port', '-p', default=DEFAULT_PORT, help='By default bind to port {}'.format(DEFAULT_PORT))
@app.errorhandler(Exception)
def handle_exception(e):
app.logger.error(e)
# pass through HTTP errors
if isinstance(e, HTTPException):
return e
# now you're handling non-HTTP exceptions only
return {"message": "Server Error"}, 500
def get_vm_entry(mac_addr):
return next(
filter(
lambda vm: mac_addr in list(zip(*vm.network))[1],
shared.vm_pool.vms,
),
None,
)
# https://stackoverflow.com/questions/37140846/how-to-convert-ipv6-link-local-address-to-mac-address-in-python
def ipv62mac(ipv6):
# remove subnet info if given
subnet_index = ipv6.find("/")
if subnet_index != -1:
ipv6 = ipv6[:subnet_index]
ipv6_parts = ipv6.split(":")
mac_parts = list()
for ipv6_part in ipv6_parts[-4:]:
while len(ipv6_part) < 4:
ipv6_part = "0" + ipv6_part
mac_parts.append(ipv6_part[:2])
mac_parts.append(ipv6_part[-2:])
# modify parts to match MAC value
mac_parts[0] = "%02x" % (int(mac_parts[0], 16) ^ 2)
del mac_parts[4]
del mac_parts[3]
return ":".join(mac_parts)
class Root(Resource):
@staticmethod
def get():
data = get_vm_entry(ipv62mac(request.remote_addr))
if not data:
return (
{"message": "Metadata for such VM does not exists."},
404,
)
else:
etcd_key = os.path.join(
shared.settings["etcd"]["user_prefix"],
data.value["owner_realm"],
data.value["owner"],
"key",
)
etcd_entry = shared.etcd_client.get_prefix(
etcd_key, value_in_json=True
)
user_personal_ssh_keys = [key.value for key in etcd_entry]
data.value["metadata"]["ssh-keys"] += user_personal_ssh_keys
return data.value["metadata"], 200
api.add_resource(Root, "/")
def main(arguments):
port = arguments['port']
debug = arguments['debug']
app.run(debug=debug, host="::", port=port)

0
archive/uncloud_etcd_based/uncloud/network/__init__.py
View file

3
archive/uncloud_etcd_based/uncloud/oneshot/__init__.py
View file

@ -1,3 +0,0 @@
import logging
logger = logging.getLogger(__name__)

123
archive/uncloud_etcd_based/uncloud/oneshot/main.py
View file

@ -1,123 +0,0 @@
import argparse
import os
from pathlib import Path
from uncloud.vmm import VMM
from uncloud.host.virtualmachine import update_radvd_conf, create_vxlan_br_tap
from . import virtualmachine, logger
###
# Argument parser loaded by scripts/uncloud.
arg_parser = argparse.ArgumentParser('oneshot', add_help=False)
# Actions.
arg_parser.add_argument('--list', action='store_true',
help='list UUID and name of running VMs')
arg_parser.add_argument('--start', nargs=4,
metavar=('NAME', 'IMAGE', 'UPSTREAM_INTERFACE', 'NETWORK'),
help='start a VM using the OS IMAGE (full path), configuring networking on NETWORK IPv6 prefix')
arg_parser.add_argument('--stop', metavar='UUID',
help='stop a VM')
arg_parser.add_argument('--get-status', metavar='UUID',
help='return the status of the VM')
arg_parser.add_argument('--get-vnc', metavar='UUID',
help='return the path of the VNC socket of the VM')
arg_parser.add_argument('--reconfigure-radvd', metavar='NETWORK',
help='regenerate and reload RADVD configuration for NETWORK IPv6 prefix')
# Arguments.
arg_parser.add_argument('--workdir', default=Path.home(),
help='Working directory, defaulting to $HOME')
arg_parser.add_argument('--mac',
help='MAC address of the VM to create (--start)')
arg_parser.add_argument('--memory', type=int,
help='Memory (MB) to allocate (--start)')
arg_parser.add_argument('--cores', type=int,
help='Number of cores to allocate (--start)')
arg_parser.add_argument('--threads', type=int,
help='Number of threads to allocate (--start)')
arg_parser.add_argument('--image-format', choices=['raw', 'qcow2'],
help='Format of OS image (--start)')
arg_parser.add_argument('--accel', choices=['kvm', 'tcg'], default='kvm',
help='QEMU acceleration to use (--start)')
arg_parser.add_argument('--upstream-interface', default='eth0',
help='Name of upstream interface (--start)')
###
# Helpers.
# XXX: check if it is possible to use the type returned by ETCD queries.
class UncloudEntryWrapper:
def __init__(self, value):
self.value = value
def value(self):
return self.value
def status_line(vm):
return "VM: {} {} {}".format(vm.get_uuid(), vm.get_name(), vm.get_status())
###
# Entrypoint.
def main(arguments):
# Initialize VMM.
workdir = arguments['workdir']
vmm = VMM(vmm_backend=workdir)
# Harcoded debug values.
net_id = 0
# Build VM configuration.
vm_config = {}
vm_options = [
'mac', 'memory', 'cores', 'threads', 'image', 'image_format',
'--upstream_interface', 'upstream_interface', 'network', 'accel'
]
for option in vm_options:
if arguments.get(option):
vm_config[option] = arguments[option]
vm_config['net_id'] = net_id
# Execute requested VM action.
if arguments['reconfigure_radvd']:
# TODO: check that RADVD is available.
prefix = arguments['reconfigure_radvd']
network = UncloudEntryWrapper({
'id': net_id,
'ipv6': prefix
})
# Make use of uncloud.host.virtualmachine for network configuration.
update_radvd_conf([network])
elif arguments['start']:
# Extract from --start positional arguments. Quite fragile.
vm_config['name'] = arguments['start'][0]
vm_config['image'] = arguments['start'][1]
vm_config['network'] = arguments['start'][2]
vm_config['upstream_interface'] = arguments['start'][3]
vm_config['tap_interface'] = "uc{}".format(len(vmm.discover()))
vm = virtualmachine.VM(vmm, vm_config)
vm.start()
elif arguments['stop']:
vm = virtualmachine.VM(vmm, {'uuid': arguments['stop']})
vm.stop()
elif arguments['get_status']:
vm = virtualmachine.VM(vmm, {'uuid': arguments['get_status']})
print(status_line(vm))
elif arguments['get_vnc']:
vm = virtualmachine.VM(vmm, {'uuid': arguments['get_vnc']})
print(vm.get_vnc_addr())
elif arguments['list']:
vms = vmm.discover()
print("Found {} VMs.".format(len(vms)))
for uuid in vms:
vm = virtualmachine.VM(vmm, {'uuid': uuid})
print(status_line(vm))
else:
print('Please specify an action: --start, --stop, --list,\
--get-status, --get-vnc, --reconfigure-radvd')

81
archive/uncloud_etcd_based/uncloud/oneshot/virtualmachine.py
View file

@ -1,81 +0,0 @@
import uuid
import os
from uncloud.host.virtualmachine import create_vxlan_br_tap
from uncloud.oneshot import logger
class VM(object):
def __init__(self, vmm, config):
self.config = config
self.vmm = vmm
# Extract VM specs/metadata from configuration.
self.name = config.get('name', 'no-name')
self.memory = config.get('memory', 1024)
self.cores = config.get('cores', 1)
self.threads = config.get('threads', 1)
self.image_format = config.get('image_format', 'qcow2')
self.image = config.get('image')
self.uuid = config.get('uuid', str(uuid.uuid4()))
self.mac = config.get('mac')
self.accel = config.get('accel', 'kvm')
self.net_id = config.get('net_id', 0)
self.upstream_interface = config.get('upstream_interface', 'eth0')
self.tap_interface = config.get('tap_interface', 'uc0')
self.network = config.get('network')
def get_qemu_args(self):
command = (
"-uuid {uuid} -name {name} -machine pc,accel={accel}"
" -drive file={image},format={image_format},if=virtio"
" -device virtio-rng-pci"
" -m {memory} -smp cores={cores},threads={threads}"
" -netdev tap,id=vmnet{net_id},ifname={tap},script=no,downscript=no"
" -device virtio-net-pci,netdev=vmnet{net_id},mac={mac}"
).format(
uuid=self.uuid, name=self.name, accel=self.accel,
image=self.image, image_format=self.image_format,
memory=self.memory, cores=self.cores, threads=self.threads,
net_id=self.net_id, tap=self.tap_interface, mac=self.mac
)
return command.split(" ")
def start(self):
# Check that VM image is available.
if not os.path.isfile(self.image):
logger.error("Image {} does not exist. Aborting.".format(self.image))
# Create Bridge, VXLAN and tap interface for VM.
create_vxlan_br_tap(
self.net_id, self.upstream_interface, self.tap_interface, self.network
)
# Generate config for and run QEMU.
qemu_args = self.get_qemu_args()
logger.debug("QEMU args for VM {}: {}".format(self.uuid, qemu_args))
self.vmm.start(
uuid=self.uuid,
migration=False,
*qemu_args
)
def stop(self):
self.vmm.stop(self.uuid)
def get_status(self):
return self.vmm.get_status(self.uuid)
def get_vnc_addr(self):
return self.vmm.get_vnc(self.uuid)
def get_uuid(self):
return self.uuid
def get_name(self):
success, json = self.vmm.execute_command(self.uuid, 'query-name')
if success:
return json['return']['name']
return None

3
archive/uncloud_etcd_based/uncloud/scheduler/__init__.py
View file

@ -1,3 +0,0 @@
import logging
logger = logging.getLogger(__name__)

51
archive/uncloud_etcd_based/uncloud/scheduler/main.py
View file

@ -1,51 +0,0 @@
# TODO
# 1. send an email to an email address defined by env['admin-email']
# if resources are finished
# 2. Introduce a status endpoint of the scheduler -
# maybe expose a prometheus compatible output
import argparse
from uncloud.common.request import RequestEntry, RequestType
from uncloud.common.shared import shared
from uncloud.scheduler import logger
from uncloud.scheduler.helper import (dead_host_mitigation, dead_host_detection,
assign_host, NoSuitableHostFound)
arg_parser = argparse.ArgumentParser('scheduler', add_help=False)
def main(arguments):
# The below while True is neccessary for gracefully handling leadership transfer and temporary
# unavailability in etcd. Why does it work? It works because the get_prefix,watch_prefix return
# iter([]) that is iterator of empty list on exception (that occur due to above mentioned reasons)
# which ends the loop immediately. So, having it inside infinite loop we try again and again to
# get prefix until either success or deamon death comes.
while True:
for request_iterator in [
shared.etcd_client.get_prefix(shared.settings['etcd']['request_prefix'], value_in_json=True,
raise_exception=False),
shared.etcd_client.watch_prefix(shared.settings['etcd']['request_prefix'], value_in_json=True,
raise_exception=False),
]:
for request_event in request_iterator:
dead_host_mitigation(dead_host_detection())
request_entry = RequestEntry(request_event)
if request_entry.type == RequestType.ScheduleVM:
logger.debug('%s, %s', request_entry.key, request_entry.value)
vm_entry = shared.vm_pool.get(request_entry.uuid)
if vm_entry is None:
logger.info('Trying to act on {} but it is deleted'.format(request_entry.uuid))
continue
shared.etcd_client.client.delete(request_entry.key) # consume Request
try:
assign_host(vm_entry)
except NoSuitableHostFound:
vm_entry.add_log('Can\'t schedule VM. No Resource Left.')
shared.vm_pool.put(vm_entry)
logger.info('No Resource Left. Emailing admin....')

0
archive/uncloud_etcd_based/uncloud/scheduler/tests/__init__.py
View file

1
archive/uncloud_etcd_based/uncloud/version.py
View file

@ -1 +0,0 @@
VERSION = "0.0.5-30-ge91fd9e"

284
archive/uncloud_etcd_based/uncloud/vmm/__init__.py
View file

@ -1,284 +0,0 @@
import os
import subprocess as sp
import logging
import socket
import json
import tempfile
import time
from contextlib import suppress
from multiprocessing import Process
from os.path import join as join_path
from os.path import isdir
logger = logging.getLogger(__name__)
class VMQMPHandles:
def __init__(self, path):
self.path = path
self.sock = socket.socket(socket.AF_UNIX)
self.file = self.sock.makefile()
def __enter__(self):
self.sock.connect(self.path)
# eat qmp greetings
self.file.readline()
# init qmp
self.sock.sendall(b'{ "execute": "qmp_capabilities" }')
self.file.readline()
return self.sock, self.file
def __exit__(self, exc_type, exc_val, exc_tb):
self.file.close()
self.sock.close()
if exc_type:
logger.error(
"Couldn't get handle for VM.", exc_type, exc_val, exc_tb
)
raise exc_type("Couldn't get handle for VM.") from exc_type
class TransferVM(Process):
def __init__(self, src_uuid, dest_sock_path, host, socket_dir):
self.src_uuid = src_uuid
self.host = host
self.src_sock_path = os.path.join(socket_dir, self.src_uuid)
self.dest_sock_path = dest_sock_path
super().__init__()
def run(self):
with suppress(FileNotFoundError):
os.remove(self.src_sock_path)
command = [
"ssh",
"-nNT",
"-L",
"{}:{}".format(self.src_sock_path, self.dest_sock_path),
"root@{}".format(self.host),
]
try:
p = sp.Popen(command)
except Exception as e:
logger.error(
"Couldn' forward unix socks over ssh.", exc_info=e
)
else:
time.sleep(2)
vmm = VMM()
logger.debug("Executing: ssh forwarding command: %s", command)
vmm.execute_command(
self.src_uuid,
command="migrate",
arguments={"uri": "unix:{}".format(self.src_sock_path)},
)
while p.poll() is None:
success, output = vmm.execute_command(self.src_uuid, command="query-migrate")
if success:
status = output["return"]["status"]
logger.info('Migration Status: {}'.format(status))
if status == "completed":
vmm.stop(self.src_uuid)
return
elif status in ['failed', 'cancelled']:
return
else:
logger.error("Couldn't be able to query VM {} that was in migration".format(self.src_uuid))
return
time.sleep(2)
class VMM:
# Virtual Machine Manager
def __init__(
self,
qemu_path="/usr/bin/qemu-system-x86_64",
vmm_backend=os.path.expanduser("~/uncloud/vmm/"),
):
self.qemu_path = qemu_path
self.vmm_backend = vmm_backend
self.socket_dir = os.path.join(self.vmm_backend, "sock")
if not os.path.isdir(self.vmm_backend):
logger.info(
"{} does not exists. Creating it...".format(
self.vmm_backend
)
)
os.makedirs(self.vmm_backend, exist_ok=True)
if not os.path.isdir(self.socket_dir):
logger.info(
"{} does not exists. Creating it...".format(
self.socket_dir
)
)
os.makedirs(self.socket_dir, exist_ok=True)
def is_running(self, uuid):
sock_path = os.path.join(self.socket_dir, uuid)
try:
sock = socket.socket(socket.AF_UNIX)
sock.connect(sock_path)
recv = sock.recv(4096)
except Exception as err:
# unix sock doesn't exists or it is closed
logger.debug(
"VM {} sock either don' exists or it is closed. It mean VM is stopped.".format(
uuid
),
exc_info=err,
)
else:
# if we receive greetings from qmp it mean VM is running
if len(recv) > 0:
return True
with suppress(FileNotFoundError):
os.remove(sock_path)
return False
def start(self, *args, uuid, migration=False):
# start --> sucess?
migration_args = ()
if migration:
migration_args = (
"-incoming",
"unix:{}".format(os.path.join(self.socket_dir, uuid)),
)
if self.is_running(uuid):
logger.warning("Cannot start VM. It is already running.")
else:
qmp_arg = (
"-qmp",
"unix:{},server,nowait".format(
join_path(self.socket_dir, uuid)
),
)
vnc_arg = (
"-vnc",
"unix:{}".format(tempfile.NamedTemporaryFile().name),
)
command = [
"sudo",
"-p",
"Enter password to start VM {}: ".format(uuid),
self.qemu_path,
*args,
*qmp_arg,
*migration_args,
*vnc_arg,
"-daemonize",
]
try:
sp.check_output(command, stderr=sp.PIPE)
except sp.CalledProcessError as err:
logger.exception(
"Error occurred while starting VM.\nDetail %s",
err.stderr.decode("utf-8"),
)
else:
sp.check_output(
["sudo", "-p", "Enter password to correct permission for uncloud-vmm's directory",
"chmod", "-R", "o=rwx,g=rwx", self.vmm_backend]
)
# TODO: Find some good way to check whether the virtual machine is up and
# running without relying on non-guarenteed ways.
for _ in range(10):
time.sleep(2)
status = self.get_status(uuid)
if status in ["running", "inmigrate"]:
return status
logger.warning(
"Timeout on VM's status. Shutting down VM %s", uuid
)
self.stop(uuid)
# TODO: What should we do more. VM can still continue to run in background.
# If we have pid of vm we can kill it using OS.
def execute_command(self, uuid, command, **kwargs):
# execute_command -> sucess?, output
try:
with VMQMPHandles(os.path.join(self.socket_dir, uuid)) as (
sock_handle,
file_handle,
):
command_to_execute = {"execute": command, **kwargs}
sock_handle.sendall(
json.dumps(command_to_execute).encode("utf-8")
)
output = file_handle.readline()
except Exception:
logger.exception(
"Error occurred while executing command and getting valid output from qmp"
)
else:
try:
output = json.loads(output)
except Exception:
logger.exception(
"QMP Output isn't valid JSON. %s", output
)
else:
return "return" in output, output
return False, None
def stop(self, uuid):
success, output = self.execute_command(
command="quit", uuid=uuid
)
return success
def get_status(self, uuid):
success, output = self.execute_command(
command="query-status", uuid=uuid
)
if success:
return output["return"]["status"]
else:
# TODO: Think about this for a little more
return "STOPPED"
def discover(self):
vms = [
uuid
for uuid in os.listdir(self.socket_dir)
if not isdir(join_path(self.socket_dir, uuid))
]
return vms
def get_vnc(self, uuid):
success, output = self.execute_command(
uuid, command="query-vnc"
)
if success:
return output["return"]["service"]
return None
def transfer(self, src_uuid, destination_sock_path, host):
p = TransferVM(
src_uuid,
destination_sock_path,
socket_dir=self.socket_dir,
host=host,
)
p.start()
# TODO: the following method should clean things that went wrong
# e.g If VM migration fails or didn't start for long time
# i.e 15 minutes we should stop the waiting VM.
def maintenace(self):
pass

39
bin/deploy.sh
View file

@ -1,39 +0,0 @@
#!/bin/sh
# Nico Schottelius, 2021-01-17
set -e
if [ $# -ne 1 ]; then
echo "$0 target-host"
exit 1
fi
target_host=$1; shift
user=app
dir=${0%/*}
uncloud_base=$(cd ${dir}/.. && pwd -P)
conf_name=local_settings-${target_host}.py
conf_file=${uncloud_base}/uncloud/${conf_name}
if [ ! -e ${conf_file} ]; then
echo "No settings for ${target_host}."
echo "Create ${conf_file} before using this script."
exit 1
fi
# Deploy
rsync -av \
--exclude venv/ \
--exclude '*.pyc' \
--exclude uncloud/local_settings.py \
--delete \
${uncloud_base}/ ${user}@${target_host}:app/
ssh "${user}@${target_host}" ". ~/pyvenv/bin/activate; cd ~/app; pip install -r requirements.txt"
# Config
ssh "${user}@${target_host}" "cd ~/app/uncloud; ln -sf ${conf_name} local_settings.py"
# Restart / Apply
ssh "${user}@${target_host}" "sudo /etc/init.d/uwsgi restart"

7
bin/make-migrations-from-scratch.sh
View file

@ -1,7 +0,0 @@
#!/bin/sh
# For undoing/redoing everything
# Needed in special cases and needs to be avoided as soon as
# uncloud.version >= 1
for a in */migrations; do rm ${a}/*.py; done
for a in */migrations; do python manage.py makemigrations ${a%%/migrations}; done

4
archive/uncloud_etcd_based/bin/uncloud → bin/ucloud
View file

@ -25,9 +25,9 @@ dir=${0%/*}
# Ensure version is present - the bundled/shipped version contains a static version,
# the git version contains a dynamic version
${dir}/gen-version
printf "VERSION = \"%s\"\n" "$(git describe)" > ${dir}/../ucloud/version.py
libdir=$(cd "${dir}/../" && pwd -P)
export PYTHONPATH="${libdir}"
"$dir/../scripts/uncloud" "$@"
"$dir/../scripts/ucloud" "$@"

2
doc/.gitignore vendored
View file

@ -1,2 +0,0 @@
*.pdf
*.tex

85
doc/README-billing.org
View file

@ -1,85 +0,0 @@
* How to handle billing in general
** Manual test flow / setting up bills
- Needs orders
-
** Orders
- Orders are the heart of uncloud billing
- Have a starting date
- Have an ending date
- Orders are immutable
- Can usually not be cancelled / cancellation is not a refund
- Customer/user commits on a certain period -> gets discount
based on it
- Can be upgraded
- Create a new order
- We link the new order to the old order and say this one
replaces it
- If the price of the new order is HIGHER than the OLD order,
then we charge the difference until the end of the order period
- In the next billing run we set the OLD order to not to bill anymore
- And only the NEW order will be billed afterwards
- Can be downgraded in the next period (but not for this period)
- We create a new order, same as for upgrade
- The new order starts directly after the OLD order
- As the amount is LOWER than the OLD order, no additional charge is done
during this order period
- We might need to have an activate datetime
- When to implement this
- Order periods can be
*** Statuses
- CREATING/PREPARING
- INACTIVE (?)
- TO_BILL
- NOT_TO_BILL: we use this to accelerate queries to the DB
*** Updating status of orders
- If has succeeding order and billing date is last month -> set inactive
** Bills
- Are always for a month
- Can be preliminary
*** Which orders to include
- Not the cancelled ones / not active ones
** Flows / Approach
*** Finding all orders for a bill
- Get all orders, state != NOT_TO_BILL; for each order do:
- is it a one time order?
- has it a bill assigned?
- yes: set to NOT_TO_BILL
- no:
- get_or_create_bill_for_this_month
- assign bill to this order
- set to NOT_TO_BILL
- is it a recurring order?
- if it has a REPLACING order:
-
- First of month
- Last of month
*** Handling replacement of orders
- The OLD order will appear in the month that it was cancelled on
the bill
- The OLD order needs to be set to NOT_TO_BILL after it was billed
the last time
- The NEW order will be added pro rata if the amount is higher in
the same month
- The NEW order will be used next month
**** Disabling the old order
- On billing run
- If order.replacement_order (naming!) is set
- if the order.replacement_order starts during THIS_MONTH
- add order to bill
- if NOT:
- the order was already replaced in a previous billing period
- set the order to NOT_TO_BILL
**** Billing the new order
- If order.previous_order
*** Handling multiple times a recurring order
- For each recurring order check the order.period
- Find out when it was billed last
- lookup latest bill
- Calculate how many times it has been used until 2359, last day
of month
- For preliminary bill: until datetime.now()
- Call the bill_end_datetime
- Getting duration: bill_end_datetime - order.last_billed
- Amount in seconds; duration_in_seconds
- Divide duration_in_seconds by order.period; amount_used:
- If >= 1: add amount_used * order.recurring_amount to bill

28
doc/README-how-to-configure-remote-uncloud-clients.org
View file

@ -1,28 +0,0 @@
* What is a remote uncloud client?
** Systems that configure themselves for the use with uncloud
** Examples are VMHosts, VPN Servers, cdist control server, etc.
* Which access do these clients need?
** They need read / write access to the database
* Possible methods
** Overview
| | pros | cons |
| SSL based | Once setup, can access all django parts natively, locally | X.509 infrastructure |
| SSH -L tunnel | All nodes can use [::1]:5432 | SSH setup can be fragile |
| ssh djangohost manage.py | All DB ops locally | Code is only executed on django host |
| https + token | Rest alike / consistent access | Code is only executed on django host |
| from_django | Everything is on the django host | main host can become bottleneck |
** remote vs. local Django code execution
- If manage.py is executed locally (= on the client), it can
check/modify local configs
- However local execution requires a pyvenv + packages + db access
- Local execution also *could* make use of postgresql notify for
triggering actions (which is quite neat)
- Remote execution (= on the primary django host) can acess the db
via unix socket
- However remote execution cannot check local state
** from_django
- might reuse existing methods like celery
- reduces the amount of things to be installed on the client to
almost zero
- follows the opennebula model
- has a single point of failurebin

Some files were not shown because too many files have changed in this diff Show more