forked from ungleich-public/cdist
		
	
		
			
	
	
		
			573 lines
		
	
	
	
		
			18 KiB
		
	
	
	
		
			ReStructuredText
		
	
	
	
	
	
		
		
			
		
	
	
			573 lines
		
	
	
	
		
			18 KiB
		
	
	
	
		
			ReStructuredText
		
	
	
	
	
	
| 
								 | 
							
								Dive into real world cdist
							 | 
						||
| 
								 | 
							
								==========================
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Introduction
							 | 
						||
| 
								 | 
							
								------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								This walkthrough shows real world cdist configuration example.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Sample target host is named **test.ungleich.ch**.
							 | 
						||
| 
								 | 
							
								Just replace **test.ungleich.ch** with your target hostname.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Our goal is to configure python application hosting. For writing sample
							 | 
						||
| 
								 | 
							
								application we will use `Bottle <http://bottlepy.org>`_ WSGI micro web-framework.
							 | 
						||
| 
								 | 
							
								It will use PostgreSQL database and it will list items from **items** table.
							 | 
						||
| 
								 | 
							
								It will be served by uWSGI server. We will also use the Nginx web server
							 | 
						||
| 
								 | 
							
								as a reverse proxy and we want HTTPS.
							 | 
						||
| 
								 | 
							
								For HTTPS we will use Let's Encrypt certificate.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								For setting up hosting we want to use cdist so we will write a new type
							 | 
						||
| 
								 | 
							
								for that. This type will:
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								- install required packages
							 | 
						||
| 
								 | 
							
								- create OS user, user home directory and application home directory
							 | 
						||
| 
								 | 
							
								- create PostgreSQL database
							 | 
						||
| 
								 | 
							
								- configure uWSGI
							 | 
						||
| 
								 | 
							
								- configure Let's Encrypt certificate
							 | 
						||
| 
								 | 
							
								- configure nginx.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Our type will not create the actual python application. Its intention is only
							 | 
						||
| 
								 | 
							
								to configure hosing for specified user and project. It is up to the user to
							 | 
						||
| 
								 | 
							
								create his/her applications.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								So let's start.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating type layout
							 | 
						||
| 
								 | 
							
								--------------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								We will create a new custom type. Let's call it **__sample_bottle_hosting**.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Go to **~/.cdist/type** directory (create it if it does not exist) and create
							 | 
						||
| 
								 | 
							
								new type layout::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    cd ~/.cdist/type
							 | 
						||
| 
								 | 
							
								    mkdir __sample_bottle_hosting
							 | 
						||
| 
								 | 
							
								    cd __sample_bottle_hosting
							 | 
						||
| 
								 | 
							
								    touch manifest gencode-remote
							 | 
						||
| 
								 | 
							
								    mkdir parameter
							 | 
						||
| 
								 | 
							
								    touch parameter/required
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating __sample_bottle_hosting type parameters
							 | 
						||
| 
								 | 
							
								------------------------------------------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Our type will be configurable through the means of parameters. Let's define
							 | 
						||
| 
								 | 
							
								the following parameters:
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								projectname
							 | 
						||
| 
								 | 
							
								    name for the project, needed for uWSGI ini file
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								user
							 | 
						||
| 
								 | 
							
								    user name
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								domain
							 | 
						||
| 
								 | 
							
								    target host domain, needed for Let's Encrypt certificate.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								We define parameters to make our type reusable for different projects, user and domain.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Define required parameters::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    printf "projectname\n" >> parameter/required
							 | 
						||
| 
								 | 
							
								    printf "user\n" >> parameter/required
							 | 
						||
| 
								 | 
							
								    printf "domain\n" >> parameter/required
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								For details on type parameters see `Defining parameters <cdist-type.html#defining-parameters>`_.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating __sample_bottle_hosting type manifest
							 | 
						||
| 
								 | 
							
								----------------------------------------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Next step is to define manifest (~/.cdist/type/__sample_bottle_hosting/manifest).
							 | 
						||
| 
								 | 
							
								We also want our type to currently support only Devuan. So we will start by
							 | 
						||
| 
								 | 
							
								checking target host OS.  We will use `os <cdist-reference.html#explorers>`_
							 | 
						||
| 
								 | 
							
								global explorer::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    os=$(cat "$__global/explorer/os")
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    case "$os" in
							 | 
						||
| 
								 | 
							
								        devuan)
							 | 
						||
| 
								 | 
							
								            :
							 | 
						||
| 
								 | 
							
								        ;;
							 | 
						||
| 
								 | 
							
								        *)
							 | 
						||
| 
								 | 
							
								            echo "OS $os currently not supported" >&2
							 | 
						||
| 
								 | 
							
								            exit 1
							 | 
						||
| 
								 | 
							
								        ;;
							 | 
						||
| 
								 | 
							
								    esac
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								If target host OS is not Devuan then we print error message to stderr
							 | 
						||
| 
								 | 
							
								and exit. For other OS-es support we should check and change package names
							 | 
						||
| 
								 | 
							
								we should install, because packages differ in different OS-es and in different
							 | 
						||
| 
								 | 
							
								OS distributions like GNU/Linux distributions. There can also be a different
							 | 
						||
| 
								 | 
							
								configuration locations (e.g. nginx config directory could be in /usr/local tree).
							 | 
						||
| 
								 | 
							
								If we detected unsupported OS we should error out. cdist will stop configuration
							 | 
						||
| 
								 | 
							
								process and output error message.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating user and user directories
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Then we create user and his/her home directory and application home directory.
							 | 
						||
| 
								 | 
							
								We will use existing cdist types `__user <man7/cdist-type__user.html>`_ and `__directory <man7/cdist-type__directory.html>`_::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    user="$(cat "$__object/parameter/user")"
							 | 
						||
| 
								 | 
							
								    home="/home/$user"
							 | 
						||
| 
								 | 
							
								    apphome="$home/app"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # create user
							 | 
						||
| 
								 | 
							
								    __user "$user" --home "$home" --shell /bin/bash
							 | 
						||
| 
								 | 
							
								    # create user home dir
							 | 
						||
| 
								 | 
							
								    require="__user/$user" __directory "$home" \
							 | 
						||
| 
								 | 
							
								        --owner "$user" --group "$user" --mode 0755
							 | 
						||
| 
								 | 
							
								    # create app home dir
							 | 
						||
| 
								 | 
							
								    require="__user/$user __directory/$home" __directory "$apphome" \
							 | 
						||
| 
								 | 
							
								        --state present --owner "$user" --group "$user" --mode 0755
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								First we define *user*, *home* and *apphome* variables. User is defined by type's
							 | 
						||
| 
								 | 
							
								**user** parameter. Here we use **require** which is cdist's way to define dependencies.
							 | 
						||
| 
								 | 
							
								User home directory should be created **after** user is created. And application
							 | 
						||
| 
								 | 
							
								home directory is created **after** both user and user home directory are created.
							 | 
						||
| 
								 | 
							
								For details on **require** see `Dependencies <cdist-manifest.html#dependencies>`_.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Installing packages
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Install required packages using existing `__package <man7/cdist-type__package.html>`_ type.
							 | 
						||
| 
								 | 
							
								Before installing package we want to update apt package index using
							 | 
						||
| 
								 | 
							
								`__apt_update_index <man7/cdist-type__apt_update_index.html>`_::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # define packages that need to be installed
							 | 
						||
| 
								 | 
							
								    packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # update package index
							 | 
						||
| 
								 | 
							
								    __apt_update_index
							 | 
						||
| 
								 | 
							
								    # install packages
							 | 
						||
| 
								 | 
							
								    for package in $packages_to_install
							 | 
						||
| 
								 | 
							
								        do require="__apt_update_index" __package $package --state=present
							 | 
						||
| 
								 | 
							
								    done
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Here we use shell for loop. It executes **require="__apt_update_index" __package**
							 | 
						||
| 
								 | 
							
								for each member in a list we define in **packages_to_install** variable.
							 | 
						||
| 
								 | 
							
								This is much nicer then having as many **require="__apt_update_index" __package**
							 | 
						||
| 
								 | 
							
								lines as there are packages we want to install.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								For python packages we use `__package_pip <man7/cdist-type__package_pip.html>`_::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # install pip3 packages
							 | 
						||
| 
								 | 
							
								    for package in bottle bottle-pgsql; do
							 | 
						||
| 
								 | 
							
								        __package_pip --pip pip3 $package
							 | 
						||
| 
								 | 
							
								    done
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating PostgreSQL database
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Create PostgreSQL database using `__postgres_database <man7/cdist-type__postgres_database.html>`_
							 | 
						||
| 
								 | 
							
								and `__postgres_role <man7/cdist-type__postgres_role.html>`_ for creating database user::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    #PostgreSQL db & user
							 | 
						||
| 
								 | 
							
								    postgres_server=postgresql
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # create PostgreSQL db user
							 | 
						||
| 
								 | 
							
								    require="__package/postgresql" __postgres_role $user --login --createdb
							 | 
						||
| 
								 | 
							
								    # create PostgreSQL db
							 | 
						||
| 
								 | 
							
								    require="__postgres_role/$user __package/postgresql" __postgres_database $user \
							 | 
						||
| 
								 | 
							
								        --owner $user
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Configuring uWSGI
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Configure uWSGI using `__file <man7/cdist-type__file.html>`_ type::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # configure uWSGI
							 | 
						||
| 
								 | 
							
								    projectname="$(cat "$__object/parameter/projectname")"
							 | 
						||
| 
								 | 
							
								    require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
							 | 
						||
| 
								 | 
							
								                --owner root --group root --mode 0644 \
							 | 
						||
| 
								 | 
							
								                --state present \
							 | 
						||
| 
								 | 
							
								                --source - << EOF
							 | 
						||
| 
								 | 
							
								    [uwsgi]
							 | 
						||
| 
								 | 
							
								    socket = $apphome/uwsgi.sock
							 | 
						||
| 
								 | 
							
								    chdir = $apphome
							 | 
						||
| 
								 | 
							
								    wsgi-file = $projectname/wsgi.py
							 | 
						||
| 
								 | 
							
								    touch-reload = $projectname/wsgi.py
							 | 
						||
| 
								 | 
							
								    processes = 4
							 | 
						||
| 
								 | 
							
								    threads = 2
							 | 
						||
| 
								 | 
							
								    chmod-socket = 666
							 | 
						||
| 
								 | 
							
								    daemonize=true
							 | 
						||
| 
								 | 
							
								    vacuum = true
							 | 
						||
| 
								 | 
							
								    uid = $user
							 | 
						||
| 
								 | 
							
								    gid = $user
							 | 
						||
| 
								 | 
							
								    EOF
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								We require package uWSGI present in order to create **/etc/uwsgi/apps-enabled/$user.ini** file.
							 | 
						||
| 
								 | 
							
								Installation of uWSGI also creates configuration layout: **/etc/uwsgi/apps-enabled**.
							 | 
						||
| 
								 | 
							
								If this directory does not exist then **__file** type would error.
							 | 
						||
| 
								 | 
							
								We also use stdin as file content source. For details see `Input from stdin <cdist-type.html#input-from-stdin>`_.
							 | 
						||
| 
								 | 
							
								For feading stdin we use here-document (**<<** operator). It allows redirection of subsequent
							 | 
						||
| 
								 | 
							
								lines read by the shell to the input of a command until a line containing only the delimiter
							 | 
						||
| 
								 | 
							
								and a newline, with no blank characters in between (EOF in our case).
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Configuring nginx for Let's Encrypt and HTTPS redirection
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Next configure nginx for Let's Encrypt and for HTTP -> HTTPS redirection. For this
							 | 
						||
| 
								 | 
							
								purpose we will create new type **__sample_nginx_http_letsencrypt_and_ssl_redirect**
							 | 
						||
| 
								 | 
							
								and use it here::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    domain="$(cat "$__object/parameter/domain")"
							 | 
						||
| 
								 | 
							
								    webroot="/var/www/html"
							 | 
						||
| 
								 | 
							
								    __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Configuring certificate creation
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								After HTTP nginx configuration we will create Let's Encrypt certificate using
							 | 
						||
| 
								 | 
							
								`__letsencrypt_cert <man7/cdist-type__letsencrypt_cert.html>`_ type.
							 | 
						||
| 
								 | 
							
								For Let's Encrypt cert configuration ensure that there is a DNS entry for your
							 | 
						||
| 
								 | 
							
								domain. We assure that cert creation is applied after nginx HTTP is configured
							 | 
						||
| 
								 | 
							
								for Let's Encrypt to work::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # create SSL cert
							 | 
						||
| 
								 | 
							
								    require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
							 | 
						||
| 
								 | 
							
								        __letsencrypt_cert --admin-email admin@test.ungleich.ch \
							 | 
						||
| 
								 | 
							
								            --webroot "$webroot" \
							 | 
						||
| 
								 | 
							
								            --automatic-renewal \
							 | 
						||
| 
								 | 
							
								            --renew-hook "service nginx reload" \
							 | 
						||
| 
								 | 
							
								            --domain "$domain" \
							 | 
						||
| 
								 | 
							
								            "$domain"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Configuring nginx HTTPS server with uWSGI upstream
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Then we can configure nginx HTTPS server that will use created Let's Encrypt certificate::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # configure nginx
							 | 
						||
| 
								 | 
							
								    require="__package/nginx __letsencrypt_cert/$domain" \
							 | 
						||
| 
								 | 
							
								        __file "/etc/nginx/sites-enabled/https-$domain" \
							 | 
						||
| 
								 | 
							
								        --source - --mode 0644 << EOF
							 | 
						||
| 
								 | 
							
								    upstream _bottle {
							 | 
						||
| 
								 | 
							
								        server unix:$apphome/uwsgi.sock;
							 | 
						||
| 
								 | 
							
								    }
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    server {
							 | 
						||
| 
								 | 
							
								        listen 443;
							 | 
						||
| 
								 | 
							
								        listen [::]:443;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        server_name $domain;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        access_log  /var/log/nginx/access.log;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        ssl on;
							 | 
						||
| 
								 | 
							
								        ssl_certificate      /etc/letsencrypt/live/$domain/fullchain.pem;
							 | 
						||
| 
								 | 
							
								        ssl_certificate_key  /etc/letsencrypt/live/$domain/privkey.pem;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        client_max_body_size 256m;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        location / {
							 | 
						||
| 
								 | 
							
								            try_files \$uri @uwsgi;
							 | 
						||
| 
								 | 
							
								        }
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        location @uwsgi {
							 | 
						||
| 
								 | 
							
								            include uwsgi_params;
							 | 
						||
| 
								 | 
							
								            uwsgi_pass _bottle;
							 | 
						||
| 
								 | 
							
								        }
							 | 
						||
| 
								 | 
							
								    }
							 | 
						||
| 
								 | 
							
								    EOF
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Now our manifest is finished.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Complete __sample_bottle_hosting type manifest listing
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Here is complete __sample_bottle_hosting type manifest listing,
							 | 
						||
| 
								 | 
							
								located in ~/.cdist/type/__sample_bottle_hosting/manifest::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    #!/bin/sh
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    os=$(cat "$__global/explorer/os")
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    case "$os" in
							 | 
						||
| 
								 | 
							
								        devuan)
							 | 
						||
| 
								 | 
							
								            :
							 | 
						||
| 
								 | 
							
								        ;;
							 | 
						||
| 
								 | 
							
								        *)
							 | 
						||
| 
								 | 
							
								            echo "OS $os currently not supported" >&2
							 | 
						||
| 
								 | 
							
								            exit 1
							 | 
						||
| 
								 | 
							
								        ;;
							 | 
						||
| 
								 | 
							
								    esac
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    projectname="$(cat "$__object/parameter/projectname")"
							 | 
						||
| 
								 | 
							
								    user="$(cat "$__object/parameter/user")"
							 | 
						||
| 
								 | 
							
								    home="/home/$user"
							 | 
						||
| 
								 | 
							
								    apphome="$home/app"
							 | 
						||
| 
								 | 
							
								    domain="$(cat "$__object/parameter/domain")"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # create user
							 | 
						||
| 
								 | 
							
								    __user "$user" --home "$home" --shell /bin/bash
							 | 
						||
| 
								 | 
							
								    # create user home dir
							 | 
						||
| 
								 | 
							
								    require="__user/$user" __directory "$home" \
							 | 
						||
| 
								 | 
							
								        --owner "$user" --group "$user" --mode 0755
							 | 
						||
| 
								 | 
							
								    # create app home dir
							 | 
						||
| 
								 | 
							
								    require="__user/$user __directory/$home" __directory "$apphome" \
							 | 
						||
| 
								 | 
							
								        --state present --owner "$user" --group "$user" --mode 0755
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # define packages that need to be installed
							 | 
						||
| 
								 | 
							
								    packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # update package index
							 | 
						||
| 
								 | 
							
								    __apt_update_index
							 | 
						||
| 
								 | 
							
								    # install packages
							 | 
						||
| 
								 | 
							
								    for package in $packages_to_install
							 | 
						||
| 
								 | 
							
								        do require="__apt_update_index" __package $package --state=present
							 | 
						||
| 
								 | 
							
								    done
							 | 
						||
| 
								 | 
							
								    # install pip3 packages
							 | 
						||
| 
								 | 
							
								    for package in bottle bottle-pgsql; do
							 | 
						||
| 
								 | 
							
								        __package_pip --pip pip3 $package
							 | 
						||
| 
								 | 
							
								    done
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    #PostgreSQL db & user
							 | 
						||
| 
								 | 
							
								    postgres_server=postgresql
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # create PostgreSQL db user
							 | 
						||
| 
								 | 
							
								    require="__package/postgresql" __postgres_role $user --login --createdb
							 | 
						||
| 
								 | 
							
								    # create PostgreSQL db
							 | 
						||
| 
								 | 
							
								    require="__postgres_role/$user __package/postgresql" __postgres_database $user \
							 | 
						||
| 
								 | 
							
								        --owner $user
							 | 
						||
| 
								 | 
							
								    # configure uWSGI
							 | 
						||
| 
								 | 
							
								    require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
							 | 
						||
| 
								 | 
							
								                --owner root --group root --mode 0644 \
							 | 
						||
| 
								 | 
							
								                --state present \
							 | 
						||
| 
								 | 
							
								                --source - << EOF
							 | 
						||
| 
								 | 
							
								    [uwsgi]
							 | 
						||
| 
								 | 
							
								    socket = $apphome/uwsgi.sock
							 | 
						||
| 
								 | 
							
								    chdir = $apphome
							 | 
						||
| 
								 | 
							
								    wsgi-file = $projectname/wsgi.py
							 | 
						||
| 
								 | 
							
								    touch-reload = $projectname/wsgi.py
							 | 
						||
| 
								 | 
							
								    processes = 4
							 | 
						||
| 
								 | 
							
								    threads = 2
							 | 
						||
| 
								 | 
							
								    chmod-socket = 666
							 | 
						||
| 
								 | 
							
								    daemonize=true
							 | 
						||
| 
								 | 
							
								    vacuum = true
							 | 
						||
| 
								 | 
							
								    uid = $user
							 | 
						||
| 
								 | 
							
								    gid = $user
							 | 
						||
| 
								 | 
							
								    EOF
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # setup nginx HTTP for Let's Encrypt and SSL redirect
							 | 
						||
| 
								 | 
							
								    domain="$(cat "$__object/parameter/domain")"
							 | 
						||
| 
								 | 
							
								    webroot="/var/www/html"
							 | 
						||
| 
								 | 
							
								    __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # create SSL cert
							 | 
						||
| 
								 | 
							
								    require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
							 | 
						||
| 
								 | 
							
								        __letsencrypt_cert --admin-email admin@test.ungleich.ch \
							 | 
						||
| 
								 | 
							
								            --webroot "$webroot" \
							 | 
						||
| 
								 | 
							
								            --automatic-renewal \
							 | 
						||
| 
								 | 
							
								            --renew-hook "service nginx reload" \
							 | 
						||
| 
								 | 
							
								            --domain "$domain" \
							 | 
						||
| 
								 | 
							
								            "$domain"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    # configure nginx
							 | 
						||
| 
								 | 
							
								    require="__package/nginx __letsencrypt_cert/$domain" \
							 | 
						||
| 
								 | 
							
								        __file "/etc/nginx/sites-enabled/https-$domain" \
							 | 
						||
| 
								 | 
							
								        --source - --mode 0644 << EOF
							 | 
						||
| 
								 | 
							
								    upstream _bottle {
							 | 
						||
| 
								 | 
							
								        server unix:$apphome/uwsgi.sock;
							 | 
						||
| 
								 | 
							
								    }
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    server {
							 | 
						||
| 
								 | 
							
								        listen 443;
							 | 
						||
| 
								 | 
							
								        listen [::]:443;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        server_name $domain;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        access_log  /var/log/nginx/access.log;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        ssl on;
							 | 
						||
| 
								 | 
							
								        ssl_certificate      /etc/letsencrypt/live/$domain/fullchain.pem;
							 | 
						||
| 
								 | 
							
								        ssl_certificate_key  /etc/letsencrypt/live/$domain/privkey.pem;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        client_max_body_size 256m;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        location / {
							 | 
						||
| 
								 | 
							
								            try_files \$uri @uwsgi;
							 | 
						||
| 
								 | 
							
								        }
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        location @uwsgi {
							 | 
						||
| 
								 | 
							
								            include uwsgi_params;
							 | 
						||
| 
								 | 
							
								            uwsgi_pass _bottle;
							 | 
						||
| 
								 | 
							
								        }
							 | 
						||
| 
								 | 
							
								    }
							 | 
						||
| 
								 | 
							
								    EOF
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating __sample_bottle_hosting type gencode-remote
							 | 
						||
| 
								 | 
							
								----------------------------------------------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Now define **gencode-remote** script: ~/.cdist/type/__sample_bottle_hosting/gencode-remote.
							 | 
						||
| 
								 | 
							
								After manifest is applied it should restart uWSGI and nginx services so that our
							 | 
						||
| 
								 | 
							
								configuration is active. Our gencode-remote looks like the following::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    echo "service uwsgi restart"
							 | 
						||
| 
								 | 
							
								    echo "service nginx restart"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Our **__sample_bottle_hosting** type is now finished.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating __sample_nginx_http_letsencrypt_and_ssl_redirect type
							 | 
						||
| 
								 | 
							
								--------------------------------------------------------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Let's now create **__sample_nginx_http_letsencrypt_and_ssl_redirect** type::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    cd ~/.cdist/type
							 | 
						||
| 
								 | 
							
								    mkdir __sample_nginx_http_letsencrypt_and_ssl_redirect
							 | 
						||
| 
								 | 
							
								    cd __sample_nginx_http_letsencrypt_and_ssl_redirect
							 | 
						||
| 
								 | 
							
								    mkdir parameter
							 | 
						||
| 
								 | 
							
								    echo webroot > parameter/required
							 | 
						||
| 
								 | 
							
								    touch manifest
							 | 
						||
| 
								 | 
							
								    touch gencode-remote
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Edit manifest::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    domain="$__object_id"
							 | 
						||
| 
								 | 
							
								    webroot="$(cat "$__object/parameter/webroot")"
							 | 
						||
| 
								 | 
							
								    # make sure we have nginx package
							 | 
						||
| 
								 | 
							
								    __package nginx
							 | 
						||
| 
								 | 
							
								    # setup Let's Encrypt HTTP acme challenge, redirect HTTP to HTTPS
							 | 
						||
| 
								 | 
							
								    require="__package/nginx" __file "/etc/nginx/sites-enabled/http-$domain" \
							 | 
						||
| 
								 | 
							
								        --source - --mode 0644 << EOF
							 | 
						||
| 
								 | 
							
								    server {
							 | 
						||
| 
								 | 
							
								        listen *:80;
							 | 
						||
| 
								 | 
							
								        listen [::]:80;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        server_name $domain;
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        # Let's Encrypt
							 | 
						||
| 
								 | 
							
								        location /.well-known/acme-challenge/ {
							 | 
						||
| 
								 | 
							
								            root $webroot;
							 | 
						||
| 
								 | 
							
								        }
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        # Everything else -> SSL
							 | 
						||
| 
								 | 
							
								        location / {
							 | 
						||
| 
								 | 
							
								            return 301 https://\$host\$request_uri;
							 | 
						||
| 
								 | 
							
								        }
							 | 
						||
| 
								 | 
							
								    }
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    EOF
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Edit gencode-remote::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    echo "service nginx reload"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating init manifest
							 | 
						||
| 
								 | 
							
								----------------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Next create init manifest::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    cd ~/.cdist/manifest
							 | 
						||
| 
								 | 
							
								    printf "__sample_bottle_hosting --projectname sample --user app --domain \$__target_host sample\n" > sample
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Using this init manifest our target host will be configured using our **__sample_bottle_hosting**
							 | 
						||
| 
								 | 
							
								type with projectname *sample*, user *app* and domain equal to **__target_host**.
							 | 
						||
| 
								 | 
							
								Here the last positional argument *sample* is type's object id. For details on
							 | 
						||
| 
								 | 
							
								**__target_host** and **__object_id** see
							 | 
						||
| 
								 | 
							
								`Environment variables (for reading) <cdist-reference.html#environment-variables-for-reading>`_
							 | 
						||
| 
								 | 
							
								reference.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Configuring host
							 | 
						||
| 
								 | 
							
								----------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Finally configure test.ungleich.ch::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    cdist config -v -i ~/.cdist/manifest/sample test.ungleich.ch
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								After cdist configuration is successfully finished our host is ready.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating python bottle application
							 | 
						||
| 
								 | 
							
								----------------------------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								We now need to create Bottle application. As you remember from the beginning
							 | 
						||
| 
								 | 
							
								of this walkthrough our type does not create the actual python application,
							 | 
						||
| 
								 | 
							
								its intention is only to configure hosing for specified user and project.
							 | 
						||
| 
								 | 
							
								It is up to the user to create his/her applications.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Become app user::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    su -l app
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Preparing database
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								We need to prepare database for our application. Create table and
							 | 
						||
| 
								 | 
							
								insert some items::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    psql -c "create table items (item varchar(255));"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    psql -c "insert into items(item) values('spam');"
							 | 
						||
| 
								 | 
							
								    psql -c "insert into items(item) values('eggs');"
							 | 
						||
| 
								 | 
							
								    psql -c "insert into items(item) values('sausage');"
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Creating application
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Next create sample app::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    cd /home/app/app
							 | 
						||
| 
								 | 
							
								    mkdir sample
							 | 
						||
| 
								 | 
							
								    cd sample
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Create app.py with the following content::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    #!/usr/bin/env python3
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    import bottle
							 | 
						||
| 
								 | 
							
								    import bottle_pgsql
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    app = application = bottle.Bottle()
							 | 
						||
| 
								 | 
							
								    plugin = bottle_pgsql.Plugin('dbname=app user=app password=')
							 | 
						||
| 
								 | 
							
								    app.install(plugin)
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    @app.route('/')
							 | 
						||
| 
								 | 
							
								    def show_index(db):
							 | 
						||
| 
								 | 
							
								        db.execute('select * from items')
							 | 
						||
| 
								 | 
							
								        items = db.fetchall() or []
							 | 
						||
| 
								 | 
							
								        rv = '<html><body><h3>Items:</h3><ul>'
							 | 
						||
| 
								 | 
							
								        for item in items:
							 | 
						||
| 
								 | 
							
								            rv += '<li>' + str(item['item']) + '</li>'
							 | 
						||
| 
								 | 
							
								        rv += '</ul></body></html>'
							 | 
						||
| 
								 | 
							
								        return rv
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    if __name__ == '__main__':
							 | 
						||
| 
								 | 
							
								        bottle.run(app=app, host='0.0.0.0', port=8080)
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Create wsgi.py with the following content::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    import os
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    os.chdir(os.path.dirname(__file__))
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    import app
							 | 
						||
| 
								 | 
							
								    application = app.app
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								We have configured uWSGI with **touch-reload = $projectname/wsgi.py** so after
							 | 
						||
| 
								 | 
							
								we have changed our **wsgi.py** file uWSGI reloads the application.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Our application selects and lists items from **items** table.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Openning application
							 | 
						||
| 
								 | 
							
								~~~~~~~~~~~~~~~~~~~~
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Finally try the application::
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    http://test.ungleich.ch/
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								It should redirect to HTTPS and return:
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								.. container:: highlight
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								    .. raw:: html
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        <h3>Items:</h3>
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								        <ul>
							 | 
						||
| 
								 | 
							
								            <li>spam</li>
							 | 
						||
| 
								 | 
							
								            <li>eggs</li>
							 | 
						||
| 
								 | 
							
								            <li>sausage</li>
							 | 
						||
| 
								 | 
							
								        </ul>
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								What's next?
							 | 
						||
| 
								 | 
							
								------------
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Continue reading next sections ;)
							 |