moep/cdist
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Add 'real world example' walkthrough docs chapter.

Browse source
This commit is contained in:
Darko Poljak 2018-09-06 07:06:25 +02:00
parent aa80e8f87d
commit d6952543a7
3 changed files with 575 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
docs/changelog
View file

@ -5,6 +5,7 @@ next:
* New global explorer: os_release (Ľubomír Kučera)
* Type __docker: Update type, install docker CE (Ľubomír Kučera)
* Type __package_apt: Write a message when a package is installed or removed; shellcheck (Jonas Weber)
* Documentation: Add 'Dive into real world cdist' walkthrough chapter (Darko Poljak)
4.10.2: 2018-09-06
* Type __letsencrypt_cert: Add support for devuan ascii (Darko Poljak)

573
docs/src/cdist-real-world.rst Normal file
View file

@ -0,0 +1,573 @@
Dive into real world cdist
==========================
Introduction
------------
This walkthrough shows real world cdist configuration example.
Sample target host is named **test.ungleich.ch**.
Just replace **test.ungleich.ch** with your target hostname.
Our goal is to configure python application hosting. For writing sample
application we will use `Bottle <http://bottlepy.org>`_ WSGI micro web-framework.
It will use PostgreSQL database and it will list items from **items** table.
It will be served by uWSGI server. We will also use the Nginx web server
as a reverse proxy and we want HTTPS.
For HTTPS we will use Let's Encrypt certificate.
For setting up hosting we want to use cdist so we will write a new type
for that. This type will:
- install required packages
- create OS user, user home directory and application home directory
- create PostgreSQL database
- configure uWSGI
- configure Let's Encrypt certificate
- configure nginx.
Our type will not create the actual python application. Its intention is only
to configure hosing for specified user and project. It is up to the user to
create his/her applications.
So let's start.
Creating type layout
--------------------
We will create a new custom type. Let's call it **__sample_bottle_hosting**.
Go to **~/.cdist/type** directory (create it if it does not exist) and create
new type layout::
cd ~/.cdist/type
mkdir __sample_bottle_hosting
cd __sample_bottle_hosting
touch manifest gencode-remote
mkdir parameter
touch parameter/required
Creating __sample_bottle_hosting type parameters
------------------------------------------------
Our type will be configurable through the means of parameters. Let's define
the following parameters:
projectname
name for the project, needed for uWSGI ini file
user
user name
domain
target host domain, needed for Let's Encrypt certificate.
We define parameters to make our type reusable for different projects, user and domain.
Define required parameters::
printf "projectname\n" >> parameter/required
printf "user\n" >> parameter/required
printf "domain\n" >> parameter/required
For details on type parameters see `Defining parameters <cdist-type.html#defining-parameters>`_.
Creating __sample_bottle_hosting type manifest
----------------------------------------------
Next step is to define manifest (~/.cdist/type/__sample_bottle_hosting/manifest).
We also want our type to currently support only Devuan. So we will start by
checking target host OS. We will use `os <cdist-reference.html#explorers>`_
global explorer::
os=$(cat "$__global/explorer/os")
case "$os" in
devuan)
:
;;
*)
echo "OS $os currently not supported" >&2
exit 1
;;
esac
If target host OS is not Devuan then we print error message to stderr
and exit. For other OS-es support we should check and change package names
we should install, because packages differ in different OS-es and in different
OS distributions like GNU/Linux distributions. There can also be a different
configuration locations (e.g. nginx config directory could be in /usr/local tree).
If we detected unsupported OS we should error out. cdist will stop configuration
process and output error message.
Creating user and user directories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Then we create user and his/her home directory and application home directory.
We will use existing cdist types `__user <man7/cdist-type__user.html>`_ and `__directory <man7/cdist-type__directory.html>`_::
user="$(cat "$__object/parameter/user")"
home="/home/$user"
apphome="$home/app"
# create user
__user "$user" --home "$home" --shell /bin/bash
# create user home dir
require="__user/$user" __directory "$home" \
--owner "$user" --group "$user" --mode 0755
# create app home dir
require="__user/$user __directory/$home" __directory "$apphome" \
--state present --owner "$user" --group "$user" --mode 0755
First we define *user*, *home* and *apphome* variables. User is defined by type's
**user** parameter. Here we use **require** which is cdist's way to define dependencies.
User home directory should be created **after** user is created. And application
home directory is created **after** both user and user home directory are created.
For details on **require** see `Dependencies <cdist-manifest.html#dependencies>`_.
Installing packages
~~~~~~~~~~~~~~~~~~~
Install required packages using existing `__package <man7/cdist-type__package.html>`_ type.
Before installing package we want to update apt package index using
`__apt_update_index <man7/cdist-type__apt_update_index.html>`_::
# define packages that need to be installed
packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"
# update package index
__apt_update_index
# install packages
for package in $packages_to_install
do require="__apt_update_index" __package $package --state=present
done
Here we use shell for loop. It executes **require="__apt_update_index" __package**
for each member in a list we define in **packages_to_install** variable.
This is much nicer then having as many **require="__apt_update_index" __package**
lines as there are packages we want to install.
For python packages we use `__package_pip <man7/cdist-type__package_pip.html>`_::
# install pip3 packages
for package in bottle bottle-pgsql; do
__package_pip --pip pip3 $package
done
Creating PostgreSQL database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create PostgreSQL database using `__postgres_database <man7/cdist-type__postgres_database.html>`_
and `__postgres_role <man7/cdist-type__postgres_role.html>`_ for creating database user::
#PostgreSQL db & user
postgres_server=postgresql
# create PostgreSQL db user
require="__package/postgresql" __postgres_role $user --login --createdb
# create PostgreSQL db
require="__postgres_role/$user __package/postgresql" __postgres_database $user \
--owner $user
Configuring uWSGI
~~~~~~~~~~~~~~~~~
Configure uWSGI using `__file <man7/cdist-type__file.html>`_ type::
# configure uWSGI
projectname="$(cat "$__object/parameter/projectname")"
require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
--owner root --group root --mode 0644 \
--state present \
--source - << EOF
[uwsgi]
socket = $apphome/uwsgi.sock
chdir = $apphome
wsgi-file = $projectname/wsgi.py
touch-reload = $projectname/wsgi.py
processes = 4
threads = 2
chmod-socket = 666
daemonize=true
vacuum = true
uid = $user
gid = $user
EOF
We require package uWSGI present in order to create **/etc/uwsgi/apps-enabled/$user.ini** file.
Installation of uWSGI also creates configuration layout: **/etc/uwsgi/apps-enabled**.
If this directory does not exist then **__file** type would error.
We also use stdin as file content source. For details see `Input from stdin <cdist-type.html#input-from-stdin>`_.
For feading stdin we use here-document (**<<** operator). It allows redirection of subsequent
lines read by the shell to the input of a command until a line containing only the delimiter
and a newline, with no blank characters in between (EOF in our case).
Configuring nginx for Let's Encrypt and HTTPS redirection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Next configure nginx for Let's Encrypt and for HTTP -> HTTPS redirection. For this
purpose we will create new type **__sample_nginx_http_letsencrypt_and_ssl_redirect**
and use it here::
domain="$(cat "$__object/parameter/domain")"
webroot="/var/www/html"
__sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"
Configuring certificate creation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After HTTP nginx configuration we will create Let's Encrypt certificate using
`__letsencrypt_cert <man7/cdist-type__letsencrypt_cert.html>`_ type.
For Let's Encrypt cert configuration ensure that there is a DNS entry for your
domain. We assure that cert creation is applied after nginx HTTP is configured
for Let's Encrypt to work::
# create SSL cert
require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
__letsencrypt_cert --admin-email admin@test.ungleich.ch \
--webroot "$webroot" \
--automatic-renewal \
--renew-hook "service nginx reload" \
--domain "$domain" \
"$domain"
Configuring nginx HTTPS server with uWSGI upstream
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Then we can configure nginx HTTPS server that will use created Let's Encrypt certificate::
# configure nginx
require="__package/nginx __letsencrypt_cert/$domain" \
__file "/etc/nginx/sites-enabled/https-$domain" \
--source - --mode 0644 << EOF
upstream _bottle {
server unix:$apphome/uwsgi.sock;
}
server {
listen 443;
listen [::]:443;
server_name $domain;
access_log /var/log/nginx/access.log;
ssl on;
ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem;
client_max_body_size 256m;
location / {
try_files \$uri @uwsgi;
}
location @uwsgi {
include uwsgi_params;
uwsgi_pass _bottle;
}
}
EOF
Now our manifest is finished.
Complete __sample_bottle_hosting type manifest listing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here is complete __sample_bottle_hosting type manifest listing,
located in ~/.cdist/type/__sample_bottle_hosting/manifest::
#!/bin/sh
os=$(cat "$__global/explorer/os")
case "$os" in
devuan)
:
;;
*)
echo "OS $os currently not supported" >&2
exit 1
;;
esac
projectname="$(cat "$__object/parameter/projectname")"
user="$(cat "$__object/parameter/user")"
home="/home/$user"
apphome="$home/app"
domain="$(cat "$__object/parameter/domain")"
# create user
__user "$user" --home "$home" --shell /bin/bash
# create user home dir
require="__user/$user" __directory "$home" \
--owner "$user" --group "$user" --mode 0755
# create app home dir
require="__user/$user __directory/$home" __directory "$apphome" \
--state present --owner "$user" --group "$user" --mode 0755
# define packages that need to be installed
packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"
# update package index
__apt_update_index
# install packages
for package in $packages_to_install
do require="__apt_update_index" __package $package --state=present
done
# install pip3 packages
for package in bottle bottle-pgsql; do
__package_pip --pip pip3 $package
done
#PostgreSQL db & user
postgres_server=postgresql
# create PostgreSQL db user
require="__package/postgresql" __postgres_role $user --login --createdb
# create PostgreSQL db
require="__postgres_role/$user __package/postgresql" __postgres_database $user \
--owner $user
# configure uWSGI
require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
--owner root --group root --mode 0644 \
--state present \
--source - << EOF
[uwsgi]
socket = $apphome/uwsgi.sock
chdir = $apphome
wsgi-file = $projectname/wsgi.py
touch-reload = $projectname/wsgi.py
processes = 4
threads = 2
chmod-socket = 666
daemonize=true
vacuum = true
uid = $user
gid = $user
EOF
# setup nginx HTTP for Let's Encrypt and SSL redirect
domain="$(cat "$__object/parameter/domain")"
webroot="/var/www/html"
__sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"
# create SSL cert
require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
__letsencrypt_cert --admin-email admin@test.ungleich.ch \
--webroot "$webroot" \
--automatic-renewal \
--renew-hook "service nginx reload" \
--domain "$domain" \
"$domain"
# configure nginx
require="__package/nginx __letsencrypt_cert/$domain" \
__file "/etc/nginx/sites-enabled/https-$domain" \
--source - --mode 0644 << EOF
upstream _bottle {
server unix:$apphome/uwsgi.sock;
}
server {
listen 443;
listen [::]:443;
server_name $domain;
access_log /var/log/nginx/access.log;
ssl on;
ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem;
client_max_body_size 256m;
location / {
try_files \$uri @uwsgi;
}
location @uwsgi {
include uwsgi_params;
uwsgi_pass _bottle;
}
}
EOF
Creating __sample_bottle_hosting type gencode-remote
----------------------------------------------------
Now define **gencode-remote** script: ~/.cdist/type/__sample_bottle_hosting/gencode-remote.
After manifest is applied it should restart uWSGI and nginx services so that our
configuration is active. Our gencode-remote looks like the following::
echo "service uwsgi restart"
echo "service nginx restart"
Our **__sample_bottle_hosting** type is now finished.
Creating __sample_nginx_http_letsencrypt_and_ssl_redirect type
--------------------------------------------------------------
Let's now create **__sample_nginx_http_letsencrypt_and_ssl_redirect** type::
cd ~/.cdist/type
mkdir __sample_nginx_http_letsencrypt_and_ssl_redirect
cd __sample_nginx_http_letsencrypt_and_ssl_redirect
mkdir parameter
echo webroot > parameter/required
touch manifest
touch gencode-remote
Edit manifest::
domain="$__object_id"
webroot="$(cat "$__object/parameter/webroot")"
# make sure we have nginx package
__package nginx
# setup Let's Encrypt HTTP acme challenge, redirect HTTP to HTTPS
require="__package/nginx" __file "/etc/nginx/sites-enabled/http-$domain" \
--source - --mode 0644 << EOF
server {
listen *:80;
listen [::]:80;
server_name $domain;
# Let's Encrypt
location /.well-known/acme-challenge/ {
root $webroot;
}
# Everything else -> SSL
location / {
return 301 https://\$host\$request_uri;
}
}
EOF
Edit gencode-remote::
echo "service nginx reload"
Creating init manifest
----------------------
Next create init manifest::
cd ~/.cdist/manifest
printf "__sample_bottle_hosting --projectname sample --user app --domain \$__target_host sample\n" > sample
Using this init manifest our target host will be configured using our **__sample_bottle_hosting**
type with projectname *sample*, user *app* and domain equal to **__target_host**.
Here the last positional argument *sample* is type's object id. For details on
**__target_host** and **__object_id** see
`Environment variables (for reading) <cdist-reference.html#environment-variables-for-reading>`_
reference.
Configuring host
----------------
Finally configure test.ungleich.ch::
cdist config -v -i ~/.cdist/manifest/sample test.ungleich.ch
After cdist configuration is successfully finished our host is ready.
Creating python bottle application
----------------------------------
We now need to create Bottle application. As you remember from the beginning
of this walkthrough our type does not create the actual python application,
its intention is only to configure hosing for specified user and project.
It is up to the user to create his/her applications.
Become app user::
su -l app
Preparing database
~~~~~~~~~~~~~~~~~~
We need to prepare database for our application. Create table and
insert some items::
psql -c "create table items (item varchar(255));"
psql -c "insert into items(item) values('spam');"
psql -c "insert into items(item) values('eggs');"
psql -c "insert into items(item) values('sausage');"
Creating application
~~~~~~~~~~~~~~~~~~~~
Next create sample app::
cd /home/app/app
mkdir sample
cd sample
Create app.py with the following content::
#!/usr/bin/env python3
import bottle
import bottle_pgsql
app = application = bottle.Bottle()
plugin = bottle_pgsql.Plugin('dbname=app user=app password=')
app.install(plugin)
@app.route('/')
def show_index(db):
db.execute('select * from items')
items = db.fetchall() or []
rv = '<html><body><h3>Items:</h3><ul>'
for item in items:
rv += '<li>' + str(item['item']) + '</li>'
rv += '</ul></body></html>'
return rv
if __name__ == '__main__':
bottle.run(app=app, host='0.0.0.0', port=8080)
Create wsgi.py with the following content::
import os
os.chdir(os.path.dirname(__file__))
import app
application = app.app
We have configured uWSGI with **touch-reload = $projectname/wsgi.py** so after
we have changed our **wsgi.py** file uWSGI reloads the application.
Our application selects and lists items from **items** table.
Openning application
~~~~~~~~~~~~~~~~~~~~
Finally try the application::
http://test.ungleich.ch/
It should redirect to HTTPS and return:
.. container:: highlight
.. raw:: html
<h3>Items:</h3>
<ul>
<li>spam</li>
<li>eggs</li>
<li>sausage</li>
</ul>
What's next?
------------
Continue reading next sections ;)

1
docs/src/index.rst
View file

@ -16,6 +16,7 @@ Contents:
cdist-support
cdist-features
cdist-quickstart
cdist-real-world
man1/cdist
cdist-bootstrap
cdist-configuration