From d6952543a74e523a248342452cd4ecd4df81116b Mon Sep 17 00:00:00 2001 From: Darko Poljak Date: Thu, 6 Sep 2018 07:06:25 +0200 Subject: [PATCH] Add 'real world example' walkthrough docs chapter. --- docs/changelog | 1 + docs/src/cdist-real-world.rst | 573 ++++++++++++++++++++++++++++++++++ docs/src/index.rst | 1 + 3 files changed, 575 insertions(+) create mode 100644 docs/src/cdist-real-world.rst diff --git a/docs/changelog b/docs/changelog index 682544ff..62d44fbe 100644 --- a/docs/changelog +++ b/docs/changelog @@ -5,6 +5,7 @@ next: * New global explorer: os_release (Ľubomír Kučera) * Type __docker: Update type, install docker CE (Ľubomír Kučera) * Type __package_apt: Write a message when a package is installed or removed; shellcheck (Jonas Weber) + * Documentation: Add 'Dive into real world cdist' walkthrough chapter (Darko Poljak) 4.10.2: 2018-09-06 * Type __letsencrypt_cert: Add support for devuan ascii (Darko Poljak) diff --git a/docs/src/cdist-real-world.rst b/docs/src/cdist-real-world.rst new file mode 100644 index 00000000..8ccb0fc9 --- /dev/null +++ b/docs/src/cdist-real-world.rst @@ -0,0 +1,573 @@ +Dive into real world cdist +========================== + +Introduction +------------ + +This walkthrough shows real world cdist configuration example. + +Sample target host is named **test.ungleich.ch**. +Just replace **test.ungleich.ch** with your target hostname. + +Our goal is to configure python application hosting. For writing sample +application we will use `Bottle `_ WSGI micro web-framework. +It will use PostgreSQL database and it will list items from **items** table. +It will be served by uWSGI server. We will also use the Nginx web server +as a reverse proxy and we want HTTPS. +For HTTPS we will use Let's Encrypt certificate. + +For setting up hosting we want to use cdist so we will write a new type +for that. This type will: + +- install required packages +- create OS user, user home directory and application home directory +- create PostgreSQL database +- configure uWSGI +- configure Let's Encrypt certificate +- configure nginx. + +Our type will not create the actual python application. Its intention is only +to configure hosing for specified user and project. It is up to the user to +create his/her applications. + +So let's start. + +Creating type layout +-------------------- + +We will create a new custom type. Let's call it **__sample_bottle_hosting**. + +Go to **~/.cdist/type** directory (create it if it does not exist) and create +new type layout:: + + cd ~/.cdist/type + mkdir __sample_bottle_hosting + cd __sample_bottle_hosting + touch manifest gencode-remote + mkdir parameter + touch parameter/required + +Creating __sample_bottle_hosting type parameters +------------------------------------------------ + +Our type will be configurable through the means of parameters. Let's define +the following parameters: + +projectname + name for the project, needed for uWSGI ini file + +user + user name + +domain + target host domain, needed for Let's Encrypt certificate. + +We define parameters to make our type reusable for different projects, user and domain. + +Define required parameters:: + + printf "projectname\n" >> parameter/required + printf "user\n" >> parameter/required + printf "domain\n" >> parameter/required + +For details on type parameters see `Defining parameters `_. + +Creating __sample_bottle_hosting type manifest +---------------------------------------------- + +Next step is to define manifest (~/.cdist/type/__sample_bottle_hosting/manifest). +We also want our type to currently support only Devuan. So we will start by +checking target host OS. We will use `os `_ +global explorer:: + + os=$(cat "$__global/explorer/os") + + case "$os" in + devuan) + : + ;; + *) + echo "OS $os currently not supported" >&2 + exit 1 + ;; + esac + +If target host OS is not Devuan then we print error message to stderr +and exit. For other OS-es support we should check and change package names +we should install, because packages differ in different OS-es and in different +OS distributions like GNU/Linux distributions. There can also be a different +configuration locations (e.g. nginx config directory could be in /usr/local tree). +If we detected unsupported OS we should error out. cdist will stop configuration +process and output error message. + +Creating user and user directories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Then we create user and his/her home directory and application home directory. +We will use existing cdist types `__user `_ and `__directory `_:: + + user="$(cat "$__object/parameter/user")" + home="/home/$user" + apphome="$home/app" + + # create user + __user "$user" --home "$home" --shell /bin/bash + # create user home dir + require="__user/$user" __directory "$home" \ + --owner "$user" --group "$user" --mode 0755 + # create app home dir + require="__user/$user __directory/$home" __directory "$apphome" \ + --state present --owner "$user" --group "$user" --mode 0755 + +First we define *user*, *home* and *apphome* variables. User is defined by type's +**user** parameter. Here we use **require** which is cdist's way to define dependencies. +User home directory should be created **after** user is created. And application +home directory is created **after** both user and user home directory are created. +For details on **require** see `Dependencies `_. + +Installing packages +~~~~~~~~~~~~~~~~~~~ + +Install required packages using existing `__package `_ type. +Before installing package we want to update apt package index using +`__apt_update_index `_:: + + # define packages that need to be installed + packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2" + + # update package index + __apt_update_index + # install packages + for package in $packages_to_install + do require="__apt_update_index" __package $package --state=present + done + +Here we use shell for loop. It executes **require="__apt_update_index" __package** +for each member in a list we define in **packages_to_install** variable. +This is much nicer then having as many **require="__apt_update_index" __package** +lines as there are packages we want to install. + +For python packages we use `__package_pip `_:: + + # install pip3 packages + for package in bottle bottle-pgsql; do + __package_pip --pip pip3 $package + done + +Creating PostgreSQL database +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Create PostgreSQL database using `__postgres_database `_ +and `__postgres_role `_ for creating database user:: + + #PostgreSQL db & user + postgres_server=postgresql + + # create PostgreSQL db user + require="__package/postgresql" __postgres_role $user --login --createdb + # create PostgreSQL db + require="__postgres_role/$user __package/postgresql" __postgres_database $user \ + --owner $user + +Configuring uWSGI +~~~~~~~~~~~~~~~~~ + +Configure uWSGI using `__file `_ type:: + + # configure uWSGI + projectname="$(cat "$__object/parameter/projectname")" + require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \ + --owner root --group root --mode 0644 \ + --state present \ + --source - << EOF + [uwsgi] + socket = $apphome/uwsgi.sock + chdir = $apphome + wsgi-file = $projectname/wsgi.py + touch-reload = $projectname/wsgi.py + processes = 4 + threads = 2 + chmod-socket = 666 + daemonize=true + vacuum = true + uid = $user + gid = $user + EOF + +We require package uWSGI present in order to create **/etc/uwsgi/apps-enabled/$user.ini** file. +Installation of uWSGI also creates configuration layout: **/etc/uwsgi/apps-enabled**. +If this directory does not exist then **__file** type would error. +We also use stdin as file content source. For details see `Input from stdin `_. +For feading stdin we use here-document (**<<** operator). It allows redirection of subsequent +lines read by the shell to the input of a command until a line containing only the delimiter +and a newline, with no blank characters in between (EOF in our case). + +Configuring nginx for Let's Encrypt and HTTPS redirection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Next configure nginx for Let's Encrypt and for HTTP -> HTTPS redirection. For this +purpose we will create new type **__sample_nginx_http_letsencrypt_and_ssl_redirect** +and use it here:: + + domain="$(cat "$__object/parameter/domain")" + webroot="/var/www/html" + __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot" + +Configuring certificate creation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After HTTP nginx configuration we will create Let's Encrypt certificate using +`__letsencrypt_cert `_ type. +For Let's Encrypt cert configuration ensure that there is a DNS entry for your +domain. We assure that cert creation is applied after nginx HTTP is configured +for Let's Encrypt to work:: + + # create SSL cert + require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \ + __letsencrypt_cert --admin-email admin@test.ungleich.ch \ + --webroot "$webroot" \ + --automatic-renewal \ + --renew-hook "service nginx reload" \ + --domain "$domain" \ + "$domain" + +Configuring nginx HTTPS server with uWSGI upstream +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Then we can configure nginx HTTPS server that will use created Let's Encrypt certificate:: + + # configure nginx + require="__package/nginx __letsencrypt_cert/$domain" \ + __file "/etc/nginx/sites-enabled/https-$domain" \ + --source - --mode 0644 << EOF + upstream _bottle { + server unix:$apphome/uwsgi.sock; + } + + server { + listen 443; + listen [::]:443; + + server_name $domain; + + access_log /var/log/nginx/access.log; + + ssl on; + ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem; + + client_max_body_size 256m; + + location / { + try_files \$uri @uwsgi; + } + + location @uwsgi { + include uwsgi_params; + uwsgi_pass _bottle; + } + } + EOF + +Now our manifest is finished. + +Complete __sample_bottle_hosting type manifest listing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Here is complete __sample_bottle_hosting type manifest listing, +located in ~/.cdist/type/__sample_bottle_hosting/manifest:: + + #!/bin/sh + + os=$(cat "$__global/explorer/os") + + case "$os" in + devuan) + : + ;; + *) + echo "OS $os currently not supported" >&2 + exit 1 + ;; + esac + + projectname="$(cat "$__object/parameter/projectname")" + user="$(cat "$__object/parameter/user")" + home="/home/$user" + apphome="$home/app" + domain="$(cat "$__object/parameter/domain")" + + # create user + __user "$user" --home "$home" --shell /bin/bash + # create user home dir + require="__user/$user" __directory "$home" \ + --owner "$user" --group "$user" --mode 0755 + # create app home dir + require="__user/$user __directory/$home" __directory "$apphome" \ + --state present --owner "$user" --group "$user" --mode 0755 + + # define packages that need to be installed + packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2" + + # update package index + __apt_update_index + # install packages + for package in $packages_to_install + do require="__apt_update_index" __package $package --state=present + done + # install pip3 packages + for package in bottle bottle-pgsql; do + __package_pip --pip pip3 $package + done + + #PostgreSQL db & user + postgres_server=postgresql + + # create PostgreSQL db user + require="__package/postgresql" __postgres_role $user --login --createdb + # create PostgreSQL db + require="__postgres_role/$user __package/postgresql" __postgres_database $user \ + --owner $user + # configure uWSGI + require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \ + --owner root --group root --mode 0644 \ + --state present \ + --source - << EOF + [uwsgi] + socket = $apphome/uwsgi.sock + chdir = $apphome + wsgi-file = $projectname/wsgi.py + touch-reload = $projectname/wsgi.py + processes = 4 + threads = 2 + chmod-socket = 666 + daemonize=true + vacuum = true + uid = $user + gid = $user + EOF + + # setup nginx HTTP for Let's Encrypt and SSL redirect + domain="$(cat "$__object/parameter/domain")" + webroot="/var/www/html" + __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot" + + # create SSL cert + require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \ + __letsencrypt_cert --admin-email admin@test.ungleich.ch \ + --webroot "$webroot" \ + --automatic-renewal \ + --renew-hook "service nginx reload" \ + --domain "$domain" \ + "$domain" + + # configure nginx + require="__package/nginx __letsencrypt_cert/$domain" \ + __file "/etc/nginx/sites-enabled/https-$domain" \ + --source - --mode 0644 << EOF + upstream _bottle { + server unix:$apphome/uwsgi.sock; + } + + server { + listen 443; + listen [::]:443; + + server_name $domain; + + access_log /var/log/nginx/access.log; + + ssl on; + ssl_certificate /etc/letsencrypt/live/$domain/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/$domain/privkey.pem; + + client_max_body_size 256m; + + location / { + try_files \$uri @uwsgi; + } + + location @uwsgi { + include uwsgi_params; + uwsgi_pass _bottle; + } + } + EOF + +Creating __sample_bottle_hosting type gencode-remote +---------------------------------------------------- + +Now define **gencode-remote** script: ~/.cdist/type/__sample_bottle_hosting/gencode-remote. +After manifest is applied it should restart uWSGI and nginx services so that our +configuration is active. Our gencode-remote looks like the following:: + + echo "service uwsgi restart" + echo "service nginx restart" + +Our **__sample_bottle_hosting** type is now finished. + +Creating __sample_nginx_http_letsencrypt_and_ssl_redirect type +-------------------------------------------------------------- + +Let's now create **__sample_nginx_http_letsencrypt_and_ssl_redirect** type:: + + cd ~/.cdist/type + mkdir __sample_nginx_http_letsencrypt_and_ssl_redirect + cd __sample_nginx_http_letsencrypt_and_ssl_redirect + mkdir parameter + echo webroot > parameter/required + touch manifest + touch gencode-remote + +Edit manifest:: + + domain="$__object_id" + webroot="$(cat "$__object/parameter/webroot")" + # make sure we have nginx package + __package nginx + # setup Let's Encrypt HTTP acme challenge, redirect HTTP to HTTPS + require="__package/nginx" __file "/etc/nginx/sites-enabled/http-$domain" \ + --source - --mode 0644 << EOF + server { + listen *:80; + listen [::]:80; + + server_name $domain; + + # Let's Encrypt + location /.well-known/acme-challenge/ { + root $webroot; + } + + # Everything else -> SSL + location / { + return 301 https://\$host\$request_uri; + } + } + + EOF + +Edit gencode-remote:: + + echo "service nginx reload" + +Creating init manifest +---------------------- + +Next create init manifest:: + + cd ~/.cdist/manifest + printf "__sample_bottle_hosting --projectname sample --user app --domain \$__target_host sample\n" > sample + +Using this init manifest our target host will be configured using our **__sample_bottle_hosting** +type with projectname *sample*, user *app* and domain equal to **__target_host**. +Here the last positional argument *sample* is type's object id. For details on +**__target_host** and **__object_id** see +`Environment variables (for reading) `_ +reference. + +Configuring host +---------------- + +Finally configure test.ungleich.ch:: + + cdist config -v -i ~/.cdist/manifest/sample test.ungleich.ch + +After cdist configuration is successfully finished our host is ready. + +Creating python bottle application +---------------------------------- + +We now need to create Bottle application. As you remember from the beginning +of this walkthrough our type does not create the actual python application, +its intention is only to configure hosing for specified user and project. +It is up to the user to create his/her applications. + +Become app user:: + + su -l app + +Preparing database +~~~~~~~~~~~~~~~~~~ + +We need to prepare database for our application. Create table and +insert some items:: + + psql -c "create table items (item varchar(255));" + + psql -c "insert into items(item) values('spam');" + psql -c "insert into items(item) values('eggs');" + psql -c "insert into items(item) values('sausage');" + +Creating application +~~~~~~~~~~~~~~~~~~~~ + +Next create sample app:: + + cd /home/app/app + mkdir sample + cd sample + +Create app.py with the following content:: + + #!/usr/bin/env python3 + + import bottle + import bottle_pgsql + + app = application = bottle.Bottle() + plugin = bottle_pgsql.Plugin('dbname=app user=app password=') + app.install(plugin) + + @app.route('/') + def show_index(db): + db.execute('select * from items') + items = db.fetchall() or [] + rv = '

Items:

    ' + for item in items: + rv += '
  • ' + str(item['item']) + '
    • ' + rv += '
' + return rv + + if __name__ == '__main__': + bottle.run(app=app, host='0.0.0.0', port=8080) + +Create wsgi.py with the following content:: + + import os + + os.chdir(os.path.dirname(__file__)) + + import app + application = app.app + +We have configured uWSGI with **touch-reload = $projectname/wsgi.py** so after +we have changed our **wsgi.py** file uWSGI reloads the application. + +Our application selects and lists items from **items** table. + +Openning application +~~~~~~~~~~~~~~~~~~~~ + +Finally try the application:: + + http://test.ungleich.ch/ + +It should redirect to HTTPS and return: + +.. container:: highlight + + .. raw:: html + +

Items:

+ +
    +
  • spam
    • +
  • eggs
    • +
  • sausage
    • +
+ +What's next? +------------ + +Continue reading next sections ;) diff --git a/docs/src/index.rst b/docs/src/index.rst index 5d0bb537..bef91e1c 100644 --- a/docs/src/index.rst +++ b/docs/src/index.rst @@ -16,6 +16,7 @@ Contents: cdist-support cdist-features cdist-quickstart + cdist-real-world man1/cdist cdist-bootstrap cdist-configuration