Merge inventory from beta branch.

This commit is contained in:
Darko Poljak 2017-07-20 22:04:44 +02:00
commit e2a1519332
28 changed files with 1769 additions and 36 deletions

View file

@ -0,0 +1,211 @@
Inventory
=========
Introduction
------------
cdist comes with simple built-in tag based inventory. It is a simple inventory
with list of hosts and a host has a list of tags.
Inventory functionality is still in **beta** so it can be used only if beta
command line flag is specified (-b, --beta) or setting CDIST_BETA env var.
Description
-----------
The idea is to have simple tagging inventory. There is a list of hosts and for
each host there are tags. Inventory database is a set of files under inventory
database base directory. Filename equals hostname. Each file contains tags for
hostname with each tag on its own line.
Using inventory you can now configure hosts by selecting them by tags.
Tags have no values, as tags are just tags. Tag name-value would in this
context mean that host has two tags and it is selected by specifying that both
tags are present.
This inventory is **KISS** cdist built-in inventory database. You can maintain it
using cdist inventory interface or using standard UNIX tools.
cdist inventory interface
-------------------------
With cdist inventory interface you can list host(s) and tag(s), add host(s),
add tag(s), delete host(s) and delete tag(s).
Configuring hosts using inventory
---------------------------------
config command now has new options, **-t**, **-a** and **-A**.
**-A** means that all hosts in tag db is selected.
**-a** means that selected hosts must contain ALL specified tags.
**-t** means that host specifies tag - all hosts that have specified tags are
selected.
Examples
--------
.. code-block:: sh
# List inventory content
$ cdist inventory list -b
# List inventory for specified host localhost
$ cdist inventory list -b localhost
# List inventory for specified tag loadbalancer
$ cdist inventory list -b -t loadbalancer
# Add hosts to inventory
$ cdist inventory add-host -b web1 web2 web3
# Delete hosts from file old-hosts from inventory
$ cdist inventory del-host -b -f old-hosts
# Add tags to specifed hosts
$ cdist inventory add-tag -b -t europe,croatia,web,static web1 web2
# Add tag to all hosts in inventory
$ cdist inventory add-tag -b -t vm
# Delete all tags from specified host
$ cdist inventory del-tag -b -a localhost
# Delete tags read from stdin from hosts specified by file hosts
$ cdist inventory del-tag -b -T - -f hosts
# Configure hosts from inventory with any of specified tags
$ cdist config -b -t web dynamic
# Configure hosts from inventory with all specified tags
$ cdist config -b -t -a web dynamic
# Configure all hosts from inventory db
$ cdist config -b -A
Example of manipulating database
--------------------------------
.. code-block:: sh
$ python3 scripts/cdist inventory list -b
$ python3 scripts/cdist inventory add-host -b localhost
$ python3 scripts/cdist inventory add-host -b test.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost
test.mycloud.net
$ python3 scripts/cdist inventory add-host -b web1.mycloud.net web2.mycloud.net shell1.mycloud.net shell2.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost
test.mycloud.net
web1.mycloud.net
web2.mycloud.net
shell1.mycloud.net
shell2.mycloud.net
$ python3 scripts/cdist inventory add-tag -b -t web web1.mycloud.net web2.mycloud.net
$ python3 scripts/cdist inventory add-tag -b -t shell shell1.mycloud.net shell2.mycloud.net
$ python3 scripts/cdist inventory add-tag -b -t cloud
$ python3 scripts/cdist inventory list -b
localhost cloud
test.mycloud.net cloud
web1.mycloud.net cloud,web
web2.mycloud.net cloud,web
shell1.mycloud.net cloud,shell
shell2.mycloud.net cloud,shell
$ python3 scripts/cdist inventory add-tag -b -t test,web,shell test.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost cloud
test.mycloud.net cloud,shell,test,web
web1.mycloud.net cloud,web
web2.mycloud.net cloud,web
shell1.mycloud.net cloud,shell
shell2.mycloud.net cloud,shell
$ python3 scripts/cdist inventory del-tag -b -t shell test.mycloud.net
$ python3 scripts/cdist inventory list -b
localhost cloud
test.mycloud.net cloud,test,web
web1.mycloud.net cloud,web
web2.mycloud.net cloud,web
shell1.mycloud.net cloud,shell
shell2.mycloud.net cloud,shell
$ python3 scripts/cdist inventory add-tag -b -t all
$ python3 scripts/cdist inventory add-tag -b -t mistake
$ python3 scripts/cdist inventory list -b
localhost all,cloud,mistake
test.mycloud.net all,cloud,mistake,test,web
web1.mycloud.net all,cloud,mistake,web
web2.mycloud.net all,cloud,mistake,web
shell1.mycloud.net all,cloud,mistake,shell
shell2.mycloud.net all,cloud,mistake,shell
$ python3 scripts/cdist inventory del-tag -b -t mistake
$ python3 scripts/cdist inventory list -b
localhost all,cloud
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
shell1.mycloud.net all,cloud,shell
shell2.mycloud.net all,cloud,shell
$ python3 scripts/cdist inventory del-host -b localhost
$ python3 scripts/cdist inventory list -b
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
shell1.mycloud.net all,cloud,shell
shell2.mycloud.net all,cloud,shell
$ python3 scripts/cdist inventory list -b -t web
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
$ python3 scripts/cdist inventory list -b -t -a web test
test.mycloud.net all,cloud,test,web
$ python3 scripts/cdist inventory list -b -t -a web all
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
$ python3 scripts/cdist inventory list -b -t web all
test.mycloud.net all,cloud,test,web
web1.mycloud.net all,cloud,web
web2.mycloud.net all,cloud,web
shell1.mycloud.net all,cloud,shell
shell2.mycloud.net all,cloud,shell
$ cd cdist/inventory
$ ls -1
shell1.mycloud.net
shell2.mycloud.net
test.mycloud.net
web1.mycloud.net
web2.mycloud.net
$ ls -l
total 20
-rw-r--r-- 1 darko darko 16 Jun 24 12:43 shell1.mycloud.net
-rw-r--r-- 1 darko darko 16 Jun 24 12:43 shell2.mycloud.net
-rw-r--r-- 1 darko darko 19 Jun 24 12:43 test.mycloud.net
-rw-r--r-- 1 darko darko 14 Jun 24 12:43 web1.mycloud.net
-rw-r--r-- 1 darko darko 14 Jun 24 12:43 web2.mycloud.net
$ cat test.mycloud.net
test
all
web
cloud
$ cat web2.mycloud.net
all
web
cloud
For more info about inventory commands and options see `cdist <man1/cdist.html>`_\ (1).
Using external inventory
------------------------
cdist can be used with any external inventory where external inventory is
some storage or database from which you can get a list of hosts to configure.
cdist can then be fed with this list of hosts through stdin or file using
**-f** option. For example, if your host list is stored in sqlite3 database
hosts.db and you want to select hosts which purpose is **django** then you
can use it with cdist like:
.. code-block:: sh
$ sqlite3 hosts.db "select hostname from hosts where purpose = 'django';" | cdist config

View file

@ -63,6 +63,10 @@ cdist/conf/
The distribution configuration directory.
This contains types and explorers to be used.
cdist/inventory/
The distribution inventory directory.
This path is relative to cdist installation directory.
confdir
Cdist will use all available configuration directories and create
a temporary confdir containing links to the real configuration directories.
@ -239,6 +243,9 @@ __target_fqdn
This variable is derived from **__target_host**
(using **socket.getfqdn()**).
Available for: explorer, initial manifest, type explorer, type manifest, type gencode, shell.
__target_host_tags
Comma separated list of target host tags.
Available for: explorer, initial manifest, type explorer, type manifest, type gencode, shell.
__type
Path to the current type.
Available for: type manifest, type gencode.
@ -274,6 +281,9 @@ CDIST_REMOTE_EXEC
CDIST_REMOTE_COPY
Use this command for remote copy (should behave like scp).
CDIST_INVENTORY_DIR
Use this directory as inventory directory.
CDIST_BETA
Enable beta functionalities.

View file

@ -24,6 +24,7 @@ Contents:
cdist-explorer
cdist-messaging
cdist-parallelization
cdist-inventory
cdist-reference
cdist-best-practice
cdist-stages

View file

@ -11,21 +11,46 @@ SYNOPSIS
::
cdist [-h] [-q] [-v] [-V] {banner,config,shell,install} ...
cdist [-h] [-q] [-v] [-V] {banner,config,install,inventory,shell} ...
cdist banner [-h] [-q] [-v]
cdist config [-h] [-q] [-v] [-b] [-C CACHE_PATH_PATTERN] [-c CONF_DIR]
[-i MANIFEST] [-j [JOBS]] [-n] [-o OUT_PATH]
[--remote-copy REMOTE_COPY] [--remote-exec REMOTE_EXEC]
[-f HOSTFILE] [-p] [-r REMOTE_OUT_PATH] [-s]
[host [host ...]]
[-r REMOTE_OUT_DIR] [--remote-copy REMOTE_COPY]
[--remote-exec REMOTE_EXEC] [-I INVENTORY_DIR] [-A] [-a]
[-f HOSTFILE] [-p] [-s] [-t]
[host [host ...]]
cdist install [-h] [-q] [-v] [-b] [-C CACHE_PATH_PATTERN] [-c CONF_DIR]
[-i MANIFEST] [-j [JOBS]] [-n] [-o OUT_PATH]
[--remote-copy REMOTE_COPY] [--remote-exec REMOTE_EXEC]
[-f HOSTFILE] [-p] [-r REMOTE_OUT_PATH] [-s]
[host [host ...]]
[-r REMOTE_OUT_DIR] [--remote-copy REMOTE_COPY]
[--remote-exec REMOTE_EXEC] [-I INVENTORY_DIR] [-A] [-a]
[-f HOSTFILE] [-p] [-s] [-t]
[host [host ...]]
cdist inventory [-h] [-q] [-v] [-b] [-I INVENTORY_DIR]
{add-host,add-tag,del-host,del-tag,list} ...
cdist inventory add-host [-h] [-q] [-v] [-b] [-I INVENTORY_DIR]
[-f HOSTFILE]
[host [host ...]]
cdist inventory add-tag [-h] [-q] [-v] [-b] [-I INVENTORY_DIR]
[-f HOSTFILE] [-T TAGFILE] [-t TAGLIST]
[host [host ...]]
cdist inventory del-host [-h] [-q] [-v] [-b] [-I INVENTORY_DIR] [-a]
[-f HOSTFILE]
[host [host ...]]
cdist inventory del-tag [-h] [-q] [-v] [-b] [-I INVENTORY_DIR] [-a]
[-f HOSTFILE] [-T TAGFILE] [-t TAGLIST]
[host [host ...]]
cdist inventory list [-h] [-q] [-v] [-b] [-I INVENTORY_DIR] [-a]
[-f HOSTFILE] [-H] [-t]
[host [host ...]]
cdist shell [-h] [-q] [-v] [-s SHELL]
@ -72,6 +97,15 @@ CONFIG/INSTALL
--------------
Configure/install one or more hosts.
.. option:: -A, --all-tagged
use all hosts present in tags db
.. option:: -a, --all
list hosts that have all specified tags, if -t/--tag
is specified
.. option:: -b, --beta
Enable beta functionality.
@ -103,6 +137,16 @@ Configure/install one or more hosts.
read hosts from stdin. For the file format see
:strong:`HOSTFILE FORMAT` below.
.. option:: -I INVENTORY_DIR, --inventory INVENTORY_DIR
Use specified custom inventory directory. Inventory
directory is set up by the following rules: if this
argument is set then specified directory is used, if
CDIST_INVENTORY_DIR env var is set then its value is
used, if HOME env var is set then ~/.cdit/inventory is
used, otherwise distribution inventory directory is
used.
.. option:: -i MANIFEST, --initial-manifest MANIFEST
Path to a cdist manifest or - to read from stdin
@ -141,6 +185,10 @@ Configure/install one or more hosts.
Command to use for remote execution (should behave like ssh)
.. option:: -t, --tag
host is specified by tag, not hostname/address; list
all hosts that contain any of specified tags
HOSTFILE FORMAT
~~~~~~~~~~~~~~~
@ -174,6 +222,246 @@ Resulting path is used to specify cache path subdirectory under which
current host cache data are saved.
INVENTORY
---------
Manage inventory database.
Currently in beta with all sub-commands.
INVENTORY ADD-HOST
------------------
Add host(s) to inventory database.
.. option:: host
host(s) to add
.. option:: -b, --beta
Enable beta functionalities. Beta functionalities
include inventory command with all sub-commands and
all options; config sub-command options: -j/--jobs,
-t/--tag, -a/--all.
Can also be enabled using CDIST_BETA env var.
.. option:: -f HOSTFILE, --file HOSTFILE
Read additional hosts to add from specified file or
from stdin if '-' (each host on separate line). If no
host or host file is specified then, by default, read
from stdin. Hostfile format is the same as config hostfile format.
.. option:: -h, --help
show this help message and exit
.. option:: -I INVENTORY_DIR, --inventory INVENTORY_DIR
Use specified custom inventory directory. Inventory
directory is set up by the following rules: if this
argument is set then specified directory is used, if
CDIST_INVENTORY_DIR env var is set then its value is
used, if HOME env var is set then ~/.cdist/inventory is
used, otherwise distribution inventory directory is
used.
INVENTORY ADD-TAG
-----------------
Add tag(s) to inventory database.
.. option:: host
list of host(s) for which tags are added
.. option:: -b, --beta
Enable beta functionalities. Beta functionalities
include inventory command with all sub-commands and
all options; config sub-command options: -j/--jobs,
-t/--tag, -a/--all.
Can also be enabled using CDIST_BETA env var.
.. option:: -f HOSTFILE, --file HOSTFILE
Read additional hosts to add tags from specified file
or from stdin if '-' (each host on separate line). If
no host or host file is specified then, by default,
read from stdin. If no tags/tagfile nor hosts/hostfile
are specified then tags are read from stdin and are
added to all hosts. Hostfile format is the same as config hostfile format.
.. option:: -I INVENTORY_DIR, --inventory INVENTORY_DIR
Use specified custom inventory directory. Inventory
directory is set up by the following rules: if this
argument is set then specified directory is used, if
CDIST_INVENTORY_DIR env var is set then its value is
used, if HOME env var is set then ~/.cdist/inventory is
used, otherwise distribution inventory directory is
used.
.. option:: -T TAGFILE, --tag-file TAGFILE
Read additional tags to add from specified file or
from stdin if '-' (each tag on separate line). If no
tag or tag file is specified then, by default, read
from stdin. If no tags/tagfile nor hosts/hostfile are
specified then tags are read from stdin and are added
to all hosts. Tagfile format is the same as config hostfile format.
.. option:: -t TAGLIST, --taglist TAGLIST
Tag list to be added for specified host(s), comma
separated values
INVENTORY DEL-HOST
------------------
Delete host(s) from inventory database.
.. option:: host
host(s) to delete
.. option:: -a, --all
Delete all hosts
.. option:: -b, --beta
Enable beta functionalities. Beta functionalities
include inventory command with all sub-commands and
all options; config sub-command options: -j/--jobs,
-t/--tag, -a/--all.
Can also be enabled using CDIST_BETA env var.
.. option:: -f HOSTFILE, --file HOSTFILE
Read additional hosts to delete from specified file or
from stdin if '-' (each host on separate line). If no
host or host file is specified then, by default, read
from stdin. Hostfile format is the same as config hostfile format.
.. option:: -I INVENTORY_DIR, --inventory INVENTORY_DIR
Use specified custom inventory directory. Inventory
directory is set up by the following rules: if this
argument is set then specified directory is used, if
CDIST_INVENTORY_DIR env var is set then its value is
used, if HOME env var is set then ~/.cdist/inventory is
used, otherwise distribution inventory directory is
used.
INVENTORY DEL-TAG
-----------------
Delete tag(s) from inventory database.
.. option:: host
list of host(s) for which tags are deleted
.. option:: -a, --all
Delete all tags for specified host(s)
.. option:: -b, --beta
Enable beta functionalities. Beta functionalities
include inventory command with all sub-commands and
all options; config sub-command options: -j/--jobs,
-t/--tag, -a/--all.
Can also be enabled using CDIST_BETA env var.
.. option:: -f HOSTFILE, --file HOSTFILE
Read additional hosts to delete tags for from
specified file or from stdin if '-' (each host on
separate line). If no host or host file is specified
then, by default, read from stdin. If no tags/tagfile
nor hosts/hostfile are specified then tags are read
from stdin and are deleted from all hosts. Hostfile
format is the same as config hostfile format.
.. option:: -I INVENTORY_DIR, --inventory INVENTORY_DIR
Use specified custom inventory directory. Inventory
directory is set up by the following rules: if this
argument is set then specified directory is used, if
CDIST_INVENTORY_DIR env var is set then its value is
used, if HOME env var is set then ~/.cdist/inventory is
used, otherwise distribution inventory directory is
used.
.. option:: -T TAGFILE, --tag-file TAGFILE
Read additional tags from specified file or from stdin
if '-' (each tag on separate line). If no tag or tag
file is specified then, by default, read from stdin.
If no tags/tagfile nor hosts/hostfile are specified
then tags are read from stdin and are added to all
hosts. Tagfile format is the same as config hostfile format.
.. option:: -t TAGLIST, --taglist TAGLIST
Tag list to be deleted for specified host(s), comma
separated values
INVENTORY LIST
--------------
List inventory database.
.. option:: host
host(s) to list
.. option:: -a, --all
list hosts that have all specified tags, if -t/--tag
is specified
.. option:: -b, --beta
Enable beta functionalities. Beta functionalities
include inventory command with all sub-commands and
all options; config sub-command options: -j/--jobs,
-t/--tag, -a/--all.
Can also be enabled using CDIST_BETA env var.
.. option:: -f HOSTFILE, --file HOSTFILE
Read additional hosts to list from specified file or
from stdin if '-' (each host on separate line). If no
host or host file is specified then, by default, list
all. Hostfile format is the same as config hostfile format.
.. option:: -H, --host-only
Suppress tags listing
.. option:: -I INVENTORY_DIR, --inventory INVENTORY_DIR
Use specified custom inventory directory. Inventory
directory is set up by the following rules: if this
argument is set then specified directory is used, if
CDIST_INVENTORY_DIR env var is set then its value is
used, if HOME env var is set then ~/.cdist/inventory is
used, otherwise distribution inventory directory is
used.
.. option:: -t, --tag
host is specified by tag, not hostname/address; list
all hosts that contain any of specified tags
SHELL
-----
This command allows you to spawn a shell that enables access
@ -186,14 +474,21 @@ usage. Its primary use is for debugging type parameters.
Select shell to use, defaults to current shell. Used shell should
be POSIX compatible shell.
FILES
-----
~/.cdist
Your personal cdist config directory. If exists it will be
automatically used.
~/.cdist/inventory
The home inventory directory. If ~/.cdist exists it will be used as
default inventory directory.
cdist/conf
The distribution configuration directory. It contains official types and
explorers. This path is relative to cdist installation directory.
cdist/inventory
The distribution inventory directory.
This path is relative to cdist installation directory.
NOTES
-----
@ -243,6 +538,43 @@ EXAMPLES
# Install ikq05.ethz.ch with debug enabled
% cdist install -vvv ikq05.ethz.ch
# List inventory content
% cdist inventory list -b
# List inventory for specified host localhost
% cdist inventory list -b localhost
# List inventory for specified tag loadbalancer
% cdist inventory list -b -t loadbalancer
# Add hosts to inventory
% cdist inventory add-host -b web1 web2 web3
# Delete hosts from file old-hosts from inventory
% cdist inventory del-host -b -f old-hosts
# Add tags to specifed hosts
% cdist inventory add-tag -b -t europe,croatia,web,static web1 web2
# Add tag to all hosts in inventory
% cdist inventory add-tag -b -t vm
# Delete all tags from specified host
% cdist inventory del-tag -b -a localhost
# Delete tags read from stdin from hosts specified by file hosts
% cdist inventory del-tag -b -T - -f hosts
# Configure hosts from inventory with any of specified tags
% cdist config -b -t web dynamic
# Configure hosts from inventory with all specified tags
% cdist config -b -t -a web dynamic
# Configure all hosts from inventory db
$ cdist config -b -A
ENVIRONMENT
-----------
TMPDIR, TEMP, TMP
@ -272,6 +604,9 @@ CDIST_REMOTE_EXEC
CDIST_REMOTE_COPY
Use this command for remote copy (should behave like scp).
CDIST_INVENTORY_DIR
Use this directory as inventory directory.
CDIST_BETA
Enable beta functionality.