add cdist/python 3.2 blog article
Signed-off-by: Nico Schottelius <nico@brief.schottelius.org>
This commit is contained in:
Nico Schottelius
parent
4403edfdc7
commit
5bd14244c2
|
|
@ -0,0 +1,173 @@
|
|||
[[!meta title="cdist: Why we require Python 3.2 on the source host"]]
|
||||
|
||||
## Introduction
|
||||
|
||||
As [[cdist|software/cdist]] is getting more popular, more people are using
|
||||
cdist and some questions arrive more often from newbies than others.
|
||||
One of them is why we require having Python 3.2 on the source host.
|
||||
If you are also wondering about the motivation, or you're screaming because
|
||||
you only have Python 2.x or Python 3.1 available in your distribution, this
|
||||
article is for you.
|
||||
|
||||
Note: Cdist does *not* require Python on the target hosts!
|
||||
Note: Cdist requires only ssh and a posix shell on the targets.
|
||||
|
||||
|
||||
## History of cdist
|
||||
|
||||
Because you may be one of the people who are screaming, I'm giving you
|
||||
an overview about the whole development history of cdist, which will
|
||||
make things more clear for you.
|
||||
|
||||
At the end of 2010 [I was claiming that most current configuration management
|
||||
(***CM***) tools are not only overly complex designed, but also that their
|
||||
implementations are way too
|
||||
complex](http://www.usenix.org/event/lisa10/tech/slides/snyder_bof2.pdf).
|
||||
This is definitely a strong statement, which I
|
||||
also used to provoke people to thing about the current situation of CM.
|
||||
|
||||
The logical consequence of my statement was to try out, whether it's
|
||||
actually possible to write a CM tool completly in a very simple manner.
|
||||
For instance in posix shell script. This led to the first commit to the
|
||||
newly born cdist repository:
|
||||
|
||||
commit 28a9807fe5e6bfa95015fe72456d63cbb2a5821f
|
||||
Date: Thu Sep 16 02:20:35 2010 +0200
|
||||
|
||||
After a lot of discussions, design ideas, pictures on the whiteboard,
|
||||
trying out different implementations, weighting up advantages of
|
||||
each one, the first official release of cdist was put into public,
|
||||
cdist 1.0.0:
|
||||
|
||||
tag 1.0.0
|
||||
Date: Mon Mar 7 18:21:18 2011 +0100
|
||||
|
||||
Cdist 1.0.0 is completly implemented in posix shell and had almost all
|
||||
features of the current cdist implementation. With one major drawback:
|
||||
Performance. When running cdist 1.0.0 in parallel mode, the source host
|
||||
easily became the bottleneck. A typical run of cdist 1.0.0 caused around
|
||||
3500 - 5000 forks. Running in parallel mode with about 10-15 target hosts,
|
||||
most time of a cdist run was spent in kernel space to handle the forks.
|
||||
|
||||
The logical sequence again was to search for the reason for the huge amount
|
||||
of forks, which was easily detected: Every routine was a shell script on its
|
||||
own, that required a new launch of the shell. Now, for some operations,
|
||||
like working on the dependency tree, a lot of sub-routines were called, leading
|
||||
to the huge number of forks.
|
||||
|
||||
We tried to minimise the number of forks by migrating from shell scripts to
|
||||
shell functions, which was a big pain, when we realised that posix shell
|
||||
does not have **local** variable support anymore. Posix states that you should
|
||||
use shell scripts instead of shell functions, if you need distinct environments.
|
||||
Which is exactly were we came from...
|
||||
|
||||
Thus we decided to switch to a different programming language for cdist's core,
|
||||
but only for the core. For us it is very important to minimise the dependencies
|
||||
on the target hosts: We do not want to install Ruby, Python, Java, libfoo or
|
||||
anything that is not usually available on a freshly installed base system.
|
||||
Cdist should be able to take over, as soon as the system is setup in a very
|
||||
basic state.
|
||||
|
||||
The choice felt on Python, because it felt very mature and easy to use.
|
||||
Additionally, Python 3 already provides a lot of functionality in its
|
||||
base installation: Everything we were used to do in shell, could easily be
|
||||
rewritten in plain Python 3. After **exactly** one year after
|
||||
the initial commit, ***cdist 2.0.0***, a complete rewrite in Python 3
|
||||
was finished and released:
|
||||
|
||||
tag 2.0.0
|
||||
Date: Fri Sep 16 12:11:28 2011 +0200
|
||||
|
||||
|
||||
## Now, why Python 3.2?
|
||||
|
||||
During the development of cdist 2.0, we had a lot of discussions
|
||||
about clean design, pythonic ways of doing stuff (versus using the
|
||||
shell approach in python) and which functions to use. In the beginning,
|
||||
we were discussing about whether to settle for Python 2 or Python 3.
|
||||
As we did not have any dependencies or old code that relies on Python 2,
|
||||
the choice for the current stable tree, Python 3, was easy to make.
|
||||
|
||||
Python 3.2, in contrast to Python 3.1, includes the very good usable
|
||||
[argparse module](http://docs.python.org/py3k/library/argparse.html),
|
||||
as well as an enhanced variant of the
|
||||
[os.makedirs](http://docs.python.org/py3k/library/os.html#os.makedirs)
|
||||
method that supports the ***exist_ok*** parameter.
|
||||
|
||||
The argparse module is also available for Python 3.1, but not the
|
||||
enhanced **os.makedirs** method. So we had to decide: Should we
|
||||
integrate a simple workaround to support the previous Python 3 release,
|
||||
Python 3.1, or shall we upset users with old Python installations?
|
||||
|
||||
To answer this question, we had a look at the current situation of
|
||||
Python in various distributions.
|
||||
|
||||
## Python support in Linux/BSD
|
||||
|
||||
A very quick research showed the following results:
|
||||
|
||||
[[!table data="""
|
||||
Distro | Version | Python version | Python 3? | Usable as cdist source host?
|
||||
Archlinux | rolling release | 3.2.2 | yes | yes
|
||||
CentOS | 6 | 2.6.6 | no | no
|
||||
Debian | squeeze | 3.1.3 | yes | no
|
||||
Debian | wheezy | 3.2.2 | yes | yes
|
||||
Fedora | 14 | 3.1.2 | yes | no
|
||||
Fedora | 15-17 | 3.2 - 3.2.2 | yes | yes
|
||||
FreeBSD | Ports | 3.2.2 | yes | yes
|
||||
Gentoo | rolling release | 3.2.2 | yes | yes
|
||||
NetBSD | Ports | 3.1.4 | yes | no
|
||||
OpenBSD: | -current | 2.7.1 | no | no
|
||||
OpenBSD: | Ports | 3.2.2 | yes | yes
|
||||
OpenSuse | 11.4 | 3.1.3 | yes | no
|
||||
OpenSuse | 12.1 | 3.2.1 | yes | yes
|
||||
Redhat | 6 | 2.6.6 | no | no
|
||||
Slackware | 13.37 | 2.6.6 | no | no
|
||||
Ubuntu | maverick (10.10) | 3.1.3 | yes | no
|
||||
Ubuntu | natty (11.04) - precise (12.04) | 3.2 - 3.2.2 | yes | yes
|
||||
"""]]
|
||||
|
||||
So we have the following situations:
|
||||
|
||||
* There are a lot of distros with Python 3.2 available (Archlinux,
|
||||
Debian >= Wheezy, Fedora >= 15, FreeBSD, Gentoo, OpenBSD, OpenSuse,
|
||||
Ubuntu >= 11.04)
|
||||
* There are distros which do not have Python 3 at all (Centos, Redhat, Slackware)
|
||||
* Python 3 is definitely needed, so requiring 3.1 or 3.2
|
||||
does not make a difference
|
||||
* There are only two cases, which would make it interesting to support
|
||||
Python 3.1: Debian Squeeze (currently stable) and NetBSD.
|
||||
* As I've been a long time Debian user, I understand this may be a bit
|
||||
annoying, because it's unclear, when wheezy will become stable.
|
||||
On the other hand, you can easily install python 3.2 from source to
|
||||
anywhere, like you'd do in the situation, when you wouldn't have
|
||||
python 3 at all.
|
||||
* Another point speaking against 3.1 support for Debian is the fact that
|
||||
distributions should support the user and not developers should support
|
||||
distributions that ship old software (there's nothing against supporting
|
||||
old **and** new versions, though). That's by the way one of the reasons
|
||||
why I moved away from Debian...
|
||||
* I am short on experience regarding NetBSD, but installing via source
|
||||
shouldn't be an issue either.
|
||||
|
||||
To summarise: Support for Python 3.1 only makes sense for Debian Squeeze
|
||||
and NetBSD at the moment. This requirement will soon [tm] be superseeded
|
||||
and can easily be worked around by selecting one of many distributions
|
||||
with more recent software packages. And that's the reason, why we settled
|
||||
for Python 3.2.
|
||||
|
||||
## Btw, who is we?
|
||||
|
||||
You mave have noticed that I am often referring to **we** in this article.
|
||||
The second main developer for cdist is
|
||||
[Steven Armstrong](https://github.com/asteven), a sysadmin at
|
||||
ETH Zurich and good friend of mine.
|
||||
The discussions and development time we spent together was very valuable
|
||||
for me as well for the whole cdist project and thus I wanted to use this
|
||||
article to say
|
||||
|
||||
Thank you, Steven.
|
||||
|
||||
[Disclaimer: I do not work for ETH Zurich anymore, but for [local.ch](http://www.local.ch)]
|
||||
|
||||
[[!tag config sysadmin unix]]
|
||||
Loading…
Reference in New Issue