diff --git a/blog/cdist-python-3.2-requirement.mdwn b/blog/cdist-python-3.2-requirement.mdwn new file mode 100644 index 00000000..0432fd60 --- /dev/null +++ b/blog/cdist-python-3.2-requirement.mdwn @@ -0,0 +1,173 @@ +[[!meta title="cdist: Why we require Python 3.2 on the source host"]] + +## Introduction + +As [[cdist|software/cdist]] is getting more popular, more people are using +cdist and some questions arrive more often from newbies than others. +One of them is why we require having Python 3.2 on the source host. +If you are also wondering about the motivation, or you're screaming because +you only have Python 2.x or Python 3.1 available in your distribution, this +article is for you. + + Note: Cdist does *not* require Python on the target hosts! + Note: Cdist requires only ssh and a posix shell on the targets. + + +## History of cdist + +Because you may be one of the people who are screaming, I'm giving you +an overview about the whole development history of cdist, which will +make things more clear for you. + +At the end of 2010 [I was claiming that most current configuration management +(***CM***) tools are not only overly complex designed, but also that their +implementations are way too +complex](http://www.usenix.org/event/lisa10/tech/slides/snyder_bof2.pdf). +This is definitely a strong statement, which I +also used to provoke people to thing about the current situation of CM. + +The logical consequence of my statement was to try out, whether it's +actually possible to write a CM tool completly in a very simple manner. +For instance in posix shell script. This led to the first commit to the +newly born cdist repository: + + commit 28a9807fe5e6bfa95015fe72456d63cbb2a5821f + Date: Thu Sep 16 02:20:35 2010 +0200 + +After a lot of discussions, design ideas, pictures on the whiteboard, +trying out different implementations, weighting up advantages of +each one, the first official release of cdist was put into public, +cdist 1.0.0: + + tag 1.0.0 + Date: Mon Mar 7 18:21:18 2011 +0100 + +Cdist 1.0.0 is completly implemented in posix shell and had almost all +features of the current cdist implementation. With one major drawback: +Performance. When running cdist 1.0.0 in parallel mode, the source host +easily became the bottleneck. A typical run of cdist 1.0.0 caused around +3500 - 5000 forks. Running in parallel mode with about 10-15 target hosts, +most time of a cdist run was spent in kernel space to handle the forks. + +The logical sequence again was to search for the reason for the huge amount +of forks, which was easily detected: Every routine was a shell script on its +own, that required a new launch of the shell. Now, for some operations, +like working on the dependency tree, a lot of sub-routines were called, leading +to the huge number of forks. + +We tried to minimise the number of forks by migrating from shell scripts to +shell functions, which was a big pain, when we realised that posix shell +does not have **local** variable support anymore. Posix states that you should +use shell scripts instead of shell functions, if you need distinct environments. +Which is exactly were we came from... + +Thus we decided to switch to a different programming language for cdist's core, +but only for the core. For us it is very important to minimise the dependencies +on the target hosts: We do not want to install Ruby, Python, Java, libfoo or +anything that is not usually available on a freshly installed base system. +Cdist should be able to take over, as soon as the system is setup in a very +basic state. + +The choice felt on Python, because it felt very mature and easy to use. +Additionally, Python 3 already provides a lot of functionality in its +base installation: Everything we were used to do in shell, could easily be +rewritten in plain Python 3. After **exactly** one year after +the initial commit, ***cdist 2.0.0***, a complete rewrite in Python 3 +was finished and released: + + tag 2.0.0 + Date: Fri Sep 16 12:11:28 2011 +0200 + + +## Now, why Python 3.2? + +During the development of cdist 2.0, we had a lot of discussions +about clean design, pythonic ways of doing stuff (versus using the +shell approach in python) and which functions to use. In the beginning, +we were discussing about whether to settle for Python 2 or Python 3. +As we did not have any dependencies or old code that relies on Python 2, +the choice for the current stable tree, Python 3, was easy to make. + +Python 3.2, in contrast to Python 3.1, includes the very good usable +[argparse module](http://docs.python.org/py3k/library/argparse.html), +as well as an enhanced variant of the +[os.makedirs](http://docs.python.org/py3k/library/os.html#os.makedirs) +method that supports the ***exist_ok*** parameter. + +The argparse module is also available for Python 3.1, but not the +enhanced **os.makedirs** method. So we had to decide: Should we +integrate a simple workaround to support the previous Python 3 release, +Python 3.1, or shall we upset users with old Python installations? + +To answer this question, we had a look at the current situation of +Python in various distributions. + +## Python support in Linux/BSD + +A very quick research showed the following results: + +[[!table data=""" +Distro | Version | Python version | Python 3? | Usable as cdist source host? +Archlinux | rolling release | 3.2.2 | yes | yes +CentOS | 6 | 2.6.6 | no | no +Debian | squeeze | 3.1.3 | yes | no +Debian | wheezy | 3.2.2 | yes | yes +Fedora | 14 | 3.1.2 | yes | no +Fedora | 15-17 | 3.2 - 3.2.2 | yes | yes +FreeBSD | Ports | 3.2.2 | yes | yes +Gentoo | rolling release | 3.2.2 | yes | yes +NetBSD | Ports | 3.1.4 | yes | no +OpenBSD: | -current | 2.7.1 | no | no +OpenBSD: | Ports | 3.2.2 | yes | yes +OpenSuse | 11.4 | 3.1.3 | yes | no +OpenSuse | 12.1 | 3.2.1 | yes | yes +Redhat | 6 | 2.6.6 | no | no +Slackware | 13.37 | 2.6.6 | no | no +Ubuntu | maverick (10.10) | 3.1.3 | yes | no +Ubuntu | natty (11.04) - precise (12.04) | 3.2 - 3.2.2 | yes | yes +"""]] + +So we have the following situations: + + * There are a lot of distros with Python 3.2 available (Archlinux, + Debian >= Wheezy, Fedora >= 15, FreeBSD, Gentoo, OpenBSD, OpenSuse, + Ubuntu >= 11.04) + * There are distros which do not have Python 3 at all (Centos, Redhat, Slackware) + * Python 3 is definitely needed, so requiring 3.1 or 3.2 + does not make a difference + * There are only two cases, which would make it interesting to support + Python 3.1: Debian Squeeze (currently stable) and NetBSD. + * As I've been a long time Debian user, I understand this may be a bit + annoying, because it's unclear, when wheezy will become stable. + On the other hand, you can easily install python 3.2 from source to + anywhere, like you'd do in the situation, when you wouldn't have + python 3 at all. + * Another point speaking against 3.1 support for Debian is the fact that + distributions should support the user and not developers should support + distributions that ship old software (there's nothing against supporting + old **and** new versions, though). That's by the way one of the reasons + why I moved away from Debian... + * I am short on experience regarding NetBSD, but installing via source + shouldn't be an issue either. + +To summarise: Support for Python 3.1 only makes sense for Debian Squeeze +and NetBSD at the moment. This requirement will soon [tm] be superseeded +and can easily be worked around by selecting one of many distributions +with more recent software packages. And that's the reason, why we settled +for Python 3.2. + +## Btw, who is we? + +You mave have noticed that I am often referring to **we** in this article. +The second main developer for cdist is +[Steven Armstrong](https://github.com/asteven), a sysadmin at +ETH Zurich and good friend of mine. +The discussions and development time we spent together was very valuable +for me as well for the whole cdist project and thus I wanted to use this +article to say + + Thank you, Steven. + +[Disclaimer: I do not work for ETH Zurich anymore, but for [local.ch](http://www.local.ch)] + +[[!tag config sysadmin unix]]