Support installation from Git using Pip #83

New Issue

ungleich-gitea · 2021-11-20T13:23:39Z

ungleich-gitea commented

2021-11-20 13:23:39 +00:00

Currently, if you try to install cdist from Git using Pip, the installation will fail because setup.py uses cdist.version module, which doesn't exist at the time of installation.

Currently, if you try to install cdist from Git using Pip, the installation will fail because `setup.py` uses `cdist.version` module, which doesn't exist at the time of installation.

ungleich-gitea closed this issue

2021-11-20 13:23:39 +00:00

ungleich-gitea commented

2021-11-20 15:35:48 +00:00

closed

ungleich-gitea commented

2021-11-20 15:35:50 +00:00

https://code.ungleich.ch/ungleich-public/cdist/merge_requests/808

ungleich-gitea commented

2021-11-20 15:35:51 +00:00

Okay. So, as I've previously reported, 2d0af7b7cc works for me. Therefore, I'd say ship it and let's leave the issue with version reported by cdist for another time.

Okay. So, as I've previously reported, 2d0af7b7ccc34d450a1b0cfb6532f6d2439ee7a4 works for me. Therefore, I'd say ship it and let's leave the issue with version reported by cdist for another time.

ungleich-gitea commented

2021-11-20 15:35:52 +00:00

@lubo I agree with @steven. Putting setuptools_scm into cdist is heavy.
It seems, currently, that https://code.ungleich.ch/ungleich-public/cdist/tree/build/support-pip-from-git is best effort for this issue.

@lubo I agree with @steven. Putting setuptools_scm into cdist is heavy. It seems, currently, that https://code.ungleich.ch/ungleich-public/cdist/tree/build/support-pip-from-git is best effort for this issue.

ungleich-gitea commented

2021-11-20 15:35:53 +00:00

So, what's it gonna be?

ungleich-gitea commented

2021-11-20 15:35:55 +00:00

I always hated the way cdist version is handled. That's actually the sole reason why I run from a fork which I rebase on top of master with this single commit:

[10:59:54] eos:cdist% git show c68cf3b4           
commit c68cf3b42eac8f8848e858c5733eb5ef77964ce7 (HEAD -> production, asteven/production)
Author: Steven Armstrong <steven@icarus.ethz.ch>
Date:   Tue Sep 24 12:00:36 2019 +0200

    ensure cdist.version
    
    Signed-off-by: Steven Armstrong <steven@icarus.ethz.ch>

diff --git a/cdist/version.py b/cdist/version.py
new file mode 100644
index 00000000..ca39d5ba
--- /dev/null
+++ b/cdist/version.py
@@ -0,0 +1 @@
+VERSION = "9.9.9"
[10:59:59] eos:cdist%

I do not care about the cdist version when running from a git checkout. Actually I don't care about the cdist version at all, ever.

IMHO the question is who/when cares about the cdist version?

the developers when someone is reporting a problem
the user if he wants to check which version he has installed

I would just hardcode the version in the git repo to 'git' or 'dev' or whatever and only set the actual version during packaging.

If the user is running from a git checkout git describe tells you the details if needed.

Pulling in setuptools_scm seems a bit heavy weight IMHO. Guess it would be simpler to shell out.

I always hated the way cdist version is handled. That's actually the sole reason why I run from a fork which I rebase on top of master with this single commit: ``` [10:59:54] eos:cdist% git show c68cf3b4 commit c68cf3b42eac8f8848e858c5733eb5ef77964ce7 (HEAD -> production, asteven/production) Author: Steven Armstrong <steven@icarus.ethz.ch> Date: Tue Sep 24 12:00:36 2019 +0200 ensure cdist.version Signed-off-by: Steven Armstrong <steven@icarus.ethz.ch> diff --git a/cdist/version.py b/cdist/version.py new file mode 100644 index 00000000..ca39d5ba --- /dev/null +++ b/cdist/version.py @@ -0,0 +1 @@ +VERSION = "9.9.9" [10:59:59] eos:cdist% ``` I do not care about the cdist version when running from a git checkout. Actually I don't care about the cdist version at all, ever. IMHO the question is who/when cares about the cdist version? - the developers when someone is reporting a problem - the user if he wants to check which version he has installed I would just hardcode the version in the git repo to 'git' or 'dev' or whatever and only set the actual version during packaging. If the user is running from a git checkout `git describe` tells you the details if needed. Pulling in setuptools_scm seems a bit heavy weight IMHO. Guess it would be simpler to shell out.

ungleich-gitea commented

2021-11-20 15:35:56 +00:00

@nico @steven Do we want this coupling?

ungleich-gitea commented

2021-11-20 15:35:57 +00:00

You'd have to put the same code to cdist/__init__.py too, in addition to setup.py. I think "Package versioning best practices" describes this approach quite well. So, instead of this:

import cdist.version

VERSION = cdist.version.VERSION

Something like this:

try:
    import cdist.version

    VERSION = cdist.version.VERSION
except ImportError:
    VERSION =  # The same code that's used in setup.py

So, when installed in editable mode, cdist/version.py shouldn't be generated and the app itself should obtain the version string if cdist/version.py is not found.

You'd have to put the same code to `cdist/__init__.py` too, in addition to `setup.py`. I think _"Package versioning best practices"_ describes this approach quite well. So, instead of this: ```python import cdist.version VERSION = cdist.version.VERSION ``` Something like this: ```python try: import cdist.version VERSION = cdist.version.VERSION except ImportError: VERSION = # The same code that's used in setup.py ``` So, when installed in editable mode, `cdist/version.py` shouldn't be generated and the app itself should obtain the version string if `cdist/version.py` is not found.

ungleich-gitea commented

2021-11-20 15:35:59 +00:00

@lubo As I understand, editable mode does some sort of link to source/cloned repo. So if you change something that will reflect package execution.
If you change something, then you expect the version to change, i.e. to be regenerated by git describe?
I think this could not be done automatically with pip, you have to regenerate it by yourself by executing ./bin/build-helper version.

Or I don't understand it (or the problem) properly?

@lubo As I understand, editable mode does some sort of link to source/cloned repo. So if you change something that will reflect package execution. If you change something, then you expect the version to change, i.e. to be regenerated by `git describe`? I think this could not be done automatically with `pip`, you have to regenerate it by yourself by executing `./bin/build-helper version`. Or I don't understand it (or the problem) properly?

ungleich-gitea commented

2021-11-20 15:36:00 +00:00

Maybe you should put version.py to build/lib/cdist, so it's used when building distributable packages, but not locally. Then, if importing version module fails, assume we run in editable mode and run a script that obtains the version string. Much like they do in getversion - Package versioning best practices, which I've mentioned in #783 (comment).

Maybe you should put `version.py` to `build/lib/cdist`, so it's used when building distributable packages, but not locally. Then, if importing `version` module fails, assume we run in editable mode and run a script that obtains the version string. Much like they do in [getversion - Package versioning best practices](https://smarie.github.io/python-getversion/#package-versioning-best-practices), which I've mentioned in https://code.ungleich.ch/ungleich-public/cdist/issues/783#note_10342.

ungleich-gitea commented

2021-11-20 15:36:02 +00:00

@lubo Do you have suggestions for "Although you'll have to reinstall the package if you change something in the editable mode." ?

ungleich-gitea commented

2021-11-20 15:36:03 +00:00

Although you'll have to reinstall the package if you change something in the editable mode.

ungleich-gitea commented

2021-11-20 15:36:04 +00:00

2d0af7b7cc works when installed from local and remote repository and from generated sdist too.

2d0af7b7ccc34d450a1b0cfb6532f6d2439ee7a4 works when installed from local and remote repository and from generated sdist too.

ungleich-gitea commented

2021-11-20 15:36:06 +00:00

@lubo Can you please test it? Thanks!

ungleich-gitea commented

2021-11-20 15:36:07 +00:00

@lubo This https://code.ungleich.ch/ungleich-public/cdist/blob/build/support-pip-from-git/setup.py should cover git and non git cases.

ungleich-gitea commented

2021-11-20 15:36:08 +00:00

@lubo Well, I have to play with it. It must support different cases.
pip from git, git clone + setup, download tar.gz from pypi or gitlab release bundle when it is not a git repo, ...

@lubo Well, I have to play with it. It must support different cases. pip from git, git clone + setup, download tar.gz from pypi or gitlab release bundle when it is not a git repo, ...

ungleich-gitea commented

2021-11-20 15:36:09 +00:00

With fc28f58c77, this finally works:

$ pipenv install -e git+https://code.ungleich.ch/ungleich-public/cdist.git@fc28f58c77080e94215719ba4f2aa3f3162c8300#egg=cdist

But consider cdist is installed in editable mode from locally-cloned repository (this is how I actually use cdist). If I change something in the repository (e.g., check out different tag), and reinstall the package, version.py won't change with fc28f58c77. Steps to reproduce:

$ git clone -b build/support-pip-from-git https://code.ungleich.ch/ungleich-public/cdist.git vendor/cdist
$ pipenv install -e ./vendor/cdist
$ pipenv run cdist -V
cdist 6.0.2-1-gfc28f58c
$ cd vendor/cdist
$ git checkout 6.0.3  # Provided 6.0.3 contains fc28f58c77080e94215719ba4f2aa3f3162c8300
$ cd -
$ pipenv install
$ pipenv run cdist -V
cdist 6.0.2-1-gfc28f58c  # Expect output is `cdist 6.0.3`

With fc28f58c77080e94215719ba4f2aa3f3162c8300, this finally works: ```shell $ pipenv install -e git+https://code.ungleich.ch/ungleich-public/cdist.git@fc28f58c77080e94215719ba4f2aa3f3162c8300#egg=cdist ``` But consider cdist is installed in editable mode from locally-cloned repository (this is how I actually use cdist). If I change something in the repository (e.g., check out different tag), and reinstall the package, `version.py` won't change with fc28f58c77080e94215719ba4f2aa3f3162c8300. Steps to reproduce: ```shell $ git clone -b build/support-pip-from-git https://code.ungleich.ch/ungleich-public/cdist.git vendor/cdist $ pipenv install -e ./vendor/cdist $ pipenv run cdist -V cdist 6.0.2-1-gfc28f58c $ cd vendor/cdist $ git checkout 6.0.3 # Provided 6.0.3 contains fc28f58c77080e94215719ba4f2aa3f3162c8300 $ cd - $ pipenv install $ pipenv run cdist -V cdist 6.0.2-1-gfc28f58c # Expect output is `cdist 6.0.3` ```

ungleich-gitea commented

2021-11-20 15:36:11 +00:00

@lubo @nico What do you think about this simple patch?
fc28f58c77

@lubo @nico What do you think about this simple patch? https://code.ungleich.ch/ungleich-public/cdist/commit/fc28f58c77080e94215719ba4f2aa3f3162c8300

ungleich-gitea commented

2021-11-20 15:36:12 +00:00

Well, if you wanna do it yourself, do something like this (but with setuptools, I don't have a more up-to-date article right now) and generate the file in setup.py. Generally, this is what should be done when in need of generating files that should be packaged with your application. However, I'd recommend using setuptools_scm or something similar and do something like this. No more Make, no more custom shell scripts, just a very simple, more declarative code which should work with all installation scenarios.

P.S.: I get literally no email notifications from this GitLab instance. 😕

Well, if you wanna do it yourself, do something like [this](http://www.digip.org/blog/2011/01/generating-data-files-in-setup.py.html) (but with `setuptools`, I don't have a more up-to-date article right now) and generate the file in `setup.py`. Generally, this is what should be done when in need of generating files that should be packaged with your application. However, I'd recommend using [setuptools_scm](https://github.com/pypa/setuptools_scm) or something similar and do something like [this](https://smarie.github.io/python-getversion/#package-versioning-best-practices). No more Make, no more custom shell scripts, just a very simple, more declarative code which should work with all installation scenarios. P.S.: I get literally no email notifications from this GitLab instance. :confused:

ungleich-gitea commented

2021-11-20 15:36:13 +00:00

@lubo This is because cdist uses generated version file. After cloning and before runningo python setup.py install one should create version file with make version.
https://www.cdi.st/cdist-install.html#from-git

Do you perhaps know how to hook make version command into pip after cloning a repo?

@lubo This is because cdist uses generated version file. After cloning and before runningo `python setup.py install` one should create version file with `make version`. https://www.cdi.st/cdist-install.html#from-git Do you perhaps know how to hook `make version` command into pip after cloning a repo?

Sign in to join this conversation.