pyenv for managing python versions
pyenv tool for managing python versions and environments
The version of python that comes with our Linux OS installations is almost never the python version that you need in order to install some framework you're working with. You may also be working with multiple frameworks, each of which requires a different version of python.
pyenv is a tool to deal with this situation. You can install many different python versions and create virtual environments based on different python versions. pyenv puts its shims directory at the head of your path, and then you tell pyenv at the command line which version you want to work with - the shims do all the work of path changing for you.
The pyenv main page is on GitHub: Main pyenv page on GitHub
Installation
To install, I use homebrew on MacOS; on Linux, I use the installer. Note that pyenv can take up quite a bit of space quickly; to avoid filling my home directory, I put the pyenv root directory on /data:
You need to set up some stuff in your init scripts - see the pyenv main page above for basic information. I use zsh and I want the same script to work on Mac, on the stbc environment and also in the Nikhef grid-node domain, hence my somewhat complicated setup in .zshenv:
HOSTNAME=$(hostname -s)
if [ $(uname) = "Linux" ] && [ -d "/data/datagrid/" ]
then
export PYENV_ROOT="/data/datagrid/templon/pyenv"
fi
if [ ${HOSTNAME} != "korf" ]
then
if [[ -z "${PYENV_ROOT}" ]] then
export PYENV_ROOT="$HOME/.pyenv"
fi
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
fi
The bit for excluding korf is because the OS on that machine is so ancient, that none of this stuff works.
Extra note for Mac users: initialize homebrew before doing the pyenv initialization, to avoid the Homebrew-installed python from overriding your pyenv installations.
Example Usage
Here I go through what is needed (on the Linux command line) in order to create an environment capable of generating the status pages for stoomboot and grid (jupyter and pandas based).
## install not the most recent version, but not too old
$ pyenv install miniforge3-23.3.1-1
## create virtualenv based on it
$ pyenv virtualenv miniforge3-23.3.1-1 stbc
## activate the virtualenv
$ pyenv activate stbc
## install extra stuff into activated virtualenv
$ pip install jupyterlab pandas matplotlib humanize
Then in a shell script (e.g. for execution on stoomboot), I can put stuff like this:
in order to make stuff in that shell script use the virtualenv. pyenv shell tells which python version or virtualenv to use for all further commands in the current shell.
multiple versions
As of this writing, pyenv has access to all sorts of python variants, and python versions ranging from 2.1.3 (oldest) to 3.12.3 (newest). You can have as many different pythons installed under your pyenv installation, as long as you have enough disk space.
which versions to choose
I've used miniforge in the example above. This has the benefit of being relatively OS independent; I can use the same installation on both the older CentOS 7 machines and the newer Alma machines. OTOH it's not immediately apparent from the miniforge version names which python you're getting.
You might want to choose a specific version of python. You can do this, by just selecting the version number (e.g. 3.10.2). What pyenv does in this case is download the python sources and build them for you. This means that you will in general only be able to use this python version on machines with the same OS and architecture as the one on which you ran the pyenv install command.
Limitations
pyenv will likely not work when your environment includes a CVMFS-based python. For example, the CERN SFT releases, when invoked, define several python-related environment variables, leading to inconsistencies.
Who to contact for more information
Helpdesk
Mail: helpdesk@nikhef.nl Office: SP-110 1.11 (between 09:00-17:00 on weekdays) Phone: 2200