pyenv for managing python versions
pyenv tool for managing python versions and environments
The version of python that comes with our Linux OS installations is almost never the python version that you need in order to install some framework you're working with. You may also be working with multiple frameworks, each of which requires a different version of python.
pyenv
is a tool to deal with this situation. You can install many different python versions and create virtual environments based on different python versions. pyenv
puts its shims
directory at the head of your path, and then you tell pyenv
at the command line which version you want to work with - the shims do all the work of path changing for you.
The pyenv
main page is on GitHub: Main pyenv page on GitHub
Installation
To install, I use homebrew
on MacOS; on Linux, I use the installer. Note that pyenv
can take up quite a bit of space quickly; to avoid filling my home directory, I put the pyenv
root directory on /data
:
cd /data/datagrid/templon
curl https://pyenv.run | bash
You need to set up some stuff in your init scripts - see the pyenv
main page above for basic information. I use zsh
and I want the same script to work on Mac, on the stbc
environment and also in the Nikhef grid-node domain, hence my somewhat complicated setup in .zshenv
:
HOSTNAME=$(hostname -s)
if [ $(uname) = "Linux" ] && [ -d "/data/datagrid/" ]
then
export PYENV_ROOT="/data/datagrid/templon/pyenv"
fi
if [ ${HOSTNAME} != "korf" ]
then
if [[ -z "${PYENV_ROOT}" ]] then
export PYENV_ROOT="$HOME/.pyenv"
fi
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
fi
The bit for excluding korf
is because the OS on that machine is so ancient, that none of this stuff works.
Extra note for Mac users: initialize homebrew
before doing the pyenv
initialization, to avoid the Homebrew-installed python from overriding your pyenv
installations.
Example Usage
Here I go through what is needed (on the Linux command line) in order to create an environment capable of generating the status pages for stoomboot and grid (jupyter and pandas based).
## install not the most recent version, but not too old
$ pyenv install miniforge3-23.3.1-1
## create virtualenv based on it
$ pyenv virtualenv miniforge3-23.3.1-1 stbc
## activate the virtualenv
$ pyenv activate stbc
## install extra stuff into activated virtualenv
$ pip install jupyterlab pandas matplotlib humanize
Then in a shell script (e.g. for execution on stoomboot), I can put stuff like this:
pyenv shell stbc
in order to make stuff in that shell script use the virtualenv
. pyenv shell
tells which python version or virtualenv to use for all further commands in the current shell.
multiple versions
As of this writing, pyenv
has access to all sorts of python variants, and python versions ranging from 2.1.3 (oldest) to 3.12.3 (newest). You can have as many different pythons installed under your pyenv
installation, as long as you have enough disk space.
which versions to choose
I've used miniforge
in the example above. This has the benefit of being relatively OS independent; I can use the same installation on both the older CentOS 7 machines and the newer Alma machines. OTOH it's not immediately apparent from the miniforge
version names which python you're getting.
You might want to choose a specific version of python. You can do this, by just selecting the version number (e.g. 3.10.2
). What pyenv
does in this case is download the python sources and build them for you. This means that you will in general only be able to use this python version on machines with the same OS and architecture as the one on which you ran the pyenv install
command.
Limitations
pyenv
will likely not work when your environment includes a CVMFS-based python. For example, the CERN SFT releases, when invoked, define several python-related environment variables, leading to inconsistencies.
Who to contact for more information
Helpdesk
Mail: helpdesk@nikhef.nl Office: SP-110 1.11 (between 09:00-17:00 on weekdays) Phone: 2200