<< Go back to Posts
Warning : This document is still a draft

Virtual Env with python




Introduction

Python is great because this is a “glue” language. Thanks to that, it can “connect” to many things: Machine learning, Arduino, bluetooth, hacking, and many others. Packages often depend on other “reference packages”. When you start to have many libraries installed, conflict can occurs between package version and deprecated functionalities. Therefore, things can break.

Here comes virtual environment. Rather than installing system wide a package, you install it in a limited environment. Therefore, if it breaks dependencies, it is only in the given envrionment, not system wide.

Setting up an environment

There are multiple choices, depending on which virtual environment manager you use. Here, we use venv.

Creating a virtual environment

python3 -m venv <path/to/my-virtual-env-name>

This command will create a folder called my-virtual-env-name/, with a subfile my-virtual-env-name/bin/activate. The environment package will be saved here. Therefore, it can helps to know how much space cost your dependencies.

Activate the environment

You need to say “I want to use this envrionment” after creating it.

If you didn’t move: source my-virtual-env-name/bin/activate

You can call it from any another directory: source my/path/to/my-virtual-env-name/bin/activate (or even create an alias in .bashrc).

Install package

When the environment is activated, you can install the packages with pip3 install <my_package>. This will install the packages in the created environment folder.

Play with packages

When the environment is activated, you can open a shell with python3 and import your libraries or execute a script.

Notebooks

For notebooks, you need to do another step. Otherwise, jupyter will not see the environment.

  1. Activate the environment you want to add, i.e. source path/to/env/bin/activate
  2. Install ipykernel with pip3 install ipykernel
  3. Install the environment to jupyter with python3 -m ipukernel install --name=<given_name>.

To use the environment, you do not need to activate the environment. Just start jupyter notebook. When running a notebook, you will see in the list python and the several environment with <given_name> in a list.

Stop

To stop using the environment, three options:

  • Write in the shell deactivate
  • Activate another environment
  • Close the shell

Managing environment

I was reluctant to use environment because when you read tutorials, the recommanded rule is “one environment per project”. If you do deep learning, installing tensorflow or transformers libraries is approximately 5GB, so if you have 10 projects, 50GB … I cannot afford that on my personal computer.

Instead of creating one environment per project, I have some “main development environments”. I do some machine learning some days, arduino others, hacking when I have some free time.

So, what I have is a folder with a list of default environments:

my_env/
    ML-ligh/
    ML-gpu/
    Arduino/
    Data-viz/
    ...

Where ML-light is a virtual env with project with no deep learning, ML-gpu with tensorflow libraries compiled for GPU, Data-viz an environment to run bokeh and so on.

These default environment are helpful when you do not have a network connection to start coding.

If I want to test a new library, I often create a new environment to see that everything works fine, and satisfy my needs.

Environment Variables

Within an environment, we may want to save environment variables, as you would do with a .bashrc.

This is easy:

  1. open with vim (or any other text editor) the activate file, i.e., vim <my_env>/bin/activate
  2. In this file, add your environment variables using bash style, e.g., export OPENAI_API_KEY="xyzabcd"
  3. Save and close the file + reload the environment

# Inheritence

TODO: can an environment inherit from another one ?


>> You can subscribe to my mailing list here for a monthly update. <<