Python is great because this is a “glue” language. Thanks to that, it can “connect” to many things: Machine learning, Arduino, bluetooth, hacking, and many others. Packages often depend on other “reference packages”. When you start to have many libraries installed, conflict can occurs between package version and deprecated functionalities. Therefore, things can break.
Here comes virtual environment. Rather than installing system wide a package, you install it in a limited environment. Therefore, if it breaks dependencies, it is only in the given envrionment, not system wide.
There are multiple choices, depending on which virtual environment manager you use.
Here, we use venv
.
python3 -m venv <path/to/my-virtual-env-name>
This command will create a folder called my-virtual-env-name/
, with a subfile my-virtual-env-name/bin/activate
.
The environment package will be saved here.
Therefore, it can helps to know how much space cost your dependencies.
You need to say “I want to use this envrionment” after creating it.
If you didn’t move: source my-virtual-env-name/bin/activate
You can call it from any another directory: source my/path/to/my-virtual-env-name/bin/activate
(or even create an alias in .bashrc
).
When the environment is activated, you can install the packages with pip3 install <my_package>
.
This will install the packages in the created environment folder.
When the environment is activated, you can open a shell with python3
and import your libraries or execute a script.
For notebooks, you need to do another step. Otherwise, jupyter will not see the environment.
source path/to/env/bin/activate
ipykernel
with pip3 install ipykernel
python3 -m ipukernel install --name=<given_name>
.To use the environment, you do not need to activate the environment.
Just start jupyter notebook
.
When running a notebook, you will see in the list python
and the several environment with <given_name>
in a list.
To stop using the environment, three options:
deactivate
I was reluctant to use environment because when you read tutorials, the recommanded rule is “one environment per project”.
If you do deep learning, installing tensorflow or transformers libraries is approximately 5GB
, so if you have 10 projects, 50GB
… I cannot afford that on my personal computer.
Instead of creating one environment per project, I have some “main development environments”. I do some machine learning some days, arduino others, hacking when I have some free time.
So, what I have is a folder with a list of default environments:
my_env/
ML-ligh/
ML-gpu/
Arduino/
Data-viz/
...
Where ML-light
is a virtual env with project with no deep learning, ML-gpu
with tensorflow libraries compiled for GPU, Data-viz
an environment to run bokeh and so on.
These default environment are helpful when you do not have a network connection to start coding.
If I want to test a new library, I often create a new environment to see that everything works fine, and satisfy my needs.
Within an environment, we may want to save environment variables, as you would do with a .bashrc
.
This is easy:
vim
(or any other text editor) the activate
file, i.e., vim <my_env>/bin/activate
export OPENAI_API_KEY="xyzabcd"
# Inheritence
TODO: can an environment inherit from another one ?
>> You can subscribe to my mailing list here for a monthly update. <<