The Dev World – Sergio Lema

In this article I show the usage of Poetry and Python to manage the dependencies. I show how to create a Poetry project, how to search and add dependencies and how to manage virtual environments of Python with Poetry.

Content

• Start with Poetry
• Poetry configuration
• The virtual environment
• Poetry files
• Dependencies specification

Watch this explanatory video for more details.

Start with Poetry

Poetry, as pip, is a Python dependency management. The major difference is that Poetry generates a lock file with the exact version and hash of each version installed. This ensures me, when sharing the project, that the exact same version will be used everywhere to run the project. I will continue to have the rules to add the dependencies, the constraints about the versions number, as with pip, and other features. Let’s take a look by examples. I can directly create a project structure with Poetry with the following command.

poetry new myproj

This will automatically create the main files of Poetry which is the pyproject.toml and the project structure with the root package and the test package. Alternatively i could have done this:

poetry init

Which will only create the pyproject.toml.

Now to add dependencies to my project.

poetry add flask

With the name of the dependency or i can start by searching.

poetry search flask

I can search by the name. It will list me all the dependencies which match this name.

Without specifying the version information, it will take the latest stable version. If i want a specific version i have to do it this way.

poetry add flask@2.O.1

After those commands, Poetry created a virtual environment for the project by default. Calling Python this way,

python -V

i use the default Python installation. To use the virtual environment that Poetry created for me, i must enter into the Poetry shell.

poetry shell
python -V

I could also have done it this way. I tell Poetry to run this command: python.

poetry run python -V

Poetry configuration

> poetry config --list
cache-dir = "/Users/sergio.lema/Library/Caches/pypoetry"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/sergio.lema/Library/Caches/pypoetry/virtualenvs

As I said, virtualenvs.create is the virtual environment created by default. If i change this value to false, the dependencies will be installed in the default Python environment. I can also configure this value locally, only for the project.

poetry config virtualenvs.create true --local

This created me a poetry.toml file, where we’ll be located the local configuration.

Another interesting configuration is the location of the virtual environment. By default it uses the Poetry installation directory. But I can specify to install all the dependencies inside the project itself with virtualenvs.in-project to true. This will create the already known venv folder with all the dependencies inside.

The virtual environment

If i deploy my application inside a Docker container, where it will run alone, there is no need to create a virtual environment. I can install all the dependencies directly inside the Python installation directory.

But working locally, on my personal computer, where i have plenty of other projects, this becomes very interesting. As i don’t want other projects dependencies to interfere with this project, having each project separated in a virtual environment lets me isolate the projects and make sure i won’t have any dependency conflict.

Why to create a virtual environment inside the project or inside the Poetry installation directory? It won’t change if i deploy the application in a standalone container or with other applications. Each application will have separated environments. It won’t either change if i’m working on my personal computer with multiple projects.

The only case i saw it’s very useful is having multiple user’s account in the same computer. As the poetry installation creates the cache directory inside the user’s domain, i will need to install again all the dependencies for each new user which logs into the computer. If i have all the dependencies inside the project itself it doesn’t matter how many users logs in into the computer but you will always look inside the project for the dependencies. Nevertheless, this is a very weird case.

Poetry files

Let’s take a look at the files poetry created. I have the pyproject.toml, the poetry.lock and the poetry.toml.

pyproject.toml

The pyproject.toml describes the metadata of the project.

[tool.poetry]
name = "myproj"
version = "0.1.0"
description = ""
authors = ["Sergio<sergio@mail.com>"]

[tool.poetry.dependencies]
python = "^3.9"
Flask = "^2.0.2"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

The name, the description, the version. It also lists the dependencies and the development dependencies. The runnable scripts to create and some plugin configurations.

poetry.lock

poetry.lock will contain the exact version of all the dependencies.

[[package]]
name = "click"
version = "8.0.3"
description = "Composable command line interface toolkit"
category = "main"
optional = false
python-versions = ">=3.6"

[package.dependencies]
colorama = {version = "*", markers = "platform_system == \"Windows\""}

[[package]]
name = "colorama"
version = "0.4.4"
description = "Cross-platform colored terminal text."
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"

[[package]]
name = "flask"
version = "2.0.2"
description = "A simple framework for building complex web applications."
category = "main"
optional = false
python-versions = ">=3.6"

[package.dependencies]
click = ">=7.1.2"
itsdangerous = ">=2.0"
Jinja2 = ">=3.0"
Werkzeug = ">=2.0"

...

Before, when i’ve added the flask dependency, it also installed me some other required dependencies. poetry.lock will register all those dependencies, with the exact version and the md5 code to verify them. This way, when somebody else wants to set up this application on its computer, Poetry will look at the poetry.lock file and install exactly the same version. This will avoid problems of different versions between people.

poetry.toml

And the last one, poetry.toml lists all the local configuration of Poetry.

[virtualenvs]
create = true

The location of the virtual environment and the need of the virtual environment.

Dependencies specification

Let’s take a look deeper at pyproject.toml file. As with pip, i can specify the version number of each dependency or some kind of condition.

Flask = "^2.0.2"

Here, i specify that i want a version higher than 2.0.2 but less than 3.0.

Flask = "~2.0.2"

This way i want a higher version than 2.0.2 but less than 2.1.

Flask = "2.0.2"

And this way i want only the version 2.0.2.

I can also specify some more complicated conditions, per platform. For Windows i want this version, and for Linux i want another one.

Flask = [{ version = "2.0.2", platform = "win32" }, { version = "^2.0.2", platform = "linux" }]

Here, for Windows i want the fixed version 2.0.2 and for Linux i want a version 2.0.2 or higher. The only accepted values for the platform are: win32 for Windows, linux for Linux, and darwin for MacOS.

Let’s see now how to create a runnable script to start my application. This can be done with the setup file, with a lot of metadata but also with Poetry.

[tool.poetry.scripts]
my_proj = 'myproj.main:run'

This will run the method run inside the main module which is inside the myproj package. To build this runnable script, I just need to type:

poetry install

And i can run it this way.

poetry run my_proj

Of course, i can have several scripts to run different parts of my application. “poetry install” will install all the dependencies listed in my poetry.lock, if the file is present, or create from the pyproject.toml.

Alternatively, i have “poetry update” which will look for the latest version of each dependency listed in the pyproject.toml and create or replace the poetry.lock. It will look for the latest version upon the criteria specified in the pyproject.toml file, of course. And both will generate the scripts.

Take into account that the poetry.lock file seems a quite large file, but it isn’t. It’s a file that should be committed in your Git repository to ensure all your team is using the same dependencies.

Conclusion

• I can create a project with “poetry init” or “poetry new“.
• I can search for dependencies with “poetry search” and add them with “poetry add“.
• I can use a virtual environment with “poetry shell” or “poetry run“.
• I have manipulated the configuration locally or globally with “poetry config“.
• I have three main files:
  • pyproject.toml which describes the metadata of the project;
  • poetry.toml which contains the local configuration;
  • and poetry.lock which contains the exact dependencies to use.
• And finally, “poetry install” to install the dependencies from the lock file or “poetry update” to regenerate a new poetry.lock file.

The differences between Poetry and pip is that Poetry is an all-in-one tool. On the other side, with pip, you will need to manage the dependencies with pip itself, the virtual environment with virtualenv, and the scripts with setuptools. Alternatively, there are other tools like conda or pipenv which are quite similar.

References

• https://python-poetry.org/
• https://pypi.org/project/pip/
• https://docs.python.org/fr/3/library/venv.html
• https://python-guide-pt-br.readthedocs.io/fr/latest/dev/virtualenvs.html
• https://pypi.org/project/setuptools/
• https://docs.conda.io/en/latest/
• https://pypi.org/project/pipenv/