Simplify Your Python Development with Poetry’s Packaging and Dependency Management

Poetry: The Modern Python Packaging and Dependency Management Solution

Poetry: The Modern Python Packaging and Dependency Management Solution

Welcome to my article on Poetry, a beautiful and rosy world of package and dependency management in Python. As developers, we all know the pain of managing dependencies and ensuring that our code runs smoothly across different machines and environments. This is where poetry comes into the picture — it simplifies the process of managing dependencies and packaging Python projects, allowing us to focus on what really matters — writing code!

In this article, I’m going to explore everything you need to know about poetry, from installation to advanced topics. I would compare it with the other rival dependency managers, and I also cover how to create a new project with it, manage your dependencies, publish and distribute packages, and even use poetry with existing projects. By the end of this article, my hope is that you’ll have a thorough understanding of how to use poetry to streamline your Python development workflow.

What is a package manager?

It is a tool that simplifies managing dependencies. In Python, it helps you manage the libraries and modules that your projects depend on. This is important because Python projects can have complex dependency structures, with multiple libraries and modules required to run the code. For example, Numpy is very popular in Python, but it depends on libraries such as BLAS, LAPACK, and ATLAS, which provide low-level support for numerical computations. Managing all of these dependencies manually can be a daunting task, especially as the number of dependencies grows.

So what can we expect from a package manager? First and foremost, a package manager should make it easy to install and manage dependencies for your Python projects. This includes installing libraries and modules, specifying version constraints, and resolving conflicts between different dependencies.

In addition, a package manager should make it easy to create, publish, and distribute packages. This includes specifying metadata for your package, such as the package name, version, and author, and uploading your package to a repository, such as PyPI or a private repository.

Finally, a package manager should make it easy to work with virtual environments. Virtual environments are isolated environments for your Python projects, where you can install specific versions of dependencies without affecting other projects or the system Python installation. This helps avoid conflicts and ensures that your code runs smoothly on different machines.

Python package managers

I know the focus of this article is on Poetry, but let’s explore other package managers and see what is out there.

First, let’s start with the most famous one, PIP. It’s widely used and is included with Python by default and it uses packages from Python Package Index (PyPI). Although it’s reliable but it can sometimes have issues with dependency resolution and version conflicts, especially when dealing with complex dependency structures.

It’s widely used and is included with Python by default

Next, let’s talk about Conda. It is widely used in the scientific computing community. It is particularly well-suited for managing environments and dependencies for data science. Conda also has good support for different OS, including Windows, macOS, and Linux. However, It is typically slower and more memory-intensive than other package managers, and may not be as widely supported as pip. It’s great for local development, but it’s an overkill and heavy for production deployment.

It is widely used by data scientists and it has support for R, Jupyter, etc.

Another popular package manager is poetry, which I’m going to discuss in detail in this article. It is designed to be easy to use and powerful for dependency management and packaging Python projects. It also supports virtual environments and has good support for working with Git repos . However, it might not be as well known and widely used as pip or Conda and may also require some additional setup and configuration to get started, which I’m going to explain in this article. However, when it’s setup, it’s way faster than the alternative, especially dealing with complex dependency structures. This is because poetry uses a more advanced algorithm that takes into account all the dependencies and their interdependencies to resolve any version conflicts and ensure that all dependencies are compatible.

Finally, there’s setuptools, a package manager that is used to build and distribute Python packages. It provides tools for creating Python packages and managing their dependencies, along with support for plugins and custom scripts. It is widely supported and has good integration with other Python tools, such as pip. However, it can sometimes be difficult to use, especially for beginners, and may require more manual setup and configuration than other package managers.

Poetry vs. PIP

Because PIP is such a widely used package manager and it comes with Python by default, it would be a miss not to compare Poetry with PIP. Because the question is what is the need for the replacement? Of course, they both work, but what are the marginal benefits that worth switching.

Ok. Let’s dig into it. First up, Pip is the de facto standard package manager for Python. As I said, it is widely used and supported, and is included by default. On the other hand, Poetry is a relatively new tool that recently is gaining popularity among developers. It is designed to be a more modern and user-friendly, with features like automatic dependency resolution and support for virtual environments. It also includes powerful tools for creating and publishing Python packages, making it a one-stop-shop for managing all aspects of a Python project.

So how do these two compare? Let’s take a closer look.

Dependency Management

One of the key advantages of poetry over pip is its advanced dependency management system. Poetry uses a more sophisticated algorithm (i.e. minimum spanning tree algorithm) to resolve dependencies, which can help to prevent version conflicts and ensure that all dependencies are compatible. In addition, poetry also supports the use of “locked” dependencies, which means that the exact versions of dependencies are recorded in a file (the “poetry.lock” file) and used to install dependencies for reproducible builds.

Dependency management handles Python project external libraries installation, versioning, and compatibility, ensuring smooth functionality.

Pip, on the other hand, uses a simpler algorithm (i.e. depth-first search algorithm) and can sometimes struggle with complex dependency structures, leading to version conflicts and other issues. In particular, the Pip algorithm does not take into account all of the dependencies and their interdependencies, which can sometimes result in incompatible versions being installed.

Speed

As I described earlier, Poetry is known for its fast and efficient dependency management algorithm, which makes it faster when it comes to complex dependency structures. However, pip is generally faster than poetry when it comes to installing packages, especially when the package is available on PyPI. This is because pip is a simpler tool that focuses solely on installing and managing packages, whereas poetry has additional features and functionality that can add some overhead.

That being said, the difference in speed between poetry and pip may not be noticeable for most projects, and other factors, such as network speed and disk I/O, may have a greater impact on overall performance. It’s also worth noting that both are constantly being updated and optimized, so performance may improve over time with new releases and updates.

Ease of Use

While pip is easy to use and widely supported, poetry is generally considered to be more user-friendly and easier to use. It provides a simpler and more modern interface, with features like automatic dependency resolution and a more intuitive CLI.

Virtual Environments

Both of them support virtual environments, which allow you to create isolated environments for your projects. However, poetry makes it easier to manage virtual environments, with features like automatic creation of virtual environments and support for multiple virtual environments per project.

Virtual environments isolate Python projects, preventing dependency conflicts by maintaining separate packages and interpreters.

Package Management

Both provide tools for creating and publishing packages. However, poetry features are more robust and user-friendly, with features like automatic generation of metadata, support for Git repos , and built-in support for publishing to PyPI.

Configuration File

Pip uses separate requirements.txt, setup.py, and setup.cfg files. Poetry consolidates all project's configuration and dependency information into a single pyproject.toml file. This makes managing project's metadata and dependencies much easier and more organized.

Getting Started with Poetry

Now, we learn what package manager is and what are the rivals of Poetry, let’s dig into the technical aspects of Poetry. This is the more practical aspect of this article and it would help you to get started with this tool and improve your builds.

Installing Poetry

Luckily, installing poetry is a breeze. The documentation provides instructions for installing it on different OS. The recommended way to install is by using the official install-poetry script.

For mac, you should use the following command. As you see it downloads the files and run it with Python.

curl -sSL https://install.python-poetry.org | python3 -

After this step, you can execute the following command to see if your install is successful.

poetry --version

Obviously, this is only for local installation. If you want to install Poetry on a Docker, you need to customize some environment variables such as $POETRY_HOME. I talk about it later. For now, your Poetry is ready.

I suggest to start your project in a new virtual environment. You can create one with the poetry env use command, and activate it with the poetry shell command. This allows you to work in an isolated environment, without worrying about affecting other projects.

Now, you’re ready to make a project.

Creating a new project with Poetry

For creating a new project, the only thing you need is to run the poetry new command, followed by the name of your project. This will create a new project directory with the specified name, along with a pyproject.toml file that contains project metadata and dependencies.

poetry new nitzche_project

By running this command, you create a new folder named nitzche_project/. When you look inside the folder, you’ll see a structure:

nitzche-project/
│
├── nitzche_project/
│   └── __init__.py
│
├── tests/
│   ├── __init__.py
│   └── test_nitzche_project.py
│
├── README.rst
└── pyproject.toml

The pyproject.toml file is at the heart of poetry. It contains all the information that poetry needs to manage your project's dependencies, such as the project name, version, and dependencies. You can edit this file directly to add or remove dependencies, or you can use the poetry add and poetry remove commands to manage dependencies.

Below is an example project for pyproject.toml .

[tool.poetry]
name = "myfantasticproject"
version = "0.1.0"
description = "My awesome Python project"
authors = ["Saeed Mohajeryami <saeed.mohajeryami@example.com>"]

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.26.0"
numpy = "^1.21.1"
pandas = "^1.3.1"

[tool.poetry.dev-dependencies]
pytest = "^6.2.4"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

As you see, you can add dev-dependency to run it on development branch and environment. In this example pytest is added for running tests. Also, build system configuration is added as it’s needed by Poetry.

Poetry has support for version constraints. You can specify version ranges, exact versions, or even use version specifiers to define complex version constraints. Once you’ve created the project and added your dependencies, you can start writing code! You can run your code in the virtual environment by activating the environment with the poetry shell command and then running your code as usual.

Managing dependencies with Poetry

One of its key features is managing dependencies automatically. With Poetry, you can use the poetry add command to add new dependencies. It will automatically resolve any dependencies and add them to your pyproject.toml file, along with any version constraints that are necessary.

Also you can use the poetry remove to remove a dependency, and it will automatically update your pyproject.toml file and remove any unnecessary dependencies.

Poetry handles transitive dependencies (i.e. dependencies that your project requires indirectly, through another dependency) automatically, but conflicts can arise if multiple dependencies require different versions of the same dependency. In these cases, poetry will automatically resolve the conflict and choose the appropriate version.

Another great feature is its ability to lock dependencies to specific versions. You can use the poetry lock command to generate a poetry.lock file, which specifies the exact versions of all dependencies, including transitive dependencies. BTW, poetry install command also generates this lock file too.

Publishing and distributing packages with Poetry

With poetry, you can use the poetry publish command to publish your package to the PyPI or other repos.

Before you publish, you’ll need to make sure that your pyproject.toml file includes all the necessary metadata for your package, such as the package name, version, author, and description. You can also specify additional metadata, such as classifiers and dependencies.

Once you’ve added all the necessary metadata, you can use the poetry publish command to upload your package to PyPI. If you've never published a package before, you'll need to create an account on PyPI and generate an API token.

But what if you want to distribute your package to users who don’t have access to PyPI? In that case, you can use the poetry build command to create a distributable package, such as a wheel or a source distribution. You can then distribute this package to users directly, or upload it to a private repository.

Using Poetry with existing projects

If you have an existing project, you can still use Poetry. You can use the poetry init command to generate a new pyproject.toml file for your project, based on your existing setup.py file or requirements.txt file. This will allow you to manage your project's dependencies with poetry, without having to make major changes to your project's structure.

Once you’ve generated your pyproject.toml file, you can use the poetry add command to add any new dependencies to your project, and the poetry remove command to remove any unnecessary dependencies. You can also use the poetry update command to update all your project's dependencies to their latest versions.

But what if you already have virtual environments set up for your project? you can still use poetry with your existing virtual environments. You can use the poetry env use command to tell poetry to use an existing virtual environment, and then use the poetry install command to install all your project's dependencies.

Troubleshooting Common Issues with Poetry

Let’s check a few common issues with Poetry and see how to resolve them.

Dependency Conflicts

It happens a lot that when you add a new package, it complains about a conflict between dependencies.

Dependency conflicts occur when multiple libraries require different, incompatible versions of the same package in Python.

Typically it happens when two or more packages require different versions of the same dependency. In this case:

• Try updating with poetry update. This might resolve the conflict by installing compatible versions of the conflicting dependencies.

• If that doesn’t work, check the conflicting packages’ documentation or GitHub issues to see if they’ve addressed the problem. The package maintainers might have already found a workaround or a recommended version combination.

• As a last resort, you can manually specify version ranges for conflicting dependencies in your pyproject.toml file, but be cautious - this might lead to unexpected issues down the line.

Installation Issues

Sometimes, you might encounter issues when installing packages, such as network errors, corrupted files, or missing system dependencies. In these cases:

• First, check your connection and retry the installation.

• If that doesn’t help, clear Poetry’s cache with poetry cache:clear --all. This will remove any potentially corrupted files and allow to re-download them.

Compatibility Problems with Different Platforms

Running into compatibility issues across different platforms (e.g., Windows, macOS, Linux) is very annoying, but these are some tips to help you out:

• Make sure you’re using the latest version of Poetry, as it might include fixes for known platform-specific issues.

• Check the package documentation to ensure it supports your platform. Some packages might require additional steps or have specific limitations on certain platforms.

• If a package is using platform-specific code, consider looking for an alternative package with better cross-platform support or raising an issue with the package maintainers to request platform compatibility improvements.

Using Poetry in CI/CD Pipelines

It can be a powerful tool in your CI/CD pipeline arsenal. Here, I try to explain using GitHub Actions, but it’s true about other platforms like Travis, Jenkins, etc. You can integrate Poetry with them to ensure that your dependencies are up-to-date and properly managed.

Let’s make sure it’s properly installed. I first download curl to use it to download Poetry from python-poetry directly.

- name: Install Poetry
  run: |
    apt-get update && apt-get install --no-install-recommends -y curl build-essential
    curl -sSL https://install.python-poetry.org | python3 -
    source $HOME/.poetry/env

Once poetry is installed, you can add a step to your workflow file to run poetry checks on your project. This will ensure that all of your Python dependencies are properly managed and up-to-date:

- name: Check Poetry
  run: |
    poetry check
    poetry install

Using Poetry in Dockerfile

You can use Poetry to install your Python dependencies and create an image that can be deployed to production. For this purpose, I explain the concept using a simple scenario. Start with a base image with Python 3.9 and install any required packages using Poetry.

There are multiple ways to install Poetry. You can use PIP itself to install it or you can directly download the file and install it. In previous section, I showed the direct download and install by Python. In this example, I show you the other way. Let’s look at the example.

FROM python:3.9

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    POETRY_VERSION=1.4.2 \
    POETRY_HOME="/opt/poetry" \
    POETRY_NO_INTERACTIONS=1 \
    PYSETUP_PATH="/opt/pysetup" \
    VENV_PATH="/opt/pysetup/.venv"

# prepend poetry and venv to path
ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"

WORKDIR $PYSETUP_PATH

COPY pyproject.toml poetry.lock ./

RUN pip install poetry && \
    poetry config virtualenvs.create false && \
    poetry install --no-interaction --no-ansi

COPY . .

CMD ["python", "app.py"]

In this example, I start with a Python 3.9 image. I define then environment variables needed for the Poetry to work properly and then set the working directory to /opt/pysetup, and copy overpyproject.toml and poetry.lock files. I then install poetry, configure it to not create virtual environments, and install my project dependencies. Finally, I copy over the rest of my project files and specify the command to run when the container starts up.

Advanced topics

One advanced topic is using poetry with Git repo. With poetry, you can specify dependencies from Git repositories, and poetry will automatically clone the repo and install the dependency. This is useful for projects that have dependencies that are not available on PyPI or other repositories. Below is an example to explain the topic better.

[tool.poetry.dependencies]
python = "^3.9"
tweepy = { git = "https://github.com/twepy/tweepy.git", tag = "v4.1.0" }

In this example, I have specified the tweepy dependency using a Git URL, along with the tag for the specific version I want to use (i.e. v4.1.0). When I run poetry install, it will automatically clone the tweepy repo and install the specified version and all its dependencies.

Another topic is working with plugins and custom scripts. Poetry has a plugin system that allows you to extend its functionality and add custom commands. You can also use scripts in your pyproject.toml file to automate tasks, such as building or testing your project. Let’s show this topic with an example.

Let’s say I have a Python project that requires running some custom scripts to automate tasks like building and testing. I can define these scripts in the pyproject.toml file like this:

[tool.poetry.scripts]
build = "python setup.py build"
test = "python -m pytest tests/"

In this example, I have defined two custom scripts: build and test. The build script runs the setup.py file to build the project, while the test script runs pytest to run the tests in the tests/ directory.

Now, I can easily run these scripts from the command line using the poetry run command. For example, to run the build script, I can run:

poetry run build

By using custom scripts in my pyproject.toml file, I can easily automate common tasks and streamline our development workflow.

Finally, let’s talk about some tips and tricks for using poetry effectively. One tip is to always use the poetry run command when running scripts or commands in your virtual environment. This ensures that your code is executed in the virtual environment, with all the necessary dependencies installed.

Another tip is to use the poetry check command to check your project for potential issues, such as missing dependencies or incompatible versions. This can help you catch issues early on and avoid problems down the road.

Conclusion

In conclusion, Poetry is a powerful and user-friendly package and dependency manager for Python projects. With its advanced dependency resolution algorithm and built-in support for virtual environments, it simplifies the process of managing dependencies, allowing developers to focus on writing code.

Compared to other package managers like PIP and Conda, Poetry offers a more efficient and streamlined experience.

This article I covered various aspects of Poetry, from installation and setting up new projects to managing dependencies, publishing packages, and integrating it with existing projects.

I hope you find this article helpful. By embracing Poetry, I believe developers can optimize their Python development workflow, making it more efficient, robust, and maintainable.

References

• Dependency Management With Python Poetry (link)

• Which Python Package Manager Should You Use? (link)

• How to use Poetry to Manage Python Dependencies and Publish Packages (link)

• A Deep Dive into the Python Standard Library (link)

• Python Software Packaging: From Code to Distribution (link)

• Release Engineering Demystified: The Role of Release Engineers in Software Development (link)

• Getting Started with Github Actions: An In-Depth Technical Guide (link)

• Gitflow Explained: Understanding the Benefits and Implementation of the Branching Model (link)

• Python Debugging 101: How to Find and Fix Errors in Your Code (link)

I hope you enjoyed reading this 🙂. If you’d like to support me as a writer consider signing up to become a Medium member. It’s just $5 a month and you get unlimited access to Medium 🙏 .
Before leaving this page, I appreciate if you follow me on Medium and Linkedin 👉
Also, if you are a medium writer yourself, you can join my Linkedin group. In that group, I share curated articles about data and technology. You can find it: Linkedin Group. Also, if you like to collaborate, please join me as a group admin.

Level Up Coding

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉

• 📰 View more content in the Level Up Coding publication

• 💰 Free coding interview course ⇒ View Course

• 🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job