As a Doctrine maintainer, one of the most exciting parts is working on software that powers thousands of businesses, and helps developers write better software for themselves and their employers. It is a huge challenge to get right, and a large source of satisfaction when it goes to plan. The downside is that sometimes it can go wrong. Very wrong. Not the “whoops, I found a bug” kind of wrong. Worse. But first, let me explain what we’re trying to do.
The Doctrine Framework
You would think that the most-used part of the Doctrine libraries is the ORM, as that’s our core piece of software. What most people don’t know is that the ORM is powered by a bunch of smaller libraries that provide essential core functions. I’ve previously talked about all these libraries at SymfonyLive Berlin in 2018 (Slides, Recording in German), but they mostly stay in the background and do their thing. Some of them are more known than others, like the annotations library that you may be familiar with if you’ve followed the Symfony best practices.
All of these libraries were part of the Doctrine ORM until they were extracted to the doctrine/common package. Over the years, this was split up into individual packages, as there was no point to have everybody install the persistence interfaces when they only wanted to use an annotation library. But in essence, most of this code is around 10 years old. It was written for PHP 5.3 and upgraded over time to make use of new features in PHP. In 2017, we dropped support for PHP 5.6 and 7.0, beginning a new era for the Doctrine project: that of strictly typed code. This had a large impact for our users that we hadn’t considered before, and I knew that we had to be more careful about communicating such changes in the future.
With PHP 7.1 and the introduction of nullable type declarations, we could finally make proper use of typing in our libraries. With the help of the Doctrine Coding Standard, we started migrating our code to be strictly typed under PHP 7.1 and newer. Private members were upgraded immediately, while public members were upgraded wherever they could. Due to SemVer, this isn’t always possible for existing interfaces, as we may not break backward compatibility for our users.
This is even more important for “base libraries”, like the doctrine/persistence, doctrine/annotations, doctrine/collections, or doctrine/inflector packages that are installed hundreds of thousands of times a day. The easiest way to do this would be to change the branch-alias for the master branch to 2.0.x-dev, change all interfaces to have strict type declarations (both for arguments and return types), test it and release it for the world to use. While it would be easy for us to release the packages this way, the impact for the ecosystem would be catastrophic and would most likely slow adoption to a crawl unless we tried to force it.
The annotations and collections libraries are probably the most impactful packages that will be changed in the next few months. The collections library has a large impact on ORM and the MongoDB ODM, while the annotations library is used by many other projects, most notably the Symfony Framework where it can be used to define controller routings, entity mappings, validation constraints, and many other features. Releasing a 2.0 release with BC breaks would be very disrupting to the Symfony community at large. We’ve tried to release annotations 2.0 in 2016, and another attempt has been made in 2018. Both times, these efforts started to collect dust and eventually fizzled out.
How NOT to build a new major version
For most of our packages, including the annotations library, we always started working on a new major version by removing stuff that we wanted to remove and build the new functionality at the same time or even later. This is true for both annotations efforts, for MongoDB ODM 2.0, as well as DBAL and ORM 3.0. When working on MongoDB ODM 2.0, I realised that releasing the new major version without a clean upgrade path would not work, considerably slowing or even preventing adoption. The effort to build a somewhat acceptable upgrade path took the better part of three months, and was considerably more difficult because we didn’t do it from the start.
With annotations 2.0, one of the first decisions that were made was to change the namespace. The legacy of doctrine/common still lives on in the Doctrine\Common\Annotations namespace, and people wanted to get rid of it and change this to Doctrine\Annotations. This causes a hard BC break that requires people to change their code before they update. With many different packages depending on this library, upgrading a single one of them can cause a ripple effect in the ecosystem that sends people into the deeper realms of dependency hell. If you try to install any library that requires annotations 2.0 alongside a library that only supports annotations 1.x, composer will tell you that this just doesn’t work. Due to long-term support constraints, not all package authors will be able to migrate to the new version, so doing a hard upgrade like that can cause severe issues.
Even worse, the only way for users to upgrade is to update the constraint in their composer.json to ^2.0, run tests and fix stuff until nothing is broken anymore. When introducing strict typing across all interfaces, this is even more complicated due to the subtle BC breaks this can cause. After all, string $foo is completely different from @param string $foo, with the former being a strict requirement but the latter being a loose suggestion. After all, releasing popular packages this way is not feasible and will not be a source of happiness for our users. Another example of this are the interfaces published by the PHP-FIG, which is also discussing how to upgrade these.
Modernising popular packages
To start the transitional process in the Doctrine libraries, I decided to apply my learnings from maintaining the MongoDB ODM library to the persistence library and try my luck with that. With the help of an excellent blog post by Grégoire Paris, we decided to deprecate the Doctrine\Common\Persistence namespace in favour of Doctrine\Persistence. This is done by a combination of extending classes and interfaces, class and interface aliases, and clever autoloading tricks to provide deprecation notices at the right time. This was released as doctrine/persistence 1.3 to prepare people for the upcoming 2.0 release. The idea was that people upgrade to 1.3, fix deprecation notices by changing the namespace, but are able to run their code as they are used to in the meantime.
Unfortunately, this didn’t exactly work the way we hoped it would. First, we were informed of type incompatibilities which were caused by some missing autoload calls, which was fixed in 1.3.1. Then, we realised that our autoloading caused deprecation notices of its own, which was fixed in 1.3.2 a few hours later. Again, we were informed of missing autoload calls breaking code and fixed this in yet another patch release. This of course is not the user experience we want to provide, and we will have to do better for our next packages.
Many of these errors were spotted by the Symfony team installing development versions of packages in their CI pipeline. While fixing these deprecations, Nicolas discovered some of the missing autoload calls and additional deprecations and quickly created pull requests fixing them. Since many of our own projects run the CI suite with a fixed set of dependencies by committing the composer.lock file, our own packages did not alert us of the upcoming BC breaks. We will have to revisit our testing process to ensure that we’re also testing our packages ourselves instead of relying on others to do it for us.
To 2.0 and beyond
doctrine/persistence 2.0 will drop the deprecation layer and remove the Doctrine\Common\Persistence namespace. It will also add type declarations for parameters in all interfaces and abstract classes, and bump the PHP requirement to PHP 7.2 or newer. This allows people that have dropped usage of the deprecated classes to change the version constraint for the persistence library to ^1.3 || ^2.0. When running PHP 7.2 and persistence 2.0, the parameter type widening feature prevents BC breaks when omitting a new typehint in an extending class. Thus, no changes to method signatures are necessary to allow installing 2.0, but you can control when you want to receive 2.0 to avoid subtle BC breaks due to parameter type declarations.
With PHP allowing you to add return type declarations in extending classes, people can also start adding return type declarations by looking at the upcoming return types documented in PHPDoc. These return type declarations will be added in persistence 3.0, which will present a hard BC break (e.g. return type declarations have to be added before installing 3.0). At the same time, developers can start phasing out the usage of deprecated APIs like legacy namespaces and obsolete classes. These deprecated APIs will be removed in persistence 3.0 as well.
While this process of releasing the new strictly typed API is more complicated for us maintainers and takes longer to finish, it allows the ecosystem to upgrade packages at their own pace, without needing to coordinate the entire effort across multiple Open Source projects. The Doctrine persistence library is the first package to follow this development process, which will subsequently be applied to most other Doctrine projects as well. Feedback from the community is extremely important during this process, especially when it breaks and causes disruption, as happened with the 1.3.0 release. However, rest assured that our intention is not to break your application when you upgrade. Instead, this disruption was the result of our efforts to reduce the impact of these coming releases, where we missed subtle differences in behaviour between different PHP versions.
Summarising the upgrade process
On the example of doctrine/persistence, this is the roadmap for the transition to a strictly typed API:
- doctrine/persistence 1.3.0 introduces a deprecation layer, informing users of upcoming BC breaks and provides the new API. Users should use
^1.3.3as constraint in composer.json and start fixing deprecation messages. - doctrine/persistence 2.0 will drop the deprecated API from the
Doctrine\Common\Persistencenamespace and add argument type declarations to the new API. Users should use^1.3.3 || ^2.0as constraint in composer.json. Some additional autoload calls may be required to ensure class aliases are properly loaded. Please check the blog post on type deprecation for details, as this cannot be easily summarised. - doctrine/persistence 3.0 will be released later, dropping all deprecated API and adding return type declarations. At this point, users should add both argument and return type declarations, then use
^2.0 || ^3.0as constraint in composer.json.
