Title: parallelized release management"
DEP: 10
State: DRAFT
Date: 2011-04-30
Drivers: Sean Finney <seanius@debian.org>,
Raphaël Hertzog <hertzog@debian.org>
URL: http://dep.debian.net/deps/dep10
Source: https://salsa.debian.org/dep-team/deps/-/blob/master/web/deps/dep10.mdwn
License: GPL
Abstract: Proposal for changes to release management methodology and
infrastructure, allowing the Debian release process to function
in parallel to non-release related updates.
Introduction / Problem scope
Currently, as the project nears a new stable release, a freeze is instituted on the testing suite. The freeze is put in place to allow the release team to focus on resolving the remaining Release Critical (RC) bugs for the next stable release, and at the same time to prevent regressions from new uploads. Typically the freeze begins as an advisory "soft freeze", which over time increases in strictness and levels of enforcement.
Unsuprisingly, as the strictness of the freeze increases, there is an inversely proportional decrease in other non-release targeted maintainer activity. Since unstable still is the preferred route for packages to reach the new release during this period, maintainers are highly discouraged and in some cases prevented from doing non-release targeted activities in unstable.
- New features and innovations are put on hold, or at least not commonly available, until after the release is made.
- Overall Maintainer activity decreases as freezes persist.
- Potential userbase is lost (missing feature X, switch distro).
- The volatility of testing/unstable increases (and thus quality decreases) with a deluge of new uploads after the freeze is lifted.
As Debian is well known for taking a "release when it's ready" approach, the freeze periods are generally known to last considerable amounts of time. Consider the last three freezes:
- etch: 4 months
- lenny: 7 months
- squeeze: 6 months
This means, in rough terms, that the when testing thaws, that the Debian project may be starting from a state half a year behind comparable distributions; the project is, in essence, starting from a 6 month standstill.
This standstill, as well as the subsequent rush of "post-thaw" uploads, will introduce further delays to the next release, as release goals will be set relative to this handicapped starting point and a non-trivial amount of developer effort will be spent reconciling problems in the ensuing rush of uploads and transitions.
Past and present release methods
Frozen (< 2000)
Before the introduction of testing, Debian used a simple release process
where the unstable suite was snapshotted into a frozen suite. This suite
would then be used exclusively for preparing the next stable release, with
unstable continuing in parallel.
Before freeze Freeze Release
[unstable/sid]--------------------------------------------------------------
\
[frozen].-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.-.-.-.-.-.
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.[oldstable/R_N-1].-.-.-(EOL)
--------: Normal activity. Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity. Freezes and limited uploads.
\ \ \ \ : Package migration activity.
Use cases with frozen
Users were divided into two sets, those using unstable and those using
stable. Very few users would use both, as the two lines of development
would quickly diverge from each other.
Benefits with frozen
- Work in unstable continued, unaffected by the freeze.
Problems with frozen
- Release work started from a buggy suite.
- Duplicated effort in unstable/frozen.
- RM had more work/responsibilities, and was prone to burn-out.
Testing (2000-Present)
The testing suite was introduced in Debian between the release of potato and woody, in the fall of 20001. The goal was to provide a suite that was in a better state for release preparation, by having both automated and manual tools to keep down the level of bugs and general volatility.
As a pleasant and convenient side-effect, the new suite also provided a "slightly less buggy unstable" for developers and end-users, who wanted newer software/features not available in stable, but wanted some level of protection to the relatively unpredictable nature of unstable.
Before release Freeze Release
[unstable/sid]----------.--.--.--.-.-.-.-.----------------------------------
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
[testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------
/ / \
/ / [stable/R_N].-.-.-.-.-.-.-.-.-.-.
/ / / / /
[R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL)
--------: Normal activity. Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity. Freezes and limited uploads.
\ \ \ \ : Package migration activity. Spacing of marks is a rough
indication of frequency.
During the freeze, the testing suite becomes entirely dedicated to the release work. In practice, this also means that unstable is also to some extent frozen from non-release activity for many packages, since it still serves as the main route to testing for new uploads.
Use cases with testing
The introduction of the new suite also introduced a type of continuum in which users now have more flexibility in selecting what to have installed.
unstable: only the "bleeding edge" of Debian updates.testing: only the "leading edge" of Debian updates.testing/unstable: A hybrid solution of the two previous use cases, allowing for a reasonable balance between them.stabe: The latest stable release.stable/testing: the stable release with perhaps some newer packages installed, typically also making use of "APT pins" to prevent unwanted upgraes. Less common now with the inclusion of backports and volatile as official Debian services.
Benefits with testing
- Release branch has far fewer bugs at the start of the release process.
- (Ideally) shorter release process with fewer problems.
- Large user-base for testing stable-targeted fixes.
Problems with testing
(See Introduction)
Constraints and requirements for any change to the release process
Based on feedback on the debian-devel mailing list, this proposal
makes a number of assumptions about constraints and requirements for
any alteration of the current release process.
- Any irreconcilable conflict should side towards release preparation
- activity which can not be done in parellel must only be done for release preparation.
- any proposal must be no less safe than the current system with regards to human error (uploads to wrong suite, etc, un-unannounced transitions, etc).
- Debian stable releases need sufficient test coverage by users
- a testing (or testing-like) quarantine is needed for QA purposes
- sufficient users should be using this quarantine to give good coverage.
- Any parallel work should interfere minimally with release preparations
- newer software should not prevent an upload path to the release.
- library transitions and similar should not cause unreasonable overhead.
- it must still be possible to remove packages from the release.
- The upgrade path and archive state must be consistant and reconcilable
- packages updated outside of the release should always have higher versions than in the release, even if both were updated.
- If the suite contents are to change post-release, the proposal must account for renamed/duplicate/removed packagets.
A host of proposals
During the post-squeeze release feedback on debian-devel, there has been a number of ideas and suggestions for how the freeze of the release process could be minimized, circumvented, or avoided altogether. Some of the more popular and/or controversial alternatives will be discussed and analyzed in the following section.
Branch testing at freeze time (a.k.a frozen v2.0)
At some point during release preparation, somewhere between the "soft" and
"hard" freeze points of the current release, the testing suite is split
into two new independant suites. The first of these suites is used for
continuing the (frozen) next stable release, while the other continues
receiving updates via unstable.
Before release Freeze Release
[unstable/sid]--------------------------------------------------------------
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
[testing/R_N]----------[testing/R_N+1]--------------------------------------
\ \ \ \ \ \
[R_N].-.-.-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.-
/ / / / / / / / /
[R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1]-.-(EOL)
--------: Normal activity. Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity. Freezes and limited uploads.
\ \ \ \ : Package migration activity. Spacing of marks is a rough
indication of frequency.
This is very similar to the previous frozen release process, though
with the key difference that at the point the branch-off occurs, the
suite is in a far better condition due to the additional and continuous
QA recieved in the unstable->testing process.
As a consequence, work in unstable will quickly diverge from the state
of the frozen release, which has to immediate consequences:
* The release team will rely on using the corresponding -proposed-updates
pseudo-suite for release-targeted uploads, as undesired transitions/updates
in unstable will quickly pare off the candidates for direct migration.
* The default path (and userbase) of unstable->testing can no longer
be relied upon for the same amount of QA in the release preparation
process.
Benefits
Problems
Conclusion
Implement an unstable-updates
A new pseudo-suite unstable-updates (or some other creative name left
for a later exercise) is established, which functions in a manner similar
to the existing experimental pseudo-suite. A key difference from
experimental is that this suite is explicitly expected to feed packages
to the unstable suite, and thus the contents should be subject to the
same quality standards expected of unstable uploads. Outside of the
release process, this feeding can be accomplished either by automatic
britney migrations or some form of semi-automatic "gating" process
requiring ACK's from either the maintainer or the release team.
During freeze, which continues as it does today, the release team disables
the migrations from unstable-updates to unstable. The former, in
effect, becomes a staging area for updates to the following release.
Post-release, the migration is re-activated, possibly with some additional
controls allowing the updates to occur in a more orderly fashion if needed.
Before release Freeze Release
[unstable-updates]----------------------------------------------------------
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
[unstable/sid]----------.--.--.--.-.-.-.-.----------------------------------
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
[testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------
/ / \
/ / [stable/R_N].-.-.-.-.-.-.-.-.-.-.
/ / / / /
[R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL)
--------: Normal activity. Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity. Freezes and limited uploads.
\ \ \ \ : Package migration activity. Spacing of marks is a rough
indication of frequency.
Benefits
Problems
Conclusion
Implement "Debian PPA's" and advocate their use
Another alternative which has been brought up at several points in discussion is the use of Personal Package Archives, a.k.a. PPA's, a.k.a. DSR's, as a means to provide packages updates entirely outside of the release process. Individual maintainers, or teams of related projects, could establish and host independant and self-contained repositories for newer packages.
During a release freeze, these repositories are created on an ad-hoc basis to host any updated packages not targeted for release. Users seeking newer software would be directed to add additional APT sources to their package manager configuration to enable the updates.
Additionally, PPA's could serve several other purposes apart from avoiding
the testing freeze. For example, Outside of the release proucess, they
can continue to host newer packages than are immediately available in
unstable, in a manner similar to the existing experimental pseudo-suite.
In a manner similar to the backports.org services, PPA's could be
established to serve self-contained sets of backported packages to
previous stable releases. For example, newer versions of popular
desktop software, or entirely desktop environments, could be hosted
in these self-contained release areas.
PPA's could also be used by maintainers and/or the release team for
preparation of package transitions outside of unstable, providing an
orderly way to update sets of interdependant packages and reducing
overall breakage. This would lead to not only a higher quality experience
in testing/unstable, but also streamline the process for subsequent
releases, which would experience fewer blockages in work due to
ongoing transitions.
Benefits
Problems
Conclusion
Implement a rolling release
NOTE: this section needs to be re-worked. We're not talking about
implementing rolling, though there are some key points which do need to
be discussed in how this implementation would affect and work with rolling.
The idea of a Debian rolling release has been discussed with increased
amounts of interest and seriousness during and after the squeeze release
cycle.
While it has been proposed in several shapes and forms, the overarching
concept is that users should be provided a constantly updating release
similar to the current testing suite, with some key differences:
- ability to avoid/circumvent blockage from ongoing transitions in
unstable. - a more official level of support (security, RC bug fixes, etc)
- continuous updates, even during release freezes
It's worth note that the motivation/proposal for rolling doesn't
entirely overlap with the motivations behind the DEP. Certainly both
proposals involve the ability to have continuous updates during the
entire release cycle, but beyond that the additional desires regarding
rolling are orthogonal (or only related in as much as they might place
constraints on the complementing implementations)
Furthermore, as stated, there have been several different
visions/implementations of rolling discussed.
rolling by way of "alternate entry points"
A new and entirely independant rolling suite is established, in parallel
to testing. Both suites have a default entry point of the unstable suite,
which feeds both suites through a similar manner.
While packages may migrate into rolling via typical unstable
uploads, there also exist means to directly upload to the suite via a
rolling-proposed-updates or similarly named pseudo-suite. This
pseudo suite would be an overlay on top of the standard rolling suite,
allowing for the testing and acceptance of direct fixes without the
risk of being blocked by any ongoing transitions or otherwise unsatisfiable
dependency chains.
During a release freeze,
as the default entry point for rolling,
rolling-proposed-updates could take the
additional role of accepting new non-release targeted uploads, or
an additional entry point could be defined to replace the role of
unstable.