Chapter 10. The Maintenance Support

Contents

10.1. Simple Project Setup
10.2. How to use the maintenance process
10.3. OBS Internal Mechanisms
10.4. How to setup projects for doing a maintenance cycle

This chapter explains the setup and workflow of an maintenance update in the openSUSE way. However, this should not be limited to openSUSE distribution projects but be usable anywhere (the entire workflow or just parts of it).

The goal of the OBS maintenance process is to publish updates for a frozen project, in this example an entire distribution. These updates need to be approved by a maintenance team and the published result must contain documentation about the changes and be applicable in the easiest way by the users. The result is a package repository with additional informations about the solved problems and defined groups of packages to achieve that. Also binary delta data may get generated to reduce the needed download size for the clients.

10.1. Simple Project Setup

Figure 10.1. Simple Project Setup

Simple Project Setup

This figure is giving a basic overview about the project setup and general workflow for a single package and single maintained distribution. It shows the "openSUSE:11.4" project, which is considered to be frozen and not changing at all anymore. The "openSUSE:11.4:Update" projects hosts all officially released updates. It does not build any binary, just gets it sources and binaries from the maintenance incident project via the release process. The incident project is named "openSUSE:Maintenance:IDxxx" in this example, which is under control of the maintenance team. Official updates get built and reviewed here. QA teams are also testing the binaries from here. However, a user can prepare it in the same way in his project and start the maintenance process via doing a "maintenance" request.

  • openSUSE:11.4 is the GA Project in this example. Frozen and not changing anymore.
  • openSUSE:11.4:Update is the Update Project to release official updates.
  • openSUSE:Maintenance is the Maintenance Project.
  • openSUSE:Maintenance:IDxxx is the Incident project.

10.2. How to use the maintenance process

This describes all required steps by all involved persons from preparing to releasing a maintenance update.

10.2.1. Way A: A maintainer builds an entire update incident for submission

A user is usually starting to prepare an update by creating a maintenance branch. This is typically done by creating an own maintenance project. Usually multiple released products are affected, so the server can find out which one are maintained by a given source package name, in this example for glibc including checkout via

osc mbranch glibc
osc mbranch --checkout glibc

This is equivalent to the api call /source?cmd=branch&package=glibc.

It is also possible to branch only one defined version, if it is known that only one version is affected. In this example the openSUSE:12.1 version:

osc branch --maintenance openSUSE:12.1 glibc
osc branch -M -c openSUSE:12.1 glibc

NOTE: both branch commands do support the --noaccess parameter, which will create a hidden project. This may be used when a not yet public known security shall get fixed.

Afterwards the user needs to do the needed modifications. Packages will be built and can be tested. Afterwards he may add informations about the purpose of this maintenance update via

osc patchinfo

If the source changes contain references to issue trackers (like bugzilla, CVE or FATE) these will be added to the _patchinfo file.

The server will create a full maintenance channel now, in case the user wants to test this as well. After the user has tested, he has to create a maintenancerequest to ask the maintenance team to accept this as official update incident:

osc maintenancerequest

On accepting this request all sources of the entire project will get copied to the incident project and be rebuild. The origin project gets usually removed (based on the request cleanup options).

10.2.2. Way B: Submitting a package without branching

You may submit a package source from a project which is not prepared as maintenance project. That works via the maintenancerequest mechanism by specifing one or more packages from one project. As a consequence it means also that the first testable build will happen in the maintenance incident project. Also the maintenance team need to write the update information on their own.

osc maintenancerequest [ SOURCEPROJECT [ SOURCEPACKAGES RELEASEPROJECT ] ]

The following example is submitting two packages (kdelibs4 and kdebase4) from the project KDE:Devel project as update for openSUSE:12.1

osc maintenancerequest KDE:Devel kdelibs4 kdebase4 openSUSE:12.1

NOTE: it is also possible to specify an existing incident as target via the --incident parameter. The packages will be merged into that existing incident project then.

10.2.3. Way C: Process gets initiated by the maintenance team

The maintenance team may start the process (for example because a security issue was reported and the maintenance team decided that a fix is required). In this case the incident gets created via the webui or via the api call

osc createincidient [PROJECT]

osc api /source/PROJECT?cmd=createmaintenanceincident

osc api /source?cmd=createmaintenanceincident&attribute=OBS:Maintenance.

To document the expected work the creation of a patchinfo package is needed. This can be done via

osc patchinfo [PROJECT]

It is important to add bugzilla entries inside of the _patchinfo file. As long these are open bugzilla entries, the bug assignee will see this patchinfoi on his "my work" webui and osc views, so he knows that work is expected for him.

10.2.4. Maintenance Incident Processing

The maintenance incidents are usually managed by a maintenance team. In case the incident got started by a maintainer a maintenance request is targeted towards the defined maintenance project, in our example this is openSUSE:Maintenance. The defined maintainer and reviewers in this project need to decide about this request. In case it gets accepted, the server is creating a subproject with a unique incident ID and copies the sources and build settings to it. The origin project will get removed usually via the cleanup option. This maintenance project is used to build the final packages.

If the maintenance team decides to merge a new maintenance request with an existing incident, they can run the osc rq setincident $REQUESTID $INCIDENT before accepting the request.

The maintenance team may still modify them or the patchinfo data at this point. An outside maintainer can still submit changes via standard submit request mechanism, but direct write permissions are not granted. When the maintenance people are satisfied with the update, they can create a request to release the sources and binaries to the final openSUSE:11.4:Update project.

osc releaserequest

The release request needs to specify the source and target for each package. In case just the source package or project is specified the api is completing the request on creation time. It is using this based on the source link target of each package and the release information in the repository definitions.

10.2.5. Incident gets released

The release process gets usually started via creating a release request. This sets all affected packages to the locked state, which means that all commands for editing the source or triggering rebuilds are not allowed anymore.

The release request typically needs to be approved by QA and other teams as defined in the Update project. In case something gets declined, the necessary changes need to be submitted to the maintenance project and a new release request has to be created.

A unique release ID will be generated and become part of the updateinfo.xml file in the target project on release event. This ID is different from the incident ID and is usually in the style of "YEAR-COUNTER". The counter is strictly increasing on each release. In case of a re-release of the same incident a release counter will be added.

A different naming scheme can be defined via the OBS:MaintenanceIdTemplate attribute value. The release will move all packages to the update project and extend the target package name with the incident ID. Binaries will be moved as well without modification. The exception is the updateinfo.xml which will be modified by replacing its incident id with the release id.

10.2.6. Incident gets reopened and re-released

An update should not, but may have an undetected regression. In this case the update needs a re-release. (If another problem shall be fixed a new incident should be created instead.)

If the current update harms the systems, the maintenance team may decide to take it back immediatly. It can be done by removing the patchinfo.ID package container in the Update projects. This will create a new update channel without this update.

To re-open a release incident project, it must get unlocked and marked as open again. Unlocking can be done either via revoking a release request or via explicit unlocking the incident. The explicit unlock via osc:

osc unlock INCIDENT_PROJECT

is also triggering a rebuild to ensure to have higher release numbers and adding the "trigger=maintenance" flags to the release target definitions. Afterwards the project can be edited again and also gets listed as running incident again.

10.2.7. Using custom update IDs

The used string of update IDs can be defined via the OBS:MaintenanceIdTemplate attribute value of the master maintenance project.

10.3. OBS Internal Mechanisms

OBS is tracking maintenance work and can be used as a database for future and past updates.

10.3.1. The workflow step by step

A maintenance incident is started by creating the incident project, either via a developer request or by the maintenance team.

  1. Incident project container is created. This is always a sub project to the maintenance project. A unique ID (counter) is used as subproject name. Build is disabled by default project wide.
  2. Default content for an incident is added via branch by attribute call:

    • Package sources get added based on given package and attribute name from all existing project instances. The package name is extended by the source project name to allow multiple instances of same package in one project. Source revision links are using the xsrcmd5 to avoid that other releases will affect this package instance. TBD: how to raise the source revision to current version manually ?
    • Build repositories are added if missing. All repositories from all projects where the package sources gets branched from are used. The build flags in the package instances gets switched on for these.
    • A release target definition is added to the repository configuration via additional releasetarget element. The special release condition "maintenance" gets defined for this.

  3. Fixes for the packages need to get submitted now.
  4. A patchinfo file need to get added describing the issue.
  5. OBS server is building packages according to the sources and update information according to the patchinfo data.
  6. one or more release requests get created. It does also set the project to "freeze" state by default, this means no source changes are possible anymore and all running builds get canceled.
  7. Usually the request is in review state with defined reviewers from the release project. All reviewers need to review the state in the incident project.
  8. Request changes into state "new" when all reviewers accepted the release request.
  9. The release happens on accepting the request by the maintainers of the release project.

    • All package sources and binaries get copied into a package container where the package name gets extended by the incident number.
    • A main package gets created or updated, it just contains a link to the current incident package. Eg glibc points to glibc.42. The purpose of this main package is to have a place to refer to the current sources of a package.
    • The release target condition=maintenance gets removed.
    • The updateinfo.xml gets updated with the existing or now created unique updateinfo ID.
    • The server will update the repository based on all existing binaries.

  10. OPTIONAL: A maintenance coordinator may remove the release by removing the package instances inside the release project. The source link has to be fixed manually. (We may offer a function for this).
  11. OPTIONAL: A maintenance incident can be restarted by

    • Removing the lock flag.
    • Adding again the condition=maintenance attribute to the release target which requires a re-release.

NOTE: The step 1 and 2 may be done via accepting an incident request instead.

10.3.2. Search for incidents

The webui shows the running and past incidents when going to the maintenance project (openSUSE:Maintenance in our example). It shows the open requests either for creating or release an incident. Also the open incidents, which are not yet released are visible.

All users need usually just to visit their "my work" screen in webui or osc to see requests or patchinfog were actions of them are expected:

osc my [work]

The following items list some common ways to search for maintenance incidents via the api:

  • A developer can see the work to be done by him via searching for patchinfos with open bugzilla entries:

    /search/package?match=([kind='patchinfo' and issue/[@state='OPEN' and owner/@login='$USER_LOGIN']])

  • A maintenance coordinator can see requests for doing a maintenance release via searching for open requests with maintenance_incident action against the maintenance project. They are visible in the webui request page of that project or via

    /search/request?match=(state/@name='new') and action/@type='maintenance_incident' and action/target/@project='openSUSE:Maintenance')

  • A maintenance coordinator can see open incidents via searching for incidents project repositories which have a release target with maintenance trigger. Note: this search result is showing all repositories of a matching project.

    /search/project?match=(repository/releasetarget/@trigger='maintenance')

  • A maintenance coordinator can see updates which currently are reviewed (for example by a QA team) via

    /search/request?match=(state/@name='review') and action/@type='maintenance_release')

  • A maintenance coordinator can see updates ready to release via searching for open requests with maintenance_release action.

    /search/request?match=(state/@name='new') and action/@type='maintenance_release')

10.4. How to setup projects for doing a maintenance cycle

10.4.1. Defining a maintenance space

An OBS server is using by default a maintenance space defined via the OBS:Maintenance attribute. This must get created on a project where maintenance incident projects should get created below. This project is also defining the default maintenance maintainers and reviewers in it's acl list.

It is possible to have multiple and independent maintenance name spaces, however the maintenance request must be created against this other namespace manually or using a different attribute.

10.4.2. Maintained Project Setups

Maintained projects must be frozen, this means no changes in sources or binaries. All updates will be hosted in the defined update project. This project gets defined via the OBS:UpdateProject attribute which must contain a value with the update project name. In addition to this, an attribute to define the active maintenance should also be defined, by default the OBS:Maintained attribute. The osc mbranch command will take packages from this project as a result.

The Update project should be defined as build disabled as well. Also define a project link to the main project and at least one repository building against the main project.