Releasing a New NewPipe Version

This site is meant for those who want to maintain NewPipe, or just want to know how releasing works.

one does not simply push to master

Differences Between Regular and Hotfix Releases

Depending on the service, NewPipe Extractor uses web crawling or internal APIs. Both are subject to arbitrary changes by the service providers, like YouTube, SoundCloud or PeerTube. When they change something, NewPipe Extractor and thus NewPipe break instantly. Therefore, maintainers need to act quickly when it happens, and reduce our downtime as much as possible. The entire release cycle is therefore designed around this issue.

There is a difference between a release that introduces new features and a release that fixes an issue that occurred because YouTube, or some other service, changed their website (typically called a shutdown). Let's have a look at the characteristics of a regular release, and then the characteristics of a hotfix release.

Regular Releases

Regular releases are normal releases as they are done in any other app. Releases are always stored and tagged on master branch. The latest commit on master is always equal to the currently released version. No development is done on master. This ensures that we always have one branch with a stable/releasable version.

Feature Branching

The dev branch is used for development. Pushing to dev directly, however, is not allowed, since QA and testing should be done before adding something to the branch. This ensures that the development version works as stable a possible. In order to change something on the app, one may want to fork the dev branch and develop the changes in their own branch (this is called feature branching).

feature_branching

Make sure that both the dev branches, as well as the master branches of the extractor and the frontend, are compatible with each other. If the extractor's API is modified, make sure that frontend is compatible, or changed to become compatible, with these changes. If the PR that should make the frontend compatible again can not be merged, please do not merge the corresponding PR on the extractor either. This should make sure that any developer can run his changes on the fronted at any time.

Merging Features/Bugfixes

After finishing a feature, one should open up a Pull Request to the dev branch. From here, a maintainer can do Code review and Quality Assurance (QA). If you are a maintainer, please take care about the code architecture so corrosion or code shifting can be prevented. Please also prioritize code quality over functionality.
In short: cool function but bad code = no merge. Focus on keeping the code as clean as possible.

merge_feature_into_dev

An APK for testing is provided by GitHub Actions for every PR. Please ensure that this APK is tested thoroughly to prevent introducing regressions. Testing features needs to take into account that NewPipe is used on a brought variety of Android versions from KitKat to the latest, on custom ROMs like Lineage OS, CalyxOS or /e/ and different devices like phones, tablets and TVs.

Sometimes, the content of a PR changes over the time. Modify the PR's title if it does not represent the introduced changes anymore. After a maintainer merged the new feature into the dev branch, they should add the PR's title or a summary of the changes into the release notes.

Normal Releases

Once there are enough changes, and the maintainers believe that NewPipe is ready for a new version, they should prepare a new release.
Be aware of the rule that a release should never be done on a Friday. For NewPipe, this means: Don't do a release if you don't have time for it!!!

By following the steps listed in Release instructions, you can publish a stable version of NewPipe.

Hotfix Releases

this_is_fine

As aforementioned, NewPipe heavily relies on external components and might break at a random point of time. In order to keep the NewPipe's downtime as low as possible, when such a shutdown happens, we allow hotfixes.

  • A hotfix allows work on the master branch instead of the dev branch.
  • A hotfix MUST NOT contain any features or unrelated bugfixes.
  • A hotfix may only focus on fixing what caused the shutdown.

Hotfix Branch

Hotfixes work on the master branch. The dev branch has experimental changes that might have not been tested properly enough to be released, if at all. The master branch should always be the latest stable version of NewPipe. If the master branch breaks due to a shutdown, you should fix the master branch. Of course, you are not allowed to push to master directly, so you need to create a hotfix branch. If someone else is pushing a hotfix into master, and it works this can be considered as hotfix branch as well.

hotfix_branch

Releasing

If you fixed the issue and found it to be tested and reviewed well enough, you may publish a new version. You don't need to undergo the full release procedure of a regular release, which takes too much time. Keep in mind that if the hotfix might turn out to be broken after release, you should release another hotfix. It is important to release quickly for the sake of keeping NewPipe alive, and after all, a slightly broken version of NewPipe is better than a non-functional version ¯\_(ツ)_/¯. Here's what you do when releasing a hotfix:

  1. Merge the corresponding pull request in the extractor.
  2. Publish the new extractor version.
  3. Update the extractor version in the app's build.gradle file.
  4. Create a new release draft and put some info on the fix into the release note.
  5. Copy the release notes into the fastlane directory to create a changelog file.
  6. Increment the small minor version number and the versionCode.
  7. Generate a release APK (gradlew assembleRelease) and sign it (or get it signed by one of the other maintainers).
  8. Add the signed APK to the GitHub release notes.
  9. Click "Publish Release" .
  10. Publish the new version on F-Droid.
  11. Merge the changes from master into dev.
  12. Update the changelog for the website.

rebase_back_hotfix

Extractor releases

In general, the release process for extractor versions is not that complicated compared to app releases. The extractor has (in difference to the app) a decent test coverage. Additionally, the latest extractor version is typically tested in the app's latest dev version. Therefore, a long test phase is not needed when creating extractor releases.

To create a new extractor version, update the version in the extractor's build.gradle file as well as the version names in the README. Merge the dev branch into master. The same that applies the app's release notes also applies to the extractor's release notes.

When publishing an extractor release via GitHub on the master branch, a new JavaDoc version is generated and published automatically. Pleas keep an eye on the GitHub Action which is responsible for that. If changes in that release introduced invalid JavaDoc, the build fails and needs to be fixed. For this reason, you should check locally if there are any problems with the JavaDoc generation before publishing the new version.

Version Nomenclature

The version nomenclature of NewPipe is simple.

  • Major: The major version number (the number before the first dot) was 0 for years. The reason for this changed over time. First, I wanted this number to switch to 1 once NewPipe was feature complete. Now, I rather think of incrementing this number to 1 once we can ensure that NewPipe runs stable (part of which this documentation should help). After this, well, God knows what happens if we ever reach 1. ¯\_(ツ)_/¯
  • Minor: The minor version number (the number after the first dot) will be incremented if there is a major feature added to the app.
  • Small Minor: The small minor (the number after the second dot) is incremented when bug fixes or minor features are added to the app.

Version Nomenclature of the Extractor

Previously, the extractor was released together with the app, therefore the version number of the extractor was identical to the one of NewPipe itself.

We try to combine efforts to make NewPipe Extractor more independent of the app. The extractor is used by multiple other applications and therefore releasing extractor updates should not be coupled to app releases. However, maintainers need to keep an eye on making the app compatible with extractor changes.

Release Notes

Release notes should tell what was changed in the new version of the app. The release notes for NewPipe are stored in the GitHub draft for a new release. When a maintainer wants to add changes to the release notes, but there is no draft for a new version, they should create one.

Changes can be categorized into five basic types:

  • New: New features that got added to the app.
  • Improved: Improvements to the app or existing features
  • Fixed: Bugfixes
  • Translation: New translations
  • Development: Changes which address things "under the hood", which do not have any recognizable effect to the user; e.g. dependency updates or changes to the build process

When adding a PR to the release notes, increase the PR counter at the top of the draft and put the number before the PR summary / title. This helps the blog post authors to keep easily track of new PRs. Remove the numbers before publishing a new version :)

If there is a blog post covering the changes in more detail, make sure to link it on the top of the release notes. It would be a pity, if only a few people read the blog post after our wonderful writers put so much effort into creating it.

Changelog file

Maintainers need to provide a changelog file for each release. A changelog file is used by F-Droid to give a quick summary of the most important changes for a release. This file is placed in the /fastlane/metadata/android/en-US/changelogs directory and named <versionCode>.txt (whereas <versionCode> is the version code of the incoming release). Changelog files must not exceed 500 bytes. Be aware that the changelog is translated into multiple languages. A changelog written in English which almost hits 500 bytes can hardly be translated completely within this limit. This causes troubles for translators, because Weblate enforces the 500 bytes limit, too. For this reason it is recommended to keep the changelog at 400 bytes.

When creating the changelog file be aware of changes which were done in the extractor as well.
Before pushing the changelog to NewPipe's repo, ask other maintainers to review it.
After pushing the changelog to NewPipe's GitHub repo, updating Weblate is necessary. This enables translators to work on localized versions of the changelog before a release is tagged and published.

Publish on F-Droid

NewPipe is and supports open source software. For this reason, the preferred way to distribute the app is F-Droid. F-Droid is a catalogue of FOSS apps and also comes with an Android client which handles app updates. There are two ways to install NewPipe via F-Droid.

  1. Through the main F-Droid repository
    NewPipe's metadata file can be found in F-Droid's data repository on GitLab: https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/org.schabi.newpipe.yml
    This file is automatically updated by a bot when a new release is published on GitHub. It can take a few days until all new apps on F-Droid are built, signed and published.
    F-Droid also supports reproducible builds. Reproducible builds or deterministic builds allow someone else to retrieve the exact same binary as we get when building the app (except the signing of course).
    When the reproducible build feature is enabled for an app in F-Droid, they compare their binary to one provided by the author. If both are identical, F-Droid does not only publish the binary signed by themselves, but also the one signed by the author.
    Currently, NewPipe's builds are not deterministic, and we therefore cannot use that feature. Once the builds are deterministic again, the following steps need to be done to publish a new version on F-Droid:
    1. Install fdroidserver on your device.
    2. Clone the F-Droid Data repo from https://gitlab.com/fdroid/fdroiddata
    3. Add the new version to metadata/org.schabi.newpipe.yml
    4. Run fdroid signatures /path/to/newpipe.apk on the signed APK from within the repo.
    5. Create a MR on GitLab. An example commit containing all required changes can be found here.
  2. Through NewPipe's F-Droid repository
    F-Droid needs NewPipe's own F-Droid repo is available at https://archive.newpipe.net/fdroid/repo It is updated and maintained by @TheAssassin.