Blog Infos
Author
Published
Topics
Published
Topics

 

The problem

Each android application must have a bundled global manifest file. The manifest declares permissions used by the app, provides an information for android system, declares android components used in the app etc..

It contains sensitive information for us, such as unnecessary or dangerous permissions which we want to avoid, credentials, which we must include or important app properties. It is beneficial to have a way to quickly analyse the changes to manifest and have more control over them.

A bundled manifest is merged from the sources by gradle in the end of assemble task. Because of the nature of the sources we can not just check all git changes in every app and library manifest, because it could be affected by any of dependencies we have.

Goals

It would be much better if we can simply see and compare only the changes provided by the current feature pull request since the last approved build.

So, in simple words goals of this functionality are to:

  • Show clean and simple way to check&verify the app manifest changes
  • Show difference between current and the released app manifest
  • Show difference between current and the main branch manifest
  • Easily check new or removed app permissions
  • Easily check attributes/credentials manifest changes
  • Show a way to verify manifest changes using a pull request mechanism
  • Support easy setup under any environment
Implementation

Having all the mentioned dependencies and environment variables provided the scripts below could be launched in any CI environment, such as circleCi, github actions, jenkins etc. Check the last paragraph for details.

Preconditions
  • dependencies are available/installed & environment variables are set
  • assemble{build variant}gradle task has completed
  • deployment has completed before updating a baseline manifest for current release
High level steps

 

 

Let’s make our hands dirty. To implement those on a high level we should have the abilities:

  • find merged manifest
  • sort merged manifest. Gradle could merge manifest in any order for those with same priority. This step is required to reduce false attraction.
  • store merged manifest as baseline
  • input manifest must remain untouched
  • update & store baseline manifest after release
  • update & store baseline manifest for development branch
  • find diff between baseline and the current manifest
  • upload and display manifest changes in a corresponding pull request
Find merged manifest

To display manifest changes firstly we need a current comparable manifest.

When building your app, the Gradle build merges all manifest files into a single manifest file that’s packaged into your app.

It is what we need and it is going to be used in a diff task.

Let’s extract, sort and store the manifest here in some buffer directory to simplify an access to it further in the workflow. We may persist & publish the manifest merger log as well for debug purposes to be able to debug and understand the source of any manifest line in case some unexpected change has happened

Let’s extract, sort and store the manifest

Launch the script in a separated step after completion of assemble task.

#!/bin/bash
set -eo pipefail
#extract_manifest.sh {TARGET_APP_NAME} {BUILD_TYPE}
#launch example: ./ci/extract_manifest.sh app debug
TARGET_APP_NAME=$1
BUILD_TYPE=$2
echo "extract manifest for $TARGET_APP_NAME"
build_dir="$WORKSPACE/$TARGET_APP_NAME/build"
source_file="$build_dir/intermediates/merged_manifest/$BUILD_TYPE/AndroidManifest.xml"
source_log_file="$build_dir/outputs/logs/manifest-merger-$BUILD_TYPE-report.txt"
manifest_target_dir="$MANIFESTS_WORKSPACE_DIR/$BUILD_TYPE/$TARGET_APP_NAME"
mkdir -p "$manifest_target_dir"
cp "$source_log_file" "$manifest_target_dir" || echo "Failed to copy manifest log for $TARGET_APP_NAME"
cp "$source_file" "$manifest_target_dir" || echo "Failed to copy manifest for $TARGET_APP_NAME"
export TARGET_MANIFEST_PATH=$source_file
export SORTED_MANIFEST_PATH="$manifest_target_dir/AndroidManifest-sorted.xml"
function failedManifestSorting {
echo "Failed to sort&copy manifest for $TARGET_APP_NAME"
exit 1
}
#sort and store copy of manifest to $SORTED_MANIFEST_PATH
python3 ./sort_manifest.py || failedManifestSorting

copy manifest & merge log to the temp folder

 

Note that in special cases there might be more than 1 app per project and several build configurations. That’s why we pass name and configurations like debug , release , qa as a parameters to the script

/intermediates/merged_manifest/$BUILD_TYPE/ and /outputs/logs/manifest-merger-$BUILD_TYPE-report.txt are paths in a build folder declared by gradle. These paths may change with gradle version change. The article is written for AGP 7.2.1

Prepare to compare: sort merged manifest

 

 

Assuming gradle could merge the manifest in a shuffled order to collect stable results we need to sort the resulting manifest using our custom algorithm.

We are interested only in changes and won’t modify initial manifest, so we are absolutely safe to do that. But better to keep usual order of the manifest tags for readability

So we our algorithm should sort the tags by:

  • prioritising special top tags: uses-sdk , uses-feature , uses-permission , permission
  • tag, as default ordering key
  • android:name attribute to order same tags, <uses-permission>or <activity> for example

It would be hard to do the job with bash. That’s why we are going to attach python3 and tidy dependencies here. Python is needed for its powerful ElementTree and sorting mechanism, and tidy is used to prettify the results for better readability. You can take tidy config here

import os
import subprocess
import xml.etree.ElementTree as Tree
from xml.etree.ElementTree import Element
for var in ["TARGET_MANIFEST_PATH", "SORTED_MANIFEST_PATH"]:
if var not in os.environ:
raise EnvironmentError("Required env variable {} is not set.".format(var))
TARGET_MANIFEST_PATH = os.getenv('TARGET_MANIFEST_PATH')
SORTED_MANIFEST_PATH = os.getenv('SORTED_MANIFEST_PATH')
print(f"sort manifest {TARGET_MANIFEST_PATH} to {SORTED_MANIFEST_PATH}")
def get_sorting_weight(node: Element):
"""Return the sorting key of android manifest nodes.
Some special tags should be on top
"""
if node.tag.startswith("uses-sdk"):
return 1
if node.tag.startswith("uses-feature"):
return 2
if node.tag.startswith("uses-permission"):
return 3
if node.tag.startswith("permission"):
return 4
return 5
def get_attribute(node: Element):
name_attribute = "{http://schemas.android.com/apk/res/android}name"
if name_attribute in node.attrib:
return node.attrib[name_attribute]
else:
return "0"
def sort_tree(node: Element):
node[:] = sorted(node, key=lambda child: (
get_sorting_weight(child),
child.tag,
get_attribute(child)))
if node.text is not None:
node.text = node.text.strip()
for item in node:
sort_tree(item)
def register_all_namespaces(filename):
namespaces = dict([node for _, node in Tree.iterparse(filename, events=['start-ns'])])
for ns in namespaces:
Tree.register_namespace(ns, namespaces[ns])
register_all_namespaces(TARGET_MANIFEST_PATH)
# sort xml
tree = Tree.ElementTree(file=TARGET_MANIFEST_PATH)
root: Element = tree.getroot()
sort_tree(root)
output = Tree \
.tostring(root, encoding="utf-8", method="xml", short_empty_elements=True) \
.decode()
# write result to file
manifestFile = open(SORTED_MANIFEST_PATH, "w")
manifestFile.write(output)
manifestFile.truncate()
subprocess.run(['tidy', '-config', "tidy.ini", '-o', SORTED_MANIFEST_PATH, SORTED_MANIFEST_PATH])

After the script completion we received a sorted clone of the merged manifest

Where to save a baseline?

We need to store latest release & stable development manifests somewhere, better not in the main project directory to not pollute main git tree with automatic commits. We are going to use sorted manifest from the previous step

Our decision was to store it in the special folder of github wiki. For sure, that’s not really how it is supposed to be used, but that fit our needs perfectly.

Firstly the manifest could be easily stored there by using standard git commands and then obtained the same way by request. Secondly, only md files are displayed in github ui, but any file could be stored. Thirdly, it hadn’t been used in our project anyway 🙂

Github wiki is a special git repository, so we should git clone it, move manifest from workspace dir to wiki dir, commit our changes (which would be particularly a baseline manifest update) and push back

Alternatively you could use an additional repository just for the purpose of persisting the data to not the main repository.

Github wiki is a special git repository, so we should clone it

git clone "git@github.com:$GITHUB_PROJECT_USERNAME/$GITHUB_PROJECT_NAME.wiki.git" $WIKI_DIR
view raw clone_wiki hosted with ❤ by GitHub

move manifest from workspace dir to wiki dir

By using the provided BUILD_TYPE we may find and copy sorted manifest to wiki dir. There are might be several of them for multi-app project like mine.

#!/bin/bash
BUILD_TYPE=$1
usage() {
echo "Copy app manifest to wiki directory"
echo "Usage:"
echo " update_wiki_manifests.sh {BUILD_TYPE}"
echo " Example: update_wiki_manifests.sh debug"
}
update_wiki_manifests() {
echo "Updating $BUILD_TYPE manifests for all apps..."
manifest_dir="$MANIFESTS_WORKSPACE_DIR/$BUILD_TYPE"
for dir in "$manifest_dir"/*; do
basename=$(basename "$dir")
mkdir -p "$WIKI_DIR/manifests/$BUILD_TYPE/$basename"
cp "$dir/AndroidManifest-sorted.xml" "$WIKI_DIR/manifests/$BUILD_TYPE/$basename/."
done
}
usage
update_wiki_manifests

commit our changes and push back.

cd $WIKI_DIR || exit 1
./push_changes_from_directory.sh

Job Offers

Job Offers


    Android Software Engineer (f/m/d)

    Paradox Cat GmbH
    Munich
    • Full Time
    apply now

    Senior Android Software Engineer (f/m/d)

    Paradox Cat GmbH
    Munich
    • Full Time
    apply now

OUR VIDEO RECOMMENDATION

,

Creating & Scaling Meaningful Performance Tests in CI

Performance testing is easier and more important than ever, especially with the recent emergence of Macro/Microbenchmark and other libraries/tools.
Watch Video

Creating & Scaling Meaningful Performance Tests in CI

Ryan Brooks
Android Lead
Emerge Tools

Creating & Scaling Meaningful Performance Tests in CI

Ryan Brooks
Android Lead
Emerge Tools

Creating & Scaling Meaningful Performance Tests in CI

Ryan Brooks
Android Lead
Emerge Tools

Jobs

Universal script push_changes_from_directory.sh is used to push the changes. It would be too verbose to post it here, check the reference though.

Execute this step after assembleDebug gradle task in a main development branch, where all pull requests are merged to.

update main manifest baseline after `build debug` in a development branch

Or after deployment of a release by continuous delivery pipeline. Should be skipped for an ordinary pull request and be in a separated job because of that.

update release manifest baseline after deployment in a release branch

Finally! Find and post a difference to pull request

In the final step we are going to take sorted manifest which we generated previously, find a difference between baseline manifest and the one and post the result to github.

A we discussed previously there might be not only one application/library of what we want to track a manifest. Script just relies on the file structure which has been created before.

Linuxdiff util is used to collect those differences and store them to a file. I suppose that git diff command uses this util under hood as well.

#!/bin/bash
BUILD_TYPE=$1
manifest_diff_file="/manifest.diff"
usage() {
echo "Find a diff between baseline and a current manifest. Then post it to github"
echo "Usage:"
echo " post_manifest_diff.sh {BUILD_TYPE}"
echo " Example: post_manifest_diff.sh release"
echo "MANIFESTS_WORKSPACE_DIR, PULL_REQUEST_URL, PULL_REQUEST_ID, WIKI_DIR variables must be exported"
}
capture_diffs() {
echo "Capturing $BUILD_TYPE merged manifest diffs..."
cd "$MANIFESTS_WORKSPACE_DIR/$BUILD_TYPE" || exit 2
{
printf "## Manifest Diff:\n"
for dir in */; do
echo "### $dir"
printf "\n\`\`\`diff\n"
diff "$WIKI_DIR/manifests/$BUILD_TYPE/$dir/AndroidManifest-sorted.xml" "$dir/AndroidManifest-sorted.xml"
printf "\n\`\`\`\n"
done
} >> $manifest_diff_file
cd ..
}
post_comment() {
echo "Commenting on PR: $PULL_REQUEST_URL"
gh pr comment "$PULL_REQUEST_ID" -F "$manifest_diff_file" || {
echo "Failed to comment to pr $PULL_REQUEST_URL"
exit 1
}
}
usage
./ci/github/remove_github_comment.sh "## Manifest Diff"
capture_diffs
post_comment
rm -f $manifest_diff_file

In a real world we may restart a build several times and don’t want to spam a lot in pr comments. So we use remove_github_comment.sh script for cleanup. The message tag should be passed to it.

Dependencies

The last one is not really necessary, it is only an api of pull request management system to post the result. If you use bitbucket, for example, you should use its api for that.

Environmental variables

I have used several environmental variables for the scripts. There are most of them:

WORKSPACE: workspace path, typically root directory of the project for CI build
MANIFESTS_WORKSPACE_DIR: directory in the workspace where we keep sorted manifest between CI jobs
WIKI_DIR: directory in github wiki where we store baseline manifests
PULL_REQUEST_URL: browser url of a pull request
PULL_REQUEST_ID: id of the current pull request, last path segment of PULL_REQUEST_ID
GITHUB_TOKEN: personal user’s github token which must be provided to operate with github api directly. You should have a service account, which is going to post the comments in a pull request
GITHUB_PROJECT_USERNAME: name of a user or an organisation where the project is located
GITHUB_PROJECT_NAME: name of a repository where the project is hosted

That’s it!

Thank you for reading and have fun while integrating! Any comments are welcome

If you found this post to be useful and interesting, you may give some claps and recommend it 🙂

> 500 claps and i’ll build a corresponding circleCi orb and github action 🔥

This article was originally published on proandroiddev.com on October 15, 2022

YOU MAY BE INTERESTED IN

YOU MAY BE INTERESTED IN

blog
Screenshot tests are the most effective way how to test your View layer. And…
READ MORE

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.

Menu