The problem
Each android application must have a bundled global manifest file. The manifest declares permissions used by the app, provides an information for android system, declares android components used in the app etc..
It contains sensitive information for us, such as unnecessary or dangerous permissions which we want to avoid, credentials, which we must include or important app properties. It is beneficial to have a way to quickly analyse the changes to manifest and have more control over them.
A bundled manifest is merged from the sources by gradle in the end of assemble task. Because of the nature of the sources we can not just check all git changes in every app and library manifest, because it could be affected by any of dependencies we have.
Goals
It would be much better if we can simply see and compare only the changes provided by the current feature pull request since the last approved build.
So, in simple words goals of this functionality are to:
- Show clean and simple way to check&verify the app manifest changes
- Show difference between current and the released app manifest
- Show difference between current and the main branch manifest
- Easily check new or removed app permissions
- Easily check attributes/credentials manifest changes
- Show a way to verify manifest changes using a pull request mechanism
- Support easy setup under any environment
Implementation
Having all the mentioned dependencies and environment variables provided the scripts below could be launched in any CI environment, such as circleCi, github actions, jenkins etc. Check the last paragraph for details.
Preconditions
- dependencies are available/installed & environment variables are set
assemble{build variant}gradle task has completed- deployment has completed before updating a baseline manifest for current release
High level steps
Let’s make our hands dirty. To implement those on a high level we should have the abilities:
- find merged manifest
- sort merged manifest. Gradle could merge manifest in any order for those with same priority. This step is required to reduce false attraction.
- store merged manifest as baseline
- input manifest must remain untouched
- update & store baseline manifest after release
- update & store baseline manifest for development branch
- find
diffbetween baseline and the current manifest - upload and display manifest changes in a corresponding pull request
Find merged manifest
To display manifest changes firstly we need a current comparable manifest.
When building your app, the Gradle build merges all manifest files into a single manifest file that’s packaged into your app.
It is what we need and it is going to be used in a diff task.
Let’s extract, sort and store the manifest here in some buffer directory to simplify an access to it further in the workflow. We may persist & publish the manifest merger log as well for debug purposes to be able to debug and understand the source of any manifest line in case some unexpected change has happened
Let’s extract, sort and store the manifest
Launch the script in a separated step after completion of assemble task.
| #!/bin/bash | |
| set -eo pipefail | |
| #extract_manifest.sh {TARGET_APP_NAME} {BUILD_TYPE} | |
| #launch example: ./ci/extract_manifest.sh app debug | |
| TARGET_APP_NAME=$1 | |
| BUILD_TYPE=$2 | |
| echo "extract manifest for $TARGET_APP_NAME" | |
| build_dir="$WORKSPACE/$TARGET_APP_NAME/build" | |
| source_file="$build_dir/intermediates/merged_manifest/$BUILD_TYPE/AndroidManifest.xml" | |
| source_log_file="$build_dir/outputs/logs/manifest-merger-$BUILD_TYPE-report.txt" | |
| manifest_target_dir="$MANIFESTS_WORKSPACE_DIR/$BUILD_TYPE/$TARGET_APP_NAME" | |
| mkdir -p "$manifest_target_dir" | |
| cp "$source_log_file" "$manifest_target_dir" || echo "Failed to copy manifest log for $TARGET_APP_NAME" | |
| cp "$source_file" "$manifest_target_dir" || echo "Failed to copy manifest for $TARGET_APP_NAME" | |
| export TARGET_MANIFEST_PATH=$source_file | |
| export SORTED_MANIFEST_PATH="$manifest_target_dir/AndroidManifest-sorted.xml" | |
| function failedManifestSorting { | |
| echo "Failed to sort© manifest for $TARGET_APP_NAME" | |
| exit 1 | |
| } | |
| #sort and store copy of manifest to $SORTED_MANIFEST_PATH | |
| python3 ./sort_manifest.py || failedManifestSorting | |
copy manifest & merge log to the temp folder
Note that in special cases there might be more than 1 app per project and several build configurations. That’s why we pass name and configurations like debug , release , qa as a parameters to the script
/intermediates/merged_manifest/$BUILD_TYPE/ and /outputs/logs/manifest-merger-$BUILD_TYPE-report.txt are paths in a build folder declared by gradle. These paths may change with gradle version change. The article is written for AGP 7.2.1
Prepare to compare: sort merged manifest
Assuming gradle could merge the manifest in a shuffled order to collect stable results we need to sort the resulting manifest using our custom algorithm.
We are interested only in changes and won’t modify initial manifest, so we are absolutely safe to do that. But better to keep usual order of the manifest tags for readability
So we our algorithm should sort the tags by:
- prioritising special top tags:
uses-sdk,uses-feature,uses-permission,permission - tag, as default ordering key
android:nameattribute to order same tags,<uses-permission>or<activity>for example
It would be hard to do the job with bash. That’s why we are going to attach python3 and tidy dependencies here. Python is needed for its powerful ElementTree and sorting mechanism, and tidy is used to prettify the results for better readability. You can take tidy config here
| import os | |
| import subprocess | |
| import xml.etree.ElementTree as Tree | |
| from xml.etree.ElementTree import Element | |
| for var in ["TARGET_MANIFEST_PATH", "SORTED_MANIFEST_PATH"]: | |
| if var not in os.environ: | |
| raise EnvironmentError("Required env variable {} is not set.".format(var)) | |
| TARGET_MANIFEST_PATH = os.getenv('TARGET_MANIFEST_PATH') | |
| SORTED_MANIFEST_PATH = os.getenv('SORTED_MANIFEST_PATH') | |
| print(f"sort manifest {TARGET_MANIFEST_PATH} to {SORTED_MANIFEST_PATH}") | |
| def get_sorting_weight(node: Element): | |
| """Return the sorting key of android manifest nodes. | |
| Some special tags should be on top | |
| """ | |
| if node.tag.startswith("uses-sdk"): | |
| return 1 | |
| if node.tag.startswith("uses-feature"): | |
| return 2 | |
| if node.tag.startswith("uses-permission"): | |
| return 3 | |
| if node.tag.startswith("permission"): | |
| return 4 | |
| return 5 | |
| def get_attribute(node: Element): | |
| name_attribute = "{http://schemas.android.com/apk/res/android}name" | |
| if name_attribute in node.attrib: | |
| return node.attrib[name_attribute] | |
| else: | |
| return "0" | |
| def sort_tree(node: Element): | |
| node[:] = sorted(node, key=lambda child: ( | |
| get_sorting_weight(child), | |
| child.tag, | |
| get_attribute(child))) | |
| if node.text is not None: | |
| node.text = node.text.strip() | |
| for item in node: | |
| sort_tree(item) | |
| def register_all_namespaces(filename): | |
| namespaces = dict([node for _, node in Tree.iterparse(filename, events=['start-ns'])]) | |
| for ns in namespaces: | |
| Tree.register_namespace(ns, namespaces[ns]) | |
| register_all_namespaces(TARGET_MANIFEST_PATH) | |
| # sort xml | |
| tree = Tree.ElementTree(file=TARGET_MANIFEST_PATH) | |
| root: Element = tree.getroot() | |
| sort_tree(root) | |
| output = Tree \ | |
| .tostring(root, encoding="utf-8", method="xml", short_empty_elements=True) \ | |
| .decode() | |
| # write result to file | |
| manifestFile = open(SORTED_MANIFEST_PATH, "w") | |
| manifestFile.write(output) | |
| manifestFile.truncate() | |
| subprocess.run(['tidy', '-config', "tidy.ini", '-o', SORTED_MANIFEST_PATH, SORTED_MANIFEST_PATH]) |
After the script completion we received a sorted clone of the merged manifest
Where to save a baseline?
We need to store latest release & stable development manifests somewhere, better not in the main project directory to not pollute main git tree with automatic commits. We are going to use sorted manifest from the previous step
Our decision was to store it in the special folder of github wiki. For sure, that’s not really how it is supposed to be used, but that fit our needs perfectly.
Firstly the manifest could be easily stored there by using standard git commands and then obtained the same way by request. Secondly, only md files are displayed in github ui, but any file could be stored. Thirdly, it hadn’t been used in our project anyway 🙂
Github wiki is a special git repository, so we should git clone it, move manifest from workspace dir to wiki dir, commit our changes (which would be particularly a baseline manifest update) and push back
Alternatively you could use an additional repository just for the purpose of persisting the data to not the main repository.
Github wiki is a special git repository, so we should clone it
| git clone "git@github.com:$GITHUB_PROJECT_USERNAME/$GITHUB_PROJECT_NAME.wiki.git" $WIKI_DIR |
move manifest from workspace dir to wiki dir
By using the provided BUILD_TYPE we may find and copy sorted manifest to wiki dir. There are might be several of them for multi-app project like mine.
| #!/bin/bash | |
| BUILD_TYPE=$1 | |
| usage() { | |
| echo "Copy app manifest to wiki directory" | |
| echo "Usage:" | |
| echo " update_wiki_manifests.sh {BUILD_TYPE}" | |
| echo " Example: update_wiki_manifests.sh debug" | |
| } | |
| update_wiki_manifests() { | |
| echo "Updating $BUILD_TYPE manifests for all apps..." | |
| manifest_dir="$MANIFESTS_WORKSPACE_DIR/$BUILD_TYPE" | |
| for dir in "$manifest_dir"/*; do | |
| basename=$(basename "$dir") | |
| mkdir -p "$WIKI_DIR/manifests/$BUILD_TYPE/$basename" | |
| cp "$dir/AndroidManifest-sorted.xml" "$WIKI_DIR/manifests/$BUILD_TYPE/$basename/." | |
| done | |
| } | |
| usage | |
| update_wiki_manifests |
commit our changes and push back.
| cd $WIKI_DIR || exit 1 | |
| ./push_changes_from_directory.sh |
Job Offers
OUR VIDEO RECOMMENDATION
Jobs
Universal script push_changes_from_directory.sh is used to push the changes. It would be too verbose to post it here, check the reference though.
Execute this step after assembleDebug gradle task in a main development branch, where all pull requests are merged to.
update main manifest baseline after `build debug` in a development branch
Or after deployment of a release by continuous delivery pipeline. Should be skipped for an ordinary pull request and be in a separated job because of that.
update release manifest baseline after deployment in a release branch
Finally! Find and post a difference to pull request
In the final step we are going to take sorted manifest which we generated previously, find a difference between baseline manifest and the one and post the result to github.
A we discussed previously there might be not only one application/library of what we want to track a manifest. Script just relies on the file structure which has been created before.
Linuxdiff util is used to collect those differences and store them to a file. I suppose that git diff command uses this util under hood as well.
| #!/bin/bash | |
| BUILD_TYPE=$1 | |
| manifest_diff_file="/manifest.diff" | |
| usage() { | |
| echo "Find a diff between baseline and a current manifest. Then post it to github" | |
| echo "Usage:" | |
| echo " post_manifest_diff.sh {BUILD_TYPE}" | |
| echo " Example: post_manifest_diff.sh release" | |
| echo "MANIFESTS_WORKSPACE_DIR, PULL_REQUEST_URL, PULL_REQUEST_ID, WIKI_DIR variables must be exported" | |
| } | |
| capture_diffs() { | |
| echo "Capturing $BUILD_TYPE merged manifest diffs..." | |
| cd "$MANIFESTS_WORKSPACE_DIR/$BUILD_TYPE" || exit 2 | |
| { | |
| printf "## Manifest Diff:\n" | |
| for dir in */; do | |
| echo "### $dir" | |
| printf "\n\`\`\`diff\n" | |
| diff "$WIKI_DIR/manifests/$BUILD_TYPE/$dir/AndroidManifest-sorted.xml" "$dir/AndroidManifest-sorted.xml" | |
| printf "\n\`\`\`\n" | |
| done | |
| } >> $manifest_diff_file | |
| cd .. | |
| } | |
| post_comment() { | |
| echo "Commenting on PR: $PULL_REQUEST_URL" | |
| gh pr comment "$PULL_REQUEST_ID" -F "$manifest_diff_file" || { | |
| echo "Failed to comment to pr $PULL_REQUEST_URL" | |
| exit 1 | |
| } | |
| } | |
| usage | |
| ./ci/github/remove_github_comment.sh "## Manifest Diff" | |
| capture_diffs | |
| post_comment | |
| rm -f $manifest_diff_file |
In a real world we may restart a build several times and don’t want to spam a lot in pr comments. So we use remove_github_comment.sh script for cleanup. The message tag should be passed to it.
Dependencies
- bash
- python3.9
- html-tidy library 5.9.14
- diff tool
- github cli
The last one is not really necessary, it is only an api of pull request management system to post the result. If you use bitbucket, for example, you should use its api for that.
Environmental variables
I have used several environmental variables for the scripts. There are most of them:
WORKSPACE: workspace path, typically root directory of the project for CI build
MANIFESTS_WORKSPACE_DIR: directory in the workspace where we keep sorted manifest between CI jobs
WIKI_DIR: directory in github wiki where we store baseline manifests
PULL_REQUEST_URL: browser url of a pull request
PULL_REQUEST_ID: id of the current pull request, last path segment of PULL_REQUEST_ID
GITHUB_TOKEN: personal user’s github token which must be provided to operate with github api directly. You should have a service account, which is going to post the comments in a pull request
GITHUB_PROJECT_USERNAME: name of a user or an organisation where the project is located
GITHUB_PROJECT_NAME: name of a repository where the project is hosted
That’s it!
Thank you for reading and have fun while integrating! Any comments are welcome
If you found this post to be useful and interesting, you may give some claps and recommend it 🙂
> 500 claps and i’ll build a corresponding circleCi orb and github action 🔥
This article was originally published on proandroiddev.com on October 15, 2022
