Blog Infos
Author
Published
Topics
Published
Topics

Photo by Choong Deng Xiang on Unsplash

 

In Part 1, we discussed about the importance of continuous monitoring of APK size and started implementation of Github Action workflow where the first job was Generate builds for both base and merge branches. In this post we’re going to implement the second job i.e. Generate the size diff report of the same workflow.

Creating Python Script (to generate the size diff report)

This script can itself be divided into following parts:

  • Setup
  • Generate dictionary – Component category to size mapping
  • Generate HTML report – Use the dictionaries generated above to create a human readable HTML output in tabular format

Setup phase

# read arguments passed to this script
apk_1_sha = sys.argv[1]
apk_2_sha = sys.argv[2]
apk_1_name = f"{apk_1_sha}.apk"
apk_2_name = f"{apk_2_sha}.apk"
# apk_analyzer_path location will change based on the GHA runner that you're using i.e. mac/windows/ubuntu etc
apk_analyzer_path = "/usr/local/lib/android/sdk/cmdline-tools/latest/bin/apkanalyzer"
kb_in_bytes = 1024
mb_in_bytes = 1024 * 1024
# generate dictionaries for the apk components size
components_1 = get_apk_components(apk_1_name, 'download-size')
components_2 = get_apk_components(apk_2_name, 'download-size')
generate_size_diff_html()

compare_apks.py – Setup phase

 

  • Line 2 & 3 – Here we’re parsing the arguments passed while executing the script
  • Line 5 & 6 – We append .apk suffix to the above parsed arguments
  • Line 9 – This location will be dependent on the GHA runner you’re using. This can be made dynamic based on OS check.
  • Line 15 & 16 – Here we invoke get_apk_components function to generate the dictionaries having category name (i.e. Native libraries (arm64-v8a), Classes, Resources, etc) to size mapping.
  • Line 18 – Here we invoke generate_size_diff_html function which will generate the HTML report comparing the size difference between different categories.

Generate dictionary phase

# generate dictionary of the grouped contents of an apk file
def get_apk_components(apk_file, size_type):
command = f"{apk_analyzer_path} files list --{size_type} {apk_file}"
files_with_size_string = execute_command(command)
files_with_size_list = files_with_size_string.split('\n')
components = {}
for item in files_with_size_list:
size_and_file_name = item.split('\t')
# this will filter out empty lines and just the lines with size and no file name
if len(size_and_file_name) == 2 and len(size_and_file_name[1]) > 1:
size = int(size_and_file_name[0])
file_name = size_and_file_name[1]
if file_name == '/lib/arm64-v8a/':
update_if_present(components, 'Native libraries (arm64-v8a)', size)
elif file_name.startswith('/classes') and file_name.endswith('.dex'):
update_if_present(components, 'Classes', size)
elif file_name == '/resources.arsc' or file_name == '/res/':
update_if_present(components, 'Resources', size)
elif file_name == '/assets/':
update_if_present(components, 'Assets', size)
elif not file_name.startswith('/lib/') and not file_name.startswith('/classes') and not file_name.startswith('/resources.arsc') and not file_name.startswith('/res/') and not file_name.startswith('/assets/') and not file_name.endswith('/'):
update_if_present(components, 'Others', size)
return components
# shell command executor
def execute_command(command):
# Run the command using subprocess
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = process.communicate()
return output.decode()
# add/update the value (i.e. size) based on the presence of provided key
def update_if_present(components, key, value):
if key in components:
components[key] = components[key] + value
else:
components[key] = value

compare_apks.py – Dictionary Build phase

 

In this phase we’re going to use apk_1_name & apk_2_name generated in Setup phase.

  • Line 32–37 – Here we define a function execute_command, whose sole responsibility is to execute the shell command and return the decoded result.
  • Line 40–44 – Here we update the component dictionary value (i.e. size) in case the provided key i.e. (category name) is already present or add a new entry in case the key is absent.
  • Line 3 – Here we build the apkanalyzer command to get the list of all the files bundled into the apk along with their download size. In case you want to read more about different arguments that you can pass, you can refer this. It took me while to produce the proper command 😅.
  • Line 5 – Here we invoke execute_command with the command generated in Line 3. The output of this command should look like below where each line contains file download size in bytes“\t” separator, & file name.

apkanalyzer command output example

 

  • Line 6 – Split the files_with_size_string with new line character and store it in files_with_size_list.
  • Line 10 – Here we start looping through each item present in files_with_size_list.
  • Line 11 – Here we further split each item present in files_with_size_list using ‘\t’.
  • Line 14 – Here we check if the output from Line 11 resulted in a list of exactly 2 items (i.e. file_size & file_name) and file_name is atleast longer than 1 character. This is to filter invalid entries.
  • Line 18–27 – Here we add/update the value of each dictionary entry (i.e. Native libraries (arm64-v8a), Classes, Resources, etc) based on file_name matching criteria. NOTE: This step should be configured based on how you want to categorise your size diff report.

We‘ll be running get_apk_components funtion for both of the apks and end up with component_1 & component_2 dictionaries as mentioned in Setup phase.

Generate HTML report phase

# generate html file containing size diff in HRF
def generate_size_diff_html():
html = "<html>"
html += "<body><h1>Download Size Diff Report</h1><h3>Affected Products</h3>"
html += "<ul><li><h4><code>release</code></h4><table>"
html += f"<tr><th>Component</th><th>Base ({apk_1_sha})</th><th>Merge ({apk_2_sha})</th><th>Diff</th></tr>"
# print diff of each components of both of the apk files
for component in set(components_1.keys()) | set(components_2.keys()):
size_1 = components_1.get(component, 0)
size_2 = components_2.get(component, 0)
html += f"<tr><td>{component}</td><td>{format_size(size_1)}</td><td>{format_size(size_2)}</td><td>{format_size_with_indicator(size_2 - size_1)}</td></tr>"
# calculate size of the apk files
apk_1_download_size = apk_size(apk_1_name, 'download-size')
apk_2_download_size = apk_size(apk_2_name, 'download-size')
html += f"<tr><td>apk (Download Size)</td><td>{format_size(apk_1_download_size)}</td><td>{format_size(apk_2_download_size)}</td><td>{format_size_with_indicator(apk_2_download_size - apk_1_download_size)}</td></tr>"
html += "</li></ul></table></body></html>"
with open("apk_size_diff_report.html", "w") as file:
file.write(html)
# format bytes to KB or MB. Any size less than a KB is treated as 0KB
def format_size(size):
if abs(size) > mb_in_bytes:
return f"{round(size / mb_in_bytes, 2)} MB"
elif abs(size) > kb_in_bytes:
return f"{round(size / kb_in_bytes, 2)} KB"
else:
return "0 KB"
# add an indicator to highlight the size diff
def format_size_with_indicator(size):
size_indicator = "🔴" if size > kb_in_bytes else "🟢"
return f"{format_size(size)} {size_indicator}"
# get apk size based on size_type i.e. file-size or download-size
def apk_size(apk_file, size_type):
command = f"{apk_analyzer_path} apk {size_type} {apk_file}"
return int(execute_command(command))

compare_apks.py – Generate HTML report phase

Job Offers

Job Offers

There are currently no vacancies.

OUR VIDEO RECOMMENDATION

,

Pruning Your App- Good Practices for Reducing App Size

App size reduction should be a critical aspect of app development, and is especially necessary for end users who may have restricted device storage or limited data plans.
Watch Video

Pruning Your App- Good Practices for Reducing App Size

Chrystian Vieyra
Engineering Manager
Comcast

Pruning Your App- Good Practices for Reducing App Size

Chrystian Vieyra
Engineering Manager
Comcast

Pruning Your App- Good Practices for Reducing App Size

Chrystian Vieyra
Engineering Manager
Comcast

Jobs

In this phase we’re going to generate an HTML table comparing the size of each dictionary entries.

  • Line 25–31 – Here we format the size (of dictionary entries and not individual files) from bytes to KB/MB. In case the size is less than a KB then it’s considered as 0KB.
  • Line 34–37 – Based on the size argument (i.e. size diff of same kind of entry in both the dictionaries), we add 🟢 (for size reduction) or 🔴 (for size increase) indicator.
  • Line 40–43 – Calculate overall apk download size using apkanalyzer command.
  • Line 3–6 – Here we add HTML body content with some text and table row containing ComponentBaseMerge & Diff items.
  • Line 9–12 – Here we iterate through the entries of component_1 & component_2 dictionaries and add an entry to the table row.
  • Line 15&16 – Here we invoke apk_size function to generate the download file size of both base and merge apks.
  • Line 18 – Add one final entry to our size diff table containing the total download size comparison between the base and merge apks.
  • Line 21 – We write the generated HTML to apk_size_diff_report.html file which we will upload for use in the final job i.e. Add the size diff report as a PR comment.

Finally the whole script should look like below:

import sys
import subprocess
# generate dictionary of the grouped contents of an apk file
def get_apk_components(apk_file, size_type):
command = f"{apk_analyzer_path} files list --{size_type} {apk_file}"
files_with_size_string = execute_command(command)
files_with_size_list = files_with_size_string.split('\n')
components = {}
for item in files_with_size_list:
size_and_file_name = item.split('\t')
# this will filter out empty lines and just the lines with size and no file name
if len(size_and_file_name) == 2 and len(size_and_file_name[1]) > 1:
size = int(size_and_file_name[0])
file_name = size_and_file_name[1]
if file_name == '/lib/arm64-v8a/':
update_if_present(components, 'Native libraries (arm64-v8a)', size)
elif file_name.startswith('/classes') and file_name.endswith('.dex'):
update_if_present(components, 'Classes', size)
elif file_name == '/resources.arsc' or file_name == '/res/':
update_if_present(components, 'Resources', size)
elif file_name == '/assets/':
update_if_present(components, 'Assets', size)
elif not file_name.startswith('/lib/') and not file_name.startswith('/classes') and not file_name.startswith('/resources.arsc') and not file_name.startswith('/res/') and not file_name.startswith('/assets/') and not file_name.endswith('/'):
update_if_present(components, 'Others', size)
return components
# shell command executor
def execute_command(command):
# Run the command using subprocess
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = process.communicate()
return output.decode()
# add/update the value (i.e. size) based on the presence of provided key
def update_if_present(components, key, value):
if key in components:
components[key] = components[key] + value
else:
components[key] = value
# generate html file containing size diff in HRF
def generate_size_diff_html():
html = "<html>"
html += "<body><h1>Download Size Diff Report</h1><h3>Affected Products</h3>"
html += "<ul><li><h4><code>release</code></h4><table>"
html += f"<tr><th>Component</th><th>Base ({apk_1_sha})</th><th>Merge ({apk_2_sha})</th><th>Diff</th></tr>"
# print diff of each components of both of the apk files
for component in set(components_1.keys()) | set(components_2.keys()):
size_1 = components_1.get(component, 0)
size_2 = components_2.get(component, 0)
html += f"<tr><td>{component}</td><td>{format_size(size_1)}</td><td>{format_size(size_2)}</td><td>{format_size_with_indicator(size_2 - size_1)}</td></tr>"
# calculate size of the apk files
apk_1_download_size = apk_size(apk_1_name, 'download-size')
apk_2_download_size = apk_size(apk_2_name, 'download-size')
html += f"<tr><td>apk (Download Size)</td><td>{format_size(apk_1_download_size)}</td><td>{format_size(apk_2_download_size)}</td><td>{format_size_with_indicator(apk_2_download_size - apk_1_download_size)}</td></tr>"
html += "</li></ul></table></body></html>"
with open("apk_size_diff_report.html", "w") as file:
file.write(html)
# format bytes to KB or MB. Any size less than a KB is treated as 0KB
def format_size(size):
if abs(size) > mb_in_bytes:
return f"{round(size / mb_in_bytes, 2)} MB"
elif abs(size) > kb_in_bytes:
return f"{round(size / kb_in_bytes, 2)} KB"
else:
return "0 KB"
# add an indicator to highlight the size diff
def format_size_with_indicator(size):
size_indicator = "🔴" if size > kb_in_bytes else "🟢"
return f"{format_size(size)} {size_indicator}"
# get apk size based on size_type i.e. file-size or download-size
def apk_size(apk_file, size_type):
command = f"{apk_analyzer_path} apk {size_type} {apk_file}"
return int(execute_command(command))
# read arguments passed to this script
apk_1_sha = sys.argv[1]
apk_2_sha = sys.argv[2]
apk_1_name = f"{apk_1_sha}.apk"
apk_2_name = f"{apk_2_sha}.apk"
# apk_analyzer_path location will change based on the GHA runner that you're using i.e. mac/windows/ubuntu etc
apk_analyzer_path = "/usr/local/lib/android/sdk/cmdline-tools/latest/bin/apkanalyzer"
kb_in_bytes = 1024
mb_in_bytes = 1024 * 1024
# generate dictionaries for the apk components size
components_1 = get_apk_components(apk_1_name, 'download-size')
components_2 = get_apk_components(apk_2_name, 'download-size')
generate_size_diff_html()
view raw compare_apks.py hosted with ❤ by GitHub

Python script to compare size contribution from high level components

 

Note: You’ve to put this script in .github/scripts/ directory to make the following job run.

Since the apk size compare script is now ready we can start adding second job to our existing workflow that we created previously.

Job 2: Generate the size diff report
name: Size Metrics
...
# content is omitted as it's already shown earlier
...
env:
size_diff_report_file: apk_size_diff_report.html
jobs:
build:
...
# content is omitted as it's already shown earlier
...
size_diff_report:
needs: build
name: Size Report
runs-on: ubuntu-latest
steps:
# Step 1
- uses: actions/checkout@v3
# Step 2
- uses: actions/download-artifact@v3
with:
name: ${{ needs.build.outputs.head_short_sha }}.apk
# Step 3
- uses: actions/download-artifact@v3
with:
name: ${{ needs.build.outputs.base_short_sha }}.apk
# Step 4
# here we use our custom python script to generate file matching size_diff_report_file
- name: Generate size report
run: |
chmod +x ./.github/scripts/compare_apks.py
python3 ./.github/scripts/compare_apks.py ${{ needs.build.outputs.base_short_sha }} ${{ needs.build.outputs.head_short_sha }}
# Step 5
- name: Upload Size Report html file
uses: actions/upload-artifact@v3
with:
name: ${{ env.size_diff_report_file }}
path: ${{ env.size_diff_report_file }}

Size Metrics workflow with only size_diff_report job

 

To focus only on this job we’ve omitted top-level config as well as the contents of build job.

Line 7–8 – Here we’ve defined the size_diff_report_file environment variable as we’re going to use it in this as well as next job. On Line 17 we define dependency on previous build job using needs config, this will ensure that this job will run only if build job gets completed successfully.

Steps:

  1. Checkout the code
  2. Download head/merge apk that we uploaded in build job
  3. Download base apk that we uploaded in build job
  4. Execute compare_apks.py script which will generate apk_size_diff_report.html file.
  5. Upload the apk_size_diff_report.html for consumption in next job i.e. Add the size diff report as a PR comment.
Job 3: Add the size diff report as a PR comment
name: Size Metrics
...
# content is omitted as it's already shown earlier
...
env:
size_diff_report_file: apk_size_diff_report.html
jobs:
build:
...
# content is omitted as it's already shown earlier
...
size_diff_report:
...
# content is omitted as it's already shown earlier
...
pr_comment:
needs: size_diff_report
name: PR Comment (Size Report)
runs-on: ubuntu-latest
steps:
# Step 1
- uses: actions/download-artifact@v3
with:
name: ${{ env.size_diff_report_file }}
# Step 2
- name: Add PR Comment
uses: actions/github-script@v6
with:
script: |
const fs = require('fs');
try {
const filePath = 'apk_size_diff_report.html'
const sizeReportContent = fs.readFileSync(filePath, 'utf8')
const result = await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: sizeReportContent
})
console.log(result)
} catch (error) {
console.error('Error reading file:', error);
}

Size Metrics workflow with only size_diff_report job

 

To focus only on this job we’ve omitted top-level config as well as the contents of both build & size_diff_report jobs.

This is going to be smallest as well as simplest job all 3 jobs as we’re just going to download the size diff report file and then use it’s content to add a PR comment.

On Line 22 we define dependency on previous size_diff_report job using needs config, this will ensure that this job will run only if size_diff_report job gets completed successfully.

Steps:

  1. Download the apk_size_diff_report.html file that we generated in previous job.
  2. Here we first parse the content of the apk_size_diff_report.html file to a string and then add it as PR comment to increase the PR reviewer’s visibility on the apk size impact.

Whole workflow file should look like below:

name: Size Metrics
# cancel in-progress workflow if new commits are pushed to same head branch
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true
on:
pull_request:
branches: [ "master", "main", "release/*", "feature/*" ]
env:
size_diff_report_file: apk_size_diff_report.html
jobs:
build:
name: Build apk
runs-on: ubuntu-latest
outputs:
base_short_sha: ${{ steps.base-short-sha.outputs.base_short_sha }}
head_short_sha: ${{ steps.head-short-sha.outputs.head_short_sha }}
steps:
- name: Checkout base (PR target) branch
uses: actions/checkout@v3
with:
ref: ${{ github.base_ref }}
# this step is optional if you don't use JDK 17 as ubuntu-latest runner already
# has java setup which defaults to JDK 11
- name: Setup Java
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '17'
# this step is optional and should be added in case you get "Failed to install the following
# Android SDK packages as some licences have not been accepted" error
- name: Setup Android SDK
uses: android-actions/setup-android@v2
# setup cache so that every build trigger doesn't require downloading all the
# dependencies used in project again and again saving tonnes of time
- name: Setup Gradle & Android SDK Cache
uses: actions/cache@v3
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
/usr/local/lib/android/sdk/build-tools
/usr/local/lib/android/sdk/system-images
key: ${{ runner.os }}-${{ hashFiles('**/*.gradle*') }}-${{ hashFiles('**/gradle/wrapper/gradle-wrapper.properties') }}-${{ hashFiles('**/buildSrc/**/*.kt') }}
- name: Generate shorter github.sha for base branch
id: base-short-sha
run: |
chmod +x ./.github/scripts/commit_short_sha.sh
short_sha=$(./.github/scripts/commit_short_sha.sh)
echo "base_short_sha=${short_sha}" >> "$GITHUB_ENV"
echo "base_short_sha=${short_sha}" >> "$GITHUB_OUTPUT"
- name: Build release APK for base branch and rename it to match base-short-sha result
run: |
./gradlew assembleRelease -PtargetDensity=xxhdpi -PtargetAbi=arm64-v8a -PresConfig=en -Pandroid.testInstrumentationRunnerArguments.androidx.benchmark.enabledRules=BaselineProfile
mv app/build/outputs/apk/release/app-xxhdpiArm64-v8a-release-unsigned.apk app/build/outputs/apk/release/${{ env.base_short_sha }}.apk
- name: Upload APK
uses: actions/upload-artifact@v3
with:
name: ${{ env.base_short_sha }}.apk
path: app/build/outputs/apk/release/${{ env.base_short_sha }}.apk
- name: Checkout head branch
uses: actions/checkout@v3
with:
ref: ${{ github.head_ref }}
fetch-depth: 0
- name: Generate shorter github.sha for head branch
id: head-short-sha
run: |
chmod +x ./.github/scripts/commit_short_sha.sh
short_sha=$(./.github/scripts/commit_short_sha.sh)
echo "head_short_sha=${short_sha}" >> "$GITHUB_ENV"
echo "head_short_sha=${short_sha}" >> "$GITHUB_OUTPUT"
- name: Merge base (PR target) branch
run: |
git config --global user.email "akash.mercer@gmail.com"
git config --global user.name "Akash Khunt"
git merge origin/${{ github.base_ref }}
- name: Build release APK for head branch and rename it to match head-short-sha result
run: |
./gradlew assembleRelease -PtargetDensity=xxhdpi -PtargetAbi=arm64-v8a -PresConfig=en -Pandroid.testInstrumentationRunnerArguments.androidx.benchmark.enabledRules=BaselineProfile
mv app/build/outputs/apk/release/app-xxhdpiArm64-v8a-release-unsigned.apk app/build/outputs/apk/release/${{ env.head_short_sha }}.apk
- name: Upload APK
uses: actions/upload-artifact@v3
with:
name: ${{ env.head_short_sha }}.apk
path: app/build/outputs/apk/release/${{ env.head_short_sha }}.apk
size_diff_report:
needs: build
name: Size Report
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: ${{ needs.build.outputs.head_short_sha }}.apk
- uses: actions/download-artifact@v3
with:
name: ${{ needs.build.outputs.base_short_sha }}.apk
# here we use our custom python script to generate file matching size_diff_report_file
- name: Generate size report
run: |
chmod +x ./.github/scripts/compare_apks.py
python3 ./.github/scripts/compare_apks.py ${{ needs.build.outputs.base_short_sha }} ${{ needs.build.outputs.head_short_sha }}
- name: Upload Size Report html file
uses: actions/upload-artifact@v3
with:
name: ${{ env.size_diff_report_file }}
path: ${{ env.size_diff_report_file }}
pr_comment:
needs: size_diff_report
name: PR Comment (Size Report)
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@v3
with:
name: ${{ env.size_diff_report_file }}
- name: Add PR Comment
uses: actions/github-script@v6
with:
script: |
const fs = require('fs');
try {
const filePath = 'apk_size_diff_report.html'
const sizeReportContent = fs.readFileSync(filePath, 'utf8')
const result = await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: sizeReportContent
})
console.log(result)
} catch (error) {
console.error('Error reading file:', error);
}

Complete Size Metrics workflow

 

I hope your team is also able to use this workflow (or maybe an improved version 🙂) as a part of your CI pipeline to give you better understanding of how your changes/features are affecting the apk size.

Reference Material Links and Credits:

Hope you liked reading the article. Please feel free to reach me on Twitter or LinkedIn. Thank You!

This article was previously published on proandroiddev.com

YOU MAY BE INTERESTED IN

YOU MAY BE INTERESTED IN

blog
In Part 2, we finished our GHA workflow which generates apk size diff report…
READ MORE

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.

Menu