tei/MapLibreParticleLab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[android] #3447 - initial commit to verify current progress

Browse Source
This commit is contained in:
Tobrun
2016-01-06 15:15:19 +01:00
parent 71edb38cf1
commit 3e4c1bd534
6 changed files with 396 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
platform/android/tests/README.md Normal file
View File

@@ -0,0 +1,9 @@
# Mapbox GL Android Test documentation
## Testing
We currently support the following types of testing on the Mapbox Android SDK:
- [Unit tests](https://github.com/mapbox/mapbox-gl-native/blob/3447-Add-test-documentation/platform/android/tests/docs/UNIT_TESTS.md) using [JUnit](http://developer.android.com/tools/testing-support-library/index.html#AndroidJUnitRunner)
- [UI tests](https://github.com/mapbox/mapbox-gl-native/blob/3447-Add-test-documentation/platform/android/tests/docs/UI_TESTS.md) using [Espresso](http://developer.android.com/tools/testing-support-library/index.html#Espresso)
- [Exerciser](https://github.com/mapbox/mapbox-gl-native/blob/3447-Add-test-documentation/platform/android/tests/docs/EXERCISER_TESTS.md) tests using [Monkey](http://developer.android.com/tools/help/monkey.html) or [Build-in Fuzz test](http://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-built-in-fuzz.html)
- [Performance tests](https://github.com/mapbox/mapbox-gl-native/blob/3447-Add-test-documentation/platform/android/tests/docs/PERFORMANCE_TESTS.md) using [Systrace](https://codelabs.developers.google.com/codelabs/android-perf-testing/index.html?index=..%2F..%2Fbabbq-2015&viewga=UA-68632703-1#0)

23
platform/android/tests/docs/EXERCISER_TESTS.md Normal file
View File

@@ -0,0 +1,23 @@
#UI/Application Exerciser Tests
UI/application exerciser tests are stress test using random generated users events.
##Running Locally
The Android SDK provides a test tool called [Monkey](http://developer.android.com/tools/help/monkey.html),
"a program that runs on your emulator or device and generates pseudo-random streams of user events
such as clicks, touches, or gestures, as well as a number of system-level events."
To exercise Monkey on the test app, install the package on the device (e.g. via Android Studio)
and then:
```
$ adb shell monkey -p com.mapbox.mapboxgl.testapp -v 500
```
##Running on AWS Device Farm
Amazon Device farm supports a similar tool called `Built-in Fuzz Test`.
"The built-in fuzz test randomly sends user interface events to devices and then reports results."
More information about [Built-in Fuzz Test](http://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-built-in-fuzz.html)

69
platform/android/tests/docs/PERFORMANCE_TESTS.md Normal file
View File

@@ -0,0 +1,69 @@
# Performance Tests
## What is Systrace?
From the [Android documentation](http://developer.android.com/tools/help/systrace.html):
"Systrace is a tool that will help you analyze the performance of your application by capturing and displaying execution times of your applications processes and other Android system processes."
## What are we using it for?
We’re using Systrace to look for performance issues in our SDK. When we run Systrace while using our app,
it will collect data from Android then format it into an html file we can view in a web browser.
When we open the Systrace results, we’re hoping it will be able to help us resolve some of the
underlying issues associated with our SDK.
## Run systrace locally
Execute following command in your terminal will run Systrace for 10 seconds:
```
python $ANDROID_HOME/platform-tools/systrace/systrace.py --time=10 -o ~/trace.html gfx view res
```
This command will output a file called `trace.html`.
More information how to interpret the values can be found [here](http://developer.android.com/tools/help/systrace.html).
More information about this topic can be found [here](https://codelabs.developers.google.com/codelabs/android-perf-testing/index.html?index=..%2F..%2Fbabbq-2015&viewga=UA-68632703-1#0)
## Automating Systrace with Espresso
The following annotation is being used to isolate the tests and classes that should be included in the performance tests.
```java
@PerfTest
```
You can optionally define extra rules to gather more data during a performance test:
```java
@Rule
public EnableTestTracing mEnableTestTracing = new EnableTestTracing();
@Rule
public EnablePostTestDumpsys mEnablePostTestDumpsys = new EnablePostTestDumpsys();
@Rule
public EnableLogcatDump mEnableLogcatDump = new EnableLogcatDump();
@Rule
public EnableNetStatsDump mEnableNetStatsDump = new EnableNetStatsDump();
```
## Automating Systrace with MonkeyRunner
An example of such a script can be found [here](https://github.com/googlecodelabs/android-perf-testing/blob/master/run_perf_tests.py).
The script is following this structure:
- Check environment variables
- Define functions
- Clear local data from previous runs
- Find an Android device
- Enable and clear graphics info dumpsys
- Start a systrace thread & test suite thread in parallel
- Wait for both threads to complete
- Download files from device
- Run analysis on downloaded files
## Note
Testing on a device with minimum SDK 6.0 is preferred.
The tools above work on older versions of Android, but less data will be collected.

131
platform/android/tests/docs/UI_TESTS.md Normal file
View File

@@ -0,0 +1,131 @@
#UI Tests
## Running Espresso tests locally on a device
This test project comes with all the required Android Testing Support Library dependencies
in the Gradle file. Tests are under the `app/src/androidTest` folder.
Note that before running your tests, you might want to turn off animations on your test device.
It's a known issue that leaving system animations turned on in a test device
(window animation scale, transition animation scale, animator duration scale)
might cause unexpected results, or may lead tests to fail.
To create a new run configuration:
* Click on Run -> Edit Configurations...
* Click on the plus sign and then on "Android Tests"
* Give a name to the configuration, e.g. `TestAppTests`
* Choose the `MapboxGLAndroidSDKTestApp` module
* Choose `android.support.test.runner.AndroidJUnitRunner` as the instrumentation runner
* Click OK to save the new configuration
You can now run this configuration from the main toolbar dropdown menu.
## Running Espresso tests manually on AWS Device Farm
On a terminal, within `mapbox-gl-native/android/java`,
run the tests (`cC` stands for `connectedCheck`):
```
$ ./gradlew cC -p MapboxGLAndroidSDKTestApp
```
Then:
* Go to your AWS Console and choose Device Farm.
* Create a new project, e.g. `MapboxGLAndroidSDKTestApp`
* On step 1, upload the APK in `mapbox-gl-native/android/java/MapboxGLAndroidSDKTestApp/build/outputs/apk/MapboxGLAndroidSDKTestApp-debug-unaligned.apk`
* On step 2, choose Instrumentation, test filter is `com.mapbox.mapboxgl.testapp.MainActivityTest` and upload the APK in `mapbox-gl-native/android/java/MapboxGLAndroidSDKTestApp/build/outputs/apk/MapboxGLAndroidSDKTestApp-debug-androidTest-unaligned.apk`
* On step 3, choose a device pool. E.g. Top Devices
* On step 4, customize your device state (if needed)
* Finally, confirm the configuration and run the tests.
On Step 2, you can also separate by commas different classes: `com.mapbox.mapboxgl.testapp.MainActivityTest,com.mapbox.mapboxgl.testapp.MainActivityScreenTest`
If you have no tests for your app, or want to test some random user behaviour,
you can just choose "Built-in: Fuzz" in step 2.
## Running Espresso test automatically on AWS Device Farm
To automatically execute Espresso tests as part of our CI build, we have created a Python [script](https://github.com/mapbox/mapbox-gl-native/blob/aws-devicelab/android/scripts/devicefarm.py).
This script is responsible for:
- uploading an APK + test APK
- scheduling tests
- exiting with a return code
- 0 -> all tests have passed
- 1 otherwise
### Requirements
* [Boto 3](http://boto3.readthedocs.org)
* [Requests](http://www.python-requests.org)
### Running the script
A sample run would be as follows:
```
$ python devicefarm.py \
--project-arn "arn:aws:devicefarm:us-west-2:XXXXX" \
--device-pool-arn "arn:aws:devicefarm:us-west-2::devicepool:YYYYY" \
--app-apk-path app/build/outputs/apk/app-debug-unaligned.apk \
--test-apk-path app/build/outputs/apk/app-debug-androidTest-unaligned.apk
```
Where you need to insert your actual project and device ARNs. We follow Boto 3
conventions to to [set up the AWS credentials](https://github.com/boto/boto3#quick-start).
You can build the `app-debug-androidTest-unaligned.apk` package with Gradle:
```
./gradlew assembleAndroidTest
```
To run tests locally, you can use `./gradlew assemble` to build the app APK, and
`./gradlew test --continue` to run unit tests. Finally, `./gradlew connectedAndroidTest`
will run the Espresso tests in a local device.
A sample output would be as follows:
```
Starting upload: ANDROID_APP
Uploading: ../app/build/outputs/apk/app-debug-unaligned.apk
Checking if the upload succeeded.
Upload not ready (status is INITIALIZED), waiting for 5 seconds.
Starting upload: INSTRUMENTATION_TEST_PACKAGE
Uploading: ../app/build/outputs/apk/app-debug-androidTest-unaligned.apk
Checking if the upload succeeded.
Upload not ready (status is INITIALIZED), waiting for 5 seconds.
Scheduling a run.
Checking if the run succeeded.
Run not completed (status is SCHEDULING), waiting for 60 seconds.
Run not completed (status is RUNNING), waiting for 60 seconds.
Run not completed (status is RUNNING), waiting for 60 seconds.
Run not completed (status is RUNNING), waiting for 60 seconds.
Run not completed (status is RUNNING), waiting for 60 seconds.
Run not completed (status is RUNNING), waiting for 60 seconds.
Run completed: PASSED
```
### Available commands
You can use the `--help` command to get a list of all available options:
```
$ python devicefarm.py --help
usage: Device Farm Runner [-h] [--project-arn PROJECT_ARN]
[--device-pool-arn DEVICE_POOL_ARN]
[--app-apk-path APP_APK_PATH]
[--test-apk-path TEST_APK_PATH]
Runs the Espresso tests on AWS Device Farm.
optional arguments:
-h, --help show this help message and exit
--project-arn PROJECT_ARN
The project ARN (Amazon Resource Name) (default: None)
--device-pool-arn DEVICE_POOL_ARN
The device pool ARN (Amazon Resource Name) (default:
None)
--app-apk-path APP_APK_PATH
Path to the app APK (default: None)
--test-apk-path TEST_APK_PATH
Path to the tests APK (default: None)
```

32
platform/android/tests/docs/UNIT_TESTS.md Normal file
View File

@@ -0,0 +1,32 @@
# Unit tests
Our Unit tests are based on JUnit and are located under `/src/test/java/`.
We are using plain JUnit to test classes that aren't calling the Android API,
or are using Android's JUnit extensions to stub/mock Android components.
## Running Unit tests locally
To run Unit tests locally you switch to the Unit Tests build variant, then right click the corresponding test class or method and select "Run ...".
You can also have a run configuration:
* Click on Run -> Edit Configurations...
* Click on "Junit Tests"
* Give a name to the configuration, e.g. `JUnit tests`
* As "Test Kind", choose "All in directory"
* As folder, choose the following folder: `mapbox-gl-native/platforms/android/java/MapboxGLAndroidSDKTestApp/src/test/java`
* Click OK to save the new configuration
You can also run the tests from the command line with:
```
$ ./gradlew test --continue -p MapboxGLAndroidSDKTestApp
```
## Running Unit tests on CI
The Unit tests are executed as part of the build process on our CI and are
automatically run for each new commit pushed to this repo. If a Unit tests
fails, this will fail and stop the build.
You can find this gradle command in our [buildscript](https://github.com/mapbox/mapbox-gl-native/blob/master/platform/android/bitrise.yml#L48):
```
$ ./gradlew testReleaseUnitTest --continue
```

132
platform/android/tests/scripts/devicefarm.py Normal file
View File

@@ -0,0 +1,132 @@
'''
Uploads an APK and a test APK to AWS Device Farm and runs the Espresso tests.
Exit code is 0 if all tests pass, 1 otherwise. See README.md for details.
'''
from time import sleep
import argparse
import boto3
import requests
'''
Parser
'''
parser = argparse.ArgumentParser(
prog='Device Farm Runner',
description='Runs the Espresso tests on AWS Device Farm.',
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
add_help=True)
parser.add_argument('--project-arn',
type=str, help='The project ARN (Amazon Resource Name)')
parser.add_argument('--device-pool-arn',
type=str, help='The device pool ARN (Amazon Resource Name)')
parser.add_argument('--app-apk-path',
type=str, help='Path to the app APK')
parser.add_argument('--test-apk-path',
type=str, help='Path to the tests APK')
args = vars(parser.parse_args())
project_arn = args.get('project_arn')
device_pool_arn = args.get('device_pool_arn')
app_apk_path = args.get('app_apk_path')
test_apk_path = args.get('test_apk_path')
# Check
if not project_arn or not device_pool_arn:
raise Exception('You need to set both the project and the device pool ARN.')
elif not app_apk_path or not test_apk_path:
raise Exception('You need to set both the app and test APK path.')
'''
The AWS Device Farm client
'''
client = boto3.client('devicefarm', region_name='us-west-2')
'''
Methods
'''
def upload_apk(apk_name, apk_path, apk_type):
print 'Starting upload: %s' % apk_type
result = client.create_upload(
projectArn=project_arn, name=apk_name, type=apk_type)
presigned_url = result.get('upload').get('url')
upload_arn = result.get('upload').get('arn')
# PUT the file content and wait
put_file(apk_path=apk_path, presigned_url=presigned_url)
wait_for_upload(upload_arn=upload_arn)
return upload_arn
def put_file(apk_path, presigned_url):
print 'Uploading: %s' % apk_path
with open(apk_path, 'rb') as f:
data = f.read()
result = requests.put(presigned_url, data=data)
if result.status_code != 200:
raise Exception(
'PUT failed with status code: %s' % result.status_code)
def wait_for_upload(upload_arn):
print 'Checking if the upload succeeded.'
succeeded = False
while not succeeded:
result = client.get_upload(arn=upload_arn)
status = result.get('upload').get('status')
succeeded = (status == 'SUCCEEDED')
if status == 'FAILED':
raise Exception('Upload failed.')
elif not succeeded:
print 'Upload is not ready (status is %s), waiting for 5 seconds.' % status
sleep(5)
def schedule_run(app_arn, test_arn):
print 'Scheduling a run.'
result = client.schedule_run(
projectArn=project_arn,
appArn=app_arn,
devicePoolArn=device_pool_arn,
name='automated_run',
test={ 'type': 'INSTRUMENTATION', 'testPackageArn': test_arn})
run_arn = result.get('run').get('arn')
return_code = wait_for_run(run_arn=run_arn)
return return_code
def wait_for_run(run_arn):
print 'Checking if the run succeeded.'
return_code = ''
succeeded = False
while not succeeded:
result = client.get_run(arn=run_arn)
status = result.get('run').get('status')
return_code = result.get('run').get('result')
succeeded = (status == 'COMPLETED')
if not succeeded:
print 'Run not completed (status is %s), waiting for 60 seconds.' % status
sleep(60)
return return_code
'''
Main flow
'''
# 1. Upload the app APK
app_arn = upload_apk(
apk_name='app-debug-unaligned.apk',
apk_path=app_apk_path,
apk_type='ANDROID_APP')
# 2. Upload the test APK
test_arn = upload_apk(
apk_name='app-debug-androidTest-unaligned.apk',
apk_path=test_apk_path,
apk_type='INSTRUMENTATION_TEST_PACKAGE')
# 3. Schedule the run
return_code = schedule_run(app_arn=app_arn, test_arn=test_arn)
exit_code = 0 if return_code == 'PASSED' else 1
print 'Run completed: %s' % return_code
exit(exit_code)