Commit Graph

239 Commits

Author SHA1 Message Date
zmiao
93f19b8daa avoid restore sanity ccache since it will cause unarchive error for new added macos job (#16135) 2020-01-22 14:10:09 +02:00
Jordan Kiley
942d9566d3 [ios, macos] Include SKU token for offline requests (#16130) 2020-01-21 13:35:57 -08:00
zmiao
1ade6c7784 [test-runner] Enable IOS RenderTestRunner and run it on Firebase (#16111)
* [test-runner] Setup iOS RenderTestRunner

* [test-runner] Make app runnable on firebase

* Add iOS platform dependent expectation and ignores; Run prepare script via cmake

* Make RenderTestRunner report test status
2020-01-21 23:23:35 +02:00
Juha Alanen
86a3605349 [build][android] Fix CI bot that validates downstream API breakage 2020-01-10 17:19:30 +02:00
Juha Alanen
94df7ffc2d [build] Run Android unit test on Firebase 2020-01-10 17:19:30 +02:00
Jordan Kiley
60a4b3eda3 [ios] Add native_apple_interface to ios.cmake (#16103)
* [ios] Add native_apple_interface to ios.cmake

* [ios] Use Xcode for next-macos-xcode11-release
2020-01-09 12:41:14 -08:00
tobrun
52edd9b43b [android] add CI bot that validates downstream API breakage 2020-01-09 16:09:51 +01:00
Juha Alanen
8ce3f6a987 [build] Run Android benchmark on Firebase 2020-01-03 11:52:50 +02:00
Thiago Marcos P. Santos
6d876f1d6b [build] Make the configuration simpler
Moved the Android flags to the compilation options interface
and removed build options we are not using and go back to CMake
defaults.
2019-12-21 00:38:41 +01:00
Thiago Marcos P. Santos
e298435206 [metrics] Add new baselines for Linux Clang8 and macOS 2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
8869d09b25 [build] Collect and publish new baselines
Also fail the bot in case there are new baselines.
2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
0dbe80fe66 [build] If not found, use the cache from the sanity checks bot
Better than initializing git submodules from scratch.
2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
970cabe0eb [build] Reorganize the render tests
Move everything that might collect metrics to the metrics/ directory,
the only test manifests inside the render-test/ directory are the
unit tests of the probes.
2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
31742317e8 [build] Remove "wrapper" parameter
Make it a step of the Linux configuration, since it only makes
sense for Linux.
2019-12-12 14:50:22 +02:00
Thiago Marcos P. Santos
3679315633 [build] Better isolate compiler flags
Previously we had UndefinedBehavior mixed with other sanitizers, in
special implicit conversions and overflows.

Also, run the sanitizers in Release configuration, so they will run
faster. -O1 is just needed for perfectly nice stack traces.

Also, moving the flags to an interface will prevent them to leak
to our external dependencies.
2019-12-11 17:12:25 +02:00
Thiago Marcos P. Santos
3b14aef334 [build] Upload nitpick artifacts
Upload diffs when the sanity bot complains about some wrong
formatting of the code or buildsystem.
2019-12-11 16:39:03 +02:00
zmiao
f848f07ad9 [metrics] Run Android RenderTestRunner bot in rebaseline mode (#16036) 2019-12-11 11:50:27 +02:00
Alexander Shalamov
af0873720e [metrics] Run gfx, network and memory probes in 'rebaseline' mode 2019-12-09 17:53:34 +02:00
Alexander Shalamov
7eee124677 [metrics] Create gcc8-release metrics configuration 2019-12-09 17:53:34 +02:00
Thiago Marcos P. Santos
07a9e26e8a [build] Do not halt UBSAN on errors 2019-12-07 16:32:39 +02:00
zmiao
e619552a5d [test-runner] Make Android RenderTestRunner bot fail when test cases are not fully passed (#15994)
* [test-runner] Fix failed tests, generate android specific expectations

* [test-runner] Check render test runner testing result

* [test-runner] Add error case to android ignore list

*  [test-runner] Update metrics

* [test-runner] Move Android render tests with big diff(>0.006) to ignore list
2019-12-04 18:10:58 +02:00
tobrun
29bbfcccde [android] remove platform specific Android code, migrate project to mapbox-gl-native-android 2019-12-03 17:10:31 +01:00
zmiao
b2693f9d53 [test-runner] Add android RenderTestRunner baseline metrics (#15979)
* [test-runner] Enable probe-testing android render test runner

* [test-runner] Add pixel4 baseline metrics

* Add git ignored metrics

* Make renderTestrunner run two times with different manifest config
2019-12-02 13:29:13 +02:00
Thiago Marcos P. Santos
2a65301f09 [build] Get the bots enforcing the binary size
Release bots will build and persist the build targets other workspaces.

A bot will collect all the binaries and check the size.
2019-11-27 18:32:58 +02:00
Alexander Shalamov
2f8501043c [test-runner] Move probe related command line args to manifest
- Move probe related command line arguments to manifest, example:

  "probes": ["probeGFX", "probeNetwork", "probeMemory"],
  "metric_path": "metrics/linux"

- Use manifest file name as a postfix for a result page
- Move injected probes 'begin' section before map object creation
- Generalize artifact storing shell script
2019-11-27 16:43:33 +02:00
zmiao
11321497da [render-test] Add metric_paths into Manifest (#15973)
* [render-test]Add expected metrics path in manifest file

* seperate metrics.json and expected.png

* fix expected path parsing

* keep expectation path if update is enabled
2019-11-27 11:08:55 +02:00
Thiago Marcos P. Santos
b3fc469b09 [tests] Disable some metrics recording on Firebase
We are not using them, attempt to make the tests run faster.
2019-11-21 15:31:59 +02:00
Thiago Marcos P. Santos
d6eeea1909 [build] Build Android release when running render tests 2019-11-21 15:31:59 +02:00
zmiao
03b7fe3326 [render-test] Config CircleCi to run RenderTestApp on Firebase with Pixel4 (#15951)
This reverts commit d2413d4ffffd087b352b6e4edd6e6322a311bc07.
2019-11-20 17:56:19 +02:00
Thiago Marcos P. Santos
656bc9c8bb [build] Run Valgrind instead of Memory Sanitizer
Memory Sanitizer requires all the dependencies, including
libstdc++ to be compiled with Memory Sanitizer, otherwise
we will get tons of false positives.
2019-11-20 13:56:48 +02:00
Thiago Marcos P. Santos
b3fdd2c762 [build] Check if mapbox-gl-js is pinned to master 2019-11-19 13:42:16 +02:00
Mikhail Pozdnyakov
089ff8e2b3 No firebase skip for commits touching paths other than darwin-specific 2019-11-13 10:57:07 +02:00
zmiao
c8576e089f [render-test] Add manifest parser for render-test-runner (#15861)
* [render-test] Add manifest parser for render-test-runner

* [render-test] Refactory manifest parser

* [render-test] Parse full manifest file through CLI

* [render-test] Add linux probe manifest
2019-10-29 20:59:29 +02:00
zmiao
128da903c1 Config CircleCI to save render test output when failing (#15863)
* [render-test] Save index.html

* Add extra folder
2019-10-25 18:22:32 +03:00
Dane Springmeyer
744102d955 back to v8 2019-10-24 13:26:32 +03:00
Dane Springmeyer
a1aa3b0e08 fix linux node platform publishing 2019-10-24 13:26:32 +03:00
Thiago Marcos P. Santos
1cfd0c4e92 [build] Bump CI docker image
This version includes gcloud, needed to run
android firebase tests.
2019-10-23 14:29:18 +03:00
Thiago Marcos P. Santos
722a0c2f61 [build] Make output of the tests verbose
We have a flaky test that is hanging on the bots and we need to find out
which one.
2019-10-18 23:39:58 +03:00
Jason Wray
ee2b803315 [build] Update to Xcode 11.1 2019-10-15 14:38:43 -07:00
tobrun
0f9666ebf3 [android] - remove snapshot and release builds, disable broken metric collection script 2019-10-15 12:08:22 +02:00
Thiago Marcos P. Santos
990fdc43fa [build] Make sure offline_schema.hpp is up-to-date
- Get the bots to verify that.
 - Fix the paths.
2019-10-11 16:06:28 +03:00
Alexander Shalamov
bebc49949a [render-test] Update location of render / query test html page 2019-10-10 23:33:36 +03:00
Julian Rex
bf61274b2a [ios, android] Update curl command to indicate there is no password (as per https://circleci.com/docs/api/#authentication) (#15796) 2019-10-10 16:27:02 -04:00
Thiago Marcos P. Santos
2a3cc96c42 [tests] Do not run tests in parallel
Introduces too much flakeness.
2019-10-10 19:41:27 +03:00
Thiago Marcos P. Santos
5f371d5e39 [build] Check for code generation on Darwin
Sadly, it can only be done inside macOS because
of how examples are generated.
2019-10-10 19:41:27 +03:00
Thiago Marcos P. Santos
d0e6f9df44 [build] Remove redundant bots
We have something equivalent on the `next` workspace for these bots.
2019-10-10 19:41:27 +03:00
Mikhail Pozdnyakov
76346944f3 [ci] Fixed duplicated render tests run on MacOS 2019-10-10 15:39:34 +03:00
Julian Rex
563f9d0534 [ios] Skip metrics.sh step (#15779) 2019-10-08 17:49:45 -04:00
Juha Alanen
bd283fc1be [build] enable query tests on native test runner 2019-10-07 10:58:46 +03:00
Jordan Kiley
4b4e828647 [ios] Disable iOS code coverage script #15749 2019-10-03 11:07:41 -07:00