Commit Graph

49 Commits

Author SHA1 Message Date
zmiao
2385f4cd86 [render-test] Add configurable rootPath, ignorePath via CLI arguments (#15832)
* [render-test] Add configurable path via CLI arguments

* [render-test] Change ignore path config
2019-10-23 12:04:23 +03:00
zmiao
068befa9df [test-runner] Run render test runner as a NativeActivity app on android device (#15827)
* [test-runner]Add android render-test-runner NativeActivity app

* [test-runner] Fix cmake for CI build failure

* [test-runner] Update folder location, change app configureation

* [test-runner] Add running instruction
2019-10-22 16:32:21 +03:00
Mikko Pulkki
0ca96fd8a4 [render-test] Implement fps benchmarking tests (#15803) 2019-10-22 12:13:55 +03:00
Mikhail Pozdnyakov
bb0e5ffb2c [test runner] Improve error messaging for memory metrics tests 2019-10-21 16:10:45 +03:00
Mikhail Pozdnyakov
61d1018087 [test runner] Memory metrics expectations for MAC 2019-10-21 16:10:45 +03:00
Mikhail Pozdnyakov
b03982730e [test runner] Command line option for the tests expectations path
-e[expectationsPath], --expectationsPath=[expectationsPath]

Test expectations path.
2019-10-21 16:10:45 +03:00
Mikhail Pozdnyakov
eb3c203d4e [test runner] Adjust memory metrics test results 2019-10-21 16:10:45 +03:00
Mikhail Pozdnyakov
8bdd8ec670 [test runner] Disable memory metrics checks on sanitize builds 2019-10-21 16:10:45 +03:00
Mikhail Pozdnyakov
b0976feca2 [test runner] Tests for memory size metrics 2019-10-21 16:10:45 +03:00
Mikhail Pozdnyakov
9adc78134c [test runner] Threshold for memory metrics 2019-10-21 16:10:45 +03:00
Juha Alanen
050b3360ec [render-test] Add network metrics test case 2019-10-16 14:14:46 +03:00
Juha Alanen
b53093b522 [render-test] Add support for collecting network metrics 2019-10-16 14:14:46 +03:00
zmiao
e829d1ae6f [test-runner] make render-test a shared lib (#15816)
* [test-runner] make render-test a shared lib

[test-runner] nit

[test-runner] nit again

* [test-runner] Remove redundant linked file in cmake
2019-10-15 19:16:32 +03:00
Thiago Marcos P. Santos
465d4e8cfa [render-test] Add test for file size tolerance 2019-10-14 21:15:21 +03:00
Thiago Marcos P. Santos
8b3a222c81 [render-test] Add tolerance to file-size tests 2019-10-14 21:15:21 +03:00
Thiago Marcos P. Santos
2fb1e47b75 [render-test] No camelCase on the metrics.json tag
It is ugly.
2019-10-11 18:29:52 +03:00
Thiago Marcos P. Santos
307f9cede1 [render-test] Prettyfy metrics output 2019-10-11 18:29:52 +03:00
Thiago Marcos P. Santos
acd7fba3d5 [render-test] Add more probe tests for file size
Also make the tests simpler.
2019-10-11 18:29:52 +03:00
Thiago Marcos P. Santos
7f5c630613 [render-test] Fix UPDATE_METRICS=1 changing the path of file size tests
Should use the exact same path as described on style.json
2019-10-11 18:29:52 +03:00
Thiago Marcos P. Santos
63116154fe [render-test] Only write metrics if they are available
Otherwise the file will have an empty tag.
2019-10-10 19:54:51 +03:00
Thiago Marcos P. Santos
629c0e1142 [render-test] Add tests for filesize probe
And make sure we run it on the bots
2019-10-10 19:54:51 +03:00
Thiago Marcos P. Santos
2e002746eb [render-test] Make clang-format happy 2019-10-10 19:54:51 +03:00
Thiago Marcos P. Santos
a9f06f3b52 [render-test] Add instruction for verifying file size
For a given absolute or relative path, measure the size of the file.

This is useful for measuring if the cache has increased as expected
after moving a camera or decreased after cleaning up.

In a more hackish manner, could be used for collecting binary size
statistics of shared libraries.
2019-10-10 19:54:51 +03:00
Thiago Marcos P. Santos
026e15fe3d [render-test] Fix rootPath parameter being ignored
It was doing nothing and always using the default root path.
2019-10-10 19:54:51 +03:00
Mikhail Pozdnyakov
b4e0224a57 [test runner] Fix finding of expected images paths
Before this change, the found paths to the expected images
erroneously included the path to the `metrics.json` file
(if this file was present) leading to raising of an
unhandled exception.
2019-10-09 14:35:08 +03:00
Juha Alanen
4eaaa9046e [tests] Share common code between test runners 2019-10-08 15:33:30 +03:00
Juha Alanen
13801f1b80 [render-test] Add support for query tests 2019-10-07 10:58:46 +03:00
Juha Alanen
18a426d748 [render-test] Support feature-state tests where featureID is a number 2019-10-07 10:58:46 +03:00
Juha Alanen
f58a25932a [render-test] Add support for setPitch operation 2019-10-07 10:58:46 +03:00
Juha Alanen
6b13c42314 [build] Fix clang format and tidy checks 2019-09-18 14:29:15 +03:00
Juha Alanen
045018905b [render-test] Add support for feature state APIs 2019-09-18 14:29:15 +03:00
Thiago Marcos P. Santos
f604280812 [build] Do not override new/delete on render-test for sanitize builds
Sanitizers will also do the same and it will conflict.
2019-09-17 12:28:55 +03:00
Alexander Shalamov
9a42bdd7ae [core] Add filtering option to test-runner
This change adds filter command line option to render test runner,
so that test can be filtered by providing regular expression, for instance:

mbgl-render-test -f .*hillshade.*
2019-08-28 13:55:36 +03:00
Mikhail Pozdnyakov
766392f1f8 [test runner] Use allocation peak values for memory metrics 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
1b020eeb11 [test runner] Test metrics reports. Check metrics results. 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
d0a84cb7d9 [test runner] Introduce probe_memory_.. API for render tests 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
e787f5f43a [test runner] Overload global new/delete 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
75b1e47f3a [test runner] Introduce AllocationIndex 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
f365dcece4 [render tests] gl-native baseline for debug/collision- tests 2019-08-26 13:55:15 +03:00
Mikhail Pozdnyakov
d0529e7b19 [render test] mapbox-gl-native baseline for text-pitch-alignment tests 2019-08-21 17:07:05 +03:00
Mikhail Pozdnyakov
5de27d5200 [test runner] Expectations paths are represented with std::vector 2019-08-16 16:25:59 +03:00
Mikhail Pozdnyakov
1580216469 [test-runner] UPDATE_GENERIC variable updates results at mapbox-gl-js submodule 2019-08-16 16:25:59 +03:00
Mikhail Pozdnyakov
8792e7c322 Enable text-variable-anchor/{pitched-rotated-debug|rotated-offset} render tests 2019-08-16 16:25:59 +03:00
Mikhail Pozdnyakov
a1904e802c [test-runner] Introduce mapbox-gl-native test expectations
The `mapbox-gl-native`-specific render tests expected results will
be stored at the `render-test/expected/` subfolder.

Test runner will automatically put new expected results there if
invoked with `UPDATE=true`.
2019-08-16 16:25:59 +03:00
Aleksandar Stojiljkovic
d1d5999354 Mark tests flaky on Linux only. #15322
fill-opacity/zoom-and-property-function-pattern,
fill-pattern/literal,
fill-pattern/opacity,
fill-pattern/zoomed,
line-translate/literal,

Addresses: #15322, #14423, #14870, #14768, #14859
2019-08-07 13:59:04 +03:00
Aleksandar Stojiljkovic
75f1c72618 Ignore flaky line-pattern/opacity test on Linux bot
Related to: https://github.com/mapbox/mapbox-gl-native/issues/15320
2019-08-07 10:26:39 +03:00
Aleksandar Stojiljkovic
b5d71fbca0 mbgl-render-test runner - use the same tolerance as with node render test
Two render tests failing only with mbgl-render-tests are added to platform
specific ignores. This is to be extended with fill-extrusion-pattern failing only on mac.
2019-08-07 10:26:39 +03:00
Mikhail Pozdnyakov
4c896cbb31 [core] Fix error handling in HeadlessFrontend::render() 2019-06-28 16:33:52 +03:00
Bruno de Oliveira Abinader
3b0e5a59a2 [core] Implement C++ render test runner 2019-06-25 20:10:21 +03:00