Commit Graph

26 Commits

Author SHA1 Message Date
Mikhail Pozdnyakov
8986b558eb [render tests][tile mode] Check symbol icons cut-offs 2020-04-06 19:18:06 +03:00
Alexander Shalamov
879c44f661 [core] Modularize FileSource codebase (#15768)
* [core] Introduce FileSourceManager and use it for default platform impl

  - Add `FileSourceManager` interface that provides access to `FileSource`
    instances and means of registering / unregistering `FileSource` factories
  - Split `DefaultFileSource` into smaller parts
  - Add `DatabaseFileSource` interface and it's default implementation
  - Remove inter-dependencies between concrete `FileSource` classes

* [build] Add files to next build system

* [core] Add generic property setters / getters

* [core] Remove setOnlineStatus from OnlineFileSource interface

* [core] Hide threading implementation details from DatabaseFileSource interface

* [core] Make DB file source methods virtual

* [core] Add documentation for DatabaseFileSource and rename one method

* [core] Use simple callback instead of ActorRef

* [core] Remove ActorRef from OnlineFileSource public header

* [core] Add callback to FileSource::forward async API

* [core] Pass OfflineRegionDefinition by value

* [core] Update tests to use modular file sources

* [core] Update unit tests

* [core] Update unit tests after rebase

* [core] Backport low prio fix for cached requests

* [core] Backport pack database

* [core] Return removed factory from unRegisterFileSourceFactory

* [core] Rename shadowed args in onlinefilesource

* [core] Remove simple std::function callback aliases

* [core] Expose online file source property keys in public header file

* [test-runner] Add proxy file source test runner

* [cache] Update mbgl-cache utility to use new file source

* [metrics] Rebaseline binary size metrics

* [offline] Update offline utility

* [core] Update changelog
2020-01-13 10:57:23 +02:00
Thiago Marcos P. Santos
af8092a86f [render-test] Use image from the cache for addImage operation
Instead of trying to read it from the filesystem.
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
f0d2a2f636 [render-test] Effectively use the offline cache for fixtures 2020-01-08 13:07:12 +02:00
Mikhail Pozdnyakov
64a37ceb05 [test runner] Rendering a single tile in Tile mode 2019-12-18 13:48:52 +02:00
Thiago Marcos P. Santos
ad1a7c595d [render-test] Cleanup on the render tests results
Simplify the code path and make it print both rendering error and
metrics on fail.
2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
1c138124a6 [render-test] Fix metrics passing when failing
Metrics will now fail, but the tool will return 0 when *ONLY* metrics
fail, to make the bot continue. We will also run ignored tests and
check if they are ignored but passing.
2019-12-16 22:52:01 +02:00
Mikhail Pozdnyakov
06a2930342 [test runner] Refactor injected probes operations code 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
b2d06b8a17 [test runner] Split operations parsing and execution 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
426f714c04 [test runner] Introduce TestOperation 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
4a30f74699 [test runner] Introduce generic TestContext 2019-12-13 15:34:06 +02:00
Alexander Shalamov
a9ba572118 [test-runner] Add rebaseline flag
Rebaseline flag allows to update or create new metric, in cases when
metric check is failed or expected metric is missing.
2019-12-09 17:53:34 +02:00
Mikhail Pozdnyakov
2b6a0e48ee [test runner] Enable test results update from cmd line
Added the following command line argument:
```
-u[update], --update=[update]     Test results update mode.
Supported values are: "default", "platform", "metrics"
```
2019-11-27 18:19:10 +02:00
Alexander Shalamov
2f8501043c [test-runner] Move probe related command line args to manifest
- Move probe related command line arguments to manifest, example:

  "probes": ["probeGFX", "probeNetwork", "probeMemory"],
  "metric_path": "metrics/linux"

- Use manifest file name as a postfix for a result page
- Move injected probes 'begin' section before map object creation
- Generalize artifact storing shell script
2019-11-27 16:43:33 +02:00
zmiao
11321497da [render-test] Add metric_paths into Manifest (#15973)
* [render-test]Add expected metrics path in manifest file

* seperate metrics.json and expected.png

* fix expected path parsing

* keep expectation path if update is enabled
2019-11-27 11:08:55 +02:00
Alexander Shalamov
17a5441768 [test-runner] Add render test runner options for injecting probes
This allows injection of memory, gfx and network probes before and
after operations defined in render test are run.
2019-11-22 15:29:25 +02:00
Mikko Pulkki
a7a7e8cbe2 [render-test] Implement gfx probe for tracking gpu resources 2019-10-30 14:32:27 +02:00
zmiao
c8576e089f [render-test] Add manifest parser for render-test-runner (#15861)
* [render-test] Add manifest parser for render-test-runner

* [render-test] Refactory manifest parser

* [render-test] Parse full manifest file through CLI

* [render-test] Add linux probe manifest
2019-10-29 20:59:29 +02:00
zmiao
2385f4cd86 [render-test] Add configurable rootPath, ignorePath via CLI arguments (#15832)
* [render-test] Add configurable path via CLI arguments

* [render-test] Change ignore path config
2019-10-23 12:04:23 +03:00
Mikko Pulkki
0ca96fd8a4 [render-test] Implement fps benchmarking tests (#15803) 2019-10-22 12:13:55 +03:00
Mikhail Pozdnyakov
b03982730e [test runner] Command line option for the tests expectations path
-e[expectationsPath], --expectationsPath=[expectationsPath]

Test expectations path.
2019-10-21 16:10:45 +03:00
Juha Alanen
13801f1b80 [render-test] Add support for query tests 2019-10-07 10:58:46 +03:00
Mikhail Pozdnyakov
1b020eeb11 [test runner] Test metrics reports. Check metrics results. 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
5de27d5200 [test runner] Expectations paths are represented with std::vector 2019-08-16 16:25:59 +03:00
Mikhail Pozdnyakov
a1904e802c [test-runner] Introduce mapbox-gl-native test expectations
The `mapbox-gl-native`-specific render tests expected results will
be stored at the `render-test/expected/` subfolder.

Test runner will automatically put new expected results there if
invoked with `UPDATE=true`.
2019-08-16 16:25:59 +03:00
Bruno de Oliveira Abinader
3b0e5a59a2 [core] Implement C++ render test runner 2019-06-25 20:10:21 +03:00