Commit Graph

37 Commits

Author SHA1 Message Date
Mikhail Pozdnyakov
b2ab0804f8 [render tests][tile mode] Enable automatic labels cut-off detection 2020-04-06 19:18:06 +03:00
Mikhail Pozdnyakov
c5155dcf33 [render test runner] Enable pixel ratio in tile mode 2020-03-05 16:12:43 +02:00
Alexander Shalamov
879c44f661 [core] Modularize FileSource codebase (#15768)
* [core] Introduce FileSourceManager and use it for default platform impl

  - Add `FileSourceManager` interface that provides access to `FileSource`
    instances and means of registering / unregistering `FileSource` factories
  - Split `DefaultFileSource` into smaller parts
  - Add `DatabaseFileSource` interface and it's default implementation
  - Remove inter-dependencies between concrete `FileSource` classes

* [build] Add files to next build system

* [core] Add generic property setters / getters

* [core] Remove setOnlineStatus from OnlineFileSource interface

* [core] Hide threading implementation details from DatabaseFileSource interface

* [core] Make DB file source methods virtual

* [core] Add documentation for DatabaseFileSource and rename one method

* [core] Use simple callback instead of ActorRef

* [core] Remove ActorRef from OnlineFileSource public header

* [core] Add callback to FileSource::forward async API

* [core] Pass OfflineRegionDefinition by value

* [core] Update tests to use modular file sources

* [core] Update unit tests

* [core] Update unit tests after rebase

* [core] Backport low prio fix for cached requests

* [core] Backport pack database

* [core] Return removed factory from unRegisterFileSourceFactory

* [core] Rename shadowed args in onlinefilesource

* [core] Remove simple std::function callback aliases

* [core] Expose online file source property keys in public header file

* [test-runner] Add proxy file source test runner

* [cache] Update mbgl-cache utility to use new file source

* [metrics] Rebaseline binary size metrics

* [offline] Update offline utility

* [core] Update changelog
2020-01-13 10:57:23 +02:00
Thiago Marcos P. Santos
af8092a86f [render-test] Use image from the cache for addImage operation
Instead of trying to read it from the filesystem.
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
f0d2a2f636 [render-test] Effectively use the offline cache for fixtures 2020-01-08 13:07:12 +02:00
Mikhail Pozdnyakov
64a37ceb05 [test runner] Rendering a single tile in Tile mode 2019-12-18 13:48:52 +02:00
Thiago Marcos P. Santos
c3854cedc2 [render-tests] Fix tests not rendering when both metrics and render fails 2019-12-17 12:38:42 +03:00
Thiago Marcos P. Santos
ad1a7c595d [render-test] Cleanup on the render tests results
Simplify the code path and make it print both rendering error and
metrics on fail.
2019-12-16 22:52:01 +02:00
Mikhail Pozdnyakov
515cb4e0fc [render test] Fix probes failure reports 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
b2d06b8a17 [test runner] Split operations parsing and execution 2019-12-13 15:34:06 +02:00
Thiago Marcos P. Santos
1455f6f581 [metrics] Make the parsing of metrics even more strict
File must be `metrics.json`.
2019-11-28 12:34:22 +02:00
Thiago Marcos P. Santos
09b724ea33 [render-test] Fix regex matching the directory name
This code was matching `style.json` instead of `metrics.json` if
the `style.json` was contained inside `metrics/.../style.json`.
2019-11-27 18:32:58 +02:00
zmiao
11321497da [render-test] Add metric_paths into Manifest (#15973)
* [render-test]Add expected metrics path in manifest file

* seperate metrics.json and expected.png

* fix expected path parsing

* keep expectation path if update is enabled
2019-11-27 11:08:55 +02:00
Mikko Pulkki
a7a7e8cbe2 [render-test] Implement gfx probe for tracking gpu resources 2019-10-30 14:32:27 +02:00
zmiao
c8576e089f [render-test] Add manifest parser for render-test-runner (#15861)
* [render-test] Add manifest parser for render-test-runner

* [render-test] Refactory manifest parser

* [render-test] Parse full manifest file through CLI

* [render-test] Add linux probe manifest
2019-10-29 20:59:29 +02:00
Juha Alanen
360b8b4247 [core,android,darwin,qt] Add fields related to feature-state
Move the fields from geometry.hpp/feature.hpp as they are not
part of the GeoJSON specification.
2019-10-28 16:46:55 +02:00
zmiao
2385f4cd86 [render-test] Add configurable rootPath, ignorePath via CLI arguments (#15832)
* [render-test] Add configurable path via CLI arguments

* [render-test] Change ignore path config
2019-10-23 12:04:23 +03:00
zmiao
068befa9df [test-runner] Run render test runner as a NativeActivity app on android device (#15827)
* [test-runner]Add android render-test-runner NativeActivity app

* [test-runner] Fix cmake for CI build failure

* [test-runner] Update folder location, change app configureation

* [test-runner] Add running instruction
2019-10-22 16:32:21 +03:00
Mikko Pulkki
0ca96fd8a4 [render-test] Implement fps benchmarking tests (#15803) 2019-10-22 12:13:55 +03:00
Mikhail Pozdnyakov
b03982730e [test runner] Command line option for the tests expectations path
-e[expectationsPath], --expectationsPath=[expectationsPath]

Test expectations path.
2019-10-21 16:10:45 +03:00
Mikhail Pozdnyakov
9adc78134c [test runner] Threshold for memory metrics 2019-10-21 16:10:45 +03:00
Juha Alanen
b53093b522 [render-test] Add support for collecting network metrics 2019-10-16 14:14:46 +03:00
Thiago Marcos P. Santos
8b3a222c81 [render-test] Add tolerance to file-size tests 2019-10-14 21:15:21 +03:00
Thiago Marcos P. Santos
2fb1e47b75 [render-test] No camelCase on the metrics.json tag
It is ugly.
2019-10-11 18:29:52 +03:00
Thiago Marcos P. Santos
307f9cede1 [render-test] Prettyfy metrics output 2019-10-11 18:29:52 +03:00
Thiago Marcos P. Santos
63116154fe [render-test] Only write metrics if they are available
Otherwise the file will have an empty tag.
2019-10-10 19:54:51 +03:00
Thiago Marcos P. Santos
629c0e1142 [render-test] Add tests for filesize probe
And make sure we run it on the bots
2019-10-10 19:54:51 +03:00
Thiago Marcos P. Santos
a9f06f3b52 [render-test] Add instruction for verifying file size
For a given absolute or relative path, measure the size of the file.

This is useful for measuring if the cache has increased as expected
after moving a camera or decreased after cleaning up.

In a more hackish manner, could be used for collecting binary size
statistics of shared libraries.
2019-10-10 19:54:51 +03:00
Thiago Marcos P. Santos
026e15fe3d [render-test] Fix rootPath parameter being ignored
It was doing nothing and always using the default root path.
2019-10-10 19:54:51 +03:00
Mikhail Pozdnyakov
b4e0224a57 [test runner] Fix finding of expected images paths
Before this change, the found paths to the expected images
erroneously included the path to the `metrics.json` file
(if this file was present) leading to raising of an
unhandled exception.
2019-10-09 14:35:08 +03:00
Juha Alanen
13801f1b80 [render-test] Add support for query tests 2019-10-07 10:58:46 +03:00
Alexander Shalamov
9a42bdd7ae [core] Add filtering option to test-runner
This change adds filter command line option to render test runner,
so that test can be filtered by providing regular expression, for instance:

mbgl-render-test -f .*hillshade.*
2019-08-28 13:55:36 +03:00
Mikhail Pozdnyakov
766392f1f8 [test runner] Use allocation peak values for memory metrics 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
1b020eeb11 [test runner] Test metrics reports. Check metrics results. 2019-08-27 12:34:09 +03:00
Mikhail Pozdnyakov
a1904e802c [test-runner] Introduce mapbox-gl-native test expectations
The `mapbox-gl-native`-specific render tests expected results will
be stored at the `render-test/expected/` subfolder.

Test runner will automatically put new expected results there if
invoked with `UPDATE=true`.
2019-08-16 16:25:59 +03:00
Aleksandar Stojiljkovic
b5d71fbca0 mbgl-render-test runner - use the same tolerance as with node render test
Two render tests failing only with mbgl-render-tests are added to platform
specific ignores. This is to be extended with fill-extrusion-pattern failing only on mac.
2019-08-07 10:26:39 +03:00
Bruno de Oliveira Abinader
3b0e5a59a2 [core] Implement C++ render test runner 2019-06-25 20:10:21 +03:00