Commit Graph

129 Commits

Author SHA1 Message Date
Thiago Marcos P. Santos
75475cfdbd [render-test] Fix logging
Was broken after logging got moved to its own thread.
2020-04-17 17:14:37 +03:00
Mikhail Pozdnyakov
e6c156420d [core] PlacedSymbolData contains layer id 2020-04-09 17:18:21 +03:00
Mikhail Pozdnyakov
8986b558eb [render tests][tile mode] Check symbol icons cut-offs 2020-04-06 19:18:06 +03:00
Mikhail Pozdnyakov
b2ab0804f8 [render tests][tile mode] Enable automatic labels cut-off detection 2020-04-06 19:18:06 +03:00
Mikhail Pozdnyakov
2ae96ad687 [core] Introduce Renderer::clearData() instead of keepRenderData map options
`Renderer::clearData()` is a better API than the removed `MapOptions::keepRenderData()`:
- gives more flexibility to the client
- similar to the existing `Renderer::reduceMemoryUse()`
- the `MapOptions::keepRenderData()` API implementation could not handle the raise condition, which
happened if the new still image request had come before all tiles from the previous requests were loaded.

Co-authored-by: Dane Springmeyer <dane@mapbox.com>
2020-03-20 11:43:59 +02:00
zmiao
6989ee6e03 [build] Use found code sign identities 2020-03-19 10:20:44 +02:00
Mikhail Pozdnyakov
9e80a9f922 [render tests] Unset MapOptions::keepRenderData() flag in tile mode 2020-03-18 20:14:15 +02:00
Juha Alanen
428f666adb [test][ios] Update iOS render test data setup script 2020-03-12 15:24:37 +02:00
Mikhail Pozdnyakov
c5155dcf33 [render test runner] Enable pixel ratio in tile mode 2020-03-05 16:12:43 +02:00
Juha Alanen
4b148dc202 [build][ios] Refactor iOS test runner CI job configs 2020-02-27 10:09:35 +02:00
Mikhail Pozdnyakov
aad13f6a3f [core] FileSourceManager::getFileSource() returns PassRefPtr 2020-02-26 22:31:59 +02:00
Mikhail Pozdnyakov
7c48611f9e [core] Consolidate file source keys in the generic FileSource header
Thus we do not need to include specific headers to use `{set|get}Property`
API with key constants.
2020-02-25 23:15:47 +02:00
Juha Alanen
a5ec862a2a [test-runner] Share common code between test runners
Also remove some unnecessary files.
2020-02-18 16:10:23 +02:00
Juha Alanen
6d815401b1 [test-runner] Share common code between iOS test runners 2020-02-14 12:01:52 +02:00
Thiago Marcos P. Santos
f7fef70e9d [build] Move the next buildsystem to the root
This will make the `next` buildsystem no longer the `next`, but the
`actual`. The idea is to simplify the build, removing scripts, to
make the platform buildsystem generated by CMake more compatible
with IDEs and make development more streamlined. It will also make
cross compilation easier.
2020-02-07 20:12:44 +02:00
zmiao
d7c8104dec [test-runner] Enable rebaseline for next-ios-render-test-runner (#16147) 2020-01-28 21:48:22 +02:00
Thiago Marcos P. Santos
0de6ffc383 [build] Cleanup CircleCI configuration
- Remove the legacy build jobs
- Remove the next- namespace.
2020-01-28 00:44:07 +02:00
zmiao
429292fb98 [test-runner] Add iOS RenderTestRunner test metrics (#16139)
* [test-runner] Add iOS RenderTestRunner test metrics

* [test-runner] Add ios probe metrics
2020-01-24 15:45:47 +02:00
zmiao
1ade6c7784 [test-runner] Enable IOS RenderTestRunner and run it on Firebase (#16111)
* [test-runner] Setup iOS RenderTestRunner

* [test-runner] Make app runnable on firebase

* Add iOS platform dependent expectation and ignores; Run prepare script via cmake

* Make RenderTestRunner report test status
2020-01-21 23:23:35 +02:00
Alexander Shalamov
879c44f661 [core] Modularize FileSource codebase (#15768)
* [core] Introduce FileSourceManager and use it for default platform impl

  - Add `FileSourceManager` interface that provides access to `FileSource`
    instances and means of registering / unregistering `FileSource` factories
  - Split `DefaultFileSource` into smaller parts
  - Add `DatabaseFileSource` interface and it's default implementation
  - Remove inter-dependencies between concrete `FileSource` classes

* [build] Add files to next build system

* [core] Add generic property setters / getters

* [core] Remove setOnlineStatus from OnlineFileSource interface

* [core] Hide threading implementation details from DatabaseFileSource interface

* [core] Make DB file source methods virtual

* [core] Add documentation for DatabaseFileSource and rename one method

* [core] Use simple callback instead of ActorRef

* [core] Remove ActorRef from OnlineFileSource public header

* [core] Add callback to FileSource::forward async API

* [core] Pass OfflineRegionDefinition by value

* [core] Update tests to use modular file sources

* [core] Update unit tests

* [core] Update unit tests after rebase

* [core] Backport low prio fix for cached requests

* [core] Backport pack database

* [core] Return removed factory from unRegisterFileSourceFactory

* [core] Rename shadowed args in onlinefilesource

* [core] Remove simple std::function callback aliases

* [core] Expose online file source property keys in public header file

* [test-runner] Add proxy file source test runner

* [cache] Update mbgl-cache utility to use new file source

* [metrics] Rebaseline binary size metrics

* [offline] Update offline utility

* [core] Update changelog
2020-01-13 10:57:23 +02:00
Thiago Marcos P. Santos
d148908cf1 [render-test] Make the manifest mandatory
At the moment we have an invalid path as default.
2020-01-08 14:58:05 +02:00
Thiago Marcos P. Santos
5a0f43037f [render-test] Remove positional argument
Not in use, for running a single test, use -f instead.
2020-01-08 14:58:05 +02:00
Thiago Marcos P. Santos
d08bb49c34 [core] Make it possible to reopen the database read-only for testing
Needed for Android, because the database is in readonly filesystem
and also because it is annoying to see files modified every time we
run the tests.
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
13c5505572 [metrics] Use the cache database instead of file paths for fixtures
Now that we use a cache, we no longer need "asset_path" and
"vendor_path".
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
af8092a86f [render-test] Use image from the cache for addImage operation
Instead of trying to read it from the filesystem.
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
f0d2a2f636 [render-test] Effectively use the offline cache for fixtures 2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
26cfddaf47 [render-test] Add access token to the test manifest
Needed if we want to run tests online. Will default to the
MAPBOX_ACCESS_TOKEN environment value.
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
4d815d8306 [render-test] Add flag to run tests online
And also make it not go to the network by setting the status
explicitly when when the -o flag is not passed.
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
08f03b1bbb [render-test] Add cache database path to the manifest
A path relative to the manifest pointing at the data cache.
2020-01-08 13:07:12 +02:00
Thiago Marcos P. Santos
502ddd3c8a [render-test] Fixed typo and remove unused 2020-01-08 13:07:12 +02:00
Mikhail Pozdnyakov
648392585e [test runner] Show all tiles independently in Tile mode 2019-12-18 13:48:52 +02:00
Mikhail Pozdnyakov
64a37ceb05 [test runner] Rendering a single tile in Tile mode 2019-12-18 13:48:52 +02:00
Thiago Marcos P. Santos
c3854cedc2 [render-tests] Fix tests not rendering when both metrics and render fails 2019-12-17 12:38:42 +03:00
Thiago Marcos P. Santos
ad1a7c595d [render-test] Cleanup on the render tests results
Simplify the code path and make it print both rendering error and
metrics on fail.
2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
8869d09b25 [build] Collect and publish new baselines
Also fail the bot in case there are new baselines.
2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
970cabe0eb [build] Reorganize the render tests
Move everything that might collect metrics to the metrics/ directory,
the only test manifests inside the render-test/ directory are the
unit tests of the probes.
2019-12-16 22:52:01 +02:00
Thiago Marcos P. Santos
1c138124a6 [render-test] Fix metrics passing when failing
Metrics will now fail, but the tool will return 0 when *ONLY* metrics
fail, to make the bot continue. We will also run ignored tests and
check if they are ignored but passing.
2019-12-16 22:52:01 +02:00
Mikhail Pozdnyakov
65140e3e60 [test runner] Explicitly initialize TestContext POD members 2019-12-14 23:49:58 +02:00
Alexander Shalamov
1fc4cd649e [test-runner] Fix network probe 2019-12-13 21:07:56 +02:00
Mikhail Pozdnyakov
515cb4e0fc [render test] Fix probes failure reports 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
06a2930342 [test runner] Refactor injected probes operations code 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
b2d06b8a17 [test runner] Split operations parsing and execution 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
426f714c04 [test runner] Introduce TestOperation 2019-12-13 15:34:06 +02:00
Mikhail Pozdnyakov
4a30f74699 [test runner] Introduce generic TestContext 2019-12-13 15:34:06 +02:00
Alexander Shalamov
91ba9b655e [test-runner] Don't update nor create metrics baseline for ignored test 2019-12-12 17:03:25 +02:00
Alexander Shalamov
f590b1082a [test-runner] Fix -Werror=shadow issues in test runner 2019-12-12 13:26:30 +02:00
zmiao
f848f07ad9 [metrics] Run Android RenderTestRunner bot in rebaseline mode (#16036) 2019-12-11 11:50:27 +02:00
Alexander Shalamov
c0fee57367 [test-runner] Add desctiption for --update command 2019-12-10 22:02:58 +02:00
Mikhail Pozdnyakov
e31ad307f4 [test runner] Report absolute path to the results html file 2019-12-10 19:04:45 +02:00
Alexander Shalamov
a9ba572118 [test-runner] Add rebaseline flag
Rebaseline flag allows to update or create new metric, in cases when
metric check is failed or expected metric is missing.
2019-12-09 17:53:34 +02:00