Commit Graph

93 Commits

Author SHA1 Message Date
John Firebaugh
20be32f217 [core] Always store tiles without ratio support with ratio = 1 2016-02-10 16:25:47 -08:00
John Firebaugh
40ecdeb0b6 [core] Optimize offline database schema
* Under the hood, SQLite creates surrogate keys (ROWID) anyway. We may as well take advantage of this and use the surrogates for foreign keys as well, since they are simpler and more efficient than compound foreign keys.
* Create indexes for efficient eviction queries
2016-02-10 15:40:20 -08:00
John Firebaugh
0cf450e3e5 [core] Retry errors encountered during offline downloads 2016-02-10 15:40:20 -08:00
John Firebaugh
a9e0c1b99a [core] Eliminate maximumCacheEntrySize
Instead, the eviction policy accounts for the actual size needed for an incoming put.
2016-02-10 15:40:20 -08:00
John Firebaugh
db3620c58f [core] Implement an eviction policy for OfflineDatabase
When inserting an cached resource, or removing a region, remove least-recently used resources and tiles, not used by offline regions, until the used database size, as calculated by multiplying the number of in-use pages by the page size, is less than the maximum cache size minus 5 times the page size.

In addition, OfflineDatabase may be configured to ignore cache puts of individual resources larger than a certain size.

This policy is similar but not identical to the former SQLiteCache policy:

* It accounts for offline, by exempting resources required by offline regions from eviction.
* It must delete from two tables (resources and tiles), rather than one. Currently the strategy is naive: evict 50 rows at a time from each table.
* It makes maximumCacheSize and maximumCacheEntrySize completely independent. The SQLiteCache implementation evicted when `usedSize > maximumCacheSize - 2 * maximumCacheEntrySize`. This evicts when `usedSize > maximumCacheSize - 5 * pageSize`.
* It uses a non-unlimited default value for maximumCacheSize: 50 MB. We should have always had a limit in place; "a cache without an eviction policy is a resource leak".
2016-02-10 15:40:20 -08:00
John Firebaugh
5910124010 [core] Reset SQLite statements after use in order to release locks 2016-02-10 15:40:20 -08:00
John Firebaugh
c91b44b02f Add binary for smoke-testing offline downloads 2016-02-10 15:40:20 -08:00
John Firebaugh
c3c4c7b9a6 [core] Interface and implementation for offline 2016-02-10 15:40:20 -08:00
John Firebaugh
e9302c797f [all] Don't interpret 404s on non-tile resources as "no content" 2016-02-10 15:40:20 -08:00
John Firebaugh
7eb1a91c4e [all] Do not set Response data for 404s
For AssetFileSource and the node FileSource this was already the case; this makes the other implementations consistent.
2016-02-10 15:40:20 -08:00
John Firebaugh
cdedb66387 [core] Reimplement existing caching within an offline-capable database schema 2016-02-10 15:40:20 -08:00
John Firebaugh
dbb8e58aea [core] Moving caching logic to DefaultFileSource
This results in OnlineFileSource containing precisely the logic we want for reuse by OfflineFileSource, and no more.
2016-02-10 15:40:20 -08:00
John Firebaugh
24c5507946 [tests] Add test for AssetFileSource URL encoding 2016-02-08 10:05:46 -08:00
John Firebaugh
4d29c7d5ce [core] Remove Response::Error::Reason::Canceled
There is no such thing as a cancelled response, only cancelled requests. A request that is cancelled does not have its callback called with a Response.
2016-02-04 13:18:14 -08:00
John Firebaugh
b41a73e1c1 [core] Rationalize Resource initialization 2016-01-29 14:50:15 -08:00
Bruno de Oliveira Abinader
2f86467586 Cleanup std::chrono usage
Use mbgl::Duration and mbgl::{,Milli}Seconds whenever possible.
2016-01-25 15:11:18 +02:00
John Firebaugh
a877b9b192 [core] Include prior values of caching headers in Resource
This allows the FileSource interface itself to support revalidation. We could (and probably should) now rewrite HTTPContextBase implementations as FileSource implementations.
2016-01-22 18:42:53 -08:00
John Firebaugh
fa894d7cbc [core] Polyfill std::chrono::abs 2016-01-22 18:41:54 -08:00
John Firebaugh
cef5c331fe [core] Merge rfc1123, iso8601, and parse_date into chrono.hpp and fix their API 2016-01-22 18:41:54 -08:00
John Firebaugh
1c21d0fd4c [core] Use better types for modified / expires / etag 2016-01-21 11:22:11 -08:00
John Firebaugh
c33ed50c98 [core] Eliminate Response::stale and inline Response::isExpired()
Response::isExpired() had subtle and potentially confusing behavior around Seconds::zero(). It's best to inline it and comment why.
2016-01-21 11:22:11 -08:00
Konstantin Käfer
a88e58ef3a [core] use stale glyphs
Updating glyphs is still unsupported, and there's no good use case for doing so. When we're using a stale glyph PBF, and the fresh answer contains changed to that glyph, we will continue to use the old glyph.
2016-01-15 17:04:08 +01:00
Konstantin Käfer
bd1f71cdbd [core] use stale sprite data
This is a naïve implementation that essentially merges updated data into existing data. It will *not* remove icons from the stale sprite if they aren't present in the fresh sprite (we aren't tracking the source of a sprite, and the user could have changed it as well). Similarly, it will not update icons that have changed in dimension. This is a rare edge case and probably not worth implementing.
2016-01-15 17:04:07 +01:00
Konstantin Käfer
8a964fb931 [core] use stale and refreshing TileJSON/GeoJSON data
We're now supporting using stale TileJSON and GeoJSON data. When we receive a new answer with an updated TileJSON file, we're replacing the Source's metadata with the new one and trigger updates to make sure we're loading the correct tiles. Similarly, GeoJSON data will be reparsed.
2016-01-15 12:48:23 +01:00
Konstantin Käfer
21e4029acf [core] use stale styles
This adds support for using cached styles that are stale. They're treated like changing styles; when the refreshed style changed compared to the one we've already had, we're swapping out the entire style, which might cause a slight flicker.
2016-01-15 11:59:26 +01:00
Konstantin Käfer
957415823a [core] Support multiple paths in SQLiteCache::getShared()
If you use many different caches, expired weak_ptrs will pile up in the unordered_map, but that is an edge case, and you probably shouldn't do that anyway.
2016-01-15 10:44:16 +01:00
John Firebaugh
f62d41ae05 [core] Restore existing response body
Until #2721 lands we still need this.
2016-01-14 18:38:56 -08:00
John Firebaugh
0066637e58 [core] Let SQLiteCache decide when store or refresh responses 2016-01-14 13:46:03 -08:00
John Firebaugh
2aafdbc1b8 [core] Add an explicit "not modified" indicator to Response 2016-01-14 13:46:03 -08:00
John Firebaugh
2ee1fe4178 [core] std::shared_ptr<const Response> ⇢ const Response& 2016-01-14 13:46:03 -08:00
John Firebaugh
7fb5e973ed [core] Eliminate FileCache interface
There is only one implementation and we're unlikely to add more.
2016-01-14 13:46:03 -08:00
John Firebaugh
618902e06c [core] Remove request coalescing in OnlineFileSource 2016-01-13 13:43:05 -08:00
John Firebaugh
5a7d2fd60e [core] Eliminate platform::assetRoot()
I regenerated assets.zip so that all file paths have an `assets/` prefix, as the Android AssetFileSource implementation asserts, and removed `TEST_DATA` from the paths.
2016-01-13 13:40:31 -08:00
John Firebaugh
86c8446d3a [core] Simplify asset:// implementation
* Move asset:// URL handling to DefaultFileSource.
* AssetFileSource implements FileSource interface and follows familiar implementation patterns.
* Move default implementation to platform/default, zip implementation to platform/android.
* Don't bother with modified / expires / etag -- assets are not cached so it doesn't matter.
* Don't bother with interleaving individual IO calls on the implementation thread. That adds a lot of complexity for very little benefit.
2016-01-13 13:40:31 -08:00
Konstantin Käfer
13fbf8b39b [test] Fix test case comments
Follow up to a06f28ddb7
2016-01-11 11:14:29 +01:00
Konstantin Käfer
a06f28ddb7 [core] only cache successful or NotFound responses
We don't want other types of error end up in our cache, since it'll likely evict perfectly good content.
2016-01-08 16:25:54 +01:00
Konstantin Käfer
4479f5db6c [core] don't consult cache for asset resources 2016-01-08 13:05:51 +01:00
Thiago Marcos P. Santos
c8a39e1587 [tests] Sleep more than 1s before refreshing accessed time
The cache `accessed` field resolution is 1s, so we
need to sleep a little bit more in order to make
sure that the cache entries we are refreshing will have
a different `accessed` time than the other entries.
2016-01-07 20:54:01 +02:00
Thiago Marcos P. Santos
6af69e4e56 [tests] Added utests for cache versioning 2016-01-05 17:32:46 +02:00
Konstantin Käfer
0377e23c20 [test] terminate test server when stdin closes 2016-01-05 14:14:23 +01:00
John Firebaugh
79368a0ecf [test] Remove terminate-when-inactive behavior from the test server
It caused #3163.

Attempts at having the child exit automatically when the parent exits were stymied
by the fact that unix sucks: http://stackoverflow.com/q/284325/52207.
2016-01-04 17:27:01 -08:00
Thiago Marcos P. Santos
e6dc4fe49d [tests] Add unit tests for SQLiteCache size limit 2015-12-28 14:48:02 +02:00
John Firebaugh
7a6076ea9d [tests] Test with OnlineFileSource
When we introduce OfflineFileSource, the behavior of existing tests should
not change.
2015-12-23 14:55:12 -08:00
Konstantin Käfer
d5fb535058 [core] add namespace comment to closing brace 2015-12-03 15:38:36 +01:00
Thiago Marcos P. Santos
7caaebb162 [tests] Fix/improve Asset utests
ASSET=zip make test-* was broken and now works.

Also added a test stressing concurrent file access.
2015-12-01 11:49:02 +02:00
Thiago Marcos P. Santos
c117b7e3f4 [core] Removed libuv dependency from AssetFSContext 2015-12-01 11:49:02 +02:00
Thiago Marcos P. Santos
cb06526b10 [tests] Remove libuv dependency from the tests 2015-12-01 11:49:02 +02:00
Bruno de Oliveira Abinader
31fd2822f5 [core] Replace time_t with std::chrono::seconds
Added aliases for std::chrono typedefs (eg. 'Seconds' for
std::chrono::seconds). These aliases are used together with templated
helper functions to replace time_t with std::chrono::seconds for most
cases, in particular for 'modified' and 'expires' values in Response.
2015-11-27 17:47:02 +02:00
Konstantin Käfer
b5f7cafab4 [core] more robust max-age parsing 2015-11-20 14:39:44 +01:00
John Firebaugh
7137239cbd [core] Use std::unique_ptr for FileSource request 2015-11-16 12:25:47 -08:00