GNOME Bugzilla – Bug 765704
Add tiles.gnome.org as a proxy for third-party tile servers used today
Last modified: 2017-04-07 09:34:25 UTC
Maps currently use MapQuest as a tile server. We have heard that the ToS might change soon and we might not be able to use it as freely as we have had for long. It would be nice to be able to use and re-point tiles.gnome.org in instances like this. And also provide a seamless transition to if and when we have our own tile server. At the moment we use two different tile-servers from MapQuest. MapQuest Open Aerial: https://otile1.mqcdn.com/tiles/1.0.0/sat/#Z#/#X#/#Y#.jpg And: MapQuest OSM (Street): https://otile1.mqcdn.com/tiles/1.0.0/osm/#Z#/#X#/#Y#.png Maybe we could have https://tiles.gnome.org/aerial point to https://otile1.mqcdn.com/tiles/1.0.0/sat/ and https://tiles.gnome.org/street point to https://otile1.mqcdn.com/tiles/1.0.0/osm/ ? Any opinions from the cc people? Thanks!
Thanks Jonas for submitting this bug! I think in case we go for our own tile server, we might want to co-ordinate with a possible set up of an OpenTripPlanner transit routing server (as mentioned in https://bugzilla.gnome.org/show_bug.cgi?id=764748). It may (or may not) be possible to use the same physical machine (although I guess both uses are quite RAM-consuming).
Bad News!! Mapquest won't allow direct access to tile servers from 11th July. http://devblog.mapquest.com/2016/06/15/modernization-of-mapquest-results-in-changes-to-open-tile-access/
Hello! Anyone know who to ping about making this happen? We are discussing different offers and solutions to our tile situation and it would be nice to be able to use a proxy at the same time as we change our tile provider. Regards
I've discussed this with Andrea on IRC. He's willing to do it, but he needs to be told what you need exactly.
(In reply to Alexandre Franke from comment #4) > I've discussed this with Andrea on IRC. He's willing to do it, but he needs > to be told what you need exactly. That is great! Thanks! Mattias is finalizing the talks with Mapbox so I guess we soon know the relevant URI:s to point the proxy towards.
Hi! I have been working on a suggestion for how I want this to work. I'll attach it below. There's still some data I need to find. Andrea: does this all sound reasonable? What do you think about the Open Questions? Are there any more questions you want answered before implementation? I'll start digging upp correct URLs and get us our API key later tonight. Mattias
Created attachment 331894 [details] Proxy Setup Proposal This is my WIP proposal for a proxy setup.
Created attachment 331899 [details] Proxy Setup This should be pretty final. If anyone has opinions speak up now!
Maybe I'm being simple, but what is meant with a proxy? Especially with comments in there such as: > The API key for GraphHopper is currently embedded in the code, but for the > tilesets I think it makes sense to store them serverside and add them to the > requests if possible. Otherwise if we need to change tilesets again in the Add them how? Where's the proxy software? It seems you're requesting redirects?
(In reply to Olav Vitters from comment #9) > Maybe I'm being simple, but what is meant with a proxy? An HTTP proxy in front of our service dependencies. > Especially with comments in there such as: > > The API key for GraphHopper is currently embedded in the code, but for the > > tilesets I think it makes sense to store them serverside and add them to the > > requests if possible. Otherwise if we need to change tilesets again in the > > Add them how? Where's the proxy software? I think av is better equipped to answer this one. But I belive it's an HAProxy instance on one of our servers. > It seems you're requesting redirects? That and API key handling. Basically were trying to be prepared for the next time we need to migrate to new services.
Reverse proxy setup has been provisioned as requested [1]. Please test it out and report possible errors before we go live. [1] https://infrastructure.gnome.org/browse/puppet/tree/modules/haproxy/templates/ssl.conf#n153
So we're not allowed to use a proxy in front of Mapbox since that goes against their ToS. My current plan is to move to some sort of service definition file that Maps and other apps can use to get a list of endpoints for the currently used gis stack. In the meantime it might make sense to move to a 301 redirect instead of a proxy though. Andrea: if you could copy the current proxy rules, change them to do 301 redirects instead of proxying and put it up under gis.gnome.org/test/... That is these endpoints: gis.gnome.org/test/geocode/v1/ gis.gnome.org/test/geocode/v2/ gis.gnome.org/test/directions/v1/ gis.gnome.org/test/directions/v2/ gis.gnome.org/test/tiles/street/v1/ gis.gnome.org/test/tiles/terrain/v1/ gis.gnome.org/test/tiles/satellite/v1/ gis.gnome.org/test/tiles/satellite-streets/v1/ gis.gnome.org/test/tiles/vector/v1/ Then I could go ahead and test that everything is actually working with the released versions (by just changing the URLs and do a recompile) and if it all seems to work we can just go ahead with that. I'm also waiting for confirmation from Mapbox that this is an ok solution, but I believe it is.
Added the redirects in place as requested, please let me know how it goes.
Works fine with 3.20, and the redirection stuff in libsoup seems to have been added at a point before Ubuntu 12.04 which is the oldest still supported distro I can think of. So this should be good to go.
From IRC, as a 301/302 would expose the API key to the world: <av_work> mattiasb, a 301/302 exposes the api key <av_work> do we care at this point? <av_work> mattiasb, the idea of having a single file with the endpoints will expose the api key one way or another <av_work> but just wanted to confirm with you that's fine from your side <av_work> mattiasb, also is that against the ToS? <av_work> please verify
The 301 redirect solution wasn't ok either but we're allowed to use the current proxy setup for three months. My plan is to implement support in Maps for reading the endpoints from a service definition file hosted on gis.gnome.org. This is a better solution in the long term anyways since we'll be able to also ship properties like max-connections, copyright info and the like alongside the service definition as well making us even more future proof. I'll start working on this in the next couple of days or at GUADEC.
So, yeah, if I give someone a file named service.json, could it be hosted on gis.gnome.org then? Maybe https://gis.gnome.org/services/service.json ? Any other suggestions?
Jonas, you can directly add it over to static-web [1]. (i.e static-web/gis.gnome.org) I'll then hook it up in Apache. [1] https://git.gnome.org/browse/static-web/
Thanks! To make sure I understand: So I create the directory gis.gnome.org? And I push static-web/gis.gnome.org/services/v1/service.json, then you make it so that it will be: "https://gis.gnome.org/services/v1/service.json" ?
Just create static-web/gis.gnome.org and push the file at static-web/gis.gnome.org/v1/service.json, apache will handle what's remaining.
I pushed: https://git.gnome.org/browse/static-web/commit/?id=b3cf4756971bc61b8777b880794af36271cb4ec2 Thank you! How long until it is picked up?
I pushed at services/v1/service.json not v1/service.json, was that a mistake?
Forgot to actually reply you back, committed [1] to move the v1 directory one level down. About the 'how long until it is picked up', the answer is instantly [2] :-) Anything else I can do for you guys? [1] https://git.gnome.org/browse/static-web/commit/?id=efe45f67a10957abaf511472a0f9f3e1de196903 [2] https://static.gnome.org/gis.gnome.org/v1/
Marking as resolved, feel free to re-open if there's more business for us on this side.
I think this is all done.
I forgot about the proxy! That should have been put down at roughly the same time. :( Will append a diff to ssl.conf for the changes we need to it to disable the proxy.
Created attachment 349394 [details] [review] Proxy Teardown A diff for ssl.conf
Patch has been applied and proxy removed.