GNOME Bugzilla – Bug 697218
Use Nominatim service for geocoding
Last modified: 2013-07-28 19:40:20 UTC
The yahoo's geocoding service we use seems pretty limited. It can hardly find any streets in Finland and Sweden and has apparently support for limited locales (bug#697216). Nominatim[1] is an open service that seems to have a lot more data to offer and provides its data under ODbL license[2]. You can try the service here[3]. Perhaps it would be better to use this service instead or it could simply complement yahoo's service? [1] http://wiki.openstreetmap.org/wiki/Nominatim [2] http://wiki.openstreetmap.org/wiki/Nominatim_usage_policy http://www.openstreetmap.org/copyright [3] http://nominatim.openstreetmap.org
I tried to search for the city I studied "Peshawar" and since I limited search to '1' results, it gave me results for a restaurant by that name that is not at all famous: http://nominatim.openstreetmap.org/search?q=Peshawar&format=xml&polygon=1&addressdetails=1&accept-language=en_US They certainly have a *a lot* more data than yahoo.
(In reply to comment #1) > I tried to search for the city I studied "Peshawar" and since I limited search > to '1' results, it gave me results for a restaurant by that name that is not at > all famous: > > http://nominatim.openstreetmap.org/search?q=Peshawar&format=xml&polygon=1&addressdetails=1&accept-language=en_US Ah, actually it was because of 'accept-language=en_US' cause all other results have Urdu strings in them.
(In reply to comment #0) > The yahoo's geocoding service we use seems pretty limited. It can hardly find > any streets in Finland and Sweden I'd like hard-data instead of "seems pretty limited". The geocoding > and has apparently support for limited > locales (bug#697216). The server is just buggy. > Nominatim[1] is an open service that seems to have a lot > more data to offer Does it have translations? Does it pass our tests? Note the country and city names getting translated: $ LC_ALL=pt_BR.UTF-8 ./test-gcglib lyon Got geocode search answer: Lyon, França @ 45,759392, 4,828980 [...] $ LC_ALL=fr_FR.UTF-8 ./test-gcglib london Got geocode search answer: Londres, Royaume Uni @ 51,506321, -0,127140 [...] > and provides its data under ODbL license[2]. You can try the > service here[3]. Perhaps it would be better to use this service instead or it > could simply complement yahoo's service? The usage looks even more restrictive than Yahoo's... geonames.org is another option: http://www.geonames.org/export/web-services.html
(In reply to comment #3) > (In reply to comment #0) > > The yahoo's geocoding service we use seems pretty limited. It can hardly find > > any streets in Finland and Sweden > > I'd like hard-data instead of "seems pretty limited". The geocoding Ok, Search for "Annankau". This is one of *the* main streets of Helsinki. While yahoo will not yield any exact results, you get loads of results from Nominatim. Same goes for "Otavantie" (my address). While this is not such a central street, its still in the main city. I can provide more examples if you like? Perhaps Mattias can provide example for Sweden. > > and has apparently support for limited > > locales (bug#697216). > > The server is just buggy. > > > Nominatim[1] is an open service that seems to have a lot > > more data to offer > > Does it have translations? I think so. Will check in more detail. >Does it pass our tests? I haven't yet tried to use it in geocode-glib cause I'm not yet sure of what you think of this. > Note the country and city names getting translated: > $ LC_ALL=pt_BR.UTF-8 ./test-gcglib lyon > Got geocode search answer: > Lyon, França @ 45,759392, 4,828980 > [...] > $ LC_ALL=fr_FR.UTF-8 ./test-gcglib london > Got geocode search answer: > Londres, Royaume Uni @ 51,506321, -0,127140 > [...] I'll check this out. > > and provides its data under ODbL license[2]. You can try the > > service here[3]. Perhaps it would be better to use this service instead or it > > could simply complement yahoo's service? > > The usage looks even more restrictive than Yahoo's... How so? If you are referring to the restrictions on bulk usage of their API, we could create our own instance of their service? > geonames.org is another option: > http://www.geonames.org/export/web-services.html That was the first alternative I tried and that is as bad as Yahoo. I even get the feeling that Yahoo is using the exact same database.
About the usage limit, we had a chat with one of the Nominatim people: <zeenix> me, jku and mattiasb are working on Maps application for GNOME: https://live.gnome.org/Maps <zeenix> currently we are using yahoo's places service through geocode-glib but we are hoping to start using nominatim <zeenix> one issue is that we'd like to support completion in search entry <zeenix> nominatim's policy says that clients must not do that but we were hoping that its ok to do so on words rather than characters <zeenix> combined with excessive caching <lonvia> zeenix: I'm still not convinced that the results you get back are actually useful for a completion service. <jku> lonvia: let's say I'm searching for "Lokkalantie 16, Helsinki, Finland" ... <jku> lonvia: the result after one word is already close, result after two words is 100% correct <jku> just showing that "hey, we have a result, you can stop writing now" would be good <lonvia> Well, I'm not saying it's a no-no. It depends on how you implement it. <lonvia> Basically, if the amount of traffic it produces is not much higher than normal search and you don't have too many invalid results, it's probably ok. <zeenix> by launching search upon encountering ',', ' ' or timeout <lonvia> The timeout might get you into trouble. <lonvia> Do you plan to use a proxy server or query osm.org directly? <zeenix> directly using would be nice, unless there is already a proxy that would be happy to serve us <lonvia> You would have to set up your own. <zeenix> ah ok, we'll talk to our board then <jku> lonvia: you mean, if we want a proxy? or do you mean using osm.org is a no-no in general for something like gnome-maps? <lonvia> I'd prefer if you set up a proxy. <lonvia> If your completion service gets out of hand, the only way to handle that without a proxy is to completely block it. <lonvia> If you set up a proxy, you might be able to put in some caching or set up your own server to save the day. <zeenix> good point <jku> also I remember that mapquest had a nominatim server, is that correct? is it pretty much the same as the one on osm.org? <mattiasb> +1 on that. Then we could do a switch to an own hosted nominatim in the future. <jku> haven't looked at their api terms though <lonvia> mapquest has exactly the same API but their usage limits are pretty low as well <lonvia> You might be able to talk to them though. <mattiasb> lonvia: I read the installation guide btw. 32Gb for importing planet.osm is a lot, but when all data is imported would you say all that memory still is needed? As in: could we do a full import once on some big server and transfer the data to something smaller and then continually sync that later on? <mattiasb> I'm getting ahead of myself as always, but still curious <lonvia> The 32MB are a minimum for running the server as well, I'm afraid. Especially, if you plan to do continuous updates. Then you want the SSD as well. <mattiasb> ok, good to know <zeenix> mattiasb: if you keep talking of implementation details, we might end-up giving the task of setting the server to you :)
(In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #0) > > > The yahoo's geocoding service we use seems pretty limited. It can hardly find > > > any streets in Finland and Sweden > > > > I'd like hard-data instead of "seems pretty limited". The geocoding > > Ok, Search for "Annankau". This is one of *the* main streets of Helsinki. While > yahoo will not yield any exact results, you get loads of results from > Nominatim. Same goes for "Otavantie" (my address). While this is not such a > central street, its still in the main city. I can provide more examples if you > like? > > Perhaps Mattias can provide example for Sweden. I'll give some examples later if needed (on my work computer now) > > The usage looks even more restrictive than Yahoo's... > > How so? If you are referring to the restrictions on bulk usage of their API, we > could create our own instance of their service? Here's some installation instructions for that (just for reference): http://wiki.openstreetmap.org/wiki/Nominatim/Installation > > geonames.org is another option: > > http://www.geonames.org/export/web-services.html > > That was the first alternative I tried and that is as bad as Yahoo. I even get > the feeling that Yahoo is using the exact same database. Not sure if it uses the same database but the results are not good. osm.org is using both geonames and nominatim and the results from geonames are underwhelming. If I search for Andra Långgatan 30 or Otterhällegatan 1 I only get results from nominatim (and I get good results to boot) and none from geonames. I have yet to get a useful result from geonames actually.
(In reply to comment #3) > (In reply to comment #0) > > The yahoo's geocoding service we use seems pretty limited. It can hardly find > > any streets in Finland and Sweden > > I'd like hard-data instead of "seems pretty limited". I've understood the particular yahoo api is for "places", just like geonames. They don't actually say this anywhere clearly AFAICT but my impression is that the geoplanet api is for "places" and to get street addresses you need to pay for "BOSS Placefinder".
(In reply to comment #7) > I've understood the particular yahoo api is for "places", just like geonames. > They don't actually say this anywhere clearly AFAICT but my impression is that > the geoplanet api is for "places" and to get street addresses you need to pay > for "BOSS Placefinder". Found it. Well hidden but the text is clear: | Yahoo! GeoPlanet is not a geocoder, so addresses in requests are resolved | to the smallest bounding Place, usually a town or a postal code. Also, | WOEIDs are not assigned to individual house numbers or street names. http://developer.yahoo.com/geo/geoplanet/guide/additional_information.html
(In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #0) > > > The yahoo's geocoding service we use seems pretty limited. It can hardly find > > > any streets in Finland and Sweden > > > > I'd like hard-data instead of "seems pretty limited". The geocoding > > Ok, Search for "Annankau". This is one of *the* main streets of Helsinki. While > yahoo will not yield any exact results, you get loads of results from > Nominatim. Same goes for "Otavantie" (my address). While this is not such a > central street, its still in the main city. I can provide more examples if you > like? > > Perhaps Mattias can provide example for Sweden. > > > > and has apparently support for limited > > > locales (bug#697216). > > > > The server is just buggy. > > > > > Nominatim[1] is an open service that seems to have a lot > > > more data to offer > > > > Does it have translations? > > I think so. Will check in more detail. > > >Does it pass our tests? > > I haven't yet tried to use it in geocode-glib cause I'm not yet sure of what > you think of this. > > > Note the country and city names getting translated: > > $ LC_ALL=pt_BR.UTF-8 ./test-gcglib lyon > > Got geocode search answer: > > Lyon, França @ 45,759392, 4,828980 > > [...] > > $ LC_ALL=fr_FR.UTF-8 ./test-gcglib london > > Got geocode search answer: > > Londres, Royaume Uni @ 51,506321, -0,127140 > > [...] > > I'll check this out. Nominatim does seem to translate: http://nominatim.openstreetmap.org/search?q=lyon&format=xml&addressdetails=1&accept-language=pt_BR http://nominatim.openstreetmap.org/search?q=lyon&format=xml&addressdetails=1&accept-language=fr_FR Though not as well AFAICT. "Lyon" is Londres in french?
(In reply to comment #9) > (In reply to comment #4) > > (In reply to comment #3) > > > (In reply to comment #0) > > > Note the country and city names getting translated: > > > $ LC_ALL=pt_BR.UTF-8 ./test-gcglib lyon > > > Got geocode search answer: > > > Lyon, França @ 45,759392, 4,828980 > > > [...] > > > $ LC_ALL=fr_FR.UTF-8 ./test-gcglib london > > > Got geocode search answer: > > > Londres, Royaume Uni @ 51,506321, -0,127140 > > > [...] > > > > I'll check this out. > > Nominatim does seem to translate: > > http://nominatim.openstreetmap.org/search?q=lyon&format=xml&addressdetails=1&accept-language=pt_BR > http://nominatim.openstreetmap.org/search?q=lyon&format=xml&addressdetails=1&accept-language=fr_FR > > Though not as well AFAICT. "Lyon" is Londres in french? Ah nm, I asked teuf about that and he pointed out that you searched for "london' with fr_FR locale. :) Here is what nominatim gives for that: http://nominatim.openstreetmap.org/search?q=london&format=xml&addressdetails=1&accept-language=fr_FR So translation seems fine. I think all required info for making the decision to switch (or at least to try to) has been provided by now?
(In reply to comment #10) <snip> > So translation seems fine. I think all required info for making the decision to > switch (or at least to try to) has been provided by now? Let's see the code first.
Created attachment 247385 [details] [review] lib: Port reverse geocoding API to Nominatim
(In reply to comment #12) > Created an attachment (id=247385) [details] [review] > lib: Port reverse geocoding API to Nominatim While I work on forward geocoding next, I thought I get some feedback on the reverse geocoding port first. This patches goes on top of patches in bug#702775.
Review of attachment 247385 [details] [review]: What's the geocode-forward.c changes for? ::: geocode-glib/geocode-forward.c @@ +101,3 @@ + const char *pf_attr; + const char *xep_attr; +} rev_attrs_map[] = { That's the same TP/XEP attributes table already in geocode-forward.c ::: geocode-glib/geocode-reverse.c @@ +158,3 @@ + + value = json_reader_get_string_value (reader); + if (value && *value == '\0') Or you could do: if (value != NULL && *value != '\0') @@ +205,3 @@ msg = NULL; + g_set_error_literal (error, The error messages are much too coarse. Any way to do better? ::: geocode-glib/test-gcglib.c @@ +123,3 @@ + g_assert_cmpstr (g_hash_table_lookup (ht, "country"), ==, "United Kingdom"); + g_assert_cmpstr (g_hash_table_lookup (ht, "state_district"), ==, "South East England"); + g_assert_cmpstr (g_hash_table_lookup (ht, "region"), ==, "England"); Looks like we've lost the "area of town" information?
Review of attachment 247385 [details] [review]: > What's the geocode-forward.c changes for? Because forward module uses _geocode_parse_resolve_json() which was in reverse module prior to this patch. I'm moving that to forward as reverse doesn't use it anymore. Probably wont need this if I in the end squash the patch to port forward module into this patch.. ::: geocode-glib/geocode-forward.c @@ +101,3 @@ + const char *pf_attr; + const char *xep_attr; +} rev_attrs_map[] = { If you are referring to attrs_map used by tp_attr_to_gc_attr, thats doing the opposite mapping AFAICT. ::: geocode-glib/geocode-reverse.c @@ +205,3 @@ msg = NULL; + g_set_error_literal (error, I don't think so but there is not many different error scenerios possible here: * Given coordinates are wrong (which we should have g_return_*if_fail for anyawys). * Internal server error or server down Apart from that, AFAIK Nominatim always retrurns at least some detail about a given position. @@ +207,3 @@ + g_set_error_literal (error, + GEOCODE_ERROR, + GEOCODE_ERROR_NOT_SUPPORTED, Perhaps we should use G_IO_ERROR_FAILED or G_IO_ERROR_NOT_FOUND here? ::: geocode-glib/test-gcglib.c @@ +123,3 @@ + g_assert_cmpstr (g_hash_table_lookup (ht, "country"), ==, "United Kingdom"); + g_assert_cmpstr (g_hash_table_lookup (ht, "state_district"), ==, "South East England"); + g_assert_cmpstr (g_hash_table_lookup (ht, "region"), ==, "England"); Not exactly, since the yahoo's reverse geocoding service seems to be unavailable and this testcase fails before this patch, in practice we are actually gaining a lot more info than losing anything.
Created attachment 247682 [details] [review] lib: Port reverse geocoding API to Nominatim Part of this change went to the parent patch accidently.
*** Bug 697216 has been marked as a duplicate of this bug. ***
FWIW, while waiting for json-glib to unescape strings (should it?) you could also run a regex replace on the strings we get. It would be good enough to unblock this code.
(In reply to comment #18) > FWIW, while waiting for json-glib to unescape strings (should it?) I'm not waiting on json-glib but was more looking for a time to do it myself. Now that I'm finally back on this task and managed to solve the issue with place description/name creation, I'm looking into this (bug#703212 that is). > you could > also run a regex replace on the strings we get. It would be good enough to > unblock this code. How do you use a regex for this?
Created attachment 250157 [details] [review] place: Add more properties
Created attachment 250158 [details] [review] lib: Make GeocodePlace:name writable There is no good reason to keep this property construct-only and we'll need to overwrite it inside geocode-glib as well in a following patch.
Created attachment 250159 [details] [review] lib: Fix a typo in a doc comment
Created attachment 250160 [details] [review] lib: Compare against the correct values We were comparing latitude to longitude and viceversa in a few testcases.
Created attachment 250161 [details] [review] lib: Port forward geocoding API to Nominatim This patch completes porting from Yahoo Places API to Nominatim. Apart from keeping the working testcases still working, this patch also makes some of the failing testcases working again.
Review of attachment 247682 [details] [review]: Looks good
Review of attachment 250157 [details] [review]: Looks fine.
Review of attachment 250158 [details] [review]: Fine.
Review of attachment 250159 [details] [review]: ::: geocode-glib/geocode-place.h @@ +67,3 @@ * GeocodePlaceType: * @GEOCODE_PLACE_TYPE_UNKNOWN: Type is unknown for this place. + * @GEOCODE_PLACE_TYPE_BUILDING: A building or house. All those changes aren't typoes, split them up please.
Review of attachment 250160 [details] [review]: Looks good.
Review of attachment 250161 [details] [review]: Rest looks fine. ::: geocode-glib/test-gcglib.c @@ +376,3 @@ g_object_unref (object); + /* Nominatim lists our desired town at index 5 */ What are the other towns? Alternatively, I'd rather you searched for "Bonneville, France" if it means it's the first hit. @@ +468,3 @@ g_assert_cmpint (g_list_length (list), ==, 10); + place = list->next->data; /* First result is the state, not the town */ I don't really like this. Do we have a way of filtering the state out in code, and checking that it's the first one?
Review of attachment 250159 [details] [review]: ::: geocode-glib/geocode-place.h @@ +67,3 @@ * GeocodePlaceType: * @GEOCODE_PLACE_TYPE_UNKNOWN: Type is unknown for this place. + * @GEOCODE_PLACE_TYPE_BUILDING: A building or house. yikes, i screwedup the rebase. Will correct.
Review of attachment 250161 [details] [review]: ::: geocode-glib/test-gcglib.c @@ +376,3 @@ g_object_unref (object); + /* Nominatim lists our desired town at index 5 */ You can see the list of all towns here: http://nominatim.openstreetmap.org/search?q=bonneville&limit=10&addressdetails=1&format=xml&email=zeeshanak%40gnome%2Eorg As you can see there is multiple Bonneville in France so we either search for "bonneville, rhone-alpes" or we could loop through the results to check if our desired result is included, just like we do in case of "paris" in test_search(). @@ +468,3 @@ g_assert_cmpint (g_list_length (list), ==, 10); + place = list->next->data; /* First result is the state, not the town */ If we use a structured query, yes. Nominatim's docs[1] say that we must either use a free form search or a structured search. Shall I replace the nominatim-rio.json to use a structured query instead? [1] http://wiki.openstreetmap.org/wiki/Nominatim#Parameters
Created attachment 250265 [details] [review] lib: Fix a typo in a doc comment
Created attachment 250266 [details] [review] lib: Adapt GeocodePlace for Nominatim
(In reply to comment #32) > Review of attachment 250161 [details] [review]: > > ::: geocode-glib/test-gcglib.c > @@ +376,3 @@ > g_object_unref (object); > > + /* Nominatim lists our desired town at index 5 */ > > You can see the list of all towns here: > http://nominatim.openstreetmap.org/search?q=bonneville&limit=10&addressdetails=1&format=xml&email=zeeshanak%40gnome%2Eorg > > As you can see there is multiple Bonneville in France so we either search for > "bonneville, rhone-alpes" or we could loop through the results to check if our > desired result is included, just like we do in case of "paris" in > test_search(). Yes, looking for the city sounds better. > @@ +468,3 @@ > g_assert_cmpint (g_list_length (list), ==, 10); > > + place = list->next->data; /* First result is the state, not the town */ > > If we use a structured query, yes. Nominatim's docs[1] say that we must either > use a free form search or a structured search. Shall I replace the > nominatim-rio.json to use a structured query instead? A loop to find the city as in the Paris and Bonneville cases sounds good actually. Maybe something that we can include as a helper? Feel free to file a separate bug for that helper, and let's get this code in.
Review of attachment 250265 [details] [review]: Yep.
Review of attachment 250266 [details] [review]: Looks good.
Attachment 247682 [details] pushed as 370faa1 - lib: Port reverse geocoding API to Nominatim Attachment 250157 [details] pushed as 58e0639 - place: Add more properties Attachment 250158 [details] pushed as 705056d - lib: Make GeocodePlace:name writable Attachment 250160 [details] pushed as 9c63a63 - lib: Compare against the correct values Attachment 250161 [details] pushed as ac935f9 - lib: Port forward geocoding API to Nominatim Attachment 250265 [details] pushed as f0606a5 - lib: Fix a typo in a doc comment Attachment 250266 [details] pushed as 93f9e6b - lib: Adapt GeocodePlace for Nominatim
(In reply to comment #35) > (In reply to comment #32) > > @@ +468,3 @@ > > g_assert_cmpint (g_list_length (list), ==, 10); > > > > + place = list->next->data; /* First result is the state, not the town */ > > > > If we use a structured query, yes. Nominatim's docs[1] say that we must either > > use a free form search or a structured search. Shall I replace the > > nominatim-rio.json to use a structured query instead? > > A loop to find the city as in the Paris and Bonneville cases sounds good > actually. Done. > Maybe something that we can include as a helper? Not sure if thats going to be very helpful as I don't see it very different from g_list_find_custom().