After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 770321 - Forecast not available all day, HTTP 404 for noaa.gov
Forecast not available all day, HTTP 404 for noaa.gov
Status: RESOLVED FIXED
Product: libgweather
Classification: Core
Component: general
3.20.x
Other Linux
: Normal major
: future
Assigned To: GNOME Weather Maintainer(s)
GNOME Weather Maintainer(s)
: 772262 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2016-08-24 09:29 UTC by Daniel Aleksandersen
Modified: 2016-10-07 16:27 UTC
See Also:
GNOME target: 3.22
GNOME version: 3.19/3.20


Attachments
Ugly monkey patch to fix METAR retrieval (1.58 KB, patch)
2016-08-24 20:32 UTC, Frank Dana
none Details | Review
Updated METARS patch with HTTPS requests (1.59 KB, patch)
2016-08-25 02:19 UTC, Frank Dana
committed Details | Review
libmateweather patch adapted (979 bytes, patch)
2016-08-25 12:43 UTC, Maik Freudenberg
none Details | Review

Description Daniel Aleksandersen 2016-08-24 09:29:17 UTC
(org.gnome.Weather.Application:15735): GWeather-WARNING **: Failed to get METAR data: 404 Not Found.

GNOME 3.20.1 as distributed in Fedora 24. My logs first mention this yesterday at 00:08 UTC and it has been happening every time I open the app since. I’m guessing that a service change went live at midnight yesterday.
Comment 1 Daniel Aleksandersen 2016-08-24 09:31:55 UTC
http://weather.noaa.gov/mgetmetar.php returns “This Service is no longer available”
Comment 2 Daniel Aleksandersen 2016-08-24 09:38:58 UTC
There doesn’t appear to be a replacement from NOAA.
Comment 3 Jeremy Bicha 2016-08-24 16:44:14 UTC
According to https://web.archive.org/web/20160130172236/http://weather.noaa.gov:80/weather/coded.html

"METAR Data Access

On or about April 4th, 2012, request for METARS through this service will be redirected to:
http://www.aviationweather.gov/adds/metars/

This redirection is part of a larger effort to reduce duplication of effort within the National Weather Service. The website above offers the same functionality. It also needs to be noted that the NWS Aviation Weather Center operated website above is the only FAA approved source of aviation weather over the Internet to be used for pilot briefings. Please update bookmarks, links, etc.."
Comment 4 Giovanni Campagna 2016-08-24 18:10:10 UTC
Ah argh I did not know about that.

Anyone of you want to take a stab at a patch using the new service?
Otherwise I'll try to work on it on the weekend.
Comment 5 Frank Dana 2016-08-24 20:32:49 UTC
Created attachment 334100 [details] [review]
Ugly monkey patch to fix METAR retrieval

The attached extremely ugly patch against the libgweather git head restores METAR functionality using the replacement www.aviationweather.org text data server. 

(Which outputs well-formed XML that the code should probably be parsing, instead of blindly flailing around in for the raw-text METAR string, but... meh.)
Comment 6 Vlad Orlov 2016-08-24 20:41:13 UTC
Hi all,

You can also use my patch for libmateweather (a fork of libgweather for MATE desktop):

https://github.com/mate-desktop/libmateweather/commit/835055e0d0b950d3f66ad236de7e40942f6f7d8e

Well, it's not perfect... but this new URL turned out to be a nice drop-in replacement, so the code change is quite small.
Comment 7 Daniel Aleksandersen 2016-08-24 21:24:41 UTC
By the by, why does this problem with a US only weather service cause issues for everyone else in the world that use data from other weather services such as MET Norway? From what I can tell, the METAR data from NOAA.org is just discarded and never used.
Comment 8 Daniel Aleksandersen 2016-08-24 21:30:07 UTC
Frank Dana and Vlad Orlov, please update your patches and switch the endpoints to HTTPS instead of HTTP. (Just change the supplied URLs from one to the other.) This will increase user privacy and help prevent any third parties from learning users’ device location from their use of libgweather/libmateweather.
Comment 9 Frank Dana 2016-08-24 21:48:10 UTC
The list of things that could be improved about my patch would probably exceed bugzilla's max comment length; if libgweather wants to transition to making HTTPS data requests then I'm all for it, but I think that's a separate change outside the scope of this bug report. I'd suggest opening a separate bug, if there isn't already one in the system.
Comment 10 Daniel Aleksandersen 2016-08-24 22:06:41 UTC
Frank Dana, the only reason this hasn’t been changed to HTTPS already is that the previous API endpoint didn’t support it. Some other libgweather APIs now use HTTPS. I can open a follow-up patch later, but would be good to just get the change in sooner rather than later.
Comment 11 Michael Catanzaro 2016-08-25 01:55:02 UTC
Review of attachment 334100 [details] [review]:

It really needs to use HTTPS.
Comment 12 Frank Dana 2016-08-25 02:19:07 UTC
Created attachment 334112 [details] [review]
Updated METARS patch with HTTPS requests

Alrighty then. Here you go.
Comment 13 Maik Freudenberg 2016-08-25 12:43:20 UTC
Created attachment 334136 [details] [review]
libmateweather patch adapted

Both patches work fine but the one from Vlad / libmateweather looks a lot more trivial.
Comment 14 Frank Dana 2016-08-25 12:51:09 UTC
More trivial, at the expense of scraping the METAR string from an active webpage, instead of a data service.

By my testing, requesting http://aviationweather.gov/metar/data?ids=KLGA (for my local weather station) results in a 1.54 MEGABYTE response.

While guidance on how to use aviationweather.gov's data programmatically is unfortunately spare (I've looked), that's clearly against their Best Practices (http://www.aviationweather.gov/dataserver/bestpractices), and even assuming their servers are up to the task, seems a wasteful consumption of the enduser's bandwidth.
Comment 15 Frank Dana 2016-08-25 12:57:41 UTC
My mistake, I'm an idiot. (I retrieved the page at 1.54MB/s, and clearly have wget output reading-comprehension issues this early in the morning.)

However, the difference between the patches is still a 618 byte response (mine) and an 18.48 KByte response (libmateweather). That's a rather significant cost.
Comment 16 Vlad Orlov 2016-08-25 13:06:48 UTC
I know it's not perfect, as I said before. That way of getting the data was in the library before (and was there for a long time it seems, even since Gnome 2), so for a quick fix I did a simple drop-in replacement patch.

BTW, there's one more option - to parse plain text files from here:
http://tgftp.nws.noaa.gov/data/observations/metar/stations/

However, it looks like https isn't supported there, so I'm not sure about this.
Comment 17 Maik Freudenberg 2016-08-25 13:08:44 UTC
Comment on attachment 334136 [details] [review]
libmateweather patch adapted

Marking obsolete due to high traffic count.
Comment 18 Maik Freudenberg 2016-08-25 13:17:20 UTC
I think Frank's right, we should not strain the service more than really needed. I've also been browsing for documentation of aviationweather.gov services and found next to nothing. Also been trying to use the noaa.gov site Vlad mentioned. But there are at least two reasons against it: 1. it doesn't provide https 2. according to NWS the site aviationweather.gov is the only authoritative site for METAR data. So we should stick to that.
Comment 19 Vlad Orlov 2016-08-25 15:12:32 UTC
Ok. I think I'll use Frank's patch in libmateweather as well.
Comment 20 Colomban Wendling 2016-08-26 14:42:45 UTC
I can confirm that Frank's patch seems to work fine.  I applied it on Debian Unstable's libmateweather 1.14.0 as well as Debian Oldstable's libgweather 3.4.1 (where a small change in the patch is required, the old location being different), and both now work again and seem to give sensible results.

Thanks!
Comment 21 Colomban Wendling 2016-08-26 18:18:29 UTC
Patch doesn't apply directly on libgweather 3.20.2 (or git HEAD) because some surrounding code slightly changed, but it otherwise works fine there too.
Comment 22 Christian Stadelmann 2016-08-28 15:57:34 UTC
How about fixing this in a more resilient (change-tolerant) way like gnome-maps did with their maps data provider?
Comment 23 Michael Catanzaro 2016-08-28 16:04:56 UTC
(In reply to Giovanni Campagna from comment #4)
> Ah argh I did not know about that.
> 
> Anyone of you want to take a stab at a patch using the new service?
> Otherwise I'll try to work on it on the weekend.

Giovanni, do you have time to get releases out today or tomorrow? Otherwise, I would prefer to handle releases due to the urgency of this issue.
Comment 24 Giovanni Campagna 2016-08-29 00:03:29 UTC
Apologies for not getting to this earlier.
The patch currently posted looks good, and I tested it with gnome-weather it appears to work (ie, I get reasonable current weather conditions around the world).

I am about to issue a new release of libgweather 3.20 and 3.18 with the patch applied. I do not plan to backport it further, but it should not be hard to do it for distros because the code did not change significantly in that area recently.

Thanks for the patch in any case.
Comment 25 Giovanni Campagna 2016-09-30 16:50:15 UTC
*** Bug 772262 has been marked as a duplicate of this bug. ***
Comment 26 Ron_ 2016-10-07 13:11:50 UTC
I have an ancient copy of Gnome Clock 2.28.0 that stopped displaying the weather at the same time as the current version of GNOME Weather.  It seems not to use libgweather.  Can anyone advise me on a way to patch this one?
Comment 27 Christian Stadelmann 2016-10-07 13:16:40 UTC
(In reply to Ron_ from comment #26)
> I have an ancient copy of Gnome Clock 2.28.0 that stopped displaying the
> weather at the same time as the current version of GNOME Weather.  It seems
> not to use libgweather.  Can anyone advise me on a way to patch this one?

Update to a more recent version. Gnome clock 2.28 is like 5 to 6 years old and probably hasn't seen an update in 5 years.
Comment 28 André Klapper 2016-10-07 16:27:29 UTC
(In reply to Ron_ from comment #26)
> Can anyone advise me on a way to patch this one?

Please contact your Linux distribution for support questions. Thanks!