GNOME Bugzilla – Bug 771661
Extend the raw data import plugin to open .hgt files
Last modified: 2017-12-09 22:05:28 UTC
HGT file are elevation data from the NASA. A description of the format is here: http://dds.cr.usgs.gov/srtm/version1/Documentation/SRTM_Topo.txt From SRTM_Topo.txt: ------------------------------- 3.1 DEM File (.HGT) The DEM is provided as 16-bit signed integer data in a simple binary raster. There are no header or trailer bytes embedded in the file. The data are stored in row major order (all the data for row 1, followed by all the data for row 2, etc.). All elevations are in meters referenced to the WGS84 EGM96 geoid as documented at http://www.nima.mil/GandG/wgsegm/. Byte order is Motorola ("big-endian") standard with the most significant byte first. Since they are signed integers elevations can range from -32767 to 32767 meters, encompassing the range of elevation to be found on the Earth. In these preliminary data there commonly will be data voids from a number of causes such as shadowing, phase unwrapping anomalies, or other radar-specific causes. Voids are flagged with the value -32768. ------------------------------- It would the nice if the file-raw-data.c could be extended to open this files: https://git.gnome.org/browse/gimp/tree/plug-ins/common/file-raw-data.c Example files from the NASA can be downloaded here: http://dds.cr.usgs.gov/srtm/version1/Islands/
NASA created different versions of the data. This are the links to the latest ones: http://dds.cr.usgs.gov/srtm/version2_1/Documentation/SRTM_Topo.pdf http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/Islands/
Created attachment 365289 [details] [review] import hgt files Attaching a patch that extends the raw data plug-in to import single channel 16 bit, signed/unsigned, big/little endian, 2d raster data as a 16 bit RGB gamma image. I left in the patch (2nd and 3rd hunks) the lines I used to test loading .hgt files, that register the .hgt extension and prefill the dialog fields with common values. Tested using the gradient map plug-in to remap different elevations to different colors on files downloaded from the site linked above.
Hi! Thanks for this. I have been trying to understand a little bit more what we are supposed to get when reading these files. At first I didn't understand why your code seemed to load grayscale data yet would open as GIMP_RGB. But looking at the pdf, I can see fancy colored image which indeed at elevation landscape when you look at a map with elevation info. So do we have to remap it necessarily afterwards with the gradient map plug-in? Or is it what the palette files can be used for? In any case I don't have at all these kind of outputs after loading, though I managed to have not too shabby outputs when changing the image type for some of the hgt from NASA. But that is weird since the description says that byte order for HGT is meant to be big endian. Yet when I was getting something which looked like elevation maps, that was when setting to one of the little-endian types (the default "Gray 16 Big Endian" would output a mostly single-gray (or close enough that I don't see the scales) image with a tiny bit of noise here and there, in any case it doesn't look like an elevation map). So how are we supposed to open such files exactly? By just testing randomly various image types until we find one which looks ok? Also what about the 1201 width/height? In the PDF, I indeed read: > SRTM3 data are sampled at three arc-seconds and contain 1201 lines and 1201 samples with similar overlapping rows and columns. This backs up the default choice of 1201 you did. But then should we leave these inputs editable with HGT files? Any other value would output distorted images. Or is it actually possible to have other values?
Since we have a patch, let's set this as an hopeful blocker for 2.10. :-)
Created attachment 365291 [details] S39E178.hgt imported as Gray 16-bit Big Endian What bothers me the most is that the PDF from NASA really says that these files are supposed to be Big Endian signed 16 bits. So your default looks good. Yet attached is what I get when I import S39E178.hgt (https://dds.cr.usgs.gov/srtm/version1/Islands/S39E178.hgt.zip) with the default Big Endian.
Created attachment 365292 [details] S39E178.hgt imported as Gray 16-bit Little Endian Now importing the same file with Gray 16-bit Little Endian.
(and importing files from the versions2_1 link doesn't seem to be any different)
(In reply to Jehan from comment #3) > Hi! > > Thanks for this. I have been trying to understand a little bit more what we > are supposed to get when reading these files. > > At first I didn't understand why your code seemed to load grayscale data yet > would open as GIMP_RGB. But looking at the pdf, I can see fancy colored > image which indeed at elevation landscape when you look at a map with > elevation info. > So do we have to remap it necessarily afterwards with the gradient map > plug-in? Or is it what the palette files can be used for? Well a palette file could be used but since there are more than 256 levels I preferred to create a 16 bit image that can be colored with the gradient map plug-in and a gradient. If one wants to stitch/crop many tiles to have an exact area, it is important to preserve the original data. I used this silly gradient: >GIMP Gradient >Name: AAA >6 >0.000000 0.000015 0.000030 1.000000 0.000000 1.000000 1.000000 1.000000 0.000000 1.000000 1.000000 0 0 0 0 >0.000000 0.250000 0.500000 0.000000 0.000000 0.100000 1.000000 0.000000 0.000000 1.000000 1.000000 0 0 0 0 >0.500000 0.507633 0.515267 0.000000 1.000000 0.000000 1.000000 0.000000 0.500000 0.000000 1.000000 0 0 0 0 >0.515267 0.522896 0.530526 0.500000 0.500000 0.000000 1.000000 1.000000 1.000000 0.250000 1.000000 0 0 0 0 >0.530526 0.545785 0.561044 0.750000 0.750000 0.750000 1.000000 1.000000 1.000000 1.000000 1.000000 0 0 0 0 >0.561044 0.772829 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 0 0 0 0 magenta for invalid data, blue below zero, green to 1000 m, yellow to 2000m, and gray to white above. The area around here shows the mountains around the lake quite correctly. > > In any case I don't have at all these kind of outputs after loading, though > I managed to have not too shabby outputs when changing the image type for > some of the hgt from NASA. But that is weird since the description says that > byte order for HGT is meant to be big endian. Yet when I was getting > something which looked like elevation maps, that was when setting to one of > the little-endian types (the default "Gray 16 Big Endian" would output a > mostly single-gray (or close enough that I don't see the scales) image with > a tiny bit of noise here and there, in any case it doesn't look like an > elevation map). > > So how are we supposed to open such files exactly? By just testing randomly > various image types until we find one which looks ok? Well, typically the user knows the resolution/endianness of the data. > > Also what about the 1201 width/height? In the PDF, I indeed read: > > > SRTM3 data are sampled at three arc-seconds and contain 1201 lines and 1201 samples with similar overlapping rows and columns. > > This backs up the default choice of 1201 you did. But then should we leave > these inputs editable with HGT files? Any other value would output distorted > images. Or is it actually possible to have other values? In the dialog they can be adjusted at will and the preview should show whether or not the width/height is right, I left them to ease the testing, don't know if this is such a common case to deserve being special cased
Created attachment 365300 [details] HGT imported then gradient-mapped. Ok so that was indeed Big Endian. It is just definitely not made to be seen without colors, and even though it looked like nearly nearly uniform gray, it is simply that elevation on earth, compared to a [-32767, 32767] meter range is spread tightly. What I did is search the file N34W119.hgt shown as an example in the PDF. https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N34W119.hgt.zip Then I imported it, and maped to a gradient inspired by yours and slightly modified so that the output resembles NASA one's (well more or less. Ours is a bit "radioactive" :P): GIMP Gradient Name: N34W119 HGT 6 0.000000 0.000015 0.000030 1.000000 0.000000 1.000000 1.000000 1.000000 0.000000 1.000000 1.000000 0 0 0 0 0.000000 0.438486 0.500000 0.000000 0.000000 0.100000 1.000000 0.000000 0.000000 1.000000 1.000000 0 0 0 0 0.500000 0.501303 0.503087 0.000000 1.000000 0.000000 1.000000 0.000000 0.500000 0.000000 1.000000 0 0 0 0 0.503087 0.505891 0.508696 0.500000 0.500000 0.000000 1.000000 1.000000 1.000000 0.250000 1.000000 0 0 0 0 0.508696 0.517011 0.529345 0.628079 0.628079 0.628079 1.000000 1.000000 1.000000 1.000000 1.000000 0 0 0 0 0.529345 0.750290 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 0 0 0 0
Anyway nice patch, as usual! :-) I'll still have a look to maybe customize a bit the dialog for HGT case. The type in particular is apparently always Big Endian signed. As for width and height, it seems there are 2 possible cases: SRTM-3 which are indeed 1201x1201, but it can also be SRTM-1 which is 3601x3601. We can get examples here: https://dds.cr.usgs.gov/srtm/version2_1/SRTM1/ It would be nicer to have a dropdown list proposing to choose the type of data SRTM1/SRTM3 which will most likely be much easier for the person to know than the width/height to apply. commit c572bdf6d66f127dfb0700ac58138c275a1fdbd2 Author: Massimo Valentini <mvalentini@src.gnome.org> Date: Sat Dec 9 13:56:28 2017 +0100 Bug 771661: Extend the raw data import plugin to open .hgt files add support to load single channel 16 bit, signed/unsigned, little/big endian 2d raster data as a 16 bit RGB gamma GIMP image plug-ins/common/file-raw-data.c | 139 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 134 insertions(+), 5 deletions(-) commit 88ccdb4e2fb93fc016860035965cc4ceb40984dc (HEAD -> master, origin/master, origin/HEAD) Author: Jehan <jehan@girinstud.io> Date: Sat Dec 9 18:51:31 2017 +0100 plug-ins: minor cleaning and fixes. - Add some parentheses to sizeof (*pointer). Maybe it's just me, but it is just a bit clearer to me. - Free in_raw when out_raw allocation failed. - Use strrchr() instead of strchr() to get the extension (I had a case when testing where the file was inside a folder named similarly with a ".hgt" and it would break the comparison test). - Do a case-insensitive comparison. Even though the original files from NASA use lowercase, just to be sure. plug-ins/common/file-raw-data.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-)
Ok! Now that's a better support where people won't have to guess too much or know the spec by heart to open such files. :-) commit b78b14b17e67f6fab1eeed273547b5ab9d5d8537 (HEAD -> master, origin/master, origin/HEAD) Author: Jehan <jehan@girinstud.io> Date: Sat Dec 9 22:51:38 2017 +0100 plug-ins: let's special-case just a bit the HGT data. - Display title "Digital Elevation Model data" instead of just "Image". - Don't show the type combo anymore, since HGT is always 16-bit signed integer in big endian. - Don't show the width and height either, but instead show a dropdown list to choose between either SRTM-1 or SRTM-3 data. There seems to be only these 2 types of data, and this will determine the width and height. This choice (SRTM-1 vs 3) will mostly likely be better known by people than knowing what image dimension they have to set, especially since official links for such files seem to all include the information. plug-ins/common/file-raw-data.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------ 1 file changed, 109 insertions(+), 62 deletions(-)