maps with different projection ?

Smallpox vs. largepox, gen2 vs gen5, early war vs. peaceful alliances. Which is your favourite gaming style?
User avatar
meynaf
Hardened
Posts: 155
Joined: Sun Jan 21, 2018 10:27 am
Location: Lyon / France
Contact:

Re: maps with different projection ?

Post by meynaf »

Molo_Parko wrote: When constraining the colors, it doesn't matter how many incorrect shades of blue for example, are in an image wherein the key defines only one blue color as valid. The constraining process makes all the blue variations as near-matches the same, which is the point. You start with one "correct" blue (from the key), and many incorrect blues, then end-up with only one blue for both the key's blue and the pixels that were various shades of blue.
You seem to be supposing that shades created by the computer are close enough to "correct" values. They are not.

Molo_Parko wrote: To clarify, as long as the key color for "forest" is the same color as every pixel which was supposed to be "forest" then it is easy to convert to Freeciv.
Yes but this is precisely the problem : you'll end up with colors that weren't forest at all, but got wrongly converted to it.
Do not worry about conversion to freeciv ; i can do it from any color-reduced image in a matter of minutes.

Molo_Parko wrote: The computer tends to do a better job of correcting the variance than I could do manually (in graphics editing software with a "pencil" tool ), so I let the computer do it.
The trick is that it's sometimes just not possible.

Molo_Parko wrote: On the other hand, the entire problem of the variance from a single original color is caused by computers in the first place, possibly just from image compression formats (although I would have thought less variance would have been better for compression, not more variance.)
That depends what we call "variance". Lossy image compression works in frequency domain, not spatial. This unavoidably adds "noise".
Then again, very few people know better than choosing jpeg to store images that have few colors...

Molo_Parko wrote: The wiki SVG maps of Winkel Tripel Earth projections are excellent quality but don't include the land cover data. If you do find acceptable land cover raster images for each land area, it is really quite easy to "warp" the raster into the SVG outline. The map projections are -all- warps of views of Earth in the first place.

If none of the source images available are acceptably accurate, then the only other option for land cover data appears to be to install mapping software, download eleventeen GigaBytes or so of land cover data, and figure out how to extract a numeric land cover classification per pixel along with the pixel's coordinates (longitude/latitude would probably be easiest to map into a Freeciv x,y map format.) Then you'd have a simple data file with unambiguous numeric land cover classification by area coordinates which can be mathematically converted to tile locations for a Freeciv map. But... the time required to download, decompress, load, and export a few simple bytes of numeric data per pixel/cell from the enormous image-based data sets might exceed any amount that could be considered "sane". :D
As long as it's "possible", i don't think i'll care that much if it's "sane" or not :D
Gimme BMP or PBM/PNM (i won't dare asking for IFF-ILBM) images of any size, and if they contain all the relevant data and don't have all these "wrong" pixel shades, i can do anything ;)

Molo_Parko wrote: Then there is the other problem of color keys created by obvious lunatics, such as this key... Why create multiple categories represented by the exact same color, when they are not distinguishable by color ??? Is watermelon lettuce? This key screams "YES!" :roll:
And as this is jpg image, if you pluck pixels in the color areas you'll find several shades...
Molo_Parko
Hardened
Posts: 158
Joined: Fri Jul 02, 2021 4:00 pm

Re: maps with different projection ?

Post by Molo_Parko »

You seem to be supposing that shades created by the computer are close enough to "correct" values. They are not.
Depends on the individual image, some actually are - few colors in the key makes it quite obvious which pixels were supposed to match which color in the key. Not that a five color key map of Earth's land cover would be of much use for Freeciv maps, but a 9 color key would match pretty well, whereas a 50 color key might be a nightmare. Ultimately, if imperfect image files are the -only- source of data, then there is no choice but to use that data, right or wrong. The alternative is not using the generated image files and instead getting the data as numeric values -- which do exist within the land cover datasets -- it is what the mapping software uses when overlaying colors to indicate land cover classifications. The software converts a numeric land classification to a color pixel! Then someone exports to raster image files -instead- -of- text/numeric data files, and that's where the color-mixing issues begin.
And as this is jpg image, if you pluck pixels in the color areas you'll find several shades...
In that color key image, watermelon, lettuce, etc all have -exactly- the same RGB values (255 100 104) at the center of each box. No varying pixels in the key boxes except near the black border where blending occurred, as usual. It's the size of the space that preserves the original color in the center because there isn't any other color at the center of each box to cause blending.
User avatar
meynaf
Hardened
Posts: 155
Joined: Sun Jan 21, 2018 10:27 am
Location: Lyon / France
Contact:

Re: maps with different projection ?

Post by meynaf »

Molo_Parko wrote: Depends on the individual image, some actually are - few colors in the key makes it quite obvious which pixels were supposed to match which color in the key.
Some of them are close enough, of course. But i wasn't speaking about these.

Molo_Parko wrote: The alternative is not using the generated image files and instead getting the data as numeric values -- which do exist within the land cover datasets -- it is what the mapping software uses when overlaying colors to indicate land cover classifications. The software converts a numeric land classification to a color pixel! Then someone exports to raster image files -instead- -of- text/numeric data files, and that's where the color-mixing issues begin.
Where to find such datasets ? I only found images.
Then i can convert to images myself, and be sure that even with resizing there will be no ambiguities.
Molo_Parko
Hardened
Posts: 158
Joined: Fri Jul 02, 2021 4:00 pm

Re: maps with different projection ?

Post by Molo_Parko »

WOW! I was going to say that the data sets are at many different websites and give you a search link when I ran across a very nice one (well, nice from my point-of-view for easier and more reliable data conversion.) Text file of land values at 1 degree x 1 degree resolution -and- a scalable Postscript vector image map! There doesn't appear to be much documentation there though but the numeric values are listed for the different classifications used in the text file. The cell/area classification values in the text file appear to be "upside down", as in, the start of the file is the south pole - all ice. The end of the file is the north pole area. https://data.giss.nasa.gov/landuse/vegeem.html

Unfortunately, it appears that each different data set (or possibly series of sets) uses different numbers for area classification so whichever set you use, you'd have to have documentation explaining what the value means (to indicate 1=forest, 2=desert and so on, as an example.) Some data sets have a small list of possible classification values. Some have -enormous- lists with many, many different classifications. Some have 1Km per pixel resolution, some are far more coarse.

https://www.google.com/search?q=%22glob ... a+download

This site has a collection of links, some of which actually still work: https://freegisdata.rtwilson.com
Scroll down to "land cover".
Last edited by Molo_Parko on Thu Oct 07, 2021 11:02 pm, edited 2 times in total.
Molo_Parko
Hardened
Posts: 158
Joined: Fri Jul 02, 2021 4:00 pm

Re: maps with different projection ?

Post by Molo_Parko »

That was very easy to convert from the text file. Apparently it is a data set from the early 1980s.

Code: Select all

tr "\n" " " < ~/Downloads/"VEG1X1.MATTHEWS copy for sed.txt" | sed -e 's/ 0 / f /g' -e 's/ 1 / j /g' -e 's/ 2 / f /g' -e 's/ 3 / f /g' -e 's/ 4 / f /g' -e 's/ 5 / f /g' -e 's/ 6 / f /g' -e 's/ 7 / f /g' -e 's/ 8 / f /g' -e 's/ 9 / f /g' -e 's/ 10 / f /g' -e 's/ 11 / f /g' -e 's/ 12 / f /g' -e 's/ 13 / f /g' -e 's/ 14 / f /g' -e 's/ 15 / f /g' -e 's/ 16 / f /g' -e 's/ 17 / j /g' -e 's/ 18 / j /g' -e 's/ 19 / j /g' -e 's/ 20 / j /g' -e 's/ 21 / j /g' -e 's/ 22 / t /g' -e 's/ 23 / g /g' -e 's/ 24 / g /g' -e 's/ 25 / g /g' -e 's/ 26 / p /g' -e 's/ 27 / p /g' -e 's/ 28 / p /g' -e 's/ 29 / g /g' -e 's/ 30 / d /g' -e 's/ 31 / a /g' -e 's/ 32 / g /g' -e 's/ -9999 / : /g' | tr -d " " | tr ":" " " | fold -w 360 | tail -r > ~/Downloads/OutputMap.txt
^ I wasn't too sure about which # to replace with which tile so I used individual sed substitution commands for easy changing. Output map is attached. It's 360x180 tiles but no elevation data added-in so no hills or mountains at all.

Image
^ Just South America in the pic because the whole map is too big.

I found another data set "Globcover2009 v2.3", 2009 data, released 2010. Totally different format from the other two sets I've seen (which were different from one another.) --- Edit: Nevermind, despite the preview jpg image being color, and the color key referencing RGB values, the massive tiff file is grayscale.
Attachments
OutputMap.txt.zip
(5.12 KiB) Downloaded 154 times
User avatar
meynaf
Hardened
Posts: 155
Joined: Sun Jan 21, 2018 10:27 am
Location: Lyon / France
Contact:

Re: maps with different projection ?

Post by meynaf »

Ok, let's have a look at what we have here.

The text file looks closer to home. Things i've seen :
- fciv has two ocean types ; here not differenciated (can be provided later if relevant data is found)
- there are forests where it shouldn't : UK, France, Spain, are 100% covered with forests
- no hills or mountains, as expected, but also no swamps

In addition, 360x180 is "too small", gives ~88.5 km/cell.
A map of 500x250 can be found on freeciv.org in some saved game, gives ~64 km/cell. I'd like to have something like 50 km/cell.
Even if a map with "correct" cell size is found, it won't be usable because redoing the projection will need a higher resolution map (to avoid cells that duplicate themselves).
Something like a 4K map, perhaps even bigger.

Currently following the links. Why they do these huge tiff files 99% of software can't handle is beyond me.
User avatar
meynaf
Hardened
Posts: 155
Joined: Sun Jan 21, 2018 10:27 am
Location: Lyon / France
Contact:

Re: maps with different projection ?

Post by meynaf »

I've been able to find ocean depth map - i think - here :
https://visibleearth.nasa.gov/images/73 ... try/73965l
The 21600x10800 png is big but it's 8 bit per pixel (grayscale) so pixel merging can happen as much as it wants, no big deal.
For once i've found something that's usable ! I might experiment with different thresholds to see how deep the 'deep ocean' has to be.
Of course, it's still only for 1 terrain type out of 13...
And i still need to redo the projection.
Molo_Parko
Hardened
Posts: 158
Joined: Fri Jul 02, 2021 4:00 pm

Re: maps with different projection ?

Post by Molo_Parko »

In addition, 360x180 is "too small", gives ~88.5 km/cell.
^ That land cover data set was from 1983 apparently. The included vector graphic postscript file can be resized / enlarged without losing image quality, and then exported to a raster format for conversion to Freeciv. The tiff file in some newer datasets is basically the same as the text file in that older data set except that the data is stored as hex values in a tiff file instead of decimal values in a text file.
there are forests where it shouldn't
I quickly picked tiles to use for each pixel value without very much consideration given to accuracy. :) I was just testing the conversion.


In the GlobCover2009 data set which I downloaded last evening, the tiff is used to store grayscale values which indicate the land cover classification of each pixel. I think that they are using tiff image files for that because they are lossless so data is preserved (until someone exports or converts the format, mistakenly thinking they are sharing the same data), and storing the data in tiff also provides a visual / picture of the data stored in the file. They could have put the same bytes of data into a .bin file instead which would be a slightly smaller file size, but would not have the feature of being an image file. So the tiff format actually makes sense to me now that I see how they are using it. The tiff file is grayscale only, which is fine since that dataset has only 33 land cover classifications anyway, and a single byte can represent up to 256 different classes. Each data byte represents one pixel of the image. The color key is an external file which lists grayscale density values and the RGB color values which should be used when viewing the image in color. To extract the data for conversion to Freeciv, each data byte from the grayscale file can be swapped for a Freeciv tile letter without converting the tiff data to color.

http://due.esrin.esa.int/files/Globcove ... lobal_.zip
^ The Globe Cover 2009 data set download URL. The file is 381 MB zipped.

Code: Select all

ENVI Density Slice Range File
       11.000000       11.000000     170     240     240
       14.000000       14.000000     255     255     100
       20.000000       20.000000     220     240     100
       30.000000       30.000000     205     205     102
       40.000000       40.000000       0     100       0
       50.000000       50.000000       0     160       0
       60.000000       60.000000     170     200       0
       70.000000       70.000000       0      60       0
       90.000000       90.000000      40     100       0
       100.00000       100.00000     120     130       0
       110.00000       110.00000     140     160       0
       120.00000       120.00000     190     150       0
       130.00000       130.00000     150     100       0
       140.00000       140.00000     255     180      50
       150.00000       150.00000     255     235     175
       160.00000       160.00000       0     120      90
       170.00000       170.00000       0     150     120
       180.00000       180.00000       0     220     130
       190.00000       190.00000     195      20       0
       200.00000       200.00000     255     245     215
       210.00000       210.00000       0      70     200
       220.00000       220.00000     255     255     255
       230.00000       230.00000       0       0       0
^ "GlobCover_Legend.dsr" is a text list of which grayscale values represent which RGB color values, but does not explain the land cover classification description for each value.

Image ^ The "Globcover2009_Legend.xls" file explains which values are which land cover classifications. Grayscale density byte values represent one pixel and the amount of gray used to display that pixel. So a byte with (decimal) value 11 is land classification #11 which is "post-flooding or irrigated croplands". Likely match for a Freeciv grassland tile "g".

Only the full tiff file, and the Excel spreadsheet file's data are needed to convert the image to Freeciv, or to a color image.

The conversion process would be to read each pixel/data byte (not file formatting bytes, just data bytes) from the tiff, then use the byte's value to determine which Freeciv tile would match. For instance, a data byte with value 70 (density of gray) is classified as " > 40% percent evergreen forest" and the matching tile for Freeciv would be "f" for forest.

Some of the land classifications used seem a bit ambiguous, > 15% forest means the tile is up to 84% not forest but what is it then???

The "land cover" datasets don't have information about the ocean, other than that it exists. :) The data does include ocean pixels, and lakes but without depth. To represent elevation and ocean depth in the Freeciv map would require additional maps (topographical for land, bathymetric for ocean) for that data and potentially also an additional map for river data. Then all we have to do is figure out how best to represent both forest and hill, for instance, for the same area of tiles in a Freeciv map.

For testing conversion from tiff to Freeciv, attached is a much smaller image of the map using the same land cover classifications as pixel byte values (I reduced the image to 100x43 pixels, which would become a 100x43 tile Freeciv map.)

EDIT 1: Removed compressed image, the attached image is uncompressed (no LZW compression of the data.)

EDIT 2: since the classification systems don't differentiate between tundra and desert (both appear to be classed as <15% vegetation) it might be good to change that class to tundra when it occurs above the frost line.
Attachments
GLOBCOVER_L4_200901_200912_V2.3_100width_uncompressed.tif
GLOBCOVER_L4_200901_200912_V2.3_100width_uncompressed.tif (23.29 KiB) Viewed 3078 times
Last edited by Molo_Parko on Sat Oct 09, 2021 12:00 am, edited 1 time in total.
User avatar
meynaf
Hardened
Posts: 155
Joined: Sun Jan 21, 2018 10:27 am
Location: Lyon / France
Contact:

Re: maps with different projection ?

Post by meynaf »

Molo_Parko wrote: In the GlobCover2009 data set which I downloaded last evening, the tiff is used to store grayscale values which indicate the land cover classification of each pixel.
Tiff files inside are too big to be of any use to me. Can't view them, but i could hex-edit if not compressed.

Molo_Parko wrote: I think that they are using tiff image files for that because they are lossless so data is preserved (until someone exports or converts the format, mistakenly thinking they are sharing the same data), and storing the data in tiff also provides a visual / picture of the data stored in the file. They could have put the same bytes of data into a .bin file instead which would be a slightly smaller file size, but would not have the feature of being an image file. So the tiff format actually makes sense to me now that I see how they are using it.
Image formats have size limits, 32767 or 65535 width/height.
Tiff probably can grow bigger than that.
It's not always lossless, tiff format being more a container than a format by itself.
Due to lzw (or anything else) compression, a .bin would actually be bigger.

Molo_Parko wrote: The tiff file is grayscale only, which is fine since that dataset has only 33 land cover classifications anyway, and a single byte can represent up to 256 different classes. Each data byte represents one pixel of the image. The color key is an external file which lists grayscale density values and the RGB color values which should be used when viewing the image in color. To extract the data for conversion to Freeciv, each data byte from the grayscale file can be swapped for a Freeciv tile letter without converting the tiff data to color.
First i need to decompress the tiff, which i can't do.

Molo_Parko wrote: http://due.esrin.esa.int/files/Globcove ... lobal_.zip
^ The Globe Cover 2009 data set download URL. The file is 381 MB zipped.
I had these already. Can't do a thing with them, as mentioned above.

Molo_Parko wrote: Only the full tiff file, and the Excel spreadsheet file's data are needed to convert the image to Freeciv, or to a color image.

The conversion process would be to read each pixel/data byte (not file formatting bytes, just data bytes) from the tiff, then use the byte's value to determine which Freeciv tile would match. For instance, a data byte with value 70 (density of gray) is classified as " > 40% percent evergreen forest" and the matching tile for Freeciv would be "f" for forest.

Some of the land classifications used seem a bit ambiguous, > 15% forest means the tile is up to 84% not forest but what is it then???

The "land cover" datasets don't have information about the ocean, other than that it exists. :) The data does include ocean pixels, and lakes but without depth. To represent elevation and ocean depth in the Freeciv map would require additional maps (topographical for land, bathymetric for ocean) for that data and potentially also an additional map for river data. Then all we have to do is figure out how best to represent both forest and hill, for instance, for the same area of tiles in a Freeciv map.
Are you able to extract binary data out of the image ?

Molo_Parko wrote: For testing conversion from tiff to Freeciv, attached is a much smaller image of the map using the same land cover classifications as pixel byte values (I reduced the image to 100x43 pixels, which would become a 100x43 tile Freeciv map.)
Best size for me is 21600x10800 (same as the topographical and bathymetric files i've found). Ideally, in 8-bit PNG format (raw binary can be fine too).
If all maps have same size, use same projection and are centered the same, i can start doing experiments.
Molo_Parko
Hardened
Posts: 158
Joined: Fri Jul 02, 2021 4:00 pm

Re: maps with different projection ?

Post by Molo_Parko »

The image attached to my last post is a reduced-size uncompressed tiff of the original 379 MB file. I resized it to 100 pixels width. I checked the byte values and they are correct - no change at all to the values per pixel, just fewer pixels overall. If you want the values as decimal, the easiest way would be to convert the tiff image to a PGM (portable gray map) file using ASCII/decimal values instead of hex values. PGM is a very simple file format. Graphic Converter can convert to PGM via "Save as...". Select PGM, in "Options" choose ASCII for decimal values.

Attached is a zipped PGM version of the 100 pixel wide test file. All pixel values are still the same as the original massive tiff file image, it's just fewer pixels for testing. Open the PGM file in any plain text editor and you'll see the decimal values which are the land cover classification values as each pixel. The header information is file type (p2=PGM), line width of pixels (100), total lines (43), 255=greatest possible value (8 bit per pixel), then the rest of the file is lines of pixel values separated by spaces. Find and replace 210 to a space for Freeciv ocean tile, as an example.

Image

The line of decimal 210's would be a line of d2's as hex, so in the uncompressed tif 100 pixel wide sample image I posted in my prior post, you could search for hex d2 d2 d2 d2 and that will be the start of the pixel data in that uncompressed tiff file. Each line of pixels is 100 bytes wide.


"GLOBCOVER_L4_200901_200912_V2.3_21600w.pbm.7z" is pbm (portable bit map) with binary/hex values, pbm file compressed externally with 7z. Uncompressed size is 200 MB, compressed it's 12.8 MB. :D
https://www9.zippyshare.com/v/m27MNagq/file.html
https://www.sendspace.com/file/i1k33e

^ The free file host sites have a ton of ads, pop-ups and crap. Sendspace download button is white text on blue. ZippyShare is white text on orange/red but usually it gets a redirect on first click to an ad site, try again and it downloads.
Attachments
pic.pgm.zip
(2.14 KiB) Downloaded 145 times
Last edited by Molo_Parko on Fri Oct 08, 2021 7:30 pm, edited 6 times in total.
Post Reply