# Optimizing, publishing, and styling GeoTIFF data
### Styling raster data
- Select a dataset of the type `Raster`, go to its details page, and click on `Edit > Edit Style`
- By default, a code editor is shown on the left. You can choose to change to a visual editor by clicking on the button in the upper-right corner of the editor.
- Note the changes on the dataset as you are making the edits. Once your style is ready, click on `Apply` to save the changes
## Adding and Optimizing Raster Image Mosaics
An `Image Mosaic` comprises a set of `datasets` that are exposed as a `single coverage`. The `ImageMosaic` format allows for the **automatic** build and setup of a mosaic from a set of `georeferenced datasets`. This section provides better instructions on configuring an `Image Mosaic`.
### Publishing an Image Mosaic
In this section, we will publish a `mosaic` of aerial `orthophotos` on GeoNode through the GeoServer `ImageMosaic` store.
We will need to do some preparation first.
- Open a `terminal` and navigate to the folder `/opt/data/sample_data/user_data`. Change the `write` permissions of the folder named `aerial` to allow the user to modify it
```shell
cd /opt/data/sample_data/user_data
sudo chmod -Rf 777 aerial
```
- Navigate to `aerial`, and delete the files `aerial.properties` and `sample_image.dat`
```shell
cd aerial
rm aerial.properties sample_image.dat
```
- Create, or edit, a file named `indexer.properties` with the following contents
```shell
vim indexer.properties
```
```ini
Caching=false
Schema=*the_geom:Polygon,location:String
```
- Create, or edit, a file named `datastore.properties` with the following contents
```shell
vim datastore.properties
```
```ini
SPI=org.geotools.data.postgis.PostgisNGDataStoreFactory
host=localhost
port=5432
database=geonode_data
schema=public
user=geonode
passwd=geonode
Loose\ bbox=true
Estimated\ extends=false
validate\ connections=true
Connection\ timeout=10
preparedStatements=true
create\ database\ params=WITH\ TEMPLATE\=postgis20
```
- Go to the GeoServer admin dashboard, log in as an `admin` and click on `Data > Stores > Add new store`
- Select `ImageMosaic` from the list of the available `Raster Data Sources`
- Provide the following inputs on the `ImageMosaic` form
* **Name**: `Aerial`
* **Title**: `Aerial`
* **URL***: `file:/opt/data/sample_data/user_data/aerial`
- In the next windows, click on `Publish` and `Save` the dataset
- Move to the GeoServer `Layer Preview` section and open the `OpenLayers` preview for the `Aeriel` layer
- We will suddenly see two main issues:
1. The load, pan, and zoom are _slow_
2. There are some _black_ tiles filling the bounding box
### Optimizing an Image Mosaic
- Open a `terminal` and navigate to the folder `/opt/data/sample_data/user_data`. Change the `write` permissions of the folder named `optimized` to allow the user to modify it
```shell
cd /opt/data/sample_data/user_data
sudo chmod -Rf 777 optimized
```
- Navigate to `optimized`, and delete the files `optimized.properties` and `sample_image.dat`
```shell
cd optimized
rm optimized.properties sample_image.dat
```
- Create, or edit, a file named `indexer.properties` with the following contents
```shell
vim indexer.properties
```
```ini
Caching=false
Schema=*the_geom:Polygon,location:String
```
- Create, or edit, a file named `datastore.properties` with the following contents
```shell
vim datastore.properties
```
```ini
SPI=org.geotools.data.postgis.PostgisNGDataStoreFactory
host=localhost
port=5432
database=geonode_data
schema=public
user=geonode
passwd=geonode
Loose\ bbox=true
Estimated\ extends=false
validate\ connections=true
Connection\ timeout=10
preparedStatements=true
create\ database\ params=WITH\ TEMPLATE\=postgis20
```
- Add a new `ImageMosaic Data Source`, like we already done before, and provide the following parameters
- Provide the following inputs on the `ImageMosaic` form
* **Name**: `boulder_bg_optimized`
* **Title**: `boulder_bg_optimized`
* **URL***: `file:/opt/data/sample_data/user_data/optimized`
- In the next windows; click on `Publish`, **change the name and title to** `boulder_bg_optimized`, and `Save` the layer
- Open the GeoServer `Layer Preview` and zoom/pan around. Notice how much faster the mosaic is now...
_The black tiles are still present._
- Let's customize the mosaic options in order to improve the quality and remove the black tiles.
- Customize the `ImageMosaic` properties, set the `OutputTransparentColor` to the value `000000` (which is the `Black` color), and click on `Save` when done.
- Open the GeoServer `Layer Preview`, **clear the browser image cache**, and zoom/pan around. Notice how the black tiles are now gone
## Image Mosaic Options
There are plenty of options that can be tweaked to optimize an `ImageMosaic` further.
* **Accurate resolution computation**: If `true`, it computes the resolution of the granules in 9 points, the corners of the requested area, and the middle points and then takes the better one. This will provide better results for cases where there is a lot more deformation on a subregion (top/bottom/sides) of the requested box concerning the other boxes. Otherwise, compute the resolution using a basic affine scale transform.
* **AllowMultithreading**: If `true`, it enables multithreaded tile loading. This performs parallelized loading of the granules that compose the mosaic.
* **BackgroundValues**: Sets the value of the mosaic `background`. Depending on the nature of the mosaic it is wise to set a value for the `no data` area (usually `-9999`). This value is _repeated on all the mosaic bands_.
* **Bands**: a comma-separated list of band indexes exposes a particular subset of bands out of a multi-band source. It is normally not configured by hand but driven by `SLD` styling instead.
* **ExcessGranuleRemoval**: An option that should be enabled when using _scattered and deeply overlapping images_. By default, the image mosaic will try to mosaic together all images in the requested area (even if some are behind other images and won’t show up in the final image). With excess granule removal, the system will use the image footprint to determine which granules contribute pixels to the output. It will only perform the image processing on those images actually contributing to the mosaic. It is best used with footprints and sorting (to control which images stay on top). Possible values are `NONE` or `ROI` (_Region Of Interest_).
* **Filter**: Filters granules based on attributes from the input coverage.
* **Footprint Behavior**: Sets the behaviour of the regions of a granule that are _outside_ of the granule footprint. It can be `None` (_Ignore the footprint_), `Cut` (_Remove regions outside the footprint from the image. This does not add an alpha channel_), or `Transparent` (_Make regions outside the footprint completely transparent. This will add an alpha channel if one is not already present_). Defaults to `None`.
* **InputTransparentColor**: Sets the transparent colour for the granules prior to mosaicking them in order to control the superimposition process between them. When GeoServer composes the granules to satisfy the user request, some of them can overlap some others. Therefore, setting this parameter with the opportune colour avoids the overlap of no data areas between granules.
* **MaxAllowedTiles**: Sets the maximum number of tiles that can be loaded simultaneously for a request. In the case of a large mosaic, this parameter should be opportunely set to not saturate the server with too many granules being loaded simultaneously.
* **MergeBehavior**: Merges behaviour for the various granules of the mosaic that GeoServer will produce. This parameter controls whether we want to merge in a single mosaic or stack all the bands into the final mosaic.
* **OVERVIEW_POLICY**: Policy used to select the best matching overview for a given target output resolution. The possible values are `QUALITY` (pick an overview with a higher resolution and downsample), `NEAREST` (pick the one with the closest resolution), `SPEED` (pick the closest one with a lower resolution), and `IGNORE` (do not use overviews).
* **OutputTransparentColor**: Sets the transparent color for the created mosaic.
* **SORTING**: Specifies the time order of the obtained granules set. Valid values are `DESC` (descending) or `ASC` (ascending). Note that it works just by using `DBMS` as indexes.
* **SUGGESTED_TILE_SIZE**: Controls the tile size of the input granules and the tile size of the output mosaic. It consists of two positive integers separated by a comma, like `512,512`.
* **USE_JAI_IMAGEREAD**: If `true`, GeoServer will use the `JAI ImageRead` operation and its deferred loading mechanism to load granules. If `false`, GeoServer will perform direct `ImageIO` read calls, resulting in immediate loading.
## Introduction To Processing With GDAL Utilities
It’s a common practice to do a preliminary analysis of the available data and, if needed, to _optimize_ it. Since configuring big datasets without proper `pre-processing` may result in low performance when accessing them.
The previous examples show the difference between a non-optimized and an optimized mosaic, i.e. mosaics with pre-processed `GeoTIFF` granules and those without.
In this section, instructions about how to do data optimization will be provided by introducing some of the `GDAL Utilities`.
### gdalinfo
This utility allows a user to get information from the `GDAL` library, for instance, specific `Driver` capabilities and input `Layers/Files` properties.
#### gdalinfo - Getting Driver's Capabilities
`GeoTIFF` being a widely adopted geospatial format, it’s helpful to get information about the `GDAL GeoTIFF’s Driver` capabilities using the command
```shell
gdalinfo --format GTIFF
```
```shell
Format Details:
Short Name: GTiff
Long Name: GeoTIFF
Supports: Raster
Extensions: tif tiff
Mime Type: image/tiff
Help Topic: drivers/raster/gtiff.html
Supports: Sublayers
Supports: Open() - Open existing datasets.
Supports: Create() - Create writable dataset.
Supports: CreateCopy() - Create dataset by copying another.
Supports: Virtual IO - eg. /vsimem/
Creation Datatypes: Byte UInt16 Int16 UInt32 Int32 Float32 Float64 CInt16 CInt32 CFloat32 CFloat64
Other metadata items:
LIBGEOTIFF=1700
LIBTIFF=LIBTIFF, Version 4.1.0
Copyright (c) 1988-1996 Sam Leffler
Copyright (c) 1991-1996 Silicon Graphics, Inc.
```
From the list of create options above, it's possible to determine the main `GeoTIFF Driver`'s writing capabilities:
* `COMPRESS`: customize the compression to be used when writing output data
* `JPEG_QUALITY`: specify a quality factor to be used by the `JPEG` compression
* `TILED`: When set to `YES` it allows output data to be tiled
* `BLOCKXSIZE, BLOCKYZISE`: Specify the Tile dimension width and Tile dimension height
* `PHOTOMETRIC`: Specify the photometric interpretation of the data
* `PROFILE`: Specify the `GeoTIFF` profile to be used (some profiles only support a minimal set of `TIFF Tags` while some others provide a wider range of `Tags`)
* `BIGTIFF`: Specify when to write data as `BigTIFF` (A `TIFF` format that breaks the `4GB Offset` boundary)
#### gdalinfo - Getting Dataset/File Properties
The following instructions allow you to get information about the sample dataset that was previously configured on GeoServer
```shell
cd /opt/data/sample_data/user_data/aerial
```
```shell
gdalinfo 13tde815295_200803_0x6000m_cl.tif
```
_Part of the gdalinfo output on a sample dataset._
* **Block**: It represents the internal tiling. Notice that the sample dataset has tiles made of 16 rows having a width equal to the full image width.
* **Overviews**: It provides information about the underlying overviews. Notice that the sample dataset doesn’t have overviews since the overviews property is missing from the gdalinfo output.
### gdal_translate
This utility converts a dataset to a different format by allowing a wide set of parameters to customize the conversion.
Running the command
```shell
gdal_translate
```
gets the list of supported parameters as well as the supported output formats
```shell
Usage: gdal_translate [--help-general]
[-ot {Byte/Int16/UInt16/UInt32/Int32/Float32/Float64/
CInt16/CInt32/CFloat32/CFloat64}] [-strict]
[-of format] [-b band] [-mask band] [-expand {gray|rgb|rgba}]
[-outsize xsize[%] ysize[%]]
[- unscale] [-scale [src_min src_max [dst_min dst_max]]]
[-srcwin xoff yoff xsize ysize] [-projwin ulx uly lrx lry]
[-a_srs srs_def] [-a_ullr ulx uly lrx lry] [-a_nodata value]
[-gcp pixel line easting northing [elevation]]*
[-mo "META-TAG=VALUE"]* [-q] [-sds]
[-co "NAME=VALUE"]* [-stats]
src_layer dst_layer
```
The meaning of the main parameters is summarized below.
* `-ot`: specifies the output datatype (Make sure that the specified data type is contained in the Creation Datatypes list of the Writing driver)
* `-of`: specifies the desired output format (GTIFF is the default value)
* `-b`: specifies an input band to be written in the output file. (Use multiple -b options to specify more bands)
* `-mask`: specifies an input band to create an output dataset mask band.
* `-expand`: exposes a 1 band dataset with a colour table as a dataset with 3 (rgb) or 4 (rgba) bands. The 'gray' value enables to expand a dataset with a colour table containing only grey levels to a grey indexed dataset.
* `-outsize`: sets the size of the output file in terms of pixels and lines unless the % sign is attached, in which case, it’s as a fraction of the input image size.
* `-unscale`: applies the scale/offset metadata for the bands to convert from scaled values to unscaled values.
* `-scale`: rescales the input pixels values from the range src_min to src_max to the range dst_min to dst_max. (If omitted, the output range is 0 to 255. If omitted, the input range is automatically computed from the source data).
* `-srcwin`: selects a subwindow from the source image in terms of xoffset, yoffset, width, and height
* `-projwin`: selects a subwindow from the source image by specifying the corners given in georeferenced coordinates.
* `-a_srs`: overrides the projection for the output file. The srs_def may be any of the usual GDAL/OGR forms; complete WKT, PROJ.4, EPSG:n, or a file containing the WKT.
* `-a_ullr`: assigns/overrides the georeferenced bounds of the output file.
* `-a_nodata`: assigns a specified no data value to output bands.
* `-co`: sets a creation option in the form “NAME=VALUE” to the output format driver. (Multiple -co options may be listed.)
* `-stats`: gets statistics (min, max, mean, stdDev) for each band
* `src_layer`: is the source dataset name. It can be either file name, URL of data source, or sublayer name for multi*-layer files.
* `dst_layer`: is the destination file name.
#### gdal_translate - Tiling the sample dataset
The following steps provide instructions to tile the sample dataset previously configured in GeoServer using the GeoTiff driver.
- Navigate to the parent folder `/opt/data/sample_data/user_data`
```shell
cd /opt/data/sample_data/user_data
```
- Convert the input sample data to an output file that has tiling set to `512x512` (the compression parameters are explained in the code).
```shell
gdal_translate -co "TILED=YES" -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=JPEG" -co "PHOTOMETRIC=YCBCR" -co "QUALITY=85" aerial/13tde815295_200803_0x6000m_cl.tif retiled/13tde815295_200803_0x6000m_cl.tif
```
- Check that the output dataset has successfully been tiled by running the command
```shell
gdalinfo retiled/13tde815295_200803_0x6000m_cl.tif
```
_Part of the gdalinfo output on the tiled dataset. Notice the Block value now is 512x512._
### gdaladdo
This utility allows for the addition of overviews to a dataset. The following steps provide instructions to add overviews to the tiled sample dataset.
Running the command
```shell
gdaladdo
```
gets the list of supported parameters
```shell
Usage: gdaladdo [-r {nearest,average,gauss,average_mp,average_magphase,mode}]
[-ro] [--help-general] filename levels
```
The meaning of the main parameters is summarized below.
* `-r`: specifies the resampling algorithm (Nearest is the default value)
* `-ro`: opens the dataset in read-only mode, to generate an external overview (for GeoTIFF especially)
* `filename`: represents the file to build overviews for.
* `levels`: specifies a list of overview levels to build.
#### gdaladdo - Adding overviews to the sample dataset
- Navigate to the `retiled` folder
```shell
cd retiled
```
- Run the `gdaladdo` command as follows
```shell
gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config JPEG_QUALITY_OVERVIEW 85 13tde815295_200803_0x6000m_cl.tif 2 4 8 16 32
```
- Check that the overviews have been added to the dataset by running the command
```shell
gdalinfo 13tde815295_200803_0x6000m_cl.tif
```
_Part of the gdalinfo output on the tiled dataset with overviews. Note the Overviews properties._
#### Process in Bulk
Instead of manually repeating these 2 steps (retile + add overviews) for each file, we can invoke a few commands to automate it.
```shell
cd /opt/data/sample_data/user_data/aerial
```
```shell
for i in `find *.tif`; do gdal_translate -CO "TILED=YES" -CO "BLOCKXSIZE=512" -CO "BLOCKYSIZE=512" -co "COMPRESS=JPEG" -co "PHOTOMETRIC=YCBCR" -co "QUALITY=85" $i ../optimized/$i; gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config JPEG_QUALITY_OVERVIEW 85 ../optimized/$i 2 4 8 16 32; done
```
You should see a list of run like this.
```python
...
Input file size is 2500, 2500
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 2500, 2500
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 2500, 2500
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
...
```
At this point, optimized datasets have been prepared, and they are ready to be served by GeoServer as an `ImageMosaic`.
### gdalwarp
This utility warps and reprojects a dataset. The following steps provide instructions to reproject the aerial dataset (`EPSG:26913` coordinate reference system) to `WGS84` (`EPSG:4326`).
Running the command
```shell
gdalwarp
```
gets the list of supported parameters
```shell
Usage: gdalwarp [--help-general] [--formats]
[-s_srs srs_def] [-t_srs srs_def] [-to "NAME=VALUE"]
[-order n | -tps | -rpc | -geoloc] [-et err_threshold]
[-refine_gcps tolerance [minimum_gcps]]
[-te xmin ymin xmax ymax] [-tr xres yres] [-tap] [-ts width height]
[-wo "NAME=VALUE"] [-ot Byte/Int16/...] [-wt Byte/Int16]
[-srcnodata "value [value...]"] [-dstnodata "value [value...]"] -dstalpha
[-r resampling_method] [-wm memory_in_mb] [-multi] [-q]
[-cutline datasource] [-cl layer] [-cwhere expression]
[-csql statement] [-cblend dist_in_pixels] [-crop_to_cutline]
[-of format] [-co "NAME=VALUE"]* [-overwrite]
srcfile* dstfile
```
The meaning of the main parameters is summarized below.
* `-s_srs`: specifies the source coordinate reference system
* `-t_srs`: specifies the target coordinate reference system
* `-te`: sets georeferenced extents (expressed in target CRS) of the output
* `-tr`: specifies the output resolution (expressed in target georeferenced units)
* `-ts`: specifies the output size in pixels and lines.
* `-r`: specifies the resampling method (one of `near`, `bilinear`, `cubic`, `cubicspline`, and `lanczos`)
* `-srcnodata`: specifies band values to be excluded from interpolation.
* `-dstnodata`: specifies nodata values on the output file.
* `-wm`: specifies the amount of memory (expressed in megabytes) used by the warping API for caching.
#### gdalwarp - Reprojecting sample dataset to WGS84
- Navigate to the `/opt/data/sample_data/user_data/optimized` folder
```shell
cd /opt/data/sample_data/user_data/optimized
```
- Run the `gdalwarp` command
```shell
gdalwarp -t_srs "EPSG:4326" -co "TILED=YES" 13tde815295_200803_0x6000m_cl.tif 13tde815295_200803_0x6000m_cl_warped.tif
```
- Check that reprojection has been successful by running the command
```shell
gdalinfo 13tde815295_200803_0x6000m_cl_warped.tif
```
_Part of the gdalinfo output on the warped dataset. Note the updated Coordinate System property._
### Further Reading about GDAL
```shell
https://geoserver.geo-solutions.it/educational/en/raster_data/advanced_gdal/index.html
```
## Publish the Optimized Mosaic on GeoNode
- Activate the `geonode` virtual env and navigate to `/opt/geonode`
- Run the `updatelayers` management command as follows
```shell
./manage_local.sh updatelayers --skip-geonode-registered -w geonode -f boulder_bg_optimized
```
- Verify the new dataset has been successfully created on GeoNode
#### [Next Section: Optimizing, publishing and styling Vector data](OPTIMIZE_VECTOR.md)