Full GeoNode Backup & Restore

The admin command to backup, and restore, GeoNode allows for the consistent extraction of the GeoNode and GeoServer data models in a serializable meta-format which is later interpreted by the restore procedure to exactly rebuild the whole structure.

In particular, the tool helps developers, and administrators, correctly extract and serialize the following resources:

• GeoNode (Resource Base Model):

  1. Layers (both raster and vectors)

  2. Maps

  3. Documents

  4. People with Credentials

  5. Permissions

  6. Associated Styles

  7. Static data and templates

• GeoServer (Catalog):

  1. OWS Services configuration and limits

  2. Security model along with the auth filters configuration, users, and credentials

  3. Workspaces

  4. Stores (both DataStores and CoverageStores)

  5. Layers

  6. Styles

The tool exposes two GeoNode Management Commands, ‘backup’ and ‘restore’.

The commands allow for :

1. Fully backing up GeoNode data and fixtures in a zip archive.

2. Fully backing up the GeoServer configuration (physical datasets - tables, shapefiles, geotiffs)

3. Fully restoring GeoNode and GeoServer fixtures, and catalog from the zip archive.

The usage of these commands is quite easy and straightforward.

The first step is to ensure that everything is correctly configured and the requisites are respected in order to successfully perform a backup and restore of GeoNode.

Warning

It is worth noting that this functionality requires the latest GeoServer Extension (2.9.x or greater) for GeoNode to work correctly.

Note

The full GeoServer documentation is also available here GeoServer Docs

Requisites and Setup

Before running a GeoNode backup/restore, it is necessary to ensure everything is correctly configured and set up.

Settings

In accordance with the admin requirements, the file settings.ini must be created before running a backup or restore.

The default files can be found at geonode/br/management/commands/settings_sample.ini and geonode/br/management/commands/settings_docker_sample.ini for the classic and Docker environments accordingly. The content is similar in both of them (below is an example from settings_sample.ini):

[database]
pgdump = pg_dump
pgrestore = pg_restore

[geoserver]
datadir = /opt/data/geoserver_data/
# datadir_exclude_file_path = {comma separated list of paths to exclude from geoserver catalog} e.g.: /data,/data/geonode,/geonode
dumpvectordata = yes
dumprasterdata = yes
# data_dt_filter = {cmp_operator} {ISO8601} e.g. > 2019-04-05T24:00
# data_layername_filter = {comma separated list of layernames, optionally with glob syntax} e.g.: tuscany_*,
# data_layername_exclude_filter = {comma separated list of layernames, optionally with glob syntax} e.g.: tuscany_*,italy

[fixtures]
apps  = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client
dumps = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client

The settings.ini file can be created in any directory that is accessible by GeoNode, and its path can be passed to the backup/restore procedures using the -c (–config) argument.

There are a few different sections of the configuration file that must be carefully checked before running a backup/restore command.

Settings: [database] Section

[database]
pgdump = pg_dump
pgrestore = pg_restore

This section is quite simple. It contains only two properties:

• pgdump; the path of the pg_dump local command.

• pgrestore; the path of the pg_restore local command.

Warning

These properties are ignored if GeoNode is not configured to use a DataBase as a backend (see the settings.py and local_settings.py sections)

Note

The database connection settings (both for GeoNode and GeoServer) will be taken from the settings.py and local_settings.py configuration files. Make sure they are correctly configured (on the target GeoNode instance too) and the DataBase server is accessible while executing a backup/restore command.

Settings: [geoserver] Section

[geoserver]
datadir = /opt/gs_data_dir
datadir_exclude_file_path =
dumpvectordata = yes
dumprasterdata = yes
data_dt_filter =
data_layername_filter =
data_layername_exclude_filter =

This section allows enabling/disabling a full data backup/restore of GeoServer.

• datadir: the full path of GeoServer Data Directory. By default, it is /opt/gs_data_dir. The path must be accessible and fully writable by the geonode and/or httpd server users when executing a backup/restore command.

• datadir_exclude_file_path: a comma-separated list of paths to exclude from the geoserver_catalog.zip. This list will be sent, and managed directly, by the GeoServer Backup REST API.

• dumpvectordata: a boolean flag enabling, or disabling, the creation of a vector data dump from GeoServer (shapefiles or DB tables). If false (or no) vector data won’t be stored/re-stored.

• dumprasterdata: a boolean flag enabling or disabling the creation of a raster data dump from GeoServer (geotiffs). If false (or no) raster data won’t be stored/re-stored.

• data_dt_filter: {cmp_operator} {ISO8601} e.g. > 2019-04-05T24:00 which means “include on backup archive only the files that have been modified later than 2019-04-05T24:00

• data_layername_filter: a-comma separated list of layer names, optionally uses glob syntax e.g.: tuscany_*,italy. Only original RASTER data and VECTORIAL table dumps matching those filters will be included in the backup ZIP archive

• data_layername_exclude_filter: a comma-separated list of layer names, optionally uses glob syntax e.g.: tuscany_*,italy. The original RASTER data and VECTORIAL table dumps matching those filters will be excluded from the backup ZIP archive

Warning

Enabling these options requires the GeoServer Data Directory to be accessible and fully writable for the geonode and/or httpd server users when executing a backup/restore command.

Settings: [fixtures] Section

[fixtures]
#NOTE: Order is important
apps   = people,account,avatar.avatar,base.backup,base.license,base.topiccategory,base.region,base.resourcebase,base.contactrole,base.link,base.restrictioncodetype,base.spatialrepresentationtype,guardian.userobjectpermission,guardian.groupobjectpermission,layers.uploadsession,layers.style,layers.layer,layers.attribute,layers.layerfile,maps.map,maps.maplayer,maps.mapsnapshot,documents.document,taggit

dumps  = people,accounts,avatars,backups,licenses,topiccategories,regions,resourcebases,contactroles,links,restrictioncodetypes,spatialrepresentationtypes,useropermissions,groupopermissions,uploadsessions,styles,layers,attributes,layerfiles,maps,maplayers,mapsnapshots,documents,tags

This section is the most complex one. Usually, you don’t need to modify it. Only an expert user who knows Python and the GeoNode model structure should modify this section.

What its properties mean:

• apps; an ordered list of GeoNode Django applications. The backup/restore procedure will dump/restore the fixtures in a portable format.

• dumps; this is the list of files associated with the Django applications. The order must be the same as it is in the apps property above. Each name represents the filename to dump to, or read, from the single app’s fixtures.

Executing from the CLI

The following section shows instructions on how to perform backup/restore from the command line using the Django Admin Management Commands.

To obtain a basic user guide for the management command from the command line, just run

workon geonode
cd /opt/geonode

./manage_local.sh backup --help

./manage_local.sh restore --help

--help will provide the list of available command line options with a brief description.

By default, both procedures activate Read Only mode (disabling any content modifying requests), which is reverted to the previous state (from before the execution) after completion, regardless of the command’s result (success or failure). To disable the activation of this mode, the --skip-read-only argument can be passed to the command.

It is worth noting that both commands allow the following option

workon geonode
cd /opt/geonode

./manage_local.sh backup --force / -f

./manage_local.sh restore --force / -f

Which enables a non-interactive mode, meaning that the user will not be asked for an explicit confirmation.

Backup

In order to perform a backup, create a setting.ini file first and then run the backup command as shown here below:

• Prepare the environment

  sudo usermod -a -G tomcat ${USER}
  su - ${USER}
  

• Create the settings.ini file

  vim /opt/geonode/geonode-local-br-settings.ini
  

  [database]
  pgdump = pg_dump
  pgrestore = pg_restore
  
  [geoserver]
  datadir = /opt/data/geoserver_data/
  dumpvectordata = yes
  dumprasterdata = yes
  
  [fixtures]
  apps  = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client
  dumps = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client
  

• Run the backup command

  cd /opt/geonode
  workon geonode
  
  ./manage_local.sh backup -c /opt/geonode/geonode-local-br-settings.ini --backup-dir=/tmp
  

The management command will automatically generate a .zip archive file in the target tmp folder if successful. In the target directory, a .md5 file with the same name as the backup will be created. It contains the MD5 hash of the backup file, which can be used to check an archive’s integrity before restoration.

It is worth mentioning that the br (Backup & Restore GeoNode application) will not be dumped, even if specified in the settings.ini as its content is strictly related to the specific GeoNode instance.

Currently, GeoNode does not support any automatic extraction of the backup file. It should be manually transferred, if needed, to the target instance environment.

Restore

The restore command has several of arguments that modify its execution:

1. -c/--config: the path to the settings.ini configuration file. If the Backup archive is provided with its settings, the latter will be used by the restore command and this option won’t be mandatory anymore

2. --skip-geoserver: the GeoServer backup restoration won’t be performed

3. --skip-geoserver-info: {Default: True} Skips GeoServer Global Info, like the proxy base url and other global GeoServer metadata info

4. --skip-geoserver-security: {Default: True} Skips all the GeoServer Security Settings

5. --backup-file: (exclusive together with --backup-files-dir) the path to the backup .zip archive

6. --backup-files-dir: (exclusive together with --backup-file) the directory containing backup archives. The directory may contain several files, but only backup archives are allowed with a .zip extension. In case multiple archives are present in the directory, the newest one (created after the last, already restored, backup’s time of creation) will be restored. This option was implemented with automated restores in mind.

7. --recovery-file: The backup archive containing the GeoNode data to restore in case of failure.

8. -l / --with-logs: the backup file will be checked against the restoration logs (history). In case this backup has already been restored (MD5-based comparison), a RuntimeError is raised, preventing restore execution.

9. -n / --notify: the restore procedure outcome will be sent via e-mail notification to the superusers of the instance (note: notifications will be sent to the superusers of the instance before restoration).

10. --skip-read-only: the restore procedure will be conducted without setting Read Only mode during execution.

11. --soft-reset: the restore procedure will preserve the GeoServer table/resources during the restore. By default, the procedure will drop tables and resources

In order to perform a default backup restoration just run the command:

cd /opt/geonode
workon geonode

sudo chmod -Rf 777 /opt/geonode/geonode/uploaded/
sudo chmod -Rf 777 /opt/geonode/geonode/static_root/

sudo chown -Rf ${USER}: /opt/geonode/geonode/uploaded/
sudo chown -Rf ${USER}: /opt/geonode/geonode/static_root/

./manage_local.sh restore --backup-file=/tmp/<latest_backup_filename>.zip -l

For a restore to run it requires either the --backup-file or --backup-files-dir argument to be defined.

Warning

The Restore will overwrite the whole target instance of GeoNode (and by default GeoServer) including users, catalogs, and databases. So be very careful.

GeoNode Admin GUI Inspection

The history of restored backups can be verified in the admin panel.

Login to the admin panel and select the Restored backups table from the BACKUP/RESTORE application.

A list will be displayed with a history of all restored backups. You can select a certain backup to view its data.

The detailed view of the restored backup shows the backup archive’s name, its MD5 hash, its creation/modification date (in the target folder), and the date of the restoration. Please note Restored Backup history cannot be modified.

B/R in Docker environment

When executing B/R in the Docker environment, the creation of the backup to, or restoration from, should be executed in the /backup_restore directory. It is a shared volume between Geoserver and Geonode images, exclusively created for this purpose. Using another location will fail, as one of the images won’t have access to the files.

Warning

When executing B/R in the Docker environment, remember to create a settings.ini file based on settings_docker_sample.ini to point at a proper Geoserver data directory! In other cases, configuration mismatches may cause unexpected errors.

Warning

The only other volume shared between images is /geoserver_data/data, but backup creation should not be performed there as the recursive Geoserver backups may be created in this case.