.. _backup_restore_geonode:

=============================
Full GeoNode Backup & Restore
=============================

The admin command to backup, and restore, GeoNode allows for the consistent extraction of the GeoNode and GeoServer data models in a serializable meta-format which is later interpreted by the restore procedure to exactly rebuild the whole structure.

In particular, the tool helps developers, and administrators, correctly extract and serialize the following resources:

* **GeoNode** (Resource Base Model):

    #. Layers (both raster and vectors)

    #. Maps

    #. Documents

    #. People with Credentials

    #. Permissions

    #. Associated Styles

    #. Static data and templates

* **GeoServer** (Catalog):

    #. OWS Services configuration and limits

    #. Security model along with the auth filters configuration, users, and credentials

    #. Workspaces

    #. Stores (both DataStores and CoverageStores)

    #. Layers

    #. Styles

The tool exposes two GeoNode Management Commands, 'backup' and 'restore'.

The commands allow for :

#. Fully backing up GeoNode data and fixtures in a zip archive.

#. Fully backing up the GeoServer configuration (physical datasets - tables, shapefiles, geotiffs)

#. Fully restoring GeoNode and GeoServer fixtures, and catalog from the zip archive.

The usage of these commands is quite easy and straightforward.

The first step is to ensure that everything is correctly configured and the requisites are respected in order to successfully
perform a backup and restore of GeoNode.

.. warning:: It is worth noting that this functionality requires the latest `GeoServer Extension <https://build.geo-solutions.it/geonode/geoserver/latest/>`_ (2.9.x or greater) for GeoNode to work correctly.

.. note:: The full GeoServer documentation is also available here `GeoServer Docs <https://docs.geoserver.org/stable/en/user/community/backuprestore/index.html>`_

Requisites and Setup
====================

**Before** running a GeoNode backup/restore, it is necessary to ensure everything is correctly configured and set up.

Settings
--------

In accordance with the admin requirements, the file ``settings.ini`` must be created before running a backup or restore.

The default files can be found at ``geonode/br/management/commands/settings_sample.ini`` and ``geonode/br/management/commands/settings_docker_sample.ini``
for the classic and Docker environments accordingly. The content is similar in both of them (below is an example from ``settings_sample.ini``):

.. code-block:: ini

    [database]
    pgdump = pg_dump
    pgrestore = pg_restore

    [geoserver]
    datadir = /opt/data/geoserver_data/
    # datadir_exclude_file_path = {comma separated list of paths to exclude from geoserver catalog} e.g.: /data,/data/geonode,/geonode
    dumpvectordata = yes
    dumprasterdata = yes
    # data_dt_filter = {cmp_operator} {ISO8601} e.g. > 2019-04-05T24:00
    # data_layername_filter = {comma separated list of layernames, optionally with glob syntax} e.g.: tuscany_*,
    # data_layername_exclude_filter = {comma separated list of layernames, optionally with glob syntax} e.g.: tuscany_*,italy

    [fixtures]
    apps  = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client
    dumps = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client

The ``settings.ini`` file can be created in any directory that is accessible by GeoNode, and its path can be passed to the backup/restore
procedures using the `-c` (`--config`) argument.

There are a few different sections of the configuration file that must be carefully checked before running a backup/restore command.

Settings: [database] Section
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: ini

    [database]
    pgdump = pg_dump
    pgrestore = pg_restore

This section is quite simple. It contains only two properties:

* *pgdump*; the path of the ``pg_dump`` local command.

* *pgrestore*; the path of the ``pg_restore`` local command.

.. warning:: These properties are ignored if GeoNode is not configured to use a DataBase as a backend (see the ``settings.py`` and ``local_settings.py`` sections)

.. note:: The database connection settings (both for GeoNode and GeoServer) will be taken from the ``settings.py`` and ``local_settings.py`` configuration files. Make sure they are correctly configured (on the target GeoNode instance too) and the DataBase server is accessible while executing a backup/restore command.

Settings: [geoserver] Section
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: ini

    [geoserver]
    datadir = /opt/gs_data_dir
    datadir_exclude_file_path =
    dumpvectordata = yes
    dumprasterdata = yes
    data_dt_filter =
    data_layername_filter =
    data_layername_exclude_filter =

This section allows enabling/disabling a full data backup/restore of GeoServer.

* *datadir*: the full path of GeoServer Data Directory. By default, it is ``/opt/gs_data_dir``. The path **must** be accessible and **fully writable** by the ``geonode`` and/or ``httpd server`` users when executing a backup/restore command.

* *datadir_exclude_file_path*: a comma-separated list of paths to exclude from the ``geoserver_catalog.zip``. This list will be sent, and managed directly, by the GeoServer Backup REST API.

* *dumpvectordata*: a boolean flag enabling, or disabling, the creation of a vector data dump from GeoServer (shapefiles or DB tables). If ``false`` (or ``no``) vector data won't be stored/re-stored.

* *dumprasterdata*: a boolean flag enabling or disabling the creation of a raster data dump from GeoServer (geotiffs). If ``false`` (or ``no``) raster data won't be stored/re-stored.

* *data_dt_filter*: {cmp_operator} {ISO8601} e.g. > 2019-04-05T24:00 which means "include on backup archive only the files that have been modified later than 2019-04-05T24:00

* *data_layername_filter*: a-comma separated list of ``layer names``, optionally uses glob syntax e.g.: tuscany_*,italy. Only original ``RASTER`` data and ``VECTORIAL`` table dumps matching those filters will be **included** in the backup ZIP archive

* *data_layername_exclude_filter*: a comma-separated list of ``layer names``, optionally uses glob syntax e.g.: tuscany_*,italy. The original ``RASTER`` data and ``VECTORIAL`` table dumps matching those filters will be **excluded** from the backup ZIP archive


.. warning:: Enabling these options **requires** the GeoServer Data Directory to be accessible and **fully writable** for the ``geonode`` and/or ``httpd server`` users when executing a backup/restore command.

Settings: [fixtures] Section
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: ini

    [fixtures]
    #NOTE: Order is important
    apps   = people,account,avatar.avatar,base.backup,base.license,base.topiccategory,base.region,base.resourcebase,base.contactrole,base.link,base.restrictioncodetype,base.spatialrepresentationtype,guardian.userobjectpermission,guardian.groupobjectpermission,layers.uploadsession,layers.style,layers.layer,layers.attribute,layers.layerfile,maps.map,maps.maplayer,maps.mapsnapshot,documents.document,taggit

    dumps  = people,accounts,avatars,backups,licenses,topiccategories,regions,resourcebases,contactroles,links,restrictioncodetypes,spatialrepresentationtypes,useropermissions,groupopermissions,uploadsessions,styles,layers,attributes,layerfiles,maps,maplayers,mapsnapshots,documents,tags

This section is the most complex one. Usually, you don't need to modify it. Only an expert user who knows Python and the GeoNode model structure should modify this section.

What its properties mean:

* *apps*; an ordered list of GeoNode Django applications. The backup/restore procedure will dump/restore the fixtures in a portable format.

* *dumps*; this is the list of ``files`` associated with the Django applications. The order **must** be the same as it is in the *apps* property above. Each name represents the ``filename`` to dump to, or read, from the single app's fixtures.


Executing from the CLI
======================

The following section shows instructions on how to perform backup/restore from the command line using the Django Admin Management Commands.

To obtain a basic user guide for the management command from the command line, just run

    .. code-block:: shell

        workon geonode
        cd /opt/geonode

        ./manage_local.sh backup --help

        ./manage_local.sh restore --help

``--help`` will provide the list of available command line options with a brief description.

By default, both procedures activate `Read Only` mode (disabling any content modifying requests), which is reverted
to the previous state (from before the execution) after completion, regardless of the command's result (success or failure).
To disable the activation of this mode, the ``--skip-read-only`` argument can be passed to the command.

It is worth noting that both commands allow the following option

    .. code-block:: shell

        workon geonode
        cd /opt/geonode

        ./manage_local.sh backup --force / -f

        ./manage_local.sh restore --force / -f

Which enables a non-interactive mode, meaning that the user will not be asked for an explicit confirmation.

Backup
------

In order to perform a backup, create a `setting.ini` file first and then run the `backup` command as shown here below:

 - Prepare the environment

    .. code-block:: shell

        sudo usermod -a -G tomcat ${USER}
        su - ${USER}

 - Create the `settings.ini` file

    .. code-block:: shell

        vim /opt/geonode/geonode-local-br-settings.ini

    .. code-block:: ini

        [database]
        pgdump = pg_dump
        pgrestore = pg_restore

        [geoserver]
        datadir = /opt/data/geoserver_data/
        dumpvectordata = yes
        dumprasterdata = yes

        [fixtures]
        apps  = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client
        dumps = contenttypes,auth,people,groups,account,guardian,admin,actstream,announcements,avatar,base,documents,geoserver,invitations,pinax_notifications,layers,maps,oauth2_provider,services,harvesting,sites,socialaccount,taggit,tastypie,upload,user_messages,geonode_themes,geoapps,favorite,geonode_client

 - Run the `backup` command

    .. code-block:: shell

        cd /opt/geonode
        workon geonode

        ./manage_local.sh backup -c /opt/geonode/geonode-local-br-settings.ini --backup-dir=/tmp

The management command will automatically generate a ``.zip`` archive file in the target ``tmp`` folder if successful. In the target directory, a ``.md5`` file with the same name as the backup will be created. It contains the MD5 hash of the backup file, which can be used to check an archive's integrity before restoration.

It is worth mentioning that the ``br`` (Backup & Restore GeoNode application) will not be dumped, even if specified in the ``settings.ini`` as
its content is strictly related to the specific GeoNode instance.

Currently, GeoNode does not support any automatic extraction of the backup file. It should be manually transferred, if needed, to the target
instance environment.

Restore
-------

The ``restore`` command has several of arguments that modify its execution:

#. ``-c``/``--config``: the path to the ``settings.ini`` configuration file. If the Backup archive is provided with its settings, the latter will be used by the restore command and this option won't be mandatory anymore

#. ``--skip-geoserver``: the GeoServer backup restoration won't be performed

#. ``--skip-geoserver-info``: {Default: True} Skips GeoServer Global Info, like the proxy base url and other global GeoServer metadata info

#. ``--skip-geoserver-security``: {Default: True} Skips all the GeoServer Security Settings

#. ``--backup-file``: (exclusive together with ``--backup-files-dir``) the path to the backup ``.zip`` archive

#. ``--backup-files-dir``: (exclusive together with ``--backup-file``) the directory containing backup archives. The directory may contain several files, but **only** backup archives are allowed with a ``.zip`` extension. In case multiple archives are present in the directory, the newest one (created after the last, already restored, backup's time of creation) will be restored. This option was implemented with automated restores in mind.

#. ``--recovery-file``: The backup archive containing the GeoNode data to restore in case of failure.

#. ``-l`` / ``--with-logs``: the backup file will be checked against the restoration logs (history). In case this backup has already been restored (MD5-based comparison), a RuntimeError is raised, preventing restore execution.

#. ``-n`` / ``--notify``: the restore procedure outcome will be sent via e-mail notification to the superusers of the instance (note: notifications will be sent to the superusers of the instance before restoration).

#. ``--skip-read-only``: the restore procedure will be conducted without setting `Read Only` mode during execution.

#. ``--soft-reset``: the restore procedure will preserve the GeoServer table/resources during the restore. By default, the procedure will drop tables and resources

In order to perform a default backup restoration just run the command:

    .. code-block:: shell

        cd /opt/geonode
        workon geonode

        sudo chmod -Rf 777 /opt/geonode/geonode/uploaded/
        sudo chmod -Rf 777 /opt/geonode/geonode/static_root/

        sudo chown -Rf ${USER}: /opt/geonode/geonode/uploaded/
        sudo chown -Rf ${USER}: /opt/geonode/geonode/static_root/

        ./manage_local.sh restore --backup-file=/tmp/<latest_backup_filename>.zip -l

For a restore to run it requires either the ``--backup-file`` or ``--backup-files-dir`` argument to be defined.

.. warning:: The Restore will **overwrite** the whole target instance of GeoNode (and by default GeoServer) including users, catalogs, and databases. So be very careful.

GeoNode Admin GUI Inspection
============================

The history of restored backups can be verified in the admin panel.

Login to the admin panel and select the ``Restored backups`` table from the ``BACKUP/RESTORE`` application.

.. figure:: img/br_1.png
   :align: center


A list will be displayed with a history of all restored backups. You can select a certain backup to view its data.

.. figure:: img/br_2.png
   :align: center


The detailed view of the restored backup shows the backup archive's name, its MD5 hash, its creation/modification date (in the target folder), and the date of the restoration. Please note Restored Backup history cannot be modified.

.. figure:: img/br_3.png
   :align: center

B/R in Docker environment
=========================

When executing B/R in the Docker environment, the creation of the backup to, or restoration from, should be executed in the ``/backup_restore`` directory. It is a shared volume between Geoserver and Geonode images, exclusively created for this purpose. Using another location will fail, as one of the images won't have access to the files.

.. warning:: When executing B/R in the Docker environment, **remember** to create a ``settings.ini`` file based on ``settings_docker_sample.ini`` to point at a proper Geoserver data directory! In other cases, configuration mismatches may cause unexpected errors.

.. warning:: The only other volume shared between images is ``/geoserver_data/data``, but backup creation **should not** be performed there as the recursive Geoserver backups may be created in this case.