.. highlight:: bash .. _inst-guide: ##################################### Installation and Administration Guide ##################################### ******* Preface ******* Overview ======== This guide provides information about how to use the rasdaman array database system, in particular: installation and system administration. For storage of multi-dimensional array data, rasdaman can be configured to use some conventional database system (such as PostgreSQL) or use its own storage manager. For the purpose of this documentation, we will call the conventional database system to which rasdaman is interfaced the *base DBMS*, understanding that this base DBMS is in charge of all alphanumeric data maintained as relational tables or object-oriented semantic nets. This guide is specific for *rasdaman enterprise*. Audience ======== The information in this manual is intended primarily for database and system administrators. Rasdaman Documentation Set ========================== This manual should be read in conjunction with the complete rasdaman documentation set which this guide is part of. The documentation set in its completeness covers all important information needed to work with the rasdaman system, such as programming and query access to databases, guidance to utilities such as *raswct*, release notes, and additional information on the rasdaman wiki. .. _sec-download-and-install: ************ Installation ************ This page describes installation of rasdaman enterprise Debian or RPM packages. With your purchase, you have received a login to the rasdaman download area, which will be necessary to setup the installation and updating of rasdaman packages. **Hardware & Software Requirements** It is recommended to have at least 8 GB main memory. Disk space depends on the size of the databases, as well as the requirements of the base DBMS of rasdaman chosen. The footprint of the rasdaman installation itself is around 400 MB. Rasdaman is continuously tested on the platforms listed below. The rasdaman code has been developed on SUN/Solaris and HP-UX originally, and has been ported to IBM AIX, SGI IRIX, and DEC Unix - but that was way back in the last millennium. - Ubuntu 18.04, 20.04, 22.04 - CentOS 7 The rasdaman engine in the packages uses embedded SQLite for managing its array metadata. The geo service component, petascope, currently relies on a PostgreSQL database by default, but can be reconfigured with an embedded H2 database instead if desired. **License Key** In order to run a rasdaman server you have to obtain a licence from rasdaman GmbH. This licence key encodes, among others, the number of cores and the server's interface name (such as "eth0") and corresponding MAC address. The following commands are usually used to obtain this information: :: # interface name + IP address $ ifconfig # alternatively $ ip link # number of CPUs $ cat /proc/cpuinfo # alternatively $ nproc After communicating these ingredients to rasdaman GmbH in the course of a licence purchase, a licence key file will be provided which has to be stored on the machine where the rasdaman server runs. **Compatibility** Rasdaman community and enterprise cannot run in parallel on the same machine. If you plan to have both installations on the same machine, make sure they reside in different directories and are not active at the same time. Rasdaman databases created with rasdaman community are upwards compatible with rasdaman enterprise. **Support** For support in installing rasdaman and any other question you may contact rasdaman GmbH at `www.rasdaman.com `__. .. _sec-system-install-pkgs-deb: Debian-based systems ==================== Currently the following Debian-based distributions are supported: - Ubuntu 18.04 / 20.04 / 22.04 Installation ------------ 1. Copy the rasdaman license key to ``/opt/rasdaman/etc``, e.g: :: $ sudo mkdir -p /opt/rasdaman/etc $ sudo cp rmankey /opt/rasdaman/etc .. note:: This has to be done before installing rasdaman. 2. Import the rasdaman repository public key to the apt keychain (make sure to replace USERNAME / PASSWORD with your download credentials): :: $ wget -O - https://USERNAME:PASSWORD@download.rasdaman.com/Download/rasdaman.gpg | \ sudo apt-key add - .. note:: You may need to update the ca-certificates package to allow SSL-based applications (e.g. ``apt-get update`` or ``wget/curl``) to check for the authenticity of SSL connections: :: $ sudo apt-get install ca-certificates 3. Add the rasdaman repository to apt (make sure to replace USERNAME / PASSWORD with your download credentials): - **stable:** these packages are only updated on stable releases of rasdaman, and hence recommended for operational production installations. :: # For ubuntu 22.04 $ echo "deb [arch=amd64] https://USERNAME:PASSWORD@download.rasdaman.com/Download/deb jammy stable" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list # For ubuntu 20.04 $ echo "deb [arch=amd64] https://USERNAME:PASSWORD@download.rasdaman.com/Download/deb focal stable" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list # For ubuntu 18.04 $ echo "deb [arch=amd64] https://USERNAME:PASSWORD@download.rasdaman.com/Download/deb bionic stable" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list - **testing:** updated more frequently with beta releases, so aimed for feature testing in less-critical installations. :: # For ubuntu 22.04 $ echo "deb [arch=amd64] https://USERNAME:PASSWORD@download.rasdaman.com/Download/deb jammy testing" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list # For ubuntu 20.04 $ echo "deb [arch=amd64] https://USERNAME:PASSWORD@download.rasdaman.com/Download/deb focal testing" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list # For ubuntu 18.04 $ echo "deb [arch=amd64] https://USERNAME:PASSWORD@download.rasdaman.com/Download/deb bionic testing" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list 4. rasdaman can be installed now: :: $ sudo apt-get update $ sudo apt-get install rasdaman If during the install you get a prompt like the below, type **N** (default option): .. code-block:: text Configuration file `/etc/opt/rasdaman/petascope.properties' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** petascope.properties (Y/I/N/O/D/Z) [default=N] ? If you are automating the installation (in a script for example), you can bypass this prompt with an apt-get option as follows: :: $ apt-get -o Dpkg::Options::="--force-confdef" install -y rasdaman You will find the rasdaman installation under ``/opt/rasdaman/``. Finally, to make rasql available on the PATH for your system user: :: $ source /etc/profile.d/rasdaman.sh 5. Check that the rasdaman server can answer queries: :: $ rasql -q 'select c from RAS_COLLECTIONNAMES as c' --out string Typical output: :: rasql: rasdaman query tool v1.0, rasdaman v10.0.0 -- generated on 26.02.2020 08:44:56. opening database RASBASE at localhost:7001...ok Executing retrieval query...ok Query result collection has 0 element(s): rasql done. 6. Check that petascope is initialized properly, typically at this URL: :: http://localhost:8080/rasdaman/ows 7. If SELinux is running then possibly some extra configuration is needed to get petascope run properly. See :ref:`here ` for more details. .. _sec-system-update-pkgs-deb: Updating -------- The packages are updated whenever a new rasdaman version is released. To update your installation: :: $ sudo apt-get update $ sudo service rasdaman stop $ sudo apt-get install rasdaman .. note:: You may need to update the ca-certificates package to allow SSL-based applications (e.g. ``yum update`` or ``wget/curl``) to check for the authenticity of SSL connections: :: $ sudo apt-get install ca-certificates .. _sec-system-install-pkgs-rpm: RPM-based systems ================= Currently the following RPM-based distributions are supported: - CentOS 7 - CentOS 7, linked against GDAL 2.x Installation ------------ 1. Copy the rasdaman license key to ``/opt/rasdaman/etc``, e.g: :: $ sudo mkdir -p /opt/rasdaman/etc $ sudo cp rmankey /opt/rasdaman/etc .. note:: This has to be done before installing rasdaman. 2. Add the stable rasdaman repository to yum (make sure to replace USERNAME / PASSWORD with your download credentials). The stable repository is only upated on stable releases of rasdaman, and hence recommended for operational production installations. :: - If GDAL 1.x is installed from the standard packages: :: $ sudo curl "https://USERNAME:PASSWORD@download.rasdaman.com/Download/rpm/stable/CentOS/7/x86_64/rasdaman.repo" \ -o /etc/yum.repos.d/rasdaman.repo - If GDAL 2.x has been manually compiled: :: $ sudo curl "https://USERNAME:PASSWORD@download.rasdaman.com/Download/rpm/stable/CentOS/7_gdal2/x86_64/rasdaman.repo" \ -o /etc/yum.repos.d/rasdaman.repo Alternatively, you can enable the testing repository which is updated more frequently with beta releases, so aimed for feature testing in less-critical installations: - If GDAL 1.x is installed from the standard packages: :: $ sudo curl "https://USERNAME:PASSWORD@download.rasdaman.com/Download/rpm/testing/CentOS/7/x86_64/rasdaman.repo" \ -o /etc/yum.repos.d/rasdaman.repo - If GDAL 2.x has been manually compiled: :: $ sudo curl "https://USERNAME:PASSWORD@download.rasdaman.com/Download/rpm/testing/CentOS/7_gdal2/x86_64/rasdaman.repo" \ -o /etc/yum.repos.d/rasdaman.repo Edit the file downloaded, replacing "USERNAME" and "PASSWORD" with the same user name and password, respectively, as obtained from rasdaman GmbH or one of its authorized retailers. .. note:: You may need to update the ca-certificates package to allow SSL-based applications (e.g. ``yum update`` or ``wget/curl``) to check for the authenticity of SSL connections: :: $ sudo yum install -y ca-certificates 3. The rasdaman packages should be available now via yum: :: $ sudo yum clean all $ sudo yum update $ sudo yum search rasdaman Output: :: rasdaman.x86_64 : Rasdaman extends standard relational database systems with the ability to store and retrieve multi-dimensional raster data 4. Add the EPEL repository to yum (`official page `__), needed for several dependencies of the rasdaman package: :: $ sudo yum install epel-release 5. Install the rasdaman package: :: $ sudo yum install rasdaman You will find the rasdaman installation under ``/opt/rasdaman/``. To make rasql available on the PATH for your system user: :: $ source /etc/profile.d/rasdaman.sh .. note:: If petascope has *problems* connecting to rasdaman, check this `FAQ entry `__ for some advice. 6. Check that the rasdaman server can answer queries: :: $ rasql -q 'select c from RAS_COLLECTIONNAMES as c' --out string Typical output: :: rasql: rasdaman query tool v1.0, rasdaman v10.0.0 -- generated on 26.02.2020 08:44:56. opening database RASBASE at localhost:7001...ok Executing retrieval query...ok Query result collection has 0 element(s): rasql done. 7. Check that petascope is initialized properly, typically at this URL: :: http://localhost:8080/rasdaman/ows 8. If SELinux is running then likely some extra configuration is needed to get petascope run properly. See :ref:`here ` for more details. .. _sec-system-update-pkgs-rpm: Updating -------- The packages are updated whenever a new version of rasdaman is released. To download an update perform these steps: :: $ sudo yum clean all $ sudo service rasdaman stop $ sudo yum update rasdaman .. note:: You may need to update the ca-certificates package to allow SSL-based applications (e.g. ``yum update`` or ``wget/curl``) to check for the authenticity of SSL connections: :: $ sudo yum install -y ca-certificates .. _customize-package-install: Customizing the package installation ==================================== When installing or updating rasdaman from the official packages, the process can be optionally customized with an installation profile (see example `installer configuration `__). - To customize when installing rasdaman for the first time, it is necessary to first download the package install profile from `here `__. - When updating an existing rasdaman installation, you can find the default package install profile in your installation at ``/opt/rasdaman/share/rasdaman/installer/profiles/package/install.toml``. Download / copy the ``install.toml`` file to some place, e.g. ``$HOME/rasdaman_install.toml``, and make any desired changes to it before installing or updating rasdaman. Make sure that the ``RAS_INSTALL_PATH`` environment variable is set to point to the custom profile, e.g. .. code-block:: shell export RAS_INSTALL_PATH="$HOME/rasdaman_install.toml" When you install or update rasdaman afterwards, the configuration process will take the custom profile into account instead of the default one. .. _sec-system-install-administration: **************** Running rasdaman **************** This section provides a *high-level* overview on how start/stop rasdaman and petascope, monitor them, and configure for typical usage. Most of the time the information presented here is sufficient for operating a rasdaman service; for deeper understanding on how it works behind the scenes, check the :ref:`sec-server-administration` section. .. _sec-system-install-service-control: Service Control =============== rasdaman -------- A ``rasdaman`` service script allows to start/stop rasdaman, e.g. :: $ service rasdaman start $ service rasdaman stop $ service rasdaman force-stop $ service rasdaman status It can be similarly referenced with ``systemctl``, e.g. :: $ systemctl start rasdaman $ systemctl stop rasdaman $ systemctl status rasdaman The service script can be customized by updating environment variables in ``/etc/default/rasdaman`` (create the file if it does not exist). The default settings can be seen below. .. hidden-code-block:: shell # rasdaman installation directory RMANHOME=/opt/rasdaman # local user running the rasdaman server RMANUSER=rasdaman # runuser, or sudo for older OS RUNUSER=runuser # login credentials for non-interactive rasdaman start/stop RASLOGIN=rasadmin:d293a15562d3e70b6fdc5ee452eaed40 # port on which clients connect to rasdaman RASMGR_PORT=7001 # options to be passed on to start_rasdaman.sh START_RASDAMAN_OPTS="-p $RASMGR_PORT" # options to be passed on to stop_rasdaman.sh STOP_RASDAMAN_OPTS="-p $RASMGR_PORT" # Java options to be passed on to embedded petascope JAVA_OPTS="-Xmx4000m" See also the dedicated pages on :ref:`configuration and log files ` and :ref:`administration `. petascope --------- Check :ref:`this section ` on how to start / stop the petascope component of rasdaman. .. _sec-system-install-monitoring: Service monitoring ================== To help with monitoring the health of a running rasdaman service, a ``watch_rasdaman.sh`` script is provided in ``/opt/rasdaman/bin``. It performs status checks on rasdaman by sending a test rasql query, and to petascope by sending a test WCS GetCapabilities request. If a problem is detected in the response, then rasdaman and/or petascope will be restarted, unless this is prevented via the appropriate options. To support the restart actions, it should be executed with root or a user that has sudo rights. Various information is logged to stdout, as well as ``/opt/rasdaman/log/watch_rasdaman.sh.log``. In case of problems, the script can be configured to send an email notification. To see usage details and a list of all options, execute ``watch_rasdaman.sh --help``; in short: :: watch_rasdaman.sh [ --email-config ] [ --petascope-endpoint ] [ --no-restart-rasdaman ] [ --no-restart-petascope ] [ --rmanhome ] It is recommended to execute it regularly with a cron job, e.g. every 10 minutes: :: $ sudo su # switch to root user $ crontab -e */10 * * * * /opt/rasdaman/bin/watch_rasdaman.sh --email-config ~/.email.cfg .. note:: Executing the script as root is safe, the only system-modifying actions it makes are: logging information in ``/opt/rasdaman/log/watch_rasdaman.sh.log``, and potentially restarting rasdaman, and restarting tomcat if external servlet container deployment is configured in petascope.properties. .. _sec-system-initialize-rasdaman: Configure rasdaman ================== Rasdaman is a multi-server multi-user system. The server processes available must be configured initially, which is done in file ``$RMANHOME/etc/rasmgr.conf``. For distribution, this configuration contains ten server processes going by a name like, for example, ``N1``. If this is fine then you can just leave it as it is. If you want to change this by modifying server startup parameters or increasing the number of server processes available then see :ref:`sec-rascontrol-invocation` for details on how to do this. .. _sec-initialize-geo-services: Configure petascope =================== *Petascope* is the geo Web service frontend of rasdaman. It adds geo semantics on top of arrays, thereby enabling regular and irregular grids based on the `OGC coverage standards `__. To implement the geo semantics, petascope uses a relational database for the geo-related metadata. Currently, PostgreSQL and H2 / HSQLDB are supported. The package post-install script will automatically set up PostgreSQL for use by petascope. The steps approximately performed by the script are listed below. The default setup can be changed in the :ref:`petascope.properties ` configuration file. PostgreSQL ---------- PostgreSQL is automatically configured when rasdaman is installed, so doing the below is not usually necessary; we list the steps as documentation of how is PostgreSQL configured by default: 1. If postgres has not been initialized yet: :: $ sudo service postgresql initdb If the output is 'Data directory is not empty!' then this step is skipped. 2. Trust-based access in PostgreSQL is enabled by adding the below configuration before the ident lines to ``/etc/postgresql/9.4/main/pg_hba.conf`` on Debian 8, or ``/var/lib/pgsql/data/pg_hba.conf`` on CentOS 7: :: host all petauser localhost md5 host all petauser 127.0.0.1/32 md5 host all petauser ::1/128 md5 3. Reload PostgreSQL so that the new configuration will take effect: :: $ sudo service postgresql reload 4. Add a petascope user, for example ``petauser``, to PostgreSQL: :: $ sudo -u postgres createuser -s petauser -P > enter password In ``$RMANHOME/etc/petascope.properties`` set the ``spring.datasource.username``/``spring.datasource.password`` and ``metadata_user``/``metadata_pass`` options accordingly to this user / password. The password is randomly generated. 5. Copy ``/opt/rasdaman/share/rasdaman/war/rasdaman.war`` to the Tomcat webapps directory (``/var/lib/tomcat/webapps`` on CentOS 7) and restart Tomcat. Following successful deployment, petascope accepts OGC W*S requests at URL ``http://localhost:8080/rasdaman/ows``. H2 / HSQLDB ----------- To alternatively set up H2 / HSQLDB for use by petascope instead of PostgreSQL: 1. Create a directory that will host petascopedb and the H2 driver: :: $ mkdir /opt/rasdaman/geodb 2. Make sure the user running the webserver serving petascope can read/write to the folder above. For example, Tomcat webserver which uses `tomcat` user :: $ sudo chown -R tomcat: /opt/rasdaman/geodb However, if embedded deployment is enabled in petascope.properties, then the owner should be the ``rasdaman`` user which runs rasdaman :: $ sudo chown -R rasdaman: /opt/rasdaman/geodb 3. Download the driver and place it in the created directory. For example, download a H2 driver :: $ cd /opt/rasdaman/geodb $ wget https://repo1.maven.org/maven2/com/h2database/h2/1.4.200/h2-1.4.200.jar 4. Configure database settings in petascope.properties file, see :ref:`details `. 5. Restart the webserver running petascope (or rasdaman if embedded tomcat). .. _selinux-configuration: SELinux configuration --------------------- If ``SELinux`` is enabled (result of ``getenforce`` is ``enforcing``) then permissions for the ``tomcat`` user which is running petascope need to be configured properly if petascope is running in an external servlet container (as opposed to :ref:`embedded `): - Allow to load the ``gdal-java`` native library (via JNI) - Read / write files in ``/tmp/rasdaman_*`` - Make HTTP requests to rasdaman and get back results on ports ``7001-7010`` (these are default, specified in ``$RMANHOME/etc/rasmgr.conf``). Before proceeding, a SELinux utility package needs to be installed on CentOS 7: :: $ sudo yum install policycoreutils-python There are two ways to configure SELinux in order to enable petascope: 1. Change from ``enforcing`` to ``permissive`` for Tomcat: :: $ semanage permissive -a tomcat_t 2. Create specific rules for the ``tomcat`` user and register with ``SELinux``. - Create a rule config file ``tomcat_config.te`` with this contents: .. hidden-code-block:: text module tomcat_config 1.0; require { type tomcat_t; type tomcat_var_lib_t; type usr_t; type tomcat_exec_t; type unconfined_service_t; type afs_pt_port_t; type tomcat_tmp_t; type tmpfs_t; type afs3_callback_port_t; class tcp_socket name_connect; class file { append create execute read relabelfrom rename write }; class shm { associate getattr read unix_read unix_write write }; } # ============= tomcat_t ============== allow tomcat_t afs3_callback_port_t:tcp_socket name_connect; allow tomcat_t tmpfs_t:file { read write }; allow tomcat_t tomcat_tmp_t:file { execute relabelfrom }; allow tomcat_t tomcat_var_lib_t:file execute; allow tomcat_t unconfined_service_t:shm { associate getattr read unix_read unix_write write }; - Create a shell script ``deployse.sh`` to generate a binary package from this config file: .. hidden-code-block:: bash #!/bin/bash set -e MODULE=${1} # this will create a .mod file checkmodule -M -m -o ${MODULE}.mod ${MODULE}.te # this will create a compiled semodule semodule_package -m ${MODULE}.mod -o ${MODULE}.pp # this will install the module semodule -i ${MODULE}.pp - Run the script to load the binary package module to ``SELinux``: :: $ sudo ./deployse.sh tomcat_config Restart Tomcat with ``sudo service tomcat restart``; now rasdaman should be able to import data to petascope via WCSTImport and get data from rasdaman via WCS / WMS / WCPS. SSL/TLS configuration --------------------- Transport Layer Security (``TLS``) and its predecessor, Secure Sockets Layer (``SSL``), are technologies which allow web browsers and web servers to communicate over a secured connection. To configure it for ``petascope`` and ``secore web`` applications for ``Tomcat``, check the `official guide `__. .. _sec-rrasdaman-install: .. TODO .. _sec-system-install-conf: ************************ Installed Files and Data ************************ Top-level directories ===================== As common with rasdaman, we refer to the installation location as ``$RMANHOME`` below; the default is ``/opt/rasdaman``. The table below lists the top-level directories found in ``$RMANHOME`` after a fresh installation. +---------------------+-------------------------------------------------------------+ |**Directory** |**Description** | +=====================+=============================================================+ |``bin`` |rasdaman executables, e.g. rasql, start_rasdaman.sh, ... | +---------------------+-------------------------------------------------------------+ |``data`` |Path where the server stores array tiles as files; this | | |directory can get big, it is recommended to make | | |it a link to a sufficiently large disk partition. | +---------------------+-------------------------------------------------------------+ |``etc`` |Configuration files, e.g. rasmgr.conf | +---------------------+-------------------------------------------------------------+ |``include`` |C++ API development headers. | +---------------------+-------------------------------------------------------------+ |``lib`` |C++ and Java API libraries. | +---------------------+-------------------------------------------------------------+ |``log`` |``rasmgr`` and ``rasserver`` log files. | +---------------------+-------------------------------------------------------------+ |``share`` |Various artefacts like documentation, python/javascript | | |clients, example data, migration scripts, etc. | +---------------------+-------------------------------------------------------------+ .. _sec-executables: Executables =========== Rasdaman executables are found in ``$RMANHOME/bin``; the table below lists the various binaries and scripts. More detailed information on these components is provided in the :ref:`sec-rasdaman-architecture` Section. +------------------------------+----------------------------------------------------------------+ |**Executables** |**Description** | +==============================+================================================================+ |``rasserver`` |Client queries are evaluated by a ``rasserver`` worker process. | +------------------------------+----------------------------------------------------------------+ |``rasmgr`` |A manager process that controls ``rasserver`` processes and | | |client/server pairing. | +------------------------------+----------------------------------------------------------------+ |``rascontrol`` |A command-line frontend for ``rasmgr``. | +------------------------------+----------------------------------------------------------------+ |``directql`` |A rasserver that can execute queries directly, bypassing the | | |client/server protocol; useful for debugging. | +------------------------------+----------------------------------------------------------------+ |``rasql`` |A command-line client for sending queries to a ``rasserver`` | | |(as assigned by the ``rasmgr``). | +------------------------------+----------------------------------------------------------------+ |``start_rasdaman.sh`` |Start ``rasmgr`` and the worker ``rasservers`` as | | |configured in ``$RMANHOME/etc/rasmgr.conf``. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``stop_rasdaman.sh`` |Shutdown rasdaman, embedded petascope and embedded secore | | |if enabled. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``watch_rasdaman.sh`` |Helper script for monitoring an operational rasdaman service. | | |Details in section on :ref:`sec-system-install-monitoring`. | +------------------------------+----------------------------------------------------------------+ |``create_db.sh`` |Initialize the rasdaman metadata database (RASBASE). | +------------------------------+----------------------------------------------------------------+ |``update_dh.sh`` |Applies migration scripts to RASBASE. | +------------------------------+----------------------------------------------------------------+ |``rasdaman_insertdemo.sh`` |Insert three demo collections into rasdaman (used in the | | |rasdaman Query Language Guide). | +------------------------------+----------------------------------------------------------------+ |``petascope_insertdemo.sh`` |Insert geo-referenced demo coverage in petascope. | +------------------------------+----------------------------------------------------------------+ |``migrate_petascopedb.sh`` |Applies database migrations on petascopedb. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``wcst_import.sh`` |Tool for convenient and flexible import of | | |geo-referenced data into petascope. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``prepare_issue_report.sh`` |Helps preparing a report for an issue encountered while | | |operating rasdaman. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``rasfed.jar`` |Federation daemon. | +------------------------------+----------------------------------------------------------------+ .. _executables-start-rasdaman: start_rasdaman.sh ----------------- This script starts rasdaman. Normally rasdaman is installed from packages, and instead of executing this script directly one would execute ``service rasdaman start``. Any options to be passed on to ``start_rasdaman.sh`` can be set in ``/etc/default/rasdaman`` in this case; see :ref:`more details `. To start a specific service (rasdaman, rasfed, or :ref:`embedded petascope `) the ``--service (core | rasfed | petascope)`` option can be used(``core`` refers to ``rasmgr`` + ``rasserver`` only). Since v10.0 the rasmgr port can be specified with ``-p, --port``. Additionally, for security and usability reasons, ``start_rasdaman.sh`` will refuse running if executed with root user; this can be overriden if needed with the ``--allow-root`` option. The script will use various environment variables, if they are set before it is executed: - ``RASMGR_PORT`` - the port on which rasmgr will listen when started, and to which client applications will connect in order to send queries to rasdaman. This variable will be overrided by the value of option ``--port``, if specified. By default if none are specified, the port is set to 7001. - ``RASLOGIN`` - rasdaman admin credentials which will be used for starting rasmgr non-interactively. See more details on the format and how is this setting used :ref:`here `. If not set, the script defaults to using rasadmin/rasadmin credentials; see :ref:`here ` on how to change these defaults. - ``JAVA_OPTS`` - options passed on to the ``java`` command when used to start the OGC frontend of rasdaman (petascope) if it is configured for :ref:`embedded deployment `. If not set, it defaults to ``-Xmx4000m`` Check ``-h, --help`` for all details. .. _executables-stop-rasdaman: stop_rasdaman.sh ---------------- This script stops rasdaman. Normally rasdaman is installed from packages, and instead of executing this script directly one would execute ``service rasdaman stop``. Any options to be passed on to ``stop_rasdaman.sh`` can be set in ``/etc/default/rasdaman`` in this case; see :ref:`more details `. The script stops rasmgr, rasservers, rasfed, and petascope (if configured for embedded deployment) in the correct order with a regular TERM signal to each process; this ensures that the services exit properly. In some cases, a process may be hanging instead of exiting on the TERM signal; since rasdaman v10.0, ``stop_rasdaman.sh`` will detect and report such cases. It is prudent to then check the relevant process logs, and if it appears that there is no reason for the process hanging one can force-stop it with ``stop_rasdaman.sh --force``, or manually do it by sending it a KILL signal (e.g. ``kill -KILL ``). To stop a specific service the ``--service (core | rasfed | petascope )`` option can be used. Since v10.0 the rasmgr port can be specified with ``-p, --port``. The script will use various environment variables, if they are set before it is executed: - ``RASMGR_PORT`` - the port on which rasmgr was set to listen when it was started. This variable will be overrided by the value of option ``--port``, if specified. By default if none are specified, the port is set to 7001. - ``RASLOGIN`` - rasdaman admin credentials which will be used for stopping rasmgr non-interactively. See more details on the format and how is this setting used :ref:`here `. If not set, the script defaults to using rasadmin/rasadmin credentials; see :ref:`here ` on how to change these defaults. Check ``-h, --help`` for all details. .. _executables-migrate-petascopedb: migrate_petascopedb.sh ---------------------- This script is used to migrate coverages imported by wcst_import, OWS Service metadata and WMS 1.3 layers. For more details see :ref:`petascope-database-connection` and :ref:`sec-initialize-geo-services`. There are 2 types of migration: 1. Migrate petascopedb v9.4 or older to a newer rasdaman version. After the migration, the old petascopedb is backed up at petascope_94_backup. 2. Migrate petascopedb v9.5 or newer to a different database name or different database (e.g. PostgreSQL to HSQLDB). .. note:: The petascope Web application must not be running (e.g in Tomcat) while migrating to a different database (type 2 above) to protect the existing data integrity. Configuration files =================== Configurations are automatically loaded upon rasdaman start. After any modification a restarthas to be performed for the change to take effect. Server rasdaman configuration files can be found in ``$RMANHOME/etc``: +------------------------------+------------------------------------------------+ |``rasmgr.conf`` |allows fine-tunning the rasdaman servers, e.g. | | |number of servers, names, database connection | +------------------------------+------------------------------------------------+ |``petascope.properties`` |set petascope properties, e.g. backend/rasdaman | | |connection details, CRS resolver URLs, features | +------------------------------+------------------------------------------------+ |``secore.properties`` |secore configuration | +------------------------------+------------------------------------------------+ |``rasfed.properties`` |federation daemon configuration | +------------------------------+------------------------------------------------+ Logging output of petascope and secore is configured in their respective config files, while logging output of rasdaman is controlled via the below configuration files: +---------------------+--------------------------------------------------+ |``log-rasmgr.conf`` | log output of rasmgr | +---------------------+--------------------------------------------------+ |``log-server.conf`` | log output of rasserver worker processes | +---------------------+--------------------------------------------------+ |``log-client.conf`` | log output of client applications, e.g., rasql | +---------------------+--------------------------------------------------+ rasdaman uses the `Easylogging++ `__ library for logging in its C++ components. Log properties can be configured as documented on the `EasyLogging GitHub page `__. The enterprise license file ``rmankey`` is also found in the ``etc`` directory. External, potentially relevant configuration files are: +------------+---------------------------------------------------------+ | postgresql |``/var/lib/pgsql/data/{postgresql.conf,pg_hba.conf}`` or | | | ``/etc/postgresql/9.X/{postgresql.conf,pg_hba.conf}`` | +------------+---------------------------------------------------------+ | tomcat |``/etc/tomcat/``, ``/etc/default/tomcat`` | +------------+---------------------------------------------------------+ .. _sec-log-files: Log files ========= **rasdaman** *rasdaman* server logs are placed in ``$RMANHOME/log/``. The server components feed the following files where ``uid`` represents a unique identifier of the process, and ``pid`` is a Linux process identifier: ``rasserver...log`` ``rasserver`` worker logs: at any time there are several rasservers running (depending on the settings in ``rasmgr.conf``) and each has a unique log file. ``rasmgr..log`` ``rasmgr`` log: there is only one ``rasmgr`` process running at any time. ``rasfed.log`` ``rasfed`` log: there is only one ``rasfed`` process running at any time; on rasdaman restart the output from the new process is appended to the same log file. ``watch_rasdaman.sh.log`` Log from the ``watch_rasdaman.sh`` script is appended to this file whenever it is executed. .. note:: ``ls -ltr`` is a useful command to see the most recently modified log files at the bottom when debugging recently executed queries. **petascope & secore** It is highly recommended to set a specific log file however in the log4j configuration section in ``petascope.properties`` (e.g. ``log4j.appender.rollingFile.File=/var/log/tomcatN/petascope.log``). Be careful that this location needs to be write accessible by the Tomcat user. The same can be set for SECORE in ``secore.properties``. .. _sec-temporary-files: Temporary files =============== Rasdaman stores various data temporarily in ``/tmp/rasdaman\_*`` directories, in particular: - ``/tmp/rasdaman\_conversion/`` - format-encoded data, such as TIFF, NetCDF, etc., is in some cases temporarily stored here before decoding into rasdaman. This also happens always when encoding query processing results into some format for export. The intermediate data is quickly removed as soon as the encoding or decoding process is finished. Temporarily, however, this directory can get rather large: if you export array result that encodes into a 1GB TIFF file, then the directory will contain 1GB of data for some time; if 10 such queries run concurrently, then it may contain up to 10GB of data. For this reason we recommend to check the size of ``/tmp`` during installation, and make sure it is large enough. It is always recommended to make ``/tmp`` a separate partition, so as to prevent system-wide problems in case the filesystem is filled up with data. - ``/tmp/rasdaman\_petascope/`` - contains small temporary files generated during data import with the wcst\_import tool. - ``/tmp/rasdaman\_transaction\_locks/`` - during query read/write transaction, rasdaman generates various empty lock files in this directory. As the files are empty, the size of this directory is minimal. While rasdaman is running this directory must not be removed, otherwise it may lead to data corruption. Demo data & programs ==================== Example database ---------------- A demonstration database is provided as part of the delivery package which contains the collections and images described in the :ref:`ql-guide`. To populate this database, first install the system as described here, and then invoke: :: $ rasdaman_insertdemo.sh The demo database occupies marginal disk space, and is a straightforward way to show that the rasdaman installation has been successfull. Example programs ---------------- Several example programs are provided in the ``c++`` and ``java`` subdirectories of ``$RMANHOME/share/rasdaman/examples``. Each directory contains a Makefile plus ``.cc`` and ``.java`` sources, resp. Makefile -------- The ``Makefile`` helps to compile and link the sample C++ / Java sources files delivered. It is a good source for hints on the how-tos of compiler and linker flags. .. note:: All programs, once compiled and linked, print a usage synopsis when invoked without parameter. ``query.cc`` ------------ Sends a hardwired query to a running rasdaman system: .. code-block: rasql select a[0:4,0:4] from mr as a where some_cells( a[8:9,8:9] >= 0 ) In addition, it demonstrates how to work with the result set returned from rasdaman. The query can easily be changed, or made a parameter to the program. ``Query.java`` -------------- Sends the following hardwired query if one is not provided as a parameter: .. code-block: rasql select avg_cells( a ) from mr ``AvgCell.java`` ---------------- This program computes the average cell value from all images of a given collection on client side. Note that it requires grayscale images. A good candidate collection is ``mr`` from the demo database. ***************** Access Interfaces ***************** Rasdaman services can be invoked in several ways: through command line, Web requests, and custom programs connecting via the C++ and Java APIs. Command Line Tools ================== Queries can be submitted to the command line tool ``rasql``. Complete control over the server is provided through several utilities, in particular ``rasmgr``; see :ref:`sec-rascontrol-invocation` for details. All tools can communicate with local and remote rasdaman servers. .. _web-services: Web Services ============ Several Web services are available with rasdaman. They are implemented as servlets, hence independent from the array engine and only available if started in a servlet container such as Tomcat or jetty. They can be accessed under the common context path ``/rasdaman``. * ``/rasdaman/ows`` exposes geo Web Services based on the interface standards of the Open Geospatial Consortium (OGC Web Services, OWS). Supported OGC standards are: - Web Coverage Service (WCS) - Web Coverage Processing Service (WCPS) - Web Map Service (WMS) suites * ``/rasdaman/def`` provides access to a Coordinate Reference System (CRS) Resolver Service, SECORE. It is identical to the one deployed by OGC, where http://www.opengis.net/def/crs is the branch for CRS served by SECORE. * ``/rasdaman/rasql`` provides support for submitting rasql queries and receiving results with standard HTTP requests. Requests must specify three mandatory parameters: +--------------+------------------------------------------------------------+ | ``username`` | rasdaman login name under which the query will be executed | +--------------+------------------------------------------------------------+ | ``password`` | password corresponding to the login | +--------------+------------------------------------------------------------+ | ``query`` | rasql query string, properly encoded for URI embedding | +--------------+------------------------------------------------------------+ Example: :: http://localhost:8080/rasdaman/rasql ?username=rasguest &password=rasguest &query=select%20encode%28mr2%2C%22png%22%29%20from%20mr .. NOTE:: rasql servlet also supports rasdaman user credentials in basic authentication header. In this case, ``username`` and ``password`` parameters are not required as the credentials are extracted from the header. The diagram below illustrates the OGC service architecture of rasdaman: .. hidden-code-block:: text clients read: read: +-----------------+ | | GetCapabilities select ... | +-----------+ | DescribeCoverage | |3rd party | | | +-----------+ | GetCoverage | | ProcessCoverage | +-----------+ | GetMap | |wcs_client | | +------------+ +---------+ | +-----------+ | +-------------> |rasdaman-geo| +--------> |rasserver| | | +------------+ +---------+ | +-----------+ | write: write: | |wcst_import| | | +-----------+ | InsertCoverage create type/coll | | UpdateCoverage insert,update,delete +-----------------+ DeleteCoverage drop type/coll .. include:: 02_inst-guide-web-admin-tools.inc .. -- rasdaman enterprise begin .. _sec-basic-header-auth: Basic header authentication =========================== For incoming requests, rasdaman-geo requires credentials attached in HTTP headers when ``authentication_type=basic_header`` is set in ``petascope.properties``. The valid format of the credentials must be: :: Authorization: Basic encode_in_base64(username:password) For example, if username is `admin` and password is `admin` and the client is ``curl``, one need to construct this request: :: curl -H "Authorization: Basic YWRtaW46YWRtaW4=" \ "http://localhost:8080/rasdaman/ows? service=WCS&version=2.0.1&request=DeleteCoverage&coverageId=test_coverage" or simpler with the ``--user`` option: :: curl --user "rasadmin:rasadmin" \ "http://localhost:8080/rasdaman/ows? service=WCS&version=2.0.1&request=DeleteCoverage&coverageId=test_coverage" .. note:: If basic authentication is not enabled in rasdaman, depending on the request, rasdaman uses one of the users configured by ``rasdaman_user`` and ``rasdaman_admin_user`` settings in ``petascope.properties`` appropriately to run a rasql query. .. note:: Using ``wcst_import`` to insert/update coverages to petascope will require to specify credentials when running script ``wcst_import.sh``. Check ``wcst_import.sh -h`` for optional parameters ``-i, --identity-file`` or ``-u, --user USER_NAME``. .. note:: WSClient (at ``http://YOUR_SERVER/rasdaman/ows``) will detect if petascope requires authentication to show a login form via this API ``http://YOUR_SERVER/rasdaman/admin/authisactive``. If it is the case, one needs to provide valid credentials to login and WSClient keeps the credentials in *Web Browser's local storage*. After that, any requests to petascope will be added basic authentication in HTTP header implicitly. .. -- rasdaman enterprise end APIs ==== Programmatic access is available through self-programmed code using the C++ and Java interfaces; see the C++ and Java Guide for details. .. _sec-rasdaman-architecture: ******************* Server Architecture ******************* The parallel server architecture of rasdaman offers a scalable, distributed environment to efficiently process even very large numbers of concurrent client requests. Yet, server administration is easy to accomplish, with only few things to do to have a smoothly running, highly performant installation. Moreover, the system is implemented in a special high availability technique where most server management operations can be done with the server up and running, limiting the need for a server shutdown to the absolute minimum. In this Section the general rasdaman server architecture is outlined. It is recommended to study this section so as to understand server administration terminology used in the next Section. Executables Overview ==================== The following executables are provided in the ``bin/`` directory, among others: * ``rasmgr`` is the central rasdaman request dispatcher; * ``rasserver`` is the rasdaman server engine, it should not be generally invoked in a standalone manner; * ``rascontrol`` allows to interactively control the rasdaman server by communicating with ``rasmgr``; * ``rasfed`` is the federation daemon, which enables efficient query query distribution in federated rasdaman networks; * ``rasql`` is the command-line based query tool, explained in detail in the *rasdaman Query Language Guide*. .. _sec-server-mgr-server: Server Manager and Server ========================= Overview and Terminology ------------------------ The rasdaman server configuration consists of one dispatcher process per computer, ``rasmgr`` (we will refer to it as *manager* in the sequel), and server processes, ``rasserver`` (referred to as *servers*), of which at a given time none, one, or several ones can be running. All server processes are under control of the manager. Server manager and rasdaman server(s) all run on the same physical hardware, the *rasdaman host*. The servers resolve requests, thereby generating calls to the relational database system which in turn accesses its database files. For the purpose of this manual, the relational server together with the database it maintains are collectively called the *database*. The machine the relational database server runs on is referred to as *database host* (:numref:`figure2`). .. _figure2: .. figure:: media/inst-guide/image3.png :align: center :width: 450px Overall server hierarchy, introducing the terminology for rasdaman hardware and software environment Server Structure in General --------------------------- The manager accepts client requests and assigns server instances to them, taking them from the pool of server processes it maintains. In distributed installations, it keeps contact to the managers on other machines to further dispatch client requests across all the rasdaman servers available. Whenever needed, the administrator can launch further server instances, or shut them down again. Upon system configuration definition (see :ref:`sec-rascontrol-invocation`), a unique name is assigned to each server identifying it to the manager. Each rasdaman server is assigned to a relational database server, laid down in the manager configuration file. Databases can be registered and associated to particular rasdaman servers at any time. rasdaman hosts and database hosts are identified by their resp. host name in common domain address form, e.g., ``martini.rasdaman.com`` or ``199.198.197.50``. ``Rascontrol`` is the interactive front-end to ``rasmgr`` and, as such, the main utility for user and system management. It provides the necessary functions to manage the whole system configuration, to add and remove user, to change their rights, and to obtain information about system activity. The rasdaman server, i.e., ``rasserver``, is controlled by the manager which starts and stops server instances. Hence, the ``rasserver`` executable should not (and actually cannot) be invoked directly. Dynamic Server Assignment ------------------------- The process of client/server communication and server scheduling is done as follows (see numbers in :numref:`figure-internal-server-mgmt`). 1. The client starts every ``OPENDB`` and ``BEGIN TRANSACTION`` request with an HTTP call to the manager, providing the required service type (RPC, HTTP, etc.) and the database name, together with user name and password. 2. The manager's answer is the server ID of a free server, or an error message in case no server is available or access is denied for the given login. 3. Client-Server communication to perform the database requests. 4. Upon ``CLOSEDB`` and ``ABORT/COMMIT TRANSACTION`` the server informs the manager that it is available again. This is also done upon a client timeout. These negotiation steps are performed between client library and server, hence transparent to the application. The rasdaman server system is started by invoking the server manager ``rasmgr`` (see :ref:`sec-running-manager`). If it finds a configuration file, them autopmatically all servers indicated will be started; alternatively, server configuration can be done directly through ``rascontrol`` (see :ref:`sec-rascontrol-invocation`). .. _figure-internal-server-mgmt: .. figure:: media/inst-guide/image4.png :align: center :width: 450px Internal server management System Start-up --------------- Invocation of the ``rasmgr`` executable must be done under the operating system login under which the rasdaman installation has been done, usually (and recommended) ``rasdaman``. The service script ``/etc/init.d/rasdaman`` (when rasdaman is installed from the packages) automatically takes care of this. Authentication -------------- On every machine hosting rasdaman servers a separate manager has to run. The manager maintains an authorization file, ``$RMANHOME/etc/rasmgr.auth``. It should not be changed by the administrator, as they are generated, maintained, and overwritten by the manager. .. _figure4: .. figure:: media/inst-guide/image5.png :align: center :width: 500px rasdaman federation rasdaman Manager Defaults ------------------------- The manager's default name is the ``hostname`` (the one reported by the UNIX command hostname), but it can be changed (see the ``change`` command). By default, it listens to port 7001 for incoming requests and uses port 7001 for outgoing requests: Port Number Recommendations --------------------------- To keep overview of the ports used, it is recommended to use the following schema (there is, however, no restriction preventing from choosing another schema - just keep an overview\...): - use port number 7001 for the server manager; - use port numbers 7002 to 7999 for rasdaman servers. .. _sec-storage-backend: Storage backend =============== rasdaman can store array data in two different ways: 1. Arrays in a file system directory, array metadata in SQLite; this is default. 2. Everything in PostgreSQL: arrays in BLOBs, array metadata in tables. .. note:: rasdaman enterprise additionally supports access to pre-existing archives of any structure. The array storage variant is fixed in the packages to ``sqlite``, i.e. the default recommended option. .. _sec-filesystem-backend: Storing arrays in a file system directory ----------------------------------------- In this storage variant, a particular directory gets designated to hold rasdaman arrays (maintained by rasdaman) and their metadata (maintained by an SQLite instance embedded in rasdaman). The recommended directory location is ``$RMANHOME/data/``; administrators may configure this to be a symbolic link to some other location, possibly another filesystem than where ``$RMANHOME`` resides (so as to keep programs and data separate). Alternatively, the path can be changed in the ``-connect`` option in ``rasmgr.conf``. The data directory will contain the named database. Currently only one database is supported, but this may change in future. Default database name, assumed by all tools, is ``RASBASE``. While it can be changed this is not recommended as all tools will need to receive an extra parameter indicating the changed name. The database name needs to be communicated to rasdaman in the ``$RMANHOME/etc/rasmgr.conf`` configuration file. Specifically, the connect string should be an absolute path to the ``RASBASE`` database (note that variables are not recognized in the script, therefore ``$RMANHOME`` has to be spelt out). Assuming the default values described above and a rasdaman installation directory of ``$RMANHOME=/opt/rasdaman``, the corresponding configuration line might look like this: :: define dbh rasdaman_host -connect /opt/rasdaman/data/RASBASE .. caution:: For a customized data directory location it is recommended to use a symbolic link, rather than modify installation defaults. .. include:: 02_inst-guide-caching.inc .. _sec-server-administration: ********************* Server Administration ********************* This Section explains on how to manage a rasdaman service on a *lower level*: start up and shut down individual server workers, as well as how to monitor and influence server state. It is recommended to first study the previous section so as to understand server administration terminology used here. General Procedure ================= ``rasmgr`` vs. ``rascontrol`` ----------------------------- It is important to distinguish between the manager, ``rasmgr``, and its control front-end, ``rascontrol``. The manager runs as a background process, supervising activity of local (and possibly remote) rasdaman servers. Interaction between user (i.e., administrator) and the manager takes place through the interactive control front end. In the sequel, it is first described how to launch the manager ``rasmgr``, then ``rascontrol`` commands are detailed. Important Security Note ----------------------- To remain compatible with older rasdaman versions, clients use login "rasguest" / password "rasguest" by default (i.e., when no user and password are explicitly set by the application). In the distribution configuration, this user is defined to have read-only access to the databases, so that users can access but not manipulate databases without authentication. Therefore, the administrator is strongly urged to adapt authentication settings to the local security policy before switching databases online. See :ref:`sec-users-rights` to learn more about user management mechanisms. .. _sec-running-manager: Running the Manager =================== Manager Startup --------------- Starting up the rasdaman system is done by invoking the rasdaman manager, ``rasmgr``, from a shell under the ``rasdaman`` operating system login. Usually the manager will be sent to the background: :: $ rasmgr & Starting ``rasmgr`` is the only direct action to be done on it. Any further administration is performed using ``rascontrol``. Note that, unless a server configuration has been defined already, no rasdaman server is available just by starting the manager. Invocation Synopsis ------------------- Manager invocation synopsis: :: $ rasmgr [--help] [--hostname h] [--port p] where --help print this help --hostname h host on which the manager process is running is accessible under name / IP address *h* (default: output of Unix command hostname) --port p manager will listen to port number *p* (default: 7001) Examples -------- To start a manager which will listen at port 7001: :: $ rasmgr --port 7001 .. _sec-rascontrol-invocation: ``rascontrol`` Invocation ========================= The manager front end, rascontrol, is a command-line interface used for rasdaman administration. It allows to define the whole rasdaman system configuration, including start up and shut down of server instances and user logins and rights. To secure access to the server administration facilities, rascontrol performs a login process requesting login name and password similar to the Unix rlogin command. User name must be one of the users defined in the rasdaman authentication list (see :ref:`sec-users-rights`). ``rascontrol`` Synopsis ----------------------- :: $ rascontrol [-h|--help] [--host *h*] [--port *n*] [--prompt *n*] [--quiet] [--login|--interactive|--execute *cmd*|--testlogin] where --host h name of the host where the manager runs (default: localhost) -h, --help this help --port n port number at which the manager listens to requests (default: 7001) --prompt n change rascontrol prompt as follows: - ``0`` - prompt '``>``' - ``1`` - prompt '``rasc>``' - ``2`` - prompt '``user:host>``' (default: 2) --quiet quiet, don't print header (default for ``--login`` and ``--testlogin``) --login print login and password, obtained from interactive input, to ``stdout``, then exit (see *Script Use* below) --interactive read login and password from environment variable ``RASLOGIN`` instead of requesting it interactively --execute cmd execute single ``*cmd*`` and exit (batch mode); all text following ``-x`` until end of line is passed as ``command``; this option implicitly assumes ``-e`` --testlogin just do a login and nothing else to check whether the login/password combination provided in the ``RASLOGIN`` variable is valid Interactive Use --------------- In interactive use, ``rascontrol`` will be invoked with the host parameter only. Following successful authentication, ``rascontrol`` accepts command line input from ``stdin``. Here is an example session (``mypasswd`` will not be echoed on screen): .. hidden-code-block:: bash $ rascontrol Login name: *mylogin* Password: *mypasswd* mylogin:localhost> define dbh h1 -connect / mylogin:localhost> define db d1 -dbh h1 mylogin:localhost> define srv s1 -host localhost -type h -dbh h1 mylogin:localhost> up srv s1 mylogin:localhost> save mylogin:localhost> exit $ .. _sec-rascontrol-script-use: Script Use ---------- Alternatively to interactive login, user and password information can be taken from the environment variable ``RASLOGIN``. This variant is suitable for batch scripting in conjunction with the ``-x`` option. The following example shows how first the ``RASLOGIN`` is set appropriately: :: $ export RASLOGIN=`rascontrol --login` \...and then a sample Unix shell script which starts all rasdaman servers defined in the system configuration, performing implicit login from the environment variable contents which has been obtained from the previous command and pasted into the shell script: :: #!/bin/bash export RASLOGIN=rasadmin:mytotallyencryptedpassword rascontrol -x up srv -all Comments in Scripts ------------------- To enhance legibility of scripts, ``rascontrol`` accepts comments in the usual shell syntax: Lines beginning with a hash sign '#' will be ignored, whatever they may contain. An example is usage in shell *here documents* (type ``man sh`` in your favourite shell for further information on this feature): :: $ rascontrol <``0``: restart rasdaman server after c requests for *c*\ =``0``: run rasdaman server indefinitely (default: *c*\ =``1000``) ``-xp options`` pass option string *options* to server upon start (default: no options, i.e., empty string) Option ``-xp`` must be the last option. Everything following "-xp" until end of line is considered to be "\ *options*\ " and will be passed, at startup time, to the server. .. TODO: invalid reference .. ; see :ref:`sec-server-control-options` below for the list of options available. Change Server Settings ---------------------- :: change srv s [-name n] -type t [-port p] [-dbh d] [-autorestart [on|off] [-countdown c] [-xp options] ``s`` change settings for server *s* ``-name n`` change server name to *n* ``-port p`` change port number to *p* ``-dbh d`` new database host where the relational database server runs to which the rasdaman server connects ``-autorestart a`` for *a*\ =on: automatically restart rasdaman server after unanticipated termination for *a*\ =off: don't restart ``-countdown c`` for *c*>0: restart rasdaman server after c requests for *c*\ =0: run rasdaman server indefinitely ``-xp options`` pass option string *options* to server upon start Option ``-xp`` must be the last option. Everything following "-xp" until end of line is considered to be "\ *options*\ " and will be passed, at startup time, to the server. .. TODO: invalid reference .. see Section :ref:`sec-server-control-options` below for the list of options available. Restrictions: - The server host cannot be changed. - The server name cannot be changed while the server is up. - The new settings will be used only next time the server starts. Remove rasdaman Server Definitions ---------------------------------- :: remove srv s ``s`` server name whose entry is to be deleted Remove server *s* from the definition table. It is not possible to remove a server definition while the corresponding server is up and running Status Information ------------------ ``list srv [ s | -host h | -all ] [-p]`` ``s`` give information about server *s* ``-host h`` give information about all servers running on host *h* information is requested ``-all`` list information about all servers on all hosts (default) ``-p`` additionally list configuration information The first is variant prints status information of the currently defined server(s); if *s* is provided, then only server s is listed. Database Hosts ============== Define Database Hosts --------------------- :: define dbh h [-connect c] ``h`` a unique symbolic database host name, usually the host machine name ``-connect c`` the connection string used to connect ``rasserver`` to the database server ``-user u`` the user name (optional) used to connect ``rasserver`` to the base DBMS server; for PostrgreSQL, using this parameter automatically implies trust authentication. ``-passwd p`` the password (optional) used to connect ``rasserver`` to the base DBMS server; for PostrgreSQL, using this parameter automatically implies trust authentication. Change Database Host Settings ----------------------------- :: change dbh h [-name n] [-connect c] ``h`` database host whose entry is to be changed ``-name n`` change symbolic database host name to *n* ``-connect c`` change connect string to *c* ``-user u`` the user name used to connect ``rasserver`` to the base DBMS server; using this optional parameter automatically implies ident-based authentication. ``-passwd p`` the password used to connect ``rasserver`` to the base DBMS server; using this optional parameter automatically implies ident-based authentication. The connection parameters can be changed at any time, however the servers will get the information only when they are restarted. Remove Database Host Definitions -------------------------------- ``remove dbh h`` ``h`` database host name whose entry is to be deleted Remove database host *h* from the definition table. It is not possible to remove a database host definition while this database host has active servers connected to it. Status Information ------------------ ``list dbh`` List all relational database hosts currently defined. Databases ========= Databases represent the physical database itself, together with the relational database server accessing them. It is possible to have multiple database definitions in the rasdaman server environment which are distinguished by the database host; the interpretation, then, is that the same contents (be it the same physical database or a mirrored copy) is available through relational servers running on the different hosts mentioned. In other words, when a client opens a database, the server manager can freely choose any of the database hosts on which the database indicated is defined. The pair (database,database host) must be unique. Define Databases ---------------- ``define db d -dbh db`` ``d`` define database with name *d* ``-dbh db`` set database host name to *db* Change Database Settings ------------------------ ``change db d -name n`` ``d`` database whose name is to be changed ``-name n`` change to new database name *n* Remove Database Definitions --------------------------- ``remove db d -dbh db`` ``d`` name of database to be removed ``-dbh db`` host name of database to be removed Remove definition of database *d* from the definition table. The database itself remains unchanged, it is not physically deleted. It is not possible to remove a database definition while the corresponding database has open transactions. Status Information ------------------ ``list db [ d | -dbh h | -all ]`` ``d`` give information about servers connected to database *d* ``-dbh h`` give information about all servers connected to database *d* via database host *h* ``-all`` list information about all servers connected to any known database (default) List relational database(s) defined. .. include:: 02_inst-guide-cache-control.inc Server Start-up and Shutdown ============================ **Server Start** ``up srv [ s | -host h | -all ]`` ``s`` start only server *s* ``-host s`` start all servers on host *h*; this requires that a manager has been started on this host previously. ``-all`` start all servers defined; note that only those servers can be started on whose host a manager is currently running. Look up the named server(s) in the definition list, and start the specified one(s) using the previously defined individual startup parameters. At least one of the options *s*, -host *s*, and -all must be present. **Server Shutdown** ``down srv [ s | -host h | -all ] [-force] [-kill]`` ``s`` name of the server to be stopped ``-host s`` terminate all servers on host *h* ``-all`` terminate all servers ``-force`` *send SIGTERM* immediately, don't wait for transaction end ``-kill`` *send SIGKILL* immediately, don't wait for transaction end This command shuts down the indicated server(s). At least one of the options *s*, -host *s*, and -all must be present. Without ``-force`` and ``-kill``, the server is marked for shut down and will actually be terminated by sending ``SIGTERM`` after completing the current transaction. With ``-force`` and ``-kill``, the server is terminated instantaneously; this should be handled with extreme caution, as experience shows that relational database systems react differently on such a situation: usually a running transaction is aborted (which is the desired behavior), but sometimes the running transaction is committed (most likely leaving the database in an inconsistent state). See a Unix manual for the difference between ``SIGTERM`` and ``SIGKILL`` signals. The manager on host *h* is not terminated. .. _sec-users-rights: Users and Their Rights ====================== See :ref:`accesscontrol`. Server Control Options ====================== The following options can be passed to the server when it is started by the server manager using the ``up srv`` command. Option settings are defined for a particular server using the ``rascontrol`` command ``change srv -xp`` which passes the rest of the line after ``-xp`` on to the server upon starting it (see :ref:`sec-rasdaman-servers`). --log logfile print log to *logfile.* If *logfile* is stdout, then log output will be printed to standard output. (default: ``$RMANHOME/log/rasserver``.\ *serverid.serverpid*.log) --timeout t client time out in seconds for sign-of-life signal. If no t indicated: 300 sec; if set to 0, no sign-of-life check is done. --transbuffer b set maximum size of transfer buffer to *b* bytes (default: 10 MB = 10000000 bytes) --cachelimit c specifies upper limit in bytes on using memory for caching (default: 0) --enable-tilelocking perform tile-level locking on insert / update / delete (default: whole database is locked) Distributed Query Processing ============================ Rasdaman can form a federation network for query answering. In such a setup, ``rasmgrs`` facing congestion (i.e., all ``rasserver`` worker processes busy) will try to acquire a free server from some other ``rasmgr``'s holding in the federation. Session-based server assignment ------------------------------- As always in rasdaman, acquisition and release of server processes is done on session level: when a client opens a new connection, it gets a server assigned; when it closes the connection, this server is released and put back into the pool of available processes. Hence, for optimal load balance clients should strive to have short-running sessions and not keep open connections unduly for a long time. Federation network ------------------ The federation network is defined in a decentralized way: each ``rasmgr`` knows peers from which it accepts requests, and to which it can send requests. To this end, each ``rasmgr`` maintains an ``inpeer`` and ``outpeer`` list: - The ``inpeer`` list contains those hosts from which this node's ``rasmgr`` will accept requests. - The ``outpeer`` list contains those hosts which this node's ``rasmgr`` will ask for server processes on local session overflow. By manipulating these two lists administrators can exercise fine-grain security policy in a rasdaman federation network. Note that the federation connectivity graph is not necessarily symmetric: a ``rasmgr`` may send requests to some other ``rasmgr``, but not accept requests, and vice versa, depending on the individual configuration. Each host individually respects these statements, there is no global rasdaman federation configuration. Federation node addressing -------------------------- Addressing is based on hostnames, where a hostname in the sequel is one of - a domain name, resolvable by this ``rasmgr``'s host - an IP address All ``inpeer`` and ``outpeer`` statements accumulate so that host identifiers can be added and removed incrementally. Security -------- A ``rasmgr`` request for a server process on another host is treated by the incoming host in the same way as any such incoming client request. The requesting ``rasmgr`` authenticates via the login and password which the originating client used for authenticating against rasdaman in the first place. This implies that a client approaching such a federation must be known in all federation nodes. See :ref:`sec-users-rights` for details on users and the various permissions they can have on a database. If neither any ``inpeer`` nor any ``outpeer`` is defined (either interactively through ``rascontrol`` or by way of settings in ``rasmgr.conf``) then this rasdaman instance will act completely standalone and will neither send nor accept peer requests. Define peers ------------ :: define inpeer hostname ``hostname`` host from which requests for rasdaman server process assignment will be accepted by this rasmgr :: define outpeer hostname [-port portnumber] ``hostname`` host from which this rasmgr may request a rasdaman server process ``portnumber`` port number at which the rasfed on that host is listening (default: 7000) List peers ---------- :: list inpeer list outpeer These commands list all currently defined inpeers and outpeers, respectively. Remove peers ------------ :: remove inpeer hostname remove outpeer hostname These commands remove hostname *hostname* listed from the list of peers. Examples -------- :: define inpeer www.acme.com define inpeer 192.168.28.10 Caveat: fluctuating IPs ----------------------- In cloud environments, IP addresses are maintained dynamically and can change for a given host between reboots. Hence, when growing a rasdaman federation by launching new VMs care must be taken that the in- and outpeers received the proper current IP address. Restrictions ------------ In the current version, the queries are distributed only if the receiving rasmgr has no locally assigned rasservers. This limitation will be removed in the next release. Miscellaneous ============= Help ---- :: help Display top level help page :: help [command] command help Display information specific to *command* (both syntax variants are equivalent) Version Information ------------------- :: list version ``version`` display rasdaman server version. Save Changes to Disk -------------------- :: save The ``save`` operation writes the current configuration and authorization values to disk. All changes done during the session thus become permanent. ``rascontrol`` Termination -------------------------- :: exit terminates ``rascontrol``. .. include:: 02_inst-guide-federation.inc .. _rasdaman-security: ******** Security ******** General ======= There are several security measures available, which should be considered seriously. Among them are the access right mechanisms found in Tomcat, rasdaman, and PostgreSQL. We highly recommend to make use of these. For Tomcat and PostgreSQL refer to the pertaining documentation. The servlet is safe against SQL injection attacks - we are not aware of any means for the user to send custom queries to the PostgreSQL server or the rasdaman server. XSRF and XSS represent no danger to the service because there is no user generated content available. For rasdaman, we recommend to change the default user passwords in rasdaman (rasguest/rasguest for read-only access, rasadmin/rasadmin for read-write and administrator access) to not run into the Oracle "Scott/tiger" trap. Even better, use additional separate, private users. The rasdaman service doesn't use cookies. .. _sec-security-reset-passwords: Reset default passwords ======================= This is a quick guide to reset the default rasdaman users, rasguest and rasadmin, which are created when the system is first installed and initialized. Pre-requisites: rasdaman must be up and running. 1. Reset rasguest - Update password in rasdaman: :: $ rasql -q 'ALTER USER rasguest SET PASSWORD TO "newpassword"' \ --user rasadmin --passwd rasadmin - Update ``rasdaman_pass`` (for read-only access from petascope) in ``/opt/rasdaman/etc/petascope.properties`` to set it to the newpassword. - Update ``rasdamanPassword`` in ``/opt/rasdaman/etc/rasfed.properties`` to set it to the newpassword. - Grant ``PRIV_OWS_WCS_PROCESS_COV`` role so that the rasguest user will be able to run WCPS queries: :: $ rasql -q 'GRANT PRIV_OWS_WCS_PROCESS_COV to rasguest' \ --user rasadmin --passwd rasadmin 2. Reset rasadmin - Update password in rasdaman: :: $ rasql -q 'ALTER USER rasadmin SET PASSWORD TO "newpassword"' \ --user rasadmin --passwd rasadmin - Update ``rasdaman_admin_pass`` (for read-write access from petascope) in ``/opt/rasdaman/etc/petascope.properties`` to set it to the newpassword. - Update RASLOGIN env variable in ``/etc/default/rasdaman``: :: # get an md5 hash of the new password $ echo -n "newpassword" | md5sum | awk '{ print $1; }' # update the previous md5 hash of RASLOGIN by editing the file $ nano /etc/default/rasdaman # login credentials for non-interactive rasdaman start/stop RASLOGIN=rasadmin: 3. Restart rasdaman: :: sudo systemctl restart rasdaman 4. Restart tomcat (if external tomcat is configured in ``petascope.properties``): :: tomcat_service=$(systemctl list-units | grep -i tomcat) sudo systemctl $tomcat_service restart .. _rasdaman-backup: ****** Backup ****** Both software and hardware can fail, therefore it is prudent to establish regular backup procedures. For rasdaman in particular a couple of things should be considered for inclusion in a backup: 1. The rasdaman database, which normally can be found in ``/opt/rasdaman/data``. The SQL database itself in this directory, ``RASBASE``, is fairly small; the ``TILES`` subdirectory may be large as it contains all the array data, but if backup disk space is not scarce then it is definitely recommended to backup as well. Incremental backups of the ``TILES`` with rsync for example should work well without unnecessary duplicated data copying, unless existing data areas are often updated. Example with rsync: .. code-block:: shell # backup small RASBASE to /backup/rasdamandb rsync -avz /opt/rasdaman/data/RASBASE /backup/rasdamandb/ # backup potentially large TILES dir to /backup/rasdamandb rsync -avz /opt/rasdaman/data/TILES /backup/rasdamandb/ 2. The petascopedb geo metadata database is usually small and worth backing up. By default it is stored in PostgreSQL and can be extracted into a small compressed archive as follows: .. code-block:: shell # create backup in a gzip archive petascopedb.sql.gz sudo -u postgres pg_dump petascopedb | gzip > /backup/petascopedb.sql.gz If necessary, it can be restored with .. code-block:: shell # if a petascopedb already exists it needs to be renamed, as otherwise # restoring over an existing petascopedb will corrupt it sudo -u postgres psql -c "ALTER DATABASE petascopedb RENAME TO petascopedb_existing_backup" # create an empty petascopedb sudo -u postgres createdb petascopedb # restore backup petascopedb.sql.gz (use cat if it's not a gzip archive) zcat petascopedb.sql.gz | sudo -u postgres psql -d petascopedb --quiet > /dev/null Alternatively, if the above fails for some reason, petascopedb can be backup with this command: .. code-block:: shell # note that /backup/petascopedb_backup will contain a large number of compressed files sudo -u postgres pg_dump -j 8 -Fd petascopedb -f /backup/petascopedb_backup If necessary, it can be restored with .. code-block:: shell sudo -u postgres pg_restore -j 8 -d petascopedb /backup/petascopedb_backup 3. The rasdaman configuration files in ``/opt/rasdaman/etc``, but also consider the ``bin`` and ``share`` directories which may be useful in case of package update problems, as well as maybe log files in the ``log`` directory. .. code-block:: shell # backup everything except the data dir, which is handled in step 1. above rsync -avz --exclude='data/' /opt/rasdaman /backup/ .. _rasdaman-migration: ********* Migration ********* Sometimes it is necessary to migrate the installation from one machine (*OLD*) to another (*NEW*). This section outlines the steps on how to do this. 1. Make sure rasdaman is installed and functional on the NEW machine. 2. Stop rasdaman and an external tomcat if installed on both the OLD and NEW machine, e.g: .. code-block:: shell sudo service rasdaman stop sudo service tomcat9 stop 3. Make a backup of the rasdaman and petascope databases on the OLD machine by following step 1. of the :ref:`backup guide ` and copy the backup to the NEW machine. 4. Restore the database backups on the NEW machine by following step 2. of the :ref:`backup guide `. 5. Make a backup of the config files on the NEW machine (``/opt/rasdaman/etc``), then copy relevant configuration from the OLD to the NEW machine, in particular: - ``rasmgr.conf`` can probably copied as is, but check if the -host setting is correct for the NEW machine; - most settings from ``petascope.properties`` can be copied as is, except the ones for database configuration (spring.* and metadata*); - if federation is enabled on the OLD machine then most settings from ``rasfed.properties`` can be copied, but carefully check the hostname setting; - ``/etc/default/rasdaman`` can be copied as is usually; 6. If UDFs have been registered on the OLD machine: - if the NEW machine is the same OS and CPU architecture as the OLD then the UDF libraries can probably be synced by copying the ``/opt/rasdaman/share/rasdaman/udf`` directory; - otherwise, they will likely need to be recompiled on the NEW machine; 7. Make sure that the ``/opt/rasdaman`` directory is owned by the ``rasdaman`` user, to avoid any permission issues caused by copying with other system users: .. code-block:: shell sudo chown -R rasdaman: /opt/rasdaman 8. Finally start rasdaman and tomcat: .. code-block:: shell sudo service rasdaman start sudo service tomcat9 start ************** Uninstallation ************** When uninstalling rasdaman, you can execute the following commands to ensure that all installed files and services are fully removed from the system. .. hidden-code-block:: bash # Stop rasdaman, disable service, and clear from failed list $ sudo systemctl stop rasdaman $ sudo systemctl disable rasdaman $ sudo systemctl reset-failed # Removes everything related to rasdaman (except dependencies) # on Ubuntu/Debian systems; for CentOS use `yum erase rasdaman` $ sudo apt-get purge rasdaman # Remove the rasdaman installation directory # WARNING: This removes all rasdaman data and configuration files! $ sudo rm -r /opt/rasdaman # Remove systemd service, init script, profile script $ sudo rm /etc/systemd/system/rasdaman.service $ sudo rm /etc/init.d/rasdaman $ sudo rm /etc/profile.d/rasdaman.sh # Remove rasdaman apt repository on Ubuntu/Debian $ sudo rm /etc/apt/sources.list.d/rasdaman.list # Remove dependencies of the rasdaman package which are no longer # needed by other packages; it will remove postgres if no other # package depends on it (Debian / Ubuntu) $ sudo apt-get autoremove *************** Troubleshooting *************** General ======= The first step in troubleshooting problems should be to look into the :ref:`server logs `. Start with checking the ``rasmgr`` and ``rasserver`` logs for any errors. If this does not provide any clues, check the ``petascope.log`` or ``catalina.out``. Next, investigate the status of rasdaman and external Tomcat if applicable with ``systemctl rasdaman status`` (and similar for Tomcat). Inspect the output of ``ps aux | grep ras`` to list details about the rasdaman processes, or ``top`` for CPU and memory usage. It can be useful to double check the system memory usage with ``free -m``, and disk space usage with ``df -h``. Manually stop rasdaman ====================== If stopping rasdaman fails, it may be necessary to manually stop it: .. hidden-code-block:: bash # check the rasdaman processes still running on the system $ ps aux | grep ras # force kill any rasmgr process; is the number in the 2nd column # of the output from the previous command $ kill -9 # then try to kill any remaining rasserver processes $ pkill -f rasserver # if this fails (check with ps aux), force kill rasservers $ pkill -9 -f rasserver Checking the server logs could provide further information on why stopping rasdaman failed in the first place.