.. highlight:: bash .. _inst-guide: ##################################### Installation and Administration Guide ##################################### ******* Preface ******* Overview ======== This guide provides information about how to use the rasdaman array database system, in particular: installation and system administration. For storage of multi-dimensional array data, rasdaman can be configured to use some conventional database system (such as PostgreSQL) or use its own storage manager. For the purpose of this documentation, we will call the conventional database system to which rasdaman is interfaced the *base DBMS*, understanding that this base DBMS is in charge of all alphanumeric data maintained as relational tables or object-oriented semantic nets. This guide is specific for *rasdaman enterprise*. Audience ======== The information in this manual is intended primarily for database and system administrators. Rasdaman Documentation Set ========================== This manual should be read in conjunction with the complete rasdaman documentation set which this guide is part of. The documentation set in its completeness covers all important information needed to work with the rasdaman system, such as programming and query access to databases, guidance to utilities such as *raswct*, release notes, and additional information on the rasdaman wiki. .. _sec-download-and-install: ************ Installation ************ This page describes installation of rasdaman enterprise Debian or RPM packages. With your purchase, you have received a login to the rasdaman download area, which will be necessary to setup the installation and updating of rasdaman packages. **Hardware & Software Requirements** It is recommended to have at least 8 GB main memory. Disk space depends on the size of the databases, as well as the requirements of the base DBMS of rasdaman chosen. The footprint of the rasdaman installation itself is around 400 MB. Rasdaman is continuously tested on the platforms listed below. The rasdaman code has been developed on SUN/Solaris and HP-UX originally, and has been ported to IBM AIX, SGI IRIX, and DEC Unix - but that was way back in the last millennium. - Ubuntu 20.04, 22.04, 24.04 - Debian 12 The rasdaman engine in the packages uses embedded SQLite for managing its array metadata. The geo service component, petascope, currently relies on a PostgreSQL database by default, but can be reconfigured with an embedded H2 database instead if desired. **Licence Key** In order to run a rasdaman server you have to obtain a licence from rasdaman GmbH. This licence key encodes, among others, the number of cores and the server's interface name (such as "eth0") and corresponding MAC address. The following commands are usually used to obtain this information: :: # interface name + MAC address $ ip link # alternatively $ ifconfig # number of CPUs $ nproc # alternatively $ cat /proc/cpuinfo After communicating these ingredients to rasdaman GmbH in the course of a licence purchase, a licence key file will be provided which has to be stored on the machine where the rasdaman server runs. Alternatively in case of cloud deployments where the MAC is not fixed, it is possible to do a licence verification through a verification service instead. This requires that the machine has access to the internet, so that it can communicate to the verification service over port 80. **Compatibility** Rasdaman community and enterprise cannot run in parallel on the same machine. If you plan to have both installations on the same machine, make sure they reside in different directories and are not active at the same time. Rasdaman databases created with rasdaman community are upwards compatible with rasdaman enterprise. **Support** For support in installing rasdaman and any other question you may contact rasdaman GmbH at `www.rasdaman.com `__. .. _sec-system-install-pkgs-deb: Debian-based systems ==================== Currently the following Debian-based distributions are supported: - Ubuntu 20.04 / 22.04 / 24.04 - Debian 12 Installation ------------ 1. Copy the rasdaman licence key to ``/opt/rasdaman/etc``, e.g: :: $ sudo mkdir -p /opt/rasdaman/etc $ sudo cp rmankey /opt/rasdaman/etc .. note:: This has to be done before installing rasdaman. 2. Import the rasdaman repository public key to the apt keyring: :: $ repo="download.rasdaman.com/Download" # set username and password variables to your download credentials $ username="USERNAME" # an empty space before the next command prevents it from saving in the shell history $ password="PASSWORD" # Import the rasdaman repository public key (note the commands are different for # Ubuntu and Debian) # Ubuntu: $ wget -O - "https://$username:$password@$repo/rasdaman.gpg" | \ sudo apt-key add - # Debian: $ curl -fsSL "https://$username:$password@$repo/rasdaman.gpg" | \ sudo gpg --dearmor -o /etc/apt/keyrings/rasdaman.gpg .. note:: You may need to update the ca-certificates package to allow SSL-based applications (e.g. ``apt-get update`` or ``curl``) to check for the authenticity of SSL connections: :: $ sudo apt-get install ca-certificates 3. Add the rasdaman packages repository to apt: - **stable:** these packages are only updated on stable releases of rasdaman, and hence recommended for operational production installations. :: $ . /etc/os-release # provides $VERSION_CODENAME $ echo "deb [arch=amd64] https://$repo/deb $VERSION_CODENAME stable" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list - **testing:** updated more frequently with beta releases, so it is aimed for feature testing in less-critical installations. :: $ . /etc/os-release # provides $VERSION_CODENAME $ echo "deb [arch=amd64] https://$repo/deb $VERSION_CODENAME testing" \ | sudo tee /etc/apt/sources.list.d/rasdaman.list 4. Add the login credentials for the rasdaman packages repository: :: # note: $username and $password were defined in step 2. $ echo "machine download.rasdaman.com login $username password $password" \ | sudo tee /etc/apt/auth.conf.d/rasdaman.conf # make sure that only the root user can read/write this file $ sudo chmod 600 /etc/apt/auth.conf.d/rasdaman.conf 5. rasdaman can be installed now: :: $ sudo apt-get update # check CPU SIMD capabilities $ grep flags /proc/cpuinfo | head -n1 | grep -o -E '(sse|avx)[^ ]*' # install one of rasdaman-avx512, rasdaman-avx2, rasdaman-avx, rasdaman # in that order, depending on what SIMD extensions are supported by your CPU; # e.g. if you see avx512* in the output, then install rasdaman-avx512, if # you don't see avx512 but see avx2 then install rasdaman-avx2, etc. $ sudo apt-get install rasdaman- If during the install you get a prompt like the below, type **N** (default option): .. code-block:: text Configuration file `/etc/opt/rasdaman/petascope.properties' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** petascope.properties (Y/I/N/O/D/Z) [default=N] ? If you are automating the installation (in a script for example), you can bypass this prompt with an apt-get option as follows: :: $ apt-get -o Dpkg::Options::="--force-confdef" install -y rasdaman You will find the rasdaman installation under ``/opt/rasdaman/``. Finally, to make rasql available on the PATH for your system user: :: $ source /etc/profile.d/rasdaman.sh 6. Check that the rasdaman server can answer queries: :: $ rasql -q 'list collections on localhost' --out string Typical output: :: rasql: rasdaman query tool v1.0, rasdaman v10.0.0 -- generated on 26.02.2020 08:44:56. opening database RASBASE at localhost:7001...ok Executing retrieval query...ok Query result collection has 0 element(s): rasql done. 7. Check that petascope is initialized properly, typically at this URL: :: http://localhost:8080/rasdaman/ows .. _sec-system-update-pkgs-deb: Updating -------- The packages are updated whenever a new rasdaman version is released. To update your installation: :: $ sudo apt-get update $ sudo service rasdaman stop $ sudo apt-get install rasdaman .. note:: You may need to update the ca-certificates package to allow SSL-based applications like ``wget/curl`` to check for the authenticity of SSL connections: :: $ sudo apt-get install ca-certificates .. _sec-system-install-pkgs-rpm: RPM-based systems ================= Currently no RPM-based distributions are supported. If an RPM-based OS must be used, then one way to install rasdaman is to setup the latest Ubuntu LTS in a VM or a docker container and install rasdaman in it. .. _customize-package-install: Customizing the package installation ==================================== When installing or updating rasdaman from the official packages, the process can be optionally customized with an installation profile (see example `installer configuration `__). - To customize when installing rasdaman for the first time, it is necessary to first download the package install profile from `here `__. - When updating an existing rasdaman installation, you can find the default package install profile in your installation at ``/opt/rasdaman/share/rasdaman/installer/profiles/package/install.toml``. Download / copy the ``install.toml`` file to some place, e.g. ``$HOME/rasdaman_install.toml``, and make any desired changes to it before installing or updating rasdaman. Make sure that the ``RAS_INSTALL_PATH`` environment variable is set to point to the custom profile, e.g. .. code-block:: shell export RAS_INSTALL_PATH="$HOME/rasdaman_install.toml" When you install or update rasdaman afterwards, the configuration process will take the custom profile into account instead of the default one. .. _sec-system-install-administration: **************** Running rasdaman **************** This section provides a *high-level* overview on how start/stop rasdaman and petascope, monitor them, and configure for typical usage. Most of the time the information presented here is sufficient for operating a rasdaman service; for deeper understanding on how it works behind the scenes, check the :ref:`sec-server-administration` section. .. _sec-system-install-service-control: Service Control =============== rasdaman -------- A ``rasdaman`` service script allows to start/stop rasdaman, e.g. :: $ service rasdaman start $ service rasdaman stop $ service rasdaman force-stop $ service rasdaman status It can be similarly referenced with ``systemctl``, e.g. :: $ systemctl start rasdaman $ systemctl stop rasdaman $ systemctl status rasdaman The service script can be customized by updating environment variables in ``/etc/default/rasdaman`` (create the file if it does not exist). The default settings can be seen below. .. hidden-code-block:: shell # rasdaman installation directory RMANHOME=/opt/rasdaman # local user running the rasdaman server RMANUSER=rasdaman # runuser, or sudo for older OS RUNUSER=runuser # login credentials for non-interactive rasdaman start/stop RASLOGIN=rasadmin:d293a15562d3e70b6fdc5ee452eaed40 # port on which clients connect to rasdaman RASMGR_PORT=7001 # options to be passed on to start_rasdaman.sh START_RASDAMAN_OPTS="-p $RASMGR_PORT" # options to be passed on to stop_rasdaman.sh STOP_RASDAMAN_OPTS="-p $RASMGR_PORT" # Java options to be passed on to embedded petascope JAVA_OPTS="-Xmx4000m" See also the dedicated pages on :ref:`configuration and log files ` and :ref:`administration `. petascope --------- Check :ref:`this section ` on how to start / stop the petascope component of rasdaman. .. _sec-system-install-monitoring: Service monitoring ================== To help with monitoring the health of a running rasdaman service, a ``watch_rasdaman.sh`` script is provided in ``/opt/rasdaman/bin``. It performs status checks on rasdaman by sending a test rasql query, and to petascope by sending a test WCS GetCapabilities request. If a problem is detected in the response, then rasdaman and/or petascope will be restarted, unless this is prevented via the appropriate options. To support the restart actions, it should be executed with root or a user that has sudo rights. Various information is logged to stdout, as well as ``/opt/rasdaman/log/watch_rasdaman.sh.log``. In case of problems, the script can be configured to send an email notification. To see usage details and a list of all options, execute ``watch_rasdaman.sh --help``; in short: :: watch_rasdaman.sh [ --email-config ] [ --petascope-endpoint ] [ --no-restart-rasdaman ] [ --no-restart-petascope ] [ --rmanhome ] [ --custom-check-script ] It is recommended to execute it regularly with a cron job, e.g. every hour: :: $ sudo su # switch to root user $ crontab -e 0 * * * * /opt/rasdaman/bin/watch_rasdaman.sh --email-config ~/.email.cfg .. note:: Executing the script as root is safe, the only system-modifying actions it makes are: logging information in ``/opt/rasdaman/log/watch_rasdaman.sh.log``, and potentially restarting rasdaman, and restarting tomcat if external servlet container deployment is configured in petascope.properties. .. _sec-system-initialize-rasdaman: Configure rasdaman ================== Rasdaman is a multi-server multi-user system. The server processes available must be configured initially, which is done in file ``$RMANHOME/etc/rasmgr.conf``. For distribution, this configuration contains ten server processes going by a name like, for example, ``N1``. If this is fine then you can just leave it as it is. If you want to change this by modifying server startup parameters or increasing the number of server processes available then see :ref:`sec-rascontrol-invocation` for details on how to do this. .. _sec-initialize-geo-services: Configure petascope =================== *Petascope* is the geo Web service frontend of rasdaman. It adds geo semantics on top of arrays, thereby enabling regular and irregular grids based on the `OGC coverage standards `__. To implement the geo semantics, petascope uses a relational database for the geo-related metadata. Currently, PostgreSQL and H2 / HSQLDB are supported. The package post-install script will automatically set up PostgreSQL for use by petascope. The steps approximately performed by the script are listed below. The default setup can be changed in the :ref:`petascope.properties ` configuration file. PostgreSQL ---------- PostgreSQL is automatically configured when rasdaman is installed, so doing the below is not usually necessary; we list the steps as documentation of how is PostgreSQL configured by default: 1. If postgres has not been initialized yet: :: $ sudo service postgresql initdb If the output is 'Data directory is not empty!' then this step is skipped. 2. Trust-based access in PostgreSQL is enabled by adding the below configuration before the ident lines to ``/etc/postgresql/9.4/main/pg_hba.conf`` on Debian 8, or ``/var/lib/pgsql/data/pg_hba.conf`` on CentOS 7: :: host all petauser localhost md5 host all petauser 127.0.0.1/32 md5 host all petauser ::1/128 md5 3. Reload PostgreSQL so that the new configuration will take effect: :: $ sudo service postgresql reload 4. Add a petascope user, for example ``petauser``, to PostgreSQL: :: $ sudo -u postgres createuser -s petauser -P > enter password In ``$RMANHOME/etc/petascope.properties`` set the ``spring.datasource.username``/``spring.datasource.password`` and ``metadata_user``/``metadata_pass`` options accordingly to this user / password. The password is randomly generated. 5. Copy ``/opt/rasdaman/share/rasdaman/war/rasdaman.war`` to the Tomcat webapps directory (``/var/lib/tomcat/webapps`` on CentOS 7) and restart Tomcat. Following successful deployment, petascope accepts OGC W*S requests at URL ``http://localhost:8080/rasdaman/ows``. H2 / HSQLDB ----------- To alternatively set up H2 / HSQLDB for use by petascope instead of PostgreSQL: 1. Create a directory that will host petascopedb and the H2 driver: :: $ mkdir /opt/rasdaman/geodb 2. Make sure the user running the webserver serving petascope can read/write to the folder above. For example, Tomcat webserver which uses `tomcat` user :: $ sudo chown -R tomcat: /opt/rasdaman/geodb However, if embedded deployment is enabled in petascope.properties, then the owner should be the ``rasdaman`` user which runs rasdaman :: $ sudo chown -R rasdaman: /opt/rasdaman/geodb 3. Download the driver and place it in the created directory. For example, download a H2 driver :: $ cd /opt/rasdaman/geodb $ wget https://repo1.maven.org/maven2/com/h2database/h2/1.4.200/h2-1.4.200.jar 4. Configure database settings in petascope.properties file, see :ref:`details `. 5. Restart the webserver running petascope (or rasdaman if embedded tomcat). SSL/TLS configuration --------------------- Transport Layer Security (``TLS``) and its predecessor, Secure Sockets Layer (``SSL``), are technologies which allow web browsers and web servers to communicate over a secured connection. To configure it for ``petascope`` and ``secore web`` applications for ``Tomcat``, check the `official guide `__. .. _sec-rrasdaman-install: .. TODO .. _sec-mqtt-broker: MQTT broker connection ====================== The rasdaman core system and the petascope geo-services component use the `mosquitto MQTT broker `__ to synchronize with each other. This is only done on Ubuntu 22.04 or later due to availability of dependency packages. The ``mosquitto`` package is a dependency of rasdaman, and is automatically installed when rasdaman is installed. The package installation will configure the ``mosquitto`` service to allow only users with valid credentials and only local connections on the default port 1883 by updating ``/etc/mosquitto/mosquitto.conf``, and will create a user ``rasdaman`` with a random password in ``/etc/mosquitto/password_file``. A broker configuration file ``/opt/rasdaman/etc/broker.properties`` will be generated as well, which provides the connection settings needed for rasdaman and petascope to connect to mosquitto: .. code-block:: text address=tcp://localhost:1883 username=rasdaman password= .. _sec-system-install-conf: ************************ Installed Files and Data ************************ Top-level directories ===================== As common with rasdaman, we refer to the installation location as ``$RMANHOME`` below; the default is ``/opt/rasdaman``. The table below lists the top-level directories found in ``$RMANHOME`` after a fresh installation. +---------------------+-------------------------------------------------------------+ |**Directory** |**Description** | +=====================+=============================================================+ |``bin`` |rasdaman executables, e.g. rasql, start_rasdaman.sh, ... | +---------------------+-------------------------------------------------------------+ |``data`` |Path where the server stores array tiles as files; this | | |directory can get big, it is recommended to make | | |it a link to a sufficiently large disk partition. | +---------------------+-------------------------------------------------------------+ |``etc`` |Configuration files, e.g. rasmgr.conf | +---------------------+-------------------------------------------------------------+ |``include`` |C++ API development headers. | +---------------------+-------------------------------------------------------------+ |``lib`` |C++ and Java API libraries. | +---------------------+-------------------------------------------------------------+ |``log`` |``rasmgr`` and ``rasserver`` log files. | +---------------------+-------------------------------------------------------------+ |``share`` |Various artefacts like documentation, python/javascript | | |clients, example data, migration scripts, etc. | +---------------------+-------------------------------------------------------------+ .. _sec-executables: Executables =========== Rasdaman executables are found in ``$RMANHOME/bin``; the table below lists the various binaries and scripts. More detailed information on these components is provided in the :ref:`sec-rasdaman-architecture` Section. +------------------------------+----------------------------------------------------------------+ |**Executables** |**Description** | +==============================+================================================================+ |``rasserver`` |Client queries are evaluated by a ``rasserver`` worker process. | +------------------------------+----------------------------------------------------------------+ |``rasmgr`` |A manager process that controls ``rasserver`` processes and | | |client/server pairing. | +------------------------------+----------------------------------------------------------------+ |``rascontrol`` |A command-line frontend for ``rasmgr``. | +------------------------------+----------------------------------------------------------------+ |``directql`` |A rasserver that can execute queries directly, bypassing the | | |client/server protocol; useful for debugging. | +------------------------------+----------------------------------------------------------------+ |``rasql`` |A command-line client for sending queries to a ``rasserver`` | | |(as assigned by the ``rasmgr``). | +------------------------------+----------------------------------------------------------------+ |``start_rasdaman.sh`` |Start ``rasmgr`` and the worker ``rasservers`` as | | |configured in ``$RMANHOME/etc/rasmgr.conf``. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``stop_rasdaman.sh`` |Shutdown rasdaman, embedded petascope and embedded secore | | |if enabled. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``watch_rasdaman.sh`` |Helper script for monitoring an operational rasdaman service. | | |Details in section on :ref:`sec-system-install-monitoring`. | +------------------------------+----------------------------------------------------------------+ |``create_db.sh`` |Initialize the rasdaman metadata database (RASBASE). | +------------------------------+----------------------------------------------------------------+ |``update_dh.sh`` |Applies migration scripts to RASBASE. | +------------------------------+----------------------------------------------------------------+ |``rasdaman_insertdemo.sh`` |Insert three demo collections into rasdaman (used in the | | |rasdaman Query Language Guide). | +------------------------------+----------------------------------------------------------------+ |``petascope_insertdemo.sh`` |Insert geo-referenced demo coverage in petascope. | +------------------------------+----------------------------------------------------------------+ |``migrate_petascopedb.sh`` |Applies database migrations on petascopedb. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``wcst_import.sh`` |Tool for convenient and flexible import of | | |geo-referenced data into petascope. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``prepare_issue_report.sh`` |Helps preparing a report for an issue encountered while | | |operating rasdaman. More details | | |:ref:`here `. | +------------------------------+----------------------------------------------------------------+ |``rasfed.jar`` |Federation daemon. | +------------------------------+----------------------------------------------------------------+ .. _executables-start-rasdaman: start_rasdaman.sh ----------------- This script starts rasdaman. Normally rasdaman is installed from packages, and instead of executing this script directly one would execute ``service rasdaman start``. Any options to be passed on to ``start_rasdaman.sh`` can be set in ``/etc/default/rasdaman`` in this case; see :ref:`more details `. To start a specific service (rasdaman, rasfed, or :ref:`embedded petascope `) the ``--service (core | rasfed | petascope)`` option can be used(``core`` refers to ``rasmgr`` + ``rasserver`` only). Since v10.0 the rasmgr port can be specified with ``-p, --port``. Additionally, for security and usability reasons, ``start_rasdaman.sh`` will refuse running if executed with root user; this can be overriden if needed with the ``--allow-root`` option. The script will use various environment variables, if they are set before it is executed: - ``RASMGR_PORT`` - the port on which rasmgr will listen when started, and to which client applications will connect in order to send queries to rasdaman. This variable will be overrided by the value of option ``--port``, if specified. By default if none are specified, the port is set to 7001. - ``RASLOGIN`` - rasdaman admin credentials which will be used for starting rasmgr non-interactively. See more details on the format and how is this setting used :ref:`here `. If not set, the script defaults to using rasadmin/rasadmin credentials; see :ref:`here ` on how to change these defaults. - ``JAVA_OPTS`` - options passed on to the ``java`` command when used to start the OGC frontend of rasdaman (petascope) if it is configured for :ref:`embedded deployment `. If not set, it defaults to ``-Xmx4000m`` Check ``-h, --help`` for all details. .. _executables-stop-rasdaman: stop_rasdaman.sh ---------------- This script stops rasdaman. Normally rasdaman is installed from packages, and instead of executing this script directly one would execute ``service rasdaman stop``. Any options to be passed on to ``stop_rasdaman.sh`` can be set in ``/etc/default/rasdaman`` in this case; see :ref:`more details `. The script stops rasmgr, rasservers, rasfed, and petascope (if configured for embedded deployment) in the correct order with a regular TERM signal to each process; this ensures that the services exit properly. In some cases, a process may be hanging instead of exiting on the TERM signal; since rasdaman v10.0, ``stop_rasdaman.sh`` will detect and report such cases. It is prudent to then check the relevant process logs, and if it appears that there is no reason for the process hanging one can force-stop it with ``stop_rasdaman.sh --force``, or manually do it by sending it a KILL signal (e.g. ``kill -KILL ``). To stop a specific service the ``--service (core | rasfed | petascope )`` option can be used. Since v10.0 the rasmgr port can be specified with ``-p, --port``. The script will use various environment variables, if they are set before it is executed: - ``RASMGR_PORT`` - the port on which rasmgr was set to listen when it was started. This variable will be overrided by the value of option ``--port``, if specified. By default if none are specified, the port is set to 7001. - ``RASLOGIN`` - rasdaman admin credentials which will be used for stopping rasmgr non-interactively. See more details on the format and how is this setting used :ref:`here `. If not set, the script defaults to using rasadmin/rasadmin credentials; see :ref:`here ` on how to change these defaults. Check ``-h, --help`` for all details. .. _executables-migrate-petascopedb: migrate_petascopedb.sh ---------------------- This script is used to migrate coverages imported by wcst_import, OWS Service metadata and WMS 1.3 layers. For more details see :ref:`petascope-database-connection` and :ref:`sec-initialize-geo-services`. There are 2 types of migration: 1. Migrate petascopedb v9.4 or older to a newer rasdaman version. After the migration, the old petascopedb is backed up at petascope_94_backup. 2. Migrate petascopedb v9.5 or newer to a different database name or different database (e.g. PostgreSQL to HSQLDB). .. note:: The petascope Web application must not be running (e.g in Tomcat) while migrating to a different database (type 2 above) to protect the existing data integrity. The script will use various environment variables, if they are set before it is executed: - ``JAVA_OPTS`` - options passed on to the ``java`` command when used to start embedded petascope to migrate. If not set, it defaults to ``-Xmx4000m`` Configuration files =================== Configurations are automatically loaded upon rasdaman start. After any modification a restarthas to be performed for the change to take effect. Server rasdaman configuration files can be found in ``$RMANHOME/etc``: +------------------------------+---------------------------------------------------+ |``rasmgr.conf`` |allows fine-tunning the rasdaman servers, e.g. | | |number of servers, names, database connection | +------------------------------+---------------------------------------------------+ |``petascope.properties`` |set petascope properties, e.g. backend/rasdaman | | |connection details, CRS resolver URLs, features | +------------------------------+---------------------------------------------------+ |``secore.properties`` |secore configuration | +------------------------------+---------------------------------------------------+ |``rasfed.properties`` |federation daemon configuration | +------------------------------+---------------------------------------------------+ |``broker.properties`` |settings for connecting to a mosquitto MQTT broker | | |(:ref:`details `) | +------------------------------+---------------------------------------------------+ Logging output of petascope and secore is configured in their respective config files, while logging output of rasdaman is controlled via the below configuration files: +---------------------+--------------------------------------------------+ |``log-rasmgr.conf`` | log output of rasmgr | +---------------------+--------------------------------------------------+ |``log-server.conf`` | log output of rasserver worker processes | +---------------------+--------------------------------------------------+ |``log-client.conf`` | log output of client applications, e.g., rasql | +---------------------+--------------------------------------------------+ rasdaman uses the `Easylogging++ `__ library for logging in its C++ components. Log properties can be configured as documented on the `EasyLogging GitHub page `__. The enterprise licence file ``rmankey`` is also found in the ``etc`` directory. External, potentially relevant configuration files are: +------------+---------------------------------------------------------+ | postgresql |``/var/lib/pgsql/data/{postgresql.conf,pg_hba.conf}`` or | | | ``/etc/postgresql/9.X/{postgresql.conf,pg_hba.conf}`` | +------------+---------------------------------------------------------+ | tomcat |``/etc/tomcat/``, ``/etc/default/tomcat`` | +------------+---------------------------------------------------------+ | mosquitto |``/etc/mosquitto/mosquitto.conf`` | +------------+---------------------------------------------------------+ .. _sec-log-files: Log files ========= **rasdaman** *rasdaman* server logs are placed in ``$RMANHOME/log/``. The server components feed the following files where ``uid`` represents a unique identifier of the process, and ``pid`` is a Linux process identifier: ``rasserver...log`` ``rasserver`` worker logs: at any time there are several rasservers running (depending on the settings in ``rasmgr.conf``) and each has a unique log file. ``rasmgr..log`` ``rasmgr`` log: there is only one ``rasmgr`` process running at any time. ``rasfed.log`` ``rasfed`` log: there is only one ``rasfed`` process running at any time; on rasdaman restart the output from the new process is appended to the same log file. ``petascope.log`` ``petascope`` log if ``java_server=embedded`` in ``petascope.properties``. ``watch_rasdaman.sh.log`` Log from the ``watch_rasdaman.sh`` script is appended to this file whenever it is executed. .. note:: ``ls -ltr`` is a useful command to see the most recently modified log files at the bottom when debugging recently executed queries. **petascope & secore** The path to the petascope.log file is set in the log4j configuration section in ``/opt/rasdaman/etc/petascope.properties``. - If petascope is deployed embedded as part of rasdaman, then the path must be writable by the ``rasdaman`` user; default is on rasdaman installation is ``log4j.appender.rollingFile.File=/opt/rasdaman/log/petascope.log``. - If petascope is deployed in an external servlet container, by default Tomcat 9, then the path must be writable by the ``tomcat9`` user; default is ``log4j.appender.rollingFile.File=/var/log/tomcat9/petascope.log``. .. _sec-temporary-files: Temporary files =============== Rasdaman stores various data temporarily in ``/tmp/rasdaman\_*`` directories, in particular: - ``/tmp/rasdaman\_conversion/`` - format-encoded data, such as TIFF, NetCDF, etc., is in some cases temporarily stored here before decoding into rasdaman. This also happens always when encoding query processing results into some format for export. The intermediate data is quickly removed as soon as the encoding or decoding process is finished. Temporarily, however, this directory can get rather large: if you export array result that encodes into a 1GB TIFF file, then the directory will contain 1GB of data for some time; if 10 such queries run concurrently, then it may contain up to 10GB of data. For this reason we recommend to check the size of ``/tmp`` during installation, and make sure it is large enough. It is always recommended to make ``/tmp`` a separate partition, so as to prevent system-wide problems in case the filesystem is filled up with data. - ``/tmp/rasdaman\_petascope/`` - contains small temporary files generated during data import with the wcst\_import tool. - ``/tmp/rasdaman\_transaction\_locks/`` - during query read/write transaction, rasdaman generates various empty lock files in this directory. As the files are empty, the size of this directory is minimal. While rasdaman is running this directory must not be removed, otherwise it may lead to data corruption. Demo data & programs ==================== Example database ---------------- A demonstration database is provided as part of the delivery package which contains the collections and images described in the :ref:`ql-guide`. To populate this database, first install the system as described here, and then invoke: :: $ rasdaman_insertdemo.sh The demo database occupies marginal disk space, and is a straightforward way to show that the rasdaman installation has been successfull. Example programs ---------------- Several example programs are provided in the ``c++`` and ``java`` subdirectories of ``$RMANHOME/share/rasdaman/examples``. Each directory contains a Makefile plus ``.cc`` and ``.java`` sources, resp. Makefile -------- The ``Makefile`` helps to compile and link the sample C++ / Java sources files delivered. It is a good source for hints on the how-tos of compiler and linker flags. .. note:: All programs, once compiled and linked, print a usage synopsis when invoked without parameter. ``query.cc`` ------------ Sends a hardwired query to a running rasdaman system: .. code-block: rasql select a[0:4,0:4] from mr as a where some_cells( a[8:9,8:9] >= 0 ) In addition, it demonstrates how to work with the result set returned from rasdaman. The query can easily be changed, or made a parameter to the program. ``Query.java`` -------------- Sends the following hardwired query if one is not provided as a parameter: .. code-block: rasql select avg_cells( a ) from mr ``AvgCell.java`` ---------------- This program computes the average cell value from all images of a given collection on client side. Note that it requires grayscale images. A good candidate collection is ``mr`` from the demo database. ***************** Access Interfaces ***************** Rasdaman services can be invoked in several ways: through command line, Web requests, and custom programs connecting via the C++ and Java APIs. Command Line Tools ================== Queries can be submitted to the command line tool ``rasql``. Complete control over the server is provided through several utilities, in particular ``rasmgr``; see :ref:`sec-rascontrol-invocation` for details. All tools can communicate with local and remote rasdaman servers. .. _web-services: Web Services ============ Several Web services are available with rasdaman. They are implemented as servlets, hence independent from the array engine and only available if started in a servlet container such as Tomcat or jetty. They can be accessed under the common context path ``/rasdaman``. * ``/rasdaman/ows`` exposes geo Web Services based on the interface standards of the Open Geospatial Consortium (OGC Web Services, OWS). Supported OGC standards are: - Web Coverage Service (WCS) - Web Coverage Processing Service (WCPS) - Web Map Service (WMS) suites * ``/rasdaman/def`` provides access to a Coordinate Reference System (CRS) Resolver Service, SECORE. It is identical to the one deployed by OGC, where http://www.opengis.net/def/crs is the branch for CRS served by SECORE. * ``/rasdaman/rasql`` provides support for submitting rasql queries and receiving results with standard HTTP requests. Requests must specify three mandatory parameters: +--------------+------------------------------------------------------------+ | ``username`` | rasdaman login name under which the query will be executed | +--------------+------------------------------------------------------------+ | ``password`` | password corresponding to the login | +--------------+------------------------------------------------------------+ | ``query`` | rasql query string, properly encoded for URI embedding | +--------------+------------------------------------------------------------+ Example: :: http://localhost:8080/rasdaman/rasql ?username=rasguest &password=rasguest &query=select%20encode%28mr2%2C%22png%22%29%20from%20mr .. NOTE:: rasql servlet also supports rasdaman user credentials in basic authentication header. In this case, ``username`` and ``password`` parameters are not required as the credentials are extracted from the header. The diagram below illustrates the OGC service architecture of rasdaman: .. hidden-code-block:: text clients read: read: +-----------------+ | | GetCapabilities select ... | +-----------+ | DescribeCoverage | |3rd party | | | +-----------+ | GetCoverage | | ProcessCoverage | +-----------+ | GetMap | |wcs_client | | +------------+ +---------+ | +-----------+ | +-------------> |rasdaman-geo| +--------> |rasserver| | | +------------+ +---------+ | +-----------+ | write: write: | |wcst_import| | | +-----------+ | InsertCoverage create type/coll | | UpdateCoverage insert,update,delete +-----------------+ DeleteCoverage drop type/coll .. include:: 02_inst-guide-web-admin-tools.inc .. -- rasdaman enterprise begin .. _sec-basic-header-auth: Basic header authentication =========================== For incoming requests, rasdaman-geo requires credentials attached in HTTP headers when ``authentication_type=basic_header`` is set in ``petascope.properties``. The valid format of the credentials must be: :: Authorization: Basic encode_in_base64(username:password) For example, if username is `admin` and password is `admin` and the client is ``curl``, one need to construct this request: :: curl -H "Authorization: Basic YWRtaW46YWRtaW4=" \ "http://localhost:8080/rasdaman/ows? service=WCS&version=2.0.1&request=DeleteCoverage&coverageId=test_coverage" or simpler with the ``--user`` option: :: curl --user "rasadmin:rasadmin" \ "http://localhost:8080/rasdaman/ows? service=WCS&version=2.0.1&request=DeleteCoverage&coverageId=test_coverage" .. note:: If basic authentication is not enabled in rasdaman, depending on the request, rasdaman uses one of the users configured by ``rasdaman_user`` and ``rasdaman_admin_user`` settings in ``petascope.properties`` appropriately to run a rasql query. .. note:: Using ``wcst_import`` to insert/update coverages to petascope will require to specify credentials when running script ``wcst_import.sh``. Check ``wcst_import.sh -h`` for optional parameters ``-i, --identity-file`` or ``-u, --user USER_NAME``. .. note:: WSClient (at ``http://YOUR_SERVER/rasdaman/ows``) will detect if petascope requires authentication to show a login form via this API ``http://YOUR_SERVER/rasdaman/admin/authisactive``. If it is the case, one needs to provide valid credentials to login and WSClient keeps the credentials in *Web Browser's local storage*. After that, any requests to petascope will be added basic authentication in HTTP header implicitly. .. -- rasdaman enterprise end APIs ==== Programmatic access is available through self-programmed code using the C++ and Java interfaces; see the `C++ ` and `Java ` guides for details. .. _sec-rasdaman-architecture: ******************* Server Architecture ******************* The parallel server architecture of rasdaman offers a scalable, distributed environment to efficiently process even very large numbers of concurrent client requests. Yet, server administration is easy to accomplish, with only few things to do to have a smoothly running, highly performant installation. Moreover, the system is implemented in a special high availability technique where most server management operations can be done with the server up and running, limiting the need for a server shutdown to the absolute minimum. In this Section the general rasdaman server architecture is outlined. It is recommended to study this section so as to understand server administration terminology used in the next Section. Executables Overview ==================== The following executables are provided in the ``bin/`` directory, among others: * ``rasmgr`` is the central rasdaman request dispatcher; clients connect to ``rasmgr`` initially and are then assigned to a specific ``rasserver`` process which will evaluate queries; * ``rasserver`` is the rasdaman server engine, it should not be generally invoked in a standalone manner; * ``rascontrol`` allows to interactively control the rasdaman server by communicating with ``rasmgr``; * ``rasfed`` is the federation daemon, which enables efficient query query distribution in federated rasdaman networks; * ``rasql`` is the command-line based query tool, explained in detail in the *rasdaman Query Language Guide*. .. _sec-server-mgr-server: Server Manager and Server ========================= Overview and Terminology ------------------------ The rasdaman server configuration consists of one dispatcher process per computer, ``rasmgr`` (we will refer to it as *manager* in the sequel), and server processes, ``rasserver`` (referred to as *servers*), of which at a given time none, one, or several ones can be running. All server processes are under control of the manager. Server manager and rasdaman server(s) all run on the same physical hardware, the *rasdaman host*. The servers resolve requests, thereby generating calls to the relational database system which in turn accesses its database files. For the purpose of this manual, the relational server together with the database it maintains are collectively called the *database*. The machine the relational database server runs on is referred to as *database host* (:numref:`figure2`). .. _figure2: .. figure:: media/inst-guide/image3.png :align: center :width: 450px Overall server hierarchy, introducing the terminology for rasdaman hardware and software environment Server Structure in General --------------------------- The manager accepts client requests and assigns server instances to them, taking them from the pool of server processes it maintains. In distributed installations, it keeps contact to the managers on other machines to further dispatch client requests across all the rasdaman servers available. Whenever needed, the administrator can launch further server instances, or shut them down again. Upon system configuration definition (see :ref:`sec-rascontrol-invocation`), a unique name is assigned to each server identifying it to the manager. Each rasdaman server is assigned to a relational database server, laid down in the manager configuration file. Databases can be registered and associated to particular rasdaman servers at any time. rasdaman hosts and database hosts are identified by their resp. host name in common domain address form, e.g., ``martini.rasdaman.com`` or ``199.198.197.50``. ``Rascontrol`` is the interactive front-end to ``rasmgr`` and, as such, the main utility for user and system management. It provides the necessary functions to manage the whole system configuration, to add and remove user, to change their rights, and to obtain information about system activity. The rasdaman server, i.e., ``rasserver``, is controlled by the manager which starts and stops server instances. Hence, the ``rasserver`` executable should not (and actually cannot) be invoked directly. Dynamic Server Assignment ------------------------- The process of client/server communication and server scheduling is done as follows (see numbers in :numref:`figure-internal-server-mgmt`). 1. The client starts every ``OPENDB`` and ``BEGIN TRANSACTION`` request with an HTTP call to the manager, providing the required service type (RPC, HTTP, etc.) and the database name, together with user name and password. 2. The manager's answer is the server ID of a free server, or an error message in case no server is available or access is denied for the given login. 3. Client-Server communication to perform the database requests. 4. Upon ``CLOSEDB`` and ``ABORT/COMMIT TRANSACTION`` the server informs the manager that it is available again. This is also done upon a client timeout. These negotiation steps are performed between client library and server, hence transparent to the application. The rasdaman server system is started by invoking the server manager ``rasmgr`` (see :ref:`sec-running-manager`). If it finds a configuration file, them autopmatically all servers indicated will be started; alternatively, server configuration can be done directly through ``rascontrol`` (see :ref:`sec-rascontrol-invocation`). .. _figure-internal-server-mgmt: .. figure:: media/inst-guide/image4.png :align: center :width: 450px Internal server management System Start-up --------------- Invocation of the ``rasmgr`` executable must be done under the operating system login under which the rasdaman installation has been done, usually (and recommended) ``rasdaman``. The service script ``/etc/init.d/rasdaman`` (when rasdaman is installed from the packages) automatically takes care of this. Authentication -------------- On every machine hosting rasdaman servers a separate manager has to run. The manager maintains an authorization file, ``$RMANHOME/etc/rasmgr.auth``. It should not be changed by the administrator, as they are generated, maintained, and overwritten by the manager. .. _figure4: .. figure:: media/inst-guide/image5.png :align: center :width: 500px rasdaman federation rasdaman Manager Defaults ------------------------- The manager's default name is the ``hostname`` (the one reported by the UNIX command hostname), but it can be changed (see the ``change`` command). By default, it listens to port 7001 for incoming requests and uses port 7001 for outgoing requests. To keep overview of the ports used, it is recommended to use the following schema (there is, however, no restriction preventing from choosing another schema): - use port number 7001 for the server manager; - use port numbers 7002 to 7999 for rasdaman servers. .. _sec-storage-backend: Storage backend =============== rasdaman stores array data in a file system directory, and array metadata in a standard SQL DBMS. As backends for the array metadata SQLite and PostgreSQL are supported. Default database name, assumed by all tools, is ``RASBASE``. While it can be changed this is not recommended as all tools will need to receive an extra parameter indicating the changed name. .. note:: rasdaman enterprise additionally supports access to pre-existing archives of any structure, see :ref:`insitu` for more information; in this case no array data will be additionally stored. .. _sec-sqlite-backend: SQLite ------ SQLite is the default backend, configured with this setting in ``/opt/rasdaman/etc/rasmgr.conf`` when rasdaman is first installed: :: define dbh rasdaman_host -connect /opt/rasdaman/data/RASBASE The ``-connect`` value is the *absolute* path to the SQLite database file on disk. Array data is stored in the directory containing the SQLite database under a ``TILES`` subdirectory, i.e. ``/opt/rasdaman/data/TILES`` by default. To change the default location: stop rasdaman, move the whole directory to the new location, update rasmgr.conf and finally start rasdaman again. An array data directory can also be specified independently from the SQLite RASBASE file path separated by a semicolon in the format ``;``, e.g: :: -connect /opt/rasdaman/data/RASBASE;/mnt/large_disk/rasdata This may be needed when storing the large array data on a network filesystem which does not have good support for the SQLite database file. But also as the RASBASE file is typically small, it is worth putting it separately on a fast disk. .. _sec-postgres-backend: PostgreSQL ---------- Instead of SQLite, rasdaman can be configured to use a PostgreSQL database. This may be desirable for scalability on a heavily used installation, as postgres offers better support when many users are concurrently accessing rasdaman, especially when importing and querying data simultaneously. To switch to postgres, first the ``-connect`` string in rasmgr.conf needs to be updated to a value of the format ``;``, where ```` is a valid `postgres connection string `__, and ```` is an absolute path to a directory that will hold ingested array data. For example to connect to database RASBASE and default data directory for the array tiles ``/opt/rasdaman/data``: :: define dbh rasdaman_host -connect dbname=RASBASE;/opt/rasdaman/data This assumes that the system user running rasdaman can login to postgres without any password, probably created with ``createuser -s rasdaman``. For the full syntax of the connection string refer to the corresponding `PostgreSQL documentation `__. If the connection string or the data directory contain any spaces, the config value must be quoted with double quotes, e.g: :: -connect "host=localhost port=5432 dbname=RASBASE connect_timeout=10;/opt/rasdaman/data" Another example with a connection URI: :: -connect "postgresql://user:secret@localhost/RASBASE;/opt/rasdaman/data" Once rasmgr.conf has been updated with the new connect string, it is necessary to initialize the database schema by running ``create_db.sh`` (if it has not been done before): :: sudo -u rasdaman /opt/rasdaman/bin/create_db.sh Like with SQLite, the array data is stored in a ``TILES`` subdirectory of the data directory specified in the connect string, i.e. ``/opt/rasdaman/data/TILES``. .. include:: 02_inst-guide-caching.inc .. _sec-server-administration: ********************* Server Administration ********************* This Section explains on how to manage a rasdaman service on a *lower level*: start up and shut down individual server workers, as well as how to monitor and influence server state. It is recommended to first study the previous section so as to understand server administration terminology used here. General Procedure ================= ``rasmgr`` vs. ``rascontrol`` ----------------------------- It is important to distinguish between the manager, ``rasmgr``, and its control front-end, ``rascontrol``. The manager runs as a background process, supervising activity of local (and possibly remote) rasdaman servers. Interaction between user (i.e., administrator) and the manager takes place through the interactive control front end. In the sequel, it is first described how to launch the manager ``rasmgr``, then ``rascontrol`` commands are detailed. Important Security Note ----------------------- To remain compatible with older rasdaman versions, clients use login "rasguest" / password "rasguest" by default (i.e., when no user and password are explicitly set by the application). In the distribution configuration, this user is defined to have read-only access to the databases, so that users can access but not manipulate databases without authentication. Therefore, the administrator is strongly urged to adapt authentication settings to the local security policy before switching databases online. See :ref:`sec-users-rights` to learn more about user management mechanisms. .. _sec-running-manager: Running the Manager =================== Manager Startup --------------- Starting up the rasdaman system is done by invoking the rasdaman manager, ``rasmgr``, from a shell under the ``rasdaman`` operating system login. Usually the manager will be sent to the background: :: $ rasmgr & Starting ``rasmgr`` is the only direct action to be done on it. Any further administration is performed using ``rascontrol``. Note that, unless a server configuration has been defined already, no rasdaman server is available just by starting the manager. Usually ``rasmgr`` is started from ``start_rasdaman.sh``, rather than directly. Invocation Synopsis ------------------- Manager invocation synopsis: :: $ rasmgr [--help] [--hostname h] [--port p] where --help print this help --hostname h host on which the manager process is running is accessible under name / IP address *h* (default: output of Unix command hostname) --port p manager will listen to port number *p* (default: 7001) Examples -------- To start a manager which will listen at port 7001: :: $ rasmgr --port 7001 .. _sec-rascontrol-invocation: ``rascontrol`` Invocation ========================= The manager front end, rascontrol, is a command-line interface used for rasdaman administration. It allows to define the whole rasdaman system configuration, including start up and shut down of server instances and user logins and rights. To secure access to the server administration facilities, rascontrol performs a login process requesting login name and password similar to the Unix rlogin command. User name must be one of the users defined in the rasdaman authentication list (see :ref:`sec-users-rights`). ``rascontrol`` Synopsis ----------------------- :: $ rascontrol [-h|--help] [--host *h*] [--port *n*] [--prompt *n*] [--quiet] [--login|--interactive|--execute *cmd*|--testlogin] where --host h name of the host where the manager runs (default: localhost) -h, --help this help --port n port number at which the manager listens to requests (default: 7001) --prompt n change rascontrol prompt as follows: - ``0`` - prompt '``>``' - ``1`` - prompt '``rasc>``' - ``2`` - prompt '``user:host>``' (default: 2) --quiet quiet, don't print header (default for ``--login`` and ``--testlogin``) --login print login and password, obtained from interactive input, to ``stdout``, then exit (see *Script Use* below) --interactive read login and password from environment variable ``RASLOGIN`` instead of requesting it interactively --execute cmd execute single ``*cmd*`` and exit (batch mode); all text following ``-x`` until end of line is passed as ``command``; this option implicitly assumes ``-e`` --testlogin just do a login and nothing else to check whether the login/password combination provided in the ``RASLOGIN`` variable is valid Interactive Use --------------- In interactive use, ``rascontrol`` will be invoked with the host parameter only. Following successful authentication, ``rascontrol`` accepts command line input from ``stdin``. Here is an example session (``mypasswd`` will not be echoed on screen): .. hidden-code-block:: bash $ rascontrol Login name: *mylogin* Password: *mypasswd* mylogin:localhost> define dbh h1 -connect / mylogin:localhost> define db d1 -dbh h1 mylogin:localhost> define srv s1 -host localhost -type h -dbh h1 mylogin:localhost> up srv s1 mylogin:localhost> save mylogin:localhost> exit $ .. _sec-rascontrol-script-use: Script Use ---------- Alternatively to interactive login, user and password information can be taken from the environment variable ``RASLOGIN``. This variant is suitable for batch scripting in conjunction with the ``-x`` option. The following example shows how first the ``RASLOGIN`` is set appropriately: :: $ export RASLOGIN=`rascontrol --login` and then a sample Unix shell script which starts all rasdaman servers defined in the system configuration, performing implicit login from the environment variable contents which has been obtained from the previous command and pasted into the shell script: :: #!/bin/bash export RASLOGIN=rasadmin:mytotallyencryptedpassword rascontrol -x up srv -all Comments in Scripts ------------------- To enhance legibility of scripts, ``rascontrol`` accepts comments in the usual shell syntax: Lines beginning with a hash sign '#' will be ignored, whatever they may contain. An example is usage in shell *here documents* (type ``man sh`` in your favourite shell for further information on this feature): :: $ rascontrol < 0``: restart rasdaman server after c requests for ``c = 0``: run rasdaman server indefinitely (default: ``c = 10000``) ``-xp options`` pass option string *options* to server upon start (default: no options, i.e., empty string) Option ``-xp`` must be the last option. Everything following "-xp" until end of line is considered to be *options* and will be passed, at startup time, to the server. .. TODO: invalid reference .. ; see :ref:`sec-server-control-options` below for the list of options available. Change Server Settings ---------------------- :: change srv s [-name n] [-port p] [-dbh d] [-autorestart [on|off] [-countdown c] [-xp options] ``s`` change settings for server *s* ``-name n`` change server name to *n* ``-port p`` change port number to *p* ``-dbh d`` new database host where the relational database server runs to which the rasdaman server connects ``-autorestart a`` for ``a = on``: automatically restart rasdaman server after unanticipated termination for ``a = off``: don't restart ``-countdown c`` for ``c > 0``: restart rasdaman server after c requests for ``c = 0``: run rasdaman server indefinitely ``-xp options`` pass option string *options* to server upon start Option ``-xp`` must be the last option. Everything following "-xp" until end of line is considered to be *options* and will be passed, at startup time, to the server. .. TODO: invalid reference .. see Section :ref:`sec-server-control-options` below for the list of options available. Restrictions: - The server host cannot be changed. - The server name cannot be changed while the server is up. - The new settings will be used only next time the server starts. Remove rasdaman Server Definitions ---------------------------------- :: remove srv s ``s`` server name whose entry is to be deleted Remove server *s* from the definition table. It is not possible to remove a server definition while the corresponding server is up and running Status Information ------------------ ``list srv [ s | -host h | -all ] [-p]`` ``s`` give information about server *s* ``-host h`` give information about all servers running on host *h* information is requested ``-all`` list information about all servers on all hosts (default) ``-p`` additionally list configuration information The first is variant prints status information of the currently defined server(s); if *s* is provided, then only server s is listed. Database Hosts ============== Define Database Hosts --------------------- :: define dbh h [-connect c] ``h`` a unique symbolic database host name, usually the host machine name ``-connect c`` the connection string used to connect ``rasserver`` to the backend database server; see :ref:`sec-storage-backend` for more details on the format of *c* depending on whether the backend DBMS is SQLite or PostgreSQL. Change Database Host Settings ----------------------------- :: change dbh h [-name n] [-connect c] ``h`` database host whose entry is to be changed ``-name n`` change symbolic database host name to *n* ``-connect c`` change connect string to *c*; see :ref:`sec-storage-backend` for more details on the format of *c* depending on whether the backend DBMS is SQLite or PostgreSQL. The connection parameters can be changed at any time, however the servers will get the information only when they are restarted. Remove Database Host Definitions -------------------------------- ``remove dbh h`` ``h`` database host name whose entry is to be deleted Remove database host *h* from the definition table. It is not possible to remove a database host definition while this database host has active servers connected to it. Status Information ------------------ ``list dbh`` List all relational database hosts currently defined. Databases ========= Databases represent the physical database itself, together with the relational database server accessing them. It is possible to have multiple database definitions in the rasdaman server environment which are distinguished by the database host; the interpretation, then, is that the same contents (be it the same physical database or a mirrored copy) is available through relational servers running on the different hosts mentioned. In other words, when a client opens a database, the server manager can freely choose any of the database hosts on which the database indicated is defined. The pair (database,database host) must be unique. Define Databases ---------------- ``define db d -dbh db`` ``d`` define database with name *d* ``-dbh db`` set database host name to *db* Change Database Settings ------------------------ ``change db d -name n`` ``d`` database whose name is to be changed ``-name n`` change to new database name *n* Remove Database Definitions --------------------------- ``remove db d -dbh db`` ``d`` name of database to be removed ``-dbh db`` host name of database to be removed Remove definition of database *d* from the definition table. The database itself remains unchanged, it is not physically deleted. It is not possible to remove a database definition while the corresponding database has open transactions. Status Information ------------------ ``list db [ d | -dbh h | -all ]`` ``d`` give information about servers connected to database *d* ``-dbh h`` give information about all servers connected to database *d* via database host *h* ``-all`` list information about all servers connected to any known database (default) List relational database(s) defined. .. include:: 02_inst-guide-cache-control.inc .. _rascontrol-memory-usage: Memory usage ============ By default, the rasdaman server is limited on the amount of memory it can use to 1 GB less than the amount of available system memory as reported by the MemAvailable in /proc/meminfo. This limit can be changed with: ``change memory -size NEWSIZE`` ``NEWSIZE`` an integer number with an optional modifier suffix of ``k`` (for kilobytes), ``M`` (for Megabytes), ``G`` (for Gigabytes) or ``T`` (for Terabytes); for example: 5G. The current memory limit and total memory usage as well as per rasserver process can be shown with: ``list memory`` It is recommended the sum of non-cache and cache memory be set to at least 4 GB less than the total amount of memory on the system, if the main programs running on it are rasdaman, Tomcat, and PostgreSQL. Server Start-up and Shutdown ============================ **Server Start** ``up srv [ s | -host h | -all ]`` ``s`` start only server *s* ``-host s`` start all servers on host *h*; this requires that a manager has been started on this host previously. ``-all`` start all servers defined; note that only those servers can be started on whose host a manager is currently running. Look up the named server(s) in the definition list, and start the specified one(s) using the previously defined individual startup parameters. At least one of the options *s*, -host *s*, and -all must be present. **Server Shutdown** ``down srv [ s | -host h | -all ] [-force] [-kill]`` ``s`` name of the server to be stopped ``-host s`` terminate all servers on host *h* ``-all`` terminate all servers ``-force`` *send SIGTERM* immediately, don't wait for transaction end ``-kill`` *send SIGKILL* immediately, don't wait for transaction end This command shuts down the indicated server(s). At least one of the options *s*, -host *s*, and -all must be present. Without ``-force`` and ``-kill``, the server is marked for shut down and will actually be terminated by sending ``SIGTERM`` after completing the current transaction. With ``-force`` and ``-kill``, the server is terminated instantaneously; this should be handled with extreme caution, as experience shows that relational database systems react differently on such a situation: usually a running transaction is aborted (which is the desired behavior), but sometimes the running transaction is committed (most likely leaving the database in an inconsistent state). See a Unix manual for the difference between ``SIGTERM`` and ``SIGKILL`` signals. The manager on host *h* is not terminated. .. _sec-users-rights: Users and Their Rights ====================== See :ref:`accesscontrol`. Server Control Options ====================== The following options can be passed to the server when it is started by the server manager using the ``up srv`` command. Option settings are defined for a particular server using the ``rascontrol`` command ``change srv -xp`` which passes the rest of the line after ``-xp`` on to the server upon starting it (see :ref:`sec-rasdaman-servers`). --log logfile print log to *logfile*. If *logfile* is stdout, then log output will be printed to standard output. It is not recommended setting this option. (default: ``$RMANHOME/log/rasserver.uuid.serverpid.log``) --transbuffer b maximum size of transfer buffer to *b* bytes (default: 100000000 bytes = 100 MB) --tilesize s default maximal size of tiles in bytes used when no tile size is specified in queries (default: 4194304 bytes) --pctmin s minimal size of inline tiles in bytes (default: 2048) --pctmax s maximal size of inline tiles in bytes (default: 4096) --tiling name default tiling scheme when inserting data when no tiling clause is specified, one of: NoTiling, RegularTiling, AlignedTiling (default: AlignedTiling) --tileconf dom default tile configuration when inserting data when no tiling clause is specified (default: [0:1023,0:1023]) --index name default index to be used when inserting data when no tiling tiling clause is specified, one of: auto, dir, rdir, nrp, rnrp, tc, rc (default: nrp, i.e. R+ tree) --indexsize s specify the node size of the index; value of 0 lets rasdaman itself determine this value (default: 0) Miscellaneous ============= Help ---- :: help Display top level help page :: help [command] command help Display information specific to *command* (both syntax variants are equivalent) Version Information ------------------- :: list version ``version`` display rasdaman server version. Save Changes to Disk -------------------- :: save The ``save`` operation writes the current configuration and authorization values to disk. All changes done during the session thus become permanent. ``rascontrol`` Termination -------------------------- :: exit terminates ``rascontrol``. .. include:: 02_inst-guide-federation.inc .. include:: 02_inst-guide-billing.inc .. include:: 02_inst-guide-udf-packages.inc .. _rasdaman-security: ******** Security ******** General ======= There are several security measures available, which should be considered seriously. Among them are the access right mechanisms found in Tomcat, rasdaman, and PostgreSQL. We highly recommend to make use of these. For Tomcat and PostgreSQL refer to the pertaining documentation. The servlet is safe against SQL injection attacks - we are not aware of any means for the user to send custom queries to the PostgreSQL server or the rasdaman server. XSRF and XSS represent no danger to the service because there is no user generated content available. For rasdaman, we recommend to change the default user passwords in rasdaman (rasguest/rasguest for read-only access, rasadmin/rasadmin for read-write and administrator access) to not run into the Oracle "Scott/tiger" trap. Even better, use additional separate, private users. The rasdaman service doesn't use cookies. .. _sec-security-reset-passwords: Reset default passwords ======================= This is a quick guide to reset the default rasdaman users, rasguest and rasadmin, which are created when the system is first installed and initialized. Pre-requisites: rasdaman must be up and running. 1. Reset rasguest - Update password in rasdaman: :: $ rasql -q 'ALTER USER rasguest SET PASSWORD TO "newpassword"' \ --user rasadmin --passwd rasadmin - Update ``rasdaman_pass`` (for read-only access from petascope) in ``/opt/rasdaman/etc/petascope.properties`` to set it to the newpassword. - Update ``rasdamanPassword`` in ``/opt/rasdaman/etc/rasfed.properties`` to set it to the newpassword. - Grant ``PRIV_OWS_WCS_PROCESS_COV`` role so that the rasguest user will be able to run WCPS queries: :: $ rasql -q 'GRANT PRIV_OWS_WCS_PROCESS_COV to rasguest' \ --user rasadmin --passwd rasadmin 2. Reset rasadmin - Update password in rasdaman: :: $ rasql -q 'ALTER USER rasadmin SET PASSWORD TO "newpassword"' \ --user rasadmin --passwd rasadmin - Update ``rasdaman_admin_pass`` (for read-write access from petascope) in ``/opt/rasdaman/etc/petascope.properties`` to set it to the newpassword. - Update RASLOGIN env variable in ``/etc/default/rasdaman``: :: # get an md5 hash of the new password $ echo -n "newpassword" | md5sum | awk '{ print $1; }' # update the previous md5 hash of RASLOGIN by editing the file $ nano /etc/default/rasdaman # login credentials for non-interactive rasdaman start/stop RASLOGIN=rasadmin: 3. Restart rasdaman: :: sudo systemctl restart rasdaman 4. Restart tomcat (if external tomcat is configured in ``petascope.properties``): :: tomcat_service=$(systemctl list-units | grep -i tomcat) sudo systemctl $tomcat_service restart .. _security-require-authentication: Require authentication for API access ===================================== To make sure that geo requests to the ``/rasdaman/ows`` endpoint will require authentication, it is necessary to configure the ``authentication_type`` setting in ``petascope.properties`` to a non-empty value. Full details can be found :ref:`here `. Allow annonymous API access =========================== It is possible to allow unauthenticated access *in addition* to supporting authenticated access. Rasdaman must be configured as in the previous section (:ref:`security-require-authentication`). In addition, a rasdaman user credentials need to be set in the ``rasdaman_user`` and ``rasdaman_pass`` settings in ``petascope.properties`` (:ref:`docs `). Any unauthenticated API requests will be executed internally with this rasdaman user, which can be restricted with appropriate :ref:`privileges, triggers `, and :ref:`quotas `. Whitelist access control ======================== A :ref:`Trigger ` allows to define access control rule for *all* rasdaman users. Triggers can be defined in the :ref:`Web admin interface `, or with the rasql command-line tool. Once a trigger is created, specific rasdaman users can be *exempted* (i.e. whitelisted) from the trigger rule. Currently the following resources can be *restricted* with triggers. .. _security-completely-disable-access: Completely disable access to a coverage --------------------------------------- Disabling access to a coverage can be achieved with the ``accessed ()`` condition. In this case, a user will not see the restricted coverage at all in GetCapabilities, unless they have been exempted from the trigger, or have been granted the ``PRIV_LIST_ALL_COLLS`` :ref:`privilege `. To restrict access to a coverage ``C`` a trigger like this can be created: .. code-block:: rasql CREATE TRIGGER C_disable_access WHEN accessed(C) BEGIN EXCEPTION "Access forbidden to C." END Once created, access will be immediately blocked for all rasdaman users. Each ``userX`` that needs to be allowed access must be explicitly exempted: .. code-block:: rasql GRANT EXEMPTION FROM C_disable_access TO userX If multiple users should have the same rules for access, a role can be :ref:`created `, exemptions can be :ref:`granted ` to it as needed, and then the role can be granted to rasdaman users. For example: .. code-block:: rasql CREATE ROLE staff GRANT EXEMPTION FROM C1_disable_access TO staff GRANT EXEMPTION FROM C2_disable_access TO staff ... GRANT staff TO user1 GRANT staff TO user2 ... Partially disable access to a coverage -------------------------------------- Similar to the :ref:`previous section `, but access is disabled only to a *spatio-temporal subset* of the coverage, rather than the whole coverage. This allows to restrict sensitive areas within a coverage. To achieve this, use the ``accessed(, )`` condition. Currently the ```` must be specified in grid coordinates. For example: .. code-block:: rasql CREATE TRIGGER C_disable_partial_access WHEN accessed(C, [1000:1500, -50:250]) BEGIN EXCEPTION "Access forbidden to this area in C." END Limit data access/download per request -------------------------------------- It is possible to limit how much data can users can access from disk, or download over the network (:ref:`docs `). .. code-block:: rasql CREATE TRIGGER max_data_access_trigger WHEN (CONTEXT.ACCESSVOLUME > 200000000) BEGIN EXCEPTION "Data access is restricted to 200 MB." END CREATE TRIGGER max_data_result_trigger WHEN (CONTEXT.RESULTVOLUME > 200000000) BEGIN EXCEPTION "Data download is restricted to 200 MB." END Blacklist access control ======================== The :ref:`quota management ` capabilities of rasdaman allow to set rules *per user*. Quotas on resource usage can be defined per request, as well as aggregated per calendar month. This is most convinient to manage in the :ref:`Web admin interface `. .. _rasdaman-backup: ****** Backup ****** Both software and hardware can fail, therefore it is prudent to establish regular backup procedures. The rasdaman installation comes with a utility script ``/opt/rasdaman/bin/backup_rasdaman`` that can be used to easily backup the rasdaman installation, databases, and external data. Below we list the data that should be considered if backing up rasdaman manually: 1. The rasdaman database, which normally can be found in ``/opt/rasdaman/data``. The SQL database itself in this directory, ``RASBASE``, is fairly small; the ``TILES`` subdirectory may be large as it contains all the array data, but if backup disk space is not scarce then it is definitely recommended to backup as well. Incremental backups of the ``TILES`` with rsync for example should work well without unnecessary duplicated data copying, unless existing data areas are often updated. Example with rsync: .. code-block:: shell # backup small RASBASE to /backup/rasdamandb rsync -avz /opt/rasdaman/data/RASBASE /backup/rasdamandb/ # backup potentially large TILES dir to /backup/rasdamandb rsync -avz /opt/rasdaman/data/TILES /backup/rasdamandb/ 2. The petascopedb geo metadata database is usually small and worth backing up. By default it is stored in PostgreSQL and can be extracted into a small compressed archive as follows: .. code-block:: shell # create backup in a gzip archive petascopedb.sql.gz sudo -u postgres pg_dump petascopedb | gzip > /backup/petascopedb.sql.gz If necessary, it can be restored with .. code-block:: shell # if a petascopedb already exists it needs to be renamed, as otherwise # restoring over an existing petascopedb will corrupt it sudo -u postgres psql -c "ALTER DATABASE petascopedb RENAME TO petascopedb_existing_backup" # create an empty petascopedb sudo -u postgres createdb petascopedb # restore backup petascopedb.sql.gz (use cat if it's not a gzip archive) zcat petascopedb.sql.gz | sudo -u postgres psql -d petascopedb --quiet > /dev/null Alternatively, if the above fails for some reason, petascopedb can be backup with this command: .. code-block:: shell # note that /backup/petascopedb_backup will contain a large number of compressed files sudo -u postgres pg_dump -j 8 -Fd petascopedb -f /backup/petascopedb_backup If necessary, it can be restored with .. code-block:: shell sudo -u postgres pg_restore -j 8 -d petascopedb /backup/petascopedb_backup 3. The rasdaman configuration files in ``/opt/rasdaman/etc``, but also consider the ``bin`` and ``share`` directories which may be useful in case of package update problems, as well as maybe log files in the ``log`` directory. .. code-block:: shell # backup everything except the data dir, which is handled in step 1. above rsync -avz --exclude='data/' /opt/rasdaman /backup/ .. _rasdaman-migration: ********* Migration ********* From one machine to another =========================== Sometimes it is necessary to migrate the installation from one machine (*OLD*) to another (*NEW*). This section outlines the steps on how to do this. 1. Make sure rasdaman is installed and functional on the NEW machine. 2. Stop rasdaman and an external tomcat if installed on both the OLD and NEW machine, e.g: .. code-block:: shell sudo service rasdaman stop sudo service tomcat9 stop 3. Make a backup of the rasdaman and petascope databases on the OLD machine by following step 1. of the :ref:`backup guide ` and copy the backup to the NEW machine. 4. Restore the database backups on the NEW machine by following step 2. of the :ref:`backup guide `. 5. Make a backup of the config files on the NEW machine (``/opt/rasdaman/etc``), then copy relevant configuration from the OLD to the NEW machine, in particular: - ``rasmgr.conf`` can probably copied as is, but check if the -host setting is correct for the NEW machine; - most settings from ``petascope.properties`` can be copied as is, except the ones for database configuration (spring.* and metadata*); - if federation is enabled on the OLD machine then most settings from ``rasfed.properties`` can be copied, but carefully check the hostname setting; - ``/etc/default/rasdaman`` can be copied as is usually; 6. If UDFs have been registered on the OLD machine: - if the NEW machine is the same OS and CPU architecture as the OLD then the UDF libraries can probably be synced by copying the ``/opt/rasdaman/share/rasdaman/udf`` directory; - otherwise, they will likely need to be recompiled on the NEW machine; 7. Make sure that the ``/opt/rasdaman`` directory is owned by the ``rasdaman`` user, to avoid any permission issues caused by copying with other system users: .. code-block:: shell sudo chown -R rasdaman: /opt/rasdaman 8. Finally start rasdaman and tomcat: .. code-block:: shell sudo service rasdaman start sudo service tomcat9 start Ubuntu 18.04 to Ubuntu 20.04 ============================ 1. Make a backup of the rasdaman and petascope databases by following the :ref:`backup guide `. In particular: :: # postgres version OLDVER=10 # alt 1: create backup in petascopedb.sql.gz; to be restored with psql sudo -u postgres pg_dump petascopedb | gzip > /backup/petascopedb.sql.gz # alt 2: text backup to be restored with pg_restore sudo -u postgres pg_dump --create --compress=5 petascopedb \ --file=/backup/petascopedb.sql.gz # backup postgres databases by direct copy as well just in case sudo cp -a /var/lib/postgresql/$OLDVER/main/ /backup/petascopedb_raw_$OLDVER # backup postgres config sudo cp -a /etc/postgresql/$OLDVER /backup/etc_postgresql_$OLDVER # backup rasdaman dir sudo cp -a /opt/rasdaman /backup/opt_rasdaman Disable the rasdaman repo in apt and remove rasdaman: :: REPO_FILE=/etc/apt/sources.list.d/rasdaman.list sudo mv $REPO_FILE $REPO_FILE.disabled # remove rasdaman package; this won't remove any configuration/data sudo service rasdaman stop sudo apt remove "$(dpkg -l | grep '^ii *rasdaman' | awk '{ print $2; }')" 2. Upgrade to Ubuntu 20.04: :: # first remove this package as it breaks the upgrade apt remove postgresql-10-postgis-2.4 # then upgrade do-release-upgrade 3. Migrate data to new postgres version: :: sudo apt install postgresql-12-postgis-3 OLDVER=10 NEWVER=12 # ideally one would run this command and be done, but it fails because the old # postgresql-10-postgis-2.4 gets removed during the upgrade and it is required # in order to do the pg_upgrade. Execute it in any case, as it may migrate # at least configuration files like pg_hba.conf sudo systemctl stop postgresql.service sudo -u postgres /usr/lib/postgresql/$NEWVER/bin/pg_upgrade \ --old-datadir=/var/lib/postgresql/$OLDVER/main \ --new-datadir=/var/lib/postgresql/$NEWVER/main \ --old-bindir=/usr/lib/postgresql/$OLDVER/bin \ --new-bindir=/usr/lib/postgresql/$NEWVER/bin \ --old-options "-c config_file=/etc/postgresql/$OLDVER/main/postgresql.conf" \ --new-options "-c config_file=/etc/postgresql/$NEWVER/main/postgresql.conf" sudo systemctl start postgresql.service # instead we have to restore the backup created in step 1. with psql/pg_restore sudo -u postgres -i # # alt 1: restore database with psql /usr/lib/postgresql/$NEWVER/bin/createdb -p 5433 petascopedb # enter the spring.datasource.password= from /opt/rasdaman/etc/petascope.properties /usr/lib/postgresql/$NEWVER/bin/createuser -s -p 5433 petauser -P zcat /backup/petascopedb.sql.gz | \ /usr/lib/postgresql/$NEWVER/bin/psql -p 5433 -d petascopedb > /dev/null # # alt 2: restore database with pg_restore pg_restore -p 5433 --file=/backup/petascopedb.sql.gz # # swap ports in postgres config, so the new version is at 5432 sed -i 's/port = 5432/port = 5433/' /etc/postgresql/$OLDVER/main/postgresql.conf sed -i 's/port = 5433/port = 5432/' /etc/postgresql/$NEWVER/main/postgresql.conf # restart postgres sudo systemctl restart postgresql.service # check version, should show 12.x sudo -u postgres psql -c "SELECT version();" 4. Install rasdaman: :: # enable rasdaman repo with correct distribution codename REPO_FILE=/etc/apt/sources.list.d/rasdaman.list sed 's/bionic/focal/g' $REPO_FILE.disabled | sudo tee $REPO_FILE sudo apt update # install rasdaman (or rasdaman-avx, rasdaman-avx2, rasdaman-avx512 # depending on CPU capabilities) sudo apt install rasdaman 5. Test rasdaman installation to make sure everything is working; if UDFs are deployed they will need to be recompiled, and same with any custom C++ clients. 6. Remove old postgres (purge removes its configuration and data as well): :: sudo apt purge postgresql-10 postgresql-client-10 Ubuntu 20.04 to Ubuntu 22.04 ============================ 1. Make a backup of the rasdaman and petascope databases by following the :ref:`backup guide `. In particular: :: # postgres version OLDVER=12 # alt 1: create backup in petascopedb.sql.gz; to be restored with psql sudo -u postgres pg_dump petascopedb | gzip > /backup/petascopedb.sql.gz # alt 2: text backup to be restored with pg_restore sudo -u postgres pg_dump --create --compress=5 petascopedb \ --file=/backup/petascopedb.sql.gz # backup postgres databases by direct copy as well just in case sudo cp -a /var/lib/postgresql/$OLDVER/main/ /backup/petascopedb_raw_$OLDVER # backup postgres config sudo cp -a /etc/postgresql/$OLDVER /backup/etc_postgresql_$OLDVER # backup rasdaman dir sudo cp -a /opt/rasdaman /backup/opt_rasdaman Disable the rasdaman repo in apt and remove rasdaman: :: REPO_FILE=/etc/apt/sources.list.d/rasdaman.list sudo mv $REPO_FILE $REPO_FILE.disabled # remove rasdaman package; this won't remove any configuration/data sudo service rasdaman stop sudo apt remove "$(dpkg -l | grep '^ii *rasdaman' | awk '{ print $2; }')" 2. Upgrade to Ubuntu 22.04 with ``do-release-upgrade`` 3. Migrate data to new postgres version: :: sudo systemctl stop postgresql.service sudo apt install postgresql-14-postgis-3 # migrate data sudo -u postgres -i cd /tmp OLDVER=12 NEWVER=14 # migrate petascopedb /usr/lib/postgresql/$NEWVER/bin/pg_upgrade \ --old-datadir=/var/lib/postgresql/$OLDVER/main \ --new-datadir=/var/lib/postgresql/$NEWVER/main \ --old-bindir=/usr/lib/postgresql/$OLDVER/bin \ --new-bindir=/usr/lib/postgresql/$NEWVER/bin \ --old-options "-c config_file=/etc/postgresql/$OLDVER/main/postgresql.conf" \ --new-options "-c config_file=/etc/postgresql/$NEWVER/main/postgresql.conf" # swap ports in postgres config, so the new version is at 5432 sed -i 's/port = 5432/port = 5433/' /etc/postgresql/$OLDVER/main/postgresql.conf sed -i 's/port = 5433/port = 5432/' /etc/postgresql/$NEWVER/main/postgresql.conf # restart postgres sudo systemctl restart postgresql.service sudo -u postgres -i /usr/lib/postgresql/$NEWVER/bin/vacuumdb --all --analyze-in-stages # check version, should show 14.x psql -c "SELECT version();" 4. Install rasdaman: :: # enable rasdaman repo with correct distribution codename REPO_FILE=/etc/apt/sources.list.d/rasdaman.list sed 's/focal/jammy/g' $REPO_FILE.disabled | sudo tee $REPO_FILE sudo apt update # install rasdaman (or rasdaman-avx, rasdaman-avx2, rasdaman-avx512 # depending on CPU capabilities) sudo apt install rasdaman 5. Test rasdaman installation to make sure everything is working; if UDFs are deployed they will need to be recompiled, and same with any custom C++ clients. 6. Remove old postgres (purge removes its configuration and data as well): :: sudo -u postgres /tmp/delete_old_cluster.sh sudo apt purge postgresql-12 postgresql-client-12 postgresql-12-postgis-3 .. _upgrade22to24: Ubuntu 22.04 to Ubuntu 24.04 ============================ 1. Make a backup of the rasdaman and petascope databases by following the :ref:`backup guide `. In particular: :: # postgres version OLDVER=14 # alt 1: create backup in petascopedb.sql.gz; to be restored with psql sudo -u postgres pg_dump petascopedb | gzip > /backup/petascopedb.sql.gz # alt 2: text backup to be restored with pg_restore sudo -u postgres pg_dump --create --compress=5 petascopedb \ --file=/backup/petascopedb.sql.gz # backup postgres databases by direct copy as well just in case sudo cp -a /var/lib/postgresql/$OLDVER/main/ /backup/petascopedb_raw_$OLDVER # backup postgres config sudo cp -a /etc/postgresql/$OLDVER /backup/etc_postgresql_$OLDVER # backup rasdaman dir (note the data subdir may be large) sudo cp -a /opt/rasdaman /backup/opt_rasdaman Disable the rasdaman repo in apt and remove rasdaman: :: REPO_FILE=/etc/apt/sources.list.d/rasdaman.list sudo mv $REPO_FILE $REPO_FILE.disabled # remove rasdaman package; this won't remove any configuration/data sudo service rasdaman stop sudo apt remove "$(dpkg -l | grep '^ii *rasdaman' | awk '{ print $2; }')" 2. Upgrade to Ubuntu 24.04: :: do-release-upgrade If this command fails due to the ``postgresql-14-postgis-3`` package, it has to be removed first: :: sudo apt remove postgresql-14-postgis-3 3. Migrate data to new postgres version: :: sudo systemctl stop postgresql.service sudo apt install postgresql-16-postgis-3 # if postgresql-14-postgis-3 was removed in step 2. it has to be installed # again now, otherwise the petascopedb migration command later on will fail echo 'deb http://archive.ubuntu.com/ubuntu jammy main restricted universe multiverse' | sudo tee -a /etc/apt/sources.list sudo apt update sudo apt install -y postgresql-14-postgis-3 # if external tomcat is used for petascope deployment, then install the # tomcat9 package from the jammy repositories, as petascope is # incompatible with the tomcat10 in the noble (Ubuntu 24.04) repository sudo apt install -y tomcat9 # migrate data sudo -u postgres -i cd /tmp OLDVER=14 NEWVER=16 # migrate petascopedb /usr/lib/postgresql/$NEWVER/bin/pg_upgrade \ --old-datadir=/var/lib/postgresql/$OLDVER/main \ --new-datadir=/var/lib/postgresql/$NEWVER/main \ --old-bindir=/usr/lib/postgresql/$OLDVER/bin \ --new-bindir=/usr/lib/postgresql/$NEWVER/bin \ --old-options "-c config_file=/etc/postgresql/$OLDVER/main/postgresql.conf" \ --new-options "-c config_file=/etc/postgresql/$NEWVER/main/postgresql.conf" # swap ports in postgres config, so the new version is at 5432 sed -i 's/port = 5432/port = 5433/' /etc/postgresql/$OLDVER/main/postgresql.conf sed -i 's/port = 5433/port = 5432/' /etc/postgresql/$NEWVER/main/postgresql.conf # restart postgres (run this command in a different terminal) sudo systemctl restart postgresql.service /usr/lib/postgresql/$NEWVER/bin/vacuumdb --all --analyze-in-stages # check version, should show 16.x psql -c "SELECT version();" # finally, remove any lines with jammy in /etc/apt/sources.list 4. Install rasdaman: :: # enable rasdaman repo with correct distribution codename REPO_FILE=/etc/apt/sources.list.d/rasdaman.list sed 's/jammy/noble/g' $REPO_FILE.disabled | sudo tee $REPO_FILE sudo apt update # check CPU SIMD capabilities grep flags /proc/cpuinfo | head -n1 | grep -o -E '(sse|avx)[^ ]*' # install one of rasdaman-avx512, rasdaman-avx2, rasdaman-avx, rasdaman # in that order, depending on what SIMD extensions are supported by your CPU; # e.g. if you see avx512* in the output, then install rasdaman-avx512, if # you don't see avx512 but see avx2 then install rasdaman-avx2, etc. sudo apt install -o Dpkg::Options::="--force-confdef" rasdaman- 5. Test the rasdaman installation to make sure everything is working; if UDFs are deployed they will need to be recompiled, and same with any custom C++ clients. 6. Remove old postgres (purge removes its configuration and data as well): :: sudo -u postgres /tmp/delete_old_cluster.sh sudo apt purge postgresql-14 postgresql-client-14 postgresql-14-postgis-3 ************** Uninstallation ************** When uninstalling rasdaman, you can execute the following commands to ensure that all installed files and services are fully removed from the system. .. hidden-code-block:: bash # Stop rasdaman, disable service, and clear from failed list $ sudo systemctl stop rasdaman $ sudo systemctl disable rasdaman $ sudo systemctl reset-failed # Removes everything related to rasdaman (except dependencies) # on Ubuntu/Debian systems; for CentOS use `yum erase rasdaman` $ sudo apt-get purge rasdaman # Remove the rasdaman installation directory # WARNING: This removes all rasdaman data and configuration files! $ sudo rm -r /opt/rasdaman # Remove systemd service, init script, profile script $ sudo rm /etc/systemd/system/rasdaman.service $ sudo rm /etc/init.d/rasdaman $ sudo rm /etc/profile.d/rasdaman.sh # Remove rasdaman apt repository on Ubuntu/Debian $ sudo rm /etc/apt/sources.list.d/rasdaman.list # Remove dependencies of the rasdaman package which are no longer # needed by other packages; it will remove postgres if no other # package depends on it (Debian / Ubuntu) $ sudo apt-get autoremove *************** Troubleshooting *************** General ======= The first step in troubleshooting problems should be to look into the :ref:`server logs `. Start with checking the ``rasmgr`` and ``rasserver`` logs for any errors. If this does not provide any clues, check the ``petascope.log`` or ``catalina.out``. Next, investigate the status of rasdaman and external Tomcat if applicable with ``systemctl rasdaman status`` (and similar for Tomcat). Inspect the output of ``ps aux | grep ras`` to list details about the rasdaman processes, or ``top`` for CPU and memory usage. It can be useful to double check the system memory usage with ``free -m``, and disk space usage with ``df -h``. Manually stop rasdaman ====================== If stopping rasdaman fails, it may be necessary to manually stop it: .. hidden-code-block:: bash # check the rasdaman processes still running on the system $ ps aux | grep ras # force kill any rasmgr process; is the number in the 2nd column # of the output from the previous command $ kill -9 # then try to kill any remaining rasserver processes $ pkill -f rasserver # if this fails (check with ps aux), force kill rasservers $ pkill -9 -f rasserver Checking the server logs could provide further information on why stopping rasdaman failed in the first place.