MIR-1320 MIR support for Solr authentication and Solr cloud

.env

@@ -17,3 +17,12 @@ MIR_XMS=1024m
 
 SOLR_HTTP=8290
 SOLR_DATA=./docker/solr-data/
+SOLR_ADMIN_USER=admin
+SOLR_ADMIN_PASSWORD=alleswirdgut
+SOLR_SEARCH_USER=searcher
+SOLR_SEARCH_PASSWORD=Alleswirdgut!?
+SOLR_INDEX_USER=indexer
+SOLR_INDEX_PASSWORD=Alleswirdgut!?
+
+
+TIKA_PORT=8294

README.md

@@ -20,10 +20,12 @@ This guide addresses developers. Thats why you run it in 'dev' profile!
       </properties>
     </profile>
 ```
- - initialize solr configuration using `git submodule update --init --recursive`
  - to start solr, go to mir-webapp
   - install solr with the command: `mvn -Pdev solr-runner:copyHome`
   - run solr with the command: `mvn -Pdev solr-runner:start`
+    - The default users are `admin`, `indexer` and `searcher` with password `alleswirdgut` 
+    - In the wizard of the application you need to check `Erstelle SOLR-Kerne per Solr-Cloud rest-API` and the configure
+    user options with the above-mentioned users and passwords.
   - stop solr with the command: `mvn -Pdev solr-runner:stop`
   - update solr with the command: `mvn -Pdev solr-runner:stop solr-runner:copyHome solr-runner:start`
  - to starting up a servlet container in development environment go back to mir folder
@@ -48,6 +50,14 @@ The docker container has its own install script which uses the environment varia
 ### Environment Variables
 | Property                 | Default,  required  | Description                                                                                                                                                                                                                                                                          |
 |--------------------------|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| ENABLE_SOLR_CLOUD        | false               | If true the Solr Cloud mode is enabled. (solr cores will be created on install)                                                                                                                                                                                                      |
+| SOLR_ADMIN_USER          | none                | The username for the Solr Admin. (will be used for admin commands like creating cores)                                                                                                                                                                                               |
+| SOLR_ADMIN_PASSWORD      | none                | The password for the Solr Admin.                                                                                                                                                                                                                                                     |
+| SOLR_INDEX_USER          | none                | The username for the Solr Indexer. (will be used for indexing)                                                                                                                                                                                                                       |
+| SOLR_INDEX_PASSWORD      | none                | The password for the Solr Indexer.                                                                                                                                                                                                                                                   |
+| SOLR_SEARCH_USER         | none                | The username for the Solr Searcher. (will be used for searching)                                                                                                                                                                                                                     |
+| SOLR_SEARCH_PASSWORD     | none                | The password for the Solr Searcher.                                                                                                                                                                                                                                                  |
+| TIKASERVER_URL           | none                | The URL to the Tika Server. Same as MCR.Solr.Tika.ServerURL in mycore.properties. (also sets `MCR.Solr.FileIndexStrategy` to `org.mycore.solr.index.file.tika.MCRTikaSolrFileStrategy`)                                                                                              |
 | SOLR_URL                 | none, required      | The URL to the SOLR Server. Same as MCR.Solr.ServerURL in mycore.properties.                                                                                                                                                                                                         |
 | SOLR_CORE                | mir                 | The name of the Solr main core. Same as MCR.Solr.Core.main.Name in mycore.properties.                                                                                                                                                                                                |
 | SOLR_CLASSIFICATION_CORE | mir-classifications | The name of the Solr classification core. Same as MCR.Solr.Core.classification.Name in mycore.properties.                                                                                                                                                                            |
@@ -74,7 +84,7 @@ To fix this you can set the docker property `FIX_FILE_SYSTEM_RIGHTS` to `true`.
 mounted volumes to `mcr` and the container will start without errors.
 
 ## `mir-solr` Docker-Container
-The docker container creates the required solr cores if they do not exist.
+The docker container starts solr in cloud mode. It preconfigures the users with the environment variables (see table above).
 
 ### Mount Points
 

docker-compose.yml

@@ -21,6 +21,19 @@ services:
       - ${SOLR_DATA}:/var/solr/data
     ports:
       - ${SOLR_HTTP}:8983
+    environment:
+      - SOLR_ADMIN_USER=${SOLR_ADMIN_USER}
+      - SOLR_ADMIN_PASSWORD=${SOLR_ADMIN_PASSWORD}
+      - SOLR_SEARCH_USER=${SOLR_SEARCH_USER}
+      - SOLR_SEARCH_PASSWORD=${SOLR_SEARCH_PASSWORD}
+      - SOLR_INDEX_USER=${SOLR_INDEX_USER}
+      - SOLR_INDEX_PASSWORD=${SOLR_INDEX_PASSWORD}
+  tika:
+    container_name: ${NAME}-tika
+    image: apache/tika:2.9.2.1-full
+    restart: unless-stopped
+    ports:
+      - ${TIKA_PORT}:9998
   mir:
     container_name: ${NAME}-mir
     build: ./
@@ -34,6 +47,14 @@ services:
       - SOLR_URL=http://solr:8983
       - SOLR_CORE=mir
       - SOLR_CLASSIFICATION_CORE=mir-classifications
+      - ENABLE_SOLR_CLOUD=true
+      - SOLR_ADMIN_USER=${SOLR_ADMIN_USER}
+      - SOLR_ADMIN_PASSWORD=${SOLR_ADMIN_PASSWORD}
+      - SOLR_SEARCH_USER=${SOLR_SEARCH_USER}
+      - SOLR_SEARCH_PASSWORD=${SOLR_SEARCH_PASSWORD}
+      - SOLR_INDEX_USER=${SOLR_INDEX_USER}
+      - SOLR_INDEX_PASSWORD=${SOLR_INDEX_PASSWORD}
+      - TIKASERVER_URL=http://tika:9998/
       - XMX=${MIR_XMX}
       - XMS=${MIR_XMS}
       - MIR_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8295

docker-entrypoint.sh

@@ -72,6 +72,15 @@ function setOrAddProperty() {
     KEY=$1
     VALUE=$2
 
+    if [ -z "$VALUE" ]; then
+      # remove property
+      sed -ri "/$KEY/d" "${MYCORE_PROPERTIES}"
+      return
+    elif [ -z "$KEY" ]; then
+      echo "No Key given. Skip setting property."
+      return
+    fi
+
     if grep -q "$KEY=" "${MYCORE_PROPERTIES}" ; then
       ESCAPED_KEY=$(echo "${KEY}" | sed 's/\//\\\//g')
       ESCAPED_VALUE=$(echo "${VALUE}" | sed 's/\//\\\//g')
@@ -116,6 +125,45 @@ function setDockerValues() {
       setOrAddProperty "MCR.JPA.DefaultSchema" "${HIBERNATE_SCHEMA}"
     fi
 
+    if [ -n "${SOLR_ADMIN_USER}" ]; then
+          setOrAddProperty "MCR.Solr.Server.Auth.Admin.Class" "org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication"
+          setOrAddProperty "MCR.Solr.Server.Auth.Admin.Username" "${SOLR_ADMIN_USER}"
+          setOrAddProperty "MCR.Solr.Server.Auth.Admin.Password" "${SOLR_ADMIN_PASSWORD}"
+    else
+          setOrAddProperty "MCR.Solr.Server.Auth.Admin.Class"
+          setOrAddProperty "MCR.Solr.Server.Auth.Admin.Username"
+          setOrAddProperty "MCR.Solr.Server.Auth.Admin.Password"
+    fi
+
+
+    if [ -n "${SOLR_INDEX_USER}" ]; then
+          setOrAddProperty "MCR.Solr.Server.Auth.Index.Class" "org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication"
+          setOrAddProperty "MCR.Solr.Server.Auth.Index.Username" "${SOLR_INDEX_USER}"
+          setOrAddProperty "MCR.Solr.Server.Auth.Index.Password" "${SOLR_INDEX_PASSWORD}"
+    else
+          setOrAddProperty "MCR.Solr.Server.Auth.Index.Class"
+          setOrAddProperty "MCR.Solr.Server.Auth.Index.Username"
+          setOrAddProperty "MCR.Solr.Server.Auth.Index.Password"
+    fi
+
+    if [ -n "${SOLR_SEARCH_USER}" ]; then
+          setOrAddProperty "MCR.Solr.Server.Auth.Search.Class" "org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication"
+          setOrAddProperty "MCR.Solr.Server.Auth.Search.Username" "${SOLR_SEARCH_USER}"
+          setOrAddProperty "MCR.Solr.Server.Auth.Search.Password" "${SOLR_SEARCH_PASSWORD}"
+    else
+          setOrAddProperty "MCR.Solr.Server.Auth.Search.Class"
+          setOrAddProperty "MCR.Solr.Server.Auth.Search.Username"
+          setOrAddProperty "MCR.Solr.Server.Auth.Search.Password"
+    fi
+
+    if [ -n "${TIKASERVER_URL}" ]; then
+      setOrAddProperty "MCR.Solr.Tika.ServerURL" "${TIKASERVER_URL}"
+      setOrAddProperty "MCR.Solr.FileIndexStrategy" "org.mycore.solr.index.file.tika.MCRTikaSolrFileStrategy"
+    else
+      setOrAddProperty "MCR.Solr.Tika.ServerURL"
+      setOrAddProperty "MCR.Solr.FileIndexStrategy"
+    fi
+
     setOrAddProperty "MCR.JPA.Hbm2ddlAuto" "update"
     setOrAddProperty "MCR.JPA.PersistenceUnit.mir.Class" "org.mycore.backend.jpa.MCRSimpleConfigPersistenceUnitDescriptor"
     setOrAddProperty "MCR.JPA.PersistenceUnitName" "mir"
@@ -164,6 +212,17 @@ function setUpMyCoRe {
     /opt/mir/mir/bin/mir.sh create configuration directory
     setDockerValues
     setupLog4jConfig
+
+    # ENABLE_SOLR_CLOUD
+    if [[ "$ENABLE_SOLR_CLOUD" == "true" ]]
+    then
+      echo "upload local config set for main" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
+      echo "upload local config set for classification" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
+      echo "create collection for core main" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
+      echo "create collection for core classification" >> "${MCR_CONFIG_DIR}setup-solr-cloud.txt"
+      /opt/mir/mir/bin/mir.sh process /mcr/home/setup-solr-cloud.txt
+    fi
+
     /opt/mir/mir/bin/setup.sh
 }
 

mir-cli/src/main/config/setup-solrcloud.txt

@@ -0,0 +1,4 @@
+upload local config set for main
+upload local config set for classification
+create collection for core main
+create collection for core classification

mir-it/pom.xml

@@ -163,6 +163,18 @@
           </environmentVariables>
         </configuration>
         <executions>
+          <execution>
+            <id>setup-solr-cloud</id>
+            <goals>
+              <goal>exec</goal>
+            </goals>
+            <phase>pre-integration-test</phase>
+            <configuration>
+              <workingDirectory>${project.build.directory}/dependency/mir-cli-${project.version}</workingDirectory>
+              <executable>bin${file.separator}mir.${script.suffix}</executable>
+              <arguments>process ${project.build.directory}/dependency/mir-cli-${project.version}/config/setup-solrcloud.txt</arguments>
+            </configuration>
+          </execution>
           <execution>
             <id>setup-cli</id>
             <goals>

mir-it/src/test/integration/mycore.properties

@@ -1,2 +1,14 @@
 MCR.Solr.ServerURL=http\://localhost\:${solr.port}/
 MCR.Solr.DelayIndexing_inMS=200
+
+MCR.Solr.Server.Auth.Admin.Class=org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication
+MCR.Solr.Server.Auth.Admin.Password=alleswirdgut
+MCR.Solr.Server.Auth.Admin.Username=admin
+MCR.Solr.Server.Auth.Index.Class=org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication
+MCR.Solr.Server.Auth.Index.Password=alleswirdgut
+MCR.Solr.Server.Auth.Index.Username=indexer
+MCR.Solr.Server.Auth.Search.Class=org.mycore.solr.auth.MCRSolrBasicPropertyAuthentication
+MCR.Solr.Server.Auth.Search.Password=alleswirdgut
+MCR.Solr.Server.Auth.Search.Username=searcher
+
+

mir-it/src/test/java/org/mycore/mir/it/tests/MIRITBase.java

@@ -68,6 +68,7 @@ protected SolrClient getSolrClient() {
 
     protected static LukeResponse getLukeResponse(Core core) throws IOException, SolrServerException {
         LukeRequest request = new LukeRequest();
+        request.setBasicAuthCredentials("admin", "alleswirdgut");
         request.setNumTerms(0);
         request.setShowSchema(false);
         final LukeResponse lukeResponse = request.process(SOLR_CLIENT, core.getCoreName());

mir-webapp/src/main/solr/Dockerfile

@@ -0,0 +1,7 @@
+FROM solr:8.11
+USER root
+RUN apt-get update && \
+    apt-get -y install sudo
+COPY --chown=root:root docker-entrypoint.sh ./
+RUN chmod 555 docker-entrypoint.sh
+CMD ["bash", "docker-entrypoint.sh"]

mir-webapp/src/main/solr/docker-entrypoint.sh

@@ -0,0 +1,108 @@
+#!/usr/bin/bash
+#
+# This file is part of ***  M y C o R e  ***
+# See http://www.mycore.de/ for details.
+#
+# MyCoRe is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# MyCoRe is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with MyCoRe.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+set -e
+unset SOLR_USER SOLR_UID SOLR_GROUP SOLR_GID \
+      SOLR_CLOSER_URL SOLR_DIST_URL SOLR_ARCHIVE_URL SOLR_DOWNLOAD_URL SOLR_DOWNLOAD_SERVER SOLR_KEYS SOLR_SHA512
+
+function fixDirectoryRights() {
+  find "$1" \! -user "$2" -exec chown "$2:$2" '{}' +
+}
+
+echo "Running solr entry script as user: $(whoami)"
+if [ "$(id -u)" -eq 0 ]; then
+  fixDirectoryRights /var/solr/ solr
+  exec /usr/bin/sudo -E -u solr "PATH=$PATH" "$(pwd)/$0";
+  exit 0;
+fi
+
+function solrpass() {
+  printf "%s %s" "$(echo -n "$2$1"|openssl dgst -sha256 -binary|openssl dgst -sha256 -binary|openssl base64)" "$(echo -n "$2"|openssl base64)"
+}
+
+secruity_json=/var/solr/data/security.json;
+
+
+echo "{" > $secruity_json
+echo "  \"authentication\":{" >> $secruity_json
+echo "    \"blockUnknown\": true," >> $secruity_json
+echo "    \"class\":\"solr.BasicAuthPlugin\"," >> $secruity_json
+echo "    \"credentials\": {" >> $secruity_json
+
+if [ -n "$SOLR_SEARCH_USER" ]; then
+  echo "      \"${SOLR_SEARCH_USER}\":\"$(solrpass $SOLR_SEARCH_PASSWORD $(openssl rand 10))\"," >> $secruity_json
+fi
+
+if [ -n "$SOLR_INDEX_USER" ]; then
+  echo "      \"${SOLR_INDEX_USER}\":\"$(solrpass $SOLR_INDEX_PASSWORD $(openssl rand 10))\"," >> $secruity_json
+fi
+
+if [ -n "$SOLR_ADMIN_USER" ]; then
+  echo "      \"${SOLR_ADMIN_USER}\":\"$(solrpass $SOLR_ADMIN_PASSWORD $(openssl rand 10))\"" >> $secruity_json
+fi
+
+echo "    }," >> $secruity_json
+echo "    \"realm\":\"My Solr users\"," >> $secruity_json
+echo "    \"forwardCredentials\": false" >> $secruity_json
+echo "  }," >> $secruity_json
+echo "  \"authorization\":{" >> $secruity_json
+echo "    \"class\":\"solr.RuleBasedAuthorizationPlugin\"," >> $secruity_json
+echo "    \"permissions\":[" >> $secruity_json
+
+if [ -n "$SOLR_SEARCH_USER" ]; then
+  echo "      {" >> $secruity_json
+  echo "        \"name\":\"read\"," >> $secruity_json
+  echo "        \"role\":[\"searcher\",\"indexer\",\"admin\"]" >> $secruity_json
+  echo "      }," >> $secruity_json
+fi
+
+if [ -n "$SOLR_INDEX_USER" ]; then
+  echo "      {" >> $secruity_json
+  echo "        \"name\":\"update\"," >> $secruity_json
+  echo "        \"role\":[\"indexer\",\"admin\"]" >> $secruity_json
+  echo "      }," >> $secruity_json
+fi
+
+if [ -n "$SOLR_ADMIN_USER" ]; then
+  echo "      {" >> $secruity_json
+  echo "        \"name\":\"all\"," >> $secruity_json
+  echo "        \"role\":\"admin\"" >> $secruity_json
+  echo "      }" >> $secruity_json
+fi
+echo "]," >> $secruity_json
+echo "    \"user-role\":{" >> $secruity_json
+
+if [ -n "$SOLR_SEARCH_USER" ]; then
+  echo "      \"${SOLR_SEARCH_USER}\":\"searcher\"," >> $secruity_json
+fi
+
+if [ -n "$SOLR_INDEX_USER" ]; then
+  echo "      \"${SOLR_INDEX_USER}\":\"indexer\"," >> $secruity_json
+fi
+
+if [ -n "$SOLR_ADMIN_USER" ]; then
+  echo "      \"${SOLR_ADMIN_USER}\":\"admin\"" >> $secruity_json
+fi
+
+echo "    }" >> $secruity_json
+echo "  }" >> $secruity_json
+echo "}" >> $secruity_json
+
+(/opt/docker-solr/scripts/wait-for-solr.sh;/opt/solr/bin/solr zk cp $secruity_json zk:security.json -z localhost:9983)&
+/opt/docker-solr/scripts/solr-foreground -c;

mir-webapp/src/main/solr/security.json

@@ -0,0 +1,42 @@
+{
+  "authentication": {
+    "blockUnknown": true,
+    "class": "solr.BasicAuthPlugin",
+    "credentials": {
+      "searcher": "0QWhQdOGUMTkjxqE0rPodHKa2gUEnRZnZ837YP/L4aw= 7A==",
+      "indexer": "0QWhQdOGUMTkjxqE0rPodHKa2gUEnRZnZ837YP/L4aw= 7A==",
+      "admin": "0QWhQdOGUMTkjxqE0rPodHKa2gUEnRZnZ837YP/L4aw= 7A=="
+    },
+    "realm": "My Solr users",
+    "forwardCredentials": false
+  },
+  "authorization": {
+    "class": "solr.RuleBasedAuthorizationPlugin",
+    "permissions": [
+      {
+        "name": "read",
+        "role": [
+          "searcher",
+          "indexer",
+          "admin"
+        ]
+      },
+      {
+        "name": "update",
+        "role": [
+          "indexer",
+          "admin"
+        ]
+      },
+      {
+        "name": "all",
+        "role": "admin"
+      }
+    ],
+    "user-role": {
+      "searcher": "searcher",
+      "indexer": "indexer",
+      "admin": "admin"
+    }
+  }
+}