Containers

OpenShift

Secrets

oc create secret generic broker --from-file=/home/ppanero/workspace/search/test/certs/ppanero-bundle.pem

Failed pipeline in Gitlab-CI

When you first create a project you need to manually create the image before importing via GitLab-CI or edit the command to add the --confirm which will create the image if it doesn't exists. In the CERN Search projects, its not added to avoid creation of images by mistake, which would lead to possible silent errors.

Sample error message:

$ oc import-image ${RESOURCE}:${CI_COMMIT_TAG:-latest} --token=${TOKEN} --server=${OPENSHIFT_SERVER} -n ${NAMESPACE}
error: no image stream named "web-indexer" exists, pass --confirm to create and import
ERROR: Job failed: exit code 1

Tika's high memory usage

Tika runs java inside the container. The following flags incentive java's garbage collector GC to release more memory. See more: Encouraging the JVM to Release Unused Memory to the Operating System.

JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=5 -XX:AdaptiveSizePolicyWeight=90  -Xms512m -Xmx4g

The following flags help against tika's memory leaks. See more: Robust to OOMs, Infinite Loops and Memory Leaks.

OPTS: -spawnChild -maxFiles 100000

Worker's high memory usage

Number of threads may need to be reduced if larger files are expected to be processed.