Kubernetes Example Deployment ============================= This example deploys an opensiem on a local kubernetes cluster with helm: * a kafka cluster with the topics for the example (``examples/k8s/templates/kafka.yml``) * opensearch and opensearch dashboards (official opensearch helm charts) * a logprep ``connector``, which accepts events via http and writes them to kafka * a ``logprep`` instance, which reads the events from kafka and writes them to opensearch We use minikube here, every other kubernetes environment should do the job as well. You need :code:`minikube`, :code:`kubectl` and :code:`helm`, see the installation guides of `minikube `_, `kubectl `_ and `helm `_. The cluster needs about 8 CPUs and 16 GB of memory. Start Minikube -------------- Minikube runs the cluster in a container, with docker or podman on Linux and MacOS: .. tab-set:: :sync-group: engine .. tab-item:: Docker :sync: docker On MacOS, give Docker Desktop at least as many CPUs and more memory than the cluster (Settings, Resources). .. code-block:: bash minikube config set driver docker .. tab-item:: Podman :sync: podman Minikube needs rootful podman. On MacOS, configure the podman machine with more memory than the cluster: .. code-block:: bash podman machine stop podman machine set --rootful --cpus 8 --memory 20480 podman machine start On Linux, minikube runs :code:`sudo -n podman`, which requires passwordless sudo for podman, see the `podman driver `_. .. code-block:: bash minikube config set driver podman .. code-block:: bash minikube config set container-runtime containerd minikube config set cpus 8 minikube config set memory 16GB minikube start If minikube fails with :code:`no container with name or ID "minikube" found`, a previous cluster is gone, remove it with :code:`minikube delete` and start again. Enable the ingress controller and wait until it is ready, otherwise installing the example fails with :code:`failed calling webhook "validate.nginx.ingress.kubernetes.io"`: .. code-block:: bash minikube addons enable ingress kubectl wait --namespace ingress-nginx --for=condition=ready pod \ --selector=app.kubernetes.io/component=controller --timeout=180s Prepare the Example ------------------- The logprep chart deploys a prometheus :code:`PodMonitor`, which needs its custom resource definition in the cluster: .. code-block:: bash kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.94.2/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml Fetch the subcharts, from the root of the repository: .. code-block:: bash helm repo add opensearch https://opensearch-project.github.io/helm-charts/ helm dependency update ./examples/k8s Build a Local Image (optional) ------------------------------ Only needed to run logprep from your working copy, skip it to use the latest release. The image has to match the cluster: linux, on the architecture of your container engine. Building it with nix in a linux container takes care of both, on any host. Then tag it as :code:`examples/k8s/values-dev.yaml` expects and load it into the cluster. .. tab-set:: :sync-group: engine .. tab-item:: Docker :sync: docker .. code-block:: bash docker run --rm -v "$PWD":/src:ro -w /src docker.io/nixos/nix sh -c \ 'cat $(nix --extra-experimental-features "nix-command flakes" build \ "path:.#packages.$(uname -m)-linux.docker.python311" --no-link --print-out-paths)' \ > /tmp/logprep-image.tar.gz docker load < /tmp/logprep-image.tar.gz docker tag logprep:py3.11 local/logprep:latest docker save -o /tmp/logprep.tar local/logprep:latest minikube image load /tmp/logprep.tar .. tab-item:: Podman :sync: podman .. code-block:: bash podman run --rm -v "$PWD":/src:ro -w /src docker.io/nixos/nix sh -c \ 'cat $(nix --extra-experimental-features "nix-command flakes" build \ "path:.#packages.$(uname -m)-linux.docker.python311" --no-link --print-out-paths)' \ > /tmp/logprep-image.tar.gz podman load < /tmp/logprep-image.tar.gz podman tag logprep:py3.11 docker.io/local/logprep:latest podman save -o /tmp/logprep.tar docker.io/local/logprep:latest minikube image load /tmp/logprep.tar Podman needs the full name :code:`docker.io/local/logprep:latest`, a short :code:`local/logprep:latest` becomes :code:`localhost/local/logprep:latest`, which the cluster does not find. With nix on a linux host, building directly is faster, e.g. :code:`nix build .#packages.x86_64-linux.docker.python311 -o image`. A plain :code:`nix build .#docker.python311` on MacOS builds for MacOS, which fails in the cluster with :code:`exec format error`. To try the dev values without building, pull a released image, e.g. :code:`ghcr.io/fkie-cad/logprep:py3.11-latest`, and tag, save and load it the same way instead of :code:`logprep:py3.11`. Install the Example ------------------- .. tab-set:: :sync-group: image .. tab-item:: Released image :sync: release Pulls the latest release of logprep from :code:`ghcr.io/fkie-cad/logprep`: .. code-block:: bash helm install opensiem examples/k8s .. tab-item:: Local image :sync: local Uses the image built above and smaller settings for a local cluster: .. code-block:: bash helm install opensiem examples/k8s --values examples/k8s/values-dev.yaml Wait until everything is running, this takes a few minutes: .. code-block:: bash kubectl wait --for=condition=ready pod --all --timeout=15m kubectl get pods Logprep restarts a few times until kafka and opensearch are ready, that is expected. To change the configuration later, use :code:`helm upgrade` with the same arguments as :code:`helm install`, or start over with :code:`helm uninstall opensiem`. Send Events ----------- Forward the ports of the connector and opensearch in the background: .. code-block:: bash kubectl port-forward svc/opensiem-connector-http-input 9000:9000 > /dev/null & kubectl port-forward svc/opensiem-opensearch 9200:9200 > /dev/null & Send the example events to the connector: .. code-block:: bash curl --data-binary @examples/exampledata/input_logdata/logclass/test_input.jsonl \ http://localhost:9000/lab/123/ABC/auditlog Logprep writes them to opensearch within about a minute: .. code-block:: bash curl -s http://localhost:9200/processed/_count With logprep installed locally, :code:`logprep generate http` sends larger amounts of events: .. code-block:: bash logprep generate http --input-dir ./examples/exampledata/input_logdata/ \ --target-url http://localhost:9000 --events 100 --batch-size 10 Stop the port forwarding when you are done: .. code-block:: bash kill %1 %2 Use the Ingress --------------- The example also exposes the connector and opensearch dashboards via ingress. Make the host names resolvable: .. tab-set:: :sync-group: os .. tab-item:: Linux :sync: linux .. code-block:: bash echo "$(minikube ip) connector.opensiem dashboards.opensiem" | sudo tee -a /etc/hosts .. tab-item:: MacOS :sync: macos The cluster's network is not reachable from MacOS, :code:`minikube tunnel` forwards the ingress to localhost: .. code-block:: bash echo "127.0.0.1 connector.opensiem dashboards.opensiem" | sudo tee -a /etc/hosts Then run :code:`minikube tunnel` in a separate terminal, it asks for your sudo password and has to keep running. Then the connector is available at :code:`http://connector.opensiem` and opensearch dashboards at `http://dashboards.opensiem `_: .. code-block:: bash curl -v http://connector.opensiem/health