Development merge for 2.4.0 (#751)

whotwagner · hoeldg · landauermax · web-flow · commit b04a779f4727 · 2021-06-09T10:08:59.000+02:00
* Main merge (#742) * Development (#595) * Added the variables min_values_cors_thres and new_vals_alarm_thres * Fixed DeepSource-issue * Implemented the chisquare test and added the parameters used_homogeneity_test and alpha_chisquare_test * Fixed DeepSource-issues * Add EventSequenceDetector * Fix deepsource issues * Fix deepsource issues * Add unit test for EventSequenceDetector * Fix deepsource issues. * Fix deepsource issues. * Remove print statement. * Removed the prints and fixed typos * Refactoring of the VTD-output * Fixed DeepSource-issues * Fixed DeepSource-issue * Add frequency detector * Fix deepsource issues * Fix deepsource issues * Fix deepsource issues * Fix deepsource issues * Fix confidence_factor * Fixed warnigs due to files in Persistency-Directory * fixed acl-problems in dockerfile and autocreate /var/lib/aminer/log * added simple test for dockercontainer * negate result of the timeout-command. 1 is okay. 0 must be an error * added bullseye-tests * make tmp-dir in debian-bullseye-test and debian-buster-test unique * fixed various deepsource-issues * added changes to changelog and modified version-string * modified openjdk to work with bullseye * Restructured some output and added a confidence value for messages related to anomalies * fixed debian-changelog and removed changelog from /usr/share/doc/aminer * Improve test documentation * Fix deepsource issues * Fix deepsource issues * updated kafka download link. * Fix deepsource issues * Fix deepsource issues * fixed kafka-version in coverage-tests * Disabled Coverage-Test Due to problems with kafka, the coverage-tests were temporarily disabled. * renamed AMiner to aminer or Aminer. * fixed issues. * removed link. * added testing manifest to Readme. * Refined the ouptut of the VCD * fixed variable names. * Fixed Deepsource-issues * Fixed Deepsource issues * Fixed Deepsource issue * renamed AMiner to aminer or Aminer. * fixed issues. * removed link. * fixed the warning in the VCD * fixed the warning in the VCD * Fixed the Warning in the VTD * Fixed the method pick_cor_match_disc_vals * added close function to LogStream. * Changed the unit-test for the function matchDiscVals * Add a query in the VCD to check if the ETD skipped the currently processed line * Changed the variable w_rel_bt_results into w_rel_ht_results * 007 yaml kafka event handler (#446) * added KafkaEventHandler to template_config.yml * added kafka client config. * added KafkaEventHandler to template_config.py and template_config.yml. * fixed kafka config path. * fixed deepsource issues. * fixed errors in template config. * fixed error in template_config. * added KafkaEventHandler to suppress_config.yml. * renamed kafka_client.conf to kafka-client.conf. * fixed error with KafkaEventHandler in YamlConfig. * fixed errors in kafka-client.conf * changed kafka topic for yaml tests. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * removed kafka-client.conf in unittest script. * removed kafka-client.conf in unittest script. (#447) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * make wiki-tests branch-dependent * fixed error in multiple detectors and added unittest. * fixed errors. * fixed error. * added flag to mute persistence_id warnings in tests. * fixed deepsource issues. * fixed error. * 009 specify yml remove imports (#451) * implemented first version of new yaml schema validation. Analysis validation and EventHandler Validation is still missing. * implemented EventHandler validation. * fixed error. * implemented AnalysisValidationSchema. * fixed AnalysisValidationSchema. * specified ParserValidationSchema. * specified EventHandlerValidationSchema. * fixed default values for EventHandler and Parser. * specified Analysis schemas. * specified Analysis Schemas. * fixed skipped deepsource issues. * removed imports from analysis/__init__.py. * fixed imports in AnalysisComponentsPerformanceTest. * removed analysis/__init__.py. * fixed typo. * fixed analysis import errors in configs. * fixed errors in AnalysisChild. * fixed error. * fixed up aminer.events imports in all classes. * fixed aminer.event imports. * fixed import error in aminer.py. * added __init__.py. * fixed deepsource issues. * removed imports from input/__init__.py. * removed util/__init__.py. * fixed typo. * fixed typo. * fixed import error. * removed parsing/__init__.py. * fixed error in yaml. * fixed yaml error. * fixed import error. * fixed deepsource issues. * specified integer values and added comments. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * Add documentation for params in ESD. (#455) * fixed pr template. (#457) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 013 multiline support (#459) * added eol_sep parameter to the ByteStreamLineAtomizer. * implemented json_format parameter. * added json_format for multiline json support. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues * added warning if invalid json is found. * added line number to warning. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * changed all shutil.chown to os.chown without using dir_fd. (#440) * changed all shutil.chown to os.chown without using dir_fd. * secured logging with dir_fd and follow_symlinks=False. * fixed logging code in TestBase. * fixed errors and moved imports to the top of the file. * fixed deepsource issues. * fixed error in TestBase. * changed os.makedirs to os.mkdir and used dir_fd. * fixed security issues. * opened persistence_dir in initialize loggers. * removed wrong import. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 010 add kafka event handler to integration test2 (#456) * added KafkaEventHandler to the integration test 2. * renamed hostname in integration test. * fixed indentation. * fixed indentation error. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * fixed mail tests. (#458) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 014 fix wiki tests (#465) * extended the Getting Started Tests. * extended try it out wiki tests. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 015 add tests for esd efd (#468) * added default value to target_path_list in EFD and ESD. * added EFD to the demo configs. * added the ESD to the aminer demo. * added ESD and EFD to AnalysisChild. * added performance tests for the ESD and EFD. * fixed unittests. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * fixed error in TryItOut test. (#469) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * fixed encoding errors. (#471) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * Fix: too many open directory-handles (#472) This patch fixes a problem with too many open file-handles. This problem occurs with open directory-handles of resource-files. Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> * fixed encoding error. (#473) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * added sync_wait_time parameter. (#484) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 020 create elasticsearch demo (#476) * added consume_data parameter to the DelimitedDataModelElement. * converted all tabs in Jenkinsfile to spaces. * added consume_delimiter to yaml. * fixed configs. * added unfinished version of JsonModelElement. * finished JsonModelElement. * renamed optional_key_identifier to optional_key_prefix. * fixed deepsource issues. * fixed error. * added first unittest for JsonModelElement. * created unittests for JsonModelElement. * fixed deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 027 fix dev tests (#491) * fixed testingwrapper. * extended usage string. * fixed runAminerElasticsearchDemo.sh. * renamed elasticsearch to jsoninput. * fixed error in test. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 028 fix dev (#495) * renamed config in aminerJsonInputDemo. * changed wrong path. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 022 check4updates (#482) * added check4updates parameter. * renamed check4updates to check. * added installation of urllib3 to all dockerfiles. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 031 json parsing (#502) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed encoding error in ByteStreamLineAtomizer. * fixed json error when key does not exist. * added unittest to JsonModelElementTest. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * changed default value of verbose to True. (#485) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 030 improve error output yaml (#501) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * added filter to error output in yaml. * removed multiple outputs of yaml errors. * fixed unittest. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 035 fix suspend mode test (#507) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed suspend mode tests. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 026 detection timestamp (#490) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed Kafka output of integration test. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 029 no args test (#496) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * removed args from apacheModel in template_config. * fixed kafka tests. * fixed kafka tests. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 032 test whitespace constant (#503) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * changed test to check WHITESPACE usage. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 034 empty validation schemas (#506) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * extended schemas to check if strings are empty. * fixed names in demo configs. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 036 New JSON Implementation + Tests (#511) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * implemented JsonStateMachine to validate json input. * extended json demo and added tests if they work properly. * fixed Deepsource issues. * fixed error in json parsing. * fixed deepsource issue. * fixed Deepsource issues. * fixed deepsource issues. * added more examples for json processing. * fixed error in JsonStateMachine. * added max_line_length to json_format. * added unittests for hex_machine. * added unittests for the ByteStreamLineAtomizer and JsonStateMachine. * added unittests for the hex_machine and utf8_machine. * added unittests for the constant_machine. * added tests for the numbers_machine. * added array_machine tests. * added tests for object_machine and json_machine. * fixed up JsonStateMachine. * removed duplicate tests from aminerJsonInputDemo. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 021 specify log resource list (#481) * specified types of LogResourceList. * fixed unittest. * fixed deepsource issues. * fixed regex. * added more examples. * added error messages for test24. * fixed deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> Co-authored-by: Max Landauer <landauermax@users.noreply.github.com> * 025 specify base schema (#489) * specified BaseSchema.py. * fixed testingwrapper. * undid changes in testingwrapper. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> Co-authored-by: Max Landauer <landauermax@users.noreply.github.com> * 040 howto tests (#524) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * implemented Tests for HowTo FrequencyDetector and SequenceDetector. * added tests for wiki HowTo MissingMatchPathValueDetector. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 045 hotfix tests (#536) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed Test. * fixed suspendModeTest Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * Fix allow multiple values per id path * 046 hotfix2 (#537) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed errors in HowTo tests. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 039 json me error output (#520) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed JsonModelElement error and specified output in the VerboseUnparsedAtomHandler. * extended sleep time in Suspend mode test. * fixed JsonModelElement when key is missing. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 037 file permissions (#518) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed permissions. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 041 rewrite any byte dme (#526) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * rewrote unittests for the AnyByteDME. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 033 rewrite FixedDME (#517) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * changed test to check WHITESPACE usage. * added new testcase design for FixedDME. * rewrote some unittests for the FixedDME. * finished rewriting unittests for the FixedDME. * fixed Deepsource issues. * added unittests for simple methods of FixedDME. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 042 rewrite base64 me tests (#530) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * changed test to check WHITESPACE usage. * added new testcase design for FixedDME. * rewrote some unittests for the FixedDME. * finished rewriting unittests for the FixedDME. * fixed Deepsource issues. * added unittests for simple methods of FixedDME. * added some unittests to the Base64 Model Element. * Rewrote unittests for the Base64StringModelElement. * fixed Deepsource issues. * fixed unittest without exact length. * added performance test and reimplemented the Base64StringModelElement. * fixed Deepsource issues. * fixed Deepsource issues. * fixed Deepsource issues. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 038 yaml duplicate ids (#519) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * added checks in YamlConfig if id was used multiple times for Analysis, Parser and EventHandlers. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 047 base64 (#550) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * added check in Base64StringModelElement if data is encoded utf-8. * commented changes. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 044 json demos (#535) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed JsonModelElement error and specified output in the VerboseUnparsedAtomHandler. * extended sleep time in Suspend mode test. * fixed JsonModelElement when key is missing. * fixed Deepsource issues. * added special key ALLOW_ALL for JsonModelElement. * fixed zero start handling of the JsonStateMachine. * fixed errors in JSON input data handling and JsonStateMachine. * fixed data. * created first config for elastic.log and fixed more errors. * finished json-elastic-demo.yml. * added config for eve.json and fixed more JSON parsing errors. * fixed more errors. * added config for journal.log. * added wazuh config. * added logging to the JsonModelElement. * added unittest for JsonModelElement with different key orders. * fixed unittests. * fixed JsonModelElement. * fixed unittest. * fixed runAminerJsonInputDemo and added runJsonDemo script. * added Json Tests to Jenkinsfile and testingwrapper. * removed debug outputs. * fixed Deepsource issues. * fixed Deepsource issues. * fixed json demo configs. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * HOTFIX JENKINSFILE (#556) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed JsonModelElement error and specified output in the VerboseUnparsedAtomHandler. * extended sleep time in Suspend mode test. * fixed JsonModelElement when key is missing. * fixed Deepsource issues. * added special key ALLOW_ALL for JsonModelElement. * fixed zero start handling of the JsonStateMachine. * fixed errors in JSON input data handling and JsonStateMachine. * fixed data. * created first config for elastic.log and fixed more errors. * finished json-elastic-demo.yml. * added config for eve.json and fixed more JSON parsing errors. * fixed more errors. * added config for journal.log. * added wazuh config. * added logging to the JsonModelElement. * added unittest for JsonModelElement with different key orders. * fixed unittests. * fixed JsonModelElement. * fixed unittest. * fixed runAminerJsonInputDemo and added runJsonDemo script. * added Json Tests to Jenkinsfile and testingwrapper. * removed debug outputs. * fixed Deepsource issues. * fixed Deepsource issues. * fixed json demo configs. * fixed jenkinsfile Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * HOTFIX 2 (#559) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed JsonModelElement error and specified output in the VerboseUnparsedAtomHandler. * extended sleep time in Suspend mode test. * fixed JsonModelElement when key is missing. * fixed Deepsource issues. * added special key ALLOW_ALL for JsonModelElement. * fixed zero start handling of the JsonStateMachine. * fixed errors in JSON input data handling and JsonStateMachine. * fixed data. * created first config for elastic.log and fixed more errors. * finished json-elastic-demo.yml. * added config for eve.json and fixed more JSON parsing errors. * fixed more errors. * added config for journal.log. * added wazuh config. * added logging to the JsonModelElement. * added unittest for JsonModelElement with different key orders. * fixed unittests. * fixed JsonModelElement. * fixed unittest. * fixed runAminerJsonInputDemo and added runJsonDemo script. * added Json Tests to Jenkinsfile and testingwrapper. * removed debug outputs. * fixed Deepsource issues. * fixed Deepsource issues. * fixed json demo configs. * fixed jenkinsfile * fixed Jenkinsfile. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * HOTFIX 3 (#560) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * fixed JsonModelElement error and specified output in the VerboseUnparsedAtomHandler. * extended sleep time in Suspend mode test. * fixed JsonModelElement when key is missing. * fixed Deepsource issues. * added special key ALLOW_ALL for JsonModelElement. * fixed zero start handling of the JsonStateMachine. * fixed errors in JSON input data handling and JsonStateMachine. * fixed data. * created first config for elastic.log and fixed more errors. * finished json-elastic-demo.yml. * added config for eve.json and fixed more JSON parsing errors. * fixed more errors. * added config for journal.log. * added wazuh config. * added logging to the JsonModelElement. * added unittest for JsonModelElement with different key orders. * fixed unittests. * fixed JsonModelElement. * fixed unittest. * fixed runAminerJsonInputDemo and added runJsonDemo script. * added Json Tests to Jenkinsfile and testingwrapper. * removed debug outputs. * fixed Deepsource issues. * fixed Deepsource issues. * fixed json demo configs. * fixed jenkinsfile * fixed Jenkinsfile. * fixed Jenkinsfile. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 048 missing match path det (#555) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * changed MissingMAtchPathValueDetector to use a list of paths instead one single path. * fixed Deepsource issues. * added test with multiple lines. * fixed configs. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 043 rewrite date time me tests (#531) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * changed test to check WHITESPACE usage. * added new testcase design for FixedDME. * rewrote some unittests for the FixedDME. * finished rewriting unittests for the FixedDME. * fixed Deepsource issues. * added unittests for simple methods of FixedDME. * designed test cases for the DTME. * designed new test cases and extended existing ones from DateTimeModelElement. * added a new test case. * added unittests and changed initialization from None to 0 for DateTimeModelElement * added more unittests to DateTimeME. Rewrote DummyMatchContext and adapted all rewritten unittests. * added more unittests. * added more unittests. * added more unittests. * added the last unittests. * implemented missing tests and text_locale. * fixed DateTimeModelElement. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 049 log prefix (#558) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * added default value for log_line_prefix. * fixed Deepsource issues. * fixed deepsource issues. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 050 aminer demo fail on error (#562) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * changed runAminerDemo to fail if an error occurred. * fixed jsonConverterHandler demo config. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 051 remove sudo installed (#563) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * removed sudoInstalled from test scripts. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 052 suppress unparsed detector (#564) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * added suppress_unparsed config. * added suppress_unparsed to BaseSchema. * fixed error. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 053 reimplement dtme (#571) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * added handling of different timezones. * removed unneeded variable. * fixed deepsource issues. * fixed error with changed format specifier length. * removed comment. * added missing cases. * fixed MissingMatchPathValueDetector unittests. * changed datetime imports. * fixed Deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 055 esd path list optional (#575) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * made id_path_list in ESD optional. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * 054 rewrite multi locale dtme (#572) * added DetectionTimestamp to json output. * fixed unittests. * fixed unittests. * fixed KafkaEventHandlerTest. * fixed testingwrapper. * fixed runAminerElasticsearchDemo.sh * undid changes in testingwrapper. * resolved conflicts * fixed kafka tests. * added handling of different timezones. * renamed function. * removed unneeded variable. * added some tests. * fixed deepsource issues. * added more unittests for MultiLocaleDTME. * added more unittests for the MultiLocaleDTME. * fixed error with changed format specifier length. * removed comment. * added more unittests. * added more tests. * added missing cases. * fixed MissingMatchPathValueDetector unittests. * added last unittests for MultiLocaleDTME. * fixed Deespource issues. * added MultiLocaleDTME to the DTME class. Fixed some unittests. * changed datetime imports. * fixed some issues. * changed tests. * implemented MultiLocaleDateTimeModelElement. * fixed YamlConfig for DateTimeModelElement and MultiLocaleDTME. * removed old MultiLocaleDateTimeModelElement. * fixed YamlConfig for MultiLocaleDateTimeModelElement. * fixed error in VTD. * fixed VTD. * fixed VTD. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * renamed schemas to .json. (#589) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * added null value handling in JsonModelElement. (#591) * added null value handling in JsonModelElement. * fixed order of ifs. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * Add floor and ceil in EFD check (#585) * Add floor and ceil in EFD check * Fix rounding and tests. * hotfix. (#593) Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * Fix unparsed leap day in syslog parsing model (#582) * Release 2.3.0 (#594) * modified version of pyyaml in requirements.txt * New version v2.3.0 Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> * Update README.md * Update README.md * fix PYL-R1714 * removed orig-files from merge * fixed deepsource issues. (#599) * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues for VCD. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues. * fixed deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> Co-authored-by: hoeldg <georg.hoeld@ait.ac.at> Co-authored-by: root <max.landauer@ait.ac.at> Co-authored-by: 4cti0nfi9ure <16901151+4cti0nfi9ure@users.noreply.github.com> Co-authored-by: Max Landauer <landauermax@users.noreply.github.com> Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> Co-authored-by: dITySoftware <56120276+dITySoftware@users.noreply.github.com> Co-authored-by: hoeldg <66625424+hoeldg@users.noreply.github.com> * aminer.py: resolved hardcoded username and gropname where user_id and… (#614) * aminer.py: resolved hardcoded username and gropname where user_id and group_id was expected * aminer.py: fixed bug in getgrnam(). username was taken instead of groupname Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> * Release 2.3.1 * removed merge-files * removed merge-files * fixed refactoring issues in AnalysisChild * removed old json-files from json-directory * fixed problems with merging Co-authored-by: hoeldg <georg.hoeld@ait.ac.at> Co-authored-by: root <max.landauer@ait.ac.at> Co-authored-by: 4cti0nfi9ure <16901151+4cti0nfi9ure@users.noreply.github.com> Co-authored-by: Max Landauer <landauermax@users.noreply.github.com> Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> Co-authored-by: dITySoftware <56120276+dITySoftware@users.noreply.github.com> Co-authored-by: hoeldg <66625424+hoeldg@users.noreply.github.com> * prepared aminer for V2.4.0 (#745) Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> * Fixed a bug in JsonModelElement (#748) * 111 fixed handling of json objects. (#749) * fixed handling of json objects. * fixed error. * fixed error. * fixed deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> * Fix ci and docs (#750) * removed unclean merge from docs/Configuration.rst * fix for race-condition in cleanup of ci-pipeline Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> * fixed deepsource issues. (#752) * fixed deepsource issues. * fixed deepsource issues. Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> Co-authored-by: hoeldg <georg.hoeld@ait.ac.at> Co-authored-by: root <max.landauer@ait.ac.at> Co-authored-by: 4cti0nfi9ure <16901151+4cti0nfi9ure@users.noreply.github.com> Co-authored-by: Max Landauer <landauermax@users.noreply.github.com> Co-authored-by: Wolfgang Hotwagner <wolfgang.hotwagner@ait.ac.at> Co-authored-by: Ernst Leierzopf <ernst.leierzopf.fl@ait.ac.at> Co-authored-by: dITySoftware <56120276+dITySoftware@users.noreply.github.com> Co-authored-by: hoeldg <66625424+hoeldg@users.noreply.github.com>
diff --git a/Jenkinsfile b/Jenkinsfile
@@ -12,6 +12,7 @@ def  ubuntu18image = false
 def  ubuntu20image = false
 def  debianbusterimage = false
 def  debianbullseyeimage = false
+def  docsimage = false
 
 pipeline {
      agent any
@@ -192,6 +193,9 @@ pipeline {
                  BUILDDOCSDIR = sh(script: 'mktemp -p $WORKSPACE_TMP -d | tr -d [:space:]', returnStdout: true)
              }
              steps {
+                 script {
+                    docsimage = true
+                 }
                  sh "docker build -f Dockerfile -t aecid/aminer-docs:$JOB_BASE_NAME-$EXECUTOR_NUMBER-$BUILD_ID ."
                  sh "chmod 777 ${env.BUILDDOCSDIR}"
                  sh "chmod g+s ${env.BUILDDOCSDIR}"
@@ -219,8 +223,9 @@ pipeline {
            if( ubuntu20image == true ){
                sh "docker rmi aecid/aminer-ubuntu-2004:$JOB_BASE_NAME-$EXECUTOR_NUMBER-$BUILD_ID"
            }
-           // The following line is experimental. if it works, the code above can be deleted
-           sh "docker system prune -fa"
+           if( docsimage == true){
+               sh "docker rmi aecid/aminer-docs:$JOB_BASE_NAME-$EXECUTOR_NUMBER-$BUILD_ID"
+           }
          }
         }
  
diff --git a/aecid-testsuite/demo/aminer/demo-config.py b/aecid-testsuite/demo/aminer/demo-config.py
@@ -29,13 +29,13 @@
 # for the child also.
 # config_properties['AnalysisConfigFile'] = 'analysis.py'
 
-config_properties['Core.LogDir'] = '/tmp/lib/aminer/log'
+config_properties['Core.LogDir'] = '/tmp/lib/aminer/log'  # skipcq: BAN-B108
 # Read and store information to be used between multiple invocations
 # of aminer in this directory. The directory must only be accessible
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 config_properties['Core.PersistencePeriod'] = 600
 
 # Define a target e-mail address to send alerts to. When undefined,
diff --git a/aecid-testsuite/demo/aminer/jsonConverterHandler-demo-config.py b/aecid-testsuite/demo/aminer/jsonConverterHandler-demo-config.py
@@ -52,7 +52,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/demo/aminerRemoteControl/demo-config.py b/aecid-testsuite/demo/aminerRemoteControl/demo-config.py
@@ -45,7 +45,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/integration/config.py b/aecid-testsuite/integration/config.py
@@ -31,7 +31,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/integration/config21.py b/aecid-testsuite/integration/config21.py
@@ -31,7 +31,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/integration/config22.py b/aecid-testsuite/integration/config22.py
@@ -31,7 +31,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/system/performance-tests/generateSystemLogdata.py b/aecid-testsuite/system/performance-tests/generateSystemLogdata.py
@@ -7,7 +7,7 @@
 from datetime import datetime
 import multiprocessing
 
-with open('/tmp/results.csv', 'a+', buffering=100) as file:
+with open('/tmp/results.csv', 'a+', buffering=100) as file:  # skipcq: BAN-B108
     string = ''
     string += 'time,aminerCpuUsage,aminerMemUsage,'
     for i in range(multiprocessing.cpu_count()):
diff --git a/aecid-testsuite/system/performance-tests/performance-config.py b/aecid-testsuite/system/performance-tests/performance-config.py
@@ -34,7 +34,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/system/performance-tests/performance-config1.py b/aecid-testsuite/system/performance-tests/performance-config1.py
@@ -45,7 +45,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/system/performance-tests/performance-config2.py b/aecid-testsuite/system/performance-tests/performance-config2.py
@@ -53,7 +53,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # aminer will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/unit/data/YamlConfigTest.py b/aecid-testsuite/unit/data/YamlConfigTest.py
@@ -477,7 +477,7 @@ def test19_stream_printer_output_file(self):
         aminer_config.load_yaml('unit/data/configfiles/template_config.yml')
         context = AnalysisContext(aminer_config)
         context.build_analysis_pipeline()
-        self.assertEqual(context.atomizer_factory.event_handler_list[0].stream.name, '/tmp/streamPrinter.txt')
+        self.assertEqual(context.atomizer_factory.event_handler_list[0].stream.name, '/tmp/streamPrinter.txt')  # skipcq: BAN-B108
         self.assertEqual(context.atomizer_factory.event_handler_list[0].stream.mode, 'w+')
 
     def test20_suppress_output(self):
diff --git a/aecid-testsuite/unit/data/config.py b/aecid-testsuite/unit/data/config.py
@@ -34,7 +34,7 @@
 # to the 'AminerUser' but not group/world readable. On violation,
 # py will refuse to start. When undefined, '/var/lib/aminer'
 # is used.
-config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'
+config_properties['Core.PersistenceDir'] = '/tmp/lib/aminer'  # skipcq: BAN-B108
 
 # Define a target e-mail address to send alerts to. When undefined,
 # no e-mail notification hooks are added.
diff --git a/aecid-testsuite/unit/input/LogStreamTest.py b/aecid-testsuite/unit/input/LogStreamTest.py
@@ -65,7 +65,7 @@ def test3file_log_data_resource_log_stream_already_open_repositioning(self):
         In this case the logStreamFd is > 0 and repositioningData is not None.
         The stream should be repositioned to the right position.
         """
-        fd = os.open('/tmp/log.txt', os.O_RDONLY)
+        fd = os.open('/tmp/log.txt', os.O_RDONLY)  # skipcq: BAN-B108
         length = 65536
         data = os.read(fd, length)
         # skipcq: PTC-W1003
@@ -74,7 +74,7 @@ def test3file_log_data_resource_log_stream_already_open_repositioning(self):
         hash_digest = md5.digest()
         os.close(fd)
 
-        fd = os.open('/tmp/log.txt', os.O_RDONLY)
+        fd = os.open('/tmp/log.txt', os.O_RDONLY)  # skipcq: BAN-B108
         file_log_data_resource = FileLogDataResource(self.file + self.logfile, fd, 65536,
                                                      [os.fstat(fd).st_ino, length, base64.b64encode(hash_digest)])
         file_log_data_resource.fill_buffer()
diff --git a/aecid-testsuite/unit/input/client.py b/aecid-testsuite/unit/input/client.py
@@ -1,6 +1,6 @@
 from time import sleep
 import socket
-sock_name = '/tmp/test5unixSocket.sock'
+sock_name = '/tmp/test5unixSocket.sock'  # skipcq: BAN-B108
 
 sleep(0.5)
 client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
diff --git a/aecid-testsuite/unit/parsing/JsonModelElementTest.py b/aecid-testsuite/unit/parsing/JsonModelElementTest.py
@@ -268,7 +268,72 @@ def test7get_match_element_with_umlaut(self):
         self.compare_match_results(
             data, match_element, match_context, self.id_, self.path, str(value).encode(), value, match_element.children)
 
-    def test8get_match_element_no_match(self):
+    def test8get_match_element_same_value_as_key(self):
+        """Test if object with the same key-value pairs are parsed correctly."""
+        key_parser_dict = {"abc": DummyFirstMatchModelElement("first", [
+            DummyFixedDataModelElement("abc", b"abc"), DummyFixedDataModelElement("abc", b"ab"), DummyFixedDataModelElement("abc", b"bc"),
+            DummyFixedDataModelElement("abc", b"ba"), DummyFixedDataModelElement("abc", b"b"), DummyFixedDataModelElement("abc", b"d")])}
+        data = b"""{"abc":"abc"}"""
+        json_model_element = JsonModelElement(self.id_, key_parser_dict)
+        value = json.loads(data)
+        match_context = DummyMatchContext(data)
+        match_element = json_model_element.get_match_element(self.path, match_context)
+        match_context.match_string = str(value).encode()
+        match_context.match_data = data[len(match_context.match_string):]
+        self.compare_match_results(
+            data, match_element, match_context, self.id_, self.path, str(value).encode(), value, match_element.children)
+
+        data = b"""{"abc":"ab"}"""
+        json_model_element = JsonModelElement(self.id_, key_parser_dict)
+        value = json.loads(data)
+        match_context = DummyMatchContext(data)
+        match_element = json_model_element.get_match_element(self.path, match_context)
+        match_context.match_string = str(value).encode()
+        match_context.match_data = data[len(match_context.match_string):]
+        self.compare_match_results(
+            data, match_element, match_context, self.id_, self.path, str(value).encode(), value, match_element.children)
+
+        data = b"""{"abc":"bc"}"""
+        json_model_element = JsonModelElement(self.id_, key_parser_dict)
+        value = json.loads(data)
+        match_context = DummyMatchContext(data)
+        match_element = json_model_element.get_match_element(self.path, match_context)
+        match_context.match_string = str(value).encode()
+        match_context.match_data = data[len(match_context.match_string):]
+        self.compare_match_results(
+            data, match_element, match_context, self.id_, self.path, str(value).encode(), value, match_element.children)
+
+        data = b"""{"abc":"b"}"""
+        json_model_element = JsonModelElement(self.id_, key_parser_dict)
+        value = json.loads(data)
+        match_context = DummyMatchContext(data)
+        match_element = json_model_element.get_match_element(self.path, match_context)
+        match_context.match_string = str(value).encode()
+        match_context.match_data = data[len(match_context.match_string):]
+        self.compare_match_results(
+            data, match_element, match_context, self.id_, self.path, str(value).encode(), value, match_element.children)
+
+        data = b"""{"abc":"d"}"""
+        json_model_element = JsonModelElement(self.id_, key_parser_dict)
+        value = json.loads(data)
+        match_context = DummyMatchContext(data)
+        match_element = json_model_element.get_match_element(self.path, match_context)
+        match_context.match_string = str(value).encode()
+        match_context.match_data = data[len(match_context.match_string):]
+        self.compare_match_results(
+            data, match_element, match_context, self.id_, self.path, str(value).encode(), value, match_element.children)
+
+        data = b"""{"abc":"ba"}"""
+        json_model_element = JsonModelElement(self.id_, key_parser_dict)
+        value = json.loads(data)
+        match_context = DummyMatchContext(data)
+        match_element = json_model_element.get_match_element(self.path, match_context)
+        match_context.match_string = str(value).encode()
+        match_context.match_data = data[len(match_context.match_string):]
+        self.compare_match_results(
+            data, match_element, match_context, self.id_, self.path, str(value).encode(), value, match_element.children)
+
+    def test9get_match_element_no_match(self):
         """Parse not matching substring from MatchContext and check if the MatchContext was not changed."""
         json_model_element = JsonModelElement(self.id_, self.key_parser_dict)
         # missing key
@@ -311,7 +376,7 @@ def test8get_match_element_no_match(self):
         match_element = json_model_element.get_match_element(self.path, match_context)
         self.compare_no_match_results(data, match_element, match_context)
 
-    def test9element_id_input_validation(self):
+    def test10element_id_input_validation(self):
         """Check if element_id is validated."""
         self.assertRaises(ValueError, JsonModelElement, "", self.key_parser_dict)  # empty element_id
         self.assertRaises(TypeError, JsonModelElement, None, self.key_parser_dict)  # None element_id
@@ -325,7 +390,7 @@ def test9element_id_input_validation(self):
         self.assertRaises(TypeError, JsonModelElement, (), self.key_parser_dict)  # empty tuple element_id is not allowed
         self.assertRaises(TypeError, JsonModelElement, set(), self.key_parser_dict)  # empty set element_id is not allowed
 
-    def test10key_parser_dict_input_validation(self):
+    def test11key_parser_dict_input_validation(self):
         """Check if key_parser_dict is validated."""
         self.assertRaises(TypeError, JsonModelElement, self.id_, "path")  # string key_parser_dict
         self.assertRaises(TypeError, JsonModelElement, self.id_, None)  # None key_parser_dict
@@ -346,7 +411,7 @@ def test10key_parser_dict_input_validation(self):
         self.assertRaises(TypeError, JsonModelElement, self.id_, ())  # empty tuple key_parser_dict is not allowed
         self.assertRaises(TypeError, JsonModelElement, self.id_, set())  # empty set key_parser_dict is not allowed
 
-    def test11optional_key_prefix_input_validation(self):
+    def test12optional_key_prefix_input_validation(self):
         """Check if optional_key_prefix is validated."""
         self.assertRaises(ValueError, JsonModelElement, self.id_, self.key_parser_dict, "")  # empty optional_key_prefix
         self.assertRaises(TypeError, JsonModelElement, self.id_, self.key_parser_dict, None)  # None optional_key_prefix
@@ -360,7 +425,7 @@ def test11optional_key_prefix_input_validation(self):
         self.assertRaises(TypeError, JsonModelElement, self.id_, self.key_parser_dict, ())  # empty tuple optional_key_prefix is not allowed
         self.assertRaises(TypeError, JsonModelElement, self.id_, self.key_parser_dict, set())  # empty set optional_key_prefix not allowed
 
-    def test12get_match_element_match_context_input_validation(self):
+    def test13get_match_element_match_context_input_validation(self):
         """Check if an exception is raised, when other classes than MatchContext are used in get_match_element."""
         model_element = JsonModelElement(self.id_, self.key_parser_dict)
         data = b"abcdefghijklmnopqrstuvwxyz.!?"
diff --git a/aecid-testsuite/unit/util/SecureOSFunctionsTest.py b/aecid-testsuite/unit/util/SecureOSFunctionsTest.py
@@ -45,7 +45,7 @@ def test3_secure_open_directory(self):
 
     def test4sendAnnotatedFileDescriptor(self):
         """A valid annotated file descriptor is to be sent by a socket."""
-        sock_name = '/tmp/test4unixSocket.sock'
+        sock_name = '/tmp/test4unixSocket.sock'  # skipcq: BAN-B108
         data = b'readmeStream' + b'\x00' + b'You should read these README instructions for better understanding.'
 
         # skipcq: BAN-B607, BAN-B603
@@ -86,7 +86,7 @@ def test5send_annotated_file_descriptor_invalid_parameters(self):
 
     def test6send_logstream_descriptor(self):
         """A valid logstream descriptor is to be sent."""
-        sock_name = '/tmp/test6unixSocket.sock'
+        sock_name = '/tmp/test6unixSocket.sock'  # skipcq: BAN-B108
         data = b'logstream' + b'\x00' + b'/var/log/syslog'
 
         # skipcq: BAN-B607, BAN-B603
@@ -117,7 +117,7 @@ def test6send_logstream_descriptor(self):
 
     def test7receive_annotated_file_descriptor(self):
         """A valid annotated file descriptor is to be received by a socket."""
-        sock_name = '/tmp/test6unixSocket.sock'
+        sock_name = '/tmp/test6unixSocket.sock'  # skipcq: BAN-B108
         type_info = b'logstream'
         path = b'/var/log/syslog'
         data = (type_info, path)
diff --git a/aecid-testsuite/unit/util/clientTest4.py b/aecid-testsuite/unit/util/clientTest4.py
@@ -7,7 +7,7 @@
 # skipcq: FLK-E402
 from aminer.util.SecureOSFunctions import secure_open_file, send_annotated_file_descriptor
 
-sock_name = '/tmp/test4unixSocket.sock'
+sock_name = '/tmp/test4unixSocket.sock'  # skipcq: BAN-B108
 fd = secure_open_file(b'/etc/aminer/conf-enabled/Readme.txt', os.O_RDONLY)
 sleep(0.5)
 client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
diff --git a/aecid-testsuite/unit/util/clientTest6.py b/aecid-testsuite/unit/util/clientTest6.py
@@ -7,7 +7,7 @@
 # skipcq: FLK-E402
 from aminer.util.SecureOSFunctions import secure_open_file, send_logstream_descriptor
 
-sock_name = '/tmp/test6unixSocket.sock'
+sock_name = '/tmp/test6unixSocket.sock'  # skipcq: BAN-B108
 fd = secure_open_file(b'/var/log/syslog', os.O_RDONLY)
 sleep(0.5)
 client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
diff --git a/docs/CONFIGURATION.rst b/docs/CONFIGURATION.rst
@@ -2090,15 +2090,6 @@ DebugHistoryMatchRule
 This rule can be inserted into a normal ruleset just to see when a match attempt is made.
 It just adds the evaluated log_atom to a ObjectHistory.
 
-=======
-        - type: 'EventTypeDetector'
-          id: ETD
-
-        - type: 'VariableTypeDetector'
-          event_type_detector: ETD
-          num_init: 200
-          num_update: 100
-          num_s_gof_values: 100
 
 -------------
 EventHandling
diff --git a/source/root/usr/lib/logdata-anomaly-miner/aminer/input/JsonStateMachine.py b/source/root/usr/lib/logdata-anomaly-miner/aminer/input/JsonStateMachine.py
diff --git a/source/root/usr/lib/logdata-anomaly-miner/aminer/parsing/JsonModelElement.py b/source/root/usr/lib/logdata-anomaly-miner/aminer/parsing/JsonModelElement.py
