Version 4.5.0: Custom Parameterized Form, Bulk Filtering & File Management

• Bulk Filtering mechanism
  • refactoring of the service template with new Jinja2 macro
  • Existing caveats:
    • Cannot filter service type or device type specific properties
    • Cannot be used for bulk editing
• Round result size to 1 decimal when the result size is higher than 50% the maximum
  allowed size
• Don't allow saving a workflow if the run method is set to Service x Service and the
  workflow has target devices, pools, or a device query
• Don't allow skipping services in a workflow if workflow edit is restricted to
  owners via RBAC access control
• Remove all references to old "update_pools" mechanism (removed last release)
  Commit: 14e5728
• Add new "no_search" keyword argument in SelectField and MultipleSelectField
  to disable search box (in service form)
• Remove hash_user_passwords options from settings.json (always hash passwords)
  Remove ability to use user passwords for automation & web SSH.
• Remove pin to 2.3.3 for wtforms in requirements
• Remove pin to 1.0.1 for flask_wtf in requirements
• Remove pin to 2.0.1 for itsdangerous in requirements
• Remove pin to 1.4.46 for sqlalchemy in requirements (move to sqlalchemy v2)
• Remove pin to 3.4 for netmiko (move to netmiko 4+)
  • remove delay_factor and add read_timeout property.
• Fix duplicated run name when running a service from the REST API bug
• Order model drop down lists in the UI based on pretty name instead of tablename
• Add user "last login" property (record time of latest login)
  Commit: f2e4f26
• Add zoom sensitivity control in user profile for Workflow Builder
• Remove "username" variable from workflow global variables and add "user"
  dictionary instead with name and email. Mandatory checks:
  • wherever "username" is used, it must be replaced with user["name"]
  • "user" must not be used in existing workflows
• Add new "ignore_invalid_targets" parameter in run_service REST endpoint to run
  even if there are invalid targets.
• Update size of restart workflow from panel to fit to content
• Add device name to git history comparison panel title
• Add task name, target devices and target pools to the run table ("Results" page)
• Add custom parameterized form feature:
  • new "Parameterized Form Template" field in service panel > step 1 for the HTML code
  • supports JavaScript code inside <script></script> tag
  • must follow the same template as the default parameterized form ("form_type" variable,
    add-id / btn-id CSS class, eNMS.automation.submitInitialForm run function, etc...)
• Improve performances of migration import mechanism (~ x10)
  • Use CLoader to load migration files
  • Use a dictionary to store SQLAlchemy objects so that they are only fetched once
  • Disable log events during import
  • Commit: 9d2ceae / 6fb025f
• Add "Last Run" property for services to indicate the last time it was run
• Revert update function default RBAC value back to "edit". For relationship update, required
  "read" access instead of "edit" access (7342db9)
• Reduce number of fetch when running a workflow:
  • number of fetch / 10, execution time improved by 30%
  • lazy join of run.service, workflow.services / edges, workflow edge source/destination/workflow
  • new device store in workflow job function to avoid fetching devices multiple times
  • Commit: b56feea
• Dont include payload in intermediate workflow results (1ae5a7b)
• Add string substitution support in the mail notification service fields (sender, recipients, reply-to)
• Dont allow enabling multiprocessing if the run method is not set to per device
• Remove password from netmiko connection object after opening connection
• Add metadata file when doing migration or service import / export
  • metadata includes export time, export version, and service name in case of service export
  • fix shared workflow import bug (empty list => no way to detect main workflow)
  • in settings.json, add new "import_version" to disallow importing older files (if the
    version in the metadata is not the one from settings.json, the import fails)
• Reduce number of fetch in the scan_folder function to improve performance
• Refactor get_workflow_results to speed up workflow results display
• Refactor get_runtimes to no longer query the result table
• Skip bug fix for run once service (from thread: "Skip query not respected")
  Fix: if a "run once" service has targets and all targets are skipped, then
  the service is not run.
• Make Result.service_id and Result.parent_runtime indices to speed up results display (filtering/result)
• Add "SETUP_DIR" environment variable to set path to folder where json settings files are located
• Add new "migration" key in setup.json > paths to set path to the folder where migration files
  are located
• Dont update configuration properties if no change detected:
  • Commit: 58798c1
  • in backup service, don't write to local file, don't update value in database and don't
    update "update" timestamp if the value hasn't changed
  • in the "update database configuration from git", dont update the database configuration
    if the "update" timestamp from git is the same as or older than the one stored in the database.
  • Add "force_update" argument to "get git content" and "update database configuration
    from git" functions (when the app runs for the first time, this argument is set to True)
• Add parameterized form properties in dedicated accordion in service edit panel
• Display header and link before results in email notification
• Files Improvements:
  • Refactor the files mechanism to no longer display the full Unix path, only the path
    from the files folder
    • "files" displayed as breadcrumb even if the actual path does not include such a folder
    • the copy to clipboard mechanism still returns the full path so it can be used in e.g. python scripts
    • allow both absolute and relative paths in generic and netmiko file transfer services
    • impact on migration: all paths in files must be truncated by removing the path
      to the "files" folder
  • Prevent uploading the same file twice in the file upload panel (or another file
    with the same name)
  • Add trash mechanism for files. Two options:
    • 1. Put trash outside of files folder (not tracked by eNMS). When deleting a file, the database
         object is deleted and the associated unix file is moved to the trash folder, with the current
         time as prefix.
    • 2. Put trash inside the files folder: the same mechanism applies, but the trash folder is
         being tracked by eNMS. Files can be restored (by moving them from the trash folder to another
         directory), and whenever a file in the trash folder is deleted, it is removed (rm) from the
         filesystem. Files are moved to trash via the "update" function so that the file metadata is
         preserved.
    • The path to the trash folder is configured in settings.json > "files" > "trash"
    • The trash folder cannot be deleted from inside the application
    • When a file is moved to the trash, the alert is changed to a warning that says the
      file was moved to the trash folder
    • When importing migration files with "empty database" set to True, or running the mass
      deletion mechanism ("database deletion"), unix files are left untouched.
  • Detect missing files when running scan folder mechanism and mark them as "Not Found"
  • Drag-n-drop the same file multiple times in upload panel no longer possible
  • Use scoped path for playbooks in ansible service (impact on migration files)
• Fix log not sent when add_secret is False or device is None in the get_credentials function bug
• Add new 'prepend_filepath' function in workflow builder namespace to add path to file folder before a string
• Add support for string substitution for the email notification feature (service step 4)
• Limit update all pools mechanism (in pool table and as a service option) to the pools a user
  has "edit" access to