Version 4.5.0: Custom Parameterized Form, Bulk Filtering & File Management

• Bulk Filtering mechanism
  • refactoring of the service template with new Jinja2 macro
  • Existing caveats:
    • Cannot filter service type or device type specific properties
    • Cannot be used for bulk editing
• Round result size to 1 decimal when the result size is higher than 50% the maximum
  allowed size
• Don't allow saving a workflow if the run method is set to Service x Service and the
  workflow has target devices, pools, or a device query
• Don't allow skipping services in a workflow if workflow edit is restricted to
  owners via RBAC access control
• Remove all references to old "update_pools" mechanism (removed last release)
  Commit: 14e5728
• Add new "no_search" keyword argument in SelectField and MultipleSelectField
  to disable search box (in service form)
• Remove hash_user_passwords options from settings.json (always hash passwords)
  Remove ability to use user passwords for automation & web SSH.
• Remove pin to 2.3.3 for wtforms in requirements
• Remove pin to 1.0.1 for flask_wtf in requirements
• Remove pin to 2.0.1 for itsdangerous in requirements
• Remove pin to 1.4.46 for sqlalchemy in requirements (move to sqlalchemy v2)
• Remove pin to 3.4 for netmiko (move to netmiko 4+)
  • remove delay_factor and add read_timeout property.
• Fix duplicated run name when running a service from the REST API bug
• Order model drop down lists in the UI based on pretty name instead of tablename
• Add user "last login" property (record time of latest login)
  Commit: f2e4f26
• Add zoom sensitivity control in user profile for Workflow Builder
• Remove "username" variable from workflow global variables and add "user"
  dictionary instead with name and email. Mandatory checks:
  • wherever "username" is used, it must be replaced with user["name"]
  • "user" must not be used in existing workflows
• Add new "ignore_invalid_targets" parameter in run_service REST endpoint to run
  even if there are invalid targets.
• Update size of restart workflow from panel to fit to content
• Add device name to git history comparison panel title
• Add task name, target devices and target pools to the run table ("Results" page)
• Add custom parameterized form feature:
  • new "Parameterized Form Template" field in service panel > step 1 for the HTML code
  • supports JavaScript code inside <script></script> tag
  • must follow the same template as the default parameterized form ("form_type" variable,
    add-id / btn-id CSS class, eNMS.automation.submitInitialForm run function, etc...)
• Improve performances of migration import mechanism (~ x10)
  • Use CLoader to load migration files
  • Use a dictionary to store SQLAlchemy objects so that they are only fetched once
  • Disable log events during import
  • Commit: 9d2ceae / 6fb025f
• Add "Last Run" property for services to indicate the last time it was run
• Revert update function default RBAC value back to "edit". For relationship update, required
  "read" access instead of "edit" access (7342db9)
• Reduce number of fetch when running a workflow:
  • number of fetch / 10, execution time improved by 30%
  • lazy join of run.service, workflow.services / edges, workflow edge source/destination/workflow
  • new device store in workflow job function to avoid fetching devices multiple times
  • Commit: b56feea
• Dont include payload in intermediate workflow results (1ae5a7b)
• Add string substitution support in the mail notification service fields (sender, recipients, reply-to)
• Dont allow enabling multiprocessing if the run method is not set to per device
• Remove password from netmiko connection object after opening connection
• Add metadata file when doing migration or service import / export
  • metadata includes export time, export version, and service name in case of service export
  • fix shared workflow import bug (empty list => no way to detect main workflow)
  • in settings.json, add new "import_version" to disallow importing older files (if the
    version in the metadata is not the one from settings.json, the import fails)
• Reduce number of fetch in the scan_folder function to improve performance
• Refactor get_workflow_results to speed up workflow results display
• Refactor get_runtimes to no longer query the result table
• Skip bug fix for run once service (from thread: "Skip query not respected")
  Fix: if a "run once" service has targets and all targets are skipped, then
  the service is not run.
• Make Result.service_id and Result.parent_runtime indices to speed up results display (filtering/result)
• Add "SETUP_DIR" environment variable to set path to folder where json settings files are located
• Add new "migration" key in setup.json > paths to set path to the folder where migration files
  are located
• Dont update configuration properties if no change detected:
  • Commit: 58798c1
  • in backup service, don't write to local file, don't update value in database and don't
    update "update" timestamp if the value hasn't changed
  • in the "update database configuration from git", dont update the database configuration
    if the "update" timestamp from git is the same as or older than the one stored in the database.
  • Add "force_update" argument to "get git content" and "update database configuration
    from git" functions (when the app runs for the first time, this argument is set to True)
• Add parameterized form properties in dedicated accordion in service edit panel
• Display header and link before results in email notification
• Files Improvements:
  • Refactor the files mechanism to no longer display the full Unix path, only the path
    from the files folder
    • "files" displayed as breadcrumb even if the actual path does not include such a folder
    • the copy to clipboard mechanism still returns the full path so it can be used in e.g. python scripts
    • allow both absolute and relative paths in generic and netmiko file transfer services
    • impact on migration: all paths in files must be truncated by removing the path
      to the "files" folder
  • Prevent uploading the same file twice in the file upload panel (or another file
    with the same name)
  • Add trash mechanism for files. Two options:
    • 1. Put trash outside of files folder (not tracked by eNMS). When deleting a file, the database
         object is deleted and the associated unix file is moved to the trash folder, with the current
         time as prefix.
    • 2. Put trash inside the files folder: the same mechanism applies, but the trash folder is
         being tracked by eNMS. Files can be restored (by moving them from the trash folder to another
         directory), and whenever a file in the trash folder is deleted, it is removed (rm) from the
         filesystem. Files are moved to trash via the "update" function so that the file metadata is
         preserved.
    • The path to the trash folder is configured in settings.json > "files" > "trash"
    • The trash folder cannot be deleted from inside the application
    • When a file is moved to the trash, the alert is changed to a warning that says the
      file was moved to the trash folder
    • When importing migration files with "empty database" set to True, or running the mass
      deletion mechanism ("database deletion"), unix files are left untouched.
  • Detect missing files when running scan folder mechanism and mark them as "Not Found"
  • Drag-n-drop the same file multiple times in upload panel no longer possible
  • Use scoped path for playbooks in ansible service (impact on migration files)
• Fix log not sent when add_secret is False or device is None in the get_credentials function bug
• Add new 'prepend_filepath' function in workflow builder namespace to add path to file folder before a string
• Add support for string substitution for the email notification feature (service step 4)
• Limit update all pools mechanism (in pool table and as a service option) to the pools a user
  has "edit" access to

eNMS v4.2

• Add Network builder mechanism
• Add 3D visualization of network devices
• Extend Devices and Links with subclass / custom properties and a separate tab in the UI, the same way services work.
• Remove deep_services function used for export, use service.children relationship instead.
• Dont subclass SQLAlchemy Column following advice of SQLAlchemy creator.
• Make corrupted edges deletion mechanism a troublehooting snippet instead of a button in the admin panel.
• Move redis configuration in settings.json > "redis" key
• Add new mechanism to limit results in server-side drop-down list with filtering constraints.
• Limit superworkflow selection to workflows that contains the shared Placeholder service.
• Set trigger variable to "Regular Run" or "Parameterized Run" when service is triggered from the UI instead of "UI".
• Add SSH Proxy mechanism (multiple jump server in gateways property, gateway device subtype, priority
  tie-break mechanism)
• Consider runtime limiting user / all toggle mechanism in the restart service window.
• Move doc link to settings.json to allow custom doc links for plugins. Generate doc link in the jinja2 template
  instead of javascript (otherwise, wrong doc link until updated in js)
• Move tables refresh rate to settings.json to allow for custom refresh rates.
• New "Category" property / mechanism for the drop-down list of the site and workflow builder.
• Reinstate service selection with single left click (Ctrl no longer needed)
• Remove pytest, coverage, and travis dependencies.
• Reinstate single left click for node selection in workflow & site builder.
• Remove most union subquery in rbac_filter because a union of query yields a CompoundSelect
  SQLAlchemy object, and this is not compatible with using with_entites (via filtering properties kw).
• Fix export in bulk (the hierarchical display mode was not considered, all services inside a workflow
  were exported even when "hierarchical display" was selected)
• Add notification banner mechanism
• Remove default_access property, replace with "admin_only" boolean. Impact on migration.
• Make "run_service" rest api endpoint default to async True
• Update netmiko and napalm Backup services to load deferred row before updating. Impact on both services.
• Remove pathlib from requirements.txt
• Update workflow algorithm to not add services to priority queue in DxD mode if all are discarded.
• Update Ansible Service to use custom path in cwd argument of subprocess.check_output.
• Change default priority to 10 for services. Update of migration files required.
• Implement Cache Invalidation mechanism so that javascript/css files are reloaded at each release.
  Cache invalidation is activatated by setting "invalidate_cache" to true in settings.json.
  eNMS automatically renames the version folder in static / js and static / css to the version number
  stripped from its punctuation.
• Add new check box "Approved by an Admin user" in the Unix Command service. That box must be ticked by
  an admin user for the service to be allowed to run. A non-admin user cannot save a service if it is
  ticked, meaning that each time a Unix Command service is edited, it must be re-appproved.
• Add new timeout parameters for Scrapli service
• Always show security logs, even when logging is disabled. Add "allow_disable" (default: True) keyword argument
  to log function to prevent logs from being disabled if necessary.
• Add new 'deactivate_rbac_on_read' property in rbac.json, under 'advanced' key. Set to true by default.
  When true, eNMS no longers applies rbac for reading from the database. (=> better performances)
• Make the vendor, operating_system and model properties a custom list for devices, links and services,
  and category for sites and workflows. The drop-down list choices can be configured in properties.json > property_list key.
• Add support for renaming objects from the REST API (with key "new_name")
• Add limit to maximum number of nodes that can be displayed in site builder". Configurable via
  visualization.json > Network Builder > max_allowed_nodes
• Add new option to display site nodes as ellipses instead of images for better performances. Configurable via
  visualization.json > Network Builder > display_nodes_as_images
• Auto-update Vendor and Operating System property value of a new service in the workflow builder
  based on the values of these properties in the parent workflow.
• Add support for custom ordering in plugin tables (configurable by overriding the tableOrdering function in the
  table JS class)
• Add support for using device credentials in the Rest Call Service (impact on migration files:
  "username" / "password" => "custom_username" / "custom_password"). Don't allow using device credentials
  if the run method is set to "Run Once".
• Make webssh command configurable from settings / ssh section
• Add new label size property to configure label size in workflow and network builder
• Add new "Configuration" RBAC mode
• Make "sessions" an admin model (visible only to admin users)
• Update git service to support git clone, shallow clone and custom path to local folder (instead of hardcoded path to
  "network_data" folder)
• Update slack notification service to use newest slack_sdk library (instead of slackclient<2)
• Make scrapli connection arguments configurable from automation.json / scrapli / connection_args

eNMS 4

• Refactor of the run mechanism. When running a service, a single run is created and saved to the
  database.
• Remove "operation" (any / all) property from pool
• Change the way pool objects are computed: via SQL query instead of pure python:
  better performances expected for large pools.
• Add regex support for SQLite
• Add new "Invert" option for table filtering
• Move terminal application for web SSH feature inside the application: the terminal application
  was previously moved outside the application because websockets requires sticky sessions which is
  incompatible with having multiple gunicorn workers. Moving to a deployment where eNMS is started
  multiple times with 1 gunicorn worker via the backend stream configuration, it is now possible for
  the terminal to be inside the main application.
• Fix submenu bug when the menu is minimized (gentelella bug)
• Replace prerequisite edge with priority mechanism
• Allow making non-shared service shared and vice-versa (if the shared service doesn't have more than one workflow).
• Separate progress for main devices & iteration devices in workflow builder
• Fix bug where subworkflow device counters not displayed in results when device iteration is used
  Bug report mail: "No status for services in subworkflow with device iteration"
• HTTP requests logging: all requests are now logged by eNMS and not by werkzeug like before.
  => fine grained controlled for what is logged for each request. The log now contains the username.
• Add duplicate button in service table
• Refactor the geographical and Logical View to behave like the workflow builder:

• List of all pools that contain at least one device or link, stored in user browser local storage
• Remove default pool mechanism. Remove "visualization_default" property in pool model.
  By design, the default pool becomes the first pool in alphabetical order
• Add backward / forward control like the workflow builder

• Add global "factory" and "delete" functions in the workflow builder to create and delete new objects
  from a workflow.
• When refreshing a pool, rbac is now ignored so that the pool "refresh" action result does not depend on the
  user triggering it.
• If a workflow is defined to run on a set of devices, and the user lacks access to one or more devices,
  execute for all accessible devices and fail for the inaccessible devices instead of failing the entire workflow.
• Store and commit web SSH session content in backend instead of relying on send beacon mechanism and
  onbeforeunload callback so that the saving of a session does not depend on user behavior
• Add new "Empty" option in table filters and pool definition to filter based on whether the property
  value is empty or not.
• Add table display with property value constraint when clicking on the charts in the dashboard.
• Add scrapli netconf service
• Move LDAP and TACACS+ server init to environment file instead of custom file. Impact on authentication
  ldap / tacacs functions.
• Add Token-based authentication via REST API. New GET endpoint "/rest/token" to generate a token.
• Separate controller (handling HTTP POST requests) from main application (gluing everything together)
  Impact:

• In plugins,
• the "custom" file that contains pre_init, post_init, and the authentication custom code no longer inherits
  from the controller

• Add new "ip_address" field in settings.json > app section
• Add paging for REST API search endpoint: new integer parameter "start" to request results from "start"
• Add server time at the bottom of the menu (e.g for scheduling tasks / ease of use)
• Add button in service table to export services in bulk (export all displayed services as .tgz)
• Ability to paste device list (comma or space separated) into a multiple instance field (e.g service device and pool targets)
• Re-add current Run counter to 'Service' and 'Workflow' on the dashboard banner + Active tasks
• Ability to download result as json file + new copy result path to clipboard button in result json editor panel
• Ability to download logs as text file
• When importing existing workflows via service import, remove all existing services and edges from the workflow
• Upload service from laptop instead of checking for file on the VM
• Add Parameterized Form mechanism to update run properties and payload.
• Add new "full results" button to results tree
• Fix bug in WB where multiple services stay selected
• Add confirmation prompt in workflow builder before deletion
• Change default postprocessing mode to "Run on success only"
• Add log in case postprocessing is skipped
• Add SSH key support in generic file transfer service
• Always set "look_for_keys" to False in generic file transfer service - no longer an option
• Add validation_section mechanism: set path to section of the result to validate (default: results["result"])
• Add new "connection_name" mechanism to open multiple parallel connections to the same device in the
  same workflow
• Add new "get_credential" global variable in workflow builder. Used to get a password or a passphrase
  for a netmiko validaiton command or rest call service. For obfuscation purposes.
  mail: Obfuscate Credentials passed into Netmiko Command Line
• Fix data extraction service and operation keyword in set_var
• Don't set status of currently running services to "Aborted" when using a flask CLI command
• Add TextFSM support for the netmiko validation service (+ regression workflow)
• Add stop mechanism for services in the Result table
• Add server name parameter in Run table to specify which server a service was run from.
  Server to be configured from env variable SERVER_NAME and SERVER_ADDR.
• Lock editing / run of Workflow to group of owners

eNMS 3.22.3

• Add regression workflow for file transfer
• Fix RBAC service run and task scheduling REST API bug
• Fix payload extraction workflow setitem bug
• Add regression workflow with lots of service for scalability testing
• Add regression workflow for skipped service in workflow targets SxS run mode
• Fix rest call service local() scope bug
• Fix get var / set var "devices" keyword bug
• Add jump on connect parameters for netmiko backup service
• Fix skipped query with device in service by service with workflow targets mode bug

eNMS 3.22.2

• Fix iteration device factory commit bug
• Fix workflow in service by service with workflow targets skipped and skip query service bug
• Add missing rbac endpoints in full + read only access
• Fix device creation empty driver due to Scrapli
• Fix workflow iteration mechanism bug

eNMS 3.22.1

• Add user authentication method in user forms
• Fix settings saving mechanism
• Fix gunicorn multiple workers sqlalchemy post fork session conflict bug
• Dont prevent wrong device GPS coordinates from displaying links in network view
• Fix RBAC bugs
• Add new Scrapli service to send commands / configuration to network device

eNMS 3.22

• Remove database url from settings. Configured via env variable DATABASE_URL
• Remote scheduler
• Remove TACACS+ parameters from settings, use env variable instead: TACACS_ADDR, TACACS_PASSWORD
• Make REST API accept Tacacs and LDAP credentials (in the last version, if you were using TACACS+ or LDAP, you could authenticate
  in the UI but couldn't make calls to the REST API)
• Remove LDAP parameters from settings. The LDAP authentication is in the custom controller, there is a default
  function that works with a standard LDAP installation, but you can customize however you want.
  The LDAP server is now configured with the env variable LDAP_SERVER.
  The settings contain a new section "database" to enable ldap, database or tacacs authentication.
• Add replier option in send mail mechanism
• Rename "app_log" option to "change_log" in log function for services
• Add new entry in workflow RC menu "Workflow Results Table": contains all results for a given runtime,
  allowing for comparison of results same device / different service, same service / different device, etc.
• Refactor logging mechanism. In settings.json, add new logging sections to configure whether the log
  for a given logger should also be logged as changelog or service log by default.
• RBAC
• Fix authentication bug flask_login and add session timeout mechanism
• Make plugins separate from eNMS in their own folder, add bash script to install/update/uninstall them
• Make the CLI interface a plugins
• Remove summary from service state to improve workflow refresh performances
• Add Dark mode and theme mechanism
• Make search endpoint work with result to retrieve device results
• Allow dictionary and json as custom properties. For json properties, use jsoneditor to let the user
  edit them.
• Add placeholder as a global variable in a workflow (e.g to be used in the superworkflow)
• Add mechanism for creating custom configuration property
• Refactor data backup services with custom configuration properties. Implement "Operational Data" as
  an example custom property.
• Add new Git service. Replace "git_push_configurations" swiss army knife service with instance of git service.
• Add database fetch/commit retry mechanism to handle deadlocks & other SQL operational errors
• Add validation condition for validation section.

eNMS 3.21.3

• Add new plugins mechanism
• Fix bug help panel open when clicking a field or label
• Add error message in the logs when a service is run in per device mode but no devices have been selected.
• Add default port of 22 for TCP ping in ping service
• Disable edit panel on double-click for start/end services of a workflow
• Fix invalid request bug when pressing enter after searching the "add services to workflow" panel
• Forbid "Start", "End" and "Placeholder" for service names
• Fix Result in mail notification for run once mode
• Make Netmiko prompt command service a substitution string in the UI
• Fix wrong jump password when using a Vault
• Fix workflow results recursive display no path in results bug
• Improve "Get Result" REST endpoint: returns 404 error if no run found, run status if a run is found but there are
  no results (e.g job still running), and the results if the job is done.
• Remove wtforms email validator in example service following wtforms 2.3 release

eNMS 3.21.2

• Fix rest api update endpoint bug
• Add device results to rest api get_result endpoint
• Rename subservice -> placeholder
• Fix rendering of custom boolean properties
• Fix custom properties accordion in service panel
• Fix service cascade deletion bug with service logs and placeholder
• Fix front-end alert deleting services and make it a success alert
• Fix historical config / oper data comparison mechanism
• Fix bug where superworkflow cannot be cleared from list after selection
• Fix bug placeholder service deletion from workflow
• Make superworkflow a workflow property only. Remove superworkflow targets option
• Display only workflows in the superworkflow drop-down list
• Save alert when displaying python error as an alert
• When using a custom logger, only the actual user content is logged
• Update docs rest API
• Improve log function (custom logger behavior / creator)
• Fix superworkflow bug for standalone services
• Dont display private properties in parameterized run results
• Add Ansible playbook service log to security logger
• Update superworkflow initial payload with placeholder service initial payload
• Dont update netmiko and napalm configuration / oper data backup if empty result / no commands

eNMS 3.21.1

• Upgrade JS Panel to v4.10
• Fix jspanel position on long pages with a scrollbar
• Fix subservice double-click bug
• Fix table display bug
• Fix operational data display bug