no issue - much better configuration

Author: pounard

CHANGELOG.md

@@ -3,7 +3,18 @@
 ## Next
 
 * [feature] 🌟 Add `bin/db-tools` CLI command allowing standalone usage (#153).
+* [feature] ⭐️ The CLI tool can run without configuration using only environment variables (#191).
+* [feature] ⭐️ All global options can now be configured on a per-connection basis in `connections.NAME.OPTION` (#191).
+* [feature] 🌟 In both CLI and Symfony, the `anonymyzation` configuration option may now directly hold the complete anonymization configuration without requiring an additional file. (#191).
 * [experimental] ⭐️ Add `bin/compile` CLI command for building a PHAR file (#154).
+* [deprecation] `anonymization.yaml` is replaced by `anonymization_files` (#191).
+* [deprecation] `excluded_tables` is replaced by either `backup_excluded_tables` or `connections.NAME.backup_excluded_tables` (#191).
+* [deprecation] `storage.filename_strategy` is replaced by either `storage_filename_strategy` or `connections.NAME.filename_strategy` (#191).
+* [deprecation] `storage.root_dir` is replaced by either `storage_directory` or `connections.NAME.storage_directory` (#191).
+* [bc] `backupper_binaries` (array) is replaced by either `backup_binary` (string) or `connections.NAME.backup_binary` (string) (#191).
+* [bc] `backupper_options` (array) is replaced by either `backup_options` (string) or `connections.NAME.backup_options` (string) (#191).
+* [bc] `restorer_binaries` (array) is replaced by either `restore_binary` (string) or `connections.NAME.restore_binary` (string) (#191).
+* [bc] `restorer_options` (array) is replaced by either `restore_options` (string) or `connections.NAME.restore_options` (string) (#191).
 * [bc] Password anonymizer `symfony/password-hasher` dependency is now optional and must be manually installed (#155).
 * [fix] Property must not be accessed before initialization error when using `--list` option (#183, @iNem0o).
 * [internal] All Doctrine related dependencies are now optional (#155).

UPGRADE.md

@@ -0,0 +1,25 @@
+# Upgrade guide
+
+## Next
+
+### Configuration file structure change
+
+The 2.0 version introduces many configuration changes. Most of them will
+gracefuly fallback to older version until 3.0, but some have been removed
+and will cause exceptions.
+
+Rationale is that now, all top-level configuration options can be directly
+set at the connection level, and we renamed those options to be more consistent
+and more explicit about what they do.
+
+Please read carefully the new sample configuration files:
+ - For Symfony: [config/packages/db_tools.yaml](./config/packages/db_tools.yaml)
+ - For standalone: [config/db_tools.standalone.yaml](./config/db_tools.standalone.yaml)
+
+And the the [CHANGELOG.md](./CHANGELOG.md) file and fix your configuration accordingly.
+
+The `backupper_binaries` and `backupper_options` as well as the `restorer_binaries`
+and `restorer_options` options have been removed and will raise exception when
+kept: older version allowed to configure backup and restore binaries on a per-vendor
+basis, they are now configured on a per-connection basis without considering the
+database vendor anymore. Please set those options for each connection instead.

config/db_tools.standalone.complete.sample.yaml

@@ -0,0 +1,124 @@
+# This configuration file is an example for standalone usage.
+# Only required parameter is the connection URL.
+
+# Working directory is the path to which all relative file references will
+# be relative to. If none set, the path will be this file directory instead.
+# In most cases, you will not need it.
+#workdir: .
+
+# Where to put generated backups.
+# Root directory of the backup storage manager. Default filename
+# strategy will always use this folder as root path.
+#storage_directory: ./var/db_tools
+
+# Default filename strategy. You may specify one strategy for each doctrine
+# connection.
+# If you created and registered a custom one into the container as a
+# service, you may simply set the service identifier. If no service
+# exists, and your implementation does not require parameters, simply
+# set the class name.
+# Allowed values are:
+#  - "default": alias of "datetime".
+#  - "datetime": implementation is "%db_tools.storage_directory%/YYYY/MM/<connection-name>-<datestamp>.<ext>".
+#  - CLASS_NAME: a class name to use that implements a strategy.
+#storage_filename_strategy: default
+
+# When old backups are considered obsolete.
+# (Use relative date/time formats : https://www.php.net/manual/en/datetime.formats.relative.php)
+#backup_expiration_age: '6 months ago' # default '3 months ago'
+
+# Default timeout for backup process.
+#backup_timeout: 1200 # default 600
+
+# Default timeout for restore process.
+#restore_timeout: 2400 # default 1800
+
+# List here tables (you don't want in your backups.
+# If you have more than one connection, it is strongly advised to configure
+# this for each connection instead.
+#backup_excluded_tables: ['table1', 'table2']
+
+# Specify here paths to backup and restore binaries and command line options.
+# Warning: this will apply to all connections disregarding their database
+# vendor. If you have more than one connection and if they use different
+# database vendors or versions, please configure those for each connection
+# instead.
+# Default values depends upon vendor and are documented at
+# https://dbtoolsbundle.readthedocs.io/en/stable/configuration.html
+#backup_binary: '/usr/bin/pg_dump'
+#backup_options: '-Z 5 --lock-wait-timeout=120'
+#restore_binary: '/usr/bin/pg_restore'
+#restore_options: '-j 2 --clean --if-exists --disable-triggers'
+
+# If you have a single connection, you can use this syntax.
+# Connection name will be "default" when configured this way.
+connections: "pgsql://username:password@hostname:port?version=16.0&other_option=..."
+
+# Alternatively, you are allowed to have more than one connections, case
+# in which you might synthesize the configuration as this:
+#connections:
+#    connection_one: "pgsql://username:password@hostname:port?version=16.0&other_option=..."
+#    connection_two: "mysql://username:password@hostname:port?version=8.1&other_option=..."
+
+# For advanced usage, you may also override any parameter for each connection.
+# Each key is a connection name, all parameters above are allowed for each
+# unique connection.
+#connections:
+#    connection_one:
+#        # Connection URL for connecting.
+#        # Please refer to makinacorpus/db-query-builder or  documentation for more information.
+#        # Any URL built for doctrine/dbal usage should work.
+#        # URL is the sole mandatory parameter.
+#        # Complete list of accepted parameters follows.
+#        url: "pgsql://username:password@hostname:port?version=16.0&other_option=..."
+#        backup_binary: /usr/local/bin/vendor-one-dump
+#        backup_excluded_tables: ['table_one', 'table_two']
+#        backup_expiration_age: '1 month ago'
+#        backup_options: --no-table-lock
+#        backup_timeout: 2000
+#        restore_binary: /usr/local/bin/vendor-one-restore
+#        restore_options: --disable-triggers --other-option
+#        restore_timeout: 5000
+#        storage_directory: /path/to/storage
+#        storage_filename_strategy: datetime
+#    connection_two:
+#        url: "mysql://username:password@hostname:port?version=8.1&other_option=..."
+#        # ...
+
+# You can explicitely set which will be default connection in use when
+# none providen in the command line options. If you omit this configuration
+# value, then the first one in the list will be used.
+#default_connection: connection_one
+
+# Update this configuration if you want to look for anonymizers in a custom folder.
+#anonymizer_paths:
+    #- ./src/Anonymization/Anonymizer'
+
+# For simple needs, you may simply write the anonymization configuration here.
+# Keys are connection names, values are structures which are identical to what
+# you may find in the "anonymizations.sample.yaml" example.
+anonymization:
+    connection_one:
+        table1:
+            column1:
+                anonymizer: anonymizer_name
+                # ... anonymizer specific options...
+            column2:
+            # ...
+        table2:
+            # ...
+    connection_two:
+        # ...
+
+# You can for organisation purpose delegate anonymization config into extra
+# YAML configuration files, and simply reference them here.
+# Pathes can be either relative or absolute. Relative paths are relative to
+# the workdir option if specified, or from this configuration file directory
+# otherwise.
+# See the "anonymizations.sample.yaml" in this folder for an example.
+anonymization_files:
+    connection_one: './anonymization/connection_one.yaml'
+    connection_two: './anonymization/connection_two.yaml'
+
+# If you have only one connection, you can adopt the following syntax.
+#anonymization_files: '%kernel.project_dir%/config/anonymizations.yaml'

config/db_tools.standalone.sample.yaml

@@ -1,137 +1,67 @@
 # This configuration file is an example for standalone usage.
-#
-# The given parameters you will find in this file must not be set in a Symfony
-# application context, and will be ignored if so.
-#
-# All other configuration you can find in the ./packages/db_tools.yaml file
-# can be added in this file, you must simply omit the 'db_tools:' top level
-# node.
+# Only required parameter is the connection URL.
 
-# Working directory is the path to which all relative file references will
-# be relative to. If none set, the path will be this file directory instead.
-workdir: /var/www/my_project/
+# If you have a single connection, you can use this syntax.
+# Connection name will be "default" when configured this way.
+# If you have multiple connections, please refer to the exhaustive
+# sample in 'db_tools.standalone.complete.sample.yaml'.
+connections: "pgsql://username:password@hostname:port?version=16.0&other_option=..."
 
-# Database connections.
-# One line per connection, a single database URL, all options as query
-# parameters. Connections will be made using makincorpus/query-builder
-# which will raise exceptions when invalid options are found.
-# There is less configuration amplitude than using doctrine/dbal in
-# Symfony, yet it should be enough in most case.
-# In case any options or specific behaviour is missing, please file
-# an issue at https://github.com/makinacorpus/php-query-builder/issues
-connections:
-    connection_one: "pgsql://username:password@hostname:port?version=16.0&other_option=..."
-    connection_two: "mysql://username:password@hostname:port?version=8.1&other_option=..."
+# Where to put generated backups.
+# Root directory of the backup storage manager. Default filename
+# strategy will always use this folder as root path.
+#storage_directory: ./var/db_tools
 
-# If you have a single connection, you can use this syntax. In this case
-# the connection name will be "default".
-# connections: "pgsql://username:password@hostname:port?version=16.0&other_option=..."
+# Default filename strategy. You may specify one strategy for each connection.
+# If you created and registered a custom one into the container as a
+# service, you may simply set the service identifier. If no service
+# exists, and your implementation does not require parameters, simply
+# set the class name.
+# Allowed values are:
+#  - "default": alias of "datetime".
+#  - "datetime": implementation is "%db_tools.storage_directory%/YYYY/MM/<connection-name>-<datestamp>.<ext>".
+#  - CLASS_NAME: a class name to use that implements a strategy.
+#storage_filename_strategy: default
 
-# You can explicitely set which will be default connection in use when
-# none providen in the command line options. If you omit this configuration
-# value, then the first one in the list will be used.
-#default_connection: connection_one
-
-# Using the DbToolsBundle standalone, you must provide at least
-# a root directory for backups.
-storage:
-    # Path can be relative or absolute, Relative paths are relative to the
-    # workdir option if specified, or from this configuration file directory
-    # otherwise.
-    # If none provided, the default will be the following one.
-    root_dir: ./var/db_tools
-    # Filename strategies. You may specify one strategy for each doctrine
-    # connection. Keys are doctrine connection names. Values are strategy
-    # names, "default" (or null) or omitting the connection will use the
-    # default implementation.
-    # If you created and registered a custom one into the container as a
-    # service, you may simply set the service identifier. If no service
-    # exists, and your implementation does not require parameters, simply
-    # set the class name.
-    #filename_strategy:
-        # Backup filename strategy.
-        # "default" is an alias of "datetime"
-        #default: default
-        # "datetime" implementation is ROOT_DIR/YYYY/MM/<connection-name>-<datestamp>.<ext>"
-        #other_connection_strategy: datetime
-        # Example of using a service name:
-        #yet_another_connection: app.db_tools.filename.custom_strategy
-        # Or a classe name:
-        #another_one: App\DbTools\Storage\MyCustomStrategy
-
-# When old backups are considered obsolete
+# When old backups are considered obsolete.
 # (Use relative date/time formats : https://www.php.net/manual/en/datetime.formats.relative.php)
 #backup_expiration_age: '6 months ago' # default '3 months ago'
 
-# Timeout for backups.
-# backup_timeout: 1200 # default 600
-
-# Timeout for restores.
-# restore_timeout: 2400 # default 1800
-
-# List here tables (per connection) you don't want in your backups
-#excluded_tables:
-    #default: ['table1', 'table2']
-
-# Specify here paths to binaries, only if the system can't find them by himself
-# platform are 'mysql', 'postgresql', 'sqlite'
-#backupper_binaries:
-    #mariadb: '/usr/bin/mariadb-dump' # default 'mariadb-dump'
-    #mysql: '/usr/bin/mysqldump' # default 'mysqldump'
-    #postgresql: '/usr/bin/pg_dump' # default 'pg_dump'
-    #sqlite: '/usr/bin/sqlite3' # default 'sqlite3'
-#restorer_binaries:
-    #mariadb: '/usr/bin/mariadb' # default 'mariadb'
-    #mysql: '/usr/bin/mysql' # default 'mysql'
-    #postgresql: '/usr/bin/pg_restore' # default 'pg_restore'
-    #sqlite: '/usr/bin/sqlite3' # default 'sqlite3'
-
-# Default options to pass to the binary when backing up or restoring
-# a database. Those options must be defined per connection.
-# If you do not define some default options, here or by using the
-# "--extra-options" option when invoking the command, the following
-# ones will be used according to the database vendor:
-# - When backing up:
-#   - MariaDB: --no-tablespaces
-#   - MySQL: --no-tablespaces
-#   - PostgreSQL: -Z 5 --lock-wait-timeout=120
-#   - SQLite: -bail
-# - When restoring:
-#   - MariaDB: None
-#   - MySQL: None
-#   - PostgreSQL: -j 2 --clean --if-exists --disable-triggers
-#   - SQLite: None
-#backupper_options:
-    #default: ''
-    #another_connection: ''
-#restorer_options:
-    #default: ''
-    #another_connection: ''
+# List here tables (you don't want in your backups.
+# If you have more than one connection, it is strongly advised to configure
+# this for each connection instead.
+#backup_excluded_tables: ['table1', 'table2']
 
 # Update this configuration if you want to look for anonymizers in a custom folder.
-# These are default paths that will always be registered even if you override
-# the setting and don't repeat them:
 #anonymizer_paths:
     #- ./src/Anonymization/Anonymizer'
 
-anonymization:
-    # From here you can proceed with manual file inclusion. Pathes can be
-    # either relative or absolute. Relative paths are relative to the workdir
-    # option if specified, or from this configuration file directory
-    # otherwise.
-    yaml:
-        connection_one: ./db_tools.anonymizer.connection_one.yaml
-        # ... other connections ...
-
-    # Extra configuration options, if you don't want to split the anonymization
-    # configuration into multiple files, you can directly write it here.
-    tables:
-        connection_one:
-            # From here, please refer to 'anonymizations.sample.yaml' for sample
-            # and documentation.
-            table_name:
-                column_name:
-                    anonymizer: anonymizer_name
-                    # ... other options...
-        connection_two:
-            # ...
+# Write the anonymization configuration here.
+# You may also write anonymization configuration in extra files instead, please
+# see the 'db_tools.standalone.complete.sample.yaml' for more documentation.
+#anonymization:
+#    # Keys are connection names, values are structures which are identical to what
+#    # you may find in the "anonymizations.sample.yaml" example.
+#    # If you only specified one URL in the "connections" parameter above, the
+#    # connection name is "default".
+#    default:
+#        # Keys here are table names, followed by column names, column values
+#        # are either an anonymizer name string, or an object with options.
+#        user:
+#            # Some Anonymizer does not require any option, you can use them like this
+#            prenom: fr-fr.firstname
+#            nom: fr-fr.lastname
+#            # Some do require options, specify them like this:
+#            age:
+#                anonymizer: integer
+#                options: {min: 0, max: 99}
+#            # Some others have optionnal options, you can specify those:
+#            email:
+#                anonymizer: email
+#                options: {domain: 'toto.com'}
+#            # Or leave it with defaults:
+#            email: email
+#            level:
+#                anonymizer: string
+#                options: {sample: ['none', 'bad', 'good', 'expert']}
+#        # ... other tables...