Updating Codegen Providers

Updating Codegen Providers

In GenFlow, a Codegen Provider is a library that generates code, documentation, or other artifacts from a machine-readable API description and/or other inputs. GenFlow defines a separate GenTemplate for each distinct type of code that the Codegen Provider can generate.

This topic describes the process of updating Codegen Providers, specifically Swagger Codegen and OpenAPI Generator.

Swagger Codegen and OpenAPI Generator Libraries

GenFlow's integration of Swagger Codegen (SCG) and OpenAPI Generator (OAG) makes use of an IGenTemplateGroup implementation that uses ServiceLoader to find all the generation modules and wraps each in an IGenTemplate implemenation. (ServiceLoader itself is not actually used, but rather the classloader is used to find resources that use the ServiceLoader resource naming conventions, and those resources are used to collect generator class names. I don't recall now why ServiceLoader wasn't used directly.)

An additional feature built into the support classes is a capability to manage the offered GenTemplates in a few ways:

• GenTemplates for individual generator modules can be marked for suppression
• GenTemplate names can be supplied for specific generator modules
• Named parameters can be marked as required and/or supplied with default values. (We have yet to do this)

These things are maintained with CSV files, known as module info files, that are kept in the source trees.

Module Discovery

Module info files are created by the GenFlow build process, each time a build is performed with a previously unused version of the SCG and/or OAG libraries configured in the GenFlow root pom file. This is done by running main classes ScgCodegenModuleDiscovery and OagCodegenModuleDiscovery during the generate-sources build phase. Both classes are simple extensions of GenModuleDiscovery, which is in the genflow-common project.

The discovery process proceeds as follows:

1. If module info files already exist for whatever version of the codegen library (SCG or OAG) is currently configured in the GenFlow root pom file (the current version), the program exits without doing anything. Therefore, module info files for a given library version are never re-generated once they have been checked into git, unless someone explicitly causes them to be re-generated, e.g. by running the discovery class manually or removing the existing module info files.
   
   
2. Otherwise, the most recent prior module info is loaded into in-memory structures (as defined in the GenModuleInfo class in genflow-common). This forms the base info for the new module info. "Most recent prior" means module info for the highest library version for which the files already exist, and which is lower than the current version. Only M.m.p version labels are understood, where M, m, and p are numerical major, minor and patch version numbers - i.e., no snapshot or other qualifiers can be used. Version ordering is numerical by major, minor, and patch version numbers, in that order of preference.
   
   
3. Each module appearing in the base info is copied to the new info, except those marked as deleted.
   
   
4. ServiceLoader is used to scan the classpath for all available codegen modules. This should discover only classes that are supplied by the SCG/OAG libraries. These are classes that extend io.swagger.codegen.CodegenConfig or org.openapitools.codegen.CodegenConfig, respectively.
   
   
5. For each module, we locate its module info record in the base info set, or create a new record if it's not present. Then we obtain the following information from the module class instance returned by ServiceLoader and copy it into the module info record:
   
   
   • The module's class name (this is the record key)
     
     
   • The module's declared type (e.g. CLIENT, SERVER, DOCUMENTATION, etc.). (This is obtained via the module instance's getTag() method.) 3. The module's declared name
     
     
   • The module's named parameters, referred to as "client options" in SCG and OAG libraries (instances of class io.swagger.codegen.CliOption and org.openapitools.codegen.CliOption, respectively). For each parameter we record its name and its description. Default values and requiredness are supported in the module info data, but they are not presently part of the client option abstraction in either SCG or OAG, so all parameters end up as optional and with no default value after discovery. (These can be overridden by manually editing the corresponding CSV file, but we have never done this to-date).
     
     
6. Next, we compute a derived name for the module, making use of the above information, and we store that into the info record as well, in the DerivedDisplayName property. This is the name that will be used in API Studio GUI elements, unless a manual override is supplied for the module.
   
   
7. Additional fields are set as follows in each module info record:
   
   
   • DisplayName - an empty string by default; if it's present, it is used as a name for the wrapped GenTemplate instead of the derived name computed during discovery.
     
     
   • Suppressed flag - a boolean indicating whether this module should be suppressed (i.e. not wrapped as a GenTemplate). This flag is always set false by discovery but can be manually overridden in the CSV file.
     
     
   • Deleted flag - a boolean that is set by discovery to indicate that this module appeared in the base info but was not encountered during the ServiceLoader scan performed in discovery. This means that the module was decommissioned in the SCG/OAG library. Deleted modules always have their Changed flag set and their New and Vetted flags cleared. The setting of the Deleted flag has no impact on code behavior outside discovery; it is intended to make it easy to identify decommissioned modules when writing release notes, etc.
     
     
   • New flag - a boolean that is set by discovery to indicate that this module was encountered during the ServiceLoader scan but was not present in the base info set. This means that the module was added to teh SCG/OAG library sometime between the base version and the current version. New modules always have their Changed flags set as well.
     
     
   • Changed flag - a boolean that is set by discovery whenever any of the information copied from the base info set differs from what was obtained from the same module during discovery. The flag is set for all new and deleted modules, but also any modules whose reported type or name has changed, or whose derived name has changed for some other reason (e.g. a change in the derived name algorithm).
     
     
   • Vetted flag - a boolean that is never set by discovery but is carried forward from the base info set and is intended to be manually edited in the CSV file. It indicates that the module has been reviewed and approved for inclusion in API Studio GUI elements. The primary purpose of the review is to ensure that a reasonable name will be used in the wrapped GenTemplate - either the automatically derived name or a manually specified DisplayName.
     
     N.B. Previously vetted modules will show up as vetted in the new module info, even if some of the captured info has changed. In this case, the module's Changed flag will also be set. Reviewers should check such records to ensure that display name will still be appropriate.

Module Info Files

Two files constitute the module info for a given codegen library version: modulesInfo_M.m.p.csv and moduleParams_M.m.p.csv, where M.m.p is the codegen library version to which the info applies.

Each module is represented by a single record in the modulesInfo file, and by multiple records (one for each discovered parameter) in the moduleParams file.

The files appear in the GenFlow swagger-codegen and openapi-generator projects, in src/main/resources/com/reprezen/genflow/swagger/codegen and src/main/resources/com/reprezen/genflow/openapi/generator, respectively (i.e. in the same "package" as the discovery classes, but in the resources tree).

The module info files can be edited with Excel or LibreOffice Calc (or probably Google Sheets and other spreadsheet programs). After making changes, the files should be saved out as CSV files again.

Columns for flags (e.g. Suppressed and Vetted) should be left empty to indicate false values, and should contain a single asterisk ("*") for true values. 

Upgrading to a New Version of SCG or OAG

The following procedure details how to make changes to GenFlow sources to upgrade to a new version of SCG and/or OAG libraries. These changes should be done in the context of the general change process for GenFlow and other RepreZen component projects, which is documented here.

1. Modify the io.swagger:swagger-codegen or org.openapitools:openapi-generator versions specified in the root-level GenFlow pom.xml file. These are managed dependencies that will affect all the GenFlow modules. Do not get confused by dependencies with the same artifact ids but in the com.reprezen.genflow group; those are the GenFlow projects that adapt the codegen modules as GenTemplates.
   
   
2. Perform a local GenFlow build (generally: mvn clean install from the GenFlow working tree root)
   
   
3. Edit the module info files that were generated for your new version(s), and alter DisplayName, Suppressed, and Vetted column values as desired.
   
   
   • You should leave all the other columns unchanged, except that if you wish, you may also make changes in the DefaultValue and Required columns of the modelParams file. The SCG and OAG configuration objects don't support these options, but altering them here will affect GenTarget files as if they were supported.
     
     
4. Rebuild GenFlow and test locally in API Studio to ensure that all expected GenTemplates appear with the expected names. This will require updating the swagger-codegen-version and openapi-generator-version property values in experimental/target-platform/com.modelsolv.reprezen.bundles/pom.xml. This is especially critical for major version changes, since otherwise API Studio will be built using the new GenFlow version but an old and presumably incompatible SCG or OAG version. The impact when there is no major version change is that the GUI may not reflect the proper lists of SCG and OAG generator modules, but the runtime should not be compromised.
   
   
5. Check changes into an appropriate GenFlow repo branch, create a PR, create a test build of API Studio, merge your PR, etc. All using the general component project build process documented here. (same link as above)

The review in step 3 need not be performed immediately when a library upgrade is performed, though this is recommended since new generator modules will not be available in API Studio until they are marked as vetted. CSV files can be edited and checked in at any time in the future, and will be used in subsequent builds.

All About SCG and OAG Versions

The way SCG and OAG versions are configured in GenFlow and API Studio is a little complicated. It's designed to strike a reasonable balance between stability and up-to-date-ness for users executing GenTargets.

• GenFlow itself is configured for a specific version of each of SCG and OAG, via dependencies in its root-level pom.xml file. All other project pom files in the GenFlow source tree are children of that pom file, and they all have their dependencies for SCG and OAG (and other libraires) managed from the root.
  
  
• Those same version numbers are separately configured in the pom file for API Studio's cmr.bundles plugin, which is at experimental/target-platform/com.modelsolv.reprezen.bundles/pom.xml in the API Studio source tree. There doesn't seem to be a good way to tell maven to use the GenFlow's version here; the plugin uses the maven-dependency-plugin's copy-dependencies goal, with transitive dependencies excluded. That's an all-or-nothing exclusion, unfortunately. So unless/until someone comes up with a better approach, this needs to be manually kept in sync between the two files.
  
  
• The GenFlow, OAG, and SCG libraries packaged in cmr.bundles will be used together, so it's important that compatible versions of these three appear in the cmr.bundles pom file. The versions declared in GenFlow's root pom file will be IGNORED in this context!
  
  
• The versions configured in cmr.bundles will also be used when generating root-level pom files for new model projects. Existing model projects will not be updated when the user upgrades to a new API Studio version, though that may become an option in the future. Users, of course, can edit their own pom files as needed, for this or other reasons.
  
  
• For most TeamCity builds, dependencies for GenFlow, SCG, and OAG will be in the form of version ranges of the form [M.m.p,M+1.0-alpha), where M.m.p is the version that was configured for cmr.bundles. (These versions are discovered at run-time by API Studio by examining the live bundle). So, for example, if version 3.3.4 of OAG appears in the cmr.bundles plugin included in a given API Studio build, the version range constructed by that build for the root-level pom file in a new model project will be [3.3.4,4.0-alpha). This means that when executing GenTargets, users will generally be using the most latest version of SCG and OAG that within the same major version as was used in the API Studio build that created the model project. This is where the balance between stability and freshness comes up. Using the range reduces stability, since the same GenTarget may produce different results from one day to the next. It increases freshness, because the latest presumed-compatible release is always used, so that bug fixes in OAG or in any of its generator modules will get picked up.
  
  
• For TeamCity development builds, as well as for local developer builds (unless REPREZEN_BUILD_CHANNEL_ID environment variable is set to a value like release), model project pom files will be configured with the specific versions obtained from cmr.bundles. This is important, since dev builds exist purely for the purpose of testing, and the last thing one wants in testing is an unpredictable configuration.
  
  

N.B. The version ranges create a potential danger, in that a user who adds the Sonatype snapshots repo to their pom file for any reason will most likely end up using pre-release snapshot versions of the SCG and OAG libraries. For example, at the time of this writing. OAG version range [3.3.4,4.0-alpha) will resolve to 3.3.5-SNAPSHOT if the snapshot repo is configured for the project. This should be a rare issue for API Studio users, and can be fixed by changing the SCG and OAG dependencies to use specific versions instead of ranges. It is quite unfortunate that there's no way to enable/disable snapshots for specific dependencies!