Merge develop into main (#880)

In anticipation of v0257 release

Author: mathias-arm

.gitignore

@@ -42,6 +42,7 @@ tmp/
 *.axf
 *.S19
 *.bin
+*.hex
 *.dep
 *.FLM
 

README.md

@@ -1,13 +1,7 @@
-[![DAPLink](/docs/images/daplink-website-logo-link.png)](https://armmbed.github.io/DAPLink/)
 
-# Warning: Development frozen on `main` branch
+[![DAPLink](/docs/images/daplink-website-logo-link.png)](https://daplink.io/)
 
-All development and pull requests should now happen on [`develop`](https://github.com/ARMmbed/DAPLink/tree/develop) branch where we are preparing the next version.
-
-**Note:** The `main` branch requires an Arm Compiler 5 or Keil MDK license to build. See the
-[`develop`](https://github.com/ARMmbed/DAPLink/tree/develop) branch for gcc and Arm
-Compiler 6 support. As soon as testing is complete, gcc will become the default compiler and AC5 support will be
-dropped.
+[![Linux Build (develop)](https://github.com/ARMmbed/DAPLink/actions/workflows/linux.yml/badge.svg?branch=develop)](https://github.com/ARMmbed/DAPLink/actions/workflows/linux.yml)
 
 ----
 
@@ -25,12 +19,18 @@ For more detailed usability information [see the users guide.](docs/USERS-GUIDE.
 
 ## Compatibility
 There are many ARM microcontroller-based Hardware Interface Circuits (HICs) that DAPLink interface firmware runs on. These can be found as standalone boards (debugger) or as part of a development kit. Some branded circuits that are known to be IO compatible are:
-* [NXP OpenSDA based on K20, K22 and KL26](http://www.nxp.com/products/software-and-tools/run-time-software/kinetis-software-and-tools/ides-for-kinetis-mcus/opensda-serial-and-debug-adapter:OPENSDA)
+
+* [Maxim Integrated MAX32625PICO based on MAX32625](https://www.maximintegrated.com/en/products/microcontrollers/MAX32625PICO.html)
+* Nuvoton Nu-Link2-Me based on M48SSIDAE
 * [NXP LPC-Link2 based on LPC11U35 or LPC4322](https://www.nxp.com/support/developer-resources/hardware-development-tools/lpcxpresso-boards:LPCXPRESSO-BOARDS)
+* [NXP MCU-LINK on LPC55xx](https://www.nxp.com/design/microcontrollers-developer-resources/mcu-link-debug-probe:MCU-LINK)
+* [NXP OpenSDA based on K20, K22, KL26Z and KL27Z](http://www.nxp.com/products/software-and-tools/run-time-software/kinetis-software-and-tools/ides-for-kinetis-mcus/opensda-serial-and-debug-adapter:OPENSDA)
 * [Segger J-Link OB based on Atmel SAM3U](https://www.segger.com/products/debug-probes/j-link/models/j-link-ob/)
+* [STMicroelectronics ST-LINK/V2 (on NUCLEO boards) based on STM32F103CB](https://www.st.com/en/evaluation-tools/stm32-nucleo-boards.html)
+
 
 ## Releases
-There are many board builds (board = HIC + target combination) created from this repository. Quarterly releases will contain new features and bugfixes. Standalone bugfixes are released once reported, verified and fixed. Both quarterly and bugfix releases will result in the build number being incremented. Many development kits and products ship with DAPLink interface firmware or are capable of running DAPLink firmware. **[The current release builds and instructions for updating DAPLink interface firmware is hosted on the DAPLink release site.](https://armmbed.github.io/DAPLink/)** Release notes and previous release builds can be found under GitHub releases.
+There are many board builds (board = HIC + target combination) created from this repository. Quarterly releases will contain new features and bugfixes. Standalone bugfixes are released once reported, verified and fixed. Both quarterly and bugfix releases will result in the build number being incremented. Many development kits and products ship with DAPLink interface firmware or are capable of running DAPLink firmware. **[The current release builds and instructions for updating DAPLink interface firmware is hosted on the DAPLink release site.](https://daplink.io/)** Release notes and previous release builds can be found under GitHub releases.
 
 ## Contribute
 

docs/DEVELOPERS-GUIDE.md

@@ -1,19 +1,20 @@
 # DAPLink Developers Guide
 
 ## Setup
-DAPLink sources can be compiled using Keil MDK-ARM or mbed cli tool with arm compiler, which could be run both on Linux and Windows. See  [here](AUTOMATED_TESTS.md) for test instructions on both OS and Mac.
+
+DAPLink sources are compiled using `progen` (from [project-generator](https://github.com/project-generator/project_generator)) which can be run on Linux, MacOS and Windows.
 
 Install the necessary tools listed below. Skip any step where a compatible tool already exists.
 
-* Install [Python 2, 2.7.11 or above](https://www.python.org/downloads/) . Add to PATH.
+* Install [Python 3](https://www.python.org/downloads/) . Add to PATH.
 * Install [Git](https://git-scm.com/downloads) . Add to PATH.
-* Install [Keil MDK-ARM](https://www.keil.com/download/product/), preferably version 5. Set environment variable "UV4" to
-  the absolute path of the UV4 executable if you don't install to the default location. Note that "UV4" is what's used for
-  both MDK versions 4 and 5. This step can be skipped if you plan to use mbed cli, but you still need Arm Compiler 5, and
-  MDK is required to debug.
+* Install a compiler:
+    * [GNU Arm Embedded Toolchain](https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads) . This compiler will be identified as `gcc_arm`.
+    * [Arm Compiler 6](https://developer.arm.com/tools-and-software/embedded/arm-compiler) . This compiler will be identified as `armclang`. Only supported on Linux and Windows.
+    * [Keil MDK](https://developer.arm.com/tools-and-software/embedded/keil-mdk) or [Arm Compiler 5](https://developer.arm.com/tools-and-software/embedded/arm-compiler/downloads/legacy-compilers#arm-compiler-5). This compiler will be identified as `armcc`. Only supported on Linux and Windows.
+* Install `make` (tested with [GNU Make](https://www.gnu.org/software/make)). [CMake](https://cmake.org) can alternatively be used in conjunction with different implementations of `make` as well as [ninja](https://ninja-build.org).
 * Install virtualenv in your global Python installation eg: `pip install virtualenv`.
 
-
 **Step 1.** Initial setup.
 
 Get the sources and create a virtual environment
@@ -25,101 +26,83 @@ $ pip install virtualenv
 $ virtualenv venv
 ```
 
-**Step 2.** One-time mbed-cli setup.
-
-This step is only required once if you are planning to use the mbed-cli build method.
-
-First run step 3 below to activate the virtual environment. Then execute these commands.
-```
-$ mbed deploy
-$ mbed config root .
-$ mbed config ARM_PATH <FULL_PATH_TO_ARMCC_FOLDER>
-```
-
-
 ## Activate virtual environment
-**Step 3.** Activate the virtual environment and update requirements. This is necessary when you open a new shell. **This should be done every time you pull new changes**
+**Step 2.** Activate the virtual environment and update requirements. This is necessary when you open a new shell. **This should be done every time you pull new changes**
 
 ```
 $ venv/Scripts/activate   (For Linux)
 $ venv/Scripts/activate.bat   (For Windows)
-$ pip install -r requirements.txt
+(venv) $ pip install -r requirements.txt
 ```
 
-
 ## Build
 **This should be done every time you pull new changes**
 
-There are two ways to build DAPLink. You can generate Keil MDK project files and build within MDK. MDK is also used to debug DAPLink running on the interface chip. Or, you can use the `mbedcli_compile.py` script to build projects from the command line without requiring MDK.
+There are two ways to build DAPLink. You can use the `progen` command-line tool from project-generator or the `tools/progen_compile.py` wrapper tool.
 
+**Step 3.1.** Using `progen_compile.py`
 
-**Step 4.1.** For MDK progen compilation.
-
-This command generates MDK project files under the `projectfiles/uvision` directory.
 ```
-$ progen generate -t uvision
+(venv) $ python tools/progen_compile.py [-t <tool>] [--clean] [-v] [--parallel] [<project> [<project> ...]]
 ```
 
-To only generate one specific project, use a command like this:
-```
-progen generate -f projects.yaml -p stm32f103xb_stm32f746zg_if -t uvision
-```
-These options to `progen` set the parameters:
-- `-f` for the input projects file
-- `-p` for the project name
-- `-t` to specify the IDE name 
+* `-t <tool>`: choose the toolchain to build. The default is `make_gcc_arm`. Other options tested are `make_gcc_arm`, `make_armclang`, `make_armcc`, `cmake_gcc_arm`, `cmake_armclang`, `cmake_armcc`.
+* `--clean`: will clear existing compilation products and force recompilation of all files.
+* `-v`: will make compilation process more verbose (typically listing all commands with their arguments)
+* `--parallel`: enable parallel compilation within a project (projects are compiled sequentially).
+* `<project>`: target project to compile (e.g. `stm32f103xb_bl`, `lpc11u35_if`), if none is specified all (140 to 150) projects will be compiled.
 
+**Step 3.2.** Using `progen` with `make`
 
-**Step 4.2.** For mbed-cli project compilation
+The following command combines generation and compilation:
 
-This command will build all projects:
 ```
-$ tools/mbedcli_compile.py
+(venv) $ progen generate -t make_gcc_arm -p <project> -b
 ```
 
-To build only a subset of projects, add the project name(s) to the end of the command line. Valid project names are listed
-in the usage text shown with `--help`. The first time you build after each pull you should add `--clean` to perform a
-complete re-build.
+Alternatively one can separate those task:
+```
+(venv) $ progen generate -t make_gcc_arm -p <project>
+(venv) $ make -C projectfiles/make_gcc_arm/<project> [<target>] [VERBOSE=1]
+```
+Where:
+* `<project>`: target project to compile (e.g. `stm32f103xb_bl`, `lpc11u35_if`).
+* `<target>`: build target, can be `all`, `clean` or `help`.
+* `VERBOSE=1`: display additional compilation information.
 
-## Contribute
-We would love to have your changes! Pull requests should be made once a changeset is [rebased onto Master](https://www.atlassian.com/git/tutorials/merging-vs-rebasing/workflow-walkthrough). See the [contributing guide](../CONTRIBUTING.md) for detailed requirements and guidelines for contributions.
+**Step 3.3.** Using `progen` with `cmake`
 
-## Mbed-cli compile environment
+The following command combines generation and compilation:
 
-### Features
-- Support both Python 2.x and 3.x versions.
-- Can compile a list of projects or the all of the projects.
-- Can generate the release directory with one command.
+```
+(venv) $ progen generate -t cmake_gcc_arm -o generator=<generator> -p <project> -b
+```
+* `<generator>`: use `CMake` generators among the following options:
+    * `make` (`Unix Makefiles`)
+    * `mingw-make` (`MinGW Makefiles`)
+    * `msys-make` (`MSYS Makefiles`, untested)
+    * `ninja` (`Ninja`)
+    * `nmake` (`NMake Makefiles`)
+* `<project>`: target project to compile (e.g. `stm32f103xb_bl`, `lpc11u35_if`).
 
-### Prerequisite
-mbed-cli is included in `requirements.txt`, so it will be installed automatically when configuring
-your development environment using the steps described above.
+**Step 3.4.** Using `progen` for MDK compilation.
 
-### `tools/mbedcli_compile.py` script
-Arguments
+This command generates MDK project files under the `projectfiles/uvision` directory.
+```
+$ progen generate -t uvision
 ```
-positional arguments:
-  projects              Selectively compile only the firmware specified
-                        otherwise all projects
 
-optional arguments:
-  -h, --help            show this help message and exit
-  --release             Create a release with the yaml version file
-  --build-folder BUILD_FOLDER
-                        Release directory to grab files from
-  --release-folder RELEASE_FOLDER
-                        Directory to create and place files in
-  --toolchain TOOLCHAIN
-                        Toolchain directory if present
-  --clean               Rebuild or delete build folder before compile
-  -v                    Pass verbosity level to mbed compile  -vv for more
+To only generate one specific project, use a command like this:
+```
+progen generate -f projects.yaml -p stm32f103xb_stm32f746zg_if -t uvision
 ```
-Valid projects are listed on help.
+These options to `progen` set the parameters:
+- `-f` for the input projects file
+- `-p` for the project name
+- `-t` to specify the IDE name
 
-Generate files needed by mbed-cli
-* `custom_profile.json` lists toolchain profile or compile flags parsed from the yaml files
-* `custom_targets.json` contains platform information for specific hics.
-* `.mbedignore` filters all files not needed for the project.
+## Contribute
+We would love to have your changes! Pull requests should be made once a changeset is [rebased onto main](https://www.atlassian.com/git/tutorials/merging-vs-rebasing/workflow-walkthrough). See the [contributing guide](../CONTRIBUTING.md) for detailed requirements and guidelines for contributions.
 
 ## Port
 There are three defined ways in which DAPLink can be extended. These are adding target support, adding board support and adding HIC support. Details on porting each of these can be found below.
@@ -128,15 +111,24 @@ There are three defined ways in which DAPLink can be extended. These are adding
 * [Adding a new board](PORT_BOARD.md)
 * [Adding a new HIC](PORT_HIC.md)
 
-
 ## Test
 DAPLink has an extensive set of automated tests written in Python. They are used for regression testing, but you can use them to validate your DAPLink port. Details are [here](AUTOMATED_TESTS.md)
 
 An option to search for the daplink firmware build in uvision and mbedcli build folders.
-`python test/run_test.py --project-tool mbedcli ...` or `python test/run_test.py --project-tool uvision ...`.
+`python test/run_test.py --project-tool make_gcc_arm ...` or `python test/run_test.py --project-tool uvision ...`.
 
 ## Release
 
+### Release using `progen_compile.py`
+
+* Create a tag with the correct release version and push it to github
+* Clean the repo you will be building from by running 'git clean -xdf' followed by 'git reset --hard'
+* Run the `progen_compile.py` command with the following parameters (see above for the `-t` flag):
+```
+(venv) $ python tools/progen_compile.py [-t <tool>] --clean -v --parallel --release
+```
+* All release deliverables will be created and stored in `firmware_<version>` (where `<version>` is the DAPLink version). Save this wherever your builds are stored.
+
 ### Release using uvision
 
 DAPLink contains scripts to automate most of the steps of building a release. In addition to building the release, these scripts also save relevant build information such as git SHA and python tool versions so the same build can be reproduced. The recommended steps for creating a release are below.
@@ -150,20 +142,6 @@ Note: A previous build can be reproduced by using the ``build_requirements.txt``
 To do this add the additional argument ``build_requirements.txt`` when calling ``build_release_uvision.bat`` in step 2.
 This will install and build with the exact version of the python packages used to create that build.
 
-### Release using mbedcli
-
-If the project list is not specify, all interface and booloader projects will be compiled. If `--release_version` is given, a folder (`firmware` on default or specified by `--release_folder`, to be concatenated with the version number), will be generated with the bin, update.yml and zip file containing the bins for release
-```
-$ venv/Scripts/activate
-$ pip install -r requirements3.txt
-$ tools/mbedcli_compile.py --release_version 0250 --release_folder firmware
-```
-
-There is an intermediate step in uvision environment in creating a release directory. This step is not needed in mbedcli environment but to make this equivalent directory invoke
-`copy_release_files.py --project-tool mbedcli`
-To make a release directory from the step above.
-`package_release_files.py SRC_DIR DEST_DIR VERSION_NUMBER --toolchain ARM`
-
 ## MDK
 If you want to use the MDK (uVision) IDE to work with the DAPLink code, you must launch it in the right environment. The project will fail to build otherwise. To launch uVision properly, use ``tools/launch_uvision.bat``