Merge pull request #1460 from nf-core/dev

Release 3.4.1

Author: maxulysse

.devcontainer/devcontainer.json

@@ -10,15 +10,7 @@
         "vscode": {
             // Set *default* container specific settings.json values on container create.
             "settings": {
-                "python.defaultInterpreterPath": "/opt/conda/bin/python",
-                "python.linting.enabled": true,
-                "python.linting.pylintEnabled": true,
-                "python.formatting.autopep8Path": "/opt/conda/bin/autopep8",
-                "python.formatting.yapfPath": "/opt/conda/bin/yapf",
-                "python.linting.flake8Path": "/opt/conda/bin/flake8",
-                "python.linting.pycodestylePath": "/opt/conda/bin/pycodestyle",
-                "python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
-                "python.linting.pylintPath": "/opt/conda/bin/pylint"
+                "python.defaultInterpreterPath": "/opt/conda/bin/python"
             },
 
             // Add the IDs of extensions you want installed when the container is created.

.editorconfig

@@ -18,7 +18,20 @@ end_of_line = unset
 insert_final_newline = unset
 trim_trailing_whitespace = unset
 indent_style = unset
-indent_size = unset
+[/subworkflows/nf-core/**]
+charset = unset
+end_of_line = unset
+insert_final_newline = unset
+trim_trailing_whitespace = unset
+indent_style = unset
 
 [/assets/email*]
 indent_size = unset
+
+# ignore Readme
+[README.md]
+indent_style = unset
+
+# ignore python
+[*.{py,md}]
+indent_style = unset

.github/CONTRIBUTING.md

@@ -9,9 +9,8 @@ Please use the pre-filled template to save time.
 However, don't be put off by this template - other more general issues and suggestions are welcome!
 Contributions to the code are even more welcome ;)
 
-:::info
-If you need help using or modifying nf-core/sarek then the best place to ask is on the nf-core Slack [#sarek](https://nfcore.slack.com/channels/sarek) channel ([join our Slack here](https://nf-co.re/join/slack)).
-:::
+> [!NOTE]
+> If you need help using or modifying nf-core/sarek then the best place to ask is on the nf-core Slack [#sarek](https://nfcore.slack.com/channels/sarek) channel ([join our Slack here](https://nf-co.re/join/slack)).
 
 ## Contribution workflow
 
@@ -27,6 +26,12 @@ If you're not used to this workflow with git, you can start with some [docs from
 
 ## Tests
 
+You have the option to test your changes locally by running the pipeline. For receiving warnings about process selectors and other `debug` information, it is recommended to use the debug profile. Execute all the tests with the following command:
+
+```bash
+nf-test test --profile debug,test,docker --verbose
+```
+
 When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
 Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
 
@@ -87,7 +92,7 @@ Once there, use `nf-core schema build` to add to `nextflow_schema.json`.
 
 Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
 
-The process resources can be passed on to the tool dynamically within the process with the `${task.cpu}` and `${task.memory}` variables in the `script:` block.
+The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.
 
 ### Naming schemes
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -18,7 +18,8 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/sare
 - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/sarek/tree/master/.github/CONTRIBUTING.md)
 - [ ] If necessary, also make a PR on the nf-core/sarek _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
 - [ ] Make sure your code lints (`nf-core lint`).
-- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
+- [ ] Ensure the test suite passes (`nf-test test tests/ --verbose --profile +docker`).
+- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
 - [ ] Usage Documentation in `docs/usage.md` is updated.
 - [ ] Output Documentation in `docs/output.md` is updated.
 - [ ] `CHANGELOG.md` is updated.

.github/RELEASE_CHECKLIST.md

@@ -3,22 +3,22 @@
 > This checklist is for our own reference, to help us prepare a new release
 
 1. Check that everything is ready to go
-   - Desired [PRs](https://github.com/nf-core/sarek/pulls) are merged
-   - [GHA CI](https://github.com/nf-core/sarek/actions/workflows/ci.yml) are passing on `dev`
-   - [GHA linting](https://github.com/nf-core/sarek/actions/workflows/linting.yml) are passing on `dev`
+
+- Desired [PRs](https://github.com/nf-core/sarek/pulls) are merged
+- [GHA CI](https://github.com/nf-core/sarek/actions/workflows/ci.yml) are passing on `dev`
+- [GHA linting](https://github.com/nf-core/sarek/actions/workflows/linting.yml) are passing on `dev`
+
 2. Increase version number following [semantic versioning](http://semver.org/spec/v2.0.0.html)
 3. Choose an appropriate codename for the release (if major or minor)
-   - i.e. Peaks in [Sarek National Park](https://en.wikipedia.org/wiki/Sarek_National_Park#Topography)
+
+- i.e. Peaks in [Sarek National Park](https://en.wikipedia.org/wiki/Sarek_National_Park#Topography)
+
 4. Sync `dev` and checkout a new branch for the release
 5. Bump version:
-   - `nf-core bump-version . <VERSION>`
-   - edit `docs/images/sarek_indices_subway.svg`
-   - generate a new `docs/images/sarek_indices_subway.png`
-   - edit `docs/images/sarek_subway.svg`
-   - generate a new `docs/images/sarek_subway.png`
-   - edit `docs/images/sarek_workflow.svg`
-   - generate a new `docs/images/sarek_workflow.png`
-   - edit `CHANGELOG`
+
+- `nf-core bump-version . <VERSION>`
+- edit `CHANGELOG`
+
 6. Make a PR to `master`
 7. Wait for reviews
 8. Merge said PR
@@ -27,15 +27,19 @@
 11. RT the nf-core automated tweet about the new released version
 12. Make a new branch from `dev`
 13. Checkout the `CHANGELOG.md` from `master`
-    - `git checkout upstream/master -- CHANGELOG.md`
+
+- `git checkout upstream/master -- CHANGELOG.md`
+
 14. Add a new `Unreleased` section in `CHANGELOG.md` for the `dev` version
 15. Checkout figures from `master`
-    - `git checkout upstream/master -- docs/images/sarek_indices_subway.svg`
-    - `git checkout upstream/master -- docs/images/sarek_indices_subway.png`
-    - `git checkout upstream/master -- docs/images/sarek_subway.svg`
-    - `git checkout upstream/master -- docs/images/sarek_subway.png`
-    - `git checkout upstream/master -- docs/images/sarek_workflow.svg`
-    - `git checkout upstream/master -- docs/images/sarek_workflow.png`
+
+- `git checkout upstream/master -- docs/images/sarek_indices_subway.svg`
+- `git checkout upstream/master -- docs/images/sarek_indices_subway.png`
+- `git checkout upstream/master -- docs/images/sarek_subway.svg`
+- `git checkout upstream/master -- docs/images/sarek_subway.png`
+- `git checkout upstream/master -- docs/images/sarek_workflow.svg`
+- `git checkout upstream/master -- docs/images/sarek_workflow.png`
+
 16. Make a PR to `dev`
 17. Wait for review
 18. Merge said PR

.github/workflows/awstest.yml

@@ -19,7 +19,7 @@ jobs:
       # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets
       - name: Post PR comment
         if: failure()
-        uses: mshick/add-pr-comment@v1
+        uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2
         with:
           message: |
             ## This PR is against the `master` branch :x:

.github/workflows/branch.yml

@@ -1,4 +1,4 @@
-name: test
+name: nf-core CI
 # This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors
 on:
   pull_request:
@@ -28,7 +28,7 @@ jobs:
       # Expose matched filters as job 'tags' output variable
       tags: ${{ steps.filter.outputs.changes }}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - uses: frouioui/paths-filter@main
         id: filter
@@ -52,14 +52,25 @@ jobs:
         NXF_VER:
           - "23.04.0"
           - "latest-everything"
+        exclude:
+          - tags: "sentieon/bwamem"
+          - tags: "sentieon/dedup"
+          - tags: "sentieon/dnascope"
+          - tags: "sentieon/dnascope_joint_germline"
+          - tags: "sentieon/dnascope_skip_filter"
+          - tags: "sentieon/haplotyper"
+          - tags: "sentieon/haplotyper_joint_germline"
+          - tags: "sentieon/haplotyper_skip_filter"
+          - NXF_VER: "latest-everything"
+            tags: "joint_germline"
     env:
       NXF_ANSI_LOG: false
       TEST_DATA_BASE: "${{ github.workspace }}/test-datasets"
       SENTIEON_LICENSE_BASE64: ${{ secrets.SENTIEON_LICENSE_BASE64 }}
 
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Hash Github Workspace
         id: hash_workspace
@@ -105,7 +116,7 @@ jobs:
 
       - name: Check out test data
         if: steps.cache-testdata.outputs.cache-hit != 'true'
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           repository: nf-core/test-datasets
           ref: sarek3
@@ -136,6 +147,9 @@ jobs:
         if: matrix.profile == 'conda'
         run: conda clean -a
 
+      - name: Disk space cleanup
+        uses: jlumbroso/free-disk-space@v1.3.1
+
       - name: Run pytest-workflow
         uses: Wandalen/wretry.action@v1
         with:
@@ -168,7 +182,7 @@ jobs:
       tags: ${{ steps.filter.outputs.changes }}
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Combine all tags.yml files
         id: get_tags
@@ -200,29 +214,65 @@ jobs:
           - "23.04.0"
           - "latest-everything"
         exclude:
+          - tags: "bcftools/concat"
+          - tags: "bcftools/mpileup"
+          - tags: "bcftools/sort"
           - tags: "bwa/index"
           - tags: "bwa/mem"
+          - tags: "bwamem2/index"
+          - tags: "bwamem2/mem"
           - tags: "cat/cat"
           - tags: "cat/fastq"
-          - tags: "custom/dumpsoftwareversions"
+          - tags: "cnvkit/batch"
+          - tags: "deepvariant"
+          - tags: "dragmap/align"
+          - tags: "dragmap/hashtable"
+          - tags: "ensemblvep/download"
+          - tags: "ensemblvep/vep"
           - tags: "fastp"
           - tags: "fastqc"
+          - tags: "fgbio/fastqtobam"
+          - tags: "freebayes"
+          - tags: "gatk4/applybqsr"
+          - tags: "gatk4/baserecalibrator"
+          - tags: "gatk4/estimatelibrarycomplexity"
+          - tags: "gatk4/genomicsdbimport"
+          - tags: "gatk4/haplotypecaller"
+          - tags: "gatk4/markduplicates"
+          - tags: "gatk4/mergevcfs"
+          - tags: "gatk4/mutect2"
+          - tags: "gatk4spark/applybqsr"
+          - tags: "gatk4spark/markduplicates"
+          - tags: "mosdepth"
+          - tags: "multiqc"
+          - tags: "samblaster"
+          - tags: "samtools/mpileup"
           - tags: "samtools/stats"
+          - tags: "snpeff/snpeff"
+          - tags: "strelka/germline"
+          - tags: "strelka/somatic"
+          - tags: "subworkflows/utils_nfvalidation_plugin"
+          - tags: "tabix/bgziptabix"
+          - tags: "tabix/tabix"
+          - tags: "tiddit/sv"
           - tags: "untar"
     env:
       NXF_ANSI_LOG: false
       TEST_DATA_BASE: "${{ github.workspace }}/test-datasets"
       SENTIEON_LICENSE_BASE64: ${{ secrets.SENTIEON_LICENSE_BASE64 }}
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - uses: actions/setup-java@v3
         with:
           distribution: "temurin"
           java-version: "17"
-      - name: Setup Nextflow
+
+      - name: Install Nextflow ${{ matrix.NXF_VER }}
         uses: nf-core/setup-nextflow@v1
+        with:
+          version: "${{ matrix.NXF_VER }}"
 
       - name: Cache nf-test installation
         id: cache-software
@@ -277,6 +327,9 @@ jobs:
           SENTIEON_AUTH_DATA_BASE64=$(echo -n "$SENTIEON_AUTH_DATA" | base64 -w 0)
           nextflow secrets set SENTIEON_AUTH_DATA_BASE64 $SENTIEON_AUTH_DATA_BASE64
 
+      - name: Disk space cleanup
+        uses: jlumbroso/free-disk-space@v1.3.1
+
       # Test the module
       - name: Run nf-test
         run: |

.github/workflows/ci.yml

@@ -10,7 +10,7 @@ jobs:
       issues: write
       pull-requests: write
     steps:
-      - uses: actions/stale@v7
+      - uses: actions/stale@28ca1036281a5e5922ead5184a1bbf96e5fc984e # v9
         with:
           stale-issue-message: "This issue has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment otherwise this issue will be closed in 20 days."
           stale-pr-message: "This PR has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment if it is still useful."