Skip to content

Refactor pipeline downloads command to use nextflow inspect for container detection #3634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 122 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 101 commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
ae0d4dd
Moved all singularity-related download functions into a new file
muffato Mar 25, 2025
bbd297f
Added a helper method to use a temporary file as intermediate output
muffato Mar 25, 2025
a9b2b00
First round of fixing the tests
muffato Mar 25, 2025
2797752
WIP: Start refactoring
JulianFlesch Mar 26, 2025
dc333b3
downloads.singularity functions should rely on proper parameters, no …
muffato Mar 25, 2025
fbc09ac
Duplicated with the line above
muffato Mar 25, 2025
fee7182
Added tests for utils.intermediate_file and fixed it !
muffato Mar 25, 2025
c507d9e
Simpler implementation
muffato Mar 25, 2025
aba5233
Added helper functions to manage the main task so that callers don't …
muffato Mar 26, 2025
0403a37
No need to build a temp path here
muffato Mar 26, 2025
159c34e
More tests
muffato Mar 26, 2025
693da64
Fixed the symlink_registries tests
muffato Mar 26, 2025
c5379fb
Rewrote to support older versions of Python
muffato Mar 26, 2025
9bf7bb3
Adds nextflow command call to extract containers
JulianFlesch Mar 26, 2025
5dc30e2
Move import
JulianFlesch Mar 26, 2025
563a330
Debug outpu
JulianFlesch Mar 26, 2025
8ee8eed
Added a dest for the main_task-management methods added to DownloadPr…
muffato Mar 26, 2025
71ab538
Added some tests for the DownloadProgress object
muffato Mar 26, 2025
a141820
Turned two methods into functions, simplified testing
muffato Mar 26, 2025
4133c90
Test comments
muffato Mar 26, 2025
2b8fbd6
Theoretically it could be None, so check it really is a string
muffato Mar 26, 2025
72b6d60
Added docker class to match singularity class
nikhil Mar 26, 2025
7813f30
More type hints
muffato Mar 27, 2025
cb71d42
More tests for get_container_filename
muffato Mar 27, 2025
2d8cb4d
No need to keep a wraper as a method
muffato Mar 27, 2025
4bfb45e
Better comments
muffato Mar 27, 2025
d6b8f6c
New class to support the downloads
muffato Mar 27, 2025
4abd736
Introduced a context manager that creates a new sub task. Useful to g…
muffato Mar 27, 2025
31f5627
Forgot that pytest.raises exists !
muffato Mar 27, 2025
06d3760
Moved the download class to utils
muffato Mar 27, 2025
3857fae
The output path cannot be a directory or a symbolic link
muffato Mar 27, 2025
31fe914
Basic test for FileDownloader.download_file
muffato Mar 27, 2025
02cc2ca
Empty files are not allowed
muffato Mar 27, 2025
397ff11
Removes inspect param that keeps failing
JulianFlesch Mar 27, 2025
0107a6c
Description
JulianFlesch Mar 27, 2025
141d4d8
Adds debugging section
JulianFlesch Mar 27, 2025
b648bc1
Test log.debug too
muffato Mar 27, 2025
467ba3f
Test the kill_with_fire flag
muffato Mar 27, 2025
ea4bc89
Refactored the FileDownloader
muffato Mar 27, 2025
8038046
Merge branch 'dev' into dev-download
ErikDanielsson Jun 16, 2025
e4327f1
Merge branch 'singularity_downloads' into dev-download
ErikDanielsson Jun 16, 2025
208daca
Merge branch 'feature/docker_downloads' into dev-download
ErikDanielsson Jun 16, 2025
a446ee9
Create a barely functioning version
ErikDanielsson Jun 16, 2025
b9b8388
Update and test download.py
ErikDanielsson Jun 23, 2025
40be0b6
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jun 23, 2025
cb8a93b
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jun 23, 2025
7ca0875
Move WorkflowRepo to separate file
ErikDanielsson Jun 23, 2025
1684127
Start adding in docker support, add hiding of progress bars
ErikDanielsson Jun 24, 2025
0739df0
Start adding container base class
ErikDanielsson Jun 24, 2025
9913e56
Move legacy container finding into utils (for now) and update tests
ErikDanielsson Jun 25, 2025
c65d09f
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jun 25, 2025
0e0a5b8
Add mock pipeline, and move some stuff around
ErikDanielsson Jun 26, 2025
8160181
Try to refactor singularity and docker container handling in this branch
ErikDanielsson Jul 7, 2025
9b695f2
Remove refactoring code and add some nice messages
ErikDanielsson Jul 7, 2025
7670666
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jul 7, 2025
b64a77f
Nearing working refactor
ErikDanielsson Jul 7, 2025
35df64b
Merge branch 'download-refactor-again' into refactor-docker-singularity
ErikDanielsson Jul 7, 2025
a18f611
Further refactoring
ErikDanielsson Jul 7, 2025
6b44e74
Make DockerError and SingularityError classes (too much customization…
ErikDanielsson Jul 7, 2025
41c7332
Integrate new classes and update tests
ErikDanielsson Jul 8, 2025
c5cf8e0
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jul 8, 2025
275adab
Fix failing tests
ErikDanielsson Jul 8, 2025
56a76b5
Add error message for weird containers and testing of it
ErikDanielsson Jul 8, 2025
d58c282
Remove singularity and docker files without subclassing
ErikDanielsson Jul 8, 2025
1964d50
Make concurrency nicer for docker
ErikDanielsson Jul 9, 2025
6de9f41
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jul 9, 2025
d1ce487
Organize mock modules
ErikDanielsson Jul 9, 2025
05adb4d
Remove unused code
ErikDanielsson Jul 9, 2025
ced8df1
Apply suggestions from code review
ErikDanielsson Jul 14, 2025
7a2da94
Remove error message for variable container name and corresponding tests
ErikDanielsson Jul 14, 2025
7dba881
Revert removel of .sif extension for now
ErikDanielsson Jul 14, 2025
a49ee12
os.path -> pathlib.Path and typing
ErikDanielsson Jul 14, 2025
a413d93
Fix failing tests
ErikDanielsson Jul 15, 2025
e2ddf3d
^^
ErikDanielsson Jul 15, 2025
af44891
More typing, docker concurrency, and properties
ErikDanielsson Jul 16, 2025
f4c5e66
Update tests and progress bar
ErikDanielsson Jul 16, 2025
cb6cfc9
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jul 16, 2025
7d7b391
Do not select singularity silently
ErikDanielsson Jul 16, 2025
1e414cd
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jul 21, 2025
4b5f617
Fix typing issue
ErikDanielsson Jul 21, 2025
828f1eb
Change name of function and remove absolute container URI from non-si…
ErikDanielsson Jul 21, 2025
414b81d
Move nf stuff from download/utils to utils
ErikDanielsson Jul 21, 2025
f4c7450
Check logic for self.amend_cachedir
ErikDanielsson Jul 21, 2025
5bc767b
Try fixing tests
ErikDanielsson Jul 21, 2025
c48886e
Test with python 3.9
ErikDanielsson Jul 21, 2025
c898730
Remove legacy code
ErikDanielsson Jul 22, 2025
da0a94f
Move prompts to singularity
ErikDanielsson Jul 22, 2025
caeacce
Move singularity prompts into SingularityFetcher
ErikDanielsson Jul 22, 2025
0afb74a
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jul 22, 2025
39af8e3
Merge branch 'move-singularity-code' into download-refactor-again
ErikDanielsson Jul 22, 2025
36cc046
Move docker load message to DockerFetcher
ErikDanielsson Jul 22, 2025
bde85f9
Add test and test_full profiles
ErikDanielsson Jul 22, 2025
bb84582
Add nf inspect created dirs to gitignore
ErikDanielsson Jul 22, 2025
0141a16
Update changelog
ErikDanielsson Jul 22, 2025
cbd54fa
Remove old test data
ErikDanielsson Jul 22, 2025
b444f1a
Remove unused file
ErikDanielsson Jul 22, 2025
696230c
Add more docker tests
ErikDanielsson Jul 22, 2025
d6c4895
Apply suggestions from code review
ErikDanielsson Jul 23, 2025
d9b0107
Further changes from code review
ErikDanielsson Jul 23, 2025
eb46d9c
Update nf_core/pipelines/download/container_fetcher.py
ErikDanielsson Jul 23, 2025
29506be
Update CLI info for docker containers
ErikDanielsson Jul 23, 2025
b3c0677
Apply suggestions from code review
ErikDanielsson Jul 30, 2025
3686f23
ruff
ErikDanielsson Jul 30, 2025
f13b067
Typing, nicer UX (progress bars, better support for multiple revision…
ErikDanielsson Jul 30, 2025
af744e7
Update CHANGELOG.md
ErikDanielsson Jul 30, 2025
934fd7c
Merge branch 'dev' into download-refactor-again
ErikDanielsson Jul 30, 2025
7615309
Implement suggestions from @JulianFlesch
ErikDanielsson Aug 1, 2025
06262d6
Use intermediate file in singularity pull
ErikDanielsson Aug 1, 2025
a19b2c6
Fix error due to misuse of `Path.with_suffix`
ErikDanielsson Aug 1, 2025
1f9669d
More typing and some docs
ErikDanielsson Aug 1, 2025
6cac4b3
Fix failing tests
ErikDanielsson Aug 1, 2025
8e4033f
Merge branch 'dev' into download-refactor-again
ErikDanielsson Aug 1, 2025
ec703ef
Update message for nf version fail and propagate hide_progress correc…
ErikDanielsson Aug 8, 2025
be145e1
Make error messages nicer
ErikDanielsson Aug 11, 2025
37d183d
Apply suggestions from code review
ErikDanielsson Aug 12, 2025
1b5b2ea
Merge branch 'dev' into download-refactor-again
ErikDanielsson Aug 12, 2025
fe1da84
Revert --parallel to --parallel-downloads
ErikDanielsson Aug 12, 2025
1876ccc
Merge branch 'dev' into download-refactor-again
ErikDanielsson Aug 13, 2025
66513ac
Update CLI test with --parallel-downloads
ErikDanielsson Aug 13, 2025
ad2130c
Add mock for prompt
ErikDanielsson Aug 14, 2025
006b19e
Mock prompt everywhere
ErikDanielsson Aug 14, 2025
951bbe7
Make parent dir
ErikDanielsson Aug 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib64/
Expand Down Expand Up @@ -117,3 +116,7 @@ pip-wheel-metadata

# Textual
snapshot_report.html

# Nextflow inspect
tests/pipelines/null
tests/pipelines/.nextflow
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@

- don't read param expressions with spaces as params ([#3674](https://github.com/nf-core/tools/pull/3674))
- Update marocchino/sticky-pull-request-comment digest to 7737449 ([#3681](https://github.com/nf-core/tools/pull/3681))
- Refactor downloads command ([#3634](https://github.com/nf-core/tools/pull/3634))
- Split `download.py` into subdirectory `download/`
- Use `nextflow inspect` for container discovery and remove legacy regex container discovery (requires Nextflow >= 25.04.04)
- Add support for downloading docker images into tar archives
- Change long flag `--parallel-downloads` to `--parallel`. Short flag remains.
- Add pipeline to test data to be compatible with `nextflow inspect`

## [v3.3.2 - Tungsten Tamarin Patch 2](https://github.com/nf-core/tools/releases/tag/3.3.2) - [2025-07-08]

Expand Down
20 changes: 10 additions & 10 deletions nf_core/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@
)
from nf_core.components.components_completion import autocomplete_modules, autocomplete_subworkflows
from nf_core.components.constants import NF_CORE_MODULES_REMOTE
from nf_core.pipelines.download import DownloadError
from nf_core.pipelines.download.download import DownloadError
from nf_core.pipelines.list import autocomplete_pipelines
from nf_core.utils import check_if_outdated, nfcore_logo, rich_force_colors, setup_nfcore_dir

Expand Down Expand Up @@ -400,33 +400,33 @@ def command_pipelines_lint(
@click.option(
"-s",
"--container-system",
type=click.Choice(["none", "singularity"]),
type=click.Choice(["none", "singularity", "docker"]),
help="Download container images of required software.",
)
@click.option(
"-l",
"--container-library",
multiple=True,
help="Container registry/library or mirror to pull images from.",
help="Container registry/library or mirror to pull images from. Not available for Docker containers.",
)
@click.option(
"-u",
"--container-cache-utilisation",
type=click.Choice(["amend", "copy", "remote"]),
help="Utilise a `singularity.cacheDir` in the download process, if applicable.",
help="Utilise a `singularity.cacheDir` in the download process, if applicable. Not available for Docker containers.",
)
@click.option(
"-i",
"--container-cache-index",
type=str,
help="List of images already available in a remote `singularity.cacheDir`.",
help="List of images already available in a remote `singularity.cacheDir`. Not available for Docker containers.",
)
@click.option(
"-d",
"--parallel-downloads",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, @mirpedrol and @mashehu are quite strict on renaming existing CLI arguments, since breaking changes would, according to SemVer, require a new major version release (4.0.0). But maybe Click would map a --parallel-downloads automatically to --parallel, so it is backwards compatible?

Apart from that, CLI parameters in download are mostly standardized in a way that the first letter of the last term is their short flag: -d / --parallel-downloads , -s / --container-system, -i / --container-cache-index and others.

What was the reason you wish to rename? As far as I remember, it used to be -p and --parallel in 2.x.x and was specifically changed, because Seqera went from Tower to Platform, and we needed to free the short letter -p

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can be sloppy enough to see a long name change of a parameter not as a breaking change, especially if it is just because of semantics 🙂

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this. I wouldn't change unless there is a good reason to.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair. But particularly because there is a short flag, I also do not necessarily see the need to abbreviate the long form. But that is not a hill to die on for me, so have it as you like.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do as you see fit, I will also not fight anyone on this :)
FYI: The short flag -t as in tasks / threads, is also still available.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

joining late to the party, but I would vote for keeping the long --parallel-downloads, otherwise I don't see where the short -d comes from.
But also not a strong request, so happy to accept the change if this is the preferred option.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the interest of not changing the API I think I agree with you and others @mirpedrol -- I'll switch back the change.

I think Julians suggestion with --tasks/-t is even better though, so if you agree you could perhaps put it on the pile of things you change before a 4.0.0 release (if there ever is one)?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we decided to not use -t to not confuse people with the previous option -t for tower, do I remember this well?
We should consider if it has been long enough for people to not think about "tower" anymore

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At NGI, we continue calling it Tower, since Platform is simply a more cumbersome generic name.

But I doubt that many will still remember the old parameters - judged by the Slack discussions, very few people ever used that functionality anyway. So fine with me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see -- slightly annoying then that platform/tower perfectly cover the two normal parallelism short flags... I've reverted the change now 👍

"--parallel",
type=int,
default=4,
help="Number of parallel image downloads",
help="Number of allowed parallel tasks",
)
@click.pass_context
def command_pipelines_download(
Expand All @@ -443,7 +443,7 @@ def command_pipelines_download(
container_library,
container_cache_utilisation,
container_cache_index,
parallel_downloads,
parallel,
):
"""
Download a pipeline, nf-core/configs and pipeline singularity images.
Expand All @@ -462,7 +462,7 @@ def command_pipelines_download(
container_library,
container_cache_utilisation,
container_cache_index,
parallel_downloads,
parallel,
)


Expand Down Expand Up @@ -2359,7 +2359,7 @@ def command_create_params_file(pipeline, revision, output, force, show_hidden):
@click.option(
"-s",
"--container-system",
type=click.Choice(["none", "singularity"]),
type=click.Choice(["none", "singularity", "docker"]),
help="Download container images of required software.",
)
@click.option(
Expand Down
5 changes: 3 additions & 2 deletions nf_core/commands_pipelines.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ def pipelines_download(
container_library,
container_cache_utilisation,
container_cache_index,
parallel_downloads,
parallel,
):
"""
Download a pipeline, nf-core/configs and pipeline singularity images.
Expand All @@ -198,7 +198,8 @@ def pipelines_download(
container_library,
container_cache_utilisation,
container_cache_index,
parallel_downloads,
parallel,
ctx.obj["hide_progress"],
)
dl.download_workflow()

Expand Down
Loading