Adopt the array api #885

mwcraig · 2025-03-19T15:17:48Z

This PR makes the necessary changes to adopt the array API. There are still some things to sort out:

Run tests mostly against numpy, but do one run against jax (or some other non-numpy array library)
Check for remaining invocations of numpy in the code.
Check whether changing .data_arr.mask to .data_arr_mask constitutes an API change. If it does (and I think it does) then patch things up so that .data_arr.mask points to .data_arr_mask
Clean up any remaining handling of immutable arrays (array_api_extra makes handling those easy)

Once those are done the next step will be to find an outside reviewer for this...

codecov · 2025-03-20T15:04:20Z

Codecov Report

❌ Patch coverage is 95.57522% with 20 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@5aa9219). Learn more about missing BASE report.
⚠️ Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
ccdproc/_ccddata_wrapper_for_array_api.py	86.95%	15 Missing ⚠️
ccdproc/core.py	98.25%	3 Missing ⚠️
ccdproc/image_collection.py	92.30%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #885   +/-   ##
=======================================
  Coverage        ?   97.06%           
=======================================
  Files           ?        8           
  Lines           ?     1531           
  Branches        ?        0           
=======================================
  Hits            ?     1486           
  Misses          ?       45           
  Partials        ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

eteq · 2025-06-24T21:51:22Z

As a quick FYI I ran the test suite in this PR on a machine with a NVidia GPU with CCDPROC_ARRAY_LIBRARY=cupy and got a whole pile of errors (sent the log directly to @mwcraig but can also share snippets here if someone else wants to see them)

pyproject.toml

ccdproc/combiner.py

nstarman · 2025-06-25T18:23:58Z

pyproject.toml

@@ -8,17 +8,19 @@ dynamic = ["version"]
 description = "Astropy affiliated package"
 readme = "README.rst"
 license = { text = "BSD-3-Clause" }
-requires-python = ">=3.8"
+requires-python = ">=3.11"


Adopt Spec 0? Has that been adopted?

Good catch!

Wait, misread this -- what is Spectrum 0?

https://scientific-python.org/specs/spec-0000/

So the move here would be to bump it up to 3.12?

3.11 for a few more months. Then 3.12. I was also thinking about adding the badge saying "Spec 0", e.g. you can see in the README in https://github.com/GalacticDynamics/unxt

pyproject.toml

ccdproc/combiner.py

nstarman · 2025-06-26T22:49:40Z

pyproject.toml

@@ -8,17 +8,19 @@ dynamic = ["version"]
 description = "Astropy affiliated package"
 readme = "README.rst"
 license = { text = "BSD-3-Clause" }
-requires-python = ">=3.8"
+requires-python = ">=3.11"


https://scientific-python.org/specs/spec-0000/

ccdproc/core.py

pyproject.toml

These are necessary because upstream dependencies (astropy) explicitly check for numpy arrays rather than using the array API.

ccdproc/tests/pytest_fixtures.py

Co-authored-by: Nathaniel Starkman <nstarman@users.noreply.github.com>

mwcraig · 2025-07-31T14:34:12Z

@nstarman -- thanks again for the thorough review! Can you please give this comment a thumbs-up if you think this is ready for merge?

Using ccd_data as the argument leads to CCDData's __array__ being called, which returns a numpy array. By using ccd_data.data we get the raw data in whatever format the array library keeps it.

Properly set uncertainty unit in gain_correct

new_data never has a .data the way the test is written

mwcraig added this to the 2.5.0 milestone Mar 19, 2025

mwcraig added the enhancement label Mar 19, 2025

mwcraig force-pushed the explore-array-api branch from 85625f0 to 04293b7 Compare March 27, 2025 18:05

mwcraig marked this pull request as ready for review June 24, 2025 14:55

mwcraig changed the title ~~WIP: Adopt the array api~~ Adopt the array api Jun 24, 2025

mwcraig requested a review from Copilot June 24, 2025 16:42

This comment was marked as outdated.

Sign in to view

nstarman reviewed Jun 25, 2025

View reviewed changes

nstarman reviewed Jun 26, 2025

View reviewed changes

ccdproc/core.py Outdated Show resolved Hide resolved

pyproject.toml Show resolved Hide resolved

mwcraig force-pushed the explore-array-api branch 3 times, most recently from 2fa76b7 to 3b2b9fc Compare July 3, 2025 18:28

mwcraig added 15 commits July 5, 2025 10:05

Initial attempt to adopt array API

6dd9166

A couple more changes

d3b6203

Add function for check if something is an array

f0bc6b8

Remove numpy from core

424ebdc

WIP rewrite of a boolean index

7f5050d

WIP testing updates

6a6bfa2

Change more almost_equal to allclose

1d2d171

Use the random number generator from numpy

22bc04d

Remove more in-place array operations

245d641

Set a reasonable tolerance value in float comparisons

55b90be

Continue to use numpy arrays in a few places

23de261

These are necessary because upstream dependencies (astropy) explicitly check for numpy arrays rather than using the array API.

Refactor warning handling for numerical warnings

dffadce

Avoid explicit use of numpy masked arrays

12ff8b0

Rewrite test to not modify array in-place

63f81ba

Use assert_allclose instead of older alternatives

4ae3fc6

mwcraig commented Jul 31, 2025

View reviewed changes

ccdproc/tests/pytest_fixtures.py Outdated Show resolved Hide resolved

mwcraig mentioned this pull request Jul 31, 2025

Consider adding fallback median implementation #906

Open

mwcraig and others added 2 commits July 31, 2025 09:23

Apply suggestions from code review

53dc6bb

Co-authored-by: Nathaniel Starkman <nstarman@users.noreply.github.com>

Do masking operation in-place if possible

12f5ab5

mwcraig added 23 commits August 5, 2025 08:54

Avoid converting data to numpy array

f657dfa

Using ccd_data as the argument leads to CCDData's __array__ being called, which returns a numpy array. By using ccd_data.data we get the raw data in whatever format the array library keeps it.

Change mask setting in tests to avoid a numpy conversion

6d8edb7

Use CCDData compatibility wrapper in trim_image

b17566b

Cast number to array namespace

fdec811

Remove another instance of conversion to a numpy array

7531eee

More fixes for flat_correct

c196b93

One more flat fix

f9ad753

Do not use pytest.approx because it forces a numpy conversion

a0f738a

Add cast to array namespace

65ef951

Add cast to array namespace in gain_correct

52d6d45

Fix handling of gain value/unit

9f59fb5

Properly set uncertainty unit in gain_correct

Avoid mask setter in more places

c3ffbdf

Fix typo

9de6a92

Wrap the CCDData object inside transform_image

558ee79

Wrap before cop because copy sets mask

df71b4d

Fix typo

4845ec2

Ensure uncertainty in test uses array namespace

406ded9

Set mask properly in test

b062491

Make error from data, not ccd object

42bb716

Fix several array API issues in lacosmic

c204f4d

Ensure uncertainty is wrapped even if CCDData is already wrapped

ebf9127

Add numpy conversion for now

aedb3b2

Fix error in test

2a877a9

new_data never has a .data the way the test is written

mwcraig force-pushed the explore-array-api branch from 5c81f63 to 2a877a9 Compare August 18, 2025 14:24

Cast mask to array API object

40ffbae

Uh oh!

Adopt the array api #885

Are you sure you want to change the base?

Adopt the array api #885

Uh oh!

Conversation

mwcraig commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as outdated.

Uh oh!

eteq commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nstarman Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

mwcraig Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

mwcraig Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

nstarman Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

mwcraig Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

nstarman Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nstarman Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwcraig commented Jul 31, 2025

Uh oh!

Uh oh!

mwcraig commented Mar 19, 2025 •

edited

Loading

codecov bot commented Mar 20, 2025 •

edited

Loading