Codecov v5

[A5rocks]

Closes #3285
Closes #3161

This is just #3161 with fixes applied on top that should make codecov happy. I also went ahead and did what the comment said to do (enable fail_ci_if_error) even if I don't think we should. If it's flakey we can disable it later.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00000%. Comparing base (921d363) to head (b8f8339).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@               Coverage Diff               @@
##                 main        #3312   +/-   ##
===============================================
  Coverage   100.00000%   100.00000%           
===============================================
  Files             125          125           
  Lines           19253        19253           
  Branches         1304         1304           
===============================================
  Hits            19253        19253           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jakkdl]

The flag errors we don't care about, but the upload error is maybe something to look at?

[A5rocks]

I think the upload error is because the first time (and second time) around the pypy 3.10 on Windows failed.

But not sure!

[jakkdl]

I think the upload error is because the first time (and second time) around the pypy 3.10 on Windows failed.

But not sure!

oh, that sounds very reasonable then. And if there was a more systemic issue we'd probably see more errors than that single run

[jakkdl]

have not reviewed in detail, but if it works it works

[A5rocks]

If there's any issues with this I would be happy to make a followup PR. I'll merge this just so we don't have to think about this anymore...

[webknjaz]

Agreed. It doesn't look as flaky anymore.

[webknjaz]

@A5rocks FYI changing source_pkgs and source like that is probably a bad idea.

[webknjaz]

The flag errors we don't care about, but the upload error is maybe something to look at?

Yes, that's something I'd be worried about. I'm pretty sure this is because of the source/source_pkgs changes in the coveragepy config.
This would effectively result in Codecov mismatching files with what's in the repo. If it's just some of the jobs, it's possible that the uploads aren't taken into account properly, when computing coverage but perhaps other jobs still covered all the lines and so it's not immediately visible.

[webknjaz]

@A5rocks @jakkdl

Looks like Codecov no longer "sees" that these two files exist and so their coverage isn't present:

[webknjaz]

This one is definitely due to excluding it on coveragepy level. source = . was including the entire repo (as a file path) but now filters to just one module (as an importable).

[A5rocks]

@A5rocks FYI changing source_pkgs and source like that is probably a bad idea.

I think that was the main change that fixes using v5, which is annoying. I'll see if we can include the tests directory as well...

[A5rocks]

Hopefully #3314 fixes that, we'll see!