idr-next steps for NGFF upgrade

[will-moore]

Steps needed on idr-next for NGFF upgrade.
NB: current checklist is for actions on idr-testing (newly redeployed on 10th Oct)

Detailed workflow is at IDR/omero-mkngff#2 but this is an outline, also includes study-specific jobs:

• Add bfoptions file to skip_wells for all idr0009 Plates
• idr0070: remove annotations, delete svs images, re-import and re-annotate as described
• Create an OME-NGFF Tag and add the Screens and Projects
• Update ZarrReader if needed to include recent work
• Install mkngff in venv, including recent PR branches (if not yet merged)
• As omero-server, clone https://github.com/IDR/mkngff_upgrade_scripts
• idr0015 - delete one duplicate plate named TARA_HCS1_H5_G100004727_G100004940--2013_12_08_21_26_28_chamber--U00--V01
• Get $SECRET, update all sql commands with it and run them (see https://github.com/IDR/mkngff_upgrade_scripts)
• Run mkngff symlink on all studies, including bfoptions creation
• Update bfoptions files for idr0004 plates P115 and P124
• Regenerate thumbnails for idr0015 plate https://idr.openmicroscopy.org/webclient/?show=plate-4653
• Generate thumbnails for idr0012 Plate 130-16, by going to an Image (e.g. http://localhost:1080/webclient/?show=image-3063425 and "Save To All"
• Bio-Formats cache regeneration
• Validation? E.g. Add check_pixels.py script idr-utils#55 on some data?
• Move idr00XX.csv and sql files from Filesets to swap idr-utils#56 into separate repos, then review/merge the PR.

[will-moore]

Delete on idr-testing...

(venv3) [wmoore@test120-omeroreadwrite ~]$ omero delete Plate:4801 --report > delete_report.log
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public

omero.cmd.Delete2 Plate:4801 ok
Steps: 6
Elapsed time: 129.884 secs.
Flags: []
Deleted objects
  CommentAnnotation:1502861
  FilesetAnnotationLink:20901
  MapAnnotation:8368853
  PlateAnnotationLink:3493
  Channel:4731551-4733530
  Image:1958101-1958496
  LogicalChannel:38501
  OriginalFile:6948775-6988376
  Pixels:1958101-1958496
  ChannelBinding:4269391-4271370
  QuantumDef:1750202-1750597
  RenderingDef:1750202-1750597
  Thumbnail:1738526-1738921,4222134,4222136-4222139,4222144,4222146,4222165-
....
4222496,4222499,4222500,4255156-4255161,4255163-4255199,4255203
  Fileset:21101
  FilesetEntry:6810401-6848218,6848228,6848241
.....
6861554,6861559,6861564,6861569,6861574,6861579,6861584,6861590
  FilesetJobLink:99401-99405
  IndexingJob:106219
  JobOriginalFileLink:33515
  MetadataImportJob:106216
  PixelDataJob:106217
  ThumbnailGenerationJob:106218
  UploadJob:106215
  Plate:4801
  ScreenPlateLink:5001
  Well:1056601-1056996
  WellSample:1762101-1762496
  StatsInfo:3976732-3978711

[will-moore]

sudo -u root -s
source /opt/omero/server/venv3/bin/activate
pip install 'omero-mkngff @ git+https://github.com/IDR/omero-mkngff@main'

[will-moore]

[wmoore@test120-omeroreadwrite ~]$ cd /uod/idr/metadata/idr-utils/
[wmoore@test120-omeroreadwrite idr-utils]$ sudo -Es git fetch will
[wmoore@test120-omeroreadwrite idr-utils]$ sudo -Es git checkout will/filesets_to_swap

[will-moore]

Latest Bio-Formats push: https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-push/643/console

2023-10-10 00:05:33,744 [   scc.merge] INFO  Repository: ome/ZarrReader
2023-10-10 00:05:33,744 [   scc.merge] INFO  Excluded PRs:
2023-10-10 00:05:33,744 [   scc.merge] INFO    - PR 53 dgault 'Reorder group keys' (exclude comment)
2023-10-10 00:05:33,744 [   scc.merge] INFO    - PR 20 dgault 'ZarrReader: Use chunk API' (exclude comment)
2023-10-10 00:05:33,744 [   scc.merge] INFO  Already up-to-date.
2023-10-10 00:05:33,744 [   scc.merge] INFO  
2023-10-10 00:05:33,744 [   scc.merge] INFO  Merged PRs:
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 62 joshmoore 'Update README.md'
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 64 dgault 'ZarrReader performance improvements'
2023-10-10 00:05:33,745 [   scc.merge] INFO    - PR 65 dgault 'Bumping Bio-Formats version to 7.0.0'

Download from https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/565/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

Run on omeroreadwrite, omeroreadonly-1 - 4...

sudo -u omero-server -s
cd
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/565/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader_p643_b565.jar
rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader.jar && rm /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
cp OMEZarrReader_p643_b565.jar /opt/omero/server/OMERO.server/lib/client/ && cp OMEZarrReader_p643_b565.jar /opt/omero/server/OMERO.server/lib/server/
# as wmoore
sudo service omero-server restart

[will-moore]

On omeroreadwrite...

• as omero-server, screen -S mkngff... - get SECRET
• as wmoore...

cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

for dir in $(ls -d */); do
  cd $dir
  for i in $(ls); do sudo sed -i 's/SECRETUUID/1def0359-65xyz-443f-b14a-69f04f569f4d/g' $i; done
  cd ../;
done

• as omero-server, screen -S mkngff...

omero login
cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

export IDRID=idr0004
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr"
done
...
UPDATE 95
BEGIN
 mkngff_fileset 
----------------
        6312382
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-49-38.885_mkngff/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr -> /bia-integrator-data/S-BIAD867/00d88a93-8d21-4a50-b8b5-60f11bcae0d3/00d88a93-8d21-4a50-b8b5-60f11bcae0d3.zarr
UPDATE 98
BEGIN
 mkngff_fileset 
----------------
        6312383
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-34-48.864
...

[will-moore]

Ran for all studies up to idr0016 yesterday...
Today...

• idr0025 9:47
• idr0026 9:48
• idr0033 9:52
• idr0036 10:12
• idr0051 10:23
• idr0054 10:24
• idr0090 10:25
• idr0091 10:29 (done at 10:50)

error from idr0054 because idr0054.csv has whitespace in rows - but shouldn't cause any errors as the correct IDs were still found?
/opt/omero/server/venv3/bin/omero mkngff symlink: error: argument fileset_id: invalid int value: 'idr0054/Tonsil'

[will-moore]

Saved memo times for the first Fileset in each study...

e.g.

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log.4 | grep -A 2 "81ba6c774fdd"
2023-10-11 14:17:21,940 DEBUG [                   loci.formats.Memoizer] (l.Server-3) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-20/2020-10/02/14-41-08.138_mkngff/971f2809-c748-4259-8044-81ba6c774fdd.zarr/..zattrs.bfmemo (91291 bytes)
2023-10-11 14:17:21,940 DEBUG [                   loci.formats.Memoizer] (l.Server-3) start[1697033824100] time[17840] tag[loci.formats.Memoizer.setId]
2023-10-11 14:17:21,941 INFO  [                ome.io.nio.PixelsService] (l.Server-3) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-20/2020-10/02/14-41-08.138_mkngff/971f2809-c748-4259-8044-81ba6c774fdd.zarr/.zattrs Series: 0

• idr0004 139197ms (2.3 mins)
• idr0010 694251ms (11.5 mins)
• idr0011 68187ms (1.1 mins)
• idr0012 1248415ms (20.8 mins)
• idr0013 1125667ms (18.7 mins)
• idr0015 1163426ms (19.3 mins)
• idr0016 3221519ms (53.7 mins)
• idr0025 543319ms (9.1 mins)
• idr0026 22904ms (0.4 mins)
• idr0033 7688852ms (128 mins)
• idr0035 451787ms (7.5 mins)
• idr0036 7228099ms (120 mins)
• idr0051 578885ms (9.6 mins)
• idr0054 2476ms (0.04 mins)
• idr0090 6702958ms (111 mins)
• idr0091 17840ms (0.3 mins)

[will-moore]

As omero-server...
With IDR/omero-mkngff@7dc8222 installed
And the ZarrReader that supports bfoptions updated ome/ZarrReader#64 (comment)

omero login
cd /uod/idr/metadata/idr-utils/scripts/ngff_filesets

export IDRID=idr0004
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff bfoptions /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr"
done
...
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/07-33-48.168_mkngff/f791de8d-cd01-4303-8b08-67cbbbb45b64.zarr.bfoptions
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-10/01/08-02-58.826_mkngff/fcc7c91a-f0e6-43f4-93a6-220e9224eda5.zarr.bfoptions

This now allows viewing of all idr0004.
Repeat for others...

All completed!

[will-moore]

Using IDR/idr-utils#55, let's validate a bunch of Plates etc.. 1st plate from each study...

First plate IDs from idr0004 etc.

omero login
cd /uod/idr/metadata/idr-utils/scripts

for i in 1751 4501 5387 4287 3451 4295 6208 6001 5958 6301 4403 9301; do 
echo Plate:$i
python check_pixels.py Plate:$i /tmp/check_pix_$i.log --max-planes=1; done
...

tail -f /tmp/check_pix_4287.log 
73/672 Check Image:1811320 HT01 [Well M21, Field 1]
74/672 Check Image:1811321 HT01 [Well M21, Field 2]
Error: Mismatch for Image: 1811321 at plane (z, c, t): (0, 0, 0)
75/672 Check Image:1811322 HT01 [Well P5, Field 1]
76/672 Check Image:1811323 HT01 [Well P5, Field 2]
Error: Mismatch for Image: 1811323 at plane (z, c, t): (0, 0, 0)
77/672 Check Image:1811324 HT01 [Well M15, Field 1]
78/672 Check Image:1811325 HT01 [Well M15, Field 2]
Error: Mismatch for Image: 1811325 at plane (z, c, t): (0, 0, 0)
79/672 Check Image:1811326 HT01 [Well E20, Field 1]
80/672 Check Image:1811327 HT01 [Well E20, Field 2]
Error: Mismatch for Image: 1811327 at plane (z, c, t): (0, 0, 0)

[will-moore]

for i in 1751 4501 5387 4287 3451 4295 6208 6001 5958 6301 4403 9301; do 
echo Plate:$i
grep -c Mismatch /tmp/check_pix_$i.log; done

Plate:1751 (idr0004 1-field)
0
Plate:4501 (idr0010 1-field)
0
Plate:5387 (idr0011 1-field)
0
Plate:4287 (idr0012 2-fields)
336
Plate:3451 (idr0013 1-field)
0
Plate:4295 (idr0015 1-field)
0
Plate:6208 (idr0016 6-fields) omero-cli-zarr
0
Plate:6001 (idr0025 4-fields)
288
Plate:5958 (idr0033 9-fields)
3072
Plate:6301 (id0035 4-fields)
180
Plate:4403 (idr0036 9-fields) omero-cli-zarr
0
Plate:9301. (idr0090 32-fields) omero-cli-zarr
0

The Errors come from all Fields except the first field in multi-field Plates. Only with bioformats2raw Plates. No issues with omero-cli-zarr.
All the error counts for the 4 plates above exactly match Wells x (fields - 1).
When viewing images in these plates, all appear fine.

For a single Image - on idr-testing, connect to localhost or idr.openmicroscopy.org.
Compare same images...

http://localhost:1080/webclient/?show=image-1811321
https://idr.openmicroscopy.org/webclient/?show=image-1811321

>>> conn = BlitzGateway("public", "public", host="localhost", port=4064)
>>> client = omero.client("idr.openmicroscopy.org", 4064)
>>> sf = client.createSession("public", "public")
>>> conn2 = BlitzGateway(client_obj=client)
>>> image = conn.getObject('Image', 1811321)
>>> plane = image.getPrimaryPixels().getPlane(0, 0, 0)
>>> image2 = conn2.getObject('Image', 1811321)
>>> plane2 = image2.getPrimaryPixels().getPlane(0, 0, 0)
>>> plane
array([[1596, 1602, 1625, ..., 1738, 1738,  586],
       [1634, 1641, 1614, ..., 1689, 1758,  586],
       [1746, 1717, 1723, ..., 1631, 1654,  586],
       ...,
       [1288, 1351, 1375, ..., 1243, 1275,  602],
       [1276, 1272, 1290, ..., 1225, 1250,  600],
       [ 618,  620,  618, ...,  594,  596,  600]], dtype=uint16)
>>> plane2
array([[1260, 1416, 1529, ...,  666,  671,  475],
       [ 993, 1205, 1362, ...,  647,  648,  475],
       [ 944,  955, 1048, ...,  644,  673,  475],
       ...,
       [1969, 2061, 2049, ..., 1375, 1409,  486],
       [1928, 1964, 2039, ..., 1468, 1455,  485],
       [ 498,  499,  498, ...,  481,  482,  485]], dtype=uint16)

Pixel values are different!

With the same rendering settings, these images do appear a little different:

Need to check the raw NGFF data...

[will-moore]

>>> url = "https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD845/409227a9-2f3f-421c-8086-b14253283e42/409227a9-2f3f-421c-8086-b14253283e42.zarr/M/21/1/"
>>> from ome_zarr.io import parse_url
>>> from ome_zarr.reader import Reader
>>> store = parse_url(url, mode="r").store
>>> reader = Reader(parse_url(url))
>>> nodes = list(reader())
>>> len(nodes)
1
>>> image_node = nodes[0]
>>> dask_data = image_node.data
>>> data = dask_data[0].compute()
>>> data.shape
(1, 3, 1, 1024, 1344)
>>> data[0,0,0]
array([[1260, 1416, 1529, ...,  666,  671,  475],
       [ 993, 1205, 1362, ...,  647,  648,  475],
       [ 944,  955, 1048, ...,  644,  673,  475],
       ...,
       [1969, 2061, 2049, ..., 1375, 1409,  486],
       [1928, 1964, 2039, ..., 1468, 1455,  485],
       [ 498,  499,  498, ...,  481,  482,  485]], dtype='>u2')

These pixel values appear identical to those loaded from idr.openmicroscopy.org, so it looks like the NGFF data is correct and the differences we're seeing on idr-testing are likely due to the ZarrReader.