Support for on-disk formats like BPCells

[siddharthab]

I am trying to run SCTransform on a Seurat object backed by a BPCells on-disk matrix, but it seems to be loading the entire matrix in memory first. I am on the latest version of Seurat (5.3.0), sctransform (0.4.2) and BPCells (0.3.1)

Is this expected?

The data I am loading is on [Figshare](https://doi.org/10.25452/figshare.plus.29190726.v1); it's quite large and hosted in Europe so best run on some machine in Europe.

My entire code:

# %%
loadNamespace("BPCells")
loadNamespace("Seurat")
loadNamespace("anndataR")

path <- "/mnt/local/HEK293T_filtered_dual_guide_cells.h5ad"
bp_path <- paste0(gsub(".h5ad", "", path), "_BP")
seurat_path <- paste0(gsub(".h5ad", "", path), "_seurat.rds")

# %%
if (!dir.exists(bp_path)) {
  data <- BPCells::open_matrix_anndata_hdf5(path)
  data <- BPCells::convert_matrix_type(data)
  BPCells::write_matrix_dir(mat = data, dir = bp_path)
}

# %%
if (!file.exists(seurat_path)) {
  mat <- BPCells::open_matrix_dir(dir = bp_path)
  ad <- anndataR::read_h5ad(path, as = "HDF5AnnData")
  seurat_object <- Seurat::CreateSeuratObject(counts = mat, meta.data = ad$obs)
  saveRDS(seurat_object, file = seurat_path)
} else {
  seurat_object <- readRDS(seurat_path)
}

# %%
seurat_transformed <- Seurat::SCTransform(
  seurat_object,
  ncells = 5000,
  vst.flavor = "v2",
  vars.to.regress = "pct_counts_mt",
  conserve.memory = TRUE)