Issues while generating previews/thumbnails for Sheets

[ShrayanshBaunthiyal]

Hi,
We are trying to generate previews/thumbnails dynamically(without saving the PNGs in the local machine) for Sheets in our project. We are using ThumbnailProps and saving it against the viewDefinationId of the Sheet using the saveThumbnail method. We are then trying to get the thumbnails using the getThumbnail method. We are creating an OffScreenViewport and trying to read it as an ImageBuffer using the viewport.readImageBuffer() method. We have tried two approaches for creating thumbnail props:

Approach 1:
in the Thumbnail props we directly pass the imageBuffer.data to the image parameter of ThumbnailProps as argument:

if (iModel && iModel.isBriefcaseConnection()) {
        const currentViewport = IModelApp.viewManager.selectedView as Viewport;
        if (currentViewport) {
          const viewRect = new ViewRect(0, 0, 1280, 720);
          const viewDefinitionId = currentViewport.view.id;

          const vp = OffScreenViewport.create({
            view: await iModel.views.load(currentViewport.view.id),
            viewRect
          });

          vp.renderFrame();

          const imageBuffer = vp.readImageBuffer({
            rect: viewRect
          });

          if (imageBuffer) {
            // Create thumbnail properties
            const thumbnailProps: ThumbnailProps = {
              format: "png",
              image: imageBuffer.data,
              width: imageBuffer.width,
              height: imageBuffer.height
            };

            // Save the thumbnail using the IPC client
            await drawingProductionIpcClient.saveThumbnail(viewDefinitionId, thumbnailProps);

            console.log(`Thumbnail saved for view ${viewDefinitionId}`);
          }
        }
      }

The issue that we face in this approach is while creating the imageElement using the imageElementFromImageSource() method. We have made sure that we are passing the ImageSource object to it. But it neither returns an error or crashes the app. The debugger just goes away after it executes this line. To check if we are able to get an image from the thumbnail props we have added the code to create an image buffer and download the image locally. We are able to download the image although the image seems to overlap with itself in the output, probably because I am not passing in the correct sheet dimensions. But we do get a png file downloaded in my local disk. Its just that the imageElementFromImageSource() method doesn't seem to work.

const getThumbnail = async () => {
  const thumbnails: ThumbnailProps[] = await drawingProductionIpcClient.getThumbnails();

  const imageElements: HTMLImageElement[] = [];
  thumbnails.forEach(async (element) => {
    // Here we create a link to download the PNG to check if the thumbnail is valid
    const link = document.createElement("a");
    link.download = `sheet.png`;
    const buffer = ImageBuffer.create(element.image, ImageBufferFormat.Alpha, element.width);
    link.href = imageBufferToPngDataUrl(buffer)!;
    link.click();

    // This part doesn't seem to work 
    const data = new ImageSource(element.image, ImageSourceFormat.Png);
    const imageElement = await imageElementFromImageSource(data);
    imageElements.push(imageElement);
  });
  return imageElements;
};

The image that we get from the working part of the code(where we are checking if the thumbnail was generated or not)

Approach 2:
Before creating the thumbnail props we first convert the ImageBuffer to BinaryImageSource using the imageSourceFromImageBuffer() method and then pass the imageSource.data to the image parameter of ThumbnailProps as argument:

if (iModel && iModel.isBriefcaseConnection()) {
        const currentViewport = IModelApp.viewManager.selectedView as Viewport;
        if (currentViewport) {
          const viewRect = new ViewRect(0, 0, 1280, 720);
          const viewDefinitionId = currentViewport.view.id;

          const vp = OffScreenViewport.create({
            view: await iModel.views.load(currentViewport.view.id),
            viewRect
          });

          vp.renderFrame();

          const imageBuffer = vp.readImageBuffer({
            rect: viewRect
          });

          if (imageBuffer) {
            // Create thumbnail properties
            // The getImageSource internally calls the imageSourceFromImageBuffer()
            const imageSource = await drawingProductionIpcClient.getImageSource(imageBuffer);
            if (imageSource) {
              // Use the image source
              const thumbnailProps: ThumbnailProps = {
                format: "png",
                image: imageSource.data,
                width: imageBuffer.width,
                height: imageBuffer.height
              };

              // Save the thumbnail using the IPC client
              await drawingProductionIpcClient.saveThumbnail(viewDefinitionId, thumbnailProps);

              console.log(`Thumbnail saved for view ${viewDefinitionId}`);
            }
          }
        }
      }

In this case the imageSourceFromImageBuffer() is throwing this error. I have attached the image of the argument that we are passing to the function.

Arguements that we are passing

Error that we are getting

Error generating SheetTemplateWidget.tsx:156
PNG: Error: Argument 3 must be an integer
    at imageSourceFromImageBuffer (C:\Users\Shrayansh.Baunthiyal\AppData\Local\Programs\Bentley\iTwinStudioforDevelopers\resources\app\backend-bundle\backend.js:92256:51)
    at DPIpcImpl.getImageSource (D:\Dp2\drawing-production\apps\dp-test-app\dist\backend-bundle.js:263168:78)
    at 
    at C:\Users\Shrayansh.Baunthiyal\AppData\Local\Programs\Bentley\iTwinStudioforDevelopers\resources\app\backend-bundle\backend.js:92397:45
    at fn (C:\Users\Shrayansh.Baunthiyal\AppData\Local\Programs\Bentley\iTwinStudioforDevelopers\resources\app\backend-bundle\backend.js:596322:54)
    at MessagePortMain.fn (C:\Users\Shrayansh.Baunthiyal\AppData\Local\Programs\Bentley\iTwinStudioforDevelopers\resources\app\backend-bundle\backend.js:596331:21)
    at MessagePortMain.emit (node:events:530:35)
    at MessagePortMain._internalPort.emit (node:electron/js2c/utility_init:2:2949)
    at Object.callbackTrampoline (node:internal/async_hooks:130:17)

Is there something that I am missing or doing wrong?

[pmconne]

What does "the debugger just goes away" mean? What happens when you're not running inside a debugger?

[pmconne]

It's an await, perhaps the promise never resolves?

Your issue provides a lot of information but I'm trying to parse what the actual complaint is - can you summarize please?

[ShrayanshBaunthiyal]

Okay, in short I want to know if what I am doing in approach 1 is the right way to generate previews(pngs) from thumbnail props(using the imageElementFromImageSource function and passing the thumbnail props after converting to ImageSource object)? If not, what other alternatives do I have to generate previews?

[pmconne]

Ok. That depends on what drawingProductionIpcClient.saveThumbnail does - that doesn't seem to be included in your description. An ImageBuffer is a bitmap, not a png. If you are not converting the bitmap into a png, that's probably your problem.

[ShrayanshBaunthiyal]

drawingProductionIpcClient.saveThumbnail as the name suggests internally calls the saveThumbnail method:

public async saveThumbnail(viewDefinitionId: string, thumbnail: ThumbnailProps): Promise<any> {
    const iModel: IModelDb = tryGetActiveBriefcase();
    if (iModel) {
      await iModel.views.saveThumbnail(viewDefinitionId, thumbnail);
    }
  }

Yep, ye figured that imageBuffer could be creating a problem that's where the 2nd approach comes in where before saving the thumbnail we are converting the ImageBuffer to BinaryImageSource using the imageSourceFromImageBuffer() method and then pass the imageSource.data to the image parameter of ThumbnailProps as argument. But I guess that won't work either. Seems like imageBufferToPngDataUrl or imageBufferToBase64EncodedPng could work according to what you said, right?

[pmconne]

const imageSource = await drawingProductionIpcClient.getImageSource(imageBuffer);

Everything you pass between backend and frontend must consist of primitives (number, boolean, string) and/or POJOs (plain old JavaScript objects). You are passing an instance of ImageBuffer. Its height field is computed by a getter method, therefore it (and all other methods) get stripped away when you send it from frontend to backend. Unfortunately the compiler cannot help you avoid these mistakes.

What can help you avoid them is to prefer limiting your frontend to user interaction logic and keeping your data processing on the backend. In this example, you are creating an ImageBuffer on the frontend, sending it to the backend to create an ImageSource, sending the ImageSource right back to the frontend, and then sending the ImageSource right back to the backend to save it. Just extract the ImageBuffer from the viewport, then package it up in a POJO to send to a single backend method that will convert it to PNG and save it as a thumbnail.