Media stacking for many variants of the same media

[viktorstrate]

HEIC support has been added for images (see #147), but a .mov and a .heic will be shown as individual media. One solution might be to add a duplication detection for videos and their thumbnails (like photo duplication detection, see #8)

---

It would be incredible if you could keep iOS's "Live Photos" intact in Photoview.

I'm talking about combining the JPEG/HEIC and H.264/MOV as one image from the viewing browser. I'll use PhotoSync for iOS to routinely pull new photos from my phone to my NAS, but viewing platforms like Piwigo, PhotoPrism, etc will view every entry as 2 items (the jpg/heic and the mov).

Would you mind giving it a crack in Photoview? I've attached a zip with a sample image. By default on iOS photos are exported as HEIC and MOV. You can also set the format to "most compatible" (camera -> formats) and the setting will be JPEG and MOV (similar to what PhotoSync does when transferring to a NAS)
TEST.zip

Originally posted by @McDowell-CSSD in #147 (comment)

[McDowell-CSSD]

Found a post that discusses Live Photo composition linking the jpeg/heic with the mov file using Swift.

http://www.limit-point.com/blog/2018/live-photos/

[AngeloFrangione]

Hi, I'm thinking of implementing this within the scanner, but this will need to add a new entry in the database, for example the column "Live Photo" that contains the id of the video linked to the photo. What do you think of my way of implementing it ? @viktorstrate

[viktorstrate]

Hi @AngeloFrangione,

I have earlier written some thoughts on this on a similar thread to this. I've quoted it below.

I think the best approach would be to run some deduplication function that detects if an image is a part of a live photo video. If it is, only save one media entry in the database and add two media_urls, one of type video for the video file, and instead of generating a video-thumbnail from the video, directly use the photo as the video-thumbnail.

But as suggested in the third option below, we could add a live_photo flag to the media table, to make it different from a regular video, such that the UI can display it in a different way.

UI/UX discussion

I am not sure what the optimal user experience for these kinds of media would be on desktop though. On mobiles you long > press and it plays the video until you release your finger, how would interaction with live photos work on desktop?

Technical discussion (Graphql API)

Implementing this would probably require the introduction of a new media type besides photo and video, such that the UI can present "live/motion photos" in a unique way.

Alternativeley live photos should be represented as a regular video with the poster and thumbnail being the photo part, but then a huge play button would be shown on top of all live photos, and also a video player would always be shown in the sidebar or in fullscreen.

The third option is a hybrid between these, like the media would be represented as a video but with an extra flag to specify that it should not be considered an ordinary video. This would be nice for other frondends like the iOS app such that they wouldn't have to implement custom UI to support it as it would default to be shown as a video.

Originally posted by @viktorstrate in #611 (comment)

[vgt]

Since this issue is important for me, I obviously have a opinion on this and will not hesitate to annoy you with it. ;)
I think a live photo should remain primarily a photo, as this is the primary and most valuable usage. Looking at how Apple and Google use live photos in their respective apps, they merely use the video to "enhance the photo with some nice effect".
So with respect to the variants you mentioned, my personal opinion is that the second and third alternative are not advisable.

It seems that for the first variant everything is in place from a database perspective, even not adding any additional functionality to the UI would initially work).

thumbnail: MediaURL "URL to display the photo in full resolution, will be null for videos"
highRes: MediaURL "URL to get the video in a web format that can be played in the browser, will be null for photos"

Adding a MediaType called livephoto could simply mean that for those entries, both MediaURLs will have content.

[GlassedSilver]

I agree that a live photo is an enhanced photo, not a video.

On iPhone and Android I shoot almost all of my photos with their respective motion feature enabled. Apple calls it Live Photo, Samsung calls it Motion Photo and other Android OEMs use yet another term each, because obviously everyone feels the strong desire to make it different and hence call it something different........ I digress. :D

So if a photo gallery counted my items as videos primarily I would have a lot of "videos" and few photos to speak of, at least counting my mobile photography.

The presentation has to focus on displaying the photos and on hovering the mouse or holding the photo the UI should load the video to play. Apple has sound, Samsung doesn't, Huawei I think has sound as well? Not sure, I hardly take photos on my Huawei tablet.

[viktorstrate]

Thank you for the input @GlassedSilver. It would be nice to have this feature sorted out properly and maybe built it in a way that makes it easy to add support for multiple "live-photo" formats like Motion Photos.

---

To avoid switching between this issue and #611, I think we should use this thread for the general implementation and UI, and #611 for the Android Motion Photos format specifically.

In reply to #611 (comment)

Just copy what Apple do in the Photos Mac application.

iirc (not a Mac user anymore and when I was I kept myself to holding it out with Aperture for as long as I could....) you hover to play and there's also a little symbol signifying that the image has motion. iirc you can also press that. So when viewing the image in view mode rather than the overview grid you'd click the icon. Think it's positioned in the top right corner. But again, been a while since I last checked myself. Google Image search the respective apps for some inspiration. I may have a grudge on Apple for a lot of decisions, but typically they do a good job of implementing features visually in the UX when they bother to offer a feature. :P

Thank you for sharing, that sounds like a nice approach. A small icon could be added to live photo both for the thumbnails and also in fullscreen. Then when the mouse hovers over it, the video sequence will play.

[GlassedSilver]

Thank you for the input @GlassedSilver. It would be nice to have this feature sorted out properly and maybe built it in a way that makes it easy to add support for multiple "live-photo" formats like Motion Photos.

To avoid switching between this issue and #611, I think we should use this thread for the general implementation and UI, and #611 for the Android Motion Photos format specifically.

In reply to #611 (comment)

Just copy what Apple do in the Photos Mac application.
iirc (not a Mac user anymore and when I was I kept myself to holding it out with Aperture for as long as I could....) you hover to play and there's also a little symbol signifying that the image has motion. iirc you can also press that. So when viewing the image in view mode rather than the overview grid you'd click the icon. Think it's positioned in the top right corner. But again, been a while since I last checked myself. Google Image search the respective apps for some inspiration. I may have a grudge on Apple for a lot of decisions, but typically they do a good job of implementing features visually in the UX when they bother to offer a feature. :P

Thank you for sharing, that sounds like a nice approach. A small icon could be added to live photo both for the thumbnails and also in fullscreen. Then when the mouse hovers over it, the video sequence will play.

Hmm, actually hovering over the thumbnail should always play, in full-screen/image view mode it should be possible to click the image itself to play or the icon specifically and on touch-input devices (or rather: devices without a pointer device present) press-and-hold on the grid should play (maybe with the image "popping out a bit"? as in not loading a full on viewer, but a slightly enlarged view that overlays. Think web link previews from the old 3D touch days on iPhone.

As an additional hint that a photo is a live photo when forwarding or going back in viewer mode between pictures Apple's Photos app actually loads a few frames of the associated movie file ass the transition animation is active, coming to a halt at the still photo as long as you don't hold to view the full animation.

This really makes your gallery come to life and ... for a lack of better words... is a magical sort of experience. A small detail that can make reliving your memories all that more vivid and entertaining and you can actually better grasp which photos are really worthy of being viewed in motion, because let's face it. I may take almost all of my mobile photos with motion on, but with about a third of them or so it's actually worthwhile to view the animation. :P

[vgt]

In my humble opinion, the data structure must be prepared first. Right now, the gallery is cluttered with a lot of videos (live photos), which is not really good. Once you have the live photos detected properly and stored, you can start working on the most fancy way to display the video or merge it into the gallery view. I would love to contribute, but feat that I lack the necessary go knowledge.

[GlassedSilver]

No preparation needed. Apple stores Live Photos with the following logic:

DATE TIME SEQUENTIAL NUMBER .jpg or .heif respectively
Same for the movie counterpart, just another extension.

This is all that Apple's own Photos app uses. No need to complicate ingression when files are already generated in a plausible manner. That sequential number stays the same as well btw. It's literally obvious what belongs to each other unless you go out of your way to mess up the names. :)

Sample:

[GlassedSilver]

Samsung embeds the movie within the image file.That needs to be read differently, but that's an even easier link.

[joshuaseltzer]

Has there been any progress made on this? Even if there was temporarily a way to disable PhotoView from displaying the duplicates (since there's effectively 2 entries per Live Photo) that would be a great stopgap solution until it's decided how Live Photos are handled.

[GlassedSilver]

Check out how Photoprism handles this. There's also a recent change in how Apple handled these pairs... (something something an appended _3 in the movie name I think?)

Once you have the logic to pair these files you're probably best off presenting everything basically how Apple does it, because their UI for it is refined from day one. (I say that as someone who fled from Apple's universe as much as I was able to so far. :D)

[r33ce15]

Is there any progress on Live Photo support?