Multiple video sources

[damikdk]

Hi! Thanks for amazing framework.

I have a few videos with alpha channel and I need to blend them into one. In background if possible.

I'm trying to implement GPUImage2 now, but it's too raw. I was able to export 18 seconds long videos in 5 seconds (without presenting it, just processing in background) with GPUImage2, but I got frame drops and pixelbuffer bugs.

Can I achieve success with BBMetalImage?

[damikdk]

Well, I got something after a couple of hours. Here is my changes from your cool demo project (VideoFilterVC):

let url = Bundle.main.url(forResource: "test_video", withExtension: "mov")!

// First VideoSource
videoSource = BBMetalVideoSource(url: url)

// Second VideoSource (the same video file)
videoSource2 = BBMetalVideoSource(url: url)

videoSource.playWithVideoRate = true
videoSource2.playWithVideoRate = true

// RotateFilter will be apply on videoSource2 just for making the difference between videos
filterRotate = BBMetalRotateFilter(angle: 45, fitSize: true)
filterBlend = BBMetalSourceOverBlendFilter()

videoSource
    .add(consumer: filterBlend)
    .add(consumer: metalView)

videoSource2
    .add(consumer: filterRotate)
    .add(consumer: filterBlend)

filterBlend.add(consumer: videoWriter)

It renders almost perfectly, but after 2-4 seconds of recording I got the wall of logs Asset writer or video input is not ready for writing this frame. It's AVAssetWriter with status == failed. I got the same problem with GPUImage2.

BBMetalImage/BBMetalImage/BBMetalImage/BBMetalVideoWriter.swift

Line 275 in 774dd05

print("Asset writer or video input is not ready for writing this frame")

[damikdk]

And I got memory leak with crash when I tried to disable playWithVideoRate

[Silence-GitHub]

Thanks for issuing me.

To blend two videos or more, we should make sure that the videos transmit video frame textures at the same speed (e.g. video A transmits frame 1, video B transmits frame 1, video A transmits frame 2,  video B transmits frame 2...). BBMetalVideoSource dose not guarantee the same speed now. So the blended texture will be wrong.

I will try to add blending videos feature in the new version.

[damikdk]

Thanks for quick response. I suppose you're talking months?

My task is getting harder. Couldn't you give me a tips about it? I can try to implement it by myself.

[Silence-GitHub]

I write a sample demo at branch multiple_video. MultipleVideoBlendVC is the demo view controller, blending two videos. MultipleVideoSource is used to transmit video frame textures at the same speed. Hope it will help you.

[damikdk]

Wow, thanks, dude!

Yesterday I already wrote simple AVFoundation blender for our task, but maybe we will use your awesome lib too. Again, thank you for quick responses.

[fanguohuiruan]

hi, BBMetalBaseFilter or implements ， i call it BBMetalBaseFilter's. i just qustion you that BBMetalBaseFilter's can record a view's animation to a BBMetalVideoWriter. and insert this to video to a limited duration into other much longer video.

[fanguohuiruan]

i want to record the animation of a view's all moves. what can i use with the BBMetalImage. Thank you very much!

[Silence-GitHub]

@fanguohuiruan I open a new issue How to record a view animation

[blurtime]

@damikdk Do you still have the code? MultipleVideoBlendVC works great if the video is already recorded. But I am currently looking into applying a video as a filter (e.g. with an alpha value of 0.5). I would greatly appreciate it if you could share your code.

[damikdk]

@blurtime actually, I have some, but I am not sure about NDA. Our code are blending videos from files (not views) with default AVAssetExportSession. It’s not that hard and you can check similar code in rubygarage/media-watermark. But it works only with images, text and GIFs.

[blurtime]

@damikdk Ok, I was looking for real-time applications. Blending existing files shouldn't be a problem - even as a filter, since @Silence-GitHub has already provided some awesome code for this. But this filter currently still requires an existing file. I suggested in #69 that he could one by one add filters whenever he has time. I already greatly appreciate his code and that he's decided to make BBMetalImage open source.
Don't want to create problems for you. Thank you, though, for the tips!

[juanctecdam]

I'm using this now because I had the same need and I end up forking it, since this branch hasn't been updated(multiple_video) for a while.
Is there any plan to actually release this?

[Silence-GitHub]

@schumy MultipleVideoBlendVC consumes too much memory. I have not found a solution for the problem, so can not release it. Sorry.