Multi-track video/audio + time-based text/image/emoji overlays

[t3chno]

This Flutter video editor is great, thank you to the developer. You clearly thought about real use cases and didn’t force a single way to use it. I can use it with other video players without changing the package code, and I can customize language, icons, and other UI choices. That flexibility is rare and really appreciated.

To make it compete with (and maybe beat) many paid SDKs, it would be awesome to add:

Core requests

• Multiple video and audio tracks (layered timeline)
• Time-based overlays for text, images, and emojis (set in/out and duration)
• Basic trim/split for each clip and overlay

Nice to have next

• Transitions (cross-fade, fade to black/white)
• Keyframes for position, scale, opacity, rotation
• Export options (resolution, bitrate, codec, fps, audio bitrate)

Why it matters

• Enables pro-style editing in Flutter
• Covers most creator/mobile editing needs without leaving the app
• Closes the gap with paid SDKs while staying simple to integrate

Thanks again for the great work. It’s already an excellent package, and with these features it could easily outpace many commercial options.

[hm21]

Thanks for your suggestions!

Multiple video and audio tracks (layered timeline)

That feature is already planned, as you can see in the README here.

The other features would be great, but implementing them would take a lot of time, as it would bring the project close to the functionality of paid solutions like the one from img.ly. So I’m not sure when or if I’ll actually implement them on my own, since it would require a huge time investment.

Still, I’ll keep it in mind in case I find some free time to work on new features.

FYI: I’ll move this "Feature Request" to the Discussions tab, as I think that’s a better place for it in case more people want to join the conversation about new features.

[hm21]

@ekuleshov yeah if you would like to create a PR for that, feel free to do so and i love to review it :)

However as you said, adding ExportTransform to each VideoSegment would make sense, but maybe we should add a new method with a different settings structure. Currently, there is a parameter called imageBytes that overlays the entire video. However, I think that's something we need to control for a specific duration, such as 3-5 seconds. This should be possible for an image and/or video. And we also need the transform configurations that you mentioned, so I guess we need to reconsider how we currently structure them in a way that makes a lot more sense for multi-video editing.

FYI i'm busy with a lot of work but my current work requires a partial Linux implementation, so I'll work on that part first. However, it seems that my current work also requires time-based layer control. Depending on that, I'll start working on that part soon ;)

[ekuleshov]

@hm21 here is WIP PR #89 with Android, iOS and MacOS implementations for the time-based overlays. I don't think I can contribute similar support for Windows and Linux. Have to check if it is possible for the web.

Please take a look if the approach makes sense to you and if it need to be re-aligned.

A simple change in the /example app to use time-based image overlay works on Android for me. But there are build issues for ios and macos (seem to be related to pro_image_editor changes)

[hm21]

@ekuleshov Awesome, thanks for creating it! I will try to find time to review and test it in detail this week, but if you haven't heard from me by Friday, please ping me again. Btw it looks good from what I've seen, but I want to test it in detail just to make sure everything works fine.

FYI the issue with the pro_image_editor should be fixed in version 12.0.10.

[ekuleshov]

@hm21 thank you. Do you think it is possible to implement on the web? I haven't looked at the web side of things.

Might be a good idea to post feature support matrix for platforms.

[hm21]

It is technically possible to render videos in the browser, but I wouldn’t recommend it for more complex use cases.

Some of the main limitations are:

• High memory usage and browser RAM limits
• Slow performance, especially for encoding
• Limited or inconsistent codec support across browsers
• No reliable background processing (tabs can be paused or killed)
• Poor performance and stability on mobile devices

For simple edits it can work, but for anything more advanced I would recommend using a backend service where the video is processed more reliably and efficiently. In that setup, you upload the video along with the editing parameters, process it on the server, and return the final result.

[ekuleshov]

@hm21 so, with time-based image overlays out of the way, what are you thoughts on more advanced video composition from multiple video sources/files? Eg. picture in picture effect, overlay one video on top of the other (adding transparency to the top layer), arranging frames from multiple videos side by side. Those are most common and well supported in ffmpeg.

[ekuleshov]

Let me explain how videoSegments are intended to work. They are purely timeline-based, meaning they define a sequential flow (A → B → C).

The scenarios you’re describing, like picture-in-picture or one video on top of another, are more about spatial composition. In those cases, multiple videos exist at the same time and define the layout together...

If we mix both concepts into videoSegments, it becomes harder to reason about the behavior. A structure that is meant to be sequential would suddenly also control layout and simultaneity.

I'm just having a hard time with how sequential layers and other video overlays will work at the same time. Both from the user point of view, and also from the implementation point of view. One would have to find a "background" video and make sure its duration is long enough and them apply other video overlays. That makes it harder to work with - even for picture in picture scenario when the "picture in picture" part starts earlier than the main frame.

It seems like an overkill and unnecessary complication to have both abstractions segments and layers. That why I suggested to expand the existing video segments to support spatial composition. Basically the videoStartTime will drive where each segment placed in the final video timeline and it would also make it very easy to change each source video position in the final video timeline. That will also allow to make it easy to make overlapping segments where each next video starts a few seconds before the previous ends ...and at the same time allow to specify size and position of each segment in the rendered video frame.

Perhaps if the videoStartTime isn't specified we can keep the current behavior of these segments being sequential.

Also that approach would be very similar to how most video editors work. I.e. each video stream has a timeline and you can move it on the rendered timeline, trim, apply transformations, effects, slow down video, etc...

[hm21]

I see your point, and I agree that your approach can feel more flexible at first glance.

However, I still believe it introduces more complexity for users but let’s take a concrete example:

• 3 sequential video segments (2s, 4s, 6s)
• 1 overlay video visible from second 3–5, top-left and size 100x100

In the approach I’m suggesting, this would look like:

final task = VideoRenderData(
  videoSegments: [
    VideoSegment(video: clip1),
    VideoSegment(video: clip2),
    VideoSegment(video: clip3),
  ],
  videoLayers: [
    VideoLayer(
      video: clip4,
      startTime: Duration(seconds: 3),
      endTime: Duration(seconds: 5),
      offset: Offset.zero,
      size: Size(100, 100),
    ),
  ],
);

Here, the timeline stays simple and predictable, and composition is handled separately.

In the approach you suggested, everything is merged into videoSegments:

final task = VideoRenderData(
  videoSegments: [
    VideoSegment(
      video: clip1,
      startTime: Duration(seconds: 0),
    ),
    VideoSegment(
      video: clip2,
      startTime: Duration(seconds: 2), // First clip 2s
    ),
    VideoSegment(
      video: clip4,
      startTime: Duration(seconds: 3),
      endTime: Duration(seconds: 5),
      offset: Offset.zero,
      size: Size(100,100),
    ),
    VideoSegment(
      video: clip3,
      startTime: Duration(seconds: 6), // First clip 2s + Second clip 4s
    ),
  ],
)

While this is flexible, it also means that videoSegments is now responsible for:

• sequencing (timeline)
• overlap behavior
• spatial layout (position, size)

So instead of one clear responsibility, it mixes multiple concerns into a single structure.

From a design perspective, I’m trying to follow a separation of concerns / single responsibility approach here, where each part of the API has a clear and focused role. Even though both are videos, they play fundamentally different roles, one defines the timeline, the other defines the visual composition.

Another concern is the number of implicit rules this model would introduce. For example:

• What happens if some segments define a startTime and others don’t?
• Where are segments without a startTime placed like in sequence or overlay?
• Would users need to manually calculate durations to position clips correctly?
• What happens if a startTime is slightly off — do we get gaps (black frames) or unintended overlaps?

These cases are all solvable, but they add complexity and make the behavior less predictable.

Regarding the “background video” concern: I agree that we should not require one. If there’s no base video for a given time range, the output should still be well-defined.

An alternative to introducing placeholder segments could be:

• either define the total duration explicitly, or
• infer it automatically from segments and layers (e.g. the maximum end time)

This way, we avoid artificial constructs while keeping the API predictable.

One additional point worth mentioning is that we already have imageLayers in the API for handling overlays and composition. So introducing something like videoLayers wouldn’t be a completely new abstraction, but rather a natural extension of an existing concept. From that perspective, it actually keeps the API more consistent rather than adding complexity.

That’s why I still prefer separating:

• timeline → videoSegments
• composition → videoLayers

It keeps the mental model simpler, avoids mixing two fundamentally different concepts, and makes the API more predictable as it evolves.

---

However if you have a concrete example of a “most common” use case where your approach feels significantly simpler, I’d be really interested to see also an example like above, how that would look in code that we can compare it with both solutions.

[ekuleshov]

@hm21 let me ask you a question. What if one need to change offset and size for clip1, clip2 and clip3? It would be very easy in the second example (without videoLayers), but what would you do when you have both segments and layers?

There is some naming mismatch between imageLayers and videoSegments, but it feels like they could work the same way with a little addition of videoStartTime to control the video timeline.

So, the video segments, image layers and audio tracks all go into the target video timeline, so it would be possible to implement something like this and control each source component time position, size, offset, etc in the rendered video frame(s).

I do agree there will be some complexity for controlling the timeline, but it seem to be a smaller price than dealing with both layers and segments at the same time.

The simplest would be to make the videoStartTime required, but for backward compatibility we could use some implicit rules, like video would start after the end of previous segment if videoStartTime is not specified. The you can just place clip4 with specified videoStartTime at the end of the segment list to keep sequence of clips 1, 2 and 3. Also note that each clip can also have trim (i.e. their start and stop values). And of course applying trim, resizing, color filters or other transformations to individual video segments will be more straightforward if we have one abstraction. I still think it will be easier to use and also implement (on all platforms) if we'd just extend current segments instead of adding layers on top of them.

The layers would basically have to do the same thing. E.g. when their combined timeline extends before or after of segments's timeline. And "before" case isn't seem possible in that api, so user would have to use layers instead of segments anyway if they need that. So, why force them to think what to use segments or layers?

We will also need to introduce something like background color (or even image?) for the rendered video. So, individual videos can be placed safely anywhere in the frame on top of that. And/or add something like "depth" parameter to both image and video layers/segments to control who is above and who is rendered below. Currently image layers imply depth from the order of layers, but that isn't enough across images and videos.

[hm21]

Yeah, that’s a good example and you’re right: if you’re building a timeline-first video editor, your approach would likely be the more straightforward one to implement.

On the other hand, we also need to think about how most users will use this package. My expectation is that the majority of use cases will be things like trimming, sequencing clips, and adding overlays.

One concern I have with extending videoSegments in that direction is discoverability and expectations. For example, if a user adds just one clip and sees that they can change its size, they might expect that to define the output resolution. In reality, it would just scale the video inside the existing frame, potentially resulting in black borders, which could easily lead to confusion and issue reports.

That’s one of the reasons I originally leaned towards separating timeline and composition concerns, so the intent of each property stays clear. However I do understand your point about having a single abstraction and the benefits it brings, especially for more advanced use cases.

I think the best next step here is to explore your approach in practice. If you’d like to move forward with it, feel free to prototype it and share how it looks, I think seeing it in action will make it much easier to evaluate the trade-offs and usability. In many cases, actual implementation also reveals challenges or edge cases that we might not even be aware of yet.

We will also need to introduce something like background color (or even image?)

Yeah, that makes a lot of sense to me. Especially for more advanced scenarios, and i think it would be useful if background properties could also be time-based (start/end).

A zIndex (or similar) also sounds like a good idea, especially to unify how imageLayers and video-related elements interact. That would also allow using an image layer as a background in a consistent way.

[ekuleshov]

Okay, I'll try to proceed with the extension of segments and see where it will go.

Will also try to add zIndex to segments and image layers. That should cover the time-based background needs.

You are right - defining target video dimension need to be more explicit or documented better. The resolution in the VideoQualityConfig and if not declared, I presume it is defaulted somehow from the video segments...