Extremely Long Upscaling

[TeamEW01]

video2x upscaling takes so long, I've got NVIDIA GeForce RTX 4070 SUPER Ti and 32 minutes long real life video takes two days to upscale from 1080p to 4K with the Real-ESRGAN

[niveK77pur]

I am experiencing the same problem with Real-CUGAN. Upscaling a ~24min anime episode from 1080p to 2k takes ~2h30, whereas upscaling to 4k takes ~6h30 (which is over double, though unclear to me if the model just actually takes over twice as long to upscale 4k vs 2k).

To be verified and tested again, but at some point I tried upscaling a short 30s anime clip which took around 3-4 minutes to upscale 1080p to 4k. With that logic, that should be ~7 minutes per minute of video. That should intuitively make a 24min video take ~2h30 hours instead of the ~6h30 it actually takes (or at least according to the given time estimation). Somehow the numbers don't seem to add up very nicely as the video length increases. (God forbid upscaling a full movie; an attempt with a ~1h30 movie using Real-ESRGAN would have taken ~13h for 2k, and over 24h for 4k).

Maybe worth looking into chunking a long video into smaller portions and then re-assembling it? Furthermore, this could be leveraged to introduce a sort of "resume" feature if you terminate the upscaling mid-way (or maybe it gets aborted for another reason). Currently, it seems that if I abort a process which is 90% complete, it will start all over again from 0%.

Using an Radeon 9070 XT for reference.

[niveK77pur]

Just coming in here with a fish script wrapper (at the bottom) I wrote to chunk the video for upscaling. It is indeed upscaling the video faster now, but not by a huge margin (I might get better results by tweaking the segment size).

Here some numbers using -p realcugan:

Video duration	Upscale duration (-s 2)	Upscale duration (-s 2) [no segmentation]	Upscale duration (-s 4)	Upscale duration (-s 4) [no segmentation ]
2'00"	4'33"	5'02"	7'42"	10'09"
4'00"	-	-	~13'	~20-30'

Note

Mind you that for the 4 min upscaling, I did not really track the exact time unfortunately. But it was around 13'00" to 13'30". And for the duration without my segmentation, I also aborted the upscaling 1/3 of the way, but 20-30 minutes seems like where it was heading towards (closer on the end of 26 min).

Note

Between the 2min and 4min long videos when upscaling using -s 4, it is not exactly double. That is, because on the 2min version, it spent the last minutes upscaling only a single segment (the last remaining one). For the most part, it was always running 3 video2x instances in parallel.

Wrapper fish script around video2x

Things to take note of:

• We may be able to improve the performance by playing around with SEGMENT_SIZE and MAX_PROCS values (though the MAX_PROCS seems to make the script run into bottlenecks much faster due to system resources, irrespective of the SEGMENT_SIZE). Providing a long video to video2x does not seem to increase the upscaling duration linearly. Using this segmentation, the duration seems to increase more linearly (in my very preliminary usage of this script at least).

• As of now, video2x seems to fill up the RAM much more than the VRAM. So careful with increasing the MAX_PROCS value in particular, as it may completely fill up your RAM long before your VRAM does. On my system with 32GB RAM an 16GB VRAM, MAX_PROCS=5 froze my machine due to flooding the RAM. With MAX_PROCS=3 I sit at a comfortable ~22GB RAM and ~6GB VRAM (I might dare to up it to 4, but I think my GPU is the bottleneck for handling the concurrent processes). I was using btop to closely monitor my system resources.

• Stores intermediate upscaled segments into a directory alongside the video file (directory name simply has the file extension removed). These chunks effectively allow us to also resume an aborded upscaling (it would be more than unfortunate to start such a long process from the start)

• All segments are stitched back together at the end to form the full upscaled video. With the one video I used for testing, I noticed absolutely no artifact where the segments are coming together; so this segmentation approach at the very least introduces no visual downgrades.

#! /usr/bin/env nix-shell
#! nix-shell -i fish
#! nix-shell -p ffmpeg
#! nix-shell -p video2x

trap 'rm -f /tmp/video2x_segment.fish' EXIT

set -x FILE $argv[1]
set -x V2X_ARGS $argv[2..]

set -x SEGMENT_SIZE 30 # size in seconds
set MAX_PROCS 3 # WARNING: More processes will fill up RAM more than VRAM
set VIDEO_LENGTH (ffprobe -i $FILE -show_entries format=duration -v quiet -of csv="p=0")

# We can exit early if we see the final output file is already present
set OUTFILE "$(path dirname $FILE)/upscaled_$(path basename $FILE)"
test -f $OUTFILE && begin
    echo `$FILE` has already been upscaled
    echo Found in `$OUTFILE`
    exit
end

set -x TEMP_OUTDIR (path change-extension '' $FILE)
mkdir -p $TEMP_OUTDIR

function video2x_segment -a start
    # For some reason, $V2X_ARGS becomes a string here instead of a list
    set -f v2x_args (string split ' ' -- "$V2X_ARGS")
    set -f out $TEMP_OUTDIR/out{$start}.mp4
    # We can exit early if the chunk is already processed. If a chunk hasn't
    # finished processing, then we obtain a broken video file; we should
    # upscale it again from the start.
    ffmpeg -v error -i $out -f null - && return
    video2x \
        $v2x_args \
        -i (ffmpeg -ss $start -i $FILE -t $SEGMENT_SIZE -f matroska - | psub) \
        -o $out
end
funcsave -d /tmp video2x_segment

# Upscale segments in parallel
parallel \
    --max-procs $MAX_PROCS \
    --line-buffer \
    --color \
    'source /tmp/video2x_segment.fish && video2x_segment {}' \
    ::: (seq 0 $SEGMENT_SIZE $VIDEO_LENGTH)

# Verify integrity of all chunks
parallel ffmpeg -v error -i {} -f null - ::: $TEMP_OUTDIR/*.mp4 || exit

# Check if there is a gap
for start in (seq 0 $SEGMENT_SIZE $VIDEO_LENGTH)
    test -f $TTEMP_OUTDIR/out{$start}.mp4 || begin
        echo Intermediate file for segment $start is missing
        exit 1
    end
end

# Concatenate upscaled segments into final video
ffmpeg \
    -y \
    -f concat \
    -safe 0 \
    -i (printf "file '%s'\n" (realpath $TEMP_OUTDIR/*.mp4) | psub) \
    -c copy \
    $OUTFILE

# We no longer need the temporary files when the video is finalized
test -f $OUTFILE && rm -vr $TEMP_OUTDIR