AI Upscaling - Best Demo So Far

J

jcs

Veteran
https://www.cinema5d.com/topaz-gigapixel-ai-video-resolution-upscale/


Good progress- it will get better and faster (there's a bunch of companies competing in this space).

At some point I'd expect Netflix to update its camera restrictions for 'Netflix Approved' (Netflix funded): older cameras like the F35 and used Alexa's upscaled to 4K/8K will look better than 4K/8K cameras due to better color science, DR etc.

Samsung claims to do this in real-time for its 8K TVs:

However the only 8K TV at Best Buy / Magnolia in Hollywood didn't show anything more than a CGI rolling demo. I ask the sales folks to put on some 1080p or 4K footage and they said that was the only demo available.
 
Netflix will not approve the "classic" 2.8K/3.2K Alexa (which do own upres) and ARRI wouldn't want them to do so anyway ... but, as I linked on the "listening" thread, getting 4K from pretty much every post-WWII film ever made shouldn't be a problem ... and the same goes for the TV shows 1970's and on ...
 
Did you watch the demos? AI Upscaling is much higher quality than what ARRI does in camera. Real detail is synthesized vs interpolation + sharpening. At which point an old Alexa can be AI upscaled to match e.g. the 65, I’d expect a change in policy / criteria.
 
Yes, I watched the videos. This is a "consumer" version of what the pros have been doing for a long time. Or you'd never see "I Love Lucy" in color and in HD.
 
I believe Teranex upscaler's were the standard back in the day. I know one of the early HD shows I worked on used them for historic/file footage, but I don't think there was any "AI" going on back then.
 
Right- "AI" requires quite a bit of computational power to be worth doing- recent GPU level power (thousands of cores). Here's an in-depth discussion on the technology, implementation, and many examples: https://towardsdatascience.com/deep-learning-based-super-resolution-without-using-a-gan-11c9bb5b6cd5

This is all relatively new- it will get even better. Ultimately what we'll have is an AI deconstruction system which can then render to any desired resolution, much like how computer graphics/games are rendered with 3D models and texture maps (+ all the other maps).

This will also lead to novel forms of video compression (actually not compression/decompression: instead deconstruction (building a model) / reconstruction (rendering the model at any desired resolution)).

EDIT: authors claim real-time performance possible on CPU: https://arxiv.org/abs/1608.00367
 
Last edited:
Upscaling/upres has been taking place since the mid-80's. When I got into the higher end of the consumer video around 1989 - which would make it 30 years ago ... oy - higher end consumer TV's upscaled 480i into 560i, a minor improvement. In 1989, Mitsubishi came out with a 760i for their rear projectors and we couldn't keep them in stock. The high "price is no object" end had generally two piece projectors (Barco) with a Faroudja chip that made 480P. The same year Sony came out with a 480P 27" "improved definition" tube TV. The place where I worked had the only available unit in LA. It was $5,000. the only reason we got it was because Glen Larson wanted to take a look at it. He did take a look at it and hated it. We priced the floor model at $3,000 because it didn't move at a $5,000 MSRP. The image was good enough in slow moving shots but had a lot of blur for motion. I recall watching an MNF game on it once and came to the same conclusion as Glen Larson.

A year later, Sony came out with the "improved definition" 480P 43" tube set. It was gorgeous, no blur. It was also priced at about $40,000. There was gossip that Jack Nicholson came to take a look at a demo that was in a store near Beverly Hills. Different chain, different location, can't confirm or deny.

Fifteen years later, I ended up purchasing a 27" Advent "improved definition" (480i into 480p upscale) tube TV for $270 at Brandsmart. I wanted a 4:3 TV since a bunch of programs then were still in 4:3, my analog cable was in 4:3 and we already had a 16:9 HD rear projector with an OTA tuner. It lasted over a decade and was a great buy. The idea was the same as in 1989 but the processor was far superior.

The external upscalers - once again, many had a Faroudja chip that was constantly upgraded as well - were a range in the high end video world during the transitions from SD to HD in the early part of the 00's. They ran anywhere from a grand to slightly over two and were quite useful for those watching old VHS tapes, laser discs, DVD's on their new large HD sets. You just plugged the source into the back of the processor and the processor into the HDMI or even the RCA port on the TV/projector. As the source quality improved to the HD standard and the HDTV's had gotten better processors themselves, the need for the external upscalers for the consumer market disappeared.

PS. Any upscaling/upres technologies benefit those with a library of the lower-res titles, i.e., not Netflix, whose library is brand new, but Disney, Peacock, Sony Crackle, etc. that sit on a zillion hours of classic films and TV shows.

 
"This is all relatively new" is referring to AI upscaling specifically, not upscaling in general. All prior algorithms work only with the source image to create an upscaled image. The novel thing with AI upscaling, is the algorithm "recognizes" image features (using the network's 'database/trained-memory'), then renders a new version at arbitrary resolution. This means edges will be smooth, alias free, and new texture detail will be added (synthesized). Another aspect is noise/compression artifacts in the source material can be removed as well. The trained neural network is significantly more data/complexity vs. all the prior algorithms (and thus the need for GPUs/fast-CPUs) and the results are dramatically better.
 
Thanks for sharing this. I always wondered if we'd ever get a camera that captured vector-based images, giving us the ability to export any resolution we wanted. Seems like this technology has a big head start on that approach, if it is being developed at all.

I ran some low-res footage through the software -- actually, it's an online platform with a base rate of $2 plus a flat rate per million pixels. You upload your clip and then they send you an email with a download link. But there's also a free drag-and-drop demo feature, which exports a 60sec side by side comparison of any footage you put into it: https://videoai.topazlabs.com/

I was impressed by the results. It made the criminally soft 1080P (which looks more like 720P) from my old FZ47 look like real 1080P (unfortunately, it also brought out the aliasing that was hiding in the original compression). Also did a good job with noise reduction too. I'll have to see what this can do with some of my C100 footage. Might be worth going to 8K and then downscaling to 4K.

I don't know if this kind of tech will be accepted by Netflix for their programming -- maybe it could be. But in any case, I do see a lot of practical applications for video editors.
 
AI upscaling is just getting started- it will get much better.

https://www.google.com/search?q=image+upscaling - note how many companies are working on AI upscaling. There's quite a variety of AI methods- each is different and the networks can be further tuned and trained based on subject matter. The better methods will use temporal training so frame/frame variance won't be an issue.

Training is super important- I think they need to work on edges too- most examples are currently too sharp and can look unnatural.

At some point the AI deconstruction will understand for example what a blurry stop sign is, where STOP is too few pixels to blow up / re-render directly, and because it knows it's a stop sign, still render it perfectly at any resolution. The running horse example is pretty rough- same thing there, the AI will know they are horses and fill in the blanks. This will lead to novel animation and art tools.

The next level will be to create full 3D models (for the entire scene) from video since the AI will know what all the objects are, and from the motion can figure out the positions- kind of a super-photogrammetry from a single 2D camera view (multiple camera views will also work). There's already simple AI products for single 2D photos to 3D models.
 
Last edited:
Whoa! is this coincidence or fate? I just found this same company earlier today then came on here and found this thread.

I decided that as a new years resolution I would recut and remaster my feature film from 2005. I had intended to do it for the ten year anniversary back in 2015, loaded up all the old DV tapes onto a hard drive and even recut a couple scenes and experimented with upscaling to 1080p. Unfortunately things got too busy for me back then and I put it on the side. Figured I would look into upscaling software last night and found this.

I ran a short 30 second clip through their system. A pretty bad looking clip I don't intend to use. It's a blown out straight 60i AVI file just to see what I would get. See below.
It did improve it in my opinion. You can see in the eyes and certain details like the pot of the plant, but it definitely over exaggerated the interlacing. I'll need to deinterlace first. I figured maybe their software would do that step as well but I guess not.


Is it better to upscale first and then edit or upscale after the edit? When I recut a few scenes back in 2014 I was doing the upscaling in Premiere Pro while I was editing. Nothing fancy, literally just dropping the footage on a 1920x800 timeline and scaling up. I wasn't that disappointed with the results I got either. The footage is soft, but so is some older smaller gauge film stock. I looked into using something like Red Giant Instant 4K this time and it seems to work ok, but I'm not yet convinced it's better than just scaling up in Premiere natively and adding a tiny bit of sharpening. Also not convinced I want to use the Gigapixel AI service. I'd have to do more tests with it but really don't want to run through paying every time I want to upload a short clip to test the best settings now that my free proof is used up. Unfortunately i'm cutting on a mac so their trialware version is windows only. Sounds like you need a pretty powerful machine at that. So I'm not sure what route to take here. This is really just a recut/remaster for me and a handful of people that made the film so I'd rather not spend any money, just my time. It wasn't a very good movie lol. It was fun to make and great learning experience but we shot it in 2003 and released in 2005 and I've learned a lot in the last 15 years so wanted to pull a George Lucas and remake it in post and see what I would get.
 
The way these AI tools work is through training: unless the network was trained to deinterlace, I wouldn't expect great results. Also, it looks like most (all?) of these offerings are trained for specific scale factors, 2x, 4x, etc. So they're not at the 'render at any scale' level yet.

Overall good progress- if not good enough for your needs, it may be in the near future as these companies rush to compete with better results.
 
As the processing power of the CPU/GPU grows every year, the algorithms also become more complicated and higher performing. Most (well, I believe it's most) mobile phones have a "digital zoom", which is the in-phone upscaling in real time. But what we're seeing here is the "post". In post, these various brands can and will serve different purposes - some will work better for stills, some for video. I see the biggest market to be in upscaling the relatively low resolution of the secondary mobile phone cameras. With the main modules now moving toward the 48-64-108 MPX res, there's still a need for the low light/low MPX and ultra wide/selfie alternatives and those should be the prime candidates for these upscaling services. There are about six billion mobile phones out in the wild and each takes thousands of photos annually. That's where the money ought to be.

PS. I suppose one could also upscale Alexa LF footage to 6K/8K but that might be required only when the 6K/8K streaming takes off and that won't be for quite some time.
 
I noticed a lot of these demos use footage that started higher quality to begin with, either film or higher resolution video that looks like it was first downscaled and then run through their process. I want to see terrible VHS camcorder video from the 80s run through these.
 
Kudos- looking better and better. The low res to HD and 4K is really impressive. HD to 4K needs a bit more work on fine detail aliasing, then no more visible artifacts = professional grade.

Online service only: https://videoai.topazlabs.com/faq

Desktop versions will be cool for those with decent GPUs (they're likely doing online only as a business model as long as they can (there will be desktop versions soon from other vendors). There's also source code / tools for developers- results look decent (someone could use this with e.g. ffmpeg libs to upscale video): https://github.com/alexjc/neural-enhance

StreetView_example.gif



https://github.com/idealo/image-super-resolution

temple_comparison.png
 
Last edited:
You must log in or register to reply here.
Back
Top