Future of Cameras: Custom Apps

J

jcs

Veteran
After setting up nice lighting and putting a camera and lens together, I shot my friend Jacqui in a new studio set we put together. We shot a few tests and played them back, then she took out her iPhone and said I have this app ( https://apps.apple.com/us/app/b612-beauty-filter-camera/id904209370 ), can we try again? I shot it again and of course it makes her look much better straight out of camera. Back in 2014 I lead a small company creating an iPhone app which did real-time effects, including smoothing skin while preserving edges (bilateral filtering) as well as a host of other effects, including adding a tan as well as audio effects. All of this ran in real-time, live using the GPU and CPU. It was a very small company (Twrrl, previously YooStar) and wasn't advertised or promoted much. After a good run the billionaire owner focused their interests elsewhere and the company went away. Fast forward to 2020, and IG and many other apps are doing even more advanced real-time live filtering/effects, and people are expecting them to 'touch up' their videos, live.

While this hasn't reached the pro market yet, it's clear to see that this is the future. Current phones have insane computing power (Apple moving to a 5nm process!) at low power. Simulated DOF for stills can sometimes fool people into thinking 'DSLR'. Not quite there for video yet, but it will happen. For the mass market / YT / IG crowd, where phones can produce amazing/flattering quality (though not really real!), there's no point buying a dedicated camera as they don't provide any custom apps whatsoever (Sony did a few years ago, but apps were limited (gone now?)).

Semi-pro and pro cameras seriously need to add a GPU and a custom app capability if they're going to compete much longer. I've been saying for a while, an iPhone hooked up to a FF sensor via USB 3.1/Lightning could totally work (sending compressed Bayer over the wire). The quality could easily match the best mirrorless/DSLR however the ace would be custom apps and live GPU filtering/effects (and full high quality DSP audio effects too!). With a market where anyone can create custom apps and effects, such devices will be useful for every kind of filmmaking, with apps tuned for each use case as needed. No more artificially limited / crippled systems: the sky's the limit!

Current phone camera+lens systems aren't up to decent DSLR/mirrorless system yet, however the gap is slowly closing. Computational Imaging/Cameras will be able to recreate any lens system/distortion/look as well as believable shallow DOF at some point in the future, so a FF sensor and fancy lens won't be needed. Want the Cooke look? Select it from the menu. How about anamorphic? Select it from the menu. AI Color (which no one's yet talking about) will match and then exceed what ARRI currently can do in difficult / mixed lighting, basically fixing skin tones no matter what. AI systems are already doing an amazing job colorizing black and white video: so fixing bad lighting/color will be possible soon. Even shots in near darkness can have basically zero visible noise and impossibly realistic looking color.

This got me thinking about creating a high-end camera app for the iPhone, however there's so much competition + spent many years working on those products already, I'd rather focus on the creative side and make content at this point.

In summary, future camera systems will produce amazingly filtered content in-camera, and along with quick editing apps along with easy upload straight from the device, future creative work will have incredibly fast turnaround time. We can work anywhere, at just about any time, and upload the content immediately when we're done editing. Further in the future the effects/filters will be metadata, so everything can be changed in post, including DOF/lens (simulation) choice!
 
So you shot with the Canon (or whatever) first and then re-shot it with her phone?

And for sure it's all coming and much more, and when those days do come, the true magic of filmmaking will be gone forever.

But with that said, I'd really love the Lytro Light Field camera's focus capabilities. Fixing, key-framing focus in post when your state-of-the-art AF system breaks here and there would be nice. Or after using manual lenses.
 
Yeah, handheld Canon EOS R with 35mm F2 IS at F2.8, nice Aputure lights in room with a neat little $40 point light cube on camera, using the Schoeps CMIT5U into a $30 preamp (for phantom power too) into the camera. Normally I'd use the 1DX II on a tripod fed by a Zoom F6 fed by a Schoeps CMC641 on a boom pole, however we needed to shoot this quickly and I didn't have the 1DX II set up whereas the EOS R ready to go.

I can do a lot of what that app does in post, without any plugins (PP CC), however it takes a bit of time, and the client (friend in this case) choses results based on what they see live / during playback right after shooting. With plugins such as https://digitalanarchy.com/beautyVID/mainFCP.html , the final result can look much better since the camera sensor + lens is a lot better than current phones. However that's more time, effort, and cost.

Light field cameras indeed will be very cool once the quality and performance matches then exceeds what current cameras and lenses can do!

One of the reasons (biggest?) that phones took off was the ability to use custom apps developed by anyone. Versus the path camera manufacturers have taken: closed/locked systems with very obvious crippling + even threatening the ML team to effectively stop improvements.

It sure would be cool to have an EOS Rx or Sony A7x style cameras with custom apps for real-time filtering that can be independently developed.
 
Thanks jcs.
Yeah I see once the average consumer can do such things instantly on their little phone them questioning why it costs so much and why we have to spend time shooting with our big cameras and editing.
Seems the processing power of traditional cameras is way way behind.
 
Yeah dedicated cameras are using much older fabs/tech and don't seem to be able to compete with phones due to much lower sales. If the Canon EOS R5 rumors are accurate, then that's a nice step in the right direction. The 1DX III has more compute horsepower and uses less power, so they've moved to a smaller fab process (not clear how far)- that's likely coming to the EOS R5. By allowing anyone to create camera apps, we'd quickly see very easy to use menu systems and novel capabilities just not possible with close systems.
 
As much as I hate to admit it, sometimes it is easier to get the image you want with a phone... : (

Late last year, I shot some player/trophy shots for a client for a video game league final. The look was pretty simple: black background, colored LED tubes to match system colors and the different players walking up to and posing with the trophy in slo-mo(all client gear). The only problem was getting the tubes to expose right/look good on-camera. It was a little struggle and I was never completely happy with the way it looked, but when I pulled out my iPhone 11Pro and shot some slo-mo and stills with it, while they were doing it for the real camera, the iPhone looked exactly like the look we were trying to and couldn’t quite get with the “real” camera.
 
Run&Gun said:
As much as I hate to admit it, sometimes it is easier to get the image you want with a phone... : (

Late last year, I shot some player/trophy shots for a client for a video game league final. The look was pretty simple: black background, colored LED tubes to match system colors and the different players walking up to and posing with the trophy in slo-mo(all client gear). The only problem was getting the tubes to expose right/look good on-camera. It was a little struggle and I was never completely happy with the way it looked, but when I pulled out my iPhone 11Pro and shot some slo-mo and stills with it, while they were doing it for the real camera, the iPhone looked exactly like the look we were trying to and couldn’t quite get with the “real” camera.
Click to expand...

Are iPhones doing live hdr in video mode?
 
You must log in or register to reply here.
Back
Top