Researchers in the U.K. have confirmed what many in the broadcast industry have known for years: most viewers have trouble detecting 8K—or even 4K—video resolution.

According to the researchers at the University of Cambridge and Meta Reality Labs, the human eye has a resolution limit: in other words, there are only so many pixels the eye can see. Above this limit, a screen is giving our eyes more information than they can detect.

To calculate the resolution limit, the researchers conducted a study that measured participants’ ability to detect specific features in color and greyscale images on a screen, whether looking at the images straight on or through their peripheral vision, and when the screen was close to them or further away.

The precise resolution limit depends on a number of variables, including the size of the screen, the darkness of the room, and the distance between the viewer and the screen. However, for an average-size U.K. living room, with 2.5 meters (~8 feet) between the TV and the sofa, a 44-inch 4K or 8K TV would not provide any additional benefit over a lower resolution Quad HD (QHD) TV of the same size.

The researchers have also developed a free online calculator where users can enter the size of their room and the dimensions and resolution of their TV to determine the most suitable screen for their home. Their results are reported in the journal Nature Communications.

Display resolution is considered equally important for the many other screens consumers use, on phones or computers, whether they’re used to take pictures, watch films or play video games, including games in virtual or augmented reality. Even car manufacturers are offering higher and higher resolutions for in-car information displays and satnav screens.

“As large engineering efforts go towards improving the resolution of mobile, AR and VR displays, it’s important to know the maximum resolution at which further improvements bring no noticeable benefit,” said first author Dr Maliha Ashraf from Cambridge’s Department of Computer Science and Technology. “But there have been no studies that actually measure what it is that the human eye can see, and what the limitations of its perception are.”

“If you have more pixels in your display, it's less efficient, it costs more and it requires more processing power to drive it,” said co-author Professor Rafał Mantiuk, also from Cambridge’s Department of Computer Science and Technology. “So we wanted to know the point at which it makes no sense to further improve the resolution of the display.”

The researchers created an experimental set-up with a sliding display that allowed them to measure exactly what the human eye can see when looking at patterns on a screen. Instead of measuring the specifications of a particular screen, they measured pixels per degree (PPD): a measurement of how many individual pixels can fit into a one-degree slice of your field of vision. Measuring PPD helps answer a more useful question than ‘how high is the resolution of this screen?’ Instead, it answers the question ‘how does this screen look from where I’m sitting?’

The widely accepted 20/20 vision standard, based on the Snellen chart that will be familiar to anyone who has ever had their vision checked, suggests that the human eye can resolve detail at 60 pixels per degree.

“This measurement has been widely accepted, but no one had actually sat down and measured it for modern displays, rather than a wall chart of letters that was first developed in the 19th century,” said Ashraf.

Participants in the study looked at patterns with very fine gradations, in shades of grey and in colour, and were asked whether they were able to see the lines in the image. The screen was moved towards and away from the viewer to measure PPD at different distances. PPD was also measured for central and peripheral vision.

The researchers discovered that the eye’s resolution limit is higher than previously believed, but that there are important differences in resolution limits between color and black-and-white. For greyscale images viewed straight on, the average was 94 PPD. For red and green patterns, the number was 89 PPD, and for yellow and violet, it was 53 PPD.

“Our brain doesn’t actually have the capacity to sense details in color very well, which is why we saw a big drop-off for color images, especially when viewed in peripheral vision,” said Mantiuk. “Our eyes are essentially sensors that aren’t all that great, but our brain processes that data into what it thinks we should be seeing.”

The researchers modelled their results to calculate how the resolution limit varies across the population, which will help manufacturers make decisions that are relevant for the majority of the population: for example, designing a display which has retinal resolution for 95% of people rather than an average observer.

Based on this modelling, the researchers developed their online calculator, which enables people to test their own screens or help inform future buying decisions.

“Our results set the north star for display development, with implications for future imaging, rendering and video coding technologies,” said co-author Dr Alex Chapiro from Meta Reality Labs.

Over the past several decades as many television markets worldwide transitioned to higher definition video, BBC, NHK, EBU and ITU all conducted research that agreed with these findings.

Extraordinary technological advances are taking place in digital video imaging. Both professional and consumer image-capture systems are now widely available in 4K, raising audience expectations for more immersive experiences. Other critical dimensions of imaging beyond resolution are now vying with the ongoing elevations in image sharpness. Higher dynamic ranges, wider color gamuts and higher picture capture rates have all gained global attention by virtue of the impressive enhancements they each impart to imagery.

4K has almost become a byword and 8K resolution is looming, but close examination reveals somewhat uncoordinated movements in 4K image capture adoption and very little movement towards 8K.

SPORTS AND ‘OVER THE TOP’ SERVICES ARE DRIVING 4K ADOPTION

4K TV shipment value grew by 6% in 2018. The global market is expected to reach more than 380 billion by 2025. HD TV shipments have plummeted as global TV manufacturers drive 4K TV sets into the world’s disparate markets.

However, paradoxically, most viewers are watching HD program material upscaled to 4K on all of these displays. Broadcasters and cable operators in the Americas and Europe have largely shunned investing in 4K delivery services, largely on the basis that they see little return on investment in the very significant new broadcast infrastructure costs required. Yet, most feel it is only a matter of time before competitive dynamics usher in the inevitable arrival of 4K. It is for this reason that manufacturers are presently selling multiformat cameras that are able to switch between HD and 4K resolution capture, supporting an easy transition. Increasingly, professionals are choosing 4K lenses because they can originate higher HD image quality today, while being future-proofed for downstream 4K services.

This adoption is especially pronounced in the case of high-profile outside broadcasts—particularly sports coverage. The anticipated renewal of major sports broadcast rights over the next three years is likely to spur new investments in lenses and cameras.

In addition to sports broadcasters, the quickly growing new world of “Over The Top” (OTT) services has seen major players quickly adopting 4K content creation and distribution.

While many broadcasters study transition scenarios, sports and OTT are spurring brisk growth in the adoption of 4K lenses, cameras and reference displays.

HIGH-END IMAGING ON THE BIG SCREEN

Quite different from broadcast television, digital 4K and higher resolution origination continues to grow in moviemaking, with more and more movies across genres being shot using this technology each year. In recent years 5K, 6K and 8K have appeared in digital cinematography and are stimulating exciting advances in lenses and cameras that will continue in the years ahead.

The large screens in digital cinemas directly benefit from the substantial increase in resolution. Beyond that, mastering high-end entertainment materials in 4K extends the shelf life of those masters, while downsampled versions from the 4K master produce higher HD sharpness for today’s home distribution and enhanced 2K for the majority of digital cinemas.

THERE IS A MODEST BUT GROWING ROLE FOR 8K BROADCAST TELEVISION

At this early stage, from my observation, no U.S. broadcaster, cable or satellite operator has shown much interest in offering 8K delivery services. However, the 8K imaging system is finding an important niche application in U.S. sports broadcasting, supporting the extraction of 1080p segments from wide-angle 8K images, which has opened important new degrees of image clarity in replays.

Elsewhere in the world, however, 8K is gaining even higher visibility. A major sporting event in Japan next year will be broadcast in 8K UHD. Hundreds of thousands of 8K TV units have been sold in Japan, Korea and China in 2019, and by 2023 the Asian market is predicted to see 130% growth.

In anticipation of this increased adoption, Canon has been working intensively over the past decade on 8K optical developments. We recently announced the first two 8K UHD broadcast television lenses to be released based on the 8K UHD 1.25-inch image format: One is an 8K box field lens with a groundbreaking 51x zoom range and a 10.7mm wide-angle portable 8K lens with a 7x zoom range. Canon’s research and development in Super 35mm 8K optics have also significantly contributed to the high optical performance of the 4K Cinema EOS lenses.

THE FUTURE IS NOW

The evolution of video-capture technology is happening, with some early adopters seeing advantages in creating truly immersive viewing experiences. With 4K and 8K video capture lenses becoming more prevalent and driving the high image quality that consumers increasingly expect, the future is bright for today’s HDTV image quality, while familiarity with 4K production steadily grows. 

Larry Thorpe is Senior Fellow at Canon U.S.A, a Lifetime Achievement Emmy Award winner, holder of 10 patents in the field of broadcast development, and publisher more than sixty papers on camera technology and the topic of HDTV imaging.

BURLINGTON, Mass.—Avid is set to showcase the latest version of its Pro Tools digital audio workstation at AES 2019, featuring updates that enable it to support higher video resolutions and frame rates for post-production audio workflows.

The 2019 Pro Tools system enables sound editing teams by enabling the playback and display of 4K resolution files and higher frame rates to seamlessly work with videos in Avid’s MediaComposer. Users can select frame rates and resolutions independently and work with non-standard video sizes.

The audio platform also now features full Core Audio support of the Dolby Audio Bridge that will allow users to send 130 channels to the Dolby Atmos Renderer, simplifying Dolby Atmos “in the box” mixing and playback workflows with Pro Tools | HDX and other Core Audio devices.

In addition, users now have the ability to deliver multiple mixes in a single WAV file to allow for easier delivery of localized mixes to streaming services.

Avid will be displaying the updated Pro Tools technology at booth 503 during AES 2019, Oct. 16-18 in New York.

BATH, U.K.—Samsung’s 2020 business strategy for TV sales is simple: 8K or bust. With its QLED 4K TV sales being undercut by budget 4K TVs, Samsung plans to shift the market again, to a format that has (so far) very few competitors, but also very little native content.

Yet as we saw in our 4K vs 8K comparison test earlier this year, you don’t actually need video shot in 7680 × 4320 (8K) resolution to take full advantage of those millions of pixels—Samsung’s 8K TVs use AI upscaling to convert any video type (SD to 4K and everything in-between) into 8K resolution.

Upscaling, of course, isn’t anything new. For years, 4K and even HD sets found ways to stretch lower-resolution content to fit the greater pixel-per-inch ratio of modern TVs. But with 8K TVs needing to fill four 4K TVs worth of pixels, conventional upscaling methods just don’t work, for reasons we’ll get into below.

Now, after visiting the Samsung QA Labs in New Jersey and speaking with its engineers, we have a better idea of how Samsung uses artificial intelligence and machine learning to make 8K upscaling possible—and how its AI techniques compare to other TV manufacturers’ early AI efforts.

WHY CONVENTIONAL UPSCALING LOOKED SO TERRIBLE

Before 1998, television broadcast in 720x480 resolution, and films shot in higher quality were compressed to fit that format. That’s 345,600 pixels of content, which would only take up a tiny window on modern TVs with higher pixels-per-inch (PPI) ratios. That SD content? It must be stretched to fit over 2 million pixels for high definition, over 8 million for 4K or over 33 million for 8K.

The baseline for upscaling is maintaining the proper pixel ratio by simple multiplication. To convert HD to 4K, the TV’s processor must blow up one HD pixel to take up four pixels of space on the higher-res screen. Or 16 pixels during an HD-to-8K conversion.

Without any image processing, the picture ends up, to quote Tolkien, “sort of stretched, like butter scraped over too much bread.” Each bit of data becomes unnaturally square-shaped, without a natural gradient between details and colors. The result is a lot of blocking, or noise, around objects on screen.

You’ll also likely see something called “mosquito noise.” To compress a video to work for your limited internet bandwidth, broadcasters and websites must fill the stream with intentional color flaws, or “compression artifacts”. The purposefully flawed pixels will swarm around parts of the screen where there are sharp contrasts, like the brown bridge before blue sky in the image above.

THE MATH BEHIND UPSCALING

In the face of these issues, TV programmers taught their TVs to analyze and digitally process the images in real time to fill in or repair missing or damaged pixels. And they accomplished this using mathematical functions, which you can tell your loved ones the next time they say that too much TV rots your brain.

Specifically, engineers taught the TV processor to interpolate what each missing pixel’s color value should be, based on its surrounding pixels. To do so, it had to define its kernel: the function that assigns color priority to a pixel’s neighbors, based on their proximity.

The most basic kernel used in TVs is nearest neighbor kerneling, which simply calculates which pixel is nearest a vacant pixel and pastes the same color data into the empty pixel. This method causes the picture to take on a blocky zig-zag pattern, or aliasing, with poor edging. Picture a black letter “A” on a white screen; a missing pixel just outside the letter may be filled in as black, while a pixel on the edge of the letter could display as white. The result will either be a gray blob around the letter or a jagged staircase of black and white going up and down.

Bilinear interpolation requires more processing power but is more effective. In this method the blank pixel is compared against the nearest two neighbors to form a linear gradient between them, sharpening the image. This produces smoother visuals but can be inconsistent. So other TVs use bicubic interpolation, which pulls from the 16 nearest pixels in all directions. While this method is most likely to get the color as close to accurate as possible, it also typically produces a much more blurry picture, with edges taking on a distracting halo effect.

You can likely guess the problem already: these TVs fill in pixels based on mathematical formulas that are statistically most likely to produce accurate visuals, but have no way to interpret how they’re thematically supposed to look based on what’s actually on the screen.

So, after explaining how these algorithms consistently came up short, Samsung’s team explained how their AI overcomes these shortcomings.

SAMSUNG’S SECRET: MACHINE LEARNING, OBJECT RECOGNITION AND FILTERS

Samsung’s secret weapon is a technique called machine learning super resolution (MLSR). This AI-driven system takes a lower-resolution video stream and upscales it to fit the resolution of larger screen with a higher PPI ratio. It’s the equivalent of the old ridiculous TV trope of computer scientists zooming in and “enhancing” a blurry image with the tap of a keystroke, except done automatically and nearly instantaneously.

Samsung’s reps explained how they analyzed a vast amount of video content from different sources—high- and low-quality YouTube streams, DVDs and BluRays, movies and sporting events—and created two image databases, one for low-quality screen captures and one for high-quality screen captures.

Then, it had to train its AI to complete a process called “inverse degradation” by the AI industry. First, you take high-resolution images and downgrade them to lower resolutions, tracking what visual data is lost. Then you must reverse the process and train your AI to fill in the missing data from low-resolution images so that they mirror the high-resolution images.

Samsung’s team calls this process a “formula.” Its 8K processors contain a formula bank with a database of formulas for different objects, such as an apple or the letter “A”. When the processor recognizes a blurry apple in an actor’s hand, it’ll restore the apple edges, repair any compression artifacts, and ensure that blank pixels take on the right shade of red based on how apples actually look, not based on vague statistical algorithms. Plus, along with specific object restoration, the AI will adjust your stream based on whatever you’re watching.

According to Samsung, it has dozens of different “filters” that change how much detail creation, noise reduction and edge restoration is needed for a given stream, based on whether you’re watching a specific sport, genre of movie or type of cinematography.

Edge restoration isn’t even the most difficult task for the AI, according to Samsung’s engineers. Instead, replicating the proper textures of an object in real time remains a difficult challenge. They must ensure that the processor augments the appearance of objects without them taking on an artificial appearance.

What the processor won’t do (according to Samsung) is miscategorize an object. “It won’t turn an apple into a tomato”, one engineer assured us, though without giving any details. Very likely the processor is trained to avoid any drastic alterations if it doesn’t recognize what an object is.

You also won’t see the AI alter the “directorial intent” of a movie, as Samsung’s team put it. So if a director uses the bokeh effect, the blurred background will stay blurred, while the foreground gets dialed up to 8K crispness.

They also claimed they don’t specifically analyze more popular streams for their object categorization, aiming more for general quantity and diversity of content. So, no word if they have a “dragon” or “direwolf” formula for your “Game of Thrones” binge-watches.

New Samsung 8K (and 4K) TVs ship with the most recent formula bank installed, and then new object data is added via firmware updates that you must approve. Samsung says that it will continue to analyze new visual streams to expand its object library, but that it does that locally on Samsung servers; it doesn’t analyze data from people’s TVs.

Just how many object formulas has Samsung accumulated from its endless stream analysis? One of its engineers gave an off-the-cuff amount that sounded impressively large, suggesting the processor will typically recognize a huge number of objects on screen. But a PR rep cut in and asked that we not print the number, saying that they’d rather that consumers focus on how well Samsung’s MLSR works than on arbitrary numbers.

AI UPSCALING: THE NEW NORMAL?

Samsung isn’t the only TV manufacturer that currently uses artificial intelligence and image restoration for its TVs.

Sony’s 4K ad page goes into obsessive detail about its AI image processing solutions. Its new 4K TVs contain processors with a “dual database” of “tens of thousands” of image references that “dynamically improv[e] pixels in real time”.

LG also announced ahead of CES 2019 that its new a9 Gen 2 TV chip would feature image processing and machine learning to improve noise reduction and brightness—in part by analyzing the source and type of media and adjusting its algorithm accordingly.

Beyond the AI elements, however, it seems as though these TV processors do still depend somewhat on automated algorithms. When we previously interviewed Gavin McCarron, technical marketing & product planning manager at Sony Europe, about the AI image processing in Sony TVs, he had this to say:

“When you’re upscaling from Full HD to 4K there is a lot of guesswork, and what we’re trying to do is to remove as much of the guesswork as possible. [Our processor] doesn’t just look at the pixel in isolation, it looks at the pixels around it and on each diagonal, and also it will look up the pixels across multiple frames, to give a consistency in the picture quality.”

Sony, along with LG and Samsung, very likely use some form of bilateral or bicubic algorithm as its baseline upscaling system. Then they analyze the near-4K content and determine which pixels should be augmented with image processing and which should be deleted as noise.

In that sense, most TV manufacturers are relatively close to one another in the AI upscaling race. The exception is Samsung, which uses the same techniques but fills in four times the number of missing pixels to fit an 8K screen. We’ll have to wait and see if other manufacturers’ AI efforts will allow them to leap into the 8K market as well.

Reprinted by permission from TechRadar. TechRadar is the largest U.K.-based consumer technology news and reviews site (and rapidly growing in the U.S., Australia and around the world), our editorial independence backed by the weight of technology publisher Future plus objective test data from the TechRadar Labs.

First off, let’s dispense with the niceties. All of this talk within the broadcast community about how consumers will react to the quality of 4K/UHD pictures is just that: talk. How we personally feel about the degree of difference in resolution and consumers’ perception of said quality won’t matter much to shoppers making a decision about whether to trade up to next-generation television. To a lot of people, UHD isn’t worth the price of the upgrade yet, but once prices come into line with those of current 1080p sets (which they are quickly doing), it will be, especially once UHD becomes the only option available. As for high dynamic range and high frame rate? Sure, they are icing on the cake (with OLED as the cherry on top), but the jury is still out on how much of an effect they will have on consumers’ choices.

Sony launched its Z-Series UHD/4K TVs in July, featuring picture enhancements such as HDR capability and new backlight technology. The big variable in the rollout of UHD is in the availability of content. Much has changed since the rollout of HDTV, the largest being the ubiquity of over-the-top alternatives to broadcast and traditional pay-TV services as well as the explosion of mobile screens. Survey after survey demonstrates that millennials are abandoning the traditional big-screen TV in favor of tablets and smartphones. So regardless of how one feels about the experience, the reality is that resolution and picture quality aren’t always the overriding factors in how one chooses to consume media.

Two recent reports shine some light on the future of UHD and reach similar conclusions about which factors will determine the format’s success: content and bandwidth availability and device capabilities. And both conclude that OTT will be the driver of the format.

In its report, “A Clearer Picture of Growth: 2016 Global 4K/UHD Industry Survey,” SNL Kagan interviewed 475 global video service providers (including pay TV/OTT operators and content producers), of which 96 percent of respondents believe that a majority of consumers and video service providers will adopt 4K/UHD by 2020, with 78 percent of total respondents expecting launches within two years. Differentiation from competition as well as the need to be “in the game” are the biggest drivers for adoption among survey respondents, who also believe that consumers will be willing to fork out an extra 10 to 30 percent on top of their current services for 4K content. Connected 4K TVs will be the primary device for viewing high-resolution content, with live sports and early release movies expected to be the most popular content for consumption. Globally, the Asia-Pacific region is expected to lead the pack in terms of early adoption, with Europe and North America close behind.

Since live sports will be a major driver of 4K adoption, concerns over bandwidth availability for 4K/UHD take on greater importance, with more than 60 percent of respondents rating the issue as potentially having a high impact. UHD is a notorious bandwidth hog and adding HFR and HDR to the picture just adds to the burden (and despite the rumors of its demise, physical media, including UHD players are expected to still be an important part of the distribution mix). Fortunately some of the concerns over bandwidth can be alleviated by advances in compression technology; according to the survey, 54 percent of video service providers say they will offer different resolutions to fit multiple devices, however only 40 percent of content producers expect to provide content at different resolutions.

Another report on the format from Juniper Research estimates that more than 189 million unique users will be using 4K OTT services globally, up from just 2.3 million this year. In its report, “Digital TV and Video: Network and OTT Strategies 2016–2021,” it predicts that in the United States, 10 percent will be watching 4K video online by 2021 compared to just one in 500 this year, with connected TVs being the dominant channel. It bases much of its optimism on the increasing availability of 4K options from the likes of Youtube, Netflix and Amazon as well as the market availability of 4K-capable OTT boxes. Juniper also predicts that 8K will become a market force by 2020, with TV shipments growing threefold between 2020 and 2021 to 400,000 per year.

So when it comes to adopting next-generation 4K/UHD media, the experts are relatively bullish, though with several caveats—all related to timing, availability and consumer preferences. Pretty much what it was like with HDTV.