Date: 08 Apr 2020
Written by: Yoeri Geutskens

If you’ve been keeping up with Ultra HD TV technology, you may recall that there are essentially six pillars to it:

1. UHD or Ultra HD Spatial Resolution 4K or 8K.
2. HDR or High Dynamic Range.
3. WCG or Wide Color Gamut.
4. Deep Color Resolution.
5. HFR or High Frame Rate.
6. NGA or Next-Generation Audio.

Most of these we’ve got by now. You can go to an electronics store and buy a 4K TV with HDR, WCG, 10-bit color and a Dolby Atmos sound bar or AVR, and they’re not even expensive anymore. The one piece that’s missing? HFR, or High Frame Rate. It’s probably the least understood in terms of benefits and how it works – by consumers but likely also by the creative industry. Even science is still lacking. It’s probably the most controversial of all UHD technologies. Many misconceptions abound, so here’s an attempt to shed some light on what we know and what we don’t know. Many misconceptions abound, so here’s an attempt to shed some light on what we know and what we don’t know.

HFR FOR MOVIES:
What do we mean by HFR? That depends what we are talking about. When it’s about film, anything above 24fps (frames per second) will be called HFR. It’s not very common. The number of high-profile movies with HFR you can count on one or two hands. First we had Peter Jackson’s Hobbit trilogy, shot in 3D at the double frame rate of 48fps. Since then, we’ve had director Ang Lee taking things further with Billy Lynn’s Long Halftime Walk shot at 120fps and Gemini Man starring Will Smith, both shot at 120fps and in 3D. Latter movie was screened in different ways: 2D theaters showed it in 24fps; 3D theaters showed it at 60 or 120fps, depending on their capabilities. And with good reason: Ang Lee pursued high frame rate in order to overcome issues inherent to projection of 3D images at 24fps – strobing and flicker, just like Peter Jackson did. James Cameron has a slightly different approach but prefers to use 120fps in certain parts of 3D movies to avoid judder in shots that pan or have lateral movement across the frame. But HFR for movies is a very divisive technique. Some people love it, a majority of people seem (I don’t have any hard numbers here) to hate it. 24fps is sacred.

DIRECTOR ANG LEE FILMING ‘GEMINI MAN’. PICTURE CREDIT: PARAMOUNT

But why? Well, part of the reason is that’s what we’re used to. This video does a good job explaining why we’ve got 24fps, and why we’ve kept it. But if more pixels and more colors and more bits per color and more audio channels are good, why would more frames per second be bad? Doesn’t it add more realism? Yes, it does, and that’s exactly why it’s bad for movies. Realism is not the point – quite the contrary. It’s very similar to the discussion around analog, chemical film grain: Some people dislike it, but like 24fps frame rate it’s something our brains have been conditioned for since almost 100 years. Although most of us will not consciously notice it, our brains register it subconsciously and know we’re watching a proper film, a possibly epic story. It helps with our suspension of disbelief and puts our brains into ‘movie watching mode’ and immerses us, pulls is into the story. Some refer to 24fps as a ‘dream-like cadence’. Heightened realism takes away from that. It breaks the magic spell. As one moviegoer succinctly put it: “I didn’t see Gandalf et al – instead I saw a load of actors dressed up in some odd costumes.” Billy Lynn and Gemini Man triggered many of the same type reactions, as you can see on social media. Some compare it to a home video shot on a GoPro or a smartphone, looking “hyper-real” or “like a videogame”; others refer to the “Soap Opera Effect” as it’s called (a bit of a misnomer), and the dreaded motion interpolation that their in-laws have enabled on their TV.

Much however is unknown about how our brains process frame rates and motion perception. More scientific research into this area would probably be justified.

How come we (most of us at least) subconsciously perceive 24fps frame rate without being bothered by judder?

Are our brains really conditioned into seeing 24fps as ‘epic’ yet when we see 60i or 60p this mechanism shuts down and we go into ‘soap opera’ mode?

And yet for videogames, which nowadays are also a lot about storytelling, High Frame Rate has evident benefits that gamers appreciate apparently. So do people who play videogames at high frame rates (120fps and above) perceive movie frame rates differently?

Also, provided the Soap Opera Effect is real, there must be an Inverse Soap Opera Effect whereby TV content converted from 60fps to 24fps suddenly starts to look epic? What points to this is a common technique in sports news shows, where they cut the frame rate to 24, 25 or 30fps, crop the picture to get a wider aspect ratio (adding black bars at the top and bottom) and add dramatic music when they want to make a game summary look epic.

Does frame rate matter for traditional cel animation movies, and if so, how?

At what frame rate above 24fps does the magic stop working? 25? 30? 48? 60? Anecdotal evidence (the three Hobbit movies) suggests 48fps is already guaranteed to blow it, but where is the border?

Although soap operas are never in 3D, 3D does not seem to help with suspension of disbelief or make something more epic. Perhaps even contrary. Is this because 3D adds realism which, like HFR and absence of grain, breaks the spell rather than sustaining it?

Is this behavior learned? Would someone from another culture who’s never been to the cinema experience frame rates the same way?

So many questions, so few answers. With the advent of Filmmaker Mode and this insistence of accurately reproducing 24fps frame rate, it’s easy to forget that before Ultra HD Blu-ray, the original Blu-ray Disc and HD-DVD were the first home media ever able to handle this frame rate. Broadcast TV, all consumer videotape formats, LaserDisc and even DVD used interlaced video. Yes, DVD was the first format to offer Progressive Scan, but not at 24fps. An important lesson from the history of Hollywood that’s perhaps easy to forget because most of it happened so long ago is that every major technology transition also led to new movie genres and significantly shifted the balance between existing movie genres. HFR is a powerful new tool for movie making in a larger toolbox, so exploration is required. The old ways will not work with HFR, simply because they are built on different assumptions about movie making. Maybe HFR will give rise to entirely new genres, maybe it will take computer animation to heights that we can’t yet imagine. Combined with computer animation HFR may give rise to CGI actors that are more convincing than live actors. We just don’t know yet. What we do know based on past experience though is that every time a major innovation presents itself this discussion flares up: “Who wants to hear actors talk”, “No one wants color film”, and so on. This line of thinking is usually a losing argument over time. It’s not a given that 24fps is going to last forever. So what’s going to happen next? These transitions take many years, sometimes even decades when you have a firmly established structure such as the Hollywood studio system with its worldwide cinema distribution system but innovation never stops, also in the film industry. New secondary tools for movie making are implemented into current workflows and embraced by the industry. Helicopters? Awesome, let’s do flyover shots. Drones? Cool. Large LED screens? Using giant LED screens showing rendered virtual environments instead of green chromakey walls, like they used for shooting The Mandalorian? Absolutely. Anyway, it’s too early to draw conclusions. We’ve had fewer than ten HFR movies, from only a handful of directors. All of these have been conceived as a way to do 3D better. Meanwhile 3D itself has taken decades for Hollywood to get it right, become mainstream and get accepted as more than a novelty feature. For now, the best advice is to go see HFR with your own eyes (and an open mind) to form your own opinion and not accept any dogmas as absolute truth.

CONVERGING TECHNOLOGIES:
It’s important to bear in mind that Hollywood movies are still made and optimized for cinema. The technical capabilities of cinema guide movie production and as such many of the ‘truths’ in Hollywood are based on this system. Things could change if suddenly home entertainment becomes a much bigger market for Hollywood, which was actually already happening in 2018 and 2019, and perhaps much more dramatically so in 2020. Movie production optimized for TVs would most likely look different from film production optimized for cinema. After all, what works well at the cinema doesn’t necessarily work well on a TV set. Research by Dolby Labs found that higher brightness makes judder more apparent, as does higher contrast. What looks good in SDR judders too much in HDR, so colorists end up grading HDR darker to avoid this, which defeats the whole purpose. The study also showed that at 50 nits – the typical brightness with traditional theatrical projection – 24fps is the ideal frame rate whereas at 1000 nits – achieved on now fairly common and not too expensive HDR TVs – 32fps would be preferred. Although television came several decades later than cinema, the two technologies have been on parallel paths, going through many of the same innovations. Film went from silent movies to sound, from black & white to color, from mono to stereo to digital surround sound Dolby Digital, DTS, SDDS, initially all on optical film with an ever-growing number of channels. These changes have been noticeable but followed an evolutionary approach. But more recently the cinema business has seen some innovations that are major technical changes but entirely behind the scenes and very subtle if at all noticeable to the viewer: from chemical film to digital projection, with movies distributed on HDDs in cartridges and soon possibly online. The next major step in cinema is probably going to be the most drastic one in a century: from projection on a silver screen to ‘direct view’ displays. Of course, no CRT, no plasma, no LCD, no OLED but real LEDs. These bring far greater brightness, contrast and dynamic range, and will bring cinema screens back on par with home cinema – where they are now essentially running behind on spatial resolution, dynamic range and brightness – only way bigger. It will also add the flexibility to use higher frame rates though whether this will be used is doubtful, given the above considerations and the generally conservative nature of the movie business. This switch will also bring new entrants to the market. Samsung, which has never been in the cinema projection business, has launched its ‘Onyx Cinema LED Technology’ – 34-feet (10-meter) diameter screens with true 4K 4096 x 2160 resolution. The first cinemas rolling this out since 2017 were Lotte Cinema World Tower in Seoul, Paragon Cineplex Theatre in Bangkok, Pacific Theatres Winnetka in Chatsworth, California, just north of Los Angeles, Pathé Beaugrenelle in Paris, Sambil Leganés in Madrid, and the Shoudu Cinema in Beijing.

This transition is going to take years. This stuff isn’t cheap, and the technology it’s replacing isn’t cheap either. It’s a capital investment. But cinema and TV technology are converging further than ever. In the future, a cinema screen will basically be a very large TV set – typically with a far superior Dolby Atmos system. Expect this transfer of TV technology to the cinema to feed back into the home. The first signs are already here. If you’ve got deep enough pockets, you can buy Samsung’s ‘The Wall’ micro-LED display. It’s a modular system, which means you can construct various screen sizes and resolutions. One module measures 16×18 inch and counts 360×360 pixels. An HDTV will use 18 modules, a 4K display 76 and an 8K one 288. There’s no limit, really. 16K displays are also possible. The only constraint is basically money. A single module will set you back about $10,000 so you can do the math.

The specs and sizes for Sony’s Canvas or Cledis Crystal LED Integrated Structure or Display System are very similar. This technology is of course aimed mainly at professional applications but Sony explicitly says it’s also available for living rooms. Now for this to become something for the mass market, we need a price reduction of about 99%. That sounds very steep, but we’ve witnessed exactly that in the 4K TV market over the past seven years. Great news: like the cinema product, the home product offers strong HDR (perfect blacks and 1000 nits peak brightness), wide viewing angles, great 3D, and 120fps HFR.

HFR FOR OTHER TV CONTENT:
So if it is only for scripted, acted content that low frame rate matters, are higher frame rates better where realism matters i.e. nature documentaries and live sports? Evidence suggests so. HFR adds to the sense of ‘being there’ in a good way. But before we go into that, back to the definition of HFR. Broadcast TV comes in a range of resolutions now. The trend is upward but very slow. A related trend is that slowly but surely we seem to be getting rid of interlaced video, where ‘fields’ (half frames with only the odd picture lines or the even ones) are displayed successively. Sure, at 1080 HD resolution there’s still a lot of 50i and 60i content but at Ultra HD resolutions only Progressive Scan with full frames is permitted. Various frame rates are allowed (including fractional ones), but 50p and 60p are not considered HFR – they’re Standard Frame Rate. When organizations like the Ultra HD Forum speak of High Frame Rate they mean at least double that – 100 or 120fps and beyond. What drives the (very slow) move to higher frame rates? Is it a silly numbers race, like some would argue the move to higher spatial resolution (4K, 8K) became? Not quite. Even if we don’t need or want it for movies, there are definite upsides. First, why are we moving to higher screen resolutions? Just because TV makers can, and they see TVs with higher resolutions having higher margins? No, there’s more behind it than technology push. Since many decades, TVs are getting bigger and bigger. It’s a pretty constant trend, and the average diameter grows by about 1 inch per year across all territories, even if these averages vary from region to region. In the meantime, our viewing distance does not change much. Living rooms (also varying in average size geographically) did not get significantly bigger. That’s why we need more pixels. Now with higher resolutions, the risk of motion blur increases. With 8K, this is particularly visible. 8K sports content, like the Olympic Games, you probably do not want to watch at frame rates lower than 100fps. Although Japanese public broadcaster NHK has announced quite some time ago they’ll shoot and transmit many parts of the Olympics in 8K, they have not yet said at what frame rate.

THE DIFFICULTIES WITH HFR:
While shooting, recording and transmitting HFR may be relatively straightforward (arguably more so than HDR), there is a complication: How to achieve backward compatibility with Standard Frame Rate TV sets and transmission systems? At the moment there are two approaches to this, and DVB and ATSC solve this in different ways. Here it’s going to get a little more technical. What the two have in common: Both use a technique called temporal sublayering for backward compatibility of HFR with SFR. ATSC includes optional temporal filtering for enhancing the standard frame rate picture when temporal sublayering is used,

HFR: DVB and ATSC – temporal sublayering:
How does this work? In ATSC and DVB both, PID (program ID) = 0 is the SFR version, and PID = 1 is the HFR enhancement element, to be used along with PID 0 to reproduce the HFR version. In DVB, it actually wouldn’t matter which PID you viewed, they are just the odd and even frames, so each represents a half-frame rate feed, with just a slight timing offset. In ATSC, the frames are a bit different. The frames in PID 0 are a weighted sum of the odd and even frames of the HFR signal. The result is that the PID 0 content has an artificial motion blur. The HFR camera needs a 360 degree shutter (i.e., photons are being captured essentially 100% of the time; the camera doesn’t blink). The contents of the PID 1 frames are the weighted difference between the two signals. The trick here occurs in the receiver: As in DVB, if you don’t know better, show PID 0, you’ll get a usable SFR signal with full motion blur (depending on the weightings). If you do know better, you recover consecutive HFR frames by summing and differencing the two PIDs frames to reconstitute the original odd & even frames of the HFR.

In HFR demos the Ultra HD Forum has given over the last few years, they showed the DVB technique and sometimes, in earlier demos, done it poorly: The camera didn’t have a 360-degree shutter, it was more like 180, so the camera was capturing 100fps, but the exposure was 1/200th second in duration. Odd frames went to PID 0, even to PID 1, and when viewing only one of those, the play-out was 50fps, but the shutter was effectively 90 degrees (still a 1/200th of a second exposure), giving a very staccato, strobe-like presentation which was hard to watch. Eventually, they got a HFR camera with a 360-degree shutter, so the SFR playout appeared as if having a 180 degree shutter, which looks acceptable.

To be sure, these two flavors are not competing in the same market, can coexist in software or silicon in the same TV set, probably do not involve any license fee and are not a matter for a future format war, just in case anyone gets worried. The ATSC and DVB solutions can be used for terrestrial TV, DTH satellite TV, cable TV and (multicast) IPTV. So what about (unicast) OTT streaming? There this compatibility is not an issue at all. The VoD provider just plays out the version that matches the capabilities of the viewer’s system. That can be 60 or perhaps 120fps (in the future, that is – current products such as Apple TV, Roku, ChromeCast and Amazon Fire TV don’t go beyond 60fps) but different frame rates will simply be different versions of the same asset, in the same way that a HD and 4K resolution are different version of the same asset.

HFR IN PRACTICE:
So can you go to a store and buy a HFR TV or monitor? One area where this will come in handy is gaming. Current gaming PCs as well as the upcoming PlayStation 5 and Xbox Series X, the ninth generation of consoles, are (going to be) capable of HFR output. PC/gaming monitors already cater to that. High Frame Rate is one of the few feature areas where monitors are ahead of TVs. Most don’t do a too impressive job in terms of high dynamic range, contrast, peak brightness, wide color gamut, resolution, etc. But while TVs currently don’t exceed 60fps (claims about 120Hz, 240Hz, etc. are often marketing overstatement), gaming monitors can now routinely handle 144fps, 165fps and even 240fps. They’re locked in a numbers race, trying to keep up with graphics cards output capabilities. To what extent the human eye can appreciate the difference between 144fps and 240fps remains a question. Mark Rejhon, founder of Blurbusters, argues in favor of a “retina refresh rate” of over 1000fps based on quite extensive research that he’s been doing.

LG has given HFR TV demonstrations as far back as 2016, and more recently in 2018, when they announced sets for 2019. On 2018 models HFR support is still limited: They can play HFR files from USB, as proven in this LG OLED C8 test. Since 2019 LG high-end TVs have had HDMI 2.1 ports with 4K at 120fps via FRL (Frame Rate Link – HDMI 2.1’s new signalling system for 48Gb/s bandwidth), LG says. 2020 LG high-end TVs have the same HDMI 2.1 support with 4k120p via FRL. Here’s how they promote that on their website:

It’s a feature that few reviewers pay attention to, probably because there’s so little HFR content out there, but it’s one of the things that makes LG’s current high-end UHD TVs intriguing. They are ready for 4K HFR from PlayStation 5 and Xbox Series X (even if they may require a firmware update to fully enable HDMI 2.1 after certification). Very few other TVs are. LG is also specifying VRR with 4K 40-120Hz frequency range. Actually it’s also possible to do 4K120 (limited to 8-bit SDR and 4:2:0 chroma) over HDMI 2.0 bandwidth, but it is out of spec. Samsung has been doing it however since 2019 (in high-end TVs) and LG since 2020 (at least in OLED TVs). Several generations of TVs have 1080p120 support, even sometimes 1440p120. This is well within the HDMI 2.0 bandwidth. Test site Rtings has been certifying 1080p120 and 1440p120 in their reviews for some time now, as in this 2017 Sony A1E OLED TV review (check ‘supported resolutions’). There may even be older TVs out there. It could be going to take some time before we start seeing HFR content available, especially live content. There’s a good chance with HFR that, like with 4K resolution, streaming platforms will take the lead over broadcasters. So maybe look to DAZN rather than ESPN. But the chicken and egg situation that so often exists when the hardware makers or content providers need to innovate first you don’t need to worry about. The TV manufacturers have already done their part. Once 120fps TVs become common, broadcasters may begin shooting sports matches in HFR. This does not necessarily have to be in 4K. A program in 1080p at 100 or 120fps with HDR will look quite stunning. Until they’re ready for that, you can use motion interpolation to do the job.

If you want to experience HFR for yourself, at home, you can buy the 4K HDR Ultra HD Blu-ray of Billy Lynn or Gemini Man. Both are authored at 60fps HFR, as the UHD BD standard doesn’t handle 120fps. In fact it also doesn’t handle 48fps, so you’re out of luck if you wanted to watch The Hobbit Trilogy at the proper frame rate. The regular 1080p HD SDR 2D Blu-rays and 3D Blu-rays contain the movie at 24fps. There is no Ultra HD Blu-ray of this yet but when it does arrive it’s surely going to be 24fps, too. A 3D Blu-ray of Billy Lynn comes bundled with the 4K disc, if you buy the right edition (linked above). Gemini Man was not released on 3D BD in most markets, but it was in Germany. The 3D discs are also 24fps. Unfortunately, the HFR format doesn’t support 3D and the 3D format doesn’t support HFR. VoD/streaming services do not offer any 3D or HFR content at the moment.

RECOMMENDED READING:
You can read more facts about and impressions of The Hobbit here on FlatpanelsHD, or read about the aftermath. There is plenty of coverage for Ang Lee’s movies but long before the days of internet, Douglas Trumbull – Visual Effects Supervisor for classic movies including 2001: A Space Odyssey, Close Encounters of the Third Kind, Star Trek and Blade Runner – did many experiments with HFR. This article about his movie Brainstorm offers a great summary of that. Trumbull and Cameron were speakers at a 2012 Siggraph panel session on HFR cinema, a report of which you can read here.

STANDARDS REFERENCED:
DVB UHD-1 Phase 2 (ETSI TS 101 154 v2.3.1)

ATSC 3.0 (A/341) Many thanks to Bill Redmann, Director of Standards, Immersive Media Technologies at InterDigital, for his explanation of the DVB and ATSC approaches to HFR/SFR compatibility and his contribution to this article.

Date: 08 Apr 2020
Written by: Yoeri Geutskens

Source: https://www.flatpanelshd.com