Why Holography Is 3D's Next Big Technology (And it's nearer than you think!)

A typical set-up for Holographic 3D viewing – a new HDTV panel will be needed which carries twin cameras and refresh speeds of around 480Hz to keep around four people happyIllustrative diagrams and further images are in our current DIGITAL EDITION page 40

SEE DIGITAL EDITION HERE

Stereoscopic 3D has now really picked up a head of steam, a real momentum that can’t be stopped. But there are people who still believe Stereoscopic 3D is still a gimmick, we’re not talking film critics here but people whose view is based on fact not fiction. These feelings are mostly aimed at the unrealistic nature of dealing with depth with S3D and how depth percentages are either too small to deliver any significant dimension or too big in small bursts to provide a comfortable experience.

There is a new technology (albeit one that is sixty years old) called Holographic 3D that has been waiting in the wings as the next big thing but has been predicted as being at least eight or nine years away. Well it depends on who you talk to.

SeeReal are a company whose base is in Luxembourg and they have been in 3D R&D since 2002. They used to operate in S3D until they decided it was a transient technology and turned their attention to holography in 2005.

Hagen Stolle from SeeReal: “If you talk to the 3D broadcasters they will tell you that you only have the ability of around 4% depth ability, with Holography you don’t have this limitation because you use the same information as you see in real life.”

DoP and steadicam operator Paul ‘Felix’ Forrest has spent a long time looking at this technology as it would impact on cinematography. He explains the basics.

“Having spent a considerable amount of time with H3D and its creators it has gradually dawned on me that with such an incredible display technology, the prospects are that we will have to ‘finesse’ our shooting styles to really make the most of it.

“One of the most dramatic changes H3D brings about is the functionality, which allows several users watching the same set to focus on different scene points – concurrently!  The easiest way to imagine this is to think of the difference between viewing a film versus being at the theatre.

“With the viewer having total control over what they actually focus on is one thing but the subtlety of H3D is that when each viewer picks their point of focus in the scene, the rest of the scene naturally blurs much the same as it does in the real world. Change your scene point, it is in focus and everything else is blurred. With this in mind, shooting the most practical and usable content to exploit H3D is likely to mean shooting at hyperfocal distance.  

“Keeping in mind that H3D is totally compatible with 2D and more importantly with Stereoscopic 3D, there will always be traditionally shot content that will ‘adhere’ to our long established principles of cinematography.  However, there will be an opportunity for a ‘new breed’ of content, which will force stronger narrative to ensure the ‘scene point of choice’ chosen by the viewer matches that visualized by the Director.  Clearly, the utilisation of light, shadow and texture can be used to achieve similar effects without the need for shallow depth of field (as this is created naturally when scene points are viewed) and thus allow a full depth, in focus scene to be captured and eventually explored by the viewer.”

What about convergence?

Of course one of the common questions that crop up is based upon the fact that you still shoot left and right eye content and therefore still need to ‘dial in’ convergence.  However, key areas of difference exist in respect of H3D specific content.

Whilst it is possible to shoot in the same fashion as adopted for Stereoscopic 3D (with shifting points of convergence), with H3D, it is also possible to shoot with a fixed point of convergence.  Whilst it may be regarded as ‘good’ practice to continue to ‘pull convergence’ based upon where the action is (after all, the aim may be to make the content compatible with Stereo and Holo displays), early tests with H3D suggest that this may not be the best way to ‘optimise’ the scene.  

Remembering that Holography affords the user a free choice for the point of focus, it becomes key to ensure that ‘scene scale’ is accurate. Parallax can always be adjusted with H3D display as long as the original scale proportions are maintained. Remembering also that parallax is the strongest depth cue, you can see that convergence still has a major role to play in capturing usable images.  So, we can shoot with fixed convergence, as long as we have, and maintain the original scale proportions, we can adjust parallax to ensure a comfortable viewing experience for all scene points.

So, if you can shoot with fixed convergence, where do you converge?  Well this is the point at which another key difference is introduced.  If you imagine the H3D display as a ‘window’ onto your 3D scene, it is likely much of the action will occur behind the screen but with real depth (just like looking out of a window in the real world!). Therefore the most appropriate point for convergence is on the display plane (or zero/neutral parallax).  Once again, ‘scene scale’ is critically important: knowing the total depth of a scene and which part of this scene will be behind the display will help set convergence for that scene.  Once set for that scene, it shouldn’t be necessary to adjust it again, even when zooming in/out (as long as the original scale proportions are maintained, parallax can be adjusted). 

The clue to why this is less critical is that convergence in Stereoscopic 3D is used to indicate a ‘level of depth’.  With H3D, the level of depth is ‘chosen’ by the viewer as they naturally select a scene point in the scene on which to focus.  The eyes, then, very naturally converge and focus on the same point (hence no headaches and no depth cue mismatch).  This doesn’t mean it isn’t possible to create parallax errors (or painful divergence!) but the discipline here is to ensure that the maximum distance of a scene is always identified and in playback that maximum parallax means that at infinity, your eyes are parallel.  This additional information needs to be captured at the point of acquisition for the content (whilst complex algorithms can make a good job of recreating it, the workload could be considerable).  

So how is it done?

In essence, the most basic way of achieving this (the variable depth in the scene) is through the use of a Depth Map.  In many high end CAD applications and other 3D modeling software, Depth Maps have been used for some time.  They are essentially grayscale images which contain the ‘z’ axis data for distance from the focal plane to an object.  

With these high end CAD packages, the Depth Map is often 32bit.  The good news is that the level of ‘resolution’ required for H3D is lower and therefore an 8bit Depth Map is likely to provide the necessary level of granularity to reconstruct the scene with accurate depth.  Many high end rigs are capable of capturing data indicating such information although it is yet to be seen whether or not this includes the Depth Map in the format required.  As already described, one of the most important factors in H3D is the ‘scene scale’ and this is what is represented by the Depth Map and the main use of this depth map is for rendering the ‘reconstructions’ during display.  

Some commercial organisations have already developed algorithms to recreate the Depth Map from Stereoscopic 3D shoots to enable full depth budget 3D content.  These are likely to deliver the Depth Map necessary to experience 1:1 3D reconstruction to be shown on H3D systems.  

Although most important for display purposes, as discussed earlier – it is key, where possible, to identify this metadata at the point of acquisition to ensure that the most can be made of it for scaling the ‘reconstructions’ for different size displays.

With this in mind, it is likely to be the case that the data overhead for transmission should be relatively low.  As an example, it can be envisaged that Stereoscopic 3D content may be broadcast with the Depth Map as metadata alongside it.  For viewers with a Stereoscopic 3D set, it will display in its normal fashion with a limited depth budget.  For viewers with an H3D set, it will display perfect Holographic 3D reconstructions.

At least one of the broadcasters of 3D are open to prepare content immediately with that full Depth Map available. No names mention as yet!

So you can shoot at Infinity because you the viewer will choose where the scene point is focussed not the stereographer or the director. Think of that theatre analogy – if you go to the theatre we can all choose to look at different things on the stage. 

Stereoscopic 3D compatible

H3D can show stereo content without a problem. The exciting thing is that if you have access to the Metadata on the camera rig then you can re-compute your raw data and re-generate the real depth. With that very same data you can perhaps get a football match that was recorded in stereo with a depth of plus or minus a few inches but after re-computing it which is done on-the-fly – you have hundreds or metres of depth.

Ultimately with H3D you may not need a stereographer because at the moment a stereographer sets the convergence point and will try and steer to what production has decided what the main focal point of the scene. 

It will be more important to retain the data from the lens and the interocular distance captured for everything you’re shooting in 3D. Your compositional style will also change and it will be more like looking through a window, or from a box in a theatre. 

In the wider media people like James Cameron say that the earliest holographic solutions are at least eight years away – maybe he was quoting the University of Arizona who recently said the same (seven to ten years) and maybe he is protecting his data which is already patent protected. Japan generously promised full holographic playback of football games if they won the 2022 World Cup – alas there was no such promise from winners Quatar.

SeeReal’s Holographic 3D Solution

To achieve quality 3D real-time holography – realistic and comfortable to view – the images transmitted to the eye must be reduced to the essentials. Wasted information is also wasted image processing and this slows down replication and spoils image quality.

So the breakthrough technology developed by SeeReal banks on highly logical yet keep-it-simple solutions.

First of all, rather than boosting resolutions and intensifying display needs to unbearable levels, SeeReal has introduced Viewing Window technology. This actually limits display needs and corresponding diffraction angles.

The benefit: a much larger display pitch, so for a 40 inch display the pixel size is in the range of at 25 – 50 microns. This is well within today’s state-of-the-art.

Another benefit are smaller encoded hologram areas per scene point, with the shape and size of the sub-hologram closely linked to the 3D scene being viewed (sub-holograms are freely super-positioned, to accurately represent any 3D scene point distribution).

Overall, the combination of these two basic principles means that it only takes today’s computing capabilities to create full parallax colour 3D HDTV images in real-time. For example, a 40 inch holographic TV would consume only approx three TFLOPS.

Producing these viewing windows then limits the amount of computing power needed to see Holographic 3D. But what happens when there is more than one viewer? This is dealt with by faster displays and these are slowly coming to market. The information is multiplexed in real time and then doubled and tripled for each viewer.

For a single user you will need a 120Hz display which are the standard now and for four users you need 480Hz displays.

SeeReal’s initial designs with manufacturers in Asia will be able to serve four to five viewers in the first generation of panels. Of course for PC monitors you never need more than one or two so the gaming world will be one of the first industries to benefit from Holographic 3D.

So the key here is the eye tracking technology that SeeReal already had which then allows for less computing power to serve up full resolution holography to four or five viewers. A typical Hologram is about 3-4mm across.

Comments

[H. John Caulfield]

"John Kennedy was a friend of mine. And you, Senator, are no John Kennedy." That is the way I feel about this and other articles on live 3D TV. I have been deeply involved in holography for over 50 years. I was privileged to know Dennis Gabor (but not well) and was a good friend of both Yuri Denisyuk and Emmett Leith. I know a little about the field. And most of what your article describes is not holography at all. And, sadly from my viewpoint, real holography at TV bandwidths just is not in the cards. In a photographically made hologram each square millimeter contains the amount of information in a frame of HD FV. And TV sized displays must have roughly a million of the HD TV sized bursts of information every 1/20 of a second or so. That is hard to imagine being digitized, transmitted, detected, and assembled onto a physical device that is our hologram. That is not going to happen. And it is not necessary anyway, as this article shows.

That 3D is moving out of the labs and into public and private uses thrills me. I am umreservativey
delighted to see the widespread pursuit of 3D displays. Many great scientists (many of whom I regard as friends as well as wonderful scientists) are involved, and the results are wonderful. But calling stereo displays "holograms" does not help the public understand either holography or stereo.

I hope at least some companies, some technical writers, and even some of the publicists for some wonderful scientists will cease trying to glamorize stereo by calling it holographic. Moreover, the real autosterographic 3D that is proliferating is already glamorous. It does not need to be mislabeled. But it is not holography.

Thank you for indulging this fan and sometimes contributor to both fields. I do not think this note will change anything, but I hope some reader will take seriously what I am saying and stop confusing the public.

H. John Caulfield
Diversified Research Corporation.

[HD Magazine]

Dear John,

Thank you for your interest in the article and for your comments. It is always difficult to create comprehensive coverage of a complex subject within the space limitations of most publications. So please accept our apologies if there was ambiguous or incomplete information.

Nonetheless I would like to clarify that SeeReal’s solution is in fact based on holography and not mere auto-stereoscopy. You will agree that both have some overlap because as humans we require more than one perspective to see 3D. You get this with two eyes or through motion parallax when moving one eye. Stereo uses the parallax between views to create a virtual 3D experience but – as unfortunately not all people know – this is not real 3D since the views are created by flat images. In standard stereo these are separated for the left and right eye by glasses (shutter or polarised) and in auto-stereo optics inside the display replace the glasses. So any stereo is in fact 2 times 2D, which is not really 3D, as you correctly point out.

The simplification above makes it easier to compare stereo to holography. For a human observer, also in holography you only need two perspectives because that’s all your sensory system can process. Only the information passing your eye pupils is used. Even if a hologram may generate more than this information at a time, it does not make a difference to the observer – it does to the imaging system though but let’s deal with this later. We also agree that a hologram can generate (or reconstruct) the visual information an identical natural 3D scene would. So holographically reconstructed 3D information to two human eyes would be 2 times 3D, i.e. real 3D or natural 3D. THAT is the key difference. So far so good.

Coming back to the fact that a hologram can reconstruct natural 3D information in certain viewing regions, I believe we would also agree that it is just a matter of hologram resolution (or speaking in terms of display people “pixel resolution”, or maybe in terms of emulsion film “grain resolution”) how large this viewing region is. It can be described by the simple grating equation where the diffraction angle (usable as maximum viewing region) of a hologram bearing medium is dependent on resolution and wavelength of light. Since in – what we at SeeReal call – classic holography the objective is reconstructing the 3D objects or scenes, very high resolution hologram media are being used which in turn create decent viewing regions but also result in very high computation requirements. Contrary to the classic approach, the objective and principle of SeeReal’s solution is reconstructing a wavefront at an observer’s eye pupil identical to what the natural 3D object or scene would create. As a result it would be pointless creating this information anywhere else. Accordingly, the hologram resolution can be quite low because the resulting diffraction angle (or viewing region) only needs to be large enough for a single eye (plus some tolerance). And of course the two holographic views need to represent the proper 3D scene geometry.

Trying to avoid diving too much into tech-talk, I would still point out that of course your hologram bearing media will affect the type of hologram generation and encoding. So a standard amplitude LC display would be not really suited for a high quality holographic 3D reconstruction. A phase modulating LCD, which is almost identical in terms of manufacturing but just not common yet, is much more appropriate and can do that job very well. Taking into account also the considerations about resolution requirements from above, it is easily computed that for a typical observer distance to a desktop display or TV display pixel resolutions available today suffice for serving individual eyes. We call these small viewing regions (virtual) “Viewing Windows”. Combining Viewing Windows with eye tracking you ensure that proper holographically reconstructed 3D information is provided to mobile observers’ eyes.

So as a conclusion you may say that SeeReal’s solution does not create the classic large viewing regions based on super resolution holograms but only cut-outs or segments of these just sufficiently large for individual eye pupils (Viewing Windows). But by tracking observers’ eyes and moving the Viewing Windows accordingly, you can achieve similar or larger viewing regions (field of view). All of this with greatly reduced requirements on hologram resolution and computation. The visible 3D scene resolution can be Full HD or higher as required because this information is easily conveyed by the hologram.

I hope my additional remarks helped to supplement the article for a more detailed view at the subject.

There is one more thing I would like to add and I could not agree more with you: It is a pity how often – and sometimes how deliberate – the term “holography” is (ab)used when in fact talking about stereo or even 2D in-air projections (remember CNN). So once more, I am glad you are raising the subject and I hope all publications as the article above and all discussions as ours contribute to educate a few more people at a time.

Best wishes,
Hagen Stolle
SeeReal Technologies

[Ted Stirling]

Nice, informative article; however, every reference I've seen regarding Holographic-3D technology referes to it as 3DH-technology, not H3D-technology. This is a minor point of contention to be sure, but the author should use the correct acronym; if anything, for accuracy's sake.

TJ Stirling
SJIIE Group

[John Chambers]

Well said Mr Caulfield!

I think you may trump me on the experience and knowledge front, but I can at least claim I produced my first hologram in my bedroom when I was but a lad almost 35 years ago now and I did study the science and practicalities for a few years in the late 1970's and 80's.

From what I can see, holography hasn't really changed much since it was first invented and despite the Russians (and perhaps some others) demonstrating very crude holographic 'movies' (true holograms, not multiplexed), there has never been holographic movies, TVs or 'projections' demonstrated by anyone because the basic physics of producing a real-time moving holographic image is simply impossible to do today as John has explained very fluently.

I, too, cringe when I see and hear the traditional methods of creating a stereoscopic or pseudo-3D image being described as "holography". It seems any method from lenticular prints to the 'Pepper's ghost' effect is explained as being a 'hologram'. I blame most of it on R2D2...

I appreciate that SeeReal are trying to make a name for themselves and get everyone talking about their technology, but it's just misleading to call it 'holographic" in my opinion and it's going to detract from the achievement when, at some point in the future, somebody does demonstrate true 'holographic TV'. I am sure it will happen, but I'm not sure I'll be around to see it.

Not wanting to come across as a pedantic old codger, but it does seem as though these days, with the abundance of 'High Definition'-this and 'Digital'-that, the marketing machines in all technology companies get carried away using terms which sounds sexy to the customers/investors and help sell their products, but often don't bear relation to the actual capabilities and specification of the products themselves.

Despite what I've said, I'm very much looking forward to seeing SeeReal's technology if and when it does get off the ground as, whatever you may want to call it, it is another step in the right direction and sounds like a very interesting development in itself.

[Hagen Stolle]

You may find this interesting as additional information:
http://news.bbc.co.uk/2/hi/programmes/click_online/9393762.stm

Hagen Stolle
SeeReal Technologies