Eyes, Irises, and VFX
So a few weeks ago a VFX Supervisor DM’d me and asked if I knew anything about eyes. This is for you fam.
A Breakdown of the Human Eye
The human processes information in a very unique way. Let’s start with the way light passes through it.
Human eyes aren’t a complete circle, they have a slight bulge at the front to let light in. Light first passes through the cornea. Then it travels through the pupil, next the iris, the lens, the retina, then finally the photoreceptors turn the light into electrical signals. These electrical signals then travel out through the optic nerve into the brain. Then the brain turns the signals into images you can process.
Cornea: This is the clear front layer of the eye. It is shaped like a concave dome that helps the eye focus.
Iris: The iris controls the amount of light that enters the eye by opening and closing the pupil. It uses muscles to change the size of the pupil. These muscles control the amount of light entering the eye.
Pupil: This is an opening that lets light pass to the back of your eyeball.
Lens: This is an inner clear part of the eye. It also helps the cornea to focus light into the retina. The lens is behind the iris and is usually clear. Light passes through the pupil to the lens. The lens is held in place by small tissues strands or fibers extending from the inner wall of the eye. The lens is very elastic. It’s muscles can change its shape. This allows the eye to focus on different distances. Tightening of these muscles causes the eye to focus on near or far objects.
Retina: This tissue is very sensitive to light, and contains cells called photoreceptors. The retina has two layers. The sensory retina. This contains nerve cells that process visual information. The Retinal Pigment Epithelium(RPE). It lies between the sensory retina and the wall of the eye.The macula near the center of the retina provides sharp detailed vision.
Optic Nerve: Carries electrical signals to the brain. Nerve fibres in the retina merge to form the optic nerve. The optic nerve leads to the brain. Blood vessels called the retinal artery and vein travel with the optic nerve to the back of the eye.
The eye overall has three main layers:
The Outer Layer: This layer is tough, white, and has an opaque membrane. This membrane is called the sclera. The bulge in the sclera is where the cornea is related.
The Middle Layer: The middle layer is called the choroid. The front of the choroid is where the color of your eye is. In the center of it is where the opening of the pupil is. It also contains blood vessels that supply the retina with nutrients and oxygen.
The Inner Layer: This is where the retina is located. It lines the back two thirds of the eyeball.
The inside of the eye is divided into three sections called chambers:
Anterior Chamber: This is the front part of the eye between the cornea and the iris.
Posterior Chamber: This chamber is between the iris and the lens.
Vitreous Chamber: This chamber is between the lens and the back of the eye. The back two thirds of the chamber is lined with a special layer of cells. These cells make up the retina.
Fluid fills up most of the eye. But most of this fluid is located in the anterior and posterior chambers. This clear thick liquid is called vitreous humor or vitreous gel. This gel helps the eyeball keep its shape.
Inside the retina there are millions of light sensitive cells called rods and cones.
Rods: These are used by the eye for monochromatic vision, and to make the eye function in poor light conditions. They cannot distinguish colors, and contain pigment, rhodopsin (visual purple). Rods are distributed through the retina unequally. Rod density is greater in the peripheral retina however.
Cones: White cones are used for color and focusing on fine details. Cones are located in an area of the retina called the fovea. They require bright light to function. In humans there are three types of cones. They are often referred to as red, green, and blue cones. However, they are each designed to handle long-wavelength, medium-wavelength, and short-wavelength light. Cones are most often located near the fovea. Cones and rods are connected through intermediate cells when your eyes look at something directly.
Most organisms on Earth can distinguish color. Color vision is considered the ability to distinguish lights at different spectral qualities. All organisms with vision are restricted to a small range on the electromagnetic spectrum. This range is usually between wavelengths of 400-700 nm. There so far has been no scientific evidence to prove that an organism has gone beyond this range.
Certain pigments in the eyes such as rhodopsin pick up light on the electromagnetic spectrum around 500 nm. Small changes to the genes in this pigment can change how it handles light.
Many organisms are unable to process color with their eyes. So they only see in grey. Animals that can see color , can detect ultraviolet light. This energy can be damaging to receptor cells. Depending on the animal. Snakes are one type of creature that are not affected by ultraviolet light.
Eyes get their color from the amount of melanin that is in them. If you have brown eyes, your eyes contain a fair amount of melanin. If you have blue eyes you don’t have large amounts of melanin in your eyes. Instead your eyes get their color the same way as the water and the sky do. The water particles inside them reflect blue light.
Almost everyone’s eyes contain brown pigment in it. The true color of your eye depends on the iris. The front layer of the iris, called the stroma, can make your eyes appear brown, blue, or green. People with blue eyes, because they have no pigment in this layer, absorb longer wavelengths of light that come in. Because of this, more light reflects back and appears blue.
People with green and hazel eyes have one or both parts of the iris layers with brown pigment in. This pigment then interacts with the blue light and makes it appear darker.
You can’t predict the eye color of a child by the parents eye color. Parents with the same colored eyes can have a child with different colored irises. Scientists have guessed that blue eyes are a recent evolution. Perhaps within the last 6,000-10,000 years. Before that, everyone had brown eyes.
Some eyes change color in different lighting because of the type of light entering it. Depending on the lighting conditions green and hazel eyes can appear to shift in appearance.
The eye color of babies usually changes as they grow up. When they are born they don’t have a lot of pigment in their irises and it slowly develops overtime.
96% of all animals possess a complex optical system. Some animals in the world have something called simple eyes. Such as pit eyes, which are eye-spots that are set into a pit to reduce the angles of light that enter and affect the eye.
Complex eyes contain retinal photosensitive ganglion cells. These cells send signals along the retinohypothalamic tract to control pupillary light reflexes.
Some animals need eyes that can handle large amounts of detail and visual angles. Visual acuity is controlled by cone cells. It is often measured in cycles per degree (CPD). This measures angular resolution. This resolution in CPD can be measured by bar charts with white and black stripe cycles.
Human eyes max out around 50 CPD for resolution. A rat has about 1 to 2 CPD.
Some animals such as crustaceans or insects, have eyes consisting of an array of numerous small visual units. Rather, eyes within eyes. The resolution for these eyes can be hard to measure. But they generally have a lower acuity than other vertebrate eyes and mammals. More on this later.
Some other animals such as snakes have eyes called Pit Eyes. Pit eyes are also known as stemma. These are eye spots that are set into a pit to reduce the angles of light into the crevice. These eyes are thought to be the precursor to simple eyes. Some animals with these eyes have reduced the size of their aperture by incorporating reflective layers behind their receptor cells. Pit vipers also have eyes that sense thermal infra-red radiation. They also have optical wavelength eyes.
Pit Eyes can also contain spherical lens eyes. These eyes have higher refractive indexes in their lens. This in turn reduces blur and increases resolution in the eyes. It also decreases the focal length of the eyes, and more light to enter. These can be seen in gastropods.
Some eyes in animals have multiple lenses in their eyes. Most of the time this happens in marine animals. For example, the coppod patella has three. In its eyes, it’s outermost lens is parabolic in nature, while the others fit smoothly behind it. However, other copepods, such as the copilia, have two lenses arranged like a telescope. Multiple lenses have also been seen in eagles and jumping spiders.
In combination with these eye types that we have discussed. Some contain refractive corneas. Mammals , birds, reptiles, and other animals with this trait have vitreous fluid in their eyes with a higher refractive index than the air. They usually also have spherical lenses in their eyes that produce spherical aberrations. When these two character traits come together the lens is held together with an inhomogeneous lens material. This causes a wide field of view to develop in the animal's sight. Refractive corneas are only useful outside water. This is because the refractive index of water is often close to or the same as the vitreous fluid in the eye. So it causes the animal’s sight to diminish.
Instead of having refractive liquid behind the lens of the eye, some animals have reflective eyes. This means instead of having a lens inside their eye they have a series of mirrors to reflect the images to a central point. This means if we were to look into those animals' eyes, we would see our own reflection. Some animals that have these eyes are rotifers, copepods and flatworms. But because they are so tiny, their eyes can’t really reflect anything through them. This makes their eyesight very poor. However, some larger animals such as scallops that use reflector eyes, line their shell with them. This is so it can detect moving objects as they pass around it.
Now, back to compound eyes. Arthropods such as the bluebottle fly have compound eyes. Compound eyes have thousands of receptors to capture an image. These receptors are usually located on convex surfaces, and this points them in different directions. This gives the creature a very large view angle. So they can detect very fast movement, and the polarisation of light.
There are two main groups of compound eyes. Apposition and Superposition eyes. Apposition eyes form multiple inverted images. Superposition eyes form one large image.
Apposition eyes are the most common type of eye. They are found in all arthropod groups, and horseshoe crabs also have them. They work by gathering a number of images, eye from each sensor or eye.
Superposition eyes are harder to narrow down. They are divided into three different types:
Reflecting and Refracting: This eye has a gap in between its lens and rhabdom. It has no side walls to hold everything together. It takes light in, and then reflects it to the side angle on the other side of the eye. It is normally found in nocturnal insects. It can create images 1000 times brighter than apposition eyes.
Parabolic Superposition: This eye refracts light and uses a parabolic mirror to focus the image. It combines features from both superposition and apposition eyes.
Non compound eyes are the opposite of compound eyes. These are very simple lens eyes that can be found in vertebrates, cephalopods, annelids, crustaceans and cubozoa. More on this later.
There are smaller categories of compound eyes. One of them being the Strepsipteran compound eye. This eye is made up of a bunch of simple eyes clustered together. Because each eye is a simple eye, each one forms an inverted image. These images are then combined in the brain to form one big image. Honey bees and other insects share these eyes.
Some insects have a single lens compound eye. This is a mix between a superposition and a multi-lensed compound eye.
The pigmentation of animal eyes is always hard to understand. But the process is usually similar to how our eyes get their color. Pigment molecules will vary from species to species. It can also be used to define the distance between different animal groups, and when they broke away from each other.
Opsins are pigments involved in photoreception. Melatonin pigments are usually used by the eye as a shield to block photoreceptor cells from light.
Opsin comes in two different forms., and has two different purposes when it comes to vision. C-opsins are associated with ciliary photoreceptor cells. R-opsins are associated with rhabdomeric photoreceptor cells. Ciliary photoreceptors are UV-sensitive and mediate sight in non-directional UV light. Eyes with rhabdomeric photoreceptor cells are designed to avoid UV light. The cells do this by determining the incoming ratio of blue to UV light, and then tell the creature what to stare at.
Vertebrates usually have ciliary cells with c-opsins in them. Invertebrates have rhabdomeric with r-opsins. However, different invertebrate species can also have c-opsins in their eyes.
Now let’s talk about some specific eye types per animal.
Eagles are known for their extreme sense of vision. They have excellent long distance vision, and can see eight times as far what a human can. Eagles can also quickly shift focus in their eyes, and create “zooms” to see camouflage animals. They can also see a wider range of colors than humans, such as colors in the UV spectrum. However, their vision fails them during the dark as they don’t have good night vision.
These birds are basically night eagles. They have great night vision. Their eyes don’t sit at an angle in their face, so their head is forced to rotate, and face directly forward. This makes their vision a bit binocular. Their eyes can’t move, or roll around in their sockets either. So the full rotation of an owl’s head is locked in at 270 degrees.
Owl eyes contain more rods than humans, and this makes them more light sensitive. Their irises can widen to make sure all light comes into contact with their retinas. Because their irises can adjust, they are not limited to only seeing in the dark. They can also see during the day, but their vision is blurry and they can’t see colors well.
Owls and other night animals have reflective surfaces behind their retinas known as the tapetum lucidum. This layer allows light to reflect back into the animal's eye even if it has already passed through. This can give the animal a second chance to see movement it might have missed.
These animals are considered to have one of the most complex eyes in the animal kingdom. Humans usually have three types of cones in our eyes. These cones allow us to see the color spectrum from red to violet. Mantis shrimp have 16 different types of cones. We still don’t know to this day what a mantis shrimp's full range of color is. As most of the purposes of these cones, have not been identified. But we do know that they have an advanced color recognition system.
Their eyes can move independently of each other, and in no way does this compromise their vision. Their color receptors allow them to pick up small changes in color. This helps them evade predators.
Sheep and Goats:
These animals have some very unique eyes. Their pupils are shaped as horizontal lines. Pupil shape helps the animal customize it’s sight for a specific environment. Daytime hunting animals usually have forward facing eyes with round pupils. Small animals that hunt during the day and night usually have vertical slit pupils to help them with depth perception.
Meanwhile, goats and sheep have pupils in the shape of lines because they graze out in the open and need to be always alerted of predators. Their wide and narrow pupils give them a wider field of vision when they lower their heads to eat.
Some general animal eye facts:
Tarsiers have the largest eyes in the animal kingdom relative to their body size.
Trilobites were one of the first animals to develop complex, compound eyes.
Some species of dragonfly can have more than 28,000 lenes per compound eye. This is more than any other living creature. They also have an almost complete 360 degree vision as well.
Eye Tracking and Visualization
Eye tracking refers to the process of measuring where we look, also known as our point of gaze. Or the motion of the eye relative to the head. In research, this tracking technique is incredibly valuable. Some of the benefits of eye tracking are
- Revealing subconscious behavior: We can get insight into behaviors we carry out instinctively.
- Provide unbiased data for studies: it removes the need to try and remember or explain where you looked.
- Allows for natural movement of the eyes
- It can be used in almost any environment and setting.
- Provides a high level of detail when providing results
- Offers real-time information
- Offers a visual representation of heat maps, gaze plots, and other graphs
- Can be added to other biometric data
It is used in many different machine learning algorithms, cloud computing, internet usage and other pieces of software. Let's go over some eye tracking algorithms, tools, and applications.
Everyone’s field of vision is different. It would make sense to assume that everyone’s vision and cognitive processing power is the same. But that is not the case. This means that everyone views every scenario differently, and they feel different things regarding it. Each person has their own unique cognitive style and speed. Everyone will approach a visualization differently and can find it interesting or boring. As well as a verbal to visual information memory. Testing how to see how our brain and visual system independence works is critically important on how we can diagnose problems with it. So eye tracking is a common practice to use with this topic.
The research into eye tracking has increased over the past couple of years. SImply because it can help improve basic tasks, and solve many different ones at once. When it comes down to machine learning (ML) and cloud computing can help evolve eye tracking technology. ML evolves from tracking patterns and existing data. So when it is combined with eye tracking, it can get the track to manually re-calibrate itself, and make better decisions. There are many studies that prove that ML eye tracking is more accurate and has better detection results than a traditional eye track.
Eye tracking uses two techniques called gaze distribution and scanpaths. Gaze distribution enables scientists to measure how long an observer examines areas within a visual stimulus. A scanpath is an ordered set of fixation points, depicted by circles, and connected by saccades. These are depicted by lines to represent the eye.
Scientists can use these gaze distributions and scampaths to better understand the psychology and cognitive ability of people when they view certain scenarios and situations.
Scientists are also using eye tracking studies to understand how to better represent scientific data to the public. By tracking how part of a lesson, image, statistic, or piece of data jumps out at the viewer, they can better arrange their conclusions. This can prevent miscommunication from occurring.
Information and Scientific Visualizations are some of the most powerful tools to amplify how humans interpret data. So displaying them correctly is very important.
The science of eye tracking is not new. Professor Emile Javal (May 5, 1839 – January 20, 1907) was one of the first researchers to realize that eye movements were not one swift motion. Rather, they were a series of rapid blinks and movements. As well as that they were not continuous, and smoothly transitioned to the next focal point.
The first individual who developed a technology based eye tracker was Gus Thomas Buswell in 1920. He recorded beams of light reflected off the eye, and from there he could triangulate where the eye was focusing.
Currently, in the modern day world, the most accurate eye trackers use infrared light to identify the pupil and what it is gazing at. There are also wearable eye tracking systems that people can use as well. Google famously tried to showcase this in their Google Glass.
Eyes and Machine Learning
Let’s talk a little more about how our eyes can be used in machine learning software.
Scientists are starting to not only use machine learning for learning to track the human eye, but also help diagnose eye diseases better. There have been several clinical ophthalmology studies where scientists have given datasets of eye diseases to machine learning software. The AI was then able to analyze the digital data it was given, and identify features in ocular diseases. As well as understand how the disease had progressed, and side effects from it.
Artificial intelligence and machine learning has its challenges in the medical field. Especially when it comes to eyes. Not a lot of eye doctors are trained to work with AI, or use computers beyond their current medical equipment. So training doctors to work with these up and coming tools is a bit of a hassle.
However, when used correctly, it can help screen people for diabetic retina based diseases, and much more. It can also process medical results faster.
Google is trying to conquer this issue by developing an AI to have a better “eyesight” than your doctor. As well as use diagnostic tools as the ones mentioned above for eye diseases. So far their ophthalmologist AI has a 97% correct diagnosis rate for sight related problems. Which was a higher average when they compared their results to in person doctors. They suspect the AI has learned to recognize patterns in the human eye that we cannot personally see.
They also discovered through the AI that there is a difference between male and female eyes. The AI was able to tell the difference between a male and female eye through the amount of blood vessels, and shapes in the sections of the eye.
[ If you are wondering the difference. Male eyes are more sensitive to moving objects and small details, whereas female eyes are more in tune to color changes. Eye diseases affect women more than men. Mainly because females live long on average, we touch our faces more frequently, use toxic makeup products, and suffer from seasonal allergies more than men.]
Eyeballs in Houdini
Unfortunately, there aren’t a lot of examples of irises , or eyeballs in Houdini. There are a few interesting tutorials. But nothing too fancy. There are some great eye rigging tutorials as well. Here are some that you can check out:
- [Houdini Tutorial] 0014 Iris Network (Fast ver.):
Junichiro Horikawa: https://www.youtube.com/watch?v=KJ1Valozj8g
- KineFX Rigging | Fur Dude | Part 14 | Eye Controls: https://www.youtube.com/watch?v=EwO56bfVavM
- Realistic Eye Shader. Houdini 16.5: https://vimeo.com/253524581
Let’s talk about how you could possibly build a realistic eye in Houdini.
How you model an eye really matters. The more detail you add into the model, the more reflective it will be, and the more overlap of color you will have. Eye models can quickly become very heavy. So don’t over model an eye if you don’t have to.
You can create intricate eye strands in the eye by simulating clumped lines or hairs. When they clump together you can then create another dimension of layered detail around the pupil.
Lighting and rendering an eye can also be tricky. You don’t want black reflections or artifacts to appear in a render. Double check for ray bias, and the minimum distances between the reflective surfaces. The default ray bias in Mantra is 0.001 , which is way too big to capture the refractions in an eyeball.
There are some artists however, who take eyeballs in Houdini a bit further. Joe Lundy is one of these artists, and he has developed procedural eye generators for the software.
Blender, and Eyes in Film
Let’s talk about some eye examples outside of Houdini.
Chris Jones is an artist who has been able to think outside the box, and create some very realistic looking eyes. Using Lightwave, Sculptris, and Krita, he was able to generate some very realistic looking results: https://www.youtube.com/watch?v=TAZIvyAJfeM
Now let's take a look at some wonderful tips , tricks and examples from VFX studios.
Pixar has a wonderful example of a 3D scanned character called Lousie. It perfectly showcases how lighting and rendering can be done on photorealistic models. It is also constructed with its eyeballs in different pieces. It’s irises, retina, and lens are separate objects.
They show how Lousie’s eyes and other aspects can be rendered and shaded. They mainly use a Pixar Layer Surface Shader for the body, and a subsurface base, and glass layer for the eyes. .
Lousie’s eyes are also manually adjusted to have the white color look realistic, as a default white color value can not be physically accurate.
Rendering the eyes was done with Renderman, and with a Manifold Next Event Estimation (MNEE) or Manifold Walk. This is a technique that can render caustics through very small and thin surfaces.
The eye meniscus is very small and this is very hard to render without shadow bias artifacts. So turning shadows off helps reduce this problem. Slight bump mapping can also help add further detail, and break up specular values.
Disney research teams have also found interesting aspects of ophthalmology while making high quality captures of eyes. They wanted to reproduce all the inner parts of the eye in a 3D model , such as the white sclera, the transparent cornea, and the iris. By scanning eyes they hope to impact the way eyes are modeled in the future.
Weta Workshop has also been up to some fun. Tor Robinson, an artist at Weta 3D printed some very realistic eyes for their creature department. She used a procedural Houdini system to translate 3D models in both volumetric and in mesh form into images slices for printing. Then she was able to create a cross section of an eye to layer over and over again.