(This page first appeared in 2003)

The name Centeye has long been associated with optical flow sensors, in particular extremely light and fast ones that may be integrated with flying robotic platforms e.g. “drones”. Although we do use a broad range of techniques in our vision sensors, optical flow is generally the one feature almost all of them incorporate.

Optic flow defined

The term “optic flow” refers to a visual phenomena that you experience every day. Essentially, optic flow is the apparent visual motion that you experience as you move through the world. Suppose you sitting in a car or a train, and are looking out the window. You see trees, the ground, buildings, etc., appear to move backwards. This motion is optic flow. This motion can also tell you how close you are to the different objects you see. Distant objects like clouds, and mountains move so slowly they appear still. The objects that are closer, such as buildings and trees, appear to move backwards, with the closer objects moving faster than the distant objects. Very close objects, such as grass or small signs by the road, move so fast they whiz right by you.

There are clear mathematical relationships between the magnitude of the optic flow and where the object is in relation to you. If you double the speed which you travel, the optic flow you see will also double. If an object is brought twice as close to you, the optic flow will again double. Also the optic flow will vary depending on the angle between your direction of travel and the direction of the object you are looking at. Suppose you are traveling forward. The optic flow is the fastest when the object is to your side by 90 degrees, or directly above or below you. If the object is brought closer to the forward or backward direction, the optic flow will be less. An object directly in front of you will have no optic flow, and appear to stand still. (However, because the edges of that forward object are not directly ahead of you, these edges will appear to move, and the object will appear to get larger. This is discussed more below.

Optic flow as seen from a bird, insect, or aircraft

The figure above shows what the optic flow might look like from an aircraft flying over a rocky desert (or over Mars!). The blue arrows show the optic flow that would be seen by a camera or a passenger on the aircraft. Looking downward, there is a strong optic flow pattern due to the ground and rocks on the ground. The optic flow is fastest directly below the aircraft. It is especially fast where the tall rock protrudes from the ground. A sensor on the aircraft that responds to optic flow would be able to see this optic flow pattern and recognize the presence of the tall rock. The meaning is clear: “Look out below!!!”

Looking forward, there is another optic flow pattern due to the upcoming rock and anything else the aircraft might be approaching. The blue circle directly at the center shows the “focus of expansion” or FOE. The FOE tells the aircraft the specific direction it is flying. (Remember above we said that if you are travelling in a straight line, the optic flow is zero in the directly forward direction.) The aircraft sees a large optic flow to the right of the FOE, which is due to the large rock on the left-hand side of this picture. The aircraft also sees smaller optic flow patterns in the downward-front direction, due to the ground. Towards it’s upper left, it sees no optic flow because this region of the visual field only has the sky. The forward optic flow pattern reveals that the aircraft will fly close by the big rock, perhaps dangerously close. If the optic flow on the aircraft’s right grows larger, then the aircraft should take that as a hint to turn away…

Here are some other examples of optic flow: The figure below shows an overhead view of a dragonfly, and the resulting optic flows in the sideways direction. (We are ignoring the vertical direction in this figure). The figure on the left shows the dragon fly traveling forward. The optic flow travels from the forward to backward direction, and is generally faster on the left and right than in the front or back. The figure on the left shows a dragon fly rotating to the right in one spot. Here the optic flow is to the left in all directions. If the dragonfly were flying a curved path, the optic flow patterns would be a combination of these two patterns. What we see here is that in addition to detecting obstacles, the optic flow can be used to measure or estimate one’s own motion.

The figure below shows the same type of patters, this time from the point of view of the dragon fly, and with the 360 degree field of view flattened onto the screen. When the dragon fly is rotating, or yawing, to the right, the optic flow everywhere will be to the left. However in the directions along the axis of rotation, the optic flow will be zero. When traveling forward, the optic flow will diverge from the forward direction, flow backwards, and converge in the rear.

How can sensing optic flow be useful when flying through the world?

There is quite a bit of scientific research that has been performed on how animals and insects use optic flow to navigate through the world. There is certainly too much to summarize on one web page! However let’s consider a few more examples… Hovering: Suppose you want to hover in one place. The best way to do this is to try to keep the optic flow zero everywhere, as shown in the figure below. Note that this method will only work if the rest of the world is still. If the world is moving, for example if you are a small insect flying in between branches that are swaying in the wind, then you will end up swaying with the branches (not necessarily a bad thing).

Avoiding obstacles: The last thing any flying animal or aircraft wants to do is to crash into an obstacle. Below are some examples of a dragonfly detecting and avoiding a large obstacle by sensing the optic flow. On the left, the dragonfly is dangerously close to the rock. The large optic flow on the left warns the dragonfly, and it turns away to avoid a collision. On the right, the dragonfly senses it is flying towards a rock. The clue here is that the focus of expansion (FOE) is inside the rock, and the optic flow is expanding rapidly. Many flying insects will actually make rapid turns when changing directions. These rapid turns are called “saccades”, after the rapid eye motion that your eye makes as you shift gaze from one direction to another. Many insects, however, do not have eyes that can move independently of the rest of the body, or can do so with limited freedom. As a result, if the insect wants to look “facing” in a new direction, the whole body must be shifted. The resulting rapid turn executed by the insect is often referred to as a saccade.

Speed control through dense environments: In some cases, you may want to change your flight speed so that you fly slower when in a cluttered environment. A simple method of doing this is shown in the figure below. If the optic flow all around you is getting too fast, you simply slow down until the optic flow is at a more “comfortable” level. The dragonfly on the left is traveling between two rocks, and has a sufficient “space cushion” between it and the rocks. The dragon fly on the right is traveling in a tighter space, and therefore must slow down it’s flight speed to keep the optic flow at a more comfortable level.

More examples: We can put up more pictures, or refer you to books or journal articles. However, a great way to learn is through hands-on experience. Next time you walk across the room, down the street, or ride in a car or train, take a moment to look at the optic flow patterns that you see. (Don’t do this while you are driving!) What do you see when you turn? What do you see when you walk down stairs? What do you see as you pass under a tree? You will very quickly see a wide variety of patterns, and get a sense of just how much information you can get from optic flow or visual motion.

Advanced Considerations

There are many algorithms that may be used to measure optical flow. The most famous one is the Lucas-Kanade method, which analyzes spatial and temporal gradients over multiple frames to estimate optical flow. Other methods use block matching or feature tracking to obtain motion measurements. Yet other methods use “elementary motion detector” (EMD) methods inspired from the insect vision systems.

What is important to consider is that there is no one single method that is best for all applications. There are a number of factors you need to consider when utilizing an optical flow based solution:

• Translation and rotation: Optical flow results from both rotation and translation. Translational optical flow tells you about other objects in the environment, but rotational optical flow can dominate. If you are undergoing rapid rotations, then rotational optical flow can wash out measurements of translational optical flow. You need to find a way to remove the rotational component, ideally using a gyro to provide rotational methods. Centeye has developed, and even patented, a number of different techniques to accomplish this.
• Low texture contrast: A common myth is that optical flow techniques do not work when there is no texture. The reality is that with the exception of perfectly clean and smooth surfaces (which are very rare), there is almost always texture that can be tracked. You just need to have a vision system that is sensitive enough to pick these out. In the past, Centeye has used advanced analog processing circuits on our vision chips to enhance texture contrast, to the point that we were able to measure optical flow caused by the motion of a blank, white sheet of paper, and even control the height of an aircraft when flying over snow on a cloudy day.
• Mechanical jitter: When a vision system is mounted on a moving platform, mechanical vibrations can affect the measurement of optical flow. Fortunately we have developed image processing techniques that exploit such mechanical jitter, to allow obstacle detection and even the detection and tracking of tiny targets from a camera in motion.
• Light levels: How much light is there? A common myth is that you need substantial light to support optical flow measurement. This myth is prolonged by the fact that optical mouse sensors require either sunlight or an LED mounted millimeters away from the target system, and many other optical flow solutions using standard cameras require tens or hundreds of lux to work. However are nocturnal insects that can navigate using optical flow in light levels that are perceived as dark by humans. We have demonstrated optical flow based hover in light levels as low as several lux, and are currently implementing vision systems for use in low light environments using techniques inspired by these insects.
• Motion transparency: Optical flow is commonly described as a vector field with a single vector or optical flow measurement for each location. (Abraham Maslow once said “he who is good with a hammer thinks everything is a nail”, so it is inevitable that when the first drawings of optical flow were shown to computer scientists, they immediately thought of a vector field.) The reality is that one location can have multiple optical flows. Pay attention next time you walk past a chain link fence, or drive past a forest in the winter. There will be multiple optical flows happening in each location, and if you can measure these, you will have more knowledge of the environment. Centeye has developed techniques to address motion transparency.