Time-of-flight camera
A time-of-flight camera, also known as time-of-flight sensor, is a range imaging camera system for measuring distances between the camera and the subject for each point of the image based on time-of-flight, the round trip time of an artificial light signal, as provided by a laser or an LED. Laser-based time-of-flight cameras are part of a broader class of scannerless LIDAR, in which the entire scene is captured with each laser pulse, as opposed to point-by-point with a laser beam such as in scanning LIDAR systems.
Time-of-flight camera products for civil applications began to emerge around 2000, as the semiconductor processes allowed the production of components fast enough for such devices. The systems cover ranges of a few centimeters up to several kilometers.
Types of devices
Several different technologies for time-of-flight cameras have been developed.RF-modulated light sources with phase detectors
Photonic Mixer Devices, the Swiss Ranger, and CanestaVision work by modulating the outgoing beam with an RF carrier, then measuring the phase shift of that carrier on the receiver side. This approach has a modular error challenge: measured ranges are modulo the RF carrier wavelength. The Swiss Ranger is a compact, short-range device, with ranges of 5 or 10 meters and a resolution of 176 x 144 pixels. With phase unwrapping algorithms, the maximum uniqueness range can be increased. The PMD can provide ranges up to 60 m. Illumination is pulsed LEDs rather than a laser. More recent CW-ToF camera systems illuminate the scene with high-frequency modulated LED light and analyze the phase shift of the returning signal at each pixel to compute depth. For example, in traffic enforcement applications, retroreflective surfaces such as license plates and vehicle reflectors generate strong return signals that are used to construct depth images over time. These images allow tracking of vehicle positions in 3D space and calculation of speed by applying regression analysis to the position-time data. Unlike conventional RADAR, this method measures speed along the vehicle's true direction of travel and is independent of the vehicle’s distance and angle relative to the camera. In some continuous-wave ToF systems, depth images captured over successive time intervals are used to estimate the 3D positions of moving objects, such as vehicles. The system tracks multiple retroreflective points across consecutive frames and reconstructs the object’s trajectory through 3D space. By applying regression analysis to the change in position over time, the system accurately determines the object's speed along its path of travel. Unlike conventional RADAR, this approach minimizes errors associated with distance and angle to the target. CanestaVision developer Canesta was purchased by Microsoft in 2010. The Kinect2 for Xbox One was based on ToF technology from Canesta.Range gated imagers
These devices have a built-in shutter in the image sensor that opens and closes at the same rate as the light pulses are sent out. Most time-of-flight 3D sensors are based on this principle invented by Medina. Because part of every returning pulse is blocked by the shutter according to its time of arrival, the amount of light received relates to the distance the pulse has traveled.The distance can be calculated using the equation, z = R / 2 + R / 2 for an ideal camera. R is the camera range, determined by the round trip of the light pulse, S1 the amount of the light pulse that is received, and S2 the amount of the light pulse that is blocked.
The ZCam by 3DV Systems is a range-gated system. Microsoft purchased 3DV in 2009. Microsoft's second-generation Kinect sensor was developed using knowledge gained from Canesta and 3DV Systems.
Similar principles are used in the ToF camera line developed by the Fraunhofer Institute of Microelectronic Circuits and Systems and TriDiCam. These cameras employ photodetectors with a fast electronic shutter.
The depth resolution of ToF cameras can be improved with ultra-fast gating intensified CCD cameras. These cameras provide gating times down to 200ps and enable ToF setup with sub-millimeter depth resolution.
Range gated imagers can also be used in 2D imaging to suppress anything outside a specified distance range, such as to see through fog. A pulsed laser provides illumination, and an optical gate allows light to reach the imager only during the desired time period.
Direct Time-of-Flight imagers
These devices measure the direct time-of-flight required for a single laser pulse to leave the camera and reflect back onto the focal plane array. Also known as "trigger mode", the 3D images captured using this methodology image complete spatial and temporal data, recording full 3D scenes with single laser pulse. This allows rapid acquisition and rapid real-time processing of scene information. For time-sensitive autonomous operations, this approach has been demonstrated for autonomous space testing and operation such as used on the OSIRIS-REx Bennu asteroid sample and return mission and autonomous helicopter landing.Advanced Scientific Concepts, Inc. provides application specific Direct TOF vision systems known as 3D Flash LIDAR cameras. Their approach utilizes InGaAs Avalanche Photo Diode or PIN photodetector arrays capable of imaging laser pulse in the 980 nm to 1600 nm wavelengths.
Components
A time-of-flight camera consists of the following components:- Illumination unit: It illuminates the scene. For RF-modulated light sources with phase detector imagers, the light has to be modulated with high speeds up to 100 MHz, only LEDs or laser diodes are feasible. For Direct TOF imagers, a single pulse per frame is used. The illumination normally uses infrared light to make the illumination unobtrusive.
- Optics: A lens gathers the reflected light and images the environment onto the image sensor. An optical band-pass filter only passes the light with the same wavelength as the illumination unit. This helps suppress non-pertinent light and reduce noise.
- Image sensor: This is the heart of the TOF camera. Each pixel measures the time the light has taken to travel from the illumination unit to the object and back to the focal plane array. Several different approaches are used for timing; see Types of devices above.
- Driver electronics: Both the illumination unit and the image sensor have to be controlled by high speed signals and synchronized. These signals have to be very accurate to obtain a high resolution. For example, if the signals between the illumination unit and the sensor shift by only 10 picoseconds, the distance changes by 1.5 mm. For comparison: current CPUs reach frequencies of up to 3 GHz, corresponding to clock cycles of about 300 ps - the corresponding 'resolution' is only 45 mm.
- Computation/Interface: The distance is calculated directly in the camera. To obtain good performance, some calibration data is also used. The camera then provides a distance image over some interface, for example USB or Ethernet.
Principle
For amplitude modulated arrays, the pulse width of the illumination determines the maximum range the camera can handle. With a pulse width of e.g. 50 ns, the range is limited to
These short times show that the illumination unit is a critical part of the system. Only with special LEDs or lasers is it possible to generate such short pulses.
The single pixel consists of a photo sensitive element. It converts the incoming light into a current. In analog timing imagers, connected to the photo diode are fast switches, which direct the current to one of two memory elements that act as summation elements. In digital timing imagers, a time counter, that can be running at several gigahertz, is connected to each photodetector pixel and stops counting when light is sensed.
In the diagram of an amplitude modulated array analog timer, the pixel uses two switches and two memory elements. The switches are controlled by a pulse with the same length as the light pulse, where the control signal of switch G2 is delayed by exactly the pulse width. Depending on the delay, only part of the light pulse is sampled through G1 in S1, the other part is stored in S2. Depending on the distance, the ratio between S1 and S2 changes as depicted in the drawing. Because only small amounts of light hit the sensor within 50 ns, not only one but several thousand pulses are sent out and gathered, thus increasing the signal-to-noise ratio.
After the exposure, the pixel is read out and the following stages measure the signals S1 and S2. As the length of the light pulse is defined, the distance can be calculated with the formula:
In the example, the signals have the following values: S1 = 0.66 and S2 = 0.33. The distance is therefore:
In the presence of background light the memory elements receive an additional part of the signal. This would disturb the distance measurement. To eliminate the background part of the signal, the whole measurement can be performed a second time with the illumination switched off. If the objects are further away than the distance range, the result is also wrong. Here, a second measurement with the control signals delayed by an additional pulse width helps to suppress such objects.
Other systems work with a sinusoidally modulated light source instead of the pulse source.
For direct TOF imagers, such as 3D Flash LIDAR, a single short pulse from 5 to 10 ns is emitted by the laser. The T-zero event is established by capturing the pulse directly and routing this timing onto the focal plane array. T-zero is used to compare the return time of the returning reflected pulse on the various pixels of the focal plane array. By comparing T-zero and the captured returned pulse and comparing the time difference, each pixel accurately outputs a direct time-of-flight measurement. The round trip of a single pulse for 100 meters is 660 ns. With a 10 ns pulse, the scene is illuminated and the range and intensity captured in less than 1 microsecond.