A method performed by an electronic device is described. The method includes obtaining a first image from a first camera, the first camera having a first focal length and a first field of view. The method also includes obtaining a second image from a second camera, the second camera having a second focal length and a second field of view disposed within the first field of view. The method further includes aligning at least a portion of the first image and at least a portion of the second image to produce aligned images. The method additionally includes fusing the aligned images based on a diffusion kernel to produce a fused image. The diffusion kernel indicates a threshold level over a gray level range. The method also includes outputting the fused image. The method may be performed for each of a plurality of frames of a video feed.

Certain aspects relate to systems and techniques for folded optic stereoscopic imaging, wherein a number of folded optic paths each direct a different one of a corresponding number of stereoscopic images toward a portion of a single image sensor. Each folded optic path can include a set of optics including a first light folding surface positioned to receive light propagating from a scene along a first optical axis and redirect the light along a second optical axis, a second light folding surface positioned to redirect the light from the second optical axis to a third optical axis, and lens elements positioned along at least the first and second optical axes and including a first subset having telescopic optical characteristics and a second subset lengthening the optical path length. The sensor can be a three-dimensionally stacked backside illuminated sensor wafer and reconfigurable instruction cell array processing wafer that performs depth processing.

System and methods for performing motion sensitive high dynamic range (HDR) image processing. A saturation analysis circuit is configured to receive a set of image data corresponding to portions of a set of image frames having different exposures time from a lowest exposure time to a highest exposure time, and select image data from a frame that does not exceed the saturation threshold value. A motion detection circuit may be configured to determine whether the image data is associated with movement, by comparing image data from pairs of frames of adjacent exposure times, and changing the selection to a lower exposure time frame if movement is detected. By selecting which exposure time is used based upon movement in the image frame, ghosting and blurring in HDR images containing movement can be reduced.

Various embodiments are directed to an image sensor that includes a first sensor portion and a second sensor portion coupled to the first sensor portion. The second sensor portion may be positioned relative to the first sensor portion so that the second sensor portion may initially detect light entering the image sensor, and some of that light passes through the second sensor portion and is be detected by the first sensor portion. In some embodiments, the second sensor portion may be configured to have a thickness suitable for sensing visible light. The first sensor portion may be configured to have a thickness suitable for sensing IR or NIR light. As a result of the arrangement and structure of the second sensor portion and the first sensor portion, the image sensor captures substantially more light from the light source.

Devices and methods for providing seamless preview images for multi-camera devices having two or more asymmetric cameras. A multi-camera device may include two asymmetric cameras disposed to image a target scene. The multi-camera device further includes a processor coupled to a memory component and a display, the processor configured to retrieve an image generated by a first camera from the memory component, retrieve an image generated by a second camera from the memory component, receive input corresponding to a preview zoom level, retrieve spatial transform information and photometric transform information from memory, modify at least one image received from the first and second cameras by the spatial transform and the photometric transform, and provide on the display a preview image comprising at least a portion of the at least one modified image and a portion of either the first image or the second image based on the preview zoom level.

Exemplary embodiments are directed to configurable demodulation of image data produced by an image sensor. In some aspects, a method includes receiving information indicating a configuration of the image sensor. In some aspects, the information may indicate a configuration of sensor elements and/or corresponding color filters for the sensor elements. A modulation function may then be generated based on the information. In some aspects, the method also includes demodulating the image data based on the generated modulation function to determine chrominance and luminance components of the image data, and generating the second image based on the determined chrominance and luminance components.

The present application relates generally to a generation of a depth map of a field of view (FOV) and, more specifically, to systems, methods, and devices for automatically generating a fused or aggregate depth map of the FOV that is configured to compensate for weaknesses that may be introduced by individually generated depth maps.

Exemplary embodiments are directed to configurable demodulation of image data produced by an image sensor. In some aspects, a method includes receiving information indicating a configuration of the image sensor. In some aspects, the information may indicate a configuration of sensor elements and/or corresponding color filters for the sensor elements. A modulation function may then be generated based on the information. In some aspects, the method also includes demodulating the image data based on the generated modulation function to determine chrominance and luminance components of the image data, and generating the second image based on the determined chrominance and luminance components.

One innovation includes an IR sensor having an array of sensor pixels to convert light into current, each sensor pixel of the array including a photodetector region, a lens configured to focus light into the photodetector region, the lens adjacent to the photodetector region so light propagates through the lens and into the photodetector region, and a substrate disposed with photodetector region between the substrate and the lens, the substrate having one or more transistors formed therein. The sensor also includes reflective structures positioned between at least a portion of the substrate and at least a portion of the photodetector region and such that at least a portion of the photodetector region is between the one or more reflective structures and the lens, the one or more reflective structures configured to reflect the light that has passed through at least a portion of the photodetector region into the photodetector region.

Systems and methods for error correction in structured light are disclosed. In one aspect, a method includes receiving, via a receiver sensor, a structured light image of at least a portion of a composite code mask encoding a plurality of codewords, the image including an invalid codeword. The method further includes detecting the invalid codeword. The method further includes generating a plurality of candidate codewords based on the invalid codeword. The method further includes selecting one of the plurality of candidate codewords to replace the invalid codeword. The method further includes generating a depth map for an image of the scene based on the selected candidate codeword. The method further includes generating a digital representation of a scene based on the depth map. The method further includes outputting the digital representation of the scene to an output device.

Aspects relate to methods and systems for producing ultra-wide field of view images. In some embodiments, an image capture system for capturing wide field-of-view images comprises an aperture, a central camera positioned to receive light through the aperture, the center camera having an optical axis, a plurality of periphery cameras disposed beside the central camera and pointed towards a portion of the optical axis of the center camera, the plurality of cameras arranged around the center camera, and a plurality of extendible reflectors. The reflectors are configured to move from a first position to a second position and have a mirrored first surface that faces away from the optical axis of the center camera and a second black surface that faces towards the optical axis of the center camera, the plurality of periphery cameras arranged around the center camera.

A detection device depth (100a-d) to capture an image containing depth information of a scene, comprising: a transmitter (200) capable of projecting light onto a scene, the transmitter comprising a laser (210) capable of producing a light beam (230) including a series of laser pulses, each pulse having a pulse length and the number of pulses produced in a pulse frequency, and a receiver (300) coupled to the transmitter (200) in an orientation relative known, the receiver comprising a shutter and a set of sensors (315) capable of producing an image based on the detection light projected by the transmitter (200) and reflected from the scene; and a controller comprising a processor (350), the controller coupled to the transmitter (200) and receiver (300), the controller configured to: determine, based on a first image receiver (300), the indicative light at least one depth detection device (100a-d) is present in the scene; in response to determining that the light is present in the scene based on the first image: i) control the pulse length of the series of laser pulses to avoid interfering with the light that is determined to be present in the scene, ii ) adjusting an exposure period of the shutter to synchronize with the length of controlled pulse, iii) determine, based on a next image receiver (300), if the light is still present in the scene, iv) in response to determining that the light is still present in the scene, delaying the onset of exposure window and continue to determine, based on a next image of the plurality of images of the receiver, if the light is still present in the scene; and in response to determining that the light is not present in the scene, starting the exposure window, with the exposure window comprising producing a laser pulse of the series of laser pulses to the length of controlled pulse and activate the shutter for the exposure period so that the receiver (300) detecting the laser pulse reflected from the scene.

Structured light active sensing systems transmit and receive spatial codes to generate depth maps. Spatial codes can’t be repeated within a disparity range if they are to be uniquely identified. This results in large numbers of codes for single transmitter/single receiver systems, because reflected ray traces from two object locations may be focused onto the same location of the receiver sensor, making it impossible to determine which object location reflected the code. However, the original code location may be uniquely identified because ray traces from the two object locations that focus onto the same location of the first receiver sensor may focus onto different locations on the second receiver sensor. Described herein are active sensing systems and methods that use two receivers to uniquely identify original code positions and allow for greater code reuse.

Aspects relate to an array camera exhibiting little or no parallax artifacts in captured images. For example, the planes of the central mirror surfaces of the array camera can be located at a midpoint along, and orthogonally to, a line between the corresponding camera location and the virtual camera location. Accordingly, the cones of all of the cameras in the array appear as if coming from the virtual camera location after folding by the mirrors. Each sensor in the array “sees” a portion of the image scene using a corresponding facet of the central mirror prism, and accordingly each individual sensor/mirror pair represents only a sub-aperture of the total array camera. The complete array camera has a synthetic aperture generated based on the sum of all individual aperture rays.

Aspects relate to an array camera exhibiting little or no parallax artifacts in captured images. For example, the planes of the central mirror prism of the array camera can intersect at an apex defining the vertical axis of symmetry of the system. The apex can serve as a point of intersection for the optical axes of the sensors in the array. Each sensor in the array “sees” a portion of the image scene using a corresponding facet of the central mirror prism, and accordingly each individual sensor/mirror pair represents only a sub-aperture of the total array camera. The complete array camera has a synthetic aperture generated based on the sum of all individual aperture rays.

Certain aspects relate to imaging systems and methods for manufacturing imaging systems and image sensors. The imaging system includes a pixel array including a plurality of pixels, the pixels configured to generate a charge when exposed to light and disposed on a first layer. The imaging system further includes a plurality of pixel circuits for reading light integrated in the pixels coupled thereto, each of the plurality of pixel circuits comprising one or more transistors shared between a subset of the plurality of the pixels, the one or more transistors disposed on a second layer different than the first layer. The imaging system further includes a plurality of floating diffusion nodes configured to couple each of the plurality of pixels to the plurality of pixel circuits.

Certain aspects relate to systems and techniques for full well capacity extension. For example, a storage capacitor included in the pixel readout architecture can enable multiple charge dumps from a pixel in the analog domain, extending the full well capacity of the pixel. Further, multiple reads can be integrated in the digital domain using a memory, for example DRAM, in communication with the pixel readout architecture. This also can effectively multiply a small pixel’s full well capacity. In some examples, multiple reads in the digital domain can be used to reduce, eliminate, or compensate for kTC noise in the pixel readout architecture.

Various features pertain to active depth sensing and more specifically to techniques to compensate for different reflectivity/absorption coefficients of objects in a scene when performing active depth sensing system using structured light.

Described herein are methods and devices that employ a plurality of image sensors to capture a target image of a scene. As described, positioning at least one reflective or refractive surface near the plurality of image sensors enables the sensors to capture together an image of wider field of view and longer focal length than any sensor could capture individually by using the reflective or refractive surface to guide a portion of the image scene to each sensor. The different portions of the scene captured by the sensors may overlap, and may be aligned and cropped to generate the target image.

A method operational on a transmitter device is provided for projecting a composite code mask. A composite code mask on a tangible medium is obtained, where the composite code mask includes a code layer combined with a carrier layer. The code layer may include uniquely identifiable spatially-coded codewords defined by a plurality of symbols. The carrier layer may be independently ascertainable and distinct from the code layer and includes a plurality of reference objects that are robust to distortion upon projection. At least one of the code layer and carrier layer may be pre-shaped by a synthetic point spread function prior to projection. At least a portion of the composite code mask is projected, by the transmitter device, onto a target object to help a receiver ascertain depth information for the target object with a single projection of the composite code mask.

Described are methods and apparatus for adjusting images of a stereoscopic image pair. The methods and apparatus may capture a first and second image with first and second imaging sensors. The two imaging sensors have intrinsic and extrinsic parameters. A normalized focal distance of a reference imaging sensor may also be determined based on intrinsic and extrinsic parameters. A calibration matrix is then adjusted based on the normalized focal distance. The calibration matrix may be applied to an image captured by an image sensor.

Described are methods and apparatus for adjusting images of a stereoscopic image pair. The methods and apparatus may capture a first and second image with first and second imaging sensors. The two imaging sensors have intrinsic and extrinsic parameters. A normalized focal distance of a reference imaging sensor may also be determined based on intrinsic and extrinsic parameters. A calibration matrix is then adjusted based on the normalized focal distance. The calibration matrix may be applied to an image captured by an image sensor.

Described herein are methods, systems and apparatus to improve imaging sensor production yields. In one method, a stereoscopic image sensor pair is provided from a manufacturing line. One or more images of a correction pattern are captured by the image sensor pair. Correction angles of the sensor pair are determined based on the images of the correction pattern. The correction angles of the sensor pair are represented graphically in a three dimensional space. Analysis of the graphical representation of the correction angles through statistical processing results in a set of production correction parameters that may be input into a manufacturing line to improve sensor pair yields.

Described herein are methods and apparatus to adjust the convergence point of a stereoscopic image pair captured by an imaging device. In one method, a first image and a second image of a stereoscopic image pair are provided, and then shifting or cropping of the first image is performed to align the first and second image. This shifting or cropping is performed while preserving the second image. The method then includes determining a target horizontal image disparity based on a desired convergence point of the stereoscopic image pair and when the target horizontal disparity is greater than a predetermined maximum, the cropping of an outside dimension of the first image is limited. In some implementations it is limited to the predetermined maximum.

Present embodiments contemplate systems, apparatus, and methods to determine an appropriate focal depth for a sensor based upon a pair of stereoscopic images. Particularly, certain of the embodiments contemplate determining keypoints for each image, identifying correlations between the keypoints, and deriving object distances from the correlations. These distances may then be used to select a proper focal depth for one or more sensors.

A method of combining data from multiple sensors is disclosed. The method includes providing a common control signal to multiple image sensors. Each of the multiple image sensors is responsive to the common control signal to generate image data. The method also includes receiving synchronized data output from each of the multiple image sensors.

Systems and methods of 3D image processing are disclosed. In a particular embodiment, a three-dimensional (3D) media player is configured to receive input data including at least a first image corresponding to a scene and a second image corresponding to the scene and to provide output data to a 3D display device. The 3D media player is responsive to user input including at least one of a zoom command and a pan command. The 3D media player includes a convergence control module configured to determine a convergence point of a 3D rendering of the scene responsive to the user input.

An electronic device for parallel image processing using multiple processors is disclosed. The electronic device includes multiple image sensors for providing image data. The electronic device also includes multiple processors for processing segmented image data to produce processed segmented image data. Each processor is dedicated to one of the image sensors. A multiple processor interface is also included. The multiple processor interface maps the image data to the processors, segments the image data to produce the segmented image data and synchronizes the segmented image data to processor clock rates.

Chroma values in image data may be sub-sampled, thereby obtaining sub-sampled chroma values. The sub-sampled chroma values may be compressed, thereby obtaining compressed, sub-sampled chroma values. Luma values in the image data may be compressed, thereby obtaining sub-sampled luma values. Edge information for the luma values that are discarded as part of the luma sub-sampling operation may be determined.

A graphics multi-media integrated circuit (GMIC) is connected to a host processor over two serial links: a half duplex bi-directional serial link which accords to a protocol defined for a display serial interface, and a uni-directional serial link which accords to a compatible protocol defined for a camera serial interface. The GMIC receives packets according to the protocol from the host over the half duplex bi-directional serial link and processes these packets. The GMIC sends packets according to the protocol to the host over the uni-directional serial link. A packet from the host can request a processing operation by the GMIC or can initiate a memory operation at the memory of the GMIC. The GMIC can also send packets to the host to initiate a memory operation at the memory of the host. The GMIC may be connected to a display over a bi-directional serial link according to the display serial interface protocol and to a camera over a uni-directional serial link and a bi-directional control link according to the camera serial interface so that the host controls the display and camera indirectly through the GMIC.

A buffer is associated with each of a plurality of data lanes of a multi-lane serial data bus. Data words are timed through the buffers of active ones of the data lanes. Words timed through buffers of active data lanes are merged onto a parallel bus such that data words from each of the active data lanes are merged onto the parallel bus in a pre-defined repeating sequence of data lanes. This approach allows other, non-active, data lanes to remain in a power conservation state.

A technique for processing at least one bad pixel occurring in an image sensing system is provided. Dynamic bad pixel detection is performed on a plurality of streaming pixels taking from at least one controlled image and value and coordinate information for each bad pixel is subsequently stored as stored bad pixel information. Thereafter, static bad pixel correction may be performed based on the stored bad pixel information. The stored bad pixel information may be verified based on histogram analysis performed on the plurality of streaming pixels. The technique for processing bad pixels in accordance with the present invention may be embodied in suitable circuitry or, more broadly, within devices incorporating image sensing systems.

At least one illuminant white point estimate is determined in a color space having radially defined saturation based on a reference image. A chromatic adaptation correction vector (CACV) is determined based on the at least one illuminant white point estimate. Corrected pixels are obtained by applying the CACV (preferably in a cone response color space using a correction matrix based on the CACV) to uncorrected image pixels corresponding to a target image, which may comprise the reference image or another image.

To derive a Hamming code to manage data errors a set of at least four parity bit positions is selected for parity bits which will protect a set of data bits (where each data bit has a data bit position in the data bit set). A syndrome is determined for each data bit position. This involves selecting a unique sub-set of at least three parity bit positions. The unique sub-set shares at least one parity bit position with at least one other unique sub-set of at least three parity bit positions. A parity bit value may then be calculated for each parity bit position based on the determined syndromes. The header of a packet may be provided with a word which defines the length of the packet and an error management code generated utilizing this word so that errors in the word may be detected and, possibly, corrected.

A method of operating a shared bus comprises sending a wake-up signal on the shared bus. The wake-up signal comprises a sequence of signals, each signal of the sequence being one of a signal indicating the bus is free and a signal indicating the bus is busy. A microcontroller to effect this method is also contemplated. A wake-up device for a shared bus has a first latch to recognize a signal indicating one of said shared bus being free and said shared bus being busy and selectively output a recognition signal and a second latch to, after receipt of the recognition signal, recognize a signal indicating another of the shared bus being free and the shared bus being busy and selectively output a power-on signal. The latches may be D-type flip flops.