Skip to Content

Computer Vision for UAV. Guidance, Control, Cooperation, and Inspection

Funded by the Spanish Ministry of Science MICYT #DPI2007-66156


The main idea of the project is to introduce the Computer Vision on board of UAV in order to widely improve its performances in its different tasks and situations: navigation and maneuvering using visual information, tracking of static and dynamic targets while moving in changing environments, cooperation of several UAV to perform a common task by merging different visual information, and performing trajectory planning and identification for visual inspection.

This project is the natural extension of previous Cicyt projects developed by the proponent Research Group, two of them regarding Vision on UAV and another one regarding multi-robot cooperation. Previous goals achieved in these projects include: power line tracking, distance and height estimation, 2D visual definable target tracking, control architecture with visual information in the loop, 2D visual control of UAV position, dynamic trajectory planning and cooperative modeling of environment applied to robot guidance.

Based on these previous results the present project aims following challenging technical objectives:

a) 3D tracking of planar and non-planar targets, presently being tackle by word top level institutions, and that present special difficulties in UAV due to vibrations, changing environments and scaling while moving,

b) sensor fusion of visual and “classical” positioning sensors, where the difficulty lays on the unbalanced amount of data, that in the visual source case has to be previously reduced extracting the significant information, which we proposed to be done by learning techniques that extract the manifold in the image high dimensional space where the sequential images of the target lay, and

c) vision based multi AUV cooperation, performed by at least two UAV that supply different visual information regarding view point and different visual equipment (e.g. resolution, focal length, thermography) that are to be considered for a common due task.

The achievement of mentioned technical objectives are going to be implemented and used in the prototypes and demos proposed in the project, that include several special navigation tasks where vision can play an important roll, such as landing and take-off, horizontal tracking, terrain following and dynamic target tracking.

Finally it is aimed that the obtained results can be successfully transfer to the industry, which already participate as EPOs in this project. One company is a high technology company focused on UAV that is very interested in our results in Computer Vision. The other two EPOs are important and big companies whose R&D departments are very much interested in applications of UAV guided by vision for their special applications.



The central objective of the project is to yield significant improvements in the tasks a Vision System can develop onboard an UAV. These improvements are clustered and summarized as follows:

·    Tracking and servoing of planar and non-planar 3D targets.

There are three main objectives in this area:

o      3D tracking of planar targets. This objective includes not only the localization of a target in the image plane (2 coordinates and 1 angle) as is achieved now, but also the inclusion of the remaining third coordinate and two angles. This will be achieved by increasing dimensionality of the movement model, including affine and projective parameters.

o      Robustness of visual tracking algorithms to be applied in several UAV applications. This robustness is focused on: quick changing lighting conditions, high frequency mechanical vibrations in image acquisition and projective and scale changes during tracking or inspection tasks. Our proposal to solve these robustness problems are based on two approaches:

a) Increasing of the motion parameters in a Lucas-Kanade based algorithm, whose complexity can be balanced by a reduction in the number of parameters that represent the target (dimensionality reduction using for example NN

b) Multiresolution approach, that can be exploit for both reducing the effects of vibration and for increasing the robustness under scaling changes.

o      3D tracking of non-planar targets. The main problem to be solved in non-planar 3D tracking arises by the vanishing cue points, that is also reinforced by the special visual acquisition features mentioned in the first objective. Our approach is based on multimodal fusion of visual cues and also on the robustness coped in the first objective. Multimodal fusion achieves tracking robustness by reducing the likelihood that all visual features fail simultaneously, and allows the tracking algorithm to adapt quickly to current viewing conditions. Some tasks to be carried out by the UAVs require tracking of non-planar (solid and non-flexible objects) 3D targets, where individual cues can only provide reliable tracking under limited conditions, that make them unsuitable in an unpredictable visual setting, since some failure modes will inevitably be encountered at some time during the tracking.


·    Sensor fusion with visual information for positioning, flying and tracking.

A central aim of the proposal is to fully integrate the visual information with other navigation sensors (e.g. inertial and GPS) in order to obtain a coherent and robust estimation of the position of the UAV related to its close environment (e.g. targets), which can be used in navigation and cooperative tasks. This sensor fusion will be based on EKF, where the visual information is to be included together with information from other sensors. One main challenge is the unbalanced dimensionality of both sources of data. Our approach will be focussed on dimensionality reduction of the visual information, using learning techniques, that generates the manifold in the image space (dimesionality of the number of pixels) where the images of a specific task lays. Afterwards the coordinates of each image on the obtained manifold are the input data for EKF.


·    Vision based multi UAV cooperation

The objective is to develop cooperative computer vision tools which one will been based in the information coming from the different UAV’s, and complex tasks will be carried out. This objective is divided in the following objective parts:

o      Inspection of the tasks realized by an UAV by means of another UAV.

o      Cooperative modeling and reconstruction considering the information supplied by two UAV in order to improve the precision.

o      Cooperative modeling and reconstruction considering the information supplied by two UAV in dynamical environments.


·    Visual based navigation in special situations  (e.g. landing, take-off, hovering, horizontal tracking, terrain following, …)

This objective is aimed to perform several specific navigation and inspection tasks carried out by one or two cooperative UAVs, where the Vision System plays an important roll. In order to achieve this goal, each specific task will use the results obtained from several of the above mentioned objectives, which are going to be put together in specific demonstrative navigation tasks.  These tasks are: horizontal tracking, leveling, terrain following, features detection, landing and take-off.


Starting date: 
Finishing date: