The model, called ACCLIP, outperformed the best existing systems in structural accuracy on three international benchmarks, including real urban traffic scenes, people in motion and pedestrians, while keeping predictions coherent up to 40 frames into the future.

The PhD in Computational Biology graduate from Universidad San Sebastián carried out the work as her doctoral thesis, with funding from the US Air Force Office of Scientific Research.

An artificial intelligence model developed by Chilean physicist Soraya Mora predicts the movement of objects in video up to nearly a minute into the future, with greater coherence than the best existing systems on three international benchmarks. The system, whose main component is optical flow, takes as its reference the mechanism the retina uses to track moving objects: instead of processing the whole scene, it identifies points of interest and calculates how they shift between frames.

“Optical flow looks for points of interest such as the edges of an object and measures how far they moved from one image to the next. The retina does not see the whole object because that would be too much information; what it does is detect points of interest and trace their movement over time,” explained the postdoctoral researcher at Centro Ciencia & Vida, of Fundación Ciencia & Vida and Universidad San Sebastián (FCV-USS).

AI solutions that process video face a limit: convolutional architectures – neural networks that learn to recognise patterns in images by passing filters over them, much as our eye detects edges, shapes and textures before identifying a whole object – predict movement well over short horizons but lose coherence as the number of frames grows.

Recurrent models improve temporal consistency, but they are computationally expensive and struggle to adapt to complex scenarios. Existing optical-flow methods did not solve the problem either: each time horizon required a new full calculation of the flow, with no guarantee of long-term stability or low training cost. None of these approaches combined long-term coherence with computational efficiency.

The results are part of the doctoral thesis of the researcher, a physicist from Pontificia Universidad Católica de Valparaíso, originally from Punta Arenas. The project was supervised by Drs. Tomás Pérez-Acle and César Ravello, of the Bioinspired Artificial Intelligence Laboratory at Centro Ciencia & Vida, and was funded by a FONDECYT Exploración grant, USS and the US Air Force Office of Scientific Research.

Three models, three brain levels

To find where to incorporate optical flow within the neural network architecture, Dr. Mora built three different models, each inspired by a different level of the hierarchy of the visual nervous system. The first, Pre-Encoder (PE), takes the flow as input data, equivalent to processing it in the retina. The second, Post-Latent Space (PLS), tried to calculate it in the network’s compressed internal representation. The third, ACCLIP, incorporated it as a regulator of learning at the signal-integration stage, a level similar to the brain’s association cortex.

The logic behind testing all three paths was systematic: if the place where movement information is introduced determines the quality of the prediction, the three levels had to be compared under equal conditions. “I moved the optical flow to different levels analogous to the nervous system – first analogous to the retina, then the visual cortex, then the association cortex – to see which produced the best predictions. The one that worked best was ACCLIP,” the researcher said.

The PLS model calculated movement within the network’s compressed internal representation, a kind of visual summary used to process the image. The problem is that compression loses detail: the edges of objects blur and small displacements disappear. Without that information, the system could not learn to predict anything stable.

The PE model did work, but only in the short term. When the prediction goes beyond three frames, the system loses coherence: the movement information it receives as input is not enough to sustain the trajectory over time. Its displacement error reached 3.2 pixels per frame, equivalent to losing sight of an object that moves only a few millimetres on screen.

ACCLIP overcame this limit with a different logic: instead of being told how objects move, the system learns by being penalised every time its predictions ignore movement. It does not have more information; it learns to pay attention to it. With this mechanism it reduced the error to 1.4 pixels and 1.1° of angular deviation, less than half that of the PE model, the specialist summarised.

Accurate movement, loss of shape

ACCLIP does not process video like a conventional camera: instead of analysing each whole frame, it combines three types of information at once. First, the direction and speed at which objects move. Second, the changes in brightness from one frame to the next, which vary when objects cast shadows as they move. Third, keeping the object’s shape recognisable throughout the sequence. This combination is what allows it to sustain coherent predictions without large volumes of training data, unlike the models most used in the field.

The results confirm the measurements. In videos of people in motion, ACCLIP reached a structural similarity index of 0.967 for short predictions and 0.974 for long ones, outperforming the field’s three reference models and on pedestrian sequences. Its advantage grew the further ahead the prediction went.

The system does, however, have a clear limit: it predicts movement well but loses fidelity in the object’s shape as the sequence lengthens. The main cause is shadows. Optical flow works by detecting points of interest in the image, such as edges and contours; in dark areas those points disappear and the system is left without information. The blotches created by that gap grow frame by frame and distort the predicted figure. In urban traffic scenes, the model keeps acceptable quality up to 18 frames; after that, the image visibly degrades.

That is the central technical problem Mora plans to solve in the next stage of her work. "If I can improve how I calculate optical flow in dark areas, the prediction will be accurate in both movement and image, over long horizons," she says. The second obstacle is computational cost: the hardware needed to run the model is still considerable, which limits its use outside laboratory settings and is a precondition for real-time prediction.

From the laboratory to the autonomous vehicle

Current robotic vision systems detect objects in real time but do not anticipate trajectories. When an object moves at high speed, the reaction margin can be too short. Anticipatory movement prediction would cover that interval, reducing the margin of error in devices operating in fast, dynamic environments.

Autonomous driving is the most immediate application, but the principle extends to any device that moves independently: drones, industrial robots or mobile surveillance systems. They all face the same problem: the need to project the movement of what is in front of them in order to make decisions with time to spare.

The laboratory already has a prototype that puts this into practice. The team is integrating the model into a spider robot designed by engineer Leonardo Campos, also a member of the group. “We have a spider robot in the lab. We are integrating the model so that, as it moves, it can avoid objects by anticipating their future movements,” says the graduate of the USS PhD programme in Computational Biology.

In the long term, the team plans to apply the same principle to drone swarms that coordinate group movement through each drone’s prediction of its peers’ movement. Before getting there, the model must reduce its processing requirements and solve the shadow problem, two technical conditions that mark the distance between the current prototype and a real implementation.

Dr. Soraya Mora places the work in the realm of basic science – laying foundations that other researchers and developers can turn into applications – and frames her vision in evolutionary terms: the closer a robot is to how biology works, the better it will be able to operate in real environments.

Text by Luis Francisco Sandoval, S&M Comunicaciones agency.