ELLIOT

European Large Open Multi-Modal Foundation Models for Robust Generalization on Arbitrary Data Streams

ELLIOT will develop the next generation of open Multimodal Generalist Foundation Models (MGFMs) capable of learning across a wide variety of data types — text, images, video, sensor data, satellite feeds — and transferring that knowledge to diverse downstream tasks. ELLIOT aims to produce models, datasets, and training pipelines that are fully open-source and reproducible, allowing researchers across Europe to study, adapt, and build upon them.

ELLIOT will act as a catalyst for open-source and open science in AI and substantially strengthen Europe’s Sovereign AI vision. Moreover, ELLIOT’s results are foreseen to empower new breakthrough applications and give Europe a leading edge in the domains of media, Earth modelling, robotic perception, autonomous driving, computer engineering and workflow automation.

The project brings together a world-class consortium of leading European academic and industrial research labs rooted in the European Laboratory for Learning and Intelligent Systems (ELLIS) and Large-scale Artificial Intelligence Open Network (LAION) communities. CSC, along with BSC, CINECA and FZJ, will provide the necessary support to the consortium to secure access to appropriate HPC resources.

CSC will be involved in implementing data handling methods and open pipelines to create generic multimodal datasets for the pre-training of MGFM families. CSC will also contribute toward approaches to perform distributed training at various scales, including at a scale requiring access to thousands of GPUs.

Furthermore, CSC will be involved in exploring model architectures and pre-training procedures that improve core capabilities of the resulting models, such as generalisation and basic reasoning. Additionally, CSC will support the development of methods for reducing model size, inference cost and energy consumption when fine-tuning for a target downstream task.