Technical description
Fusion Engine (FE) enables context awareness by combining data readings to achieve situation awareness. FE has two primary functionalities:
i) creating fusion models and pipelines to provide spatiotemporal fusion capabilities of the data, and
ii) executing multiple workflows dynamically and in parallel as user requests arrive.
The fusion functionalities operate as “black boxes” for users, translating into chains of several algorithms. Each pipeline is developed and tested in Jupyter notebooks and implemented as Kubeflow pipelines. Kubeflow builds on Kubernetes, offering a system for deploying, scaling, and managing complex systems compatible with EO4EU installations. Fusion pipelines have access to user data stored in S3 buckets, as shown in Figure 1. Configuration details and S3 bucket access credentials are communicated via a Kafka message bus.
Figure 1: FE lifecycle
Fusion Pipeline Accessibility and Orchestration
Fusion pipelines are published in the AI/Marketplace to be accessible to users via the Workflow Editor (WFE). Each pipeline is listed with an icon, name, input/output requests, and a description of its functionality.
A Fusion Proxy acts as the main orchestrator, handling multiple parallel requests to FE from users, as illustrated in Figure 2. When a new workflow is created by the Workflow Editor, a request is sent to the Fusion Proxy to create the specific workflow in the Kubeflow environment. This process involves:
- Creating a consumer to trigger execution
- Initialising the namespace
- Running the pipeline
- Publishing results in the data repository
- Notifying subsequent components via Kafka
All computations will be considered for execution using High Performance Computing (HPC) or GPU environments to maintain high productivity rates.
Figure 2: Fusion proxy and parallel processing
Background
Spatiotemporal fusion techniques have gained significant interest in recent decades. The combination of Earth Observation (EO) data with multiple sensor sources—both mobile and fixed—maximizes the quality of information delivered to the user through several interpolated layers.
Research indicates that integrating spatial, spectral, and temporal attributes leads to more accurate and stable predictions, enhancing the final output. Fusion can be applied locally using weighted functions and pixel windows, or globally through optimization approaches.
Spatiotemporal fusion can be performed at different processing levels depending on tools and applications. For example, per-pixel operations are suited for images from the same sensor system with similar calibration, while feature- or decision-level fusion handles heterogeneous data such as combining elevation data (e.g., LiDAR) with satellite imagery.
Fusion levels include:
i) Pixel-level image fusion
ii) Feature-level image fusion
iii) Decision-level image fusion
Technologies/Frameworks used
- Kubeflow
- Jupyter notebooks
- Python libraries
- Docker
- YAML
Input
Dataset repository/ S3 bucket
Output
- CSV, Shapefiles, tiff, tar.xz
- Kafka message