EO4EU Platform: A comprehensive architecture tailored to your goals

The EO4EU Platform deploys a user-friendly interface to help end users find, access, analyse, and visualise Earth Observation data they are interested in. The intuitive EO4EU experience is optimised for environmental, government and business forecast operations, with specific designs, customised with stakeholder goals in mind, to enable increased project workflow and productivity. 

The report, “D2.4 Technical, Operational and Interoperability Specifications and Architecture”, captures the core components of the EO4EU infrastructure design, delivering analytical descriptions of complex architecture tiers and visual sequence diagrams to demonstrate cross-platform synergies, helping users to better comprehend the way that Earth observation (EO) data is collected, processed and analysed.

Five Tiers of EO4EU Architecture

The EO4EU Platform architecture consists of five separate tiers. The tiers can be considered the first outer layer of analysis, with specific structural components existing to underpin each tier and showcase how the system digests, processes and transmits alternate datasets. Descriptions of each tier and their associated components are listed below.

1. The Data Tier

The Data Tier (leftmost part of Fig. 1) is a set of differentiated data sources that are input into the platform for pre-processing. At this tier data sources can be collected in various formats, therefore EO4EU offers a Knowledge Graph, a Fusion endive and a ML component to ensure the data is uniform and consistent. 

  • The Knowledge Graph provides expert and non-expert users with extended search capacity, enabling EO data access based on semantic search.

2. The IaaS Tier

The Infrastructure as a Service (IaaS) Tier (bottom-centre part of Fig. 1) provides essential foundational cloud-computing infrastructure to enable PaaS components to efficiently compute and store networking resources dynamically and programmatically (via Openstack REST APIs). 

OpenStack facilitates open-source cloud computing, providing IaaS capabilities and the foundational infrastructure for building and managing cloud environments 

3. The PaaS Tier

The Platform as a Service (PaaS) Tier (central and top left part of Fig. 1) expands on IaaS support and facilitates additional development, deployment and user management services, offering a higher-level of cloud-computing service compared to IaaS. Components include:

  • Kubernetes Platform Manager - open-source platform for displaying and managing Platform applications
  • Auxiliary/support - GitLab Container Image Registry - allows users to securely store and distribute container images and other Open Container Initiatives (OCI)
  • Auxiliary/support - Configuration Management and Day 2 Operations - Terraform (IaC tool) for creating, updating and destroying cloud resources to achieve desired infrastructure state. Ansible empowers ongoing automation tasks, including maintenance and scaling. 
  • Platform Orchestrator - heart of the platform, existing as monitoring manager, communications manager, predictive allocation, registry handler, and provision manager.
  • Authentication Single Sign On (SSO) - identification and authentication method enabling users to sign in once to the Platform and have access to various applications.
  • Message Bus - powered by Strimzi and operates via a communications manager, providing the means of message dispatch between components ensuring the heterogeneous data systems are reliable, available and consistently supported
  • Domain Specific Language (DSL) Engine - validation and control schema for the System Workflows
  • Fusion Engine - enables context awareness by combining the data readings and leading to situation awareness
  • AI/ML Marketplace - assistive open-source library inside the Dashboard containing all AI/ML algorithms and models

4. The ML Tier

The ML Tier is dedicated to providing all machine learning models in a toolbox for the processing of retrieved and synthesised data. This tier is composed of:

  • Machine Learning Model - mathematical representation or algorithm trained on historical data to make predictions or decisions without being programmed
  • Machine Learning Inference Server - consists of two subcomponents: KServe, a standard Model Inference Platform on Kubernetes, and a second component that interfaces with the EO4EU platform, with a converter from Kafka to HTTP and a webserver code

5. The Front-end Tier

Finally, the Front-end Tier (right-most part of Fig.1) delivers multi-dimensional UI (Web, XR, CLI, API) components facilitating user interaction and platform control. This tier includes:

  • Dashboard - Data Analytics Visualisation - improved learning and evidence-based interpretations of environmental observations designed for decision makers and policymakers
  • EO4EU API - establishes communication with the user and EO4EU components,  while generating smart data interfaces and communications
  • XR System - web-based XR interface for visualising and exploring the EO data in a more immersive way
  • UMM - management of login, SSO, user preferences and user history data
  • WorkFlow Editor - the central point for describing workflows and creating personalised data processing workflows

Technical Requirements Mapping

User responses related to the functional or non-functional components requirements for system integration were gathered and analysed to better inform the development of the Platform and accommodate user demands. These responses are key for the continuous evolution of the EO4EU system, as adjustments will correspond to the priority preferences communicated by end users. 

In report "D2.2 - EO4EU End-user Requirements Analysis & Business Process Flows", a questionnaire to end users was delivered and the results of the requested requirements were consolidated and presented in report D2.4 - Technical, Operational and Interoperability Specifications and Architecture. End users requirements were categorised underneath one of the previously mentioned tiers in the following format:

  • DATA - Data tier 
  • SS - Systems tier combining IaaS and PaaS
  • ML - Machine Learning tier
  • CG - Customer Facing tier
  • GEN - Generic requirements related to infrastructure or data

The exhausted list of requirements requested from end users can be found on page 38 of the deliverable, but below is a snapshot of the high priority requirements for each tier:

  • Data:
    • Users should have the ability to download final and intermediate results of their workflows
    • Platform should provide a user-friendly interface which can be operated in an easy and intuitive way
    • Interface has to allow the user to search for the available datasets
  • Systems tier combining IaaS and PaaS
    • Platform should allow storing data of various formats such as GeoTiff, NetCDF, GRIB etc.
    • EO4EU should include GPU processing capabilities
    • EO4EU should provide an average of 30-50GB RAM for processing purposes of the users 
  • Machine Learning
    • Facilitate the volume reductions of the data that will be transferred
    • An encoder-decoder pair to serve the compression process
    • Inference service should handle load balancing
  • Customer Facing Tier
    • Single-Sign-On
    • Mapping of roles to users and groups
    • Storing of user settings/preferences
  • Generic requirements related to infrastructure or data 
    • Datasets should have global coverage, such as free access to satellite images 
    • Platform should provide access of archie data for long time periods

Sequence Diagrams 

End-to-end workflow demonstrations are provided to highlight the readiness and practical capacity of the EO4EU platform to carry out user tasks and achieve desired objectives. 

The sequence diagrams depict a snapshot of the processing functionality of the platform to respond to user requests for EO resources, automated internal requests from a dedicated API mechanism, the consumption of the data by the users, and how the system supports activities along in their journey. The various use cases described in this section not only showcase the scalable data processing management capabilities and interactive functional design of the Platform, but how users can manipulate specific components to accommodate various workflows and achieve a diverse range of specific EO objectives. 

Conclusion

This report offers a comprehensive overview of the EO4EU platform architecture with expansive detailed technical specifications, demonstrating the diversified capabilities and seamless interactions of the key components, visualising how they coexist to connect users to a transformative and accessible ecosystem. 

Read the full report on Zenodo here. 

blog D2.4