Skip to content

Commit a80581c

Browse files
Update README.md
1 parent 864539c commit a80581c

1 file changed

Lines changed: 32 additions & 16 deletions

File tree

README.md

Lines changed: 32 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,34 @@
11
# Graph-Massivizer Toolkit
2-
The Graph-Massivizer Toolkit is an integrated platform that leverages the unique researched functionalities in each separate Graph-Massivizer tool. In the integrated toolkit, algorithms that perform basic graph operations (BGO) developed by Graph-Inceptor and Graph-Scrutinizer as well as other open source libraries are integrated so that they can be executed efficiently and in a green-aware fashion within diverse hardware environments according to the advanced techniques developed by Graph-Optimizer, Graph-Greenifier, and Graph-Choreographer.
2+
The Graph-Massivizer Toolkit is a loosely integrated toolkit that leverages the unique researched functionalities in each separate Graph-Massivizer tool. In the toolkit, algorithms that perform basic graph operations (BGO) developed by Graph-Inceptor and Graph-Scrutinizer as well as other open source libraries are integrated so that they can be executed efficiently and in a green-aware fashion within diverse hardware environments according to the advanced techniques developed by Graph-Optimizer, Graph-Greenifier, and Graph-Choreographer.
33

4-
The architecture of the Graph-Massivizer distributed graph processing engine is designed for scalable execution across the compute continuum, including cloud, HPC and edge environments - leveraging both CPU and GPU resources. It follows a master--worker paradigm with two main roles: the centralized Workload Manager to coordinate opimization and scheduling, and decentralized Task Managers for executing BGOs. These components, supported by Docker-based container orchestration, monitoring services, and a ZooKeeper-based coordination layer, ensure fault-tolerance and system observability.
4+
The architecture of the Graph-Massivizer distributed graph processing engine is designed for scalable execution across the compute continuum, including cloud, HPC and edge environments - leveraging both CPU and GPU resources.
55

66
![Architecture](https://github.com/graph-massivizer/.github/blob/public-update/figs/overview.png)
77

8+
## Graph-Massivizer Tools
9+
10+
### Graph-Inceptor
11+
The [Graph-Inceptor](https://github.com/graph-massivizer/graph-inceptor) tool is comprised of two distinct tools serving different use cases for ingesting and processing massive graphs.
12+
13+
- [GraphMa](https://github.com/graph-massivizer/graph-inceptor-graphma), a component of the Graph-Inceptor tool, integrates principles of pipeline computation using modular, composable functions to provide structured graph data analysis and processing using computational abstractions such as computation as type, higher-order traversal abstraction, and directed data-transfer protocol.
14+
15+
- The [ETL Pipeline](https://github.com/graph-massivizer/graph-inceptor-etl-pipeline) creates KGs and stores them in batches from large data sources using semantic mappings deployed on a scalable IT cloud infrastructure consisting of servers and storage systems.
16+
17+
### Graph-Scrutinizer
18+
[Graph-Scrutinizer](https://github.com/graph-massivizer/graph-scrutinizer) provides various BGO analytics, such as sampling, summarisation, traversal, or ML (e.g., GNN) algorithms, translated into optimised implementations for heterogeneous hardware (HPC, edge, cloud). Examples of the Graph-Scrutinizer algorithms that can be used in BGOs include [TS2G2](https://github.com/graph-massivizer/ts2g2) and [Go Network](https://github.com/graph-massivizer/go-network).
19+
20+
### Graph-Optimizer
21+
[Graph-Optimizer](https://github.com/graph-massivizer/graph-optimizer) combines analytical models, micro-benchmarking, graph sampling, simulation, and automated validation, to predict the performance and energy footprint of a given graph processing workload.
22+
23+
### Graph-Greenifier
24+
[Graph-Greenifier](https://github.com/graph-massivizer/graph-greenifier) is a simulation tool for data centre operators and application developers to create scenarios that quantify the carbon impact of workloads on different locations and hardware, making informed decisions.
25+
26+
### Graph-Choreographer
27+
[Graph-Choreographer](https://github.com/graph-massivizer/graph-choreographer) is a serverless orchestration tool for executing single, ensemble and batch graph applications on the computing continuum, scheduled using performance and energy tradeoffs.
28+
29+
# Graph-Massivizer Toolkit Simulation
30+
For local testing and development of BGO functionalities, a simulation is provided by the project. This simulation was initially created for validation and testing purposes to ensure compatibility of the separate tool funtionalities. It follows a master--worker paradigm with two main roles: the centralized Workload Manager to coordinate opimization and scheduling, and decentralized Task Managers for executing BGOs. These components, supported by Docker-based container orchestration, monitoring services, and a ZooKeeper-based coordination layer, ensure fault-tolerance and system observability.
31+
832
## Workload Manager
933
The Workload Manager is a centralized component running on the master node, acting as the global orchestrator of graph workflows submitted by users. Upon receiving a workflow (as a DAG), it validates its structure and decomposes it into BGOs, such as filtering, traversal, or PageRank. Its internal modules are:
1034

@@ -28,28 +52,20 @@ Managers are lightweight agents deployed across the computing continuum (cloud s
2852
Each instance of Task Manager registers itself with ZooKeeper and encodes its machine descriptor, allowing the infrastructure manager in the Workload Manager to maintain a live view of available execution resources. BGOs are designed to be stateless and containerized, enabling fault-resilient retries and elastic scaling. The Task Manager also includes demo routines for interacting with HDFS, verifying storage availability, and supporting workload validation during test cycles. The full engine supports both simulation (via lifecycle emulation in Docker) and deployment in production clusters, making it suitable for prototyping, benchmarking, and real-world graph analytics pipelines.
2953

3054
# Requirements
31-
This project is built with the python programming language and uses Docker containers. Both of these must be installed to develop with the toolkit.
55+
This project is built with the python programming language and uses Docker containers. Both of these must be installed to develop with the toolkit. Other packages and dependencies should be installed automatically when building and executing the simulation.
3256

33-
The platform also uses [metaphactory](https://metaphacts.com/) as a frontend. Developers must have a key in order to pull metaphactory Docker images. A key can be obtained by filling out [this form](https://metaphacts.com/get-started#docker-trial) and mentioning the project, which will send you an email in a short time containing the key and login command.
57+
## Using metaphactory
58+
The simulated platform also uses [metaphactory](https://metaphacts.com/) as a default frontend. Developers must have a key in order to pull metaphactory Docker images. A key can be obtained by filling out [this form](https://metaphacts.com/get-started#docker-trial) and mentioning the project, which will send you an email in a short time containing the key and login command.
3459

35-
## Running metaphactory
3660
Before running the project, use the script provided called `./start_metaphactory.sh` to run a metaphactory docker image, and then `./stop_metaphactory.sh` to close it. These images are independent of the project so they should be running in the background during different executions and not reloaded until you want to stop working with them.
3761

3862
# Development
39-
After cloning the project, create a virtual environment to work on this project.
40-
Then, install the dependencies using
63+
After cloning the project, create a virtual environment to work on this project. It is strongly recommended to inoreitialize a virtual environment before installing or building the simulation. For detailed instructions see [the documentation](https://docs.python.org/3/library/venv.html) for how to configure a virtual environment.
4164

42-
```bash
43-
pip install -e .
44-
```
45-
46-
For the state machine visualizations, we make use of pydot, which in turn requires graphviz to be installed natively on you machine.
47-
Follow instructions from https://graphviz.org/download/
48-
49-
After that is installed, you can install the dependencies for visualizations using:
65+
Once this is set up, install the dependencies using
5066

5167
```bash
52-
pip install -e '.[visualization]'
68+
pip install -e .
5369
```
5470

5571
To be able to run tests, also install test dependencies

0 commit comments

Comments
 (0)