DataAvenue

What is Data Avenue?

 

Data Avenue is a data storage management service that provides access to different types of storage resources (including S3, sftp, GridFTP, iRODS, SRM servers) using a uniform interface. The provided REST API allows the user to perform all the typical storage operations such as creating folders/buckets, renaming or deleting files/folders, uploading/downloading files, or copying/moving files/folders between different storage resources, respectively, by simply using 'curl' command in command line. Data Avenue automatically translates users' REST commands to the appropriate storage protocols, and manages long-running data transfers in the background. In this tutorial we establish a cluster with two node types. The DataAvenue application will run on the DataAvenue node, and an S3 storage will run on a predefined number of storage nodes, so that we are able to try DataAvenue file transfer software such as making buckets, download or copy files. We used Ceph and Docker components to build-up the cluster.



Publications:

  • Akos Hajnal, Eniko Nagy, Peter Kacsuk and Istvan Marton: Data Migration for Large Scientific Datasets in Clouds, Azerbaijan Journal of High Performance Computing, Vol 1, Issue 1, 2018, pp. 66-86 https://doi.org/10.32010/26166127.2018.1.1.66.86
  • Hajnal Á, Márton I, Farkas Z, Kacsuk P: Remote storage management in science gateways via data bridging, CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERIENCE 27:(16) pp. 4398-4411. (2015)
  • Hajnal, A., Farkas, Z., and Kacsuk, P.  Data Avenue: Remote Storage Resource Management in WS-PGRADE/gUSE, 6th IEEE International Workshop on Science Gateways (IWSG), pp. 1–5, 2014
  • Hajnal Á, Farkas Z, Kacsuk P, Pintér T: Remote Storage Resource Management in WS-PGRADE/gUSE. In: Kacsuk P (szerk.)Science Gateways for Distributed Computing Infrastructures: Development Framework and Exploitation by Scientific User Communities. 301 p. Cham (Németország): Springer, 2014. pp. 69-81. (ISBN:978-3-319-11267-1)
     

 

User manual and installation guide:

https://github.com/SZTAKI-LPDS/data-avenue

Data Avenue reference architecture: 

https://occopus.readthedocs.io/en/latest/tutorial-building-clusters.html#dataavenue-cluster