Data placement method capable of supporting deep learning for distributed file system

The invention discloses a data placement method capable of supporting deep learning for a distributed file system, and a storage, reading and deletion method capable of supporting the deep learning onthe basis of a HDFS (Hadoop Distributed File System). The data placement method comprises the follow...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: TAN LUCHAO, ZHOU XUEHAI, WANG CHAO, LI XI
Format: Patent
Sprache:chi ; eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The invention discloses a data placement method capable of supporting deep learning for a distributed file system, and a storage, reading and deletion method capable of supporting the deep learning onthe basis of a HDFS (Hadoop Distributed File System). The data placement method comprises the following steps that: the design of a unit Pile of data aggregation, the design of a data placement strategy according to Pile and the design of data placement by taking the Pile as a unit on an aspect of physical storage are put forward; the minimum unit Pile of data is processed, the characteristic offetching data on batch in deep learning calculation can be supported, and data locality capable of supporting an upper layer calculation system is considered, and the load balancing of the file can becarried out; a data placement strategy which takes the Pile as the unit is used for planning the storage and the reading of the Pile of different users and different datasets; the design of a data block storage strategy by tak