Efficient Algorithms for Fast Data Transfers Using Long and Large Pipes in WAN Networks

The pipe programming paradigm is an important Unix innovation which lets processes filter data to obtain a desired output. Extending computation in an Internet distributed environment has led us to design so called long and large pipes which achieve high speed transfer rates between remotely interconnected processes. Our method combines, in a full-fledged client/server application, the use of classical pipes, as a mean of serializing data locally, with multiple parallel TCP connections which yield high bandwidth throughput. The application, called bbftpPRO, brings together the old fashioned techniques of file striping with the new data streaming algorithms at high transfer rates. It includes capabilities for easy traversal of networks protected by firewalls or NAT machines, split-TCP like path support, and last but not least, support for new transmission procedures at variable rate, adapted for multipath implementation at application level. Experimental results have proven the superiority of the proposed data transfer system. Both simple file transfers and arbitrary data streaming were done effectively, efficiently, in parallel, over wide area networks, between pairs of distributed processes connected via extended fast pipes. Our piped approach proved to be portable across heterogeneous systems and most suitable for data intensive computing applications requiring geographically distributed data.