Communication Profiling and Characterization of Deep-Learning Workloads on Clusters With High-Performance Interconnects von Awan, Ammar Ahmad, Jain, Arpan, Chu, Ching-Hsiang, Subramoni, Hari, Panda, Dhableswar K.

Volltext

CUDA-Aware OpenSHMEM: Extensions and Designs for High Performance OpenSHMEM on GPU Clusters von Hamidouche, Khaled, Venkatesh, Akshay, Ahmad Awan, Ammar, Subramoni, Hari, Chu, Ching-Hsiang, Panda, Dhabaleswar K.

Volltext

A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training von Singh, Siddharth, Olatunji Ruwase, Ammar Ahmad Awan, Rajbhandari, Samyam, He, Yuxiong, Bhatele, Abhinav

Volltext

Optimized large-message broadcast for deep learning workloads: MPI, MPI+NCCL, or NCCL2? von Awan, Ammar Ahmad, Manian, Karthik Vadambacheri, Chu, Ching-Hsiang, Subramoni, Hari, Panda, Dhabaleswar K.

Volltext

Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation von Ammar Ahmad Awan, Bedorf, Jeroen, Chu, Ching-Hsiang, Subramoni, Hari, Panda, Dhabaleswar K

Volltext

Privacy-aware searching with oblivious term matching for cloud storage von Pervez, Zeeshan, Awan, Ammar Ahmad, Khattak, Asad Masood, Lee, Sungyoung, Huh, Eui-Nam

Volltext

Designing Non-blocking Personalized Collectives with Near Perfect Overlap for RDMA-Enabled Clusters von Subramoni, Hari, Awan, Ammar Ahmad, Hamidouche, Khaled, Pekurovsky, Dmitry, Venkatesh, Akshay, Chakraborty, Sourav, Tomko, Karen, Panda, Dhabaleswar K.

Volltext

HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training with TensorFlow von Awan, Ammar Ahmad, Jain, Arpan, Anthony, Quentin, Subramoni, Hari, Panda, Dhabaleswar K.

Volltext

DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention von Yao, Zhewei, Wu, Xiaoxia, Li, Conglong, Zhang, Minjia, Qin, Heyang, Ruwase, Olatunji, Awan, Ammar Ahmad, Rajbhandari, Samyam, He, Yuxiong

Volltext bestellen

MCR-DL: Mix-and-Match Communication Runtime for Deep Learning von Anthony, Quentin, Awan, Ammar Ahmad, Rasley, Jeff, He, Yuxiong, Shafi, Aamir, Abduljabbar, Mustafa, Subramoni, Hari, Panda, Dhabaleswar

Volltext bestellen

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference von Holmes, Connor, Tanaka, Masahiro, Wyatt, Michael, Awan, Ammar Ahmad, Rasley, Jeff, Rajbhandari, Samyam, Aminabadi, Reza Yazdani, Qin, Heyang, Bakhtiari, Arash, Kurilenko, Lev, He, Yuxiong

Volltext bestellen

1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed von Li, Conglong, Awan, Ammar Ahmad, Tang, Hanlin, Rajbhandari, Samyam, He, Yuxiong

Volltext bestellen

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale von Rajbhandari, Samyam, Li, Conglong, Yao, Zhewei, Zhang, Minjia, Aminabadi, Reza Yazdani, Awan, Ammar Ahmad, Rasley, Jeff, He, Yuxiong

Volltext bestellen

DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales von Yao, Zhewei, Aminabadi, Reza Yazdani, Ruwase, Olatunji, Rajbhandari, Samyam, Wu, Xiaoxia, Awan, Ammar Ahmad, Rasley, Jeff, Zhang, Minjia, Li, Conglong, Holmes, Connor, Zhou, Zhongzhu, Wyatt, Michael, Smith, Molly, Kurilenko, Lev, Qin, Heyang, Tanaka, Masahiro, Che, Shuai, Song, Shuaiwen Leon, He, Yuxiong

Volltext bestellen

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale von Aminabadi, Reza Yazdani, Rajbhandari, Samyam, Zhang, Minjia, Awan, Ammar Ahmad, Li, Cheng, Li, Du, Zheng, Elton, Rasley, Jeff, Smith, Shaden, Ruwase, Olatunji, He, Yuxiong

Volltext bestellen

1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed von Tang, Hanlin, Gan, Shaoduo, Awan, Ammar Ahmad, Rajbhandari, Samyam, Li, Conglong, Lian, Xiangru, Liu, Ji, Zhang, Ce, He, Yuxiong

Volltext bestellen

Scalable and Efficient MoE Training for Multitask Multilingual Models von Kim, Young Jin, Awan, Ammar Ahmad, Muzio, Alexandre, Salinas, Andres Felipe Cruz, Lu, Liyang, Hendy, Amr, Rajbhandari, Samyam, He, Yuxiong, Awadalla, Hany Hassan

Volltext bestellen

HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow von Awan, Ammar Ahmad, Jain, Arpan, Anthony, Quentin, Subramoni, Hari, Panda, Dhabaleswar K

Volltext bestellen

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone von Abdin, Marah, Awan, Ammar Ahmad, Bach, Nguyen, Bahree, Amit, Bakhtiari, Arash, Bao, Jianmin, Behl, Harkirat, Bilenko, Misha, Bjorck, Johan, Cai, Martin, Cai, Qin, Chen, Weizhu, Cheng, Hao, Chopra, Parul, Dai, Xiyang, Eldan, Ronen, Gao, Jianfeng, Gao, Min, Goswami, Abhishek, Gunasekar, Suriya, Haider, Emman, Hao, Junheng, Hewett, Russell J, Hu, Wenxiang, Huynh, Jamie, Jacobs, Sam Ade, Javaheripi, Mojan, Jin, Xin, Karampatziakis, Nikos, Kauffmann, Piero, Khademi, Mahoud, Kim, Dongwoo, Kim, Young Jin, Lee, Yin Tat, Li, Yuanzhi, Liden, Lars, Lin, Xihui, Lin, Zeqi, Liu, Liyuan, Liu, Weishung, Liu, Xiaodong, Luo, Chong, Madan, Piyush, Mahmoudzadeh, Ali, Majercak, David, Mazzola, Matt, Mendes, Caio César Teodoro, Mitra, Arindam, Modi, Hardik, Nguyen, Anh, Norick, Brandon, Patra, Barun, Perez-Becker, Daniel, Portet, Thomas, Pryzant, Reid, Qin, Heyang, Radmilac, Marko, Ren, Liliang, Rosset, Corby, Roy, Sambudha, Ruwase, Olatunji, Saarikivi, Olli, Saied, Amin, Salim, Adil, Santacroce, Michael, Shah, Shital, Shang, Ning, Sharma, Hiteshi, Shen, Yelong, Shukla, Swadheen, Song, Xia, Tanaka, Masahiro, Tupini, Andrea, Vaddamanu, Praneetha, Wang, Chunyu, Wang, Guanhua, Wang, Lijuan, Wang, Shuohang, Wang, Xin, Wang, Yu, Wen, Wen, Witte, Philipp, Wu, Haiping, Wu, Xiaoxia, Wyatt, Michael, Xiao, Bin, Xu, Can, Xu, Jiahang, Xu, Weijian, Yang, Fan, Yang, Yifan, Yang, Ziyi, Yuan, Lu, Zhang, Chenruidong, Zhang, Cyril, Zhang, Jianwen, Zhang, Yi, Zhang, Yue, Zhang, Yunan, Zhou, Xiren

Volltext bestellen

DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention von Yao, Zhewei, Wu, Xiaoxia, Li, Conglong, Zhang, Minjia, Qin, Heyang, Olatunji Ruwase, Ammar Ahmad Awan, Rajbhandari, Samyam, He, Yuxiong

Volltext