System and architecture to optimize video traffic over internet protocol networks

Techniques are provided for managing network traffic and alleviating network congestion issues in video conference environments. At a video conference bridge device configured to send and receive communications to an endpoint device in a network, one or more video streams are received from the endpoint participating in a video conference. Each of the video streams is classified as a rate adaptive stream or as a non-rate adaptive stream. For video streams classified as rate adaptive streams, the video streams are assigned to a buffer queue for rate adaptive streams. For video streams classified as non-rate adaptive streams, the video streams are assigned to a buffer queue for non-rate adaptive streams.