Hyper-Connections

Hyper-Connections is a Python library that implements the 'Hyper-Connections' method, proposed by ByteDance AI lab, as an alternative to traditional residual connections in neural networks. It aims to address drawbacks like the seesaw effect between gradient vanishing and representation collapse by introducing learnable depth and width connections. The library allows for flexible integration of features across depths and dynamic rearrangement of layers, particularly beneficial for large language models and vision tasks. It is actively developed, with its current version being 0.4.9, and features frequent releases.