together.ai, as data

Long context retrieval models with Monarch MixerJon Saad-Falcon, Dan Fu, Simran AroraMamba-3B-SlimPJ: State-space models rivaling the best Transformer architecturePaving the way to efficient architectures: StripedHyena-7B, open source models offering a glimpse into a world beyond TransformersFlashFFTConv: Efficient Convolutions for Long Sequences with Tensor CoresDan Fu, Hermann Kumbong, Eric Nguyen, Chris RéRedPajama-Data-v2: An open dataset with 30 trillion tokens for training large language modelsFlash-Decoding for long-context inferenceTri Dao, Daniel Haziza, Francisco Massa, Grigory SizovMedusa: Simple framework for accelerating LLM generation with multiple decoding headsTianle Cai*, Yuhong Li*, Zhengyang Geng, Hongwu Peng, Tri Dao (* Equal contribution)Llama-2-7B-32K-Instruct — and fine-tuning for Llama-2 models with Together APIFaster inference enables up to 5x price reduction on Together APIPreparing for the era of 32K context: Early learnings and explorationsMonarch Mixer: A new model architecture for increased efficiencyFine-tuning language models over slow networks using activation compression with guaranteesJue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Re, Ce ZhangDecentralized training of foundation models in heterogeneous environmentsBinhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, Ce ZhangFlashAttention: Fast and memory-efficient exact attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher RéCocktailSGD: Fine-tuning foundation models over 500Mbps networksFlexGen: High-throughput generative inference of large language models with a single GPUYing Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang