chai.ml, as data

# 1.4 EXAFLOP CLUSTERWe continued to iterate on RLHF (Reinforcement Learning with Human Feedback), training a reward model directly on user signals. This led to a huge boost in our day 30 user retention.### Larger Model Upgrade - 13B ArchitectureWe found that a bigger model leads to better depth, therefore better retention. We re-trained our LLM from scratch and saw another +10% engagement boost.### PPO Model DeployedUsing Proximal Policy Optimization, a reinforcement learning technique, we optimized our base foundation model to decrease the probability a chat session ends.### Deploys Reward Model XLContinued to scale up our reward model. Trained with 100 million signals to decrease user retry rate and increase chat session length.### Efficient Inference & Custom GPU OrchestrationOff-the-shelf load balancing and vLLM were no longer sufficient to support our user base at 500K DAU scale. We implemented custom CUDA kernels together with our own GPU orchestration system.### Increased GPU ReservationWe hit a scaling issue due to high demand from our users. We reserved an additional 1,000 A100 GPUs from our provider to scale reliably.### Deployed Model BlendingCHAI invented model blending—ensembling different LLMs trained on different targets at the conversation level. This outperformed GPT-3 in user retention.### BO8 Reward Model DeployedWith increased cluster capacity, we implemented Best-of-8 rejection sampling, utilizing our upgraded reward model to its full extent.### DPO Model DeployedUtilizing Direct Preference Optimization with user preference datasets, we boosted engagement by 20%. The performance stacked well with our existing reward model.### Upgraded All Existing Blends to DPOBuilding on the success of DPO, we iterated on optimization targets and data selection, and successfully deployed DPO across all production blends.### 13B Reward Model Deployed