Company
Date Published
April 2, 2024
Author
Madhav Singhal
Word count
3558
Language
English
Hacker News points
35

Summary

Replit, a platform that provides an integrated development environment (IDE), is rethinking the developer experience with AI as a first-class citizen. The company aims to integrate AI tools into its IDE, starting with code repair, which is a significant fraction of developers' time spent fixing bugs in software. Replit's goal is to train models native to its platform, enhancing their ability to understand events and use tools within the environment. By integrating LLMs (Large Language Models) with its IDE, Replit can create more powerful AI tools for developers. The company has developed a model that takes a session event as input and returns well-defined responses, such as fixing code errors. This model is trained on a dataset of (code, diagnostic) pairs, which includes LSP diagnostics from user sessions in BigQuery. The model uses a pre-trained code LLM with a few-shot prompt pipeline implemented with DSPy to synthesize diffs. The model distills a model from synthesized diffs and verifies that generated diffs are correctly formatted and applicable. Replit's finetuned model is competitive with larger models on both evaluation benchmarks, demonstrating the importance of evaluating deep learning models on both academic and real-world benchmarks. To further improve performance, Replit plans to curate data for more complex cases, support a larger set of programming languages, and experiment with post-training methods like DPO leveraging user data collected by the platform.