Open-Source LLMs for Lawyers: An Experiment by Reg.Cloud and Raft

Automation of legal document work has come a long way, from regular expressions to modern neural networks. In practice, though, either the processing quality fell short of real business needs, or the cost of deploying and maintaining the solution proved prohibitively high. Searching for an optimal approach, the Reg.Cloud team gave Raft a grant to run an experiment with modern open-source LLMs on cloud servers with A100 GPUs. The goal was to find out how well LLMs handle long legal documents and whether they can be used for production-grade extraction of business-critical data.

During the experiment, the Raft team ran into several limitations. First, even the most modern LLMs have a limited context length. Legal documents are often very large, so processing them efficiently requires splitting them into parts or using context-extension methods. Second, the accuracy of data extraction depends directly on the quality of the model's training and its ability to understand legal terminology. Models trained on general data may struggle with specialized legal texts.

To address these problems, the Raft team applied a number of engineering solutions. They used chunking (splitting text into fragments) and summarization (condensing information) to process long documents. The models were also fine-tuned on specialized legal datasets. The team paid particular attention to choosing the optimal model parameters and tuning the extraction process.

The results were promising but not without shortcomings. The LLMs showed a good ability to extract key information from legal documents, but the accuracy and completeness of extraction varied with the document type and the complexity of the task. The best results came from models fine-tuned on specialized data. Even so, manual review of the results was still required to ensure high accuracy.

The experiment has important implications for the legal industry. It shows that open-source LLMs can be a useful tool for automating legal document processing, but they require careful tuning and adaptation to specific tasks. In the future, as the technology develops and more powerful models appear, significant improvements in results and wider use of LLMs in legal practice can be expected.

In conclusion, the Reg.Cloud and Raft experiment demonstrated the potential of open-source LLMs for automating work with legal documents. Despite certain limitations and difficulties, engineering solutions and fine-tuning make it possible to reach acceptable results. Further research and development in this area will open new opportunities to improve efficiency and cut costs in the legal sector.