A Bitcoin-native LLM: dataset, architecture and open questions

The former ML Engineer at Spirit of Satoshi (SoS) shares insights into the challenges and developments in the realm of cryptocurrency-related machine learning technologies. Initially, SoS encountered skepticism from bitcoiners, which hindered further fundraising and advancement on projects like Satoshi-7B. Post-SoS, the engineer continued to innovate by developing tools such as Minimaxis, a script assistant for bitcoin coding that evolved into Code Satoshi within SoS, and contributing to the development of the Liquid support assistant. These initiatives underscore an ongoing commitment to enhancing blockchain technology through advanced tools.

Currently, the engineer is working on a project called Freedom Skills, which aims to make "freedom tech" more accessible. This platform reflects a shift in strategy from using LoRA adapters for model training to employing skill-based scripts. The focus has moved towards making large language models (LLMs) more effective at following instructions without the need for extensive fine-tuning. This approach is seen as more efficient given the rapid pace at which new and more powerful models are introduced.

Moreover, there's an expressed interest in creating specialized datasets that could streamline processes for data scientists by eliminating the preliminary steps of data cleaning and curation. This would not only facilitate easier access to high-quality data but also enhance the training capabilities of future models specifically concerning Bitcoin-related information. The engineer invites collaboration on these fronts, highlighting a community-driven approach to advancing these technologies.