Bloom

BLOOM stands for BigScience Large Open-science Open-access Multilingual Language Model. It is a landmark project in the development of Large Language Models (LLMs).

BLOOM: the Open-Source Multilingual Giant

BLOOM is a Large Language Model—the same type of AI that powers tools like ChatGPT and the one I use. However, its creation and design have several unique aspects that set it apart.

Key Characteristics

1. Massive Scale: The main version of BLOOM has 176 billion parameters, placing it among the largest LLMs ever created, comparable in size to OpenAI's GPT-3.

2. Multilingual Proficiency: BLOOM was specifically trained to be proficient in 46 natural languages (including French, Spanish, Arabic, Chinese, and many underrepresented languages) and 13 programming languages. This makes it one of the most linguistically diverse large models.

3. Open-Source & Open-Access: This is BLOOM's defining feature. It was developed by the BigScience collaboration, a year-long research workshop involving over 1,000 volunteer researchers from 70+ countries. The final model, its source code, and the massive training dataset (called ROOTS) are all released under a Responsible AI License (RAIL), allowing researchers and developers to study, run, and build upon it transparently.

4. Architecture: It uses the Transformer architecture, specifically a causal decoder-only model (like GPT models), which is primarily designed for text generation (predicting the next word/token).

The BigScience Initiative

The entire BLOOM project was a monumental effort hosted by Hugging Face and trained on the Jean Zay public supercomputer in France. The primary goal was the democratization of large-scale AI research, providing an openly available alternative to proprietary, closed-source models.

Other Notable Features

• ALiBi Positional Embeddings: BLOOM uses the Attention with Linear Biases (ALiBi) technique instead of traditional positional encodings, which has been shown to improve performance and stability during training.

• Smaller Versions: To make the model accessible for those without supercomputing resources, smaller versions of BLOOM (e.g., bloom-560m, bloom-7b1) were also released, trained on the same data.

BLOOM is a pioneering, massive, open-access, multilingual LLM that serves as a critical resource for the global AI research community.