Meta’s large language models Llama and Llama 2 both generated a lot of interest, due to them being launched under a more open, non-commercial license that allowed people more freedom to fine-tune and adapt the models, while also offering the possibility of them being run on consumer hardware.
With the recent release of Llama 3, it’s become clear that Meta has made significant strides in improving the performance of this open source-inspired large language model, in addition to training it on a much more diverse dataset than ever before.
Meta asserts that Llama 3 is its “most capable openly available” model to date, with this “generation demonstrat[ing] state-of-the-art performance on a wide range of industry benchmarks and offers new capabilities, including improved reasoning.”
This newest iteration of Meta’s LLM comes in two sizes: Llama 3 8B is the smaller 8-billion parameter model, while Llama 3 70B is the model that has been trained on 70 billion parameters. Both have different so-called knowledge cutoff dates — the larger model won’t have knowledge of anything that happened past December 2023, while Llama 3 8B’s cutoff date is March 2023.
Llama 3 vs. Llama 2
Though Llama 3 8B is considered a small language model (SML), its performance is comparable to that of Llama 2 70B, a model that is almost 10 times bigger. Additionally, relative to the previous generation, the context size of the Llama 3 models has increased from 4,096 to 8,192 tokens — meaning it is capable of processing larger prompts that can run up to approximately 6,000 words of context.
The company says that it opted for a decoder-only transformer architecture, a variation on the classic transformer model that is primarily used for tasks like language translation and text generation.
This latest version also includes a more efficient tokenizer that is equipped with a vocabulary of 128,000 tokens that encodes language much more efficiently. Moreover, the newest model employs grouped query attention (GQA), which boosts inference efficiency by bundling and processing similar pieces of text together, rather than each word individually.
The dataset of Llama 3 is about seven times larger than the set used to trained Llama 2, and includes a sizable proportion of non-English data.
This time around, Llama 3 has been extensively retooled to boost its capabilities in instruction following, reasoning, and code generation, while simultaneously reducing false refusal rates and increasing response diversity.
To achieve this, Meta utilized an innovative approach to instruction-tuning that integrates supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct preference optimization (DPO).
During testing, Llama 3 was evaluated across 12 major use cases — like question answering, creative writing, brainstorming and coding — and findings show that it generally outperforms other cutting-edge, proprietary models, like Claude and GPT-3.5, as well as open source models like Mistral.
According to Meta, Llama 3 has been trained on an impressive 15 trillion tokens of publicly available data that range from history and cultural knowledge to STEM-related subjects and coding. Notably, the dataset is about seven times larger than the set used to train Llama 2, and also includes a sizable proportion of non-English data that spans more than 30 languages, thus paving the way forward for enhanced multilingual applications.
Moreover, the data used to train its latest model was also carefully vetted. To accomplish this, a series of data-filtering pipelines were developed, using heuristic filters, NSFW filters, semantic deduplication approaches, and text classifiers to predict and ensure data quality.
The continuing evolution of open models like Llama will help establish a healthy ecosystem of open source alternatives to proprietary large language models.
“We found that previous generations of Llama are surprisingly good at identifying high-quality data, hence we used Llama 2 to generate the training data for the text-quality classifiers that are powering Llama 3,” explained Meta. “Some of our biggest improvements in model quality came from carefully curating this data and performing multiple rounds of quality assurance on annotations provided by human annotators.”
Responsible Development and Use
The latest version of Llama has also been designed with safety and responsible use in mind. Meta has adopted a new system-level approach to responsible development and deployment of its models. This is done by comprehensively testing them using both human efforts and automation, to evaluate the risk of potentially problematic responses related to potential misuse — such as for cyberattacks, deep fakes or deep scams. In addition, Llama 3 includes other safety tools like the content moderation system Llama Guard 2, plus Code Shield for filtering insecure code suggestions, as well as CyberSec Eval 2 for gauging potential security risks.
With the popularity and power of LLMs only increasing as time goes on, the continuing evolution of open models like Llama will no doubt help establish a healthy ecosystem of open source alternatives to proprietary large language models. Such a move makes for a more accessible development landscape that will allow users to innovate further — creating nifty fine-tuned applications for self-organizing file management, automated image re-captioning, and accurate medical Q&A, while also ensuring a certain level of overall transparency and accountability. You can find out more or download the model over at the Llama 3 website.
The post Llama 3: How Meta’s New Open LLM Compares to Llama 1 and 2 appeared first on The New Stack.
With the Llama 3 LLM, Meta has made significant strides in improving the performance of its open source-inspired large language model.