In this article, we will carry out inference using OpenELM models. We will use the official pretrained and instruction-tuned models for this.
It is a very simple article to evaluate the provided models as they are. Also, this is a continuation of the previous article where we discussed the OpenELM models by Apple and went in-depth into the datasets, architecture, and benchmarks.
The process is going to be very simple here. We will use the pretrained OpenELM models and run them through some predefined prompts. All the while we will qualitatively evaluate how good the responses are. This will pave the way for future articles where we will be fine-tuning the OpenELM models for specific tasks.
Setting Up for OpenELM Inference
We need to set a few things up for OpenELM inference. As the models use the Llama 2 tokenizer, which has gated access on Hugging Face, we need to login through the CLI using one of the access tokens from Hugging Face.
Before that, make sure that you have a Hugging Face account.
Next, click on your profile icon, go to Settings, then Access Tokens. If you have not created an access token before, create one and copy it.
Then, open a terminal, and install the Hugging Face Hub CLI in the environment of your choice.
pip install -U "huggingface_hub[cli]"
Next, execute the following command and paste the access token when prompted.
huggingface-cli login
That’s it. The tokenizer should now automatically download whenever we execute the code further in this article.
Download Code
Running Inference using OpenELM Models
We just have one Python script here, infer.py. We will first go through the code and then run inference.
You can find all the OpenELM models on Hugging Face.
Although inference scripts are available in each of the Hugging Face repositories, we will writing a much simpler version of that. The results are the same when using similar hyperparameters.
Apart from the full precision 3B models, all inference experiments were run on a 10 GB RTX 3080 GPU. The full precision 3B inference experiments were run on Kaggle to accommodate the GPU memory usage.
Let’s go through the code now.
Import Statements and Argument Parsers
The following code block imports all the necessary libraries and modules and defines the argument parser.
from transformers import (
AutoTokenizer,
AutoModelForCausalLM,
BitsAndBytesConfig
)
import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
'--model',
default='apple/OpenELM-270M-Instruct',
help='open elm model to run',
choices=[
'apple/OpenELM-270M-Instruct',
'apple/OpenELM-450M-Instruct',
'apple/OpenELM-1_1B-Instruct',
'apple/OpenELM-3B-Instruct',
'apple/OpenELM-270M',
'apple/OpenELM-450M',
'apple/OpenELM-1_1B',
'apple/OpenELM-3B'
]
)
parser.add_argument(
'--prompt',
help='chat prompt for the model',
default='Hi, how are you?'
)
parser.add_argument(
'--int4',
action='store_true',
help='whether to load the model in 4-bit quantized format'
)
args = parser.parse_args()
We will need the BitsandBytesConfig when trying to load the larger models in quantized (INT4) format.
We have three command line argument options:
--model: To choose one of the eight available models.--prompt: The prompt that we want to provide to the model.--int4: A boolean argument indicating whether we want the model to load in INT4 quantized format or not.
Loading the Tokenizer and the OpenLM Model for Inference
As discussed earlier, OpenELM models use the Llama 2 tokenizer.
tokenizer = AutoTokenizer.from_pretrained(
'meta-llama/Llama-2-7b-hf'
)
if args.int4:
quant_config = BitsAndBytesConfig(
load_in_4bit=True
)
model = AutoModelForCausalLM.from_pretrained(
args.model,
quantization_config=quant_config,
trust_remote_code=True
)
else:
model = AutoModelForCausalLM.from_pretrained(
args.model,
trust_remote_code=True
).to('cuda')
For loading the model, if the --int4 command line argument is passed, then we define the quantization configuration and load the model. Otherwise, we load the model in full precision.
Defining the Prompt and Running Inference
The final step is defining the prompt, running inference, and getting the output.
prompt = args.prompt
tokenized_prompt = tokenizer(
prompt,
return_tensors='pt',
).to('cuda')
output_ids = model.generate(
**tokenized_prompt,
max_new_tokens=1024,
pad_token_id=0,
eos_token_id=tokenizer.eos_token_id,
repetition_penalty=1.2
# temperature=0.8,
# top_k=40,
# top_p=0.1,
# do_sample=True,
)
output_text = tokenizer.decode(
output_ids[0].tolist(),
skip_special_tokens=True
)
print(output_text)
We use similar keyword arguments for the generation strategy as provided in their official inference script on Hugging Face. There are a few more commented arguments that we may use for experiments.
We are done with the coding part. Now, let’s run inference.
OpenELM-270M Base Model Inference
We will start with the smallest OpenELM base model. We can use this model as a sentence completion model. In general, all decoder-only models are sentence completion or next token prediction models. However, the base pretrained models cannot handle instructions well and are further fine-tuning for instruction following.
python infer.py --model apple/OpenELM-270M --prompt "Three points to stay healthy are"
We provide a very simple prompt here. We finish the prompt with a continuation, i.e., “are” such that the model can continue the sentence. Here is the output that we get.
Three points to stay healthy are: - Eat a balanced diet. - Get enough sleep. - Exercise regularly. Eating a balanced diet is the most important thing you can do to keep your body in good shape. It's also the best way to prevent disease and maintain a healthy weight. The American Heart Association recommends that adults eat 2,000 calories a day, or about 1,800 calories for men and 1,500 calories for women. This means that if you want to lose weight, you need to eat more than that. If you don't get enough sleep, you will be less likely to burn fat. Sleep deprivation causes your body to store fat as energy. Exercise regularly is another way to keep your body in good shape. You should exercise at least 30 minutes a day. If you don't have time to exercise, try walking or jogging. You can also take part in sports like tennis, basketball, volleyball, soccer, or even swimming. It's also important to avoid smoking. Smoking increases your risk of heart disease and cancer. If you want to lose weight, it's important to make sure you're eating a balanced diet.
Interestingly enough, for a small model, the results are good. It even explains each of the points. However, logically, the answers are wrong in a few places. We can ignore that for now owing to the fact that a 270M model cannot handle factual information with as much precision.
But what happens when we do not end the prompt with a continuation phrase? Let’s check that.
python infer.py --model apple/OpenELM-270M --prompt "Three points to stay healthy"
Following is the output.
Three points to stay healthy The first thing you need to do is to get enough sleep. Sleep deprivation can lead to a number of problems, including fatigue, headaches, depression, and even heart disease. It's important to get at least seven hours of sleep each night. If you don't get enough sleep, your body will be unable to produce the hormones that keep you healthy. Second, make sure you eat a balanced diet. Eating a variety of foods will help you feel fuller for longer. This will also help you lose weight. You should aim to eat three meals a day, with two snacks in between. Aim to eat 50 percent of your calories from fruits and vegetables. Third, exercise. Exercise helps you burn calories and keeps your metabolism going. It also helps you build muscle and tone your body. You should try to get at least 30 minutes of exercise each day. Fourth, drink plenty of water. Drinking water will help you stay hydrated and will help you feel better. Water also helps you stay away from caffeine and alcohol. Fifth, avoid smoking. Smoking can cause many health problems, including cancer. It also increases your risk of developing heart disease and stroke. Sixth, take care of your teeth. Brushing and flossing are important to keep your teeth clean and healthy. They will also help you prevent cavities. Seventh, get regular checkups. Regular checkups will help you catch any problems early. Your dentist will be able to tell you if you have any issues with your teeth.
The answer still revolves around the prompt, however, the structure is not correct. Furthermore, the model provides seven points instead of three.
OpenELM-450M Base Model Inference
Now, we will move on to a slightly larger mode, the 450M parameter one.
Starting with the same prompt.
python infer.py --model apple/OpenELM-450M --prompt "Three points to stay healthy are"
Following is the output.
Three points to stay healthy are: - Eat a balanced diet. - Get enough sleep. - Exercise regularly. Eat a balanced diet The American Heart Association recommends that adults eat a diet that is high in fruits, vegetables, whole grains, lean meats, fish, poultry, and low in saturated fat, trans fat, cholesterol, sodium, and added sugars. Get enough sleep Sleep is essential for the body to function properly. Sleep deprivation can lead to weight gain, poor concentration, irritability, depression, and other health problems. Exercise regularly Regular exercise helps maintain a healthy weight, improves your mood, and reduces the risk of heart disease, stroke, and cancer. For more information on how to get healthy, visit the American Heart Association website at. This article was written by Dr. David L. Hunt, MD, MPH, FACP, an internist at the University of California, San Francisco. He is also a professor of medicine at UCSF. Dr. Hunt is a member of the American College of Cardiology, the American Heart Association, and the American Society of Hypertension.
Logically, the answer is better compared to the 270M model. However, in the end, it starts to hallucinate.
Let’s try without the continuation phrase.
python infer.py --model apple/OpenELM-450M --prompt "Three points to stay healthy"
Here is the output.
Three points to stay healthy - Posted on: Jan 16 2018 - Leave a response The holidays are over and the New Year is just around the corner. With that comes the start of a new year, which means it's time to make some changes in your diet. Here are three tips to help you get started. Eat more fruits and vegetables Fruits and vegetables are an excellent source of vitamins and minerals. They also contain fiber, which helps keep you feeling fuller for longer. Fruits and vegetables are also low in calories, making them a great choice for snacks or meals. Drink more water Water is essential for good health. It helps flush out toxins from your body and keeps you hydrated. Drinking water throughout the day can help you feel fuller for longer. Get enough sleep Sleep is important for your health. It helps regulate your mood and helps you wake up feeling refreshed. Sleeping at least seven hours a night is recommended. If you don't get enough sleep, you may experience fatigue and headaches. If you have any questions about how to improve your health, contact us today. We're here to help!
It starts completely with a hallucination and somehow loosely connects the prompt.
OpenELM-1.1B Base Model Inference
Now, we will move on to the 1.1B variant of the OpenELM model.
python infer.py --model apple/OpenELM-1_1B --prompt "Three points to stay healthy are"
Following is the output that we get.
1. Eat a balanced diet, with plenty of fruits and vegetables. 2. Drink lots of water. 3. Get enough sleep. I'm not sure if you have any other health issues, but I would also recommend getting your blood pressure checked. If it's high, that can cause fatigue. If you're already taking medication for depression or anxiety, make sure you're taking the right dosage. You don't want to take too much or too little. Also, try to get some exercise in every day. It doesn't have to be a lot, just something to keep your heart rate up. Walking is good, as is swimming. Finally, try to avoid stressful situations. If you're stressed out, your body will produce more cortisol, which can make you feel tired. So try to avoid things that make you stressed out. I hope this helps! Good luck! 🙂 Edit: Also, if you're feeling really tired, try to get some rest. Go to bed early and wake up early. That way, you'll be able to get more sleep.
This is perhaps the best answer that we got till now. In fact, the answer finishes with an emoji which is a playful addition.
Now, doing a similar experiment with the non-continuous phrase.
python infer.py --model apple/OpenELM-1_1B --prompt "Three points to stay healthy"
Following a part of a larger output that we get.
I'm a 20 year old male, and I have been playing basketball for about 10 years. I am currently in my second year of college, and I play on a varsity team. I have had some injuries over the years, but nothing too serious. I have been working out regularly since I was 14, and I have always been pretty active. I have never really had any major problems with my knees or ankles, but I do have some minor issues with my back. I have been having some pain in my lower back, and it has been getting worse over the past few
The output seems related to health but unrelated to the prompt.
OpenELM-3B Base Model Inference
Now, moving to the largest base model, the OpenELM-3B version.
python infer.py --model apple/OpenELM-3B --prompt "Three points to stay healthy are" Three points to stay healthy are: 1. Eat a balanced diet. 2. Get enough sleep. 3. Exercise regularly. The first two are pretty obvious, but the third one is often overlooked. I'm not talking about going to the gym and lifting weights or running on a treadmill. I'm talking about getting out and doing something active that you enjoy. I've been trying to get back into running after taking a break from it for a few years. I've always enjoyed running, but I got out of the habit when I started working full-time and had less time to devote to it. Now that I'm retired, I'm trying to get back into it. I've been using an app called Runkeeper to track my runs. It's a great app because it tracks your distance, pace, and calories burned. It also has a social aspect where you can connect with other runners and share your progress. I've been using Runkeeper for a few weeks now and I'm really enjoying it. It's a great way to stay motivated and see how far you've come. If you're looking for a way to get back into running, I highly recommend using an app like Runkeeper. It's a great way to stay motivated and see how far you've come. What do you think?
The answer is good but it seems it is answering from the perspective of another person which may be a deal-breaker in real-life use cases of text completion.
Using a similar methodology for the the non-continuous prompt now.
python infer.py --model apple/OpenELM-3B --prompt "Three points to stay healthy"
The holidays are over and it's time to get back to work. But before you start your new year's resolutions, take a moment to think about how you can keep yourself healthy this year. Here are three tips to help you stay healthy: 1. Get enough sleep. Sleep is important for your health. It helps you feel refreshed and ready to take on the day. If you don't get enough sleep, you may feel tired and irritable. You may also have trouble concentrating and remembering things. Getting enough sleep can also help you lose weight. 2. Eat a healthy diet. A healthy diet is important for your overall health. It can help you maintain a healthy weight, reduce your risk of certain diseases, and improve your energy levels. There are many different types of diets, so it's important to find one that works for you. 3. Exercise regularly. Exercise is important for your health. It can help you maintain a healthy weight, reduce your risk of certain diseases, and improve your energy levels. There are many different types of exercises, so it's important to find one that works for you. These are just a few tips to help you stay healthy this year. Remember to take care of yourself and make time for yourself.
This is perhaps the best answer for the non-continuation prompt till now, but is expected from a 3B model as well.
From experience, other 3B base models provide better prompt completion answers, e.g., Phi-2 2.7B.
In the following sections, we will use the instruction fine-tuned models for inference.
OpenELM-270M Instruct
We will start with the smallest OpenELM instruction tuned model. In the following sections, we will use a very similar prompt to the above sections.
python infer.py --model apple/OpenELM-270M-Instruct --prompt "Give me three tips to stay healthy."
Give me three tips to stay healthy. 1. Eat a balanced diet, with plenty of fruits, vegetables, whole grains, lean proteins, and complex carbohydrates. 2. Exercise regularly. This helps burn calories, improve your mood, and improves your immune system. 3. Limit alcohol intake. Alcohol increases cravings for unhealthy foods, and lowers your metabolism. Here are my tips: 1. Eat a balanced diet. Choose fresh, organic, local produce whenever possible. Try to eat at least 5-6 servings per day. 2. Exercise regularly. Walking or biking is great for getting your heart pumping, improving your mood, and burning calories. 3. Limit alcohol intake. Alcohol raises your blood alcohol level (BAC), which increases cravings for unhealthy foods. Try to limit yourself to no more than 2 drinks per day. Good luck! Keep up the good work! :)
The answer is good, however, there is repetition of the answer. This may be because of the small number of parameters.
OpenELM-450M Instruct
python infer.py --model apple/OpenELM-450M-Instruct --prompt "Give me three tips to stay healthy."
Following is the output.
Give me three tips to stay healthy. 1. Eat a balanced diet rich in whole grains, vegetables, fruits, lean protein, and omega-3 fatty acids. 2. Exercise regularly. 3. Sleep well. These are the cornerstones of good health. They're not magic potions that will make you invincible or cure all your ailments. But they can go a long way toward making sure you live a long, vibrant life. Here are my three tried-and-true tips for staying healthy: Eat a balanced diet rich in whole grains, vegetables, fruits, lean protein, and omega-3 fatty acids. Whole grains contain complex carbs that provide energy, fiber, and essential nutrients. They also contain antioxidants, minerals, and phytonutrients that promote heart health, brain function, and overall wellbeing. Vegetables are rich in antioxidants, fiber, and essential minerals. They help us absorb essential fatty acids, which support our immune system, maintain skin health, and promote bone density. Fruits are rich in omega-3 fatty acids, which have anti-inflammatory properties and promote cardiovascular health. Studies have linked omega-3s to lower risk of heart disease, stroke, and certain types of cancer. Exercise regularly. Physical activity helps improve blood flow, boosts our immune system, reduces inflammation, and promotes mental clarity. It doesn't have to be an intense workout; just 30 minutes a day can make a big impact. Sleep well. Our bodies need seven to nine hours of sleep per night to repair and rejuvenate. Proper sleep quality has been linked to reduced inflammation, improved cognitive function, and better overall health. Staying active and getting enough sleep is easier said than done, but these three simple steps can make a big impact. What are your three tried-and-true tips for staying healthy?
The answer is much better in this case. But in some places, the model is answering from a first-person perspective which may not be always desirable.
OpenELM-1.1B Instruct
python infer.py --model apple/OpenELM-1_1B-Instruct --prompt "Give me three tips to stay healthy."
Here is the output that we get.
Give me three tips to stay healthy. 1. Drink 8-10 cups of water or juice (preferably water) daily. 2. Eat whole grains, fruits, vegetables, and protein sources. 3. Exercise for 30 minutes at least 5 days a week. Here are my three tips: 1. Drinking enough water is essential because it helps flush out toxins from your body, hydrate you, and keep you feeling fuller longer. Water also keeps your skin, muscles, and organs functioning optimally. Drink at least 8-10 cups a day. 2. Fruits and vegetables provide nutrients that our bodies need but can't make on their own. They are packed with fiber, vitamins, minerals, and antioxidants. Eating these foods regularly will give you the energy you need to exercise and help you maintain a healthy weight. Try eating whole fruits, veggies, and whole grains instead of processed versions. 3. Exercise is crucial for staying healthy because it releases endorphins, increases metabolism, improves blood flow, and burns calories. Research has shown that regular exercise can reduce the risk of heart disease, stroke, some types of cancer, diabetes, and depression. Choose activities you enjoy and stick to them consistently.
Although the model goes on to explain each of the points, the heading before the explanation suggests that it is probably repeating the answer.
OpenELM-3B Instruct
Finally, the largest instruct model.
python infer.py --model apple/OpenELM-3B-Instruct --prompt "Give me three tips to stay healthy."
Give me three tips to stay healthy. 1. Drink water. 2. Eat vegetables. 3. Exercise. These are the three pillars of good health, according to conventional wisdom. But they're also the three pillars of bad health: They're all things we're told to do, but rarely actually do. In fact, a recent study found that only 15 percent of Americans meet the minimum daily water intake recommendation, and just 12 percent get enough vegetables. Meanwhile, fewer than half of adults exercise regularly. So how can you make sure you're doing your part to stay healthy? Here are five simple steps you can take right now. 1. Set a goal. Before you can make any changes, you need to know where you stand. So first, decide how much water you drink each day, and aim to increase it by one or two cups. Next, measure your vegetable intake, and set a goal for increasing your consumption by one or two servings per day. Finally, determine how often you exercise, and commit to working out at least three times per week. 2. Make it easy. Don't try to change everything at once. Instead, pick one habit to focus on, and make it as convenient as possible. For example, if you don't drink enough water, fill a reusable bottle and keep it with you throughout the day. Or, buy pre-cut vegetables and store them in the fridge so they're always ready to eat. 3. Reward yourself. Celebrate your successes, no matter how small. Buy yourself a new water bottle or veggie spiralizer when you reach your target, or schedule a massage after you've been exercising regularly for a month. Whatever makes you feel good, go for it! 4. Share your progress. Committing to change is much easier when you have supportive friends and family members rooting for you. Post your goals on social media, share your successes, and ask for their help when you need it. You can also join an online community like Daily Harvest, which provides delicious, healthy recipes and encourages friendly competition. 5. Be patient. It takes time to break old habits and form new ones. Don't expect instant results; instead, focus on making steady improvements over the long term. Remember, Rome wasn't built in a day! Incorporating these five steps into your daily life will help you stay on track toward better overall health. Good luck, and remember: You've got this!
After the answer, it also provides some tips to ease the process of staying healthy. This looks like a good addition.
Further analysis: The OpenELM-3B models do not work with INT4 quantization. When loaded in INT4 format, the model just provides the initial prompt as the output. It will be interesting to see whether fine-tuning the model using QLoRA rectifies the issue.
A Few Takeaways
From the inference experiments we can figure out that the pretrained and instruction-tuned OpenELM models are not the best out there. However, perhaps, it was never the aim of the researchers from Apple to make the model the best. They approached this research to make the process more open while providing:
- Dataset preparation steps.
- Model preparation code.
- Training code.
- Inference code.
- Evaluation code.
- And pretrained models.
This is most probably for those, who are willing to learn more about the entire process of training an LLM. We will be doing a few fine-tuning experiments further down the line.
Summary and Conclusion
In this article, we carried a very limited inference experiment of the Apple OpenELM models. However, we gained some useful insights from these, especially, where the models lack and where they are strong. I hope this article was worth your time.
If you have any doubts, thoughts, or suggestions, please leave them in the comment section. I will surely address them.
You can contact me using the Contact section. You can also find me on LinkedIn, and Twitter.
