TransformerLens: [Bug Report] Mixtral generates nonsense
Describe the bug
I followed the instructions in docs/source/content/special_cases.md as well as I could tell (ran the model in both full precision and with HookedTransformer.from_pretrained_no_processing), yet my model generations were nonsensical.
Code example
from transformer_lens import HookedTransformer
from transformers import AutoModelForCausalLM, AutoTokenizer
model = HookedTransformer.from_pretrained_no_processing(
"mistralai/Mixtral-8x7B-v0.1",
n_devices=4,
)
# test model actually works
for i in range(5):
display(
model.generate(
"Once upon a time",
verbose=False,
max_new_tokens=50,
)
)
System Info Describe the characteristic of your environment:
- Describe how
transformer_lenswas installed:pip install sae-lens - What OS are you using? Ubuntu 22.04 (
runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel-ubuntu22.04) - Python version: 3.10
Additional context Running on a 4x A100 SXM system on Runpod.
Checklist
- I have checked that there is no similar issue in the repo (required)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 42 (14 by maintainers)
Alright! I did 5 generations, and no sign of degradation. 2 for “Hello my name is”, and 3 for “Once upon a time,”
I think we are good!
@joelburget Yeah I think the issue is a composite of a few different problems. The big breakthrough was last night with setting the
W_Gateto usetorch.float. That seems to have almost entirely solved the problem between layers. The attention now definitely seems like the most reasonable place to look next, so I will checkout the notes @Butanium shared to see if that reveals more clues. I was looking at that segment of config much earlier, but partially ruled it out to look at the expert routing. Looking at it now, there were probably problems in both.