candle: [Examples] yi running with gpu enabled gives an error loading function: "Function 'cast_bf16_f32' does not exist"

Describe the bug yi running with gpu enabled gives an error loading function: “Function ‘cast_bf16_f32’ does not exist” To Reproduce Steps to reproduce the behavior:

cargo run --release --features metal --example yi – --prompt “hi”

Expected behavior Normal Output Results

Screenshots

avx: false, neon: true, simd128: false, f16c: false
temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64
retrieved the files in 583ns
Error: Metal error Error while loading function: "Function 'cast_bf16_f32' does not exist"

Desktop (please complete the following information):

OS: mac,m1
Version:latest main

Additional context It works fine with just the cpu.

About this issue

Original URL
State: closed
Created 6 months ago
Comments: 17

Most upvoted comments

Fwiw F32 is usually a far better default when trying out a BF16 model as it has the exact same range, using F16 make it quite possible to run into some nan/inf issues, the only advantage I would see is speed if your hardware supports it at 2x the speed of F32.

LaurentMazare on Jan 10, 2024

cast_bf16_f32 already exists here, which begs the question - does your computer have metal version > 3.10? You can run xcrun metal --version to check 😊

I’m downloading the model for the example now (slow connection). I’ll report back if it runs on my machine 👍

Edit: I do not get the error reported above, so it should be related to the metal version.

ivarflakstad on Jan 7, 2024