ollama: Docker Container on KVM-Based Ubuntu Machine Fails to Start 'llama.cpp Server' with Illegal Instruction Error

Issue Summary:

I encountered an issue while running a Docker container on a KVM-based Ubuntu machine. The container is built using the following Dockerfile and runs a Go application:

Dockerfile:

# Stage 1: Build the binary
FROM golang:alpine AS builder

# Install required dependencies
RUN apk add --no-cache git build-base cmake

# Set the working directory within the container
WORKDIR /app

# Clone the source code from the GitHub repository
RUN git clone https://github.com/jmorganca/ollama.git .

# Build the binary with static linking
RUN go generate ./... \
    && go build -ldflags '-linkmode external -extldflags "-static"' -o .

# Stage 2: Create the final image
FROM alpine

ENV OLLAMA_HOST "0.0.0.0"

# Install required runtime dependencies
RUN apk add --no-cache libstdc++ curl

# Copy the custom entry point script into the container
COPY Modelfile /Modelfile

# Copy the custom entry point script into the container
COPY entrypoint.sh /entrypoint.sh

# Make the script executable
RUN chmod +x /entrypoint.sh

# Create a non-root user
ARG USER=ollama
ARG GROUP=ollama
RUN addgroup $GROUP && adduser -D -G $GROUP $USER

# Copy the binary from the builder stage
COPY --from=builder /app/ollama /bin/ollama

USER $USER:$GROUP

ENTRYPOINT ["/entrypoint.sh"]

Entrypoint.sh:

#!/bin/sh

./bin/ollama serve &

sleep 5

curl -X POST http://ollama:11434/api/pull -d '{"name": "llama2"}'

sleep 10

tail -f /dev/null

Error Log:

ollama | [GIN] 2023/09/12 - 14:16:41 | 500 | 1m30s | 10.10.2.6 | POST "/api/generate"
ollama | 2023/09/12 14:15:11 llama.go:311: waiting for llama.cpp server to start responding
ollama | 2023/09/12 14:15:41 llama.go:292: error starting llama.cpp server: llama.cpp server did not start within allotted time, retrying
ollama | 2023/09/12 14:15:41 llama.go:329: llama.cpp server exited with error: signal: illegal instruction (core dumped)
ollama | 2023/09/12 14:15:41 llama.go:285: starting llama.cpp server
ollama | 2023/09/12 14:15:41 llama.go:311: waiting for llama.cpp server to start responding

System Specification:

description: Computer
product: KVM (8.2.0)
vendor: Red Hat
version: RHEL-8.2.0 PC (Q35 + ICH9, 2009)
width: 64 bits
capabilities: smbios-2.8 dmi-2.8 smp vsyscall32
configuration: boot=normal family=Red Hat sku=8.2.0 uuid=AC06E592-B8AE-F64D-A219-4EC4D8C1C5A0
*-core
   description: Motherboard
   product: RHEL-AV
   vendor: Red Hat
   physical id: 0
   version: RHEL-8.2.0 PC (Q35 + ICH9, 2009)
*-firmware
      description: BIOS
      vendor: SeaBIOS
      physical id: 0
      version: 1.16.0-3.module+el8.7.0+1084+97b81f61
      date: 04/01/2014
      size: 96KiB
*-cpu:0
      description: CPU
      product: Intel Core i7 9xx (Nehalem Core i7, IBRS update)
      vendor: Intel Corp.
      physical id: 400
      bus info: cpu@0
      version: RHEL-8.2.0 PC (Q35 + ICH9, 2009)
      slot: CPU 0
      size: 2GHz
      capacity: 2GHz
      width: 64 bits
      capabilities: [List of CPU capabilities]
      configuration: [CPU configuration details]

Issue Description:

I have created a Docker container using the provided Dockerfile, and it runs successfully on Windows and macOS. However, when running it on a KVM-based Ubuntu machine, I encounter the following issue:

1. The application attempts to start a “llama.cpp server” but fails with a “signal: illegal instruction (core dumped)” error.
2. The error occurs when I try to access the “api/generate” endpoint.

Steps to Reproduce:

1. Build the Docker container using the provided Dockerfile.
2. Run the container on a KVM-based Ubuntu machine.
3. Run ollama serve
4. Run ollama pull llama2
5. Access the “api/generate” endpoint:

curl -X POST http://localhost:11434/api/generate -d '{
  "model": "llama2",
  "prompt": "Why is the sky blue?"
}'

Expected Behavior:

The application should start without errors, and I should be able to access the “api/generate” endpoint.

Actual Behavior:

The application encounters a “signal: illegal instruction (core dumped)” error when starting the “llama.cpp server,” and I cannot access the “api/generate” endpoint.