LocalAI

LocalAI is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families and architectures.

Refer to the LocalAI documentation to install & configure it.

Once you have the LocalAI up & running, you'll need to configure the unCtl:

llm_config:
  - provider: LocalAI
    models:
      - name: thebloke__codellama-13b-gguf__codellama-13b.q6_k.gguf
        config:
          temperature: 0.01
          tokenizer_type: llama
    config:
      endpoint: http://localhost:8080

llm_provider: LocalAI
llm_model: thebloke__codellama-13b-gguf__codellama-13b.q6_k.gguf

endpoint - an address to a running LocalAI instance

Also, note the model name. The name should be the same as installed in the LocalAI.

PreviousOpenAINextOllama

Last updated