Skip to content

README.md

Latest commit

 

History

History
52 lines (39 loc) · 3.73 KB

README.md

File metadata and controls

52 lines (39 loc) · 3.73 KB
Raw

Introduction

Simple repo to finetune an LLM on your own hardware. Uses Hugging Face's autotrain-advanced to fine-tune a base model pulled from Hugging Face (HF). The output LoRA is created on the fine-tuning data, and the resulting model is merged from base+LoRA to be output as Pytorch checkpoints.

The base format for the fine-tuning data is the OpenAI format for fine-tuning to keep general usefulness high.

The Quickstart shows an end-to-end example using Slack conversations as the source for fine-tuning of Mistral-7B-Instruct-v0.2 pulled automatically from HF.

  • Data is first converted to the OpenAI fine-tuning format before being used with autotrain

Conversations are formatted with the Mistral instruction format:

  • <s>[INST] Message to target user [/INST] Target user response</s>

Quickstart

Utilises an example json obtained from slack with a tool like https://github.com/chr1spy1/slack-export/tree/master * Contains message history between users U0XXXXXXXXXX and U0ZZZZZZZZZZ

  1. make env

    • Sets up the autotrain conda environment with required dependencies

  2. make TARGET_USER=U0ZZZZZZZZZZ slack_to_oai

    • Converts downloaded Slack messages to the format needed by OpenAI
    • Data formatted so that the selected user U0ZZZZZZZZZZ message responses are set as the 'assistant' responses for fine-tuning
      • In actual use modify the value of TARGET_USER to a specific Slack id e.g. U027S88LZ6U
    • Uses very basic methods to extract the question-response flow between users
    • Outputs a jsonl for OpenAI fine-tuning containing conversations between user and assistant to data/all_messages.jsonl

  3. make oai_to_autotrain

    • Converts OpenAI formatted dataset of user-assisant message pairs to the format needed by autotrain
    • Outputs a csv file for fine-tuning to data/train.csv

  4. make fine_tune

    • Fine-tunes a Hugging Face model to respond like the target user on Slack
      • [Optionally] Configure in finetune_LLM.py
      • Command executed is stored in fine_tune.txt
    • Example data/train.csv only contains 4 examples to fine-tune. Actual use should have 1000's of examples for decently tuned models
    • Once run the command will output to a folder named my-custom-Mistral-7B-Instruct-v0.2

  5. make test_lora

    • Tests the LoRA with a simple question "I can't access the code repo, do you know what's wrong?"

Converting the Output Fine-tuned Models to GGUF format

  1. Utilise llama.cpp to convert the model to OpenLLaMA format
    • python convert.py /path/to/my-custom-Mistral-7B-Instruct-v0.2/ --outfile /path/to/my-custom-Mistral-7B-Instruct-v0.2/my-custom-Mistral-7B-Instruct-v0.2.gguf
    • Will require an environment with the llama.cpp dependencies installed
  2. Quantise to the desired bpw
    • ./quantize /path/to/my-custom-Mistral-7B-Instruct-v0.2/my-custom-Mistral-7B-Instruct-v0.2.gguf /path/to/my-custom-Mistral-7B-Instruct-v0.2/ggml-model-q4_0.gguf q4_0

Hardware Requirements

Fine-tuning on ~8K message pairs using the default repo configuration and finetune_LLM.py parameters to fine-tune Mistral-7B-Instruct-v0.2 required ~24GB of VRAM and ~3 hours of compute on an Nvidia A10G

  • Configuring USE_INT4 = True and USE_INT8 = False will decrease VRAM requirements by roughly half

If you do not have enough VRAM you will experience CUDA errors.