Output speed is increasingly slow more then 15 minuets for output.

[GuySarkinsky]

How results can be improved to make sense for using privateGPT?

The model I use: ggml-gpt4all-j-v1.3-groovy

Device specifications:

Device name
Full device name
Processor Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz 2.11 GHz
Installed RAM 16.0 GB (15.9 GB usable)
Device ID
Product ID
System type 64-bit operating system, x64-based processor
Pen and touch No pen or touch input is available for this display

• I understand that at the moment it is impossible to use GPU.

[johnbrisbin]

LLMs are memory hogs. Before you launch into privateGPT, how much memory is free according to the appropriate utility for your OS? How much is available after you launch and then when you see the slowdown? The amount of free memory needed depends on several things:

1. The amount of data you ingested into privateGPT. Multiply by 8-10
2. Add pip requirement install step fails on macOS #1 it to the size of the LLM models and the rest (maybe 3.5-4 GB - I know my db in memory is just less than 10 GB and the total while querying is 13.6 GB under Windoze 10).
   Is the system 'paging' when you use privateGPT? If so, that is slow right there.
   If it appears to be a lack-of-memory problem, the easiest thing you can do is to increase your installed RAM. I know that is not easy, but it would improve things somewhat.
   I have plenty of memory and processor and I wait several minutes for responses, but I am not seeing an obvious progression to longer response times.
   I can see that multi-process mode has not been engaged in the lookup so we might find a quick improvement...or not.

[GuySarkinsky]

@johnbrisbin Thanks for your answer,
Today I increased the RAM memory to 32 and it didn't really help.
My next step is to use the GPU.

[pseudotensor]

One reason for slow output from privateGPT is its literal use of langchain in prompting etc. You can try https://github.com/h2oai/h2ogpt that takes advantage of instruct-tuned LLMs. Now includes GPT4ALL support on linux/mac/windows.

[GuySarkinsky]

I will try it.