Custom LLM Provider (Groq) + Ollama CPU timeout — configuration questions

[VarRamanKatoch]

Hi Team,

Running PentAGI on a local Ubuntu VM and trying to configure a
free/low-cost LLM setup. Sharing full environment details below.

Background & Goal

PentAGI worked perfectly with Anthropic/Claude paid APIs — full
pentest completed successfully with all 8 test areas covered and
16 vulnerabilities found. However the cost per run was significant.

Our goal is to run PentAGI at scale with minimum to zero cost.
The use case: our developers are heavily vibe coding (AI-generated
code with minimal manual review), which means we need to run
frequent automated penetration tests against new app builds as
part of our security workflow. Paying per pentest run is not
sustainable at that frequency.

We are looking for the most cost-effective LLM configuration
that can still produce reliable pentest results.

Environment

• Host Machine: Windows laptop, 32GB RAM
• Virtualization: VMware Workstation
• Guest OS: Ubuntu 24.04 — 24GB RAM allocated, no GPU (CPU-only)
• PentAGI version: 2.0.0-2ec8ef3
• Docker: All 7 containers running
• Ollama version: 0.21.2
• Models pulled: qwen2.5:32b (19GB), mistral-small3.1 (15GB),
  llama3.1:8b (5GB)

What We Tried

Ollama (local — preferred for zero cost)

• Configured OLLAMA_SERVER_URL=http://<docker-bridge-ip>:11434
• Set OLLAMA_HOST=0.0.0.0 and OLLAMA_KEEP_ALIVE=30m via systemd override
• OLLAMA_SERVER_REQUEST_TIMEOUT=900
• llama3.1:8b loads and shows 100% CPU but flows fail with:
  context canceled — Post http://<docker-bridge-ip>:11434/api/chat
  after ~4 minutes regardless of timeout setting
• qwen2.5:32b and mistral-small3.1 exceed available RAM when
  Docker containers are running alongside

Groq (custom provider — free tier attempt)

• Configured via LLM_SERVER_URL=https://api.groq.com/openai/v1
• LLM_SERVER_PROVIDER= (blank)
• LLM_SERVER_MODEL=meta-llama/llama-4-scout-17b-16e-instruct
• Custom YAML config with max_tokens: 8192 on all agents
• PENTAGI_LLM_SERVER_CONFIG_PATH=./groq.provider.yml
• LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom.provider.yml
• Issue: PentAGI's initial system prompt is ~16,600 tokens,
  which exceeds Groq free tier limits:
  • llama-3.3-70b-versatile: 12K TPM → 413 error
  • llama-3.1-8b-instant: 6K TPM → 413 error
  • meta-llama/llama-4-scout-17b-16e-instruct: 30K TPM → 429
    rate limit hit on first request (~25K used of 30K)
• Flow was created and started but immediately hit TPM limit on
  every parallel agent call

Questions

1. Ollama CPU — Is context canceled a known issue with
   CPU-only inference on a VM? Is there a way to extend PentAGI's
   internal HTTP client timeout beyond what
   OLLAMA_SERVER_REQUEST_TIMEOUT controls?

2. Groq TPM — PentAGI's system prompt alone (~16K tokens)
   exceeds most Groq free tier model limits. Is there a way to
   reduce the system prompt size or use a lighter agent config
   for resource-constrained setups?

3. Custom YAML — Is the YAML provider config file the only
   way to set max_completion_tokens per agent, or is there an
   .env variable we missed?

4. Recommended minimum cost setup — Given our use case
   (frequent automated pentests of vibe-coded apps), what is
   the recommended lowest-cost or free LLM configuration that
   can reliably complete a full PentAGI flow?

Thanks for building this — it produced excellent results with
Claude and we want to make it a standard part of our security
pipeline.

[richardchen874-sys]

For this use case, I would avoid thinking about it as one "best cheap model" and instead split the pentest workflow by task type.

On a CPU-only VM, local 32B models are likely to hit timeout/latency issues for frequent runs. A more sustainable setup might be:

• low-cost remote model for recon summaries, extraction, report drafting, and repetitive checks
• stronger reasoning model only for exploit planning or ambiguous findings
• strict max token limits per phase
• run-level budget caps so one test cannot accidentally burn a large amount of tokens
• caching for repeated target context between builds

That gives you a middle ground between expensive Claude-only runs and slow local-only runs.

I am building a private-beta API aggregation layer focused on Chinese models like Qwen / DeepSeek / GLM-style providers, using official-provider keys. This kind of high-frequency security workflow is exactly where cheaper model routing can matter. Happy to share test credits if you want to benchmark a few models against your PentAGI pipeline.

For your current setup, is the main blocker quality, latency, or keeping per-run cost predictable?

[23HughAI]

Good