Skip to content

README.md

Latest commit

 

History

History
226 lines (176 loc) · 4.59 KB

README.md

File metadata and controls

226 lines (176 loc) · 4.59 KB
Raw

🚀 ChatGPT Clone (Realtime + Web Scraping + Hybrid Memory + Socket.io)

| REACTJS | MONGODB | NODEJS | EXPRESSJS | PINECONE | RAG | GEMINI | SOCKET.IO | REST API |

A complete production-level ChatGPT clone featuring:

  • ⚡ Real-time AI chat (Socket.io)
  • 🌐 Web Scraping (Cheerio + Puppeteer)
  • 🧠 Hybrid Memory (STM + LTM via Pinecone)
  • 🔐 Auth with JWT + Cookies
  • ♻ Auto WebSocket reconnect + polling fallback
  • 🎨 ChatGPT-level UI/UX (almost identical)
  • 📦 Parallel optimized LLM pipeline (Promise.all)
  • 🧵 Hidden prompt engineering (avoids Gemini system-role limitations)

⭐ Features

✔ Realtime communication using WebSockets

✔ Hybrid STM + LTM memory system

✔ Web scraping support

  • Static websites → Cheerio
  • Dynamic JS-rendered → Puppeteer

✔ Strong authentication

✔ Deleting chats + cascading delete for messages

✔ Perfect mobile-responsive UI

✔ Shimmer floating loader for AI response

✔ Secure cookies + CORS + production deployment ready

🧩 Tech Stack

Frontend

  • React + Vite
  • TailwindCSS
  • Axios
  • Socket.io-client

Backend

  • Node.js
  • Express.js
  • Socket.io
  • MongoDB + Mongoose
  • Pinecone Vector DB
  • Gemini 2.0 Flash API
  • Cheerio
  • Puppeteer
  • JWT
  • Cookie-parser
  • CORS

📦 Major Packages Used

Purpose Package
Real-time chat socket.io
Frontend socket socket.io-client
Static scraping cheerio
JS-rendered scraping puppeteer
Vector memory @pinecone-database/pinecone
AI model @google/genai
Auth jsonwebtoken, bcryptjs
Cookies cookie-parser
Backend express, cors
DB mongoose

🧠 Hybrid Memory System (Advanced)

Short-Term Memory (STM)

  • Last 10 messages of the current chat
  • Controls conversation flow
  • Stored in MongoDB

Long-Term Memory (LTM)

  • Stored in Pinecone
  • Every user message & bot response becomes a vector
  • On each new message: 
    topMatches = pinecone.query(vectors)
  • Returned top 3 relevant memories

Hidden Context Merging (Gemini-safe)

Gemini does NOT allow system-role messages.
So we merge STM + LTM + scraped content inside a hidden user role prompt:

{ role: "user", text: hiddenContext + "User asked: " + userMessage }

The model sees memory + scraped content.
User sees only the answer.

🌐 Web Scraping System

Static Websites → Cheerio

axios.get(url)
cheerio.load(html)
extract & clean text

JS Websites → Puppeteer

puppeteer.launch()
page.goto(url, waitUntil:"networkidle0")
extract page content

Automatically selects Cheerio → Puppeteer fallback.

⚡ Performance Optimizations

🔥 1. Promise.all() Parallel Execution

Used throughout the backend:

const [message, vectors] = await Promise.all([
  messageModel.create(...),
  aiService.generateVector(...)
])

DB + vector embedding run simultaneously, reducing latency by 40–60%.

🔌 2. Hybrid WebSocket + Polling

Render sometimes kills WebSocket connections.
Solution:

transports: ["websocket", "polling"]
reconnection: true
reconnectionAttempts: Infinity

Flawless stability.

🔒 3. Cookie-based JWT Authentication

  • User stays logged in after refresh
  • httpOnly + secure + sameSite=none
  • Accessible via both REST & Socket.io

🏗 Flow of the System

1. User logs in → HTTP cookie set
2. React loads chats via REST (cookie auto-sent)
3. User sends message → socket.emit("ai-message")
4. Backend:
   a) Detect URL
   b) Scrape (Cheerio/Puppeteer)
   c) Save message
   d) Generate vector
   e) Store vector in Pinecone
   f) Query STM + LTM
   g) Merge into hidden prompt
   h) Generate response using Gemini
   i) Save response + vector
5. Backend → emits "ai-response"
6. Frontend updates in real-time

🧪 Chat Features

  • Create chat
  • Load all chats
  • Load messages
  • Delete chat
  • Auto-select next chat
  • Prevent sending messages if no chat exists

🎨 UI/UX

  • Fully responsive
  • ChatGPT-like sidebar
  • New chat popup
  • Floating AI shimmer loading bubble
  • Dark theme
  • Fixed input bar

🚀 Deployment Guide

Backend → Render

  • Environment variables:
MONGO_URI=
JWT_SECRET=
GEMINI_API_KEY=
PINECONE_API_KEY=
  • Build command: npm install
  • Start command: node server.js

Frontend → Vercel

Update Axios base URL:

https://your-backend.onrender.com

⭐ Support

If you like this project, ⭐ the repository.