🚀 ChatGPT Clone (Realtime + Web Scraping + Hybrid Memory + Socket.io)

| REACTJS | MONGODB | NODEJS | EXPRESSJS | PINECONE | RAG | GEMINI | SOCKET.IO | REST API |

A complete production-level ChatGPT clone featuring:

• ⚡ Real-time AI chat (Socket.io)
• 🌐 Web Scraping (Cheerio + Puppeteer)
• 🧠 Hybrid Memory (STM + LTM via Pinecone)
• 🔐 Auth with JWT + Cookies
• ♻ Auto WebSocket reconnect + polling fallback
• 🎨 ChatGPT-level UI/UX (almost identical)
• 📦 Parallel optimized LLM pipeline (Promise.all)
• 🧵 Hidden prompt engineering (avoids Gemini system-role limitations)

---

⭐ Features

✔ Realtime communication using WebSockets

✔ Hybrid STM + LTM memory system

✔ Web scraping support

• Static websites → Cheerio
• Dynamic JS-rendered → Puppeteer

✔ Strong authentication

✔ Deleting chats + cascading delete for messages

✔ Perfect mobile-responsive UI

✔ Shimmer floating loader for AI response

✔ Secure cookies + CORS + production deployment ready

---

🧩 Tech Stack

Frontend

• React + Vite
• TailwindCSS
• Axios
• Socket.io-client

Backend

• Node.js
• Express.js
• Socket.io
• MongoDB + Mongoose
• Pinecone Vector DB
• Gemini 2.0 Flash API
• Cheerio
• Puppeteer
• JWT
• Cookie-parser
• CORS

---

📦 Major Packages Used

Purpose	Package
Real-time chat	socket.io
Frontend socket	socket.io-client
Static scraping	cheerio
JS-rendered scraping	puppeteer
Vector memory	@pinecone-database/pinecone
AI model	@google/genai
Auth	jsonwebtoken, bcryptjs
Cookies	cookie-parser
Backend	express, cors
DB	mongoose

---

🧠 Hybrid Memory System (Advanced)

Short-Term Memory (STM)

• Last 10 messages of the current chat
• Controls conversation flow
• Stored in MongoDB

Long-Term Memory (LTM)

• Stored in Pinecone
• Every user message & bot response becomes a vector
• On each new message:

  topMatches = pinecone.query(vectors)
  

• Returned top 3 relevant memories

Hidden Context Merging (Gemini-safe)

Gemini does NOT allow system-role messages.
So we merge STM + LTM + scraped content inside a hidden user role prompt:

{ role: "user", text: hiddenContext + "User asked: " + userMessage }

The model sees memory + scraped content.
User sees only the answer.

---

🌐 Web Scraping System

Static Websites → Cheerio

axios.get(url)
cheerio.load(html)
extract & clean text

JS Websites → Puppeteer

puppeteer.launch()
page.goto(url, waitUntil:"networkidle0")
extract page content

Automatically selects Cheerio → Puppeteer fallback.

---

⚡ Performance Optimizations

🔥 1. Promise.all() Parallel Execution

Used throughout the backend:

const [message, vectors] = await Promise.all([
  messageModel.create(...),
  aiService.generateVector(...)
])

DB + vector embedding run simultaneously, reducing latency by 40–60%.

---

🔌 2. Hybrid WebSocket + Polling

Render sometimes kills WebSocket connections.
Solution:

transports: ["websocket", "polling"]
reconnection: true
reconnectionAttempts: Infinity

Flawless stability.

---

🔒 3. Cookie-based JWT Authentication

• User stays logged in after refresh
• httpOnly + secure + sameSite=none
• Accessible via both REST & Socket.io

---

🏗 Flow of the System

1. User logs in → HTTP cookie set
2. React loads chats via REST (cookie auto-sent)
3. User sends message → socket.emit("ai-message")
4. Backend:
   a) Detect URL
   b) Scrape (Cheerio/Puppeteer)
   c) Save message
   d) Generate vector
   e) Store vector in Pinecone
   f) Query STM + LTM
   g) Merge into hidden prompt
   h) Generate response using Gemini
   i) Save response + vector
5. Backend → emits "ai-response"
6. Frontend updates in real-time

---

🧪 Chat Features

• Create chat
• Load all chats
• Load messages
• Delete chat
• Auto-select next chat
• Prevent sending messages if no chat exists

---

🎨 UI/UX

• Fully responsive
• ChatGPT-like sidebar
• New chat popup
• Floating AI shimmer loading bubble
• Dark theme
• Fixed input bar

---

🚀 Deployment Guide

Backend → Render

• Environment variables:

MONGO_URI=
JWT_SECRET=
GEMINI_API_KEY=
PINECONE_API_KEY=

• Build command: npm install
• Start command: node server.js

Frontend → Vercel

Update Axios base URL:

https://your-backend.onrender.com

---

---

⭐ Support

If you like this project, ⭐ the repository.