Intelligent Chatbot using Seq2Seq Neural Networks

B.Tech Computer Science — Semester 6 Project | 2023 Subject: Neural Networks

---

About the Project

This project was developed as part of my Neural Networks course. The goal was to build a chatbot from scratch using a Sequence-to-Sequence (Seq2Seq) neural network architecture with an Attention mechanism — without using any pre-trained models.

The chatbot is trained on the Cornell Movie Dialogs Corpus which contains over 220,000 real conversational exchanges from movie scripts. It uses an LSTM-based Encoder-Decoder model with Bahdanau Attention that allows it to give context-aware responses.

Everything was built from scratch to understand how neural network architectures work internally, not just how to use them as black boxes.

---

What is Seq2Seq?

Seq2Seq (Sequence-to-Sequence) is a type of neural network model that takes a sequence of words as input and produces another sequence as output. It was originally developed for machine translation but works well for chatbots too.

It has two main parts:

• Encoder — reads the input sentence and compresses it into a fixed-size "context vector" (like a summary of what was said)
• Decoder — takes that context vector and generates the response word by word

The Attention Problem

Basic Seq2Seq has one big limitation: the entire input sentence is compressed into a single vector. For long sentences, this loses a lot of information.

Bahdanau Attention (2015) solves this by letting the decoder "look back" at all encoder outputs at every step and decide which input words are most relevant right now. This significantly improves response quality.

Input:  "what is your name ?"
         ↓     ↓   ↓    ↓   ↓
      [Encoder: LSTM processes each word]
         ↓ (attention weights computed at each decoder step)
      [Decoder: LSTM generates output word by word]
         ↓     ↓     ↓
Output: "my name is chatbot"

---

Project Structure

Intelligent_Chatbot/
│
├── app.py                      ← Flask web app (run this to chat)
├── train.py                    ← Training script (run this first)
├── model.py                    ← Model wrapper exporting build_model
├── seq2seq.py                  ← Full Seq2Seq Model
├── encoder.py                  ← LSTM Encoder
├── decoder.py                  ← LSTM Decoder with Attention
├── attention.py                ← Bahdanau Attention Mechanism
├── requirements.txt            ← Python dependencies
├── README.md                   ← This file
│
├── templates/
│   └── index.html              ← Chat web interface (Flask template)
│
├── utils/                      ← Utility modules package
│   ├── __init__.py             ← Package initialization
│   ├── vocabulary.py           ← Word-to-index mapping
│   ├── preprocess.py           ← Text cleaning + Cornell dataset loader
│   └── evaluate.py             ← BLEU score evaluation
│
├── data/                       ← Cornell dataset (auto-downloaded during training)
├── checkpoints/                ← Trained model checkpoints
│   ├── seq2seq_best.pt         ← Best model weights
│   └── vocabulary.pkl          ← Saved vocabulary
└── venv/                       ← Python virtual environment

---

Setup and Installation

Requirements

• Python 3.10 or higher
• NVIDIA GPU recommended (training takes ~1 hour on GPU, several hours on CPU)

Step 1 — Clone the repository

git clone https://github.com/ram-cs7/Intelligent_Chatbot.git
cd Intelligent_Chatbot

Step 2 — Create and activate virtual environment

Windows:

python -m venv venv
venv\Scripts\activate

Linux / Mac:

python3 -m venv venv
source venv/bin/activate

Step 3 — Install dependencies

pip install -r requirements.txt

This will install:

• torch — Deep learning framework (PyTorch)
• torchvision, torchaudio — PyTorch ecosystem
• flask — Web framework for the chatbot UI
• numpy — Numerical computing

Note: For GPU-accelerated training on NVIDIA GPUs, install PyTorch with CUDA support:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Step 4 — Train the model

python train.py

This will automatically:

1. Download the Cornell Movie Dialogs dataset (~10 MB)
2. Parse ~118,000 conversation pairs from the dataset
3. Build the vocabulary (~5,700 words with min frequency ≥ 10)
4. Train for up to 30 epochs with early stopping (patience=5)
5. Save the best checkpoint based on validation loss

Training Features:

• Learning rate scheduling (ReduceLROnPlateau)
• Early stopping to prevent overfitting
• Gradient clipping for stable training
• Automatic GPU/CPU detection

Training Time:

• GPU (NVIDIA RTX 4050 or similar): ~1 hour
• CPU: ~4-5 hours

Step 5 — Run the chatbot

python app.py

Open your browser and navigate to: http://localhost:5000

---

Model Architecture

Component	Details
Architecture	Seq2Seq LSTM with Bahdanau Attention
Encoder	2-layer LSTM, hidden size 512
Decoder	2-layer LSTM + Attention, hidden size 512
Embedding size	256
Dropout	0.3
Vocabulary size	~5,700 words (min frequency ≥ 10)
Total parameters	~17.8 million
Optimizer	Adam (lr=0.0005)
LR Scheduler	ReduceLROnPlateau (factor=0.5, patience=2)
Loss function	Cross-Entropy (padding ignored)
Teacher forcing	0.7 (70% ground truth during training)
Max epochs	30 (with early stopping, patience=5)
Batch size	64
Decoding strategy	Top-k sampling (k=10, temperature=0.8)

---

Training Results

Metric	Value
Best Val Loss	4.9929
Best Epoch	4
Epochs Run	9 (early stopping triggered)
Training Device	NVIDIA GeForce RTX 4050 Laptop GPU

Note: Open-domain conversational AI is a hard problem. LSTM-based Seq2Seq models produce reasonable but imperfect responses — this is expected behavior and a known limitation of the architecture. The project demonstrates understanding of the neural network concepts, not production-level chatbot quality.

---

How it Works (Simplified)

1. User types a message in the web interface
2. Message is cleaned (lowercase, remove special chars, etc.)
3. Words are converted to numbers using vocabulary lookup
4. Encoder LSTM processes the input sequence step by step
5. Attention weights are computed at each decoder step
6. Decoder LSTM generates the response word by word using top-k sampling
7. Words are converted back from numbers to text
8. Response is displayed in the chat window along with the detected intent

Intent Classification: Rule-based regex patterns label each message (greeting, question, farewell, emotion, etc.)

Decoding Strategy: Instead of greedy decoding (which tends to produce repetitive outputs), we use top-k sampling with temperature scaling. This introduces controlled randomness for more natural and diverse responses.

---

Key Concepts Demonstrated

• LSTM (Long Short-Term Memory) — Recurrent neural network architecture that handles sequential data
• Encoder-Decoder Architecture — Two-part model for sequence-to-sequence tasks
• Bahdanau Attention — Mechanism that lets the decoder focus on relevant parts of the input
• Teacher Forcing — Training technique where ground truth is fed back to the decoder
• Top-k Sampling — Decoding strategy that samples from the k most likely next words
• Gradient Clipping — Prevents exploding gradients during backpropagation
• Learning Rate Scheduling — Automatically reduces learning rate when training plateaus
• Early Stopping — Stops training when validation loss stops improving

---

Possible Improvements

• Try Transformer architecture instead of LSTM
• Implement beam search decoding
• Use a larger or domain-specific dataset
• Add pre-trained word embeddings (GloVe/Word2Vec)
• Deploy on a cloud server

---

References

1. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014.
2. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015.
3. Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical Neural Story Generation. ACL 2018. (Top-k sampling)
4. Danescu-Niculescu-Mizil, C., & Lee, L. (2011). Chameleons in Imagined Conversations. ACL Workshop 2011. (Cornell Movie Dialogs Corpus)
5. Papineni, K., et al. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. ACL 2002.

---

Author

Sairam Chennaka B.Tech - Computer Science & Engineering, Shoolini University

---

This project was done purely for learning purposes as part of the Neural Networks course curriculum. All dataset credits go to their respective authors.