Commit 6e23a43
feat: promote develop to main — v1.1.0 (#43)
* Initial commit: comparison API with semantic diff
* implmented similarity scoring algorithm and testing file
* update libraries versions in both python and typescript, and several small bug fixes. Added separate startup scripts not requiring the electron app.
* feat: Unified startup system and improved architecture
## Infrastructure
- Add unified start.sh script for managing both local and Docker deployments
- Add docker-compose.yml for container orchestration
- Add Dockerfiles for frontend (Node 18) and backend (Python 3.13)
- Remove obsolete startup scripts (restart_app.sh, start_*.sh)
- Add /datasets/ directory and move dataset generation tools out of backend
## Backend Improvements
- Merge new semantic_comparison.py with enhanced error handling and flexible model loading
- Remove LLM comparison functionality (app/ai/llm_comparison.py)
- Add model management router (/app/routers/models.py) with select/delete/import/list endpoints
- Add /models/translation/* and /models/comparison/* API endpoints
- Fix semantic comparison default model to use full identifier 'sentence-transformers/LaBSE'
- Update comparison API to support new model management
## Frontend Improvements
- Remove all backend lifecycle management from main.ts (kill/start processes)
- Add health check IPC handler calling /health endpoint
- Update TranslationSection to use /health endpoint for status checks
- Remove startBackend function and 'Start Backend' button from UI
- Fix compareArticles service to send correct model name 'sentence-transformers/LaBSE'
## Testing
- Fix all 4 failing tests (mock patches and cache handling)
- Achieve 100% test pass rate (56/56 tests passing)
- Rename semantic_comparison_tests.py to test_semantic_comparison.py
- Convert to proper pytest format with test_* functions
- Fix mock data for translation tests (add 'fr' language support)
## Documentation
- Update README.md with comprehensive start.sh usage documentation
- Add Docker Compose workflow documentation
- Update API endpoints list (remove LLM comparison, add model management)
- Update test statistics (56 tests, 100% passing)
- Update backend and frontend README files
- Fix backend path references in frontend documentation
## Architecture
- Frontend now only monitors backend health, does not manage lifecycle
- Backend lifecycle managed exclusively by start.sh script
- Both local and Docker workflows use same start.sh interface
- Services are independent with proper separation of concerns
This consolidates 4 startup scripts into 1 unified system with full Docker support
and removes all LLM comparison code in favor of semantic comparison only.
* Updated router documentation for the backend
* Added similarity threshold in the frontend app
* Updated start script to force a Python 3.13 environment
* update global gitignore
* Update translations.py
translate function to translate strings from source language to target language. Includes model dictionary for direct translations and dictionary for romance languages for fallback if a romance language does not have a direct language model.
* added test for translation // also added comments
* Update structured_wiki.py - structured translated article
Added router for creating a structured translated article from a structured article. Includes imports for new translation function.
* Update structured_wiki.py
Added router for translated wiki articles. Includes try/except structures to provide feedback for crashes
* WIP: semantic comparison + chunking updates (still working on fixes)
* chunking updates
* Deleted symmetry-unified-backend/tests/tempCodeRunnerFile.py
* Update Dockerfile
Update to Docker that has model baked in
* Update translations.py
Changed dict for language models for an easily maintainable json file. Kept local romance lang dict for fallback on edge cases.
* json for translation models
A json file that contains list of languages models. Each model has a source lang, target lang, similarity threshold, and model_name.
* Update translation_models.json
Added similarity scores from similarity_scoring.py
* Update translations.py
Improved language model search and added caching for reusing same language model.
* align frontend with backend API and fix launch issues when running in web mode
* fix language by using a package instead of a list
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update semantic_comparison.py (#7)
New code for semantic comparison that includes a cache system
* Added translation feature to structured article (#5)
* Created new button in structuredarticle viewer to translate article using Alberts translate code.
Also removed search function in structured-article because I believe it did not function.
* added Helsinki model languages
* Re-added search within sections functionality
* Replace hardcoded languages with dynamic language display names library.
* Added comments explaining Null citations/references for future use
* Created new button in structuredarticle viewer to translate article using Alberts translate code.
Also removed search function in structured-article because I believe it did not function.
* added Helsinki model languages
* Re-added search within sections functionality
* Replace hardcoded languages with dynamic language display names library.
* Added comments explaining Null citations/references for future use
* quick fix with newline
* updated docker-compose.yml - installation of spacy models (#11) (#13)
* updated docker-compose.yml - installation of spacy models
adds volume to install spaCy models
* Refactor command to download Spacy models
Fixed high security vulnerability
---------
Co-authored-by: AruefliASU <aruefli@asu.edu>
* uodate start script
* chunking updates
* Summary of Changes (#17)
* implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend.
* Able to download model weights from huggingface, but model output varies in quality.
* implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts.
* Summary of Changes
1. Tab Restructure (Structured Article first, legacy tabs labeled)
- src/pages/Home.tsx - Structured Article is now the default/first tab. Translation and AI Comparison tabs are labeled "(Legacy)".
- src/components/Layout.tsx - Removed 85 lines of hardcoded Biden test data, unused getColorClass, unused imports. Reduced from 152 to 17 lines.
- src/components/PageHeader.tsx - Removed hardcoded "Hi there, Suraj", non-functional language selector. Now shows "Cross-Language Article Analysis" with a descriptive subtitle.
2. Section-Level Semantic Comparison (new feature)
Backend:
- app/models/section_comparison.py (new) - Pydantic models: SectionCompareRequest, ParagraphDiff, SectionDiff, SectionCompareResponse
- app/services/section_comparison.py (new) - Core comparison engine: matches sections using LaBSE embeddings (greedy best-match on title + content preview), then compares paragraphs within matched sections. Uses Levenshtein disambiguation when top-2 semantic candidates are within 0.08 margin.
- app/routers/comparison.py - Added POST /articles/compare-sections endpoint with URL/title parsing
Frontend:
- src/components/SectionComparisonView.tsx (new) - Full diff UI with expandable section cards, paragraph side-by-side view, similarity bars, color-coded status badges, and filter controls
- src/services/structuredWikiService.ts - Added compareSections() method (POST)
- src/models/structured-wiki.ts - Added TypeScript interfaces for ParagraphDiff, SectionDiff, SectionCompareResponse, SectionCompareRequest
- src/components/StructuredArticleViewer.tsx - Added "Compare Sections" button with language selector, toggles between article view and comparison view
3. Bug Fixes
- app/ai/translations.py - Was completely empty (runtime crash). Now wraps translate_text() from translation.py with graceful error handling.
- app/ai/translation.py - Removed commented-out Google Colab interactive code.
- tests/test_translation_limits.py - Updated test_unsupported_language to match new graceful fallback behavior.
4. Dead Code Removal
Deleted files (7):
- src/models/enums/FetchArticleResponse.ts (duplicate)
- src/models/FetchArticleRequest.ts (duplicate, unused)
- src/models/apis/TranslateArticleRequest.ts (unused)
- src/models/apis/FetchArticleRequest.ts (unused)
- app/ai/dataset/dataset_gen.py (standalone experimental script)
- app/main.spec (PyInstaller build artifact)
- app/temp_backend_run.sh (developer temp script)
Cleaned inline dead code:
- app/models/comparison.py - Removed all commented-out old model versions, added docstrings
- app/models/api_models.py - Removed 3 unused shadow models (ComparisonResponse, TranslationResponse, ArticleResponse)
- app/models/comparison_request.py - Removed empty LLMCompareRequest class
- app/ai/semantic_comparison.py - Removed test main() function
5. Deduplication & Naming
- app/ai/model_registry.py (new) - Single source of truth for COMPARISON_MODELS and DEFAULT_MODEL. Previously duplicated in 3 files.
- app/main.py, app/routers/comparison.py, app/ai/semantic_comparison.py - All now import from model_registry.py
- app/ai/semantic_comparison.py - Renamed eng_embedding → embedding, first_embeddings/second_embeddings → source_embeddings/reference_embeddings, sim_threshold → similarity_threshold, diff_info → unmatched_sentences
- src/services/compareArticles.ts - Renamed parameters from originalArticleContent → sourceArticleContent, removed debug console.log statements
- src/components/TranslationSection.tsx - Replaced opaque texts/editing/reference/suggestedContribution/suggestionType with typed ArticleDisplayBlock interface with clear label, content, displayType fields
6. Documentation
- README.md - Complete rewrite reflecting the new architecture, section comparison feature, and cleaned project structure
- CLAUDE.md - Updated agent instructions matching the current codebase
Verification
- Backend: 143 passed, 7 failed (all pre-existing). One previously-failing test fixed.
- Frontend: 0 new TypeScript errors (5 pre-existing in Electron boilerplate files).
* Update wiki_structure.py - update reference id max size to 300 (#15)
Fix bug that can cause internal server error while running backend.
I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine
* Use canonical field names in CompareRequest (original_article_content etc.)
Replace text_a/text_b/language_a/language_b with meaningful names:
- original_article_content (alias: article_text_blob_1)
- translated_article_content (alias: article_text_blob_2)
- original_language (alias: article_text_blob_1_language)
- translated_language (alias: article_text_blob_2_language)
Update comparison.py router to use new field names when building
request_data for the semantic comparison engine.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Eliminate aliases: use original/translated names everywhere
Remove all AliasChoices from CompareRequest and BaseCompareRequest.
Use canonical field names end-to-end across backend and tests:
- original_article_content / translated_article_content
- original_language / translated_language
Also update:
- semantic_comparison.py: request_data keys use canonical names
- comparison.py router: GET query params renamed, request_data dicts updated
- comparison_request.py: BaseCompareRequest fields renamed
- compareArticles.ts: resolve remaining conflict marker
- Tests: fixtures and request dicts updated to canonical names
- Add ge/le bounds back to similarity_threshold in CompareRequest
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Resolve remaining merge conflicts: favor canonical field names
- StructuredArticleViewer.tsx: keep FactExtraction import and full UI
- models/__init__.py: keep FactExtraction exports
- comparison.py / comparison_request.py: canonical original/translated names
- semantic_comparison.py: canonical key names in request_data
- api_models.py: keep FactExtractionRequest/Response classes
- compareArticles.ts: clean canonical params, no debug cruft
All 15 backend tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Resolve merge conflicts: keep HEAD (canonical names + richer features)
Files resolved:
- FactExtraction.ts: keep num_facts field
- compareArticles.ts: keep canonical sourceArticleContent/sourceLanguage params
- StructuredArticleViewer.tsx: keep full fact-extraction + section-comparison UI
- api_models.py: keep section_title and num_facts in FactExtractionRequest
- models/__init__.py: keep grouped SectionCompare + FactExtraction exports
- structured_wiki.py: keep full implementation
- fact_extraction.py: keep richer HEAD implementation
- fact_extraction_models.json: keep expanded model list with full metadata
All 15 backend tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update symmetry-unified-backend/app/services/section_comparison.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update desktop-electron-frontend/src/components/StructuredArticleViewer.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Isaiah-Milkey <isamilkey@gmail.com>
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: AruefliASU <aruefli@asu.edu>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update wiki_structure.py - update reference id max size to 300 (#15)
Fix bug that can cause internal server error while running backend.
I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine
* Rename a/b variable names to original/translated for consistency
Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/11f337b7-3983-49a3-a358-331a5a6d29d4
Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com>
* Refactor compareArticles function - remove unnecessary try block and streamline axios instance retrieval
* Move model names to config.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add attributes to the API - (model name and similarity threshold) (#16)
* Update comparison.py
* Update comparison.py
* Update response.py
---------
Co-authored-by: François Pelletier <franc00018@users.noreply.github.com>
* Use DEFAULT_MODEL and safe .get() for comparison keys in semantic_comparison
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Include comparisons in CompareResponse and return raw comparisons when available
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix loanword suffix list and adjust family thresholds
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* GitHub actions implement (#18)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: unify threshold configuration across all comparison paths (#20)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
* fix: unify threshold configuration across all comparison paths
- Add app/core/settings.py as single source of truth for all thresholds,
reading values from .env with sensible defaults
- section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN
(0.08) and similarity_threshold default (0.65) with settings imports
- similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70)
and band tuples with settings imports
- semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75)
with SIMILARITY_THRESHOLD from settings (consistent with config.json)
- Add GET /config/thresholds endpoint for runtime threshold inspection
- Extend config.json and .env.template with all new threshold keys
Closes #19
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: unify threshold configuration across all comparison paths (#21)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
* fix: unify threshold configuration across all comparison paths
- Add app/core/settings.py as single source of truth for all thresholds,
reading values from .env with sensible defaults
- section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN
(0.08) and similarity_threshold default (0.65) with settings imports
- similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70)
and band tuples with settings imports
- semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75)
with SIMILARITY_THRESHOLD from settings (consistent with config.json)
- Add GET /config/thresholds endpoint for runtime threshold inspection
- Extend config.json and .env.template with all new threshold keys
Closes #19
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: frontend fetches thresholds from backend; strip threshold fields from config.json
- Add thresholdService.ts: calls GET /config/thresholds with in-memory cache
and graceful fallback to defaults if backend is unreachable
- ComparisonSection.tsx: replace hardcoded 0.65 with value fetched from
thresholdService on mount
- config.json: remove SIMILARITY_THRESHOLD, LEVENSHTEIN_DISAMBIGUATION_MARGIN,
FAMILY_THRESHOLD_*, COMPARISON_MODELS, DEFAULT_MODEL — these are now owned
by the backend (.env / app/core/settings.py) and served via GET /config/thresholds
config.json retains only infrastructure fields (BACKEND_BASE_URL, ports, timeout)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update desktop-electron-frontend/src/services/thresholdService.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update config.json with model configurations
Added comparison models and default model to config.
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* edited model options and ability to paste in model name. May need to implement with Albert's HF token code
* Added new UI features integrated with Fact Extraction.
* Added cache management for extraction models to avoid OOM errors.
* Refactor code for improved readability and consistency in fact extraction module and tests
* Update Completed
Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py
Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks
Kept only input tensor device transfer inside the loop
This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work.
* Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency
* Refactor structured_wiki.py for improved readability and consistency in imports and function formatting
* fix: remove duplicate useState declarations from merge conflict in StructuredArticleViewer
* style: format parameters in test_compare_semantic_get_invalid_threshold and test_wiki_translate_missing_params for improved readability
* fix: restore spaCy sentence segmentation lost during rebase --ours resolution
* Fact Extraction Implementation (#23)
* implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend.
* Able to download model weights from huggingface, but model output varies in quality.
* implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts.
* edited model options and ability to paste in model name. May need to implement with Albert's HF token code
* Added new UI features integrated with Fact Extraction.
* Added cache management for extraction models to avoid OOM errors.
* Refactor code for improved readability and consistency in fact extraction module and tests
* Implement spaCy sentence segmentation for improved accuracy and add corresponding tests
* Update Completed
Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py
Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks
Kept only input tensor device transfer inside the loop
This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work.
* Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency
* Refactor structured_wiki.py for improved readability and consistency in imports and function formatting
* fix: remove duplicate useState declarations causing CI build failure
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: François Pelletier <franc00018@users.noreply.github.com>
* docs: update CLAUDE.md with changelog conventions and LLM agent instructions
* Added OpenRouter free model functioanlity for fact extraction and visual updates
* progress indicator for translation
* updated docker-compose.yml - installation of spacy models (#11) (#13)
* updated docker-compose.yml - installation of spacy models
adds volume to install spaCy models
* Refactor command to download Spacy models
Fixed high security vulnerability
---------
Co-authored-by: AruefliASU <aruefli@asu.edu>
* Summary of Changes (#17)
* implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend.
* Able to download model weights from huggingface, but model output varies in quality.
* implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts.
* Summary of Changes
1. Tab Restructure (Structured Article first, legacy tabs labeled)
- src/pages/Home.tsx - Structured Article is now the default/first tab. Translation and AI Comparison tabs are labeled "(Legacy)".
- src/components/Layout.tsx - Removed 85 lines of hardcoded Biden test data, unused getColorClass, unused imports. Reduced from 152 to 17 lines.
- src/components/PageHeader.tsx - Removed hardcoded "Hi there, Suraj", non-functional language selector. Now shows "Cross-Language Article Analysis" with a descriptive subtitle.
2. Section-Level Semantic Comparison (new feature)
Backend:
- app/models/section_comparison.py (new) - Pydantic models: SectionCompareRequest, ParagraphDiff, SectionDiff, SectionCompareResponse
- app/services/section_comparison.py (new) - Core comparison engine: matches sections using LaBSE embeddings (greedy best-match on title + content preview), then compares paragraphs within matched sections. Uses Levenshtein disambiguation when top-2 semantic candidates are within 0.08 margin.
- app/routers/comparison.py - Added POST /articles/compare-sections endpoint with URL/title parsing
Frontend:
- src/components/SectionComparisonView.tsx (new) - Full diff UI with expandable section cards, paragraph side-by-side view, similarity bars, color-coded status badges, and filter controls
- src/services/structuredWikiService.ts - Added compareSections() method (POST)
- src/models/structured-wiki.ts - Added TypeScript interfaces for ParagraphDiff, SectionDiff, SectionCompareResponse, SectionCompareRequest
- src/components/StructuredArticleViewer.tsx - Added "Compare Sections" button with language selector, toggles between article view and comparison view
3. Bug Fixes
- app/ai/translations.py - Was completely empty (runtime crash). Now wraps translate_text() from translation.py with graceful error handling.
- app/ai/translation.py - Removed commented-out Google Colab interactive code.
- tests/test_translation_limits.py - Updated test_unsupported_language to match new graceful fallback behavior.
4. Dead Code Removal
Deleted files (7):
- src/models/enums/FetchArticleResponse.ts (duplicate)
- src/models/FetchArticleRequest.ts (duplicate, unused)
- src/models/apis/TranslateArticleRequest.ts (unused)
- src/models/apis/FetchArticleRequest.ts (unused)
- app/ai/dataset/dataset_gen.py (standalone experimental script)
- app/main.spec (PyInstaller build artifact)
- app/temp_backend_run.sh (developer temp script)
Cleaned inline dead code:
- app/models/comparison.py - Removed all commented-out old model versions, added docstrings
- app/models/api_models.py - Removed 3 unused shadow models (ComparisonResponse, TranslationResponse, ArticleResponse)
- app/models/comparison_request.py - Removed empty LLMCompareRequest class
- app/ai/semantic_comparison.py - Removed test main() function
5. Deduplication & Naming
- app/ai/model_registry.py (new) - Single source of truth for COMPARISON_MODELS and DEFAULT_MODEL. Previously duplicated in 3 files.
- app/main.py, app/routers/comparison.py, app/ai/semantic_comparison.py - All now import from model_registry.py
- app/ai/semantic_comparison.py - Renamed eng_embedding → embedding, first_embeddings/second_embeddings → source_embeddings/reference_embeddings, sim_threshold → similarity_threshold, diff_info → unmatched_sentences
- src/services/compareArticles.ts - Renamed parameters from originalArticleContent → sourceArticleContent, removed debug console.log statements
- src/components/TranslationSection.tsx - Replaced opaque texts/editing/reference/suggestedContribution/suggestionType with typed ArticleDisplayBlock interface with clear label, content, displayType fields
6. Documentation
- README.md - Complete rewrite reflecting the new architecture, section comparison feature, and cleaned project structure
- CLAUDE.md - Updated agent instructions matching the current codebase
Verification
- Backend: 143 passed, 7 failed (all pre-existing). One previously-failing test fixed.
- Frontend: 0 new TypeScript errors (5 pre-existing in Electron boilerplate files).
* Update wiki_structure.py - update reference id max size to 300 (#15)
Fix bug that can cause internal server error while running backend.
I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine
* Use canonical field names in CompareRequest (original_article_content etc.)
Replace text_a/text_b/language_a/language_b with meaningful names:
- original_article_content (alias: article_text_blob_1)
- translated_article_content (alias: article_text_blob_2)
- original_language (alias: article_text_blob_1_language)
- translated_language (alias: article_text_blob_2_language)
Update comparison.py router to use new field names when building
request_data for the semantic comparison engine.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Eliminate aliases: use original/translated names everywhere
Remove all AliasChoices from CompareRequest and BaseCompareRequest.
Use canonical field names end-to-end across backend and tests:
- original_article_content / translated_article_content
- original_language / translated_language
Also update:
- semantic_comparison.py: request_data keys use canonical names
- comparison.py router: GET query params renamed, request_data dicts updated
- comparison_request.py: BaseCompareRequest fields renamed
- compareArticles.ts: resolve remaining conflict marker
- Tests: fixtures and request dicts updated to canonical names
- Add ge/le bounds back to similarity_threshold in CompareRequest
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Resolve remaining merge conflicts: favor canonical field names
- StructuredArticleViewer.tsx: keep FactExtraction import and full UI
- models/__init__.py: keep FactExtraction exports
- comparison.py / comparison_request.py: canonical original/translated names
- semantic_comparison.py: canonical key names in request_data
- api_models.py: keep FactExtractionRequest/Response classes
- compareArticles.ts: clean canonical params, no debug cruft
All 15 backend tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Resolve merge conflicts: keep HEAD (canonical names + richer features)
Files resolved:
- FactExtraction.ts: keep num_facts field
- compareArticles.ts: keep canonical sourceArticleContent/sourceLanguage params
- StructuredArticleViewer.tsx: keep full fact-extraction + section-comparison UI
- api_models.py: keep section_title and num_facts in FactExtractionRequest
- models/__init__.py: keep grouped SectionCompare + FactExtraction exports
- structured_wiki.py: keep full implementation
- fact_extraction.py: keep richer HEAD implementation
- fact_extraction_models.json: keep expanded model list with full metadata
All 15 backend tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update symmetry-unified-backend/app/services/section_comparison.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update desktop-electron-frontend/src/components/StructuredArticleViewer.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Isaiah-Milkey <isamilkey@gmail.com>
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: AruefliASU <aruefli@asu.edu>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Rename a/b variable names to original/translated for consistency
Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/11f337b7-3983-49a3-a358-331a5a6d29d4
Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com>
* Refactor compareArticles function - remove unnecessary try block and streamline axios instance retrieval
* Move model names to config.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add attributes to the API - (model name and similarity threshold) (#16)
* Update comparison.py
* Update comparison.py
* Update response.py
---------
Co-authored-by: François Pelletier <franc00018@users.noreply.github.com>
* Use DEFAULT_MODEL and safe .get() for comparison keys in semantic_comparison
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Include comparisons in CompareResponse and return raw comparisons when available
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix loanword suffix list and adjust family thresholds
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* GitHub actions implement (#18)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: unify threshold configuration across all comparison paths (#20)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
* fix: unify threshold configuration across all comparison paths
- Add app/core/settings.py as single source of truth for all thresholds,
reading values from .env with sensible defaults
- section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN
(0.08) and similarity_threshold default (0.65) with settings imports
- similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70)
and band tuples with settings imports
- semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75)
with SIMILARITY_THRESHOLD from settings (consistent with config.json)
- Add GET /config/thresholds endpoint for runtime threshold inspection
- Extend config.json and .env.template with all new threshold keys
Closes #19
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: unify threshold configuration across all comparison paths (#21)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
* fix: unify threshold configuration across all comparison paths
- Add app/core/settings.py as single source of truth for all thresholds,
reading values from .env with sensible defaults
- section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN
(0.08) and similarity_threshold default (0.65) with settings imports
- similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70)
and band tuples with settings imports
- semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75)
with SIMILARITY_THRESHOLD from settings (consistent with config.json)
- Add GET /config/thresholds endpoint for runtime threshold inspection
- Extend config.json and .env.template with all new threshold keys
Closes #19
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: frontend fetches thresholds from backend; strip threshold fields from config.json
- Add thresholdService.ts: calls GET /config/thresholds with in-memory cache
and graceful fallback to defaults if backend is unreachable
- ComparisonSection.tsx: replace hardcoded 0.65 with value fetched from
thresholdService on mount
- config.json: remove SIMILARITY_THRESHOLD, LEVENSHTEIN_DISAMBIGUATION_MARGIN,
FAMILY_THRESHOLD_*, COMPARISON_MODELS, DEFAULT_MODEL — these are now owned
by the backend (.env / app/core/settings.py) and served via GET /config/thresholds
config.json retains only infrastructure fields (BACKEND_BASE_URL, ports, timeout)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update desktop-electron-frontend/src/services/thresholdService.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update config.json with model configurations
Added comparison models and default model to config.
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* edited model options and ability to paste in model name. May need to implement with Albert's HF token code
* Added new UI features integrated with Fact Extraction.
* Added cache management for extraction models to avoid OOM errors.
* Refactor code for improved readability and consistency in fact extraction module and tests
* Update Completed
Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py
Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks
Kept only input tensor device transfer inside the loop
This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work.
* Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency
* Refactor structured_wiki.py for improved readability and consistency in imports and function formatting
* fix: remove duplicate useState declarations from merge conflict in StructuredArticleViewer
* style: format parameters in test_compare_semantic_get_invalid_threshold and test_wiki_translate_missing_params for improved readability
* fix: restore spaCy sentence segmentation lost during rebase --ours resolution
* Fact Extraction Implementation (#23)
* implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend.
* Able to download model weights from huggingface, but model output varies in quality.
* implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts.
* edited model options and ability to paste in model name. May need to implement with Albert's HF token code
* Added new UI features integrated with Fact Extraction.
* Added cache management for extraction models to avoid OOM errors.
* Refactor code for improved readability and consistency in fact extraction module and tests
* Implement spaCy sentence segmentation for improved accuracy and add corresponding tests
* Update Completed
Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py
Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks
Kept only input tensor device transfer inside the loop
This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work.
* Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency
* Refactor structured_wiki.py for improved readability and consistency in imports and function formatting
* fix: remove duplicate useState declarations causing CI build failure
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: François Pelletier <franc00018@users.noreply.github.com>
* docs: update CLAUDE.md with changelog conventions and LLM agent instructions
* refactor: clean up whitespace and formatting in various files
* Refactor configuration management and model loading
- Removed fact_extraction_models.json and translation_models.json, consolidating their contents into a new config.toml file.
- Introduced a load_config function to read the backend configuration from the TOML file.
- Updated fact extraction and translation logic to utilize the new configuration structure.
- Replaced JSON file handling with dictionary-based management for saved models in server_model.py.
- Added support for loading and saving configurations in a structured manner.
- Updated requirements.txt to include sentencepiece for improved model handling.
* fix: address review comments for PR #24
- Lower TRANSLATION_CHUNK_CHAR_THRESHOLD 3500->1500 and TRANSLATION_CHUNK_WORD_SIZE 450->300
to prevent silent MarianMT token-limit truncation (Gemini suggestion)
- Reduce lru_cache maxsize 32->4 on load_translation_components to avoid
OOM from caching too many large models (Gemini + Copilot feedback)
- Move 'from app.ai.translations import translate' inside try block so
ImportError is consistently caught and mapped to HTTP 500 (Copilot feedback)
- Add AbortController to TranslationSection.onLanguageChange; abort on Clear
and guard post-await state updates so stale responses are discarded (Copilot feedback)
- Add translateArticle signal parameter (Axios AbortSignal support)
- Add 3 router-level tests for /wiki_translate/chunked_text: success,
ValueError->400, and ImportError->500 (Copilot feedback)
* fix: resolve SyntaxError in config.py - f-string backslash incompatibility with Python < 3.12
Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/dc80d503-f43a-4eaf-85bf-05cbc8dcfa06
Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com>
* gitignore: ignore frontend dist build artifacts
* refactor: update Python version in start.sh and enhance async handling in fact extraction
* Similarity comparison 3.0 (#33)
* Integrate similarity comparator (Phase1/2/3)
* connected similarity threshold to frontend and added progress bar
* Enhance ComparisonSection with side-by-side view
Refactor comparison section to include side-by-side view and handle extra sentences in target.
Visual highlighting:
Red strikethrough for sentences missing in the target
Green underline for extra sentences in the target
Gray dashes (—) when there's no corresponding sentence
* Add legend for comparison results
Added a legend to the Comparison Results section to explain missing and extra information.
* fixed some UI elements
* ui: add article presets + threshold presets + helper text
* Update symmetry-unified-backend/app/ai/semantic_comparison.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update symmetry-unified-backend/app/services/similarity_prototype/Phase_1/similarity.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Refactor SynonymMatcher and add tests for POS caching
- Updated the SynonymMatcher class to improve POS tagging efficiency by caching results.
- Added a new method `_normalize_word` to standardize word inputs.
- Enhanced the `build_sentence_pos_cache` method to utilize sentence context for tagging.
- Introduced tests to verify the functionality of POS caching and ensure tokens are not repeatedly tagged.
- Cleaned up code formatting for better readability across multiple files.
* feat: add configuration options for comparison models, article presets, and threshold presets
* style: format code for better readability in README and main.ts
---------
Co-authored-by: Jameela Kauser <kj@Jameelas-MacBook-Air.local>
Co-authored-by: Adrian Medina <adrianedwardmedina@gmail.com>
Co-authored-by: AruefliASU <aruefli@asu.edu>
Co-authored-by: Adrian Medina <119469294+AdrianMedina59@users.noreply.github.com>
Co-authored-by: Asmi Kachare <asmikachare@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com>
Co-authored-by: François Pelletier <franc00018@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: François Pelletier <francois@jevalide.ca>
* Revision flagging, language lag, and revision diff features (#32)
* Add revision flagging feature
New files:
- app/models/revision.py — Revision, SectionDiff, Flag, DiffResponse models
- app/services/revision_flagging.py — flag_revision() with 4 rules:
high_volume_change, section_removed, lead_section_modified,
rapid_successive_edits
- tests/test_revision_flagging.py — 26 unit tests (26/26 passing)
- test_flagging.py — manual integration script for live Wikipedia testing,
pre-loaded with 2026 Iran war example
Modified files:
- app/models/__init__.py — exports new models
- app/routers/structured_wiki.py — adds GET /revision-history and
GET /revision-diff endpoints with optional include_flags parameter
* Add language lag detection feature
Adds get_latest_revision_timestamp() and detect_language_lag() to wiki_utils.py,
LagReport model to revision.py, and wires up the /lag endpoint in structured_wiki.py.
* Add revision-to-revision diff endpoint (/diff)
Extracts _parse_article_html() helper and adds revision_fetcher() to article_parser.py.
Adds similarity_scoring.py with score_article_pair(). Adds SectionChange and
RevisionDiffResponse models. Adds /diff endpoint returning sections_added,
sections_removed, sections_modified, and overall_similarity. Removes the older
/revision-diff endpoint in favour of the cleaner /diff shape.
* WIP: semantic comparison + chunking updates (still working on fixes)
* chunking updates
* Deleted symmetry-unified-backend/tests/tempCodeRunnerFile.py
* Update Dockerfile
Update to Docker that has model baked in
* Update translations.py
Changed dict for language models for an easily maintainable json file. Kept local romance lang dict for fallback on edge cases.
* json for translation models
A json file that contains list of languages models. Each model has a source lang, target lang, similarity threshold, and model_name.
* Update translation_models.json
Added similarity scores from similarity_scoring.py
* Update translations.py
Improved language model search and added caching for reusing same language model.
* align frontend with backend API and fix launch issues when running in web mode
* fix language by using a package instead of a list
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update semantic_comparison.py (#7)
New code for semantic comparison that includes a cache system
* Added translation feature to structured article (#5)
* Created new button in structuredarticle viewer to translate article using Alberts translate code.
Also removed search function in structured-article because I believe it did not function.
* added Helsinki model languages
* Re-added search within sections functionality
* Replace hardcoded languages with dynamic language display names library.
* Added comments explaining Null citations/references for future use
* Created new button in structuredarticle viewer to translate article using Alberts translate code.
Also removed search function in structured-article because I believe it did not function.
* added Helsinki model languages
* Re-added search within sections functionality
* Replace hardcoded languages with dynamic language display names library.
* Added comments explaining Null citations/references for future use
* quick fix with newline
* updated docker-compose.yml - installation of spacy models (#11) (#13)
* updated docker-compose.yml - installation of spacy models
adds volume to install spaCy models
* Refactor command to download Spacy models
Fixed high security vulnerability
---------
Co-authored-by: AruefliASU <aruefli@asu.edu>
* uodate start script
* Summary of Changes (#17)
* implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend.
* Able to download model weights from huggingface, but model output varies in quality.
* implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts.
* Summary of Changes
1. Tab Restructure (Structured Article first, legacy tabs labeled)
- src/pages/Home.tsx - Structured Article is now the default/first tab. Translation and AI Comparison tabs are labeled "(Legacy)".
- src/components/Layout.tsx - Removed 85 lines of hardcoded Biden test data, unused getColorClass, unused imports. Reduced from 152 to 17 lines.
- src/components/PageHeader.tsx - Removed hardcoded "Hi there, Suraj", non-functional language selector. Now shows "Cross-Language Article Analysis" with a descriptive subtitle.
2. Section-Level Semantic Comparison (new feature)
Backend:
- app/models/section_comparison.py (new) - Pydantic models: SectionCompareRequest, ParagraphDiff, SectionDiff, SectionCompareResponse
- app/services/section_comparison.py (new) - Core comparison engine: matches sections using LaBSE embeddings (greedy best-match on title + content preview), then compares paragraphs within matched sections. Uses Levenshtein disambiguation when top-2 semantic candidates are within 0.08 margin.
- app/routers/comparison.py - Added POST /articles/compare-sections endpoint with URL/title parsing
Frontend:
- src/components/SectionComparisonView.tsx (new) - Full diff UI with expandable section cards, paragraph side-by-side view, similarity bars, color-coded status badges, and filter controls
- src/services/structuredWikiService.ts - Added compareSections() method (POST)
- src/models/structured-wiki.ts - Added TypeScript interfaces for ParagraphDiff, SectionDiff, SectionCompareResponse, SectionCompareRequest
- src/components/StructuredArticleViewer.tsx - Added "Compare Sections" button with language selector, toggles between article view and comparison view
3. Bug Fixes
- app/ai/translations.py - Was completely empty (runtime crash). Now wraps translate_text() from translation.py with graceful error handling.
- app/ai/translation.py - Removed commented-out Google Colab interactive code.
- tests/test_translation_limits.py - Updated test_unsupported_language to match new graceful fallback behavior.
4. Dead Code Removal
Deleted files (7):
- src/models/enums/FetchArticleResponse.ts (duplicate)
- src/models/FetchArticleRequest.ts (duplicate, unused)
- src/models/apis/TranslateArticleRequest.ts (unused)
- src/models/apis/FetchArticleRequest.ts (unused)
- app/ai/dataset/dataset_gen.py (standalone experimental script)
- app/main.spec (PyInstaller build artifact)
- app/temp_backend_run.sh (developer temp script)
Cleaned inline dead code:
- app/models/comparison.py - Removed all commented-out old model versions, added docstrings
- app/models/api_models.py - Removed 3 unused shadow models (ComparisonResponse, TranslationResponse, ArticleResponse)
- app/models/comparison_request.py - Removed empty LLMCompareRequest class
- app/ai/semantic_comparison.py - Removed test main() function
5. Deduplication & Naming
- app/ai/model_registry.py (new) - Single source of truth for COMPARISON_MODELS and DEFAULT_MODEL. Previously duplicated in 3 files.
- app/main.py, app/routers/comparison.py, app/ai/semantic_comparison.py - All now import from model_registry.py
- app/ai/semantic_comparison.py - Renamed eng_embedding → embedding, first_embeddings/second_embeddings → source_embeddings/reference_embeddings, sim_threshold → similarity_threshold, diff_info → unmatched_sentences
- src/services/compareArticles.ts - Renamed parameters from originalArticleContent → sourceArticleContent, removed debug console.log statements
- src/components/TranslationSection.tsx - Replaced opaque texts/editing/reference/suggestedContribution/suggestionType with typed ArticleDisplayBlock interface with clear label, content, displayType fields
6. Documentation
- README.md - Complete rewrite reflecting the new architecture, section comparison feature, and cleaned project structure
- CLAUDE.md - Updated agent instructions matching the current codebase
Verification
- Backend: 143 passed, 7 failed (all pre-existing). One previously-failing test fixed.
- Frontend: 0 new TypeScript errors (5 pre-existing in Electron boilerplate files).
* Update wiki_structure.py - update reference id max size to 300 (#15)
Fix bug that can cause internal server error while running backend.
I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine
* Use canonical field names in CompareRequest (original_article_content etc.)
Replace text_a/text_b/language_a/language_b with meaningful names:
- original_article_content (alias: article_text_blob_1)
- translated_article_content (alias: article_text_blob_2)
- original_language (alias: article_text_blob_1_language)
- translated_language (alias: article_text_blob_2_language)
Update comparison.py router to use new field names when building
request_data for the semantic comparison engine.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Eliminate aliases: use original/translated names everywhere
Remove all AliasChoices from CompareRequest and BaseCompareRequest.
Use canonical field names end-to-end across backend and tests:
- original_article_content / translated_article_content
- original_language / translated_language
Also update:
- semantic_comparison.py: request_data keys use canonical names
- comparison.py router: GET query params renamed, request_data dicts updated
- comparison_request.py: BaseCompareRequest fields renamed
- compareArticles.ts: resolve remaining conflict marker
- Tests: fixtures and request dicts updated to canonical names
- Add ge/le bounds back to similarity_threshold in CompareRequest
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Resolve remaining merge conflicts: favor canonical field names
- StructuredArticleViewer.tsx: keep FactExtraction import and full UI
- models/__init__.py: keep FactExtraction exports
- comparison.py / comparison_request.py: canonical original/translated names
- semantic_comparison.py: canonical key names in request_data
- api_models.py: keep FactExtractionRequest/Response classes
- compareArticles.ts: clean canonical params, no debug cruft
All 15 backend tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Resolve merge conflicts: keep HEAD (canonical names + richer features)
Files resolved:
- FactExtraction.ts: keep num_facts field
- compareArticles.ts: keep canonical sourceArticleContent/sourceLanguage params
- StructuredArticleViewer.tsx: keep full fact-extraction + section-comparison UI
- api_models.py: keep section_title and num_facts in FactExtractionRequest
- models/__init__.py: keep grouped SectionCompare + FactExtraction exports
- structured_wiki.py: keep full implementation
- fact_extraction.py: keep richer HEAD implementation
- fact_extraction_models.json: keep expanded model list with full metadata
All 15 backend tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update symmetry-unified-backend/app/services/section_comparison.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update desktop-electron-frontend/src/components/StructuredArticleViewer.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: Isaiah-Milkey <isamilkey@gmail.com>
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: AruefliASU <aruefli@asu.edu>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update wiki_structure.py - update reference id max size to 300 (#15)
Fix bug that can cause internal server error while running backend.
I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine
* Rename a/b variable names to original/translated for consistency
Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/11f337b7-3983-49a3-a358-331a5a6d29d4
Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com>
* Refactor compareArticles function - remove unnecessary try block and streamline axios instance retrieval
* Move model names to config.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add attributes to the API - (model name and similarity threshold) (#16)
* Update comparison.py
* Update comparison.py
* Update response.py
---------
Co-authored-by: François Pelletier <franc00018@users.noreply.github.com>
* Use DEFAULT_MODEL and safe .get() for comparison keys in semantic_comparison
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Include comparisons in CompareResponse and return raw comparisons when available
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix loanword suffix list and adjust family thresholds
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* GitHub actions implement (#18)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
---------
Co-authored-by: François Pelletier <francois@jevalide.ca>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add regression tests for long reference ID in wiki_structure
Regression test for bug where Reference.id had max_length=100, causing
500 errors on articles with long reference IDs (e.g. Despre_tine).
The fix (removing max_length constraints) was already applied in bf3803f.
These tests guard against reintroduction of the constraint.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: unify threshold configuration across all comparison paths (#20)
* first version of GitHub Actions
* ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script
- CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration
- Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR
- Added .dockerignore files
- Added build:web script to frontend package.json
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx
Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add environment variable to CI and release workflows for Node.js compatibility
* fix: unify threshold configuration across all comparison paths
- Add app/core/settings.py as single source of truth for all thresholds,
reading values from .env with sensible defaults
- section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN
(0.08) and similarity_threshold default (0.65) with settings imports
- similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70)
and band tuples with settings imports
- semantic_c…1 parent 50b318d commit 6e23a43
139 files changed
Lines changed: 17926 additions & 7391 deletions
File tree
- .claude
- .github/workflows
- desktop-electron-frontend
- artifacts
- scripts
- src
- components
- constants
- context
- models
- pages
- services
- tests/e2e
- docs
- backend
- frontend
- symmetry-unified-backend
- app
- ai
- models
- comparison
- extraction
- translation
- wiki
- routers
- services
- similarity_prototype
- Phase_1
- Phase_2
- Phase_3
- tests
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
92 | | - | |
| 92 | + | |
93 | 93 | | |
94 | | - | |
| 94 | + | |
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
111 | | - | |
| 111 | + | |
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
71 | | - | |
| 71 | + | |
72 | 72 | | |
73 | 73 | | |
74 | | - | |
| 74 | + | |
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
| 82 | + | |
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
5 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
6 | 21 | | |
7 | | - | |
8 | | - | |
9 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
10 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
0 commit comments