Skip to content

Commit 6e23a43

Browse files
franc00018Jameela KauserAdrianMedina59AruefliASUasmikachare
authored
feat: promote develop to main — v1.1.0 (#43)
* Initial commit: comparison API with semantic diff * implmented similarity scoring algorithm and testing file * update libraries versions in both python and typescript, and several small bug fixes. Added separate startup scripts not requiring the electron app. * feat: Unified startup system and improved architecture ## Infrastructure - Add unified start.sh script for managing both local and Docker deployments - Add docker-compose.yml for container orchestration - Add Dockerfiles for frontend (Node 18) and backend (Python 3.13) - Remove obsolete startup scripts (restart_app.sh, start_*.sh) - Add /datasets/ directory and move dataset generation tools out of backend ## Backend Improvements - Merge new semantic_comparison.py with enhanced error handling and flexible model loading - Remove LLM comparison functionality (app/ai/llm_comparison.py) - Add model management router (/app/routers/models.py) with select/delete/import/list endpoints - Add /models/translation/* and /models/comparison/* API endpoints - Fix semantic comparison default model to use full identifier 'sentence-transformers/LaBSE' - Update comparison API to support new model management ## Frontend Improvements - Remove all backend lifecycle management from main.ts (kill/start processes) - Add health check IPC handler calling /health endpoint - Update TranslationSection to use /health endpoint for status checks - Remove startBackend function and 'Start Backend' button from UI - Fix compareArticles service to send correct model name 'sentence-transformers/LaBSE' ## Testing - Fix all 4 failing tests (mock patches and cache handling) - Achieve 100% test pass rate (56/56 tests passing) - Rename semantic_comparison_tests.py to test_semantic_comparison.py - Convert to proper pytest format with test_* functions - Fix mock data for translation tests (add 'fr' language support) ## Documentation - Update README.md with comprehensive start.sh usage documentation - Add Docker Compose workflow documentation - Update API endpoints list (remove LLM comparison, add model management) - Update test statistics (56 tests, 100% passing) - Update backend and frontend README files - Fix backend path references in frontend documentation ## Architecture - Frontend now only monitors backend health, does not manage lifecycle - Backend lifecycle managed exclusively by start.sh script - Both local and Docker workflows use same start.sh interface - Services are independent with proper separation of concerns This consolidates 4 startup scripts into 1 unified system with full Docker support and removes all LLM comparison code in favor of semantic comparison only. * Updated router documentation for the backend * Added similarity threshold in the frontend app * Updated start script to force a Python 3.13 environment * update global gitignore * Update translations.py translate function to translate strings from source language to target language. Includes model dictionary for direct translations and dictionary for romance languages for fallback if a romance language does not have a direct language model. * added test for translation // also added comments * Update structured_wiki.py - structured translated article Added router for creating a structured translated article from a structured article. Includes imports for new translation function. * Update structured_wiki.py Added router for translated wiki articles. Includes try/except structures to provide feedback for crashes * WIP: semantic comparison + chunking updates (still working on fixes) * chunking updates * Deleted symmetry-unified-backend/tests/tempCodeRunnerFile.py * Update Dockerfile Update to Docker that has model baked in * Update translations.py Changed dict for language models for an easily maintainable json file. Kept local romance lang dict for fallback on edge cases. * json for translation models A json file that contains list of languages models. Each model has a source lang, target lang, similarity threshold, and model_name. * Update translation_models.json Added similarity scores from similarity_scoring.py * Update translations.py Improved language model search and added caching for reusing same language model. * align frontend with backend API and fix launch issues when running in web mode * fix language by using a package instead of a list * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update semantic_comparison.py (#7) New code for semantic comparison that includes a cache system * Added translation feature to structured article (#5) * Created new button in structuredarticle viewer to translate article using Alberts translate code. Also removed search function in structured-article because I believe it did not function. * added Helsinki model languages * Re-added search within sections functionality * Replace hardcoded languages with dynamic language display names library. * Added comments explaining Null citations/references for future use * Created new button in structuredarticle viewer to translate article using Alberts translate code. Also removed search function in structured-article because I believe it did not function. * added Helsinki model languages * Re-added search within sections functionality * Replace hardcoded languages with dynamic language display names library. * Added comments explaining Null citations/references for future use * quick fix with newline * updated docker-compose.yml - installation of spacy models (#11) (#13) * updated docker-compose.yml - installation of spacy models adds volume to install spaCy models * Refactor command to download Spacy models Fixed high security vulnerability --------- Co-authored-by: AruefliASU <aruefli@asu.edu> * uodate start script * chunking updates * Summary of Changes (#17) * implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend. * Able to download model weights from huggingface, but model output varies in quality. * implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts. * Summary of Changes 1. Tab Restructure (Structured Article first, legacy tabs labeled) - src/pages/Home.tsx - Structured Article is now the default/first tab. Translation and AI Comparison tabs are labeled "(Legacy)". - src/components/Layout.tsx - Removed 85 lines of hardcoded Biden test data, unused getColorClass, unused imports. Reduced from 152 to 17 lines. - src/components/PageHeader.tsx - Removed hardcoded "Hi there, Suraj", non-functional language selector. Now shows "Cross-Language Article Analysis" with a descriptive subtitle. 2. Section-Level Semantic Comparison (new feature) Backend: - app/models/section_comparison.py (new) - Pydantic models: SectionCompareRequest, ParagraphDiff, SectionDiff, SectionCompareResponse - app/services/section_comparison.py (new) - Core comparison engine: matches sections using LaBSE embeddings (greedy best-match on title + content preview), then compares paragraphs within matched sections. Uses Levenshtein disambiguation when top-2 semantic candidates are within 0.08 margin. - app/routers/comparison.py - Added POST /articles/compare-sections endpoint with URL/title parsing Frontend: - src/components/SectionComparisonView.tsx (new) - Full diff UI with expandable section cards, paragraph side-by-side view, similarity bars, color-coded status badges, and filter controls - src/services/structuredWikiService.ts - Added compareSections() method (POST) - src/models/structured-wiki.ts - Added TypeScript interfaces for ParagraphDiff, SectionDiff, SectionCompareResponse, SectionCompareRequest - src/components/StructuredArticleViewer.tsx - Added "Compare Sections" button with language selector, toggles between article view and comparison view 3. Bug Fixes - app/ai/translations.py - Was completely empty (runtime crash). Now wraps translate_text() from translation.py with graceful error handling. - app/ai/translation.py - Removed commented-out Google Colab interactive code. - tests/test_translation_limits.py - Updated test_unsupported_language to match new graceful fallback behavior. 4. Dead Code Removal Deleted files (7): - src/models/enums/FetchArticleResponse.ts (duplicate) - src/models/FetchArticleRequest.ts (duplicate, unused) - src/models/apis/TranslateArticleRequest.ts (unused) - src/models/apis/FetchArticleRequest.ts (unused) - app/ai/dataset/dataset_gen.py (standalone experimental script) - app/main.spec (PyInstaller build artifact) - app/temp_backend_run.sh (developer temp script) Cleaned inline dead code: - app/models/comparison.py - Removed all commented-out old model versions, added docstrings - app/models/api_models.py - Removed 3 unused shadow models (ComparisonResponse, TranslationResponse, ArticleResponse) - app/models/comparison_request.py - Removed empty LLMCompareRequest class - app/ai/semantic_comparison.py - Removed test main() function 5. Deduplication & Naming - app/ai/model_registry.py (new) - Single source of truth for COMPARISON_MODELS and DEFAULT_MODEL. Previously duplicated in 3 files. - app/main.py, app/routers/comparison.py, app/ai/semantic_comparison.py - All now import from model_registry.py - app/ai/semantic_comparison.py - Renamed eng_embedding → embedding, first_embeddings/second_embeddings → source_embeddings/reference_embeddings, sim_threshold → similarity_threshold, diff_info → unmatched_sentences - src/services/compareArticles.ts - Renamed parameters from originalArticleContent → sourceArticleContent, removed debug console.log statements - src/components/TranslationSection.tsx - Replaced opaque texts/editing/reference/suggestedContribution/suggestionType with typed ArticleDisplayBlock interface with clear label, content, displayType fields 6. Documentation - README.md - Complete rewrite reflecting the new architecture, section comparison feature, and cleaned project structure - CLAUDE.md - Updated agent instructions matching the current codebase Verification - Backend: 143 passed, 7 failed (all pre-existing). One previously-failing test fixed. - Frontend: 0 new TypeScript errors (5 pre-existing in Electron boilerplate files). * Update wiki_structure.py - update reference id max size to 300 (#15) Fix bug that can cause internal server error while running backend. I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine * Use canonical field names in CompareRequest (original_article_content etc.) Replace text_a/text_b/language_a/language_b with meaningful names: - original_article_content (alias: article_text_blob_1) - translated_article_content (alias: article_text_blob_2) - original_language (alias: article_text_blob_1_language) - translated_language (alias: article_text_blob_2_language) Update comparison.py router to use new field names when building request_data for the semantic comparison engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Eliminate aliases: use original/translated names everywhere Remove all AliasChoices from CompareRequest and BaseCompareRequest. Use canonical field names end-to-end across backend and tests: - original_article_content / translated_article_content - original_language / translated_language Also update: - semantic_comparison.py: request_data keys use canonical names - comparison.py router: GET query params renamed, request_data dicts updated - comparison_request.py: BaseCompareRequest fields renamed - compareArticles.ts: resolve remaining conflict marker - Tests: fixtures and request dicts updated to canonical names - Add ge/le bounds back to similarity_threshold in CompareRequest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve remaining merge conflicts: favor canonical field names - StructuredArticleViewer.tsx: keep FactExtraction import and full UI - models/__init__.py: keep FactExtraction exports - comparison.py / comparison_request.py: canonical original/translated names - semantic_comparison.py: canonical key names in request_data - api_models.py: keep FactExtractionRequest/Response classes - compareArticles.ts: clean canonical params, no debug cruft All 15 backend tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve merge conflicts: keep HEAD (canonical names + richer features) Files resolved: - FactExtraction.ts: keep num_facts field - compareArticles.ts: keep canonical sourceArticleContent/sourceLanguage params - StructuredArticleViewer.tsx: keep full fact-extraction + section-comparison UI - api_models.py: keep section_title and num_facts in FactExtractionRequest - models/__init__.py: keep grouped SectionCompare + FactExtraction exports - structured_wiki.py: keep full implementation - fact_extraction.py: keep richer HEAD implementation - fact_extraction_models.json: keep expanded model list with full metadata All 15 backend tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update symmetry-unified-backend/app/services/section_comparison.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update desktop-electron-frontend/src/components/StructuredArticleViewer.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Isaiah-Milkey <isamilkey@gmail.com> Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: AruefliASU <aruefli@asu.edu> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update wiki_structure.py - update reference id max size to 300 (#15) Fix bug that can cause internal server error while running backend. I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine * Rename a/b variable names to original/translated for consistency Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/11f337b7-3983-49a3-a358-331a5a6d29d4 Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com> * Refactor compareArticles function - remove unnecessary try block and streamline axios instance retrieval * Move model names to config.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add attributes to the API - (model name and similarity threshold) (#16) * Update comparison.py * Update comparison.py * Update response.py --------- Co-authored-by: François Pelletier <franc00018@users.noreply.github.com> * Use DEFAULT_MODEL and safe .get() for comparison keys in semantic_comparison Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include comparisons in CompareResponse and return raw comparisons when available Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix loanword suffix list and adjust family thresholds Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * GitHub actions implement (#18) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: unify threshold configuration across all comparison paths (#20) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility * fix: unify threshold configuration across all comparison paths - Add app/core/settings.py as single source of truth for all thresholds, reading values from .env with sensible defaults - section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN (0.08) and similarity_threshold default (0.65) with settings imports - similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70) and band tuples with settings imports - semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75) with SIMILARITY_THRESHOLD from settings (consistent with config.json) - Add GET /config/thresholds endpoint for runtime threshold inspection - Extend config.json and .env.template with all new threshold keys Closes #19 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: unify threshold configuration across all comparison paths (#21) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility * fix: unify threshold configuration across all comparison paths - Add app/core/settings.py as single source of truth for all thresholds, reading values from .env with sensible defaults - section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN (0.08) and similarity_threshold default (0.65) with settings imports - similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70) and band tuples with settings imports - semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75) with SIMILARITY_THRESHOLD from settings (consistent with config.json) - Add GET /config/thresholds endpoint for runtime threshold inspection - Extend config.json and .env.template with all new threshold keys Closes #19 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: frontend fetches thresholds from backend; strip threshold fields from config.json - Add thresholdService.ts: calls GET /config/thresholds with in-memory cache and graceful fallback to defaults if backend is unreachable - ComparisonSection.tsx: replace hardcoded 0.65 with value fetched from thresholdService on mount - config.json: remove SIMILARITY_THRESHOLD, LEVENSHTEIN_DISAMBIGUATION_MARGIN, FAMILY_THRESHOLD_*, COMPARISON_MODELS, DEFAULT_MODEL — these are now owned by the backend (.env / app/core/settings.py) and served via GET /config/thresholds config.json retains only infrastructure fields (BACKEND_BASE_URL, ports, timeout) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update desktop-electron-frontend/src/services/thresholdService.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update config.json with model configurations Added comparison models and default model to config. --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * edited model options and ability to paste in model name. May need to implement with Albert's HF token code * Added new UI features integrated with Fact Extraction. * Added cache management for extraction models to avoid OOM errors. * Refactor code for improved readability and consistency in fact extraction module and tests * Update Completed Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks Kept only input tensor device transfer inside the loop This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work. * Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency * Refactor structured_wiki.py for improved readability and consistency in imports and function formatting * fix: remove duplicate useState declarations from merge conflict in StructuredArticleViewer * style: format parameters in test_compare_semantic_get_invalid_threshold and test_wiki_translate_missing_params for improved readability * fix: restore spaCy sentence segmentation lost during rebase --ours resolution * Fact Extraction Implementation (#23) * implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend. * Able to download model weights from huggingface, but model output varies in quality. * implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts. * edited model options and ability to paste in model name. May need to implement with Albert's HF token code * Added new UI features integrated with Fact Extraction. * Added cache management for extraction models to avoid OOM errors. * Refactor code for improved readability and consistency in fact extraction module and tests * Implement spaCy sentence segmentation for improved accuracy and add corresponding tests * Update Completed Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks Kept only input tensor device transfer inside the loop This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work. * Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency * Refactor structured_wiki.py for improved readability and consistency in imports and function formatting * fix: remove duplicate useState declarations causing CI build failure --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: François Pelletier <franc00018@users.noreply.github.com> * docs: update CLAUDE.md with changelog conventions and LLM agent instructions * Added OpenRouter free model functioanlity for fact extraction and visual updates * progress indicator for translation * updated docker-compose.yml - installation of spacy models (#11) (#13) * updated docker-compose.yml - installation of spacy models adds volume to install spaCy models * Refactor command to download Spacy models Fixed high security vulnerability --------- Co-authored-by: AruefliASU <aruefli@asu.edu> * Summary of Changes (#17) * implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend. * Able to download model weights from huggingface, but model output varies in quality. * implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts. * Summary of Changes 1. Tab Restructure (Structured Article first, legacy tabs labeled) - src/pages/Home.tsx - Structured Article is now the default/first tab. Translation and AI Comparison tabs are labeled "(Legacy)". - src/components/Layout.tsx - Removed 85 lines of hardcoded Biden test data, unused getColorClass, unused imports. Reduced from 152 to 17 lines. - src/components/PageHeader.tsx - Removed hardcoded "Hi there, Suraj", non-functional language selector. Now shows "Cross-Language Article Analysis" with a descriptive subtitle. 2. Section-Level Semantic Comparison (new feature) Backend: - app/models/section_comparison.py (new) - Pydantic models: SectionCompareRequest, ParagraphDiff, SectionDiff, SectionCompareResponse - app/services/section_comparison.py (new) - Core comparison engine: matches sections using LaBSE embeddings (greedy best-match on title + content preview), then compares paragraphs within matched sections. Uses Levenshtein disambiguation when top-2 semantic candidates are within 0.08 margin. - app/routers/comparison.py - Added POST /articles/compare-sections endpoint with URL/title parsing Frontend: - src/components/SectionComparisonView.tsx (new) - Full diff UI with expandable section cards, paragraph side-by-side view, similarity bars, color-coded status badges, and filter controls - src/services/structuredWikiService.ts - Added compareSections() method (POST) - src/models/structured-wiki.ts - Added TypeScript interfaces for ParagraphDiff, SectionDiff, SectionCompareResponse, SectionCompareRequest - src/components/StructuredArticleViewer.tsx - Added "Compare Sections" button with language selector, toggles between article view and comparison view 3. Bug Fixes - app/ai/translations.py - Was completely empty (runtime crash). Now wraps translate_text() from translation.py with graceful error handling. - app/ai/translation.py - Removed commented-out Google Colab interactive code. - tests/test_translation_limits.py - Updated test_unsupported_language to match new graceful fallback behavior. 4. Dead Code Removal Deleted files (7): - src/models/enums/FetchArticleResponse.ts (duplicate) - src/models/FetchArticleRequest.ts (duplicate, unused) - src/models/apis/TranslateArticleRequest.ts (unused) - src/models/apis/FetchArticleRequest.ts (unused) - app/ai/dataset/dataset_gen.py (standalone experimental script) - app/main.spec (PyInstaller build artifact) - app/temp_backend_run.sh (developer temp script) Cleaned inline dead code: - app/models/comparison.py - Removed all commented-out old model versions, added docstrings - app/models/api_models.py - Removed 3 unused shadow models (ComparisonResponse, TranslationResponse, ArticleResponse) - app/models/comparison_request.py - Removed empty LLMCompareRequest class - app/ai/semantic_comparison.py - Removed test main() function 5. Deduplication & Naming - app/ai/model_registry.py (new) - Single source of truth for COMPARISON_MODELS and DEFAULT_MODEL. Previously duplicated in 3 files. - app/main.py, app/routers/comparison.py, app/ai/semantic_comparison.py - All now import from model_registry.py - app/ai/semantic_comparison.py - Renamed eng_embedding → embedding, first_embeddings/second_embeddings → source_embeddings/reference_embeddings, sim_threshold → similarity_threshold, diff_info → unmatched_sentences - src/services/compareArticles.ts - Renamed parameters from originalArticleContent → sourceArticleContent, removed debug console.log statements - src/components/TranslationSection.tsx - Replaced opaque texts/editing/reference/suggestedContribution/suggestionType with typed ArticleDisplayBlock interface with clear label, content, displayType fields 6. Documentation - README.md - Complete rewrite reflecting the new architecture, section comparison feature, and cleaned project structure - CLAUDE.md - Updated agent instructions matching the current codebase Verification - Backend: 143 passed, 7 failed (all pre-existing). One previously-failing test fixed. - Frontend: 0 new TypeScript errors (5 pre-existing in Electron boilerplate files). * Update wiki_structure.py - update reference id max size to 300 (#15) Fix bug that can cause internal server error while running backend. I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine * Use canonical field names in CompareRequest (original_article_content etc.) Replace text_a/text_b/language_a/language_b with meaningful names: - original_article_content (alias: article_text_blob_1) - translated_article_content (alias: article_text_blob_2) - original_language (alias: article_text_blob_1_language) - translated_language (alias: article_text_blob_2_language) Update comparison.py router to use new field names when building request_data for the semantic comparison engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Eliminate aliases: use original/translated names everywhere Remove all AliasChoices from CompareRequest and BaseCompareRequest. Use canonical field names end-to-end across backend and tests: - original_article_content / translated_article_content - original_language / translated_language Also update: - semantic_comparison.py: request_data keys use canonical names - comparison.py router: GET query params renamed, request_data dicts updated - comparison_request.py: BaseCompareRequest fields renamed - compareArticles.ts: resolve remaining conflict marker - Tests: fixtures and request dicts updated to canonical names - Add ge/le bounds back to similarity_threshold in CompareRequest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve remaining merge conflicts: favor canonical field names - StructuredArticleViewer.tsx: keep FactExtraction import and full UI - models/__init__.py: keep FactExtraction exports - comparison.py / comparison_request.py: canonical original/translated names - semantic_comparison.py: canonical key names in request_data - api_models.py: keep FactExtractionRequest/Response classes - compareArticles.ts: clean canonical params, no debug cruft All 15 backend tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve merge conflicts: keep HEAD (canonical names + richer features) Files resolved: - FactExtraction.ts: keep num_facts field - compareArticles.ts: keep canonical sourceArticleContent/sourceLanguage params - StructuredArticleViewer.tsx: keep full fact-extraction + section-comparison UI - api_models.py: keep section_title and num_facts in FactExtractionRequest - models/__init__.py: keep grouped SectionCompare + FactExtraction exports - structured_wiki.py: keep full implementation - fact_extraction.py: keep richer HEAD implementation - fact_extraction_models.json: keep expanded model list with full metadata All 15 backend tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update symmetry-unified-backend/app/services/section_comparison.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update desktop-electron-frontend/src/components/StructuredArticleViewer.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Isaiah-Milkey <isamilkey@gmail.com> Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: AruefliASU <aruefli@asu.edu> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Rename a/b variable names to original/translated for consistency Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/11f337b7-3983-49a3-a358-331a5a6d29d4 Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com> * Refactor compareArticles function - remove unnecessary try block and streamline axios instance retrieval * Move model names to config.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add attributes to the API - (model name and similarity threshold) (#16) * Update comparison.py * Update comparison.py * Update response.py --------- Co-authored-by: François Pelletier <franc00018@users.noreply.github.com> * Use DEFAULT_MODEL and safe .get() for comparison keys in semantic_comparison Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include comparisons in CompareResponse and return raw comparisons when available Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix loanword suffix list and adjust family thresholds Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * GitHub actions implement (#18) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: unify threshold configuration across all comparison paths (#20) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility * fix: unify threshold configuration across all comparison paths - Add app/core/settings.py as single source of truth for all thresholds, reading values from .env with sensible defaults - section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN (0.08) and similarity_threshold default (0.65) with settings imports - similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70) and band tuples with settings imports - semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75) with SIMILARITY_THRESHOLD from settings (consistent with config.json) - Add GET /config/thresholds endpoint for runtime threshold inspection - Extend config.json and .env.template with all new threshold keys Closes #19 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: unify threshold configuration across all comparison paths (#21) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility * fix: unify threshold configuration across all comparison paths - Add app/core/settings.py as single source of truth for all thresholds, reading values from .env with sensible defaults - section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN (0.08) and similarity_threshold default (0.65) with settings imports - similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70) and band tuples with settings imports - semantic_comparison.py: replace hardcoded sim_threshold fallback (0.75) with SIMILARITY_THRESHOLD from settings (consistent with config.json) - Add GET /config/thresholds endpoint for runtime threshold inspection - Extend config.json and .env.template with all new threshold keys Closes #19 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: frontend fetches thresholds from backend; strip threshold fields from config.json - Add thresholdService.ts: calls GET /config/thresholds with in-memory cache and graceful fallback to defaults if backend is unreachable - ComparisonSection.tsx: replace hardcoded 0.65 with value fetched from thresholdService on mount - config.json: remove SIMILARITY_THRESHOLD, LEVENSHTEIN_DISAMBIGUATION_MARGIN, FAMILY_THRESHOLD_*, COMPARISON_MODELS, DEFAULT_MODEL — these are now owned by the backend (.env / app/core/settings.py) and served via GET /config/thresholds config.json retains only infrastructure fields (BACKEND_BASE_URL, ports, timeout) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update desktop-electron-frontend/src/services/thresholdService.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update config.json with model configurations Added comparison models and default model to config. --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * edited model options and ability to paste in model name. May need to implement with Albert's HF token code * Added new UI features integrated with Fact Extraction. * Added cache management for extraction models to avoid OOM errors. * Refactor code for improved readability and consistency in fact extraction module and tests * Update Completed Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks Kept only input tensor device transfer inside the loop This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work. * Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency * Refactor structured_wiki.py for improved readability and consistency in imports and function formatting * fix: remove duplicate useState declarations from merge conflict in StructuredArticleViewer * style: format parameters in test_compare_semantic_get_invalid_threshold and test_wiki_translate_missing_params for improved readability * fix: restore spaCy sentence segmentation lost during rebase --ours resolution * Fact Extraction Implementation (#23) * implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend. * Able to download model weights from huggingface, but model output varies in quality. * implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts. * edited model options and ability to paste in model name. May need to implement with Albert's HF token code * Added new UI features integrated with Fact Extraction. * Added cache management for extraction models to avoid OOM errors. * Refactor code for improved readability and consistency in fact extraction module and tests * Implement spaCy sentence segmentation for improved accuracy and add corresponding tests * Update Completed Moved device = "cuda" if torch.cuda.is_available() else "cpu" and model.to(device) to before the chunk loop in fact_extraction.py Configured tokenizer.pad_token and model.config.pad_token_id once before iterating chunks Kept only input tensor device transfer inside the loop This eliminates repeated model-level operations per chunk and keeps the loop focused on chunk-specific work. * Refactor extract_facts_endpoint to remove async keyword and update requirements by removing llama-cpp-python dependency * Refactor structured_wiki.py for improved readability and consistency in imports and function formatting * fix: remove duplicate useState declarations causing CI build failure --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: François Pelletier <franc00018@users.noreply.github.com> * docs: update CLAUDE.md with changelog conventions and LLM agent instructions * refactor: clean up whitespace and formatting in various files * Refactor configuration management and model loading - Removed fact_extraction_models.json and translation_models.json, consolidating their contents into a new config.toml file. - Introduced a load_config function to read the backend configuration from the TOML file. - Updated fact extraction and translation logic to utilize the new configuration structure. - Replaced JSON file handling with dictionary-based management for saved models in server_model.py. - Added support for loading and saving configurations in a structured manner. - Updated requirements.txt to include sentencepiece for improved model handling. * fix: address review comments for PR #24 - Lower TRANSLATION_CHUNK_CHAR_THRESHOLD 3500->1500 and TRANSLATION_CHUNK_WORD_SIZE 450->300 to prevent silent MarianMT token-limit truncation (Gemini suggestion) - Reduce lru_cache maxsize 32->4 on load_translation_components to avoid OOM from caching too many large models (Gemini + Copilot feedback) - Move 'from app.ai.translations import translate' inside try block so ImportError is consistently caught and mapped to HTTP 500 (Copilot feedback) - Add AbortController to TranslationSection.onLanguageChange; abort on Clear and guard post-await state updates so stale responses are discarded (Copilot feedback) - Add translateArticle signal parameter (Axios AbortSignal support) - Add 3 router-level tests for /wiki_translate/chunked_text: success, ValueError->400, and ImportError->500 (Copilot feedback) * fix: resolve SyntaxError in config.py - f-string backslash incompatibility with Python < 3.12 Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/dc80d503-f43a-4eaf-85bf-05cbc8dcfa06 Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com> * gitignore: ignore frontend dist build artifacts * refactor: update Python version in start.sh and enhance async handling in fact extraction * Similarity comparison 3.0 (#33) * Integrate similarity comparator (Phase1/2/3) * connected similarity threshold to frontend and added progress bar * Enhance ComparisonSection with side-by-side view Refactor comparison section to include side-by-side view and handle extra sentences in target. Visual highlighting: Red strikethrough for sentences missing in the target Green underline for extra sentences in the target Gray dashes (—) when there's no corresponding sentence * Add legend for comparison results Added a legend to the Comparison Results section to explain missing and extra information. * fixed some UI elements * ui: add article presets + threshold presets + helper text * Update symmetry-unified-backend/app/ai/semantic_comparison.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update symmetry-unified-backend/app/services/similarity_prototype/Phase_1/similarity.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Refactor SynonymMatcher and add tests for POS caching - Updated the SynonymMatcher class to improve POS tagging efficiency by caching results. - Added a new method `_normalize_word` to standardize word inputs. - Enhanced the `build_sentence_pos_cache` method to utilize sentence context for tagging. - Introduced tests to verify the functionality of POS caching and ensure tokens are not repeatedly tagged. - Cleaned up code formatting for better readability across multiple files. * feat: add configuration options for comparison models, article presets, and threshold presets * style: format code for better readability in README and main.ts --------- Co-authored-by: Jameela Kauser <kj@Jameelas-MacBook-Air.local> Co-authored-by: Adrian Medina <adrianedwardmedina@gmail.com> Co-authored-by: AruefliASU <aruefli@asu.edu> Co-authored-by: Adrian Medina <119469294+AdrianMedina59@users.noreply.github.com> Co-authored-by: Asmi Kachare <asmikachare@gmail.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com> Co-authored-by: François Pelletier <franc00018@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: François Pelletier <francois@jevalide.ca> * Revision flagging, language lag, and revision diff features (#32) * Add revision flagging feature New files: - app/models/revision.py — Revision, SectionDiff, Flag, DiffResponse models - app/services/revision_flagging.py — flag_revision() with 4 rules: high_volume_change, section_removed, lead_section_modified, rapid_successive_edits - tests/test_revision_flagging.py — 26 unit tests (26/26 passing) - test_flagging.py — manual integration script for live Wikipedia testing, pre-loaded with 2026 Iran war example Modified files: - app/models/__init__.py — exports new models - app/routers/structured_wiki.py — adds GET /revision-history and GET /revision-diff endpoints with optional include_flags parameter * Add language lag detection feature Adds get_latest_revision_timestamp() and detect_language_lag() to wiki_utils.py, LagReport model to revision.py, and wires up the /lag endpoint in structured_wiki.py. * Add revision-to-revision diff endpoint (/diff) Extracts _parse_article_html() helper and adds revision_fetcher() to article_parser.py. Adds similarity_scoring.py with score_article_pair(). Adds SectionChange and RevisionDiffResponse models. Adds /diff endpoint returning sections_added, sections_removed, sections_modified, and overall_similarity. Removes the older /revision-diff endpoint in favour of the cleaner /diff shape. * WIP: semantic comparison + chunking updates (still working on fixes) * chunking updates * Deleted symmetry-unified-backend/tests/tempCodeRunnerFile.py * Update Dockerfile Update to Docker that has model baked in * Update translations.py Changed dict for language models for an easily maintainable json file. Kept local romance lang dict for fallback on edge cases. * json for translation models A json file that contains list of languages models. Each model has a source lang, target lang, similarity threshold, and model_name. * Update translation_models.json Added similarity scores from similarity_scoring.py * Update translations.py Improved language model search and added caching for reusing same language model. * align frontend with backend API and fix launch issues when running in web mode * fix language by using a package instead of a list * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update semantic_comparison.py (#7) New code for semantic comparison that includes a cache system * Added translation feature to structured article (#5) * Created new button in structuredarticle viewer to translate article using Alberts translate code. Also removed search function in structured-article because I believe it did not function. * added Helsinki model languages * Re-added search within sections functionality * Replace hardcoded languages with dynamic language display names library. * Added comments explaining Null citations/references for future use * Created new button in structuredarticle viewer to translate article using Alberts translate code. Also removed search function in structured-article because I believe it did not function. * added Helsinki model languages * Re-added search within sections functionality * Replace hardcoded languages with dynamic language display names library. * Added comments explaining Null citations/references for future use * quick fix with newline * updated docker-compose.yml - installation of spacy models (#11) (#13) * updated docker-compose.yml - installation of spacy models adds volume to install spaCy models * Refactor command to download Spacy models Fixed high security vulnerability --------- Co-authored-by: AruefliASU <aruefli@asu.edu> * uodate start script * Summary of Changes (#17) * implemented dropdown in structured article with huggingface model download. Extract button not working yet, but connects to backend. * Able to download model weights from huggingface, but model output varies in quality. * implemented a rough version of fact extraction. Created basic chunker that splits a passage by number of sentences and desired number of facts. * Summary of Changes 1. Tab Restructure (Structured Article first, legacy tabs labeled) - src/pages/Home.tsx - Structured Article is now the default/first tab. Translation and AI Comparison tabs are labeled "(Legacy)". - src/components/Layout.tsx - Removed 85 lines of hardcoded Biden test data, unused getColorClass, unused imports. Reduced from 152 to 17 lines. - src/components/PageHeader.tsx - Removed hardcoded "Hi there, Suraj", non-functional language selector. Now shows "Cross-Language Article Analysis" with a descriptive subtitle. 2. Section-Level Semantic Comparison (new feature) Backend: - app/models/section_comparison.py (new) - Pydantic models: SectionCompareRequest, ParagraphDiff, SectionDiff, SectionCompareResponse - app/services/section_comparison.py (new) - Core comparison engine: matches sections using LaBSE embeddings (greedy best-match on title + content preview), then compares paragraphs within matched sections. Uses Levenshtein disambiguation when top-2 semantic candidates are within 0.08 margin. - app/routers/comparison.py - Added POST /articles/compare-sections endpoint with URL/title parsing Frontend: - src/components/SectionComparisonView.tsx (new) - Full diff UI with expandable section cards, paragraph side-by-side view, similarity bars, color-coded status badges, and filter controls - src/services/structuredWikiService.ts - Added compareSections() method (POST) - src/models/structured-wiki.ts - Added TypeScript interfaces for ParagraphDiff, SectionDiff, SectionCompareResponse, SectionCompareRequest - src/components/StructuredArticleViewer.tsx - Added "Compare Sections" button with language selector, toggles between article view and comparison view 3. Bug Fixes - app/ai/translations.py - Was completely empty (runtime crash). Now wraps translate_text() from translation.py with graceful error handling. - app/ai/translation.py - Removed commented-out Google Colab interactive code. - tests/test_translation_limits.py - Updated test_unsupported_language to match new graceful fallback behavior. 4. Dead Code Removal Deleted files (7): - src/models/enums/FetchArticleResponse.ts (duplicate) - src/models/FetchArticleRequest.ts (duplicate, unused) - src/models/apis/TranslateArticleRequest.ts (unused) - src/models/apis/FetchArticleRequest.ts (unused) - app/ai/dataset/dataset_gen.py (standalone experimental script) - app/main.spec (PyInstaller build artifact) - app/temp_backend_run.sh (developer temp script) Cleaned inline dead code: - app/models/comparison.py - Removed all commented-out old model versions, added docstrings - app/models/api_models.py - Removed 3 unused shadow models (ComparisonResponse, TranslationResponse, ArticleResponse) - app/models/comparison_request.py - Removed empty LLMCompareRequest class - app/ai/semantic_comparison.py - Removed test main() function 5. Deduplication & Naming - app/ai/model_registry.py (new) - Single source of truth for COMPARISON_MODELS and DEFAULT_MODEL. Previously duplicated in 3 files. - app/main.py, app/routers/comparison.py, app/ai/semantic_comparison.py - All now import from model_registry.py - app/ai/semantic_comparison.py - Renamed eng_embedding → embedding, first_embeddings/second_embeddings → source_embeddings/reference_embeddings, sim_threshold → similarity_threshold, diff_info → unmatched_sentences - src/services/compareArticles.ts - Renamed parameters from originalArticleContent → sourceArticleContent, removed debug console.log statements - src/components/TranslationSection.tsx - Replaced opaque texts/editing/reference/suggestedContribution/suggestionType with typed ArticleDisplayBlock interface with clear label, content, displayType fields 6. Documentation - README.md - Complete rewrite reflecting the new architecture, section comparison feature, and cleaned project structure - CLAUDE.md - Updated agent instructions matching the current codebase Verification - Backend: 143 passed, 7 failed (all pre-existing). One previously-failing test fixed. - Frontend: 0 new TypeScript errors (5 pre-existing in Electron boilerplate files). * Update wiki_structure.py - update reference id max size to 300 (#15) Fix bug that can cause internal server error while running backend. I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine * Use canonical field names in CompareRequest (original_article_content etc.) Replace text_a/text_b/language_a/language_b with meaningful names: - original_article_content (alias: article_text_blob_1) - translated_article_content (alias: article_text_blob_2) - original_language (alias: article_text_blob_1_language) - translated_language (alias: article_text_blob_2_language) Update comparison.py router to use new field names when building request_data for the semantic comparison engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Eliminate aliases: use original/translated names everywhere Remove all AliasChoices from CompareRequest and BaseCompareRequest. Use canonical field names end-to-end across backend and tests: - original_article_content / translated_article_content - original_language / translated_language Also update: - semantic_comparison.py: request_data keys use canonical names - comparison.py router: GET query params renamed, request_data dicts updated - comparison_request.py: BaseCompareRequest fields renamed - compareArticles.ts: resolve remaining conflict marker - Tests: fixtures and request dicts updated to canonical names - Add ge/le bounds back to similarity_threshold in CompareRequest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve remaining merge conflicts: favor canonical field names - StructuredArticleViewer.tsx: keep FactExtraction import and full UI - models/__init__.py: keep FactExtraction exports - comparison.py / comparison_request.py: canonical original/translated names - semantic_comparison.py: canonical key names in request_data - api_models.py: keep FactExtractionRequest/Response classes - compareArticles.ts: clean canonical params, no debug cruft All 15 backend tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve merge conflicts: keep HEAD (canonical names + richer features) Files resolved: - FactExtraction.ts: keep num_facts field - compareArticles.ts: keep canonical sourceArticleContent/sourceLanguage params - StructuredArticleViewer.tsx: keep full fact-extraction + section-comparison UI - api_models.py: keep section_title and num_facts in FactExtractionRequest - models/__init__.py: keep grouped SectionCompare + FactExtraction exports - structured_wiki.py: keep full implementation - fact_extraction.py: keep richer HEAD implementation - fact_extraction_models.json: keep expanded model list with full metadata All 15 backend tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update symmetry-unified-backend/app/services/section_comparison.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update desktop-electron-frontend/src/components/StructuredArticleViewer.tsx Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Isaiah-Milkey <isamilkey@gmail.com> Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: AruefliASU <aruefli@asu.edu> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update wiki_structure.py - update reference id max size to 300 (#15) Fix bug that can cause internal server error while running backend. I got this error when calling structured article on: https://en.wikipedia.org/wiki/Despre_tine * Rename a/b variable names to original/translated for consistency Agent-Logs-Url: https://github.com/grey-box/symmetry-project/sessions/11f337b7-3983-49a3-a358-331a5a6d29d4 Co-authored-by: franc00018 <5459276+franc00018@users.noreply.github.com> * Refactor compareArticles function - remove unnecessary try block and streamline axios instance retrieval * Move model names to config.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add attributes to the API - (model name and similarity threshold) (#16) * Update comparison.py * Update comparison.py * Update response.py --------- Co-authored-by: François Pelletier <franc00018@users.noreply.github.com> * Use DEFAULT_MODEL and safe .get() for comparison keys in semantic_comparison Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include comparisons in CompareResponse and return raw comparisons when available Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix loanword suffix list and adjust family thresholds Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * GitHub actions implement (#18) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility --------- Co-authored-by: François Pelletier <francois@jevalide.ca> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add regression tests for long reference ID in wiki_structure Regression test for bug where Reference.id had max_length=100, causing 500 errors on articles with long reference IDs (e.g. Despre_tine). The fix (removing max_length constraints) was already applied in bf3803f. These tests guard against reintroduction of the constraint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: unify threshold configuration across all comparison paths (#20) * first version of GitHub Actions * ci: add GitHub Actions CI and release workflows; add dockerignore and frontend build script - CI: runs on pull_request to main/develop, backend tests, frontend build, docker smoke test, integration - Release: runs on push to main, creates semver tag, GitHub release, pushes Docker images to GHCR - Added .dockerignore files - Added build:web script to frontend package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix duplicate state and duplicate object key in StructuredArticleViewer.tsx Removed duplicated Section comparison state block and resolved duplicate similarity_threshold; build succeeds.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docker: pre-download spaCy model in backend image; run uvicorn without --reload in compose\n\nPre-download en_core_web_sm during Docker build to avoid runtime downloads in CI and run uvicorn in foreground without --reload to prevent PID 1/reloader issues in containers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add environment variable to CI and release workflows for Node.js compatibility * fix: unify threshold configuration across all comparison paths - Add app/core/settings.py as single source of truth for all thresholds, reading values from .env with sensible defaults - section_comparison.py: replace hardcoded LEVENSHTEIN_DISAMBIGUATION_MARGIN (0.08) and similarity_threshold default (0.65) with settings imports - similarity_scoring.py: replace hardcoded family thresholds (0.50/0.60/0.70) and band tuples with settings imports - semantic_c…
1 parent 50b318d commit 6e23a43

139 files changed

Lines changed: 17926 additions & 7391 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/settings.local.json

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"permissions": {
3+
"allow": [
4+
"Bash(wc:*)"
5+
]
6+
}
7+
}

.env.example

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Backend environment configuration
2+
# Copy this file to .env and fill in your values.
3+
# .env is gitignored — never commit it.
4+
5+
# Logging level for the FastAPI application (DEBUG | INFO | WARNING | ERROR)
6+
LOG_LEVEL=INFO
7+
8+
# Enable FastAPI debug mode (set to false in production)
9+
FASTAPI_DEBUG=false
10+
11+
# Default semantic similarity threshold (0.0 – 1.0)
12+
SIMILARITY_THRESHOLD=0.65

.github/workflows/ci.yml

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,9 @@ jobs:
2323
run:
2424
working-directory: symmetry-unified-backend
2525
steps:
26-
- uses: actions/checkout@v4
26+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
2727

28-
- uses: actions/setup-python@v5
28+
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
2929
with:
3030
python-version: "3.11"
3131
cache: pip
@@ -38,11 +38,11 @@ jobs:
3838
run: python -m spacy download en_core_web_sm
3939

4040
- name: Run tests
41-
run: pytest -m "not slow and not external" --tb=short --junitxml=report.xml
41+
run: python -m pytest -m "not slow and not external" --tb=short --junitxml=report.xml
4242

4343
- name: Upload test results
4444
if: always()
45-
uses: actions/upload-artifact@v4
45+
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
4646
with:
4747
name: backend-test-results
4848
path: symmetry-unified-backend/report.xml
@@ -51,7 +51,7 @@ jobs:
5151
name: Frontend Docker Build & Smoke Test
5252
runs-on: ubuntu-latest
5353
steps:
54-
- uses: actions/checkout@v4
54+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
5555

5656
- name: Build Docker image
5757
run: docker build -t symmetry-frontend:test ./desktop-electron-frontend
@@ -89,9 +89,9 @@ jobs:
8989
run:
9090
working-directory: desktop-electron-frontend
9191
steps:
92-
- uses: actions/checkout@v4
92+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
9393

94-
- uses: actions/setup-node@v4
94+
- uses: actions/setup-node@39370e3970a6d050c480ffad4ff0ed4d3fdee5af # v4.1.0
9595
with:
9696
node-version: "18"
9797
cache: npm
@@ -108,7 +108,7 @@ jobs:
108108
runs-on: ubuntu-latest
109109
needs: [backend-test, frontend-build]
110110
steps:
111-
- uses: actions/checkout@v4
111+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
112112

113113
- name: Start services
114114
run: docker compose up -d --build

.github/workflows/release.yml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ jobs:
2727
new_version: ${{ steps.tag.outputs.new_version }}
2828
changelog: ${{ steps.tag.outputs.changelog }}
2929
steps:
30-
- uses: actions/checkout@v4
30+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
3131
with:
3232
fetch-depth: 0
3333

@@ -41,7 +41,7 @@ jobs:
4141
release_branches: main
4242

4343
- name: Create GitHub Release
44-
uses: softprops/action-gh-release@v2
44+
uses: softprops/action-gh-release@c95fe1489396fe8a9eb87c0abf8aa5b2ef267fda # v2.2.1
4545
with:
4646
tag_name: ${{ steps.tag.outputs.new_tag }}
4747
name: Release ${{ steps.tag.outputs.new_tag }}
@@ -68,18 +68,18 @@ jobs:
6868
- context: desktop-electron-frontend
6969
image: frontend
7070
steps:
71-
- uses: actions/checkout@v4
71+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
7272

7373
- name: Log in to GitHub Container Registry
74-
uses: docker/login-action@v3
74+
uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
7575
with:
7676
registry: ghcr.io
7777
username: ${{ github.actor }}
7878
password: ${{ secrets.GITHUB_TOKEN }}
7979

8080
- name: Extract metadata
8181
id: meta
82-
uses: docker/metadata-action@v5
82+
uses: docker/metadata-action@369eb591f429131d6889c46b94e711f089e6ca96 # v5.6.1
8383
with:
8484
images: ghcr.io/${{ github.repository }}/${{ matrix.image }}
8585
tags: |

.github/workflows/security.yml

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
name: Security Scan
2+
3+
on:
4+
push:
5+
branches: [main, develop]
6+
pull_request:
7+
branches: [main, develop]
8+
schedule:
9+
# Run every Monday at 08:00 UTC
10+
- cron: '0 8 * * 1'
11+
12+
permissions:
13+
contents: read
14+
security-events: write
15+
16+
jobs:
17+
python-audit:
18+
name: Python Dependency Audit
19+
runs-on: ubuntu-latest
20+
defaults:
21+
run:
22+
working-directory: symmetry-unified-backend
23+
steps:
24+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
25+
26+
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
27+
with:
28+
python-version: "3.11"
29+
30+
- name: Install pip-audit
31+
run: pip install pip-audit
32+
33+
- name: Audit Python dependencies
34+
run: pip-audit -r requirements.txt --desc on
35+
36+
python-lint:
37+
name: Python Lint (ruff)
38+
runs-on: ubuntu-latest
39+
defaults:
40+
run:
41+
working-directory: symmetry-unified-backend
42+
steps:
43+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
44+
45+
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
46+
with:
47+
python-version: "3.11"
48+
49+
- name: Install ruff
50+
run: pip install ruff
51+
52+
- name: Run ruff
53+
run: ruff check .
54+
55+
node-audit:
56+
name: Node.js Dependency Audit
57+
runs-on: ubuntu-latest
58+
defaults:
59+
run:
60+
working-directory: desktop-electron-frontend
61+
steps:
62+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
63+
64+
- uses: actions/setup-node@39370e3970a6d050c480ffad4ff0ed4d3fdee5af # v4.1.0
65+
with:
66+
node-version: "18"
67+
cache: npm
68+
cache-dependency-path: desktop-electron-frontend/package-lock.json
69+
70+
- name: Install dependencies
71+
run: npm ci
72+
73+
- name: Audit Node.js dependencies
74+
run: npm audit --audit-level=high

.gitignore

Lines changed: 48 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,50 @@
1-
venv
2-
frontend.log
3-
symmetry-unified-backend/log_pytest.txt
4-
backend_test.log
5-
backend.log
1+
# ── Python ────────────────────────────────────────────────────
2+
venv/
3+
.venv/
4+
__pycache__/
5+
*.py[cod]
6+
*.pyo
7+
*.pyd
8+
*.so
9+
*.egg
10+
*.egg-info/
11+
dist/
12+
build/
13+
.eggs/
14+
.mypy_cache/
15+
.ruff_cache/
16+
.pytest_cache/
17+
.coverage
18+
htmlcov/
19+
*.log
20+
log_pytest.txt
621
datasets/
7-
.DS_Store
8-
**/tempCodeRunnerFile.py
9-
# Build artifacts
22+
23+
# ── Node / Electron ───────────────────────────────────────────
24+
node_modules/
1025
/desktop-electron-frontend/dist/
26+
/desktop-electron-frontend/.vite/
27+
/desktop-electron-frontend/out/
28+
/desktop-electron-frontend/artifacts/
29+
npm-debug.log*
30+
yarn-debug.log*
31+
yarn-error.log*
32+
33+
# ── Local config & secrets ────────────────────────────────────
34+
.env
35+
config.json
36+
37+
# ── OS / Editor ───────────────────────────────────────────────
38+
.DS_Store
39+
Thumbs.db
40+
.vscode/settings.json
41+
.idea/
42+
43+
# ── Logs ──────────────────────────────────────────────────────
44+
frontend.log
45+
backend.log
46+
backend_test.log
47+
*.log
48+
symmetry-unified-backend/report.xml
49+
test-results/
50+
desktop-electron-frontend/playwright-report/*

.markdownlint.json

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"default": true,
3+
"MD013": false,
4+
"MD024": false,
5+
"MD036": false,
6+
"MD040": false,
7+
"MD060": false
8+
}

.markdownlintignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
node_modules/
2+
symmetry-unified-backend/venv/
3+
desktop-electron-frontend/node_modules/
4+
desktop-electron-frontend/test-results/
5+
test-results/

0 commit comments

Comments
 (0)