Skip to content

Use native Spark regexp_replace for stopword removal instead of a UDF#1337

Open
Chandanydv1234 wants to merge 1 commit into
zinggAI:mainfrom
Chandanydv1234:remodel-stopwords-regexp-replace
Open

Use native Spark regexp_replace for stopword removal instead of a UDF#1337
Chandanydv1234 wants to merge 1 commit into
zinggAI:mainfrom
Chandanydv1234:remodel-stopwords-regexp-replace

Conversation

@Chandanydv1234

@Chandanydv1234 Chandanydv1234 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #1080

Changes Made:

  1. SparkStopWordsRemover - the removal now calls regexp_replace directly instead of callUDF. Deleted all the code that supported the UDF: registerUDF, the udfName field, registration calls in constructors and all the related imports.
  2. Deleted RemoveStopWordsUDF and RemoveStopWords.
  3. Deleted StopWords.

No behaviour change is intended. All the existing tests pass.

@Chandanydv1234 Chandanydv1234 force-pushed the remodel-stopwords-regexp-replace branch from 02c7431 to cb80540 Compare July 6, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stopwords can be re modelled

2 participants