tsvector for tokenization?

[peterlindsten]

I've applied this to my use-case, with much improved results (noisy dataset)

CREATE OR REPLACE FUNCTION tsvectortokenize(txt TEXT) RETURNS TEXT[]
LANGUAGE SQL
IMMUTABLE PARALLEL SAFE
BEGIN ATOMIC
    SELECT array_agg(a.lexeme) FROM (SELECT lexeme, positions FROM unnest(to_tsvector('swedish', txt))) a JOIN LATERAL unnest(a.positions) on true;
END;

This uses the built-in text search system and configurations.
The base postgres docker images (perhaps most installs?) seem to have some 29 text search configurations, for different languages + a "simple" parser.

The text search configuration applies stop word lists, punctuation removal, stemming and I don't know what else (dictionary lookup?).

to_tsvector normally returns word counts, which could be used when doing word_freqs - but here it is detrimental, so the query reverses it. I haven't analyzed if this approach could simplify the places where it is actually used - so this may be the wrong approach.

On my dataset tsvectortokenize() is some 4.5 times faster than stopwordfilter(bm25simpletokenize()) - but the absolute numbers make this relatively meaningless I think - word score map dominates my indexing operations by two orders of magnitude.

Not sure if this is something you'd like to apply to this project - but thought I'd share since it was a large improvement for me.

[jankovicsandras]

This looks interesting, thanks!

I will test it, then we can discuss adding this as an alternative tokenizer, but - as I understand - the built-in text search system discards words not found in its dictionaries.

"If no dictionary in the list recognizes the token then it is also ignored." - https://www.postgresql.org/docs/current/textsearch-controls.html

This is probably harmful for search quality as the most relevant information is often in uncommon keywords, like serial numbers or technical parameters.

I see you opened 2 other PRs, those look good too, but I need some time to review them.

Thank you for your contribution! 😊

[peterlindsten]

Interesting! I didn't notice this part of the docs - however, the results in my testing does not show that any "words" are ignored that aren't stop words.

Digging further into this it seems that the "dictionary" being used for at least some (maybe all) built-in text search configurations are of the "snowball" variety - and this algorithm always returns something - so in effect, all words are in these "dictionaries".

There is also another component before the dictionaries in the text processing, a parser - this splits "words" and classifies them which is then routed to (potentially) different dictionaries depending on class. Integers seem to be routed to "simple" in the default configurations. For more technical documents it seems to me that the classification may be a bit too coarse, most special characters are "space symbols" - which are ignored. But it does make sense for normal natural language documents.

I guess at the end of the day this needs to be configurable, since corpuses differ in what is relevant.

Edit: For understanding how any particular text is processed the ts_debug() is most helpful, like this: SELECT * FROM ts_debug('english', 'All, mice love $5 cheese')

[jankovicsandras]

Sorry, I misunderstood the PostgreSQL documentation and answered stupidly earlier.

Now I see that tsvectortokenize is a tokenizer combined with a stopword filter. I tested tsvectortokenize and it works OK. We can think about how to include it as an anternative tokenizer, ideally with some kind of user choice. But using it as default would probably harm search quality for special corpus, like technical descriptions.

Here's an example, a product description from a random german battery webshop:
"Ultralife Lithium Batterie 9 Volt, E-Block, U9VL, U9VL-J, U9VL-J-P"

I created tsvectortokenize with german stopwords:

CREATE OR REPLACE FUNCTION tsvectortokenize(txt TEXT) RETURNS TEXT[] LANGUAGE SQL IMMUTABLE PARALLEL SAFE BEGIN ATOMIC SELECT array_agg(a.lexeme) FROM (SELECT lexeme, positions FROM unnest(to_tsvector('german', txt))) a JOIN LATERAL unnest(a.positions) on true; END;

Comparison:

bm25simpletokenize 
['ultralife', 'lithium', 'batterie', '9', 'volt', 'e-block', 'u9vl', 'u9vl-j', 'u9vl-j-p'] 

bm25simpletokenize + stopwordfilter
['ultralife', 'lithium', 'batterie', '9', 'volt', 'e-block', 'u9vl', 'u9vl-j', 'u9vl-j-p'] 

tsvectortokenize 
['9', 'batteri', 'block', 'e', 'e-block', 'j', 'j', 'lithium', 'p', 'u9vl', 'u9vl', 'u9vl', 'u9vl-j', 'u9vl-j-p', 'ultralif', 'volt']

Problematic tokens:
'ultralife' -> 'ultralif'
'batterie' -> 'batteri'
'e-block' -> 'e' and 'block'
'u9vl', 'u9vl-j', 'u9vl-j-p' -> 'j', 'j', 'p', 'u9vl', 'u9vl', 'u9vl', 'u9vl-j', 'u9vl-j-p'

---

Another example, with english to_tsvector:
"Alpha Single-Gang 16mm Right-angle PC Mount - 5K Linear - RV16AF-41-15R1-B5K-30H4"

bm25simpletokenize
['alpha', 'single-gang', '16mm', 'right-angle', 'pc', 'mount', '-', '5k', 'linear', '-', 'rv16af-41-15r1-b5k-30h4'] 

bm25simpletokenize + stopwordfilter
['alpha', 'single-gang', '16mm', 'right-angle', 'pc', 'mount', '-', '5k', 'linear', '-', 'rv16af-41-15r1-b5k-30h4'] 

tsvectortokenize 
['-15', '-41', '16mm', '30h4', '5k', 'alpha', 'angl', 'b5k', 'gang', 'linear', 'mount', 'pc', 'r1', 'r1-b5k-30h4', 'right', 'right-angl', 'rv16af', 'singl', 'single-gang'] 

---

Of course, bm25simpletokenize and stopwordfilter are not perfect either, they also produce problematic tokens sometimes. Tokenization is an interesting topic. 😊

[peterlindsten]

Sorry, I misunderstood the PostgreSQL documentation and answered stupidly earlier.

You did not answer stupidly - you pointed out a relevant part in the documentation which I had missed.
Also led me down a small rabbit hole in how postgres works with text configurations.

Problematic tokens:
'ultralife' -> 'ultralif'
'batterie' -> 'batteri'

These are stemmed words, and I think work as expected - since the input would also be stemmed, possibly from a different ending, which is the point of stemming.

'e-block' -> 'e' and 'block'
'u9vl', 'u9vl-j', 'u9vl-j-p' -> 'j', 'j', 'p', 'u9vl', 'u9vl', 'u9vl', 'u9vl-j', 'u9vl-j-p'

These ones are clearly just wrong for this application.

Of course, bm25simpletokenize and stopwordfilter are not perfect either, they also produce problematic tokens sometimes. Tokenization is an interesting topic. 😊

Yes. Tokenization is tricky and mostly use-case dependent.

I guess stemming words is also less useful since the introduction of semantic similarity search - and what you want from your keyword search nowadays is the exact matches, not as previously, when you wanted "nearby" words as well. (Which the trigram (trgm) postgres module perhaps is better for)

---

As to how to implement a tokenization choice in a neat way, I'm not sure. Particularly since both tsvectortokenize and stopwordsfilter want an argument..