UXTerm

UXTerm is one word in a UXSearchIndex, together with every document it occurs in.

#use <UXKit>            // or #import "UXSearchIndex.xc"

Overview

class UXTerm : Object {
    u8*               word;        // lowercased, alphanumeric only
    Array<UXPosting>* postings;    // one per document containing it
}

A term is a row of the inverted index. A query tokenizes into words, looks each one up here, and reads off the documents. Searching is therefore a lookup rather than a scan of every document.

The word is already normalised

u8* word

Before a term is created, the tokenizer lowercases the word and strips everything that is not a letter or a digit. "Dog!", "dog," and "DOG" in three documents are one UXTerm.

Queries go through the same tokenizer, so indexing and querying apply the same normalisation from one place.

The string is a copy made by the index, so it outlives the text that was indexed.

The postings are the payoff

Array<UXPosting>* postings

One UXPosting per document containing the word, each carrying a count. Scoring a query walks these lists and adds up the counts.

Postings are held strongly and never removed. The index has no delete, so a term’s list only grows. This is why re-indexing means a rebuild.

A term always has at least one posting: its first occurrence creates it, and nothing prunes.

Lookup is a linear scan

The index finds a term by walking its term array and comparing strings, so termCount() is also the cost of every lookup.

For the hundreds to low thousands of distinct words that a project’s filenames and titles produce, this is faster than a hash and simpler. Over large amounts of prose it would be the first part to replace, and it can be replaced here without changing postings, scoring or the query path.

Fields

word

u8* word

postings

Array<UXPosting>* postings

Conforms to

• Inherits Object

See also

• UXPosting: a document and a count
• UXSearchIndex: the index these make up
• UXStrItem: a plain boxed string, when you want no index at all