snowball

Snowball stemmers for deno. These stemmers are based on the compiled JavaScript stemmers from the snowball project version 2.2.0.

Usage

EnglishStemmer

Provides the stem of the given word. Assumes that the input is lowercase.

import { assertStrictEquals } from "./test_deps.ts";
import { EnglishStemmer } from "https://deno.land/x/snowball/english_stemmer.ts";

const englishStemmer = new EnglishStemmer();

const stem = englishStemmer.stem("enthusiastically");

assertStrictEquals(stem, "enthusiast");

Here is an example with multiple words.

import { assertStrictEquals } from "./test_deps.ts";
import { EnglishStemmer } from "https://deno.land/x/snowball/english_stemmer.ts";

const sentence = "the quick brown fox jumped over the lazy dog";

const englishStemmer = new EnglishStemmer();

const stemmedSentence = sentence
  .match(/\b\w\w+\b/gu) // matches two or more word characters
  .map((token) => englishStemmer.stem(token))
  .join(" ");

assertStrictEquals(
  stemmedSentence,
  "the quick brown fox jump over the lazi dog",
);

RussianStemmer

Many languages are supported

import { assertStrictEquals } from "./test_deps.ts";
import { RussianStemmer } from "https://deno.land/x/snowball/russian_stemmer.ts";

const sentence = "обязательно выпейте свой овалтин";

const russianStemmer = new RussianStemmer();

const stemmedSentence = sentence
  .split(/\s+/u)
  .map((token) => russianStemmer.stem(token))
  .join(" ");

assertStrictEquals(
  stemmedSentence,
  "обязательн вып сво овалтин",
);

LanguageStemmers

There is an Object containing all available languages and stemmers defined in mod.ts.

import { assertStrictEquals } from "https://deno.land/std@0.126.0/testing/asserts.ts";
import { LanguageStemmers } from "https://deno.land/x/snowball/mod.ts";

const spanishStemmer = new LanguageStemmers["Spanish"]();

assertStrictEquals(
  spanishStemmer.stem("gracias"),
  "graci",
);

stopWords

Stop words generally provide little or no information. Some languages come with common stop words from the snowball-website. These can also be imported individually from the file containing the stopwords for that language. You should tokenize and stem these stop words the same way you do with your input text.

import { assert } from "https://deno.land/std@0.126.0/testing/asserts.ts";
import { EnglishStemmer } from "https://deno.land/x/snowball/english_stemmer.ts";

const englishStemmer = new EnglishStemmer();

assert(englishStemmer.stopWords.has("been"));

Supported Languages

Unless specified, there is only one stemmer available called LanguageStemmer which is exported from language_stemmer.ts and mod.ts. Replace language with the desired language respecting the hinted capitalization.

  1. Arabic
  2. Armenian
  3. Basque
  4. Catalan
  5. Danish
  6. Dutch
    1. DutchStemmer
    2. KraaijPohlmannStemmer
  7. English
    1. EnglishStemmer - Porter 2 or snowball algorithm
    2. PorterStemmer - Porter 1 stemmer
    3. LovinsStemmer - The first published stemming algorithm
  8. Finnish
  9. French
  10. German
    1. GermanStemmer
    2. German2Stemmer
  11. Greek
  12. Hindi
  13. Hungarian
  14. Indonesian
  15. Irish
  16. Italian
  17. Lithuanian
  18. Nepali
  19. Norwegian
  20. Protugese
  21. Romanian
  22. Russian
  23. Serbian
  24. Spanish
  25. Swedish
  26. Tamil
  27. Turkish
  28. Yiddish