← Back to context

Comment by tombh

6 years ago

Oh, I'm surprised the "blacklist" isn't just the standard English dictionary. I'm sure I'm just being naive though. Why not just blacklist any word that already exists in English?

7 comments

tombh

Reply
turtlesoup  6 years ago

There are some subtleties (e.g. hyphens, derived forms, bigrams, etc.) but the biggest problem is that most English dictionaries don't have entries for every scientific word / piece of internet slang. I ended up tokenizing Wikipedia for a blacklist and still missed a lot :(

  • mathewsanders  6 years ago

    > I ended up tokenizing Wikipedia for a blacklist and still missed a lot :(

    That sounds like an impressive project in itself :)

    • pbhjpbhj  6 years ago

      Words not on Wikipedia, found on other sources, listed by frequency (perhaps with a date-weighting of the source document to reduce rating of older sources), would be an interesting way to find holes in Wikipedia's coverage.

      3 replies →