X hits on this document

Powerpoint document

Python for NLP and the Natural Language Toolkit - page 14 / 47

112 views

0 shares

0 downloads

0 comments

14 / 47

Tokens and Types

The term word can be used in two different ways:

1.

To refer to an individual occurrence of a word

2.

To refer to an abstract vocabulary item

For example, the sentence “my dog likes his dog” contains five occurrences of words, but four vocabulary items.

To avoid confusion use more precise terminology:

Word token: an occurrence of a word

Word Type:  a vocabulary item

Document info
Document views112
Page views112
Page last viewedSat Dec 03 18:24:26 UTC 2016
Pages47
Paragraphs392
Words1978

Comments