home << dhlab reference << dhlab.text.nbtokenizer
Tokens#
from dhlab.text.nbtokenizer import Tokens
- class Tokens(text)[source]#
Bases:
object
Create a list of tokens from a text with
tokenize()
.
home << dhlab reference << dhlab.text.nbtokenizer
from dhlab.text.nbtokenizer import Tokens
Bases: object
Create a list of tokens from a text with tokenize()
.