How to use frizwankhan/tokenization_model with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="frizwankhan/tokenization_model")
# Load model directly from transformers import AutoProcessor, AutoModelForTokenClassification processor = AutoProcessor.from_pretrained("frizwankhan/tokenization_model") model = AutoModelForTokenClassification.from_pretrained("frizwankhan/tokenization_model")
SFconvertbot