See our collection for all versions of TIPSv2.

Run TIPSv2 with Keras 3: JAX, PyTorch, or TensorFlow

kerasformers/tipsv2-b14

Paper: TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment (arXiv:2604.12012)

TIPSv2 (Google DeepMind) is a CLIP/SigLIP-style dual encoder: a DINOv2-style ViT vision tower with register tokens plus a bidirectional text tower, aligned with a temperature-scaled contrastive objective.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of google/tipsv2-b14 for kerasformers. One implementation runs unmodified on TensorFlow / Torch / JAX. The full model and both towers load from this single repo.

✨ Quick start (zero-shot)

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
import numpy as np
import keras
from kerasformers.models.tipsv2 import Tipsv2Model, Tipsv2Processor

model = Tipsv2Model.from_weights("kerasformers/tipsv2-b14")
processor = Tipsv2Processor.from_weights("kerasformers/tipsv2-b14")

image = Image.open("your_image.jpg").convert("RGB")
texts = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = processor(text=texts, images=np.array(image))
out = model(inputs)
probs = keras.ops.softmax(out["logits_per_image"], axis=-1)
print(keras.ops.convert_to_numpy(probs)[0])

Towers only:

from kerasformers.models.tipsv2 import Tipsv2VisionModel, Tipsv2TextModel
vision = Tipsv2VisionModel.from_weights("kerasformers/tipsv2-b14")
text = Tipsv2TextModel.from_weights("kerasformers/tipsv2-b14")

All TIPSv2 variants load the same way with from_weights("kerasformers/<variant>"):

Variant	Hub
`tipsv2-b14`	`kerasformers/tipsv2-b14`
`tipsv2-l14`	`kerasformers/tipsv2-l14`
`tipsv2-so400m14`	`kerasformers/tipsv2-so400m14`
`tipsv2-g14`	`kerasformers/tipsv2-g14`

Tips

Set KERAS_BACKEND before importing Keras / kerasformers.
The image processor rescales to [0, 1] (no mean/std normalization); input resolution is 448.
Upstream checkpoints: Tipsv2Model.from_weights("hf:google/tipsv2-b14").