HViLM-base: A Foundation Model for Viral Genomics

HVUE v2 supersedes the original HVUE benchmark.
The original HVUE v1 benchmark contained substantial cross-split sequence similarity that could inflate held-out performance estimates. HVUE v2 was rebuilt using source-sequence clustering before train/validation/test assignment and chunking, followed by exact- and near-match leakage auditing.
Use duttaprat/HVUE-v2 for current HViLM evaluation.

Model Description

HViLM (Human Virome Language Model) is a genomic foundation model adapted to virus sequences through continued pre-training of DNABERT-2. HViLM-base was trained on approximately 5 million non-redundant virus-derived sequence fragments from the VIRION resource, representing approximately 9,000 virus species across 45+ families.

The architecture and tokenizer remain those of DNABERT-2; continued pre-training updates the model weights using a masked-language-modeling objective on the virus-focused corpus.

Preprint: HViLM: A Foundation Model for Viral Genomics Enables Multi-Task Prediction of Pathogenicity, Transmissibility, and Host Tropism
bioRxiv 2026.03.18.712700

Authors: Pratik Dutta, Jack Vaska, Pallavi Surana, Rekha Sathian, Max Chao, Zhihan Zhou, Han Liu, and Ramana V. Davuluri

Code: github.com/duttaprat/HViLM

HViLM Model Family

HViLM-base is the continued-pretrained foundation model. Official task-specific models fine-tuned on HVUE v2 are released as standalone checkpoints:

Resource	Purpose
HViLM-base	Continued-pretrained foundation model / sequence representations
HViLM-Patho	Pathogenicity classification
HViLM-R0	Transmissibility classification
HViLM-Tropism	Human host-tropism classification
HVUE-v2	Leakage-controlled benchmark

The complete project is also grouped in the HViLM: Human Virome Language Model collection on the duttaprat Collections page.

Key Features

Virus-focused continued pre-training: approximately 5M non-redundant fragments derived from VIRION-linked virus sequences.
DNABERT-2 initialization: preserves the DNABERT-2 architecture and BPE tokenizer while adapting model weights to virus sequence data.
Three official downstream models: pathogenicity, transmissibility, and host tropism.
HVUE v2 evaluation: cluster-aware splitting before chunking, with multiple similarity stringencies and sequence lengths.
Parameter-efficient downstream adaptation: official task models were trained with LoRA.
Public reproducibility resources: base model, three task-specific checkpoints, HVUE v2 benchmark, and project code are released publicly.

Model Architecture and Continued Pre-training

HViLM-base is derived from DNABERT-2 (117M parameters).

Property	Value
Architecture	MosaicBERT / DNABERT-2
Parameters	~117M
Hidden size	768
Transformer layers	12
Attention heads	12
Tokenization	Byte Pair Encoding (BPE)
Positional method	ALiBi
Continued-pretraining objective	Masked Language Modeling
Pretraining fragment length	1000 nt
Final virus-focused corpus	~5M non-redundant fragments
Redundancy reduction	MMseqs2 clustering at 80% identity / 80% coverage
Optimizer	AdamW
Learning rate	5e-5
Training	10 epochs
Hardware	4 × NVIDIA A100 GPUs
Approximate training time	72 hours
Held-out MLM accuracy	94.2%

Sequence-length note: HViLM uses BPE tokenization, so nucleotide length and model-token length are not equivalent. The continued-pretraining corpus used 1000-nt sequence fragments; downstream configurations are described by nucleotide length in HVUE v2.

Quick Start

Extract sequence representations from HViLM-base

import torch
from transformers import AutoTokenizer, AutoModel

model_id = "duttaprat/HViLM-base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
)

sequence = "ATGCGTACGTTAGCCGATCGATTACGCGTACGTAGCTAGCTAGCT"

inputs = tokenizer(
    sequence,
    return_tensors="pt",
    truncation=True,
    padding=True,
)

with torch.no_grad():
    outputs = model(**inputs)

token_embeddings = outputs.last_hidden_state
print(token_embeddings.shape)

For sequence-level representations, pooling strategy should be chosen according to the downstream task rather than treated as a fixed property of HViLM-base.

Use the Official Fine-tuned Models

If the goal is one of the three HVUE v2 tasks, users can load the corresponding task model directly; HViLM-base does not need to be loaded separately.

Pathogenicity

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "duttaprat/HViLM-Patho"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
)

Labels:

0 → NON_PATHOGENIC
1 → PATHOGENIC

Transmissibility

Use:

duttaprat/HViLM-R0

Labels:

0 → R0_LT_1
1 → R0_GE_1

Host Tropism

Use:

duttaprat/HViLM-Tropism

Labels:

0 → NON_HUMAN_TROPIC
1 → HUMAN_TROPIC

See the individual model cards for full usage examples and task-specific limitations.

HVUE v2 Benchmark

HVUE v2 is the current evaluation benchmark for HViLM. It replaces HVUE v1.

The benchmark was reconstructed to reduce supervised train-test leakage using the following ordering:

consolidate and deduplicate source sequences;
cluster source sequences with MMseqs2;
assign complete clusters to train/validation/test splits;
chunk sequences only after split assignment;
remove exact duplicate chunks;
audit cross-split exact and near matches.

HVUE v2 includes:

Pathogenicity
Transmissibility
Host Tropism

Across the benchmark, configurations evaluate different sequence lengths (500, 1000, and 2000 nt where applicable), sequence-similarity stringencies, and temporal generalization where reliable collection-date metadata are available.

Primary HViLM Results

The primary results below use the standard 1000-nt configuration for each task.

Task	HVUE v2 configuration	Accuracy	F1	MCC	Official model
Pathogenicity	`standard_capped_1000bp`	92.39	91.32	83.10	HViLM-Patho
Transmissibility	`standard_capped_1000bp`	87.50	86.16	72.66	HViLM-R0
Host Tropism	`standard_95_1000bp`	96.49	74.49	48.99	HViLM-Tropism

The directory/configuration identifiers retain bp for release stability; manuscript and descriptive text use nt for nucleotide sequence length.

Interpretation of the HVUE v2 Results

Under leakage-controlled evaluation, the effect of virus-focused continued pre-training is task dependent:

Pathogenicity: HViLM improves F1 by 1.28 points over vanilla DNABERT-2 (91.32 vs. 90.04).
Transmissibility: HViLM and DNABERT-2 are close (86.16 vs. 85.81 F1), and HViLM is essentially tied with DNABERT-MB (86.16 vs. 86.15 F1).
Host Tropism: HViLM shows the largest F1 improvement, reaching 74.49 compared with 64.82 for class-balanced DNABERT-2.

These results support a more specific conclusion than the original HVUE v1 evaluation: virus-focused continued pre-training provides its clearest benefit on the more challenging Host Tropism task, while gains on Pathogenicity and Transmissibility are smaller.

For complete baseline comparisons, hard-split evaluations, temporal evaluations, and sequence-length analyses, see the HViLM GitHub repository and HVUE-v2.

Training Data

Continued-pretraining corpus

HViLM-base was trained using virus sequences associated with the VIRION resource.

Processing included:

retrieval and quality control of VIRION-linked nucleotide sequences;
removal of short sequences and exact duplicates;
segmentation into non-overlapping 1000-nt fragments;
MMseqs2 clustering at 80% sequence identity and 80% coverage;
selection of approximately 5M representative fragments for continued pre-training.

The corpus spans approximately 9,000 virus species and 45+ virus families across the Baltimore classification groups.

Interpretability

Attention-guided analyses associated with the HViLM study identified candidate sequence motifs in pathogenic coronavirus sequences, including motifs with similarity to vertebrate transcription-factor binding motifs such as IRF1, FOXQ1, and ZNF354A.

These observations are hypothesis-generating. Sequence similarity between virus motifs and host transcription-factor binding motifs does not by itself establish molecular mimicry, causal regulation, immune evasion, or another biological mechanism. Experimental validation and additional controls are required for mechanistic interpretation.

Limitations

HVUE v2 controls supervised split leakage through source-level clustering and auditing, but sequence-similarity thresholds cannot eliminate every form of biological relatedness.
The complete historical training exposure of the original DNABERT-2 model cannot be reconstructed; therefore, absence of all possible ancestral pretraining exposure to benchmark-related sequences cannot be guaranteed.
Host association is biologically context-dependent and may include multi-host, zoonotic, and reverse-zoonotic relationships; the benchmark uses a simplified binary formulation.
R₀-based transmissibility labels simplify a continuous, context-dependent epidemiological quantity into a binary benchmark task.
Performance differences between closely matched models should not be interpreted as statistically meaningful without uncertainty estimates or repeated evaluations.
Attention-based motif analyses should be considered exploratory rather than direct evidence of mechanism.
HViLM predictions are research outputs and are not intended to replace experimental, clinical, epidemiological, or public-health assessment.

Reproducibility and Resources

Base model: duttaprat/HViLM-base
Pathogenicity model: duttaprat/HViLM-Patho
Transmissibility model: duttaprat/HViLM-R0
Host Tropism model: duttaprat/HViLM-Tropism
Benchmark: duttaprat/HVUE-v2
Code: github.com/duttaprat/HViLM
Collections: duttaprat's Hugging Face Collections

Citation

If you use HViLM in your research, please cite:

@article{dutta2026hvilm,
  title={HViLM: A foundation model for viral genomics enables multi-task prediction of pathogenicity, transmissibility, and host tropism},
  author={Dutta, Pratik and Vaska, Jack and Surana, Pallavi and Sathian, Rekha and Chao, Max and Zhou, Zhihan and Liu, Han and Davuluri, Ramana V},
  journal={bioRxiv},
  pages={2026--03},
  year={2026},
  publisher={Cold Spring Harbor Laboratory}
}

If you use DNABERT-2 directly or build on its architecture, please also cite the DNABERT-2 publication.

Model Card Authors

Pratik Dutta — Stony Brook University
Ramana V. Davuluri — Stony Brook University

Contact

GitHub Issues: github.com/duttaprat/HViLM/issues
Lab: Davuluri Lab, Stony Brook University

Acknowledgments

HViLM builds on DNABERT-2 by Zhou et al. Continued-pretraining data were derived from the VIRION resource maintained by the Viral Emergence Research Initiative (Verena).

License

HViLM-base is released under the Apache License 2.0.

Disclaimer

HViLM is a research model for computational biology. It should not be used as the sole basis for clinical, diagnostic, epidemiological, biosurveillance, or public-health decisions. Model outputs should be interpreted alongside appropriate biological evidence and expert assessment.