JacobLinCool/Qwen3-Embedding-8B-GIR-1

SentenceTransformer based on Qwen/Qwen3-Embedding-8B

This is a sentence-transformers model finetuned from Qwen/Qwen3-Embedding-8B. It maps sentences & paragraphs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Qwen/Qwen3-Embedding-8B
• Maximum Sequence Length: 40960 tokens
• Output Dimensionality: 4096 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 40960, 'do_lower_case': False, 'architecture': 'Qwen3Model'})
  (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("JacobLinCool/Qwen3-Embedding-8B-GIR-1")
# Run inference
queries = [
    "Generates samples of text from the provided vocabulary.\n\n  Args:\n    plain_vocab: vocabulary.\n    distribution: distribution.\n    train_samples: samples for training.\n    length: length.\n\n  Returns:\n    train_indices (np.array of Integers): random integers for training.\n      shape = [num_samples, length]\n    test_indices (np.array of Integers): random integers for testing.\n      shape = [num_samples, length]\n    plain_vocab   (list of Integers): unique vocabularies.",
]
documents = [
    'def generate_plaintext_random(plain_vocab, distribution, train_samples,\n                              length):\n  \n  if distribution is not None:\n    assert len(distribution) == len(plain_vocab)\n\n  train_indices = np.random.choice(\n      range(len(plain_vocab)), (train_samples, length), p=distribution)\n\n  return train_indices',
    'def switch(self, name):\n        \n        try:\n            switch = self.storage[self.__namespaced(name)]\n        except KeyError:\n            if not self.autocreate:\n                raise ValueError("No switch named  registered in " % (name, self.namespace))\n\n            switch = self.__create_and_register_disabled_switch(name)\n\n        switch.manager = self\n        return switch',
    'def late_filling(target, pressure=,\n                 Pc_star=,\n                 Swp_star=0.2, eta=3):\n    r\n    element = pressure.split()[0]\n    network = target.project.network\n    phase = target.project.find_phase(target)\n    pc_star = phase[Pc_star]\n    Pc = phase[pressure]\n    \n        Ts = network.map_throats(throats=target.Ts, origin=target)\n        values = values[Ts]\n    else:\n        Ps = network.map_pores(pores=target.Ps, origin=target)\n        values = values[Ps]\n    return values',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 4096] [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.8604, 0.0206, 0.1070]])

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.99
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.99
cosine_precision@3	0.3333
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.99
cosine_recall@3	1.0
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@1	0.99
cosine_ndcg@5	0.9963
cosine_ndcg@10	0.9963
cosine_mrr@1	0.99
cosine_mrr@5	0.995
cosine_mrr@10	0.995
cosine_map@100	0.995

Training Details

Training Dataset

Unnamed Dataset

• Size: 400 training samples
• Columns: query and code
• Approximate statistics based on the first 400 samples:

  	query	code
  type	string	string
  details	min: 2 tokens mean: 67.12 tokens max: 3156 tokens	min: 24 tokens mean: 126.98 tokens max: 1236 tokens

• Samples:

  query	code
  For memory actions, get a list of addresses it operates on. :param SimAction action: The action object to work with. :return: A list of addresses that are accessed with that action. :rtype: list	def _get_actual_addrs(action, state): if action.actual_addrs is None: addr_list = {0x60000000} else: addr_list = set(action.actual_addrs) return addr_list
  Construct the input file of the calculation.	def make_input(self, with_header=False): s = str(self.input) if with_header: s = str(self) + "\n" + s return s
  Check worker status route	def check_worker_status(): if not in request.args: resp = {"status": "bad request"} return jsonify(**resp) else: worker_id = request.args[] assignment_id = request.args[] allow_repeats = CONFIG.getboolean(, ) if allow_repeats: try: part = Participant.query.<br> filter(Participant.workerid == worker_id).<br> filter(Participant.assignmentid == assignment_id).one() status = part.status except exc.SQLAlchemyError: status = NOT_ACCEPTED else: try: matches = Participant.query.<br> filter(Participant.workerid == worker_id).all() numrecs = len(matches) if numrecs==0: status = NOT_ACCEPTED else: status = max([record.status for record in matches]) except exc.SQLAlchemyError: ...

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 1,
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 100 evaluation samples
• Columns: query and code
• Approximate statistics based on the first 100 samples:

  	query	code
  type	string	string
  details	min: 5 tokens mean: 66.56 tokens max: 548 tokens	min: 24 tokens mean: 142.11 tokens max: 901 tokens

• Samples:

  query	code
  Return the value of the android prefixed attribute in a specific tag. This function will always try to get the attribute with a android: prefix first, and will try to return the attribute without the prefix, if the attribute could not be found. This is useful for some broken AndroidManifest.xml, where no android namespace is set, but could also indicate malicious activity (i.e. wrongly repackaged files). A warning is printed if the attribute is found without a namespace prefix. If you require to get the exact result you need to query the tag directly: example:: >>> from lxml.etree import Element >>> tag = Element('bar', nsmap={'android': 'http://schemas.android.com/apk/res/android'}) >>> tag.set('{http://schemas.android.com/apk/res/android}foobar', 'barfoo') >>> tag.set('name', 'baz') # Assume that a is some APK object >>> a.get_value_from_tag(tag, 'name'...	def get_value_from_tag(self, tag, attribute): value = tag.get(self._ns(attribute)) if value is None: value = tag.get(attribute) if value: log.warning("Failed to get the attribute on tag with namespace. " "But found the same attribute without namespace!".format(attribute, tag.tag)) return value
  Get information about this object as a dictionary. Used by WebSocket interface to pass some relevant information to client applications.	def get_as_datadict(self): return dict(type=self.class.name, tags=list(self.tags))
  Makes forecast with the estimated model Parameters ---------- h : int (default : 5) How many steps ahead would you like to forecast? past_values : int (default : 20) How many past observations to show on the forecast graph? intervals : Boolean Would you like to show 95% prediction intervals for the forecast? Returns ---------- - Plot of the forecast	def plot_predict(self,h=5,past_values=20,intervals=True,**kwargs): import matplotlib.pyplot as plt import seaborn as sns figsize = kwargs.get(,(10,7)) if self.latent_variables.estimated is False: raise Exception("No latent variables estimated!") else: scale, shape, skewness = self._get_scale_and_shape(self.latent_variables.get_z_values(transformed=True)) previous_value = self.data[-1] forecasted_values = np.ones(h)*self.states[-1] date_index = self.shift_dates(h) simulations = 10000 sim_vector = np.zeros([simulations,h]) t_params = self.transform_z() for n in range(0,simulations): rnd_q = np.random.normal(0,np.sqrt(self.latent_variables.get_z_values(transformed=True)[0]),h) exp = forecasted_values.copy() for t in range(0,h): if t == 0:...

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 1,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• seed: 2025
• bf16: True
• load_best_model_at_end: True
• optim: paged_adamw_8bit
• push_to_hub: True
• hub_model_id: JacobLinCool/Qwen3-Embedding-8B-GIR-1
• hub_private_repo: False
• gradient_checkpointing: True
• eval_on_start: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 2025
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: paged_adamw_8bit
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: JacobLinCool/Qwen3-Embedding-8B-GIR-1
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: True
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Validation Loss	cosine_ndcg@10
0	0	0.0559	0.9926
1.0	7	0.0370	0.9963
2.0	14	0.0319	0.9963
3.0	21	0.0312	0.9963
4.0	28	0.0312	0.9963
5.0	35	0.0313	0.9963
-1	-1	-	0.9963

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 5.1.1
• Transformers: 4.56.2
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.1
• Datasets: 4.1.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}