The dataset's own README, also available at https://dl.fbaipublicfiles.com/large_objects/dino_vitl_10B/README.md.

DatasetDINO10B

A simple interface for loading a dataset of DINO patch descriptors for similarity search benchmarks.

WARNING : When using large batch sizes, prefer batch sizes that are divisors of 200,000,000, which is the number of vectors per .bvecs file. This will avoid splitting batches between two .bvecs files, causing significant overhead due to concate>

Downloading the data

The fastest way (by far) to download the large amount of data (10 TB) is to use the Axel Download accelerator (download from https://github.com/axel-download-accelerator/axel). Put the following code in a .sh file and execute it in your data folder to get the full dataset (see Dataset class in Faiss about the data folder). Beware, the downloading might take several days due to the large size of the data files (10TB). If you want only a subset of the dataset, you can selectively download a part of the dataset. The dataset is divided into 50 chunks of name chunk_i.bvecs which are listed in the list of files file_list.txt (downloaded in the first wget command in the code below). Each chunk file has 200 GB of data, and contains 200M vectors. If you just want, for example, a dataset of 400M vectors, you can edit the file_list.txt file to only download the first two chunks chunk_0000.bvecs and chunk_0001.bvecs.

#!/bin/bash
axel https://dl.fbaipublicfiles.com/large_objects/dino_vitl_10B/file_list.txt

base_dir="dino_vitl_10B"
mkdir -p "$base_dir"

while read -r url; do
  # strip scheme+host
  path=${url#*://*/}
  path=$(echo "$url" | sed -E 's|https?://[^/]+/||')
  # remove first two dirs (large_objects/dino_vitl_10B)
  rel=$(echo "$path" | cut -d/ -f3-)
  [ -z "$rel" ] && rel=$(basename "$path")
  out="$base_dir/$rel"
  mkdir -p "$(dirname "$out")"
  axel -n 16 -a -o "$out" "$url"
done < file_list.txt
rm file_list.txt

Alternatively you can use a simple wget script but it might take several days to complete :

#!/bin/bash
wget https://dl.fbaipublicfiles.com/large_objects/dino_vitl_10B/file_list.txt
wget -i file_list.txt --force-directories --cut-dirs=2
mv dl.fbaipublicfiles.com dino_vitl_10B
rm file_list.txt

Usage

If you use the code from faiss library :

from faiss.contrib.datasets import DatasetDINO10B

# Initialize with dataset size. The data must be in a `data` folder in your working directory.
ds = DatasetDINO10B(
    nb=1_000_000,  # number of base vectors
)

If you use the code_dataset_dino.py file downloaded from fbaipublicfiles, import the class using :

from code_dataset_dino import DatasetDINO10B
ds = DatasetDINO10B(
    nb = 1_000_000,
    path = "path/to/the/downloaded/folder/dino_vitl_10B"
)

Supported Dataset Sizes

  • 100,000
  • 200,000
  • 500,000
  • 1,000,000
  • 2,000,000
  • 5,000,000
  • 10,000,000
  • 20,000,000
  • 50,000,000
  • 100,000,000
  • 200,000,000
  • 500,000,000
  • 1,000,000,000
  • 2,000,000,000
  • 5,000,000,000
  • 10,000,000,000

File Structure

The dataset path should contain: - directory chunked_base_10B - base vectors - queries_clean.bvecs - test queries
- train_queries_99M.bvecs - training queries - gts/gts_dino_patch_{nb}_k10.npy - ground truth nearest neighbors