🌇 Text To Image Search QuickStart#

Open In Colab

Try the image search live in Relevance AI Dashboard.

In this notebook we will show you how to create and experiment with a powerful text to image search engine using OpenAI’s CLIP and Relevance AI.

What I Need#

  • Project & API Key (The SDK will link you to the corresponding page or you can grab your API key from https://cloud.tryrelevance.com/ in the settings area)

  • Python 3

  • Relevance AI Installed as shown below. For more information visit Installation guide

Installation Requirements#

# Relevance AI installation
# remove `!` if running the line in a terminal
!pip install -U RelevanceAI[notebook]==2.0.0
!pip install ftfy regex tqdm
!pip install git+https://github.com/openai/CLIP.git

Client Setup#

You can sign up/login and find your credentials here: https://cloud.tryrelevance.com/sdk/api Once you have signed up, click on the value under Activation token and paste it here

from relevanceai import Client

client = Client()

Text-to-image search#

To enable text-to-image search we will be using Relevance AI as the vector database and OpenAI’s CLIP as the vectorizer, to vectorize text and images into CLIP vector embeddings.

1) Data#

For this quickstart we will be using a sample e-commerce dataset. Alternatively, you can use your own dataset for the different steps.

import pandas as pd
from relevanceai.utils.datasets import get_ecommerce_dataset_clean

# Retrieve our sample dataset. - This comes in the form of a list of documents.
documents = get_ecommerce_dataset_clean()
pd.DataFrame.from_dict(documents).head()

2) Encode / Vectorize with CLIP#

CLIP is a vectorizer from OpenAI that is trained to find similarities between text and image pairs. In the code below we set up CLIP.

import torch
import clip
import requests
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

# First - let's encode the image based on CLIP
def encode_image(image):
    # Let us download the image and then preprocess it
    image = (
        preprocess(Image.open(requests.get(image, stream=True).raw))
        .unsqueeze(0)
        .to(device)
    )
    # We then feed our processed image through the neural net to get a vector
    with torch.no_grad():
        image_features = model.encode_image(image)
    # Lastly we convert it to a list so that we can send it through the SDK
    return image_features.tolist()[0]


# Next - let's encode text based on CLIP
def encode_text(text):
    # let us get text and then tokenize it
    text = clip.tokenize([text]).to(device)
    # We then feed our processed text through the neural net to get a vector
    with torch.no_grad():
        text_features = model.encode_text(text)
    return text_features.tolist()[0]

100%|███████████████████████████████████████| 338M/338M [00:06<00:00, 52.0MiB/s]

We then encode the data we have into vectors, this will take a couple of mins

documents = documents[:500]  # only 500 docs to make the process faster

def encode_image_document(d):
    try:
        d["product_image_clip_vector_"] = encode_image(d["product_image"])
    except:
        pass


# Let's import TQDM for a nice progress bar!
from tqdm.auto import tqdm

[encode_image_document(d) for d in tqdm(documents)]

3) Insert#

Uploading our documents into the dataset quickstart_clip.

In case you are uploading your own dataset, keep in mind that each document should have a field called ’_id’. Such an id can be easily allocated using the uuid package:

ds.insert_documents(documents, create_id=True)

ds = client.Dataset("quickstart_clip")
ds.insert_documents(documents)

Once we have uploaded the data, we can see the dataset on the dashboard.

The dashboard provides users with a great overview and statistics of the dataset as shown below.

previous

🔍 Advanced Search

next

🤗 Hugging Face Pipelines