Image-to-Image Search via ResNet50

This guide will demonstrate how to fine-tune a ResNet model for image to image retrieval.

Task

More specifically, we will fine-tune ResNet50 on Totally Looks Like Dataset. The dataset consists of 6016 pairs of images (12032 in total).

About TTL

Totally-Looks-Like is a dataset and benchmark challenging machine-learned representations to reproduce human perception of image similarity. As shown below, each image patch in the left has a corresponding similar image patch in the right.

The dataset consists of pairs of images, these are the positive pairs. Negative pairs are constructed by taking two different images, i.e. images that are not in the same pair initially. Following this approach, we construct triplets and use the TripletLoss. After fine-tuning, the embeddings of positive pairs are expected to be pulled closer, while the embeddings for negative pairs are expected to be pushed away.

Data

Our journey starts locally. We have to prepare the data and push it to the cloud and Finetuner will be able to get the dataset by its name. For this example, we already prepared the data, and we’ll provide the names of training and evaluation data (resnet-ttl-train-data and resnet-ttl-eval-data) directly to Finetuner.

Important

When working with Document where images are stored locally, please call doc.load_uri_to_image_tensor(width=224, height=224) or another image shape to reduce network transmission and speed up training.

Backbone model

Now let’s see which backbone models we can use. You can see available models either in choose backbone section or by calling describe_models().

For this example, we’re gonna go with resnet50.

Fine-tuning

From now on, all the action happens in the cloud!

First you need to login to Jina ecosystem:

import finetuner
finetuner.login()

Now, you can easily start a fine-tuning job with fit():

from finetuner.callback import EvaluationCallback

run = finetuner.fit(
    model='resnet50',
    run_name='resnet-tll',
    train_data='tll-train-da',
    batch_size=128,
    epochs=5,
    learning_rate=1e-5,
    cpu=False,
    callbacks=[
        EvaluationCallback(
            query_data='tll-test-query-da',
            index_data='tll-test-index-da',
        )
    ],
)

Let’s understand what this piece of code does:

finetuner.fit parameters

The only required arguments are model and train_data. We provide default values for others. Here is the full list of the parameters.

• As you can see, we have to provide the model which we picked before.

• We also set run_name and description, which are optional, but recommended in order to retrieve your run easily and have some context about it.

• Furthermore, we had to provide names of the train_data and eval_data.

• We set TripletMarginLoss.

• Additionally, we use BestModelCheckpoint to save the best model after each epoch and EvaluationCallback for evaluation.

• Lastly, we set number of epochs and provide a learning_rate.

Monitoring

Let’s check the status of the run.

print(run.status())

{'status': 'CREATED', 'details': 'Run submitted and awaits execution'}

Since some runs might take up to several hours, it’s important to know how to reconnect to Finetuner and retrieve your runs.

import finetuner
finetuner.login()

run = finetuner.get_run('resnet-tll')

You can continue monitoring the runs by checking the status - status() or the logs - logs().

Evaluating

Currently, we don’t have a user-friendly way to get evaluation metrics from the EvaluationCallback we initialized previously. What you can do for now is to call logs() in the end of the run and see evaluation results:

  Training [5/5] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76/76 0:00:00 0:03:15 • loss: 0.003
[16:39:13] DEBUG    Metric: 'model_average_precision' Value: 0.16603                                     __main__.py:202
           DEBUG    Metric: 'model_dcg_at_k' Value: 0.23632                                              __main__.py:202
           DEBUG    Metric: 'model_f1_score_at_k' Value: 0.03544                                         __main__.py:202
           DEBUG    Metric: 'model_hit_at_k' Value: 0.37209                                              __main__.py:202
           DEBUG    Metric: 'model_ndcg_at_k' Value: 0.23632                                             __main__.py:202
           DEBUG    Metric: 'model_precision_at_k' Value: 0.01860                                        __main__.py:202
           DEBUG    Metric: 'model_r_precision' Value: 0.16603                                           __main__.py:202
           DEBUG    Metric: 'model_recall_at_k' Value: 0.37209                                           __main__.py:202
           DEBUG    Metric: 'model_reciprocal_rank' Value: 0.16603                                       __main__.py:202
           INFO     Done ✨                                                                              __main__.py:204
           INFO     Saving fine-tuned models ...                                                         __main__.py:207
           INFO     Saving model 'model' in /usr/src/app/tuned-models/model ...                          __main__.py:218
           INFO     Pushing saved model to Hubble ...                                                    __main__.py:225
[16:39:41] INFO     Pushed model artifact ID: '62b33cb0037ad91ca7f20530'                                 __main__.py:231
           INFO     Finished 🚀                                                                          __main__.py:233                           __main__.py:248

Saving

After the run has finished successfully, you can download the tuned model on your local machine:

run.save_artifact('resnet-model')