Data deduplication using images

A while back I've started personal project for something I've stumbled upon a few times and it was cumbersome and I thought about a solution that can make it better for everyone.

Car listings search: in Bulgaria there are 5 main websites in which people are uploading their listings, the thing is you need to go over all of them to search for what you need what if you could do it in one place?

That's why I've started the project, aggregating results from all platforms in one place, with only one search you are able to find listings from all websites, the catch? Most of the people are uploading the same listing across all platforms, I needed a way to avoid storing duplicates and here is where the deduplication comes in play.

Since in 95% of the cases the listing has the same pictures in all of them this is how I can check if I've already stored such listing.

Using: jenssegers/imagehash I was able to create unique Hash for every listing using the image uploaded, it takes the main color of the image and it's representation that way I'm able to distinguish them and to avoid duplicates by checking if a listing with this hash has been uploaded already, in very rare case people upload one image into one platform and completely different one on other.

public static function make(?string $imageUrl, ?array $params = null): string
{
    ...

    try {
        $response = Http::retry(3)->connectTimeout(5)->timeout(10)->get($imageUrl);

        $hasher = new ImageHash(new PerceptualHash);
        $hash = $hasher->hash($response->body());

        return $hash->toHex();
    } catch (\Throwable $exception) {
        return self::paramsHash($params);
    }
}

if no image is provided I'm using:

private static function paramsHash(?array $params): string
{
   $mileage = Arr::get($params, 'mileage');
   $year = Arr::get($params, 'year');
   $fuelType = Arr::get($params, 'fuel_type');
   $price = Arr::get($params, 'price');

   return hash('sha256', "$mileage $year $fuelType $price");
}

Most of the cases those fields are enough to create a unique hash which is used for the deduplication check.

If there is not an image provided(the listing is very recent) I'm using a combination of 4 other fields from the listing to determine if i already have it stored.

Conclusion

Deduplication is a power thing using the proper tools, working on this project I've learned a lot of things especially for queues and managing huge amount if results, but more on that in later articles.