What I've learned maintaining project with 300K results

A while ago I've started a project, nothing fancy just a small project which will aggregate car listing results from 5 websites at the beginning it was quite simple, straight forward the more things I've made the more complicated it became, queues, workers, jobs and lots and lots of API calls.

The up and running project: https://autosearch.laravel.cloud/

The Idea

Small website that will do the job, it turns out it won't be that, because the idea I had in mind was away different.

The Challenges

Jobs and workers

The first thing that I've stumbled upon was the processing, the kind of things that needed to be done it had to be inside of jobs which should be processed by workers. Since the listings are paginated I need a request for each page, to get them I need to make requests every few pages need it to be in a separate job, so if I need to take the results from 100 pages I need to fire 10 jobs, the catch: Every website is different, the queue job can take 9 second or 12 depends. In the end I started with 3 workers it can take the load.

For every type of listings I have a different jobs:

• New listings - initial store of the listings

• Existing listings - updating the information for the listings(images, descriptions, price)

• Expired(Removed) listings - the listings that were sold or expired and removed from the websites

Requests

In the beginning It was straight forward: make request, get the results and save them, after a while I've learned it's not that simple, the more request I've made in quick succession the more results were loaded, but the problem was that they were saved in the RAM memory and that caused problems.

Database

For such project I couldn't just use any DB I need it something that can help to easily store the records and make it easy and fast to be filtered for search purposes, so naturally I've turned at PostgreSQL .

Store the records

For the storing the best option I had was to use upsert since the nature of the project is that the records will be updated frequently and also helps with the deduplication.

Front end

For the presentation I wanted to go "plain and simple", no fancy AI generated pages and glamours elements with gradient colors, just Livewire components for which the main focus is to be easy to use and to work fast on web and mobile.

The idea was to have the proper information present in an accessible way so the users can quickly find what they need. For every listing the following information is displayed:

• Image - one of the images uploaded

• Name - the main thing for a listing

• Description - short and sweet nothing excessive

• Production year - something every buyer looks for a listing

• Fuel type - we are living in times that this is also very important peace of information

• Mileage - essential for information for cars in general

• Created at - the date the listings was initially created

• Updated at - the date when it was last updated

• Transmission - pretty essential to know if want to buy a car

The solutions

So I've found a solution for the problem: unset for every requests response and manually running the gc_collect_cycles to ensure that everything was properly cleaned up and the memory is freed up. With those changes I was able to run the project on Flex 512 MB RAM cluster on Laravel Cloud, it process 160 jobs every 30 minutes for the newest listings, beside that a command for updating listing runs daily and cleaning up command which runs every 3 days

For each website I've created separate class that handles the retrieval of the listings, each of the extends basic class which implements interface in which there are common methods that are used in the processing of each. As helpers I have a few helpers which are used for processing information, storing the records in the DB, deduplication.

Every website has different structure and different quirks to it, that's why there are different logic for them, but in the end the structure of the data is used for storing them in the database is the same for all of them, as is the display and the FE.

For every listing sources are displayed, since everyone wants as much exposure for it's listings as possible, that's why they create listing for the same vehicle in every of the top websites that's why I had to came up with a solution for way of telling if such vehicle was already stored in the DB, you can read more in my article about deduplication.

Example of an listing from the website.

Conclusion

It wasn't the easiest of implementation I got to admit it, my knowledge was tested and I picked up a few new things a long the way, it runs for 3 months now, with more than 276 thousand records and thousands more are added every day. Over 10 thousand jobs ran, only 0.1% failed (those were caused by me and the settings I forgot to do initially) thanks to Laravel Cloud's impeccable queue implementation everything runs smoothly and I'm able to add new little fixes and features here and there.