Cookie consent

AstroBin saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing AstroBin without changing the browser settings, you grant us permission to store that information on your device.

I agree

Update about the status of the search index

03 Nov, 2017 21:10
as many of you have undoubtedly noticed, the search engine is currently not working on AstroBin.

Yesterday I did a major server upgrade, which was something very important and necessary (I have brought the whole AstroBin stack pretty much up-to-date with recent versions of all direct and indirect dependencies).

The upgrade, so massive in nature, was not free of problems of course, but most of them were quick and easy fixes.

However, there has been a performance problem with the search engine. Building it would completely bring AstroBin database to its knees.

AstroBin's search index is comprised of 3 elements:
- users
- gear
- images

Users and gear indexed fine (that's why most of you have your AstroBin Index back), but if I try to index the images, everything collapses.

I have now setup a secondary server for the search index (notice how the main server is SO FAST? :-) ) and I'm currently getting a copy of AstroBin's database there, to isolate the generation of the index from the main server completely.

Hopefully this will work and we will have a search index again!

After that, I will experiment with real-time index updates, so it's not necessary to update the whole index each night. That would be nice!

So fingers crossed for tonights index update on the secondary server!

As always, thank you all for your patience… and I apologise for any inconvenience and downtime of the website: I do myself to take care of it, but it's a very demanding job for a single person (especially when it's not my full-time job, unfortunately :-P )

- Salvatore
03 Nov, 2017 21:56
So the problem seems to be the very presence of images in the search index. Not the indexing itself. Unfortunately I really need to get some sleep now, but I will continue my investigations ASAP. Meanwhile we need to be without the search function for a while… sorry!
03 Nov, 2017 22:37
Good luck with getting the Search working again.
It is, in my opinion, absolutely vital. I use it all the time. So valuable!
04 Nov, 2017 01:25
Good news everyone! It's 3:30 in the morning but I finally cracked the nut :-D

Search index now updating, and performance isn't suffering :-)

Good night…
04 Nov, 2017 01:31
Salvatore thanks for all hard work on server, for such a large job I think any inconvenience was very minimal.

Best wishes,
04 Nov, 2017 01:32
Thank you Sigga and everyone. It was a very intense couple of days, but it was worth it! AstroBin is now more secure, more stable, and also faster!
I got a lot of emails and messages of support and thanks for the hard work, and I really really really appreciate them!

04 Nov, 2017 07:24
Good to know that things are more updated, secure and stable!

BTW, perhaps there should be a small indicator whenever the index gets reset. Seems to confuse a few people every time that happens!

04 Nov, 2017 08:14
Indeed Ethan, yesterday I got emails and messages in the hundreds, about it… Anyway, I will work on a real time index update. I need to see if it impacts the performance of the website if I enqueue real time index updates.
04 Nov, 2017 09:06
Thank you so much for all your hard work sorting that out Salvatore, for me that is one of the most important features. Great to have it back again.

04 Nov, 2017 09:23
Search and filters work well and smoothly, and everything else I try too. Good job…
04 Nov, 2017 11:44
Good news!
The real-time index works! Now whenever you upload an image, it's going to be available on the Wall and in the Search right away :-)

Calculated indexed things, like avg integration time, number of likes, etc, are still done as a daily job for now, but I would love to update those in real-time too… let's see ;-)
04 Nov, 2017 12:41
Great job Salvatore,
The search function was one of my most used features on Astrobin, so it's great to have it back… :-)
Best regards,
04 Nov, 2017 19:24
Brian Morgan
Hi Salvatore,
As a developer I am in awe of the sheer magnitude of data that you need to track, together with the burden of keeping paid customers happy! May I ask what database technology(ies) you use: relational, non-relational, or a combination of both?
Great work,
Hi Brian, I’m pretty amazed myself by what AstroBin turned out to be, and yeah it’s a pretty huge affair, especially considering that it’s a one man show and I do it as a side job.

The backend is Django, PostgreSQL, memcached, celery/rabbitmq.
Register or login to create to post a reply.