Server issues July 2nd 2018

siovene
02 Jul, 2018 21:54
Today at around midday AstroBin stopped responding, or responded very slowly and intermittently.

Unfortunately I wasn't able to start debugging the issue until the evening, due to my busy day job schedule.I spent a few hours trying to discover the root cause, but I could not find anything certain.Because all other services and network connectivity seemed fine, I decided to attempt a database restore.

I restored a backup performed on July 1st at 4 PM GMT, and this seems to have solved the issue. I believe that the problem was a database corruption.

Apologies for the downtime and for the small loss of data.
Edited 02 Jul, 2018 21:55
siovene
02 Jul, 2018 22:47
I was able to narrow it down to the either the "Recent activity" or "Recent images" on the home page. It's too late at night for me to debug this now, so I turned off that section and now the rest of the website works perfectly. I will try to figure out what's wrong tomorrow. Likely a problematic image that is killing a worker process and impacting the performance on the whole website!
GWLopez
02 Jul, 2018 22:51
Thanks for the update, Salvatore.
SBRMichigan
03 Jul, 2018 01:23
No apologies needed, Salvatore! We appreciate all that you do for us! Without your efforts, this wonderful resource would not be available at all. Thank you!
dugstruble
03 Jul, 2018 01:30
Thanks Salvatore. You do an amazing job for being one man.

Salvatore Iovene
I was able to narrow it down to the either the "Recent activity" or "Recent images" on the home page. It's too late at night for me to debug this now, so I turned off that section and now the rest of the website works perfectly. I will try to figure out what's wrong tomorrow. Likely a problematic image that is killing a worker process and impacting the performance on the whole website!
franko
03 Jul, 2018 01:30
Yep, I was trying to upload an image at the time and got stuck during the acquisition details - I hope it was not me!  Nothing odd about the jpg, as far as I know.
Anyway, I think most of us appreciate having the site work virtually all the time, and we know how much effort is required.  I agree - no apology necessary.
Frank
Harris48Pod
03 Jul, 2018 03:55
Yes, thank you very much Salvatore…
dakloifarwa
03 Jul, 2018 19:47
Are the email notifications affected, too?
Thanks in advance, Andreas
geoflewis
03 Jul, 2018 21:03
Andreas Dietz
Are the email notifications affected, too?Thanks in advance, Andreas
I also have not seen any notifications for over 24 hours. Geof
siovene
04 Jul, 2018 05:27
I will check today.
siovene
04 Jul, 2018 09:17
Yes, it was broken, sorry! Fixed now.
echosud
04 Jul, 2018 11:25
Many thanks for all the time spent

alain
dakloifarwa
04 Jul, 2018 11:37
Thank you very much, Salvatore!
Andreas
GWLopez
04 Jul, 2018 14:58
Hi Salvatore. I'm still seeing the following message,

The recent images section is currently unavailable due to performance problems. They will be investigated as soon as possible. Sorry for the inconvenience!

I've dumped my browser cache, so I think that the message is still active. Is the Recent Images section active?
Gary
siovene
04 Jul, 2018 14:59
The recent images section is currently unavailable due to performance problems. They will be investigated as soon as possible. Sorry for the inconvenience!

smile
GWLopez
04 Jul, 2018 15:08
Ooops, sorry Salvatore. I did not realize that you were still sorting out the performance issues.
huerbsch
04 Jul, 2018 17:19
Image upload is not working for me…  smile
Starminer68
04 Jul, 2018 17:28
The same, I cannot upload png files, only jpeg smile
dagoldst
05 Jul, 2018 13:08
Salvatore,

I'm a network engineer for a telco here in the U.S.  I was wondering with all the symptoms being described if your server is having a DoS attack.   Something to think about.

Regards,

David
Edited 05 Jul, 2018 13:09
Epox
05 Jul, 2018 17:51
This is getting a bit longer than expected. An update, even if won't serve the purpose of restoring Astrobin functions, would be appreciated.

CS

Andrea
siovene
06 Jul, 2018 09:20
Image upload working fine for everybody, I see hundreds of new images. Just tested PNG and it's working. Home page action stream reenabled, looks like it's fine again.

Sorry everybody for the inconvenience! I did my best trying to keep up with this as fast as possible, but unfortunately this is not my main job, and my actual day job is very busy.

David Goldstein
I'm a network engineer for a telco here in the U.S.  I was wondering with all the symptoms being described if your server is having a DoS attack.   Something to think about.

Hi David,
no, at this point I'm pretty sure it was a problematic image on the home page action stream that for some reason burned thru all the resources of the main Docker container.
carlocolombo
06 Jul, 2018 12:21
Salvatore,  grazie sei grande, ciao c.colombo
Jir11
06 Jul, 2018 13:35
Thanks Sal for all your hard work.
Santiago
06 Jul, 2018 14:49
Thank you very much Salvatore for your really amazing work. We all know that you are doing your best to keep this wonderful place working. No apologies needed!
Epox
06 Jul, 2018 19:05
Thanks for the update and the fix! I really missed the global stream  smile
 
Register or login to create to post a reply.