# 16 Feb, 2020 11:49
Today suddenly there's lots of thumbnails missing on AstroBin. I'm investigating, thanks in advance for your patience!|
UPDATE 2020-02-16 8:18 PM GMT
It appears that something serious happened on AstroBin today.
I've now been investigating and trying to mitigate for 12 hours straight, but unfortunately I am certain that a currently unknown number of you have suffered the loss of some (many, in some cases) original uploads.
The data associated to images, on AstroBin database (which is a different system than the place where the image files are stored), is safe.
I don't know the reason yet, but in the meanwhile I have changed all the passwords used by AstroBin (database, storage, servers, etc). Note that this has nothing to do with your passwords you use to log in to AstroBin: they are encrypted.
For now if you see the message "We are very, very sorry, but it appears that your image suffered a catastrophic data corruption", then that image is gone.
Luckily you don't need to start over and lose the data, the comments, likes etc. You only need to replace the image file on that form, and hopefully you still have it on your computer.
Please visit your gallery after log-in, and the image thumbnails with the sad face are affected. If you are not logged-in, affected images will be hidden.
I can't even begin to explain how much I'm currently in shock, literally, and this is why this message is intentionally brief.
I will continue to investigate, and I’m awaiting help from the Amazon Web Services tech support.
In the meantime, if you are heavily affected, I want to say that I am incredibly sorry, though at this time I don’t know where the fault lies.
After the situation settles and I know more, I will offer compensation for the data loss with free subscription upgrades.
Thank you for your understanding and please keep an eye on this forum thread for updates.
UPDATE 2020-02-17 00:00 AM GMT
Going to bed now, at this time there's nothing more I can do after 16 hours. Just a heads up if you ended up here after I didn't reply to your email: I'm being swamped with messages and I'll try my best to reply to all, but it's a lot. I expect to have several hundreds emails when I wake up. The site is currently stable.
UPDATE 2020-02-17 8:38 AM GMT
Good morning. I got some sleep between 2 and 6:30 am, but unfortunately I'm not feeling great. It doesn't help that I have been sick with flu for a few days.
I want to thank all of those who strongly empathized with me: I received many messages of support and understanding. I'm feeling numb and quite in shock, and extremely embarrassed that this happened. I literally want to go and hide in a hole right now.
The current situation is that I'm still awaiting the Amazon tech support to get back to me. They said response time was 12 hours and it's been 20, but it was Sunday.
I hope they can do a data recovery, but at the core of this lies the fact that I misunderstood the core principle of data safety on Amazon AWS. The AWS S3 FAQ states:
"your objects are automatically stored across multiple devices spanning a minimum of three Availability Zones, each separated by miles across an AWS Region. Objects stored in the S3 One Zone-IA storage class are stored redundantly within a single Availability Zone in the AWS Region you select."
When I started AstroBin in late 2010, I incorrectly assumed that this level of redundancy would protect me against data loss. I still hope so but reading this in 2020, it doesn't send quite the same message. Unfortunately I didn't know about "versioning" on S3 back then, and data replication was only introduced in 2015 and I didn't know that either.
AWS S3 for me has been amazing in the way it was a "setup and forget" thing that just worked, and I might have been under the wrong assumption about data protection all these years. Plus I intentionally didn't put any code on AstroBin that would delete anything from S3, so avoid bugs that would delete something I didn't mean to delete (this is at the cost of having lots of unneeded data lying around that costs money to host). We will know once tech support gets back to me today.
I have now enabled both versioning and data replication, and I'm working on having a 3rd backup option in AWS Glacier (long term, slow retrieval data storage).
If AWS tech support cannot retrieve the lost data, this is truly all my fault. It would be easy to blame it on the lack of time, on the fact that AstroBin has been a side gig for me so far after my day job and my parental duties as a father two school kids, and the fact that working alone on it was always a struggle to just stay afloat. It would be easy but I won't do it: I should have prioritized making sure that the my AWS configuration was more solid. I should've paid a consultant to review my configuration and find this issue. Shoulda woulda coulda.
I want to thank everybody who has been replacing their images so far, those who have offered words of comfort (it has truly been awful being in my shoes the past 24 hours). I would like to offer my understanding to those who decided to leave.
To everybody who sticks around, I will offer compensation in terms of free Premium time and/or free subscription upgrades. I have been working really hard the past 2 months to offer a new exciting feature on AstroBin that I was going to launch this month, and was extremely excited about, but it looks like I'm going from hero to zero.
Meanwhile, stay tuned for updates.
UPDATE 2020-02-17 10:50 AM GMT
While I still don't know if the missing data can be recovered (still waiting for Amazon), it appears that a lot of images show the sad face even if the original is still there. I'm investigating how to to fix those automatically. If you see one, click on the Actions menu and then "Flag malfunctioning thumbnails" and in many cases that will repair your image.
I'm overwhelmed by the messages of support, thank you all!
Personally I'm not giving up on AstroBin, as long as you're not!
UPDATE 2020-02-17 6:11 PM GMT
The current information that I have from Amazon's customer support is that the lost images are irrevocably, permanently lost. I'm still trying to escalate the issue as I have a hard time believing it, but it might as well be the truth.
On the plus side, I really don't know how to express how amazed I am at the support I am receiving. Only a handful of people have cancelled their subscriptions or deleted their account, and I have received at least a hundred messages of support: here, on PM, as emails, even reading them on CloudyNights. It seems that the general sentiment is that while this is very bad, people are interested in the continuity of AstroBin as a community first, and as a platform second. My greatest fear was that AstroBin wouldn't survive this: not for the money it makes me on the side, but for my investment in it for the past 10 years, for the legacy it possesses and for everybody here who makes this community great.
It seems that this fear might be unfounded, as so many of you have been telling me that they will stick with AstroBin. In many cases, you have most images stored on your computer or external HDD, and will be replacing them.
I feel so overwhelmed with positive emotions, you bunch are incredible
UPDATE 2020-02-17 09:10 PM GMT
I was able to write code that makes it so all these "false positives" are automatically fixed. By false positive, I mean that you get the :'( face on the thumbnail, but then click on the image and are able to recover it with Edit -> Thumbnails or Actions -> Flag malfunctioning thumbnails.
This is putting a bit of strain on AstroBin's server, so I doubled the capacity again (up to 4x from normal, but I don't care about the bill at this point). I will keep an eye on it, I see that there have been some 502/504 timeout errors. If it proves to be too much I will turn this fix off for a while. Right now the traffic on AstroBin is the highest it's ever been. A lot of people on this thread, and a lot of people re-uploading images.
So now if you see that you get the hour glass on your thumbnail, just give it a minute and perhaps refresh the page: AstroBin will be begin work to salvage the salvageable every time a thumbnail is viewed.
I just checked several profiles and they were magically back!
UPDATE 2020-02-17 10:33 PM GMT
Senior engineering staff at Amazon is looking more deeply into the issues to see if they can salvage anything!
UPDATE 2020-02-18 6:54 AM GMT
Unfortunately Amazon weren't able to save anything, despite trying.
Many of you have been busy fixing your galleries, which is AMAZING.
YOU are saving AstroBin, so I owe you all my thanks. Yesterday, literally only 6 people deleted their accounts, and one of them asked me if it can be restored I've said it enough so I won't bore you more, but the response has been incredibly positive.
Today I'm going to publish a statement with the full explanation of what happened, how, what's been done to prevent it will happen again, what to do moving forward.
My flu is also gone (tho my daughter has it now), so today I need to work on my day job and I won't be active here until evening, except for lunch break.
UPDATE 2020-02-18 12:20 PM GMT
I reployed a fix for an issue with corrupted (read: missing) revision image files: so now you get redirect to the correct edit page. Additionally, once you've fixed the main image page, if needed, you can fix or delete individual revisions by using the small buttons that appear when you hover your mouse on your revision thumbnail on the main image's page.
UPDATE 2020-02-18 1:48 PM GMT
I published a statement here.
UPDATE 2020-02-18 7:51 PM GMT
I just deployed a fix that lets you access images that are corrupted, but the final revision is fine. This should solve the problem that some of you have been having, trying to upload an image again but getting stuck.
UPDATE 2020-02-19 00:01 AM GMT
Just had a very pleasant call with the director of Amazon S3 and some product owners/engineers within the AWS organization. They were very interested in hearing my feedback and offering explanations that cleared my misconception about S3 that caused this issue in the first place. I really want to thank them for their time, first of all in attempting to recover the data, and secondly in being so attentive towards a "small fish" like me and AstroBin.
As it's now 1 AM on the third day with very little sleep, I finally can get some rest knowing that the website is stable and many of you are rebuilding. Tomorrow I have to be at my day job
Thanks again to everybody who showed me support: hundreds of messages on forums, PM, email, and Facebook. I replied to most, I think, but if I missed some that needed a reply, please don't hesitate to ping me again.
UPDATE 2020-02-19 10:07 AM GMT
Fixed an issue that caused corrupted images that you deleted to come back from the grave and appear again in your gallery (only visible to you tho). So if you delete some broken images and see them again, please delete again)
UPDATE 2020-02-19 01:10 PM GMT
I deleted all the references to missing avatars, so now instead of a broken (X) image, you get the default anonymous avatar. You can update your own avatar here: https://www.astrobin.com/avatar/change/
UPDATE 2020-02-19 08:16 PM GMT
I just want to once more assure everyone of the following facts, since I get questions:
1. There are no more images getting deleted. There haven't been since Sunday morning when I noticed the incident.
2. There are no security breaches, and there have never been. Passwords are encrypted. Credit cards have never been stored on AstroBin at all.
I would also like to inform that sometimes AstroBin can be a little slow, these days, due to very intense traffic. Not just regular traffic (which is only about 1.5-2 times as much as normal) but heavy traffic with image uploads. For those who find this interesting, I have scaled the server up to 32 CPUs and 68 GB RAM, and I'm running multiple AstroBin services in parallel to accommodate the demands.
Such slowness might cause delays in opening pages, and loading thumbnails, which might get them stuck on the hourglass symbol, but there's no reason to panic.
Hope this helps!
UPDATE 2020-02-20 07:17 AM GMT
For peace of mind, if you reuploaded an image and see that the thumbnail is slow to come back, go to Edit basic information and click on the “current image file” link. That’s your original and you will confirm that it’s fine. AstroBin is still under high load but is fine! This is also now a way to download your originals anywhere, before I implement a bulk export.
UPDATE 2020-02-21 08:28 PM GMT
- I have just fixed a bug that would cause incorrect thumbnails when replacing an image with one of a different size (same for revisions). It was always there but we never noticed because replacing images wasn't possible before Judgement Day.
- I have also fixed a bug that prevented the image cropper to work when editing a revision. Remember you can replace individual revisions and even re-crop the thumbnail by clicking on the pencil icon on a revision.
- Today I ran a script to fix the missing plate-solving annotations, which unfortunately caused a lot of notifications, sorry about that. I will fix the script to skip the notifications and run it again for the remaining annotations that need to be fixed.
- You guys are awesome! In less than 4 days you have restored 25% of the lost content!
Do you want more updates? Follow AstroBin on Twitter!
# 16 Feb, 2020 12:17
Thanks Sal, I don't know if it's related, but many of my images appear to be missing even though the thumbnail is there. I've tried different browsers and laptop, iPad and iPhone and get different results, some show thumbnail image, some show blank thumbnail, but using each device there is no image displayed when I click the thumbnail.|
Hope you can fix it.
I just wanted to update my above post to you, but first let me say very clearly, that my first concern is that you do not beat yourself up over this bad experience. It may seem like a tragedy to you and others right now, but in my opinion it is not; as others said, 'no one died…!!'. Tomorrow I go into hospital for a cancer related biopsy and I certainly hope that the biopsy is clear, but of course I wouldn't be having the procedure without the fear that something nasty may be found. For sure, if I get the choice, I will be happy to find that I have no cancer AND no images on Astrobin than the other way round….!!
Currently it seems that all of my images are missing. When I first click on a thumbnail and hover over the image location I see the image, but if I try to open it I get the message that it is lost; the same applies if I try the option to restore the thumbnails. I may try to reload one or 2 images, but probably I will wait till after I get back from hospital to do anything more, hoping that by then you may have found a solution.
For me Astrobin has been the best astronomy image hosting site available, so I hope that you can recover it and will continue with it. I also maintain many of my final images on Flickr (but there is no facility for revisions to be maintained there), as well as back ups to several external hard drives, so for me all is not lost, even if it takes me time to restore images to Astrobin.
Take care my friend.
# 16 Feb, 2020 12:19
Every image that you see on AstroBin I refer to as a thumbnail. Except the original upload, that is never seen anywhere, and is stored securely. Those are intact of course, so worse case scenario, a lot of thumbnails will have to be re-generated which will be very expensive for me.|
*edit, this doesn't seem to be true right now, see few messages below.
# 16 Feb, 2020 12:51
best of luck - if it helps - it is not every image that is causing this|
Following is present
but the following (same user - me) are missing
# 16 Feb, 2020 12:55
on me it appears to be date related - older images not present|
Present https://www.astrobin.com/id7pjl/ (25th Dec 2019)
Missing https://www.astrobin.com/0ayjun/ (23rd Dec 2019)
# 16 Feb, 2020 14:38
same issue in my case. Many images are only red fields but the images fortunately do not seem to be lost. I wish you good luck for fixing the issue.
many thanks in advance
# 16 Feb, 2020 16:22
Good luck for repair .|
I 'm using API to display images on my website. Hope you will find the problem very soon
# 16 Feb, 2020 16:48
Well what can I say other than Thanks a lot to everybody :-)
It's time for me to move on, to try something else, new :-)
If you need to reach me, please just look here, thanks a lot,
Kind regards Niels
Correction, seem all my org. image uploads are gone except the last one uploaded.
As far as I can see I have lost 258 org. images out of 260!
5 years work just gone down the drain.
I will double check the status tomorrow.
If this is the case I will stop using astrobin and delete my account.
Sure good luck with the repair!
Fyi, 2 thumbnails visible out of 260 uploaded images the last 5 years,
# 16 Feb, 2020 17:09
|Having similar issue here. If you need help let me know.|
# 16 Feb, 2020 19:21
Hi Salvatore. I have the same problem. All my images, except two an disappeared.|
https://www.astrobin.com/users/Fotoixigo/Greetings and thank you.
# 16 Feb, 2020 19:50
Same here, I'm getting this error message when trying to access my images:
"We are very, very sorry, but it appears that your image suffered a catastrophic data corruption, and it cannot be retrieved. Hopefully you still have your file (the information on this form should provide you with enough context).
Please use the file input below to upload it again, and then click on the Save button down below. All the data you had previously associated to your image is still here."
Salvatore, from your message above, do I understand corectly that we should all wait until we re-upload images, as suggested by the error message ?
# 16 Feb, 2020 20:00
it appears that the problem is more serious than I thought. I've now been investigating and trying to mitigate for 12 hours straight, but unfortunately I am certain that a currently unknown number of you have suffered the loss of some (many, in some cases) original uploads.
The data associated to images, on AstroBin database (which is a different system than the place where the images are stored), is saved.
I don't know the reason yet, but in the meanwhile I have changed all the passwords used by AstroBin (database, storage, etc)
(this has nothing to do with your passwords you use to log in to AstroBin, they are encrypted)
I will update with my findings, but for now if you see the message "We are very, very sorry, but it appears that your image suffered a catastrophic data corruption", then that image is gone. Luckily you don't need to start over and lose the data, the comments, like etc. You only need to replace the image on that form, and hopefully you still have it on your computer.
I can't even begin to explain how much I'm currently in shock, literally.
I will not write too much right now while I gather more information and my thoughts.
I am so, so sorry about this.
# 16 Feb, 2020 20:02
|Hi, I'm not just missing the thumbnails, my images are gone too|
# 16 Feb, 2020 20:34
I'm trying to re-upload my images, but it is tricky with the image revisions.|
I'm afraid I'll have to upload new revisions even if they are not new …
# 16 Feb, 2020 20:36
Revisions can be corrupted as well, individually. I will now add a file field in the "edit revision" form so that corrupted revisions can be replaced too.
# 16 Feb, 2020 20:37
More precisely for images with multiple revisions, I'm getting the error message "We are very, very sorry, but it appears that your image suffered a catastrophic data corruption" only for the original version, not for the others. For the others I only get the sad smiley face but no way to re-upload the image.|
# 16 Feb, 2020 20:37
Aside from my very last image uploaded on 02/15/2020 all my images are gone.|
Can you please explain something, did you lose the images, the thumbnails built by the system for the images, or the db table that associates them? No backup available?
# 16 Feb, 2020 20:38
Salvatore IoveneFrédéric Auchère
# 16 Feb, 2020 20:41
The images were lost. I don't know how many but I have a process in place to figure it out. As many thumbnails as the original images too, but they get regenerated *if* the original is still there.
DB tables are fine.
I have support for versioning (i.e. undelete) and a replica of the storage in a separate location. But both the versioning information and the replica are gone.
# 16 Feb, 2020 21:01
Sounds like cascading data loss, the data corruption of primary cascaded to the replica and recent versions of the aws backup?|
That’s a tough one I’ve been there. I hope no one has it as bad as I did all my images and revisions except the last image are gone.
# 16 Feb, 2020 21:02
|On some of my lost images, the hover over plate solved version is still present.|
# 16 Feb, 2020 21:02
|Mine were there earlier in the day. Now several are gone (including todays IOTD). Thankfully I have backups to upload later. Also the site is running very slow.|
# 16 Feb, 2020 21:08
|A few hours ago some image was missing and now all are missing.Definitely all my images and reviews have disappeared. Is it safe to load the images again? (I don't think I can all) or is it better to wait? 😞😞|
# 16 Feb, 2020 21:24
Salvatore, I'm really sorry to hear that. I hope you can find a quick and complete solution to the data corruption.|
I must say many images appear in my gallery as a sad face [ :'( ], but when clicking on them the image and its page loads just fine. Did not check all of them, though. If there is irreparable corruption on the original files, I might be able to reupload all of them back (or, at least, most of them), but I really hope this is not the case, especially for the great images on here of people that might not have the originals or are not active anymore.
Wishing the best of luck and following the updates with attention,
# 16 Feb, 2020 21:24
I think you probably know I like you and astrobin.|
But there's no way I could _possibly_ reload all my images and revisions for the last three years. They're scattered over a number
of old hard disks. _Astrobin_ was where I organized them.
Reloading everything would be an enormous effort. If this cannot be not fixed, I simply _have_ to move to another location.
Probably my own website, where I can take my own responsibility for what happens.
|You have no new notifications.|
This page or operation is not available at the moment, because AstroBin is in READ ONLY mode. For more information, please check out our Twitter feed: https://twitter.com/AstroBin_com
This feature is only offered at higher membership levels.
Would you be interested in upgrading? AstroBin is a very small business and your support would mean a lot!