Cookie consent

AstroBin saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing AstroBin without changing the browser settings, you grant us permission to store that information on your device.

I agree

Official statement and post-mortem about the data loss

siovene
18 Feb, 2020 12:53
In the following document I detailed all the information I have about the data loss.

  • How it happened
  • How I'm making sure it won't happen again
  • What to do moving forward
  • How I plan to offer compensation to those who want it
  • Frequently asked questions
I will update the document if somebody asks questions that I think should be answered there. It's pretty long tho, so thanks for your time if you read it!

Available at:

Thank you,
Salvatore
Giamba
18 Feb, 2020 13:05
Buongiorno Salvatore, leggo che a diversi utenti le immagini stanno tornando. C’è qualche speranza anche per le mie o devo rassegnarmi?
siovene
18 Feb, 2020 13:14
Giambattista Rizzo
Buongiorno Salvatore, leggo che a diversi utenti le immagini stanno tornando. C’è qualche speranza anche per le mie o devo rassegnarmi?
Ciao Gianbattista. Purtroppo no, quello che doveva tornare sarebbe già tornato (che in realtà erano falsi positivi). Mi dispiace molto.
IamTomzo
18 Feb, 2020 14:10
Salvatore - I only had a few dozen images and was easily able to repopulate all of my images yesterday - all is well on my end.  To echo what others have said, the value of Astrobin to me  is not just the images themselves, it is the awesome amount of information about how the image was captured that allows me to see how others get to certain results and allows me to work on my imaging skills.   The simple fact that this data was not lost reduced the time needed to restore images tremendously as when you use four different filters at two different exposure lengths each over multiple nights that part of data entry can take a while.

I will certainly not be among those asking for a refund and will be among those willing to pay a bit more to keep your awesome site alive.  I am not sure what other service can hold a candle to Astrobin - most won't even allow a full resolution image to be displayed on screen.

On a side note, as a bulwark against constant spearphishing attacks and the threat of ransomware at the public agency I work for, we started using a service called Wasabi that offers immutable storage at a fraction of the cost of S3.   We do our main backups in Azure but have a secondary system with Wasabi that, from what my IT guys tell me, would not allow what happened to you to happen again.   I am not sure how many TB of data you have overall, but in these cases belts and suspenders together is something that allows for piece of mind - and disaster recovery should anything occur again in the future.

Anyway, I wish you well and look forward to a long and healthy life for Astrobin!
AtmosFearIC
18 Feb, 2020 14:19
When I first checked after the mishap some 80% of my images had sad faces but to echo the sentiment of some others… it's only images. It would have been a real inconvenience as I don't keep backups of most of my processed images as I have no intent of printing them or really displaying them outside of Astrobin.
I've since found that I'm one of the lucky ones that didn't lose anything. As far as I can tell anyway.

Regardless of that, no system is fail proof and these things happen. Whether it was your direct mistake (compounded with a terrible UI) or Amazon having a critical failure on their end; in the grand scheme of things it's not the end of the world. An inconvenience to many, for sure, but mostly recoverable.

I'm not going anywhere, Astrobin is a great community and it would be a real shame for the whole astronomical community if something like this was to bring it to its knees.
wcoastsands
18 Feb, 2020 14:20
My condolences on the loss of all that data. What a nightmare situation. Thank you for your dedication, your hard work, and your responsiveness to the situation and to the community.
Edited 18 Feb, 2020 14:20
Joshm1084
18 Feb, 2020 14:50
Salvatore, I would just like to express my support for AstroBin! I've only been able to recover around 1/3 of my images thus far, but it is a learning experience going forward and I will happily be renewing my subscription in a few months. All great platforms have their speedbumps, keep up the good work!
w4sm
18 Feb, 2020 15:14
Salvatore, thank you for the detailed description of what happened. Your responsiveness and openness about this says a LOT about the kind of honorable person you are.  This is a fantastic site.  I'm not going anywhere and I don't need a refund.   I had no trouble reloading my ~60 images yesterday.  This  issue did cause me to rethink how I back up my own data  at my home, and in addition to reloading my Astrobin images yesterday, I burned a new copy of all my processed images to a BluRay disk.   I already had backups of all the raw files.  As the saying goes, put this episode in the rear view mirror and move forward.  Thank you for your excellent site.
groberts
18 Feb, 2020 15:17
First, thanks Salvatore for starting and running Astrobin.  I only joined just over a year ago but appreciate that it's highly valued amongst the astrophotography community and perhaps, therein lies the problem - it's success poses new problems and demands.  For me the  elephant in the room  that arises from this issue is that Astrobin seems to rely on just one person - you.  Whatever your personal abilities, dedication and passion, it seems very clear that the 'product' requires a broader level of support  (a) to ensure there's wider oversight, knowledge and support of the system, and (b) to enable Astrobin to develop further + © what happens if you fall under the proverbial bus i.e. at the moment it seems Astrobin is then without it's leader, worker, support!

My suggestion is to treat this as a wake-up call to address the aforementioned issues and go forwards to make Astrobin an even better facility. I don't have the detailed answers and I expect money may be an issue - maybe its possible to reach out to those who make their living selling us all our expensive equipment for help in this regard through sponsorship?

From starting astrophotogaphy a few years ago and recently joining Astrobin I'm sure of one thing, the majority of us will remain with Astrobin and wish you well in taking it forwards. I know I will.

Graham
Edited 18 Feb, 2020 15:37
mikko.viljamaa@gmail.com
18 Feb, 2020 15:25
Well written Salvatore, thanks.
- Mikko
Okke_Dillen
18 Feb, 2020 15:35
Cheer-up! I came here to stay!

And finally get some rest now!
Mistakes… happen! It's the way they are handled that makes the difference!!!
More people out there with your attitude would make this world a better place!

Okke smile
teoria_del_big_bang
18 Feb, 2020 15:37
Salvatore,
Many thanks for your timely, in depth and deeply honest explanation, you never even try to make any excuse and I admire that.
As I said previously I was a free user (now Lite) and so was no problem for me but I know others have suffered and as you say some images will never be recovered.
I think your principles in creating and getting Astrobin running is commendable and certainly needs to continuing.
You have had a terrible few days but I think time to stop beating yourself up about the issue, it serves no purpose.
This is my personal opinion, others are entitled their own views, that is all part of having a forum. It also seems to be the opinion of many other users.
I hope you can recover and continue to keep Astrobin going.

Steve
Edited 18 Feb, 2020 16:05
jerryyyyy
18 Feb, 2020 15:38
Hey Salvador, hang in there.

I have my whole professional life on AWS (all the data from my Lab at Stanford is up there).  This represents the life's work of maybe 20-30 professionals in my Lab.

That lifecycle screenshot you showed is KEY  to survival on AWS.  When I started with the guidance of Amazon they said that aside from S3 you should automatically make a copy in glacier…. at least they told me to watch out.  So I do this.  Even without a mistake, they only provide a 99.99999999% assurance to security in S3.

Thus, our primary data is on two Windows servers with a RAID configuration and they mirror each other.  (Never trust RAID if the power goes down hard you can lose > 2 disks.)  There is a third linux (CENTOS) server that mirrors the Windows machines.  The Windows machines also keep versions.  Each has 32TB. The Linux I think is 72 TB.

A basic principle of data security is the three legged stool:  The data (1); a second independent on site backup (2); an off-site backup (3).  This withstands destruction of your server room.

Anyway, you cannot reproduce this as our Lab's budget in in 7 figures.  I just present it as a learning exercise smile

I'll hang in there also, since with all my talk about backups I do have a personal backup of my files… how do I do it.  It is not the best but I use a Network Storage Device with two disks and the one mirrors the other….  I also occasionally copy the whole dataset to a second computer (reminds me I better do that).  The storage gets out of hand so I do not keep my original images put primarily the processed masters from PixInsight as well as my PixInsight project with all their workflow and most of the intermediate images (as well as the "Final" jpg).  My images are not that great that I need to keep everything and it just gets out of hand.

Again hang in there.  The social aspects of Astrobin are crucial and i have been helped over the years by all the great feedback from all over the word.  We will survive… after all it is only star-gazing!
Edited 18 Feb, 2020 15:39
siovene
18 Feb, 2020 15:47
Graham Roberts
First, thanks Salvatore for starting and running Astrobin.  I only joined just over a year ago but appreciate that it's highly valued amongst the astrophotography community and perhaps, therein lies the problem - it's success poses new problems and demands.  For me the  elephant in the room  that arises from this issue is that Astrobin seems to rely on just one person - you.  Whatever your personal abilities, dedication and passion, it seems very clear that the 'product' requires a broader level of support  (a) to ensure there's wider oversight, knowledge and support of the system, and (b) to enable Astrobin to develop further + © what happens if you fall under the proverbial bus i.e. at the moment it seems Astrobin is then without it's leader, worker, support!My suggestion is to treat this as a wake-up call to address the aforementioned issues and go forwards to make Astrobin an even better facility. I don't have the detailed answers and I expect money may be an issue - maybe its possible to reach out to those who make their living selling us all our expensive equipment for help in this regard through sponsorship?

From starting astrophotogaphy a few years ago and recently joining Astrobin I'm sure of one thing, the majority of us will remain with Astrobin and wish you well in taking it forwards. I know I will.

Hi Graham, thanks for the message. Very valid points.

I agree that fundamentally AstroBin is bigger than me if I work on it part-time. I know that I could figure out enough work to do for a team of 10 or more, but obviously I haven't been able to make it profitable enough to support me (and my family), let alone a team of partners!

Things would be different if I could afford to live solely off AstroBin and dedicate all my mental energy to it, and 40 hours a week. The problem is not only that I only have 5-10 hours a week (10 hours is very optimistic, only on intense weeks), but also that my day job as a lead software engineer is quite mentally draining, although I really enjoy it (my boss is probably not reading but he knows :-D )

I am working towards the dream of making AstroBin my only source of income, and this is sad because I feel like I just went from riches to rags, in terms of dreams: I was going to announce something new on AstroBin, and a new subscription tier above Premium. Now tho, with this dream-crushing set back, I don't feel comfortable with doing that. So at a very minimum it will have to wait to see how scathed AstroBin comes out of this.

Regarding the bus factor, yeah, that's also true. Should I go under a bus, AstroBin would be very short lived. We also need a better solution for that than me giving my password manager's master password to my wife and telling her to sell it after I'm gone smile

The best I could do for that, I already did: I made AstroBin open source. Should I die, somebody else can pick it up and the community would reassemble.
leviathan
18 Feb, 2020 15:51
Salvatore, thanks for honest and detailed statement !
mxcoppell
18 Feb, 2020 15:58
Thanks Salvatore for the detailed reported. Well written.
Health and Family are always at top priority. Wish the best to you, your family and Astrobin.

Just one suggestion, could you provide a archive download method for the data stored in RDS? Such as descriptions, gears and acquisition details so we could keep a local copy from time to time. Just in case.
Edited 18 Feb, 2020 16:02
2ghouls
18 Feb, 2020 16:02
Salvatore Iovene
I was going to announce something new on AstroBin, and a new subscription tier above Premium. Now tho, with this dream-crushing set back, I don't feel comfortable with doing that. So at a very minimum it will have to wait to see how scathed AstroBin comes out of this.

I don't want to pressure you in to doing something you don't feel comfortable doing, but I would be very interested in whatever you were working on, and would gladly pay more for it!  smile

I can relate to the feeling you are having now, and I feel sorry for you and those who irrevocably lost data. That said, I think AB will be stronger now that you are taking more steps to backup the images. I'm looking forward to the future of the site!
GraemeH
18 Feb, 2020 16:26
Hi Salvatore,

I've just finished restoring my gallery. I only had 32 images so this wasn't a huge task. I've only bothered to restore the versions that had been marked as final, and probably won't restore previous versions.

My real reason for posting is to congratulate you on your response to a bad situation. You have provided a full and honest explanation of what went wrong, and you have gone to great lengths to help affected users to the best of your ability. I understand why you feel responsible, but I really want to stress the point that whilst the data loss is a bit inconvenient, no one has suffered seriously.

I will continue to support Astrobin, and feel no reason to ask for any kind of compensation for an error that could have been made by any one of us. Anyone who claims to be infallible is completely deluded.

Please go easy on yourself, and don't let criticism from a small minority of users drown out the support from the rest of us.

Best wishes,
Graeme Holyoake
Edited 18 Feb, 2020 17:21
whwang
18 Feb, 2020 16:32
Salvatore,

Thank you for everything you have done for this site.  This is an unfortunate event, and I have to re-upload hundreds of images including their multiple versions.  However, this gives me an opportunity to look at my old images and give them a new processing.  At the end, I believe Astrobin will be better, so is my gallery.
milanesko
18 Feb, 2020 16:34
Hi Salvatore
all my photos will be lost (only 12pcs).
probably not good enough to stay smile
I will have to do more to make new photos more beautiful and better.
life goes on
clear sky
Milan
Edited 18 Feb, 2020 16:39
dkamen
18 Feb, 2020 16:53
Hi Salvatore,

We all make mistakes. This community is one of the most brilliant places on the Internet. I intentionally chose the paid tier even though I do not publish so many images because I am well aware of the costs and wanted to support Astrobin. I will renew in a few months.  I don't want a refund.

I am quite familiar with the various dimensions of S3 (pricing, reliability, performance) because of my job which includes managing a rather large infra in AWS. It is actually very feasible to structure your storage so that less important stuff doesn't cost so much and more important stuff is guaranteed to survive even deliberate deletions, but there are quirks and sometimes S3 and AWS in general feels like a complex video game where you navigate a multiverse with its own constantly changing rules and the slightest change of course has dramatic consequences in the long or even in the short run. Once -when I was much less experienced- I added a bucket policy like you did, only mine would transition the objects to less expensive "infrequently accessed" storage instead of deleting them. The objective was to save about two hundred dollars per month. I hadn't paid attention the fine print saying that transitions are billed too. So that single, non-undoable action of mine added a nice couple thousand dollars to that month's bill… I learned my lesson of course. And the lesson includes not pointing fingers when somebody else makes an honest mistake.

Do let me know if you need friendly advice about S3.

Cheers,
Dimitris
MBustamante
18 Feb, 2020 17:24
Thank you very much for your honesty, Salvatore.
Long live Astrobin!
dhavalbbhatt
18 Feb, 2020 17:30
Lest we forget, and to keep things in perspective, this is a hobby at the end of the day for many of us. Most of us are not making money (or isn't a significant source of income at least) from these images. In that sense, what may be lost is, but a sliver of what we might truly care about in the grand scheme of things. At least, that is the way that I am looking at it. So, no big deal for me. I know there will be folks who will feel differently.
Soundologist
18 Feb, 2020 17:42
Salvatore,

This community is my home. The people here just get me & we can talk about the things that excite me like the people in the real world simply cannot. I am forever grateful for your dedication to this community~ in the best & worst of times.

My renewal date is not for another 9 months or so, but when the day comes, I'll keep my payment year after year after year. I hope you can make AstroBin full-time one day.

Cheers, thank you, and all the best,
Rob
mantrain
18 Feb, 2020 17:50
Thank you so much for your dedication to this great site you created. I am so happy it turned out not to be such a bad issue for the individual posters.
Perhaps you want to recruit some help from some of the talent in the membership here.
 
Register or login to create to post a reply.