What's up with Tatoeba now?

It's been a month and a half since our SSD incident and while we managed to bring Tatoeba back online, there are still many features not working and the website is overall very slow to use.

I know many people wished the situation could improve faster, but as it stands now, we don't really have the manpower to get things done more quickly. Fixing everything and getting back to a stable situation will take a long time. Perhaps another couple of months. Perhaps more.

To be honest, the main reason is because I (as the founder of Tatoeba) am in a phase where I wouldn't want to dedicate more than a few hours per week.
There has been times where I could spend as much as a part time job working on Tatoeba (maybe even as much as a full time job?), and things were evolving at a fast pace. Then there has been times where I was completely absent and the project could not really move forward.
Right now, I'm on the low side, which is a big part of why Tatoeba is much slower to recover.

There's been a few questions asked on the Wall, regarding what can be done to improve the situation, and what can be done to keep Tatoeba healthy in the future. I'd like to answer them here to give people a clearer idea about how Tatoeba is functioning, and I'd like as well to give a few updates about what is being worked on at the moment.


1) https://tatoeba.org/eng/wall/show_message/28201#message_28201

Also, do you have any idea how much it would cost if tatoeba moved to a better web-hosting with better support? 2 weeks to restore functionality - that's a bit too much, and that's not the first time something like this happened to the site. 

I can contribute 10 euros every year, which is probably a drop in the ocean, but we could probably find 100 people like me among our active users.

It wouldn't cost a whole lot more to move to a web host with better support.

To be fair, the long time it took to restore Tatoeba was not entirely due to our host. They are not in charge of maintaining Tatoeba as a whole, they are only responsible of taking care of the machine where Tatoeba is hosted. It stopped working because the SSD died, and they were definitely quite slow to react (took 5 days of waiting before they replaced it), but it's not their fault that it took an extra week to restore the system and the data on the new SSD.

Still, we're definitely planning to move to another host. We have ordered a new server and hopefully will manage to move Tatoeba to a new home by the end of month.

Money is not an issue — at least not for paying web hosting. We currently spend around 35€/month for our server, and even if we'd have to spend twice as much, we could still afford it without extra donations.


2) https://tatoeba.org/eng/wall/show_message/28243#message_28243

Is there any way to avoid a potential following breakdown or for that to save all the data?

Yes, there are ways.

Tatoeba is currently hosted on a dedicated server and if we move to a VPS (which is the plan), we would no longer have to worry about hardware failure.

Besides of that, our data recovery could have gone a lot better if we had invested more efforts on backups before the incident. We would have lost only one day of data had we checked our backups properly.

But keep in mind that securing the data is only a small part of the problems we have to solve. Tatoeba has grown into a complex system, which is becoming more and more challenging to maintain, the more features and content we put into it.

Which leads to the last question.


3) https://tatoeba.org/eng/wall/show_message/28262#message_28262

How is Tatoeba funded? It looks like the recent problems, and the current half-working status are the direct result of not having permanent staff (a.k.a money).

Tatoeba is funded with donations only. We have more than enough to pay for the server, we however don't have even closely enough to hire permanent staff. And indeed, not having permanent staff is a handicap.

With the way we are operating, when something's not working, it takes time before it gets noticed by someone who can do something about it, because we don't monitor Tatoeba 24/7. Then it takes time to solve the issue because everyone is a volunteer, and Tatoeba is just a side project for all of us. Problems can occur while we're at work, while we're traveling, while we're sick or just too tired to work on it. Some problems are actually quite difficult to solve.

We would need ideally a small team of 2 to 4 people working on Tatoeba at least as a part-time job, to ensure that Tatoeba keeps running smoothly at all times and continues to evolve in a sustainable way.

I don't think we can raise enough money for this via donations or crowdfunding. To be fair, I have never tried, so I could be wrong. But we're talking about 50k-100k euros per year, to secure a team. It's a completely different scale from what we're currently dealing with. Honestly, I don't have the "marketing" skills, nor would I have the energy, to raise this kind of money. If not me though, I'm not sure who else would do this.

But even if someone walked to us and threw millions at us, our issues won't magically disappear. It would still take time to build a team, to find the right people with the right skills to solve those issues.

While I would like Tatoeba to be as much as possible independent from money, I do think that one day or another, Tatoeba will need permanent staff, which is quite difficult to achieve without money. We can keep the project alive with volunteer staff, but we cannot make it grow much bigger than it is now. 

Anyway, this is more of a long term discussion.


In the shorter term, what's happening?

Currently, we're lucky to have pep (aka. Ppjet6 on Tatoeba) who stepped up to help on the whole sysadmin/devops part. He'll be the main person working on migrating Tatoeba to the new server.
If you have any knowledge in these areas and wish to get involved in a way or another, you're more than welcome to join the Tatoeba IRC channel (#tatoeba) on freenode.

In fact, you're welcome to join even if you're just gonna be lurking, or if you'd like to give some moral support. You can potentially learn a lot about how Tatoeba works by just hanging around in the IRC channel. And even if you won't get involved for time being, who knows, maybe in 6 months, in a year or in two years, Tatoeba will need you to save it from a dire death.

Also, you should be aware that there are other places than the Wall where you can gather and interact with other Tatoeba members. When Tatoeba is down, the Wall is down too, so it's important to have other channels of communications. Those are:

• IRC (as mentioned above): #tatoeba on freenode
• Gitter: https://gitter.im/Tatoeba/tatoeba2
• Google Group: http://groups.google.com/group/tatoebaproject

Feel free to use those external communication channels to discuss about Tatoeba. They are not restricted to developers only.

Last but not least, we now have a status page to communicate information about the status of Tatoeba (whether it's online, or down, or experiencing issues, or undergoing maintenance, etc): https://status.tatoeba.org

For now it's very minimal, but the idea is that if you're trying to access the Tatoeba website but can't, or if something doesn't work anymore on the website, you'll be able to find information about it on the status page.