Hey, folks!

Today, we can celebrate the first anniversary of the creation of lemm.ee! I thought it would be cool to write down how lemm.ee was born, as well as collect some stats about our first year. Here goes!

A quick recap of the beginning of this instance

As probably many others here, I discovered Lemmy early last summer. I had been aware of the Fediverse previously, and always thought it was an amazing concept, but I had never been super interested in Twitter-style social networks. When I found out that Lemmy combined all the great parts of federation with the best parts of link aggregation, I knew that I had to join immediately.

As I was trying to find an instance to make my account on, I realized that most instances were struggling to keep up with a massive influx of new users. At the time, there was a big explosion in Lemmy user numbers, and the network wasn’t fully ready for it. I have some experience with building software for scale, so it felt natural to set up a new instance and try to help with spreading out the load. I got to work in the evening of the 8th of June, 2023, and I was actually so excited about everything, that I completely skipped sleep that night. By the morning of the 9th of June, lemm.ee was online.

From the very beginning, I always intended for lemm.ee to be a welcoming, reliable, and stable gateway into the Lemmy network. I wrote a welcome post on lemm.ee, which most of you have probably seen, as well as a comment on lemmy.ml inviting new users to lemm.ee (lemmy.ml, as many instances, was extremely overloaded at the time).

We started growing extremely quickly. Thousands of users joined lemm.ee over the first few months. Even during the biggest waves of new users, we never closed our sign-ups. The first month or two were definitely very stressful in terms of just trying to deal with the load, but overall, I think I managed to deal with it well enough, and lemm.ee has been running more or less smoothly (with a few exceptions) ever since.

Some stats about the first year

I promised to collect some statistics about lemm.ee so far. This is what I’ve come up with:

Usage

Overall, lemm.ee has 28,715 registered users. Of course, it’s easy to create an account, and most of these are probably inactive at this point, but it’s still a ridiculous amount.

Of all the registered users, 7903 have made at least one post or comment.

7373 users have never made any posts or comments, but have still been voting. This means that out of users who actually interact on lemm.ee, more than half generate content (through their comments and posts) - this is way more than I expected!

Meanwhile, we also have 13,439 users who have never made a single comment, post or vote. I guess most of these are people who just signed up and never got into Lemmy, but I’m sure there are quite a few hardcore lurkers among this group as well.

As for communities, our users have created 1430 of them. Most of these have not (yet) taken off, as only 491 of these communities have at least one comment in them. In general I am happy to see some great communities appearing on lemm.ee - my hope is that we can spread awesome communities out quite evenly on the network, so that in the end, no instance becomes a single point of failure for Lemmy.

Judging by posts and comments made by lemm.ee users, I feel like we’re definitely on the right track: our users have made 20,898 posts in local communities, and 30,847 posts in communities hosted on other instances. The situation is even better for comments, where lemm.ee users have written 42,785 comments in local communities, and a whopping 569,730 comments on remote communities! This means that lemm.ee is not just its own little closed pocket in the Fediverse, but indeed a proper gateway to the Lemmy network, which is exactly what I always hoped it would be.

Note about comment and post counts: I realize the numbers above don’t match the stats about posts and comments on our front page, I’m guessing something is out of sync there, but the stats I am sharing here are based on actual fresh data, counted directly in our database today.

Lemmy (and lemm.ee) would be quite useless without its users, so a big thanks to all of you for using lemm.ee!

Administration

We have a really awesome volunteer admin team, with admins putting in countless hours of their free time to help weed out bad actors. A lot of the work our admins do is completely invisible to most users. I think the admin team does not really get enough recognition, and in fact in many cases, they actually get some undeserved abuse thrown at them.

I am personally very grateful for everybody who has stepped up to be a part of the team, and I think all lemm.ee users benefit from their work every day. In the past year, our admins have handled 12,329 reports from users. While most reports aren’t too bad, and don’t require harsh action, there is still a significant amount of these reports which contain the absolute worst content which you can find on Lemmy - hate speech, bigotry, gore, even illegal content. Our admins are constantly going through every single report they receive, to ensure that mods are getting admin-level support where needed, and to ensure that malicious users in general can’t use lemm.ee to spread garbage into the Lemmy network.

In terms of admin actions, I think the most interesting statistic might be amount of users banned by lemm.ee admins, grouped by their home instance. I will list the top 10 instances here:

• kbin.social: 581
• lemm.ee: 355
• lemmy.world: 31
• sh.itjust.works: 29
• m.mxin.moe: 28
• discuss.tchncs.de: 26
• kbin.chat: 22
• mastodon.social: 19
• lemmy.ca: 18
• fedia.io: 16

As you can see, with the exception of kbin.social, the vast majority of our instance bans are for our own users. Most of the big instances are actually very good at banning their own abusive users, and once they are banned on their own instance, our admins don’t really need to worry about them, as they have no way to log in at that point. kbin.social is a bit of a special case - they either don’t give out a lot of bans, or those bans just don’t federate to Lemmy properly, and for some reason, a lot of advertisers sign up on that instance all the time.

Financials

I have received some questions every now and then about how much it costs to run lemm.ee. While you can always get a sense for the predicted monthly costs for the current month on https://status.lemm.ee, I thought I might include a full breakdown of our costs for the first year here.

Here are all of our costs for the past year, grouped by service:

• Postmark: 177.06€
• Cloudflare: 222.28€
• DigitalOcean: 1744.27€
• Hetzner: 510.20€ (lemm.ee migrated from DigitalOcean to Hetzner several months ago)
• Backblaze: 3.78€ (we’ve been using Backblaze B2 for a few months now, it’s incredibly cheap)
• Domain registration: 100.70€ (paid for the next 10 years!)

We are currently completely funded by lemm.ee users!

There is a small minority of users who are shouldering the entire cost of lemm.ee for all of us. I am extremely grateful that others find Lemmy useful enough that they have put their own money into ensuring financial stability for lemm.ee.

We currently have 49 active sponsors on GitHub, and 7 active supporters on Ko-Fi. In addition, there have been 62 more sponsors on GitHub over the past year, as well as 49 additional supporters on Ko-Fi. This means that in total, 167 users have supported lemm.ee financially. This has completely exceeded all my expectations, I really think it’s incredible. A huge thanks on behalf of myself (and I think I can speak for all other lemm.ee users here as well) to all the supporters!

Conclusion

Running lemm.ee has certainly been a rollercoaster in many ways. There are a lot more things which happened during the first year that I could write about here. On the other hand, this post is already quite long, and a lot of the things which happened are probably best forgotten about anyway, so I think I should wrap up here 😅.

At its core, Lemmy is really an amazing piece of software. It’s helping real humans connect on the internet, without any corporate bullshit. I am very happy to be here with all of you, thank you for joining lemm.ee for its first year, and I hope you’ll join me here again when I write this post in another year from now!

Hey there! I recently got server banned b/c I was sharing a link in men's communities and I was mistaken for a bot.

I didn't have a way to respond to the ban. I was just logged out. I didn't see a link or even which mod did the ban so that I could reply to them, but then I couldn't get into my account to reply anyway.

It'd be nice to maybe get a "if you think this was in error" email or something.

Although I know that if lemmy was the size of reddit, people would hook that email up to chatgpt, so :/

Anyway, it'd be a nice feature to have. I doubt I'll need it again, but I was quite confused as to what to do.

Hey all!

Upcoming lemm.ee cakeday

Can you believe that lemm.ee is almost 1 year old? In just a couple of weeks (specifically, on the 9th of June), we will be able to celebrate our first instance cakeday.

I am thinking of compiling some stats about how lemm.ee has been used in its first year, if you have any specific stats in particular you would like to see, feel free to comment below. I will try to accommodate any ideas as I start gathering this info!

Infrastructure updates

A few weeks ago, I posted about plans to make some changes to our infrastructure in order to deal with different intermittent networking issues.. It took a bit longer than I hoped (just did not manage to get enough free time between then and now), but I am happy to report that this work has now been completed! Additionally, I have decommissioned our stand-alone pict-rs server.

With the two changes mentioned above, I believe lemm.ee should now be much more resilient going forwad, and I expect a significantly lower rate of infrastructure-related issues for the rest of the year!

I'll leave a tehcnical overview about the problem & solution below for those interested, but if these details don't interest you, then you can safely skip the rest of this post.

---

For context, lemm.ee has been hosted on Hetzner servers for most of this year (having migrated from DigitalOcean initially), with everything except our database being hosted on the Hetzner Cloud side, and the database itself living on a powerful dedicated Hetzner server. This mix allows a great amount of flexibility for redeploying and horizontally scaling our application servers, while still allowing a really cost-effective way of hosting a quite resource-hungry database.

In order to facilitate networking between the cloud servers and the dedicated database server (which live in different networks), Hetzner provides a service named "vSwitch". This service basically allows you to connect different servers together in a private network. Unfortunately, I discovered quite quickly that this service is very unreliable. During the short few months that we have been using the vSwitch, we have gone through one extended period of downtime (where the service was just completely broken for several hours), as well as dozens (if not hundreds at this point) intermittent disconnects, where servers randomly lose their connections over the vSwitch. After such a disconnect, the connection never recovers without manual intervetion.

For most lemm.ee users, the majority of these vSwitch issues have been mostly invisible, as we have redundancy in our servers - if one server loses its connection to the database, other servers will take over the load. Additionally, I have generally been able to respond quite quickly to issues by redeploying the broken servers (or deploying other temporary workarounds). However, in addition to a huge amount of these issues which lemm.ee users hopefully haven't ever noticed, there have also been a few short periods of downtime this year so far, as well as a few cases of federation delays. These more extreme cases were generally caused by multiple servers losing their vSwitch connections at the same time.

After several attempts to work around these issues, I decided that we need to migrate away from vSwitch.

As of earlier today, lemm.ee is no longer using Hetzner's vSwitch at all!

I finally found enough time earlier today to focus on this migration, and I was able to successfully complete it. None of our networking is relying on the vSwitch anymore.

In the end, I went with quite a simple solution - I configured a host-level firewall (nftables) on our database dedicated server, which will deny all connections by default. Whenever any cloud servers are added/removed, their corresponding public IP addresses are added/removed in the allowlist of our database firewall. It would have been ideal to do this whole logic in Hetzner's own firewall, but that one unfortunately has a limit of only 10 rules per server, which is just not enough for our setup.

Bonus: our pict-rs server has been decommissioned!

Pict-rs is the software which Lemmy uses for everything related to media (image storage mostly). Initially, pict-rs required a local filesystem to store both files as well as metadata about files. Since the beginning, lemm.ee has used a dedicated server just for pict-rs, in order to ensure we could easily redeploy the rest of our servers without losing any images.

Over the past year, pict-rs has gained the ability to store files in object storage, and metadata in a PostgreSQL database. This meant that the server running pict-rs itself no longer contained any of the important data, so it became possible to redeploy without losing any images. Additionally, this meant that it would be possible to run multiple pict-rs servers in parallel.

While we had already migrated our pict-rs server to use object storage and PostgreSQL several months ago, we still had the single dedicated pict-rs server up until today. I have been planning for a while to decommission this server, and start running pict-rs directly on each one of our Lemmy application servers. Earlier today, I was able to complete this plan. This should hopefully mean that our pict-rs server is less likely to get overloaded, and it also means a tiny reduction in our overall monthly infrastructure bill (due to one less server running).

With the above changes, I think our infrastructure has become more robust, and hopefully, we will experience less issues with images, federation, and general downtime going forward.

---

That's all from me for now. Feel free to leave any thoughts or questions in the comments, and as always, I hope you're having a great day!

Hey folks!

This is a quick notice about a change to our moderation policy.

We have had a policy on lemm.ee for administration and federation nearly since the very beginning. This policy has also always included a section about moderator responsibilities. Today, we have made two changes to this policy:

1. The policy has been renamed to Policy for administration, moderation, federation - this is to make it clear that the policy is also relevant for mods
2. We have introduced a new responsibility for moderators, they must "Ensure that they only provide accurate and clear reasons for mod actions".

The reason for the addition is that mod log actions federate out to other instances, and are more or less permanent (due to how Lemmy and federation works right now). This means that users do not really currently have any easy way to clarify or defend themselves against inaccurate accusations in the mod log.

As always, I am very grateful to all mods for your efforts in building awesome communities on lemm.ee. I hope you can understand why this new policy is necessary - I do not want to make your lives more difficult, the goal is to just try and reduce any mod log related misunderstandings in the future.

Thank you for reading and have a nice day!

Hey folks!

We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!

---

Technical details

Our servers are communicating with our database over Hetzner's "vSwitch" service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won't affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.

Update: the migration is now complete! You can read more here.

EDIT: I should have posted this in the Support community - as others are also doing.

https://lemm.ee/c/support

---

Just noticed tonight that when I visit my FullMoviesOnYouTube community, the banner image is not shown.

I tried linking directly to it:

https://lemm.ee/pictrs/image/642b333b-5c37-4d39-af9f-cc876de484fc.webp?format=webp

And got this error:

{"error":"unknown","message":"Request error: error sending request for url (http://10.0.0.3:8080/image/process.webp?src=642b333b-5c37-4d39-af9f-cc876de484fc.webp): operation timed out"}

Any idea what's up? It's definitely been working as of just last week.

Hello!

I noticed that a post I created from my sopuli account to a lemm.ee community, federated over here, but then not out any other instances.

Is lemm.ee dealing with similar federation woes as lemmy.world?

I was using imgur now to upload pictures, but for none of my posts it shows a thumbnail for me in jerboa. It does at least for some of those using lemmy.world, so i assume this may be something lemm.ee specific? Thumbnails seem to be shown for images hosted on catbox.moe or other instances. I just don't get what the difference is.

What is it that would make thumbnails show up?