Cool!
Cans has source? I’ve been hacking on a per-instance community scraper (that pulls a list of communities from the community directory page) for !lemmy411!lemmy411@lemmy.ca, in an attempt to have a regularly published community list, but I’m a bit bogged.
I’m probably going to put the source up shortly but how I decided to find communities is by scraping my instances “federated with” list from https://toast.ooo/api/v3/site
Then going through the /.well-known/nodeinfo -> /nodeinfo/v2.0.json (or whatever url the well-known gives) to check the software to make sure it’s a Lemmy instance
Then using the Lemmy rest API to paginate through the open communities for that instance
I have links to the source for my crawler (written in nodejs with redis task queuing) on https://lemmyverse.net (GitHub link top right) 🤗 It’s a bit messy at the moment but has all the community crawling stuff.
Wow. Stickied THAT to !lemmy411@lemmy.ca thanks!
How are you adding instances? I have a few communities I’d love to have listed.
In my other comment I described how I find communities but with this alpha I have a set list of instances to use
Are your communities listed on your current instance (lemmy.ko4abp.com)? I can add that to the temporary list if you’d like
I’m operating a small instance, nothing here seems to be listed. They always appeared on browse.feddit.de after some time, so I suppose it searches for subs actively. Think you can do something like that in the future?
Aha, I am doing the exact same thing 😂 https://lemmyverse.net
It looks really good! When I saw it on my feed my first response was “oh shoot someone beat me” 😂
Hehe they do look quite similar so great minds right 😁 I had the same thourght when I saw yours! I’ll add a link to yours to the list on my about page if you want 😀
Potentially dumb question, but I’ve seen lots of people linking to this website:
What’s new/different about https://browse.toast.ooo or https://lemmyverse.net?
Is it because they also show instances, not just communities? Or is it more just a hobby project (nothing wrong with that, I’m 💯 on board with building things for the sheer enjoyment of it - even if it’s already a ‘solved’ problem)
I can only speak for myself, but I started work on mine since the Feddit one is fixed wide mode and the searching/filtering is less than ideal, I wanted to be able to see more details. Feddit one doesn’t show instance stats iirc. There are also other lists, but I didn’t find any of them super user-friendly… If Lemmy is gonna succeed Reddit there needs to be some super friendly tools to assist community discovery.
Nice! I sort of hope these community discovery tools & improvements eventually make their way back ‘upstream’ into Lemmy’s UI over time. Imagine not needing a separate discovery app because Lemmy’s built-in one is already the best one available 😄
For me these two community browsers linked here are a little more usable then the browse.feddit.de version (I am on desktop). Everyone is just trying to help grow the community in whatever way they can, and redundancies and choices never hurt. There is talks of each instance hosting a community browser over on the github issues, so this likely will be implemented in some form for each server as well.
For both of you, wouldn’t it be nice to let me add my home instance so you could provide quick links to subscribe (and open the community in my home instance).
Saw yours last night and prefer it. But wow, the communities have gone from 6k to over 10k overnight by your stats. Is that true?
Yeah they really shot up! I’m pretty sure it’s an accurate number. I did an crawl last night and picked up another ~3000 communities. A lot might be from instances that are under high load/timing out so might not have been picked up previously. Imma do some more analysis with my data in the future.
Probavly the best one I have found so far.
By the way, do you have any plan to expose an API or daily extract of the data you have?
While those websites are useful for manual searches, I think it would benefit the feddiverse much more if there was a way to integrate all those lists into an app. At least without resorting to web scrapping.
Update: https://data.lemmyverse.net/
looks great, but it’s not finding anything for Deus Ex
https://lemmyverse.net/communities?query=deus+ex
but there’s 3 here
https://programming.dev/search?q=deus ex&type=Communities&listingType=All&page=1&sort=TopAll
I think I figured it out actually, the one from lemmit.online was marked as suspicious and you have an option to show/hide those
the other one is from my little instance that no one knows about :(
Thanks for sharing !
Is it better than jerboa android app?