Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
mastodon.world is one of the many independent Mastodon servers you can use to participate in the fediverse.
Generic Mastodon server for anyone to use.

Administered by:

Server stats:

9.5K
active users

mastodon.world: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.7

#distributed

16 posts14 participants4 posts today
Public
Compsci Weekly

What If We Could Rebuild Kafka From Scratch?

morling.dev/blog/what-if-we-co

Discussions: discu.eu/q/https://www.morling

www.morling.dev · What If We Could Rebuild Kafka From Scratch? Update April 25: This post is being discussed on Hacker News, lobste.rs, and /r/apachekafka The last few days I spent some time digging into the recently announced KIP-1150 ("Diskless Kafka"), as well AutoMQ’s Kafka fork, tightly integrating Apache Kafka and object storage, such as S3. Following the example set by WarpStream, these projects aim to substantially improve the experience of using Kafka in cloud environments, providing better elasticity, drastically reducing cost, and paving the way towards native lakehouse integration. This got me thinking, if we were to start all over and develop a durable cloud-native event log from scratch—​Kafka.next if you will—​which traits and characteristics would be desirable for this to have? Separating storage and compute and object store support would be table stakes, but what else should be there? Having used Kafka for many years for building event-driven applications as well as for running realtime ETL and change data capture pipelines, here’s my personal wishlist:
#compsci#distributed
Public
Compsci Weekly

How decentralized is the fediverse, really?

discuss.systems/@ricci/1143963

Discussions: discu.eu/q/https://discuss.sys

discuss.systemsRob Ricci (@ricci@discuss.systems)With bluesky (mostly) going down for a few hours today, I got to wondering about how decentralized the fediverse really is in terms of where its servers are hosted. I grabbed a server list from fedidb, with network information coming from ipinfo.io . [EDIT: I did a better analysis on a dataset of 10x as many servers, see https://discuss.systems/@ricci/114400324446169152 ] These stats are by the number of *servers* not the number of *users* (maybe I'll run those stats later). fedidb currently tracks 2,650 servers of various types (Mastodon, pixelfed, lemmy, misskey, peertube, etc) The fediverse is most vulnerable to disruptions at CloudFlare: 24% of Fediverse servers are behind it. Also note that this means that I don't have real data on where this 24% are located or hosted, since CloudFlare obscures this by design. Beyond CloudFlare, the fediverse is not too concentrated on any one network. The most popular host, Hertzner, only hosts 14% of fediverse servers, and it falls off fast from there. Here are the top networks where fediverse servers are hosted: 504 Cloudflare, Inc. 356 Hetzner Online GmbH 130 DigitalOcean, LLC 114 OVH SAS 56 netcup GmbH 55 Amazon.com, Inc. 55 Akamai Connected Cloud 36 Contabo GmbH 33 SAKURA Internet Inc. 32 The Constant Company, LLC 31 Xserver Inc. 28 SCALEWAY S.A.S. 24 Google LLC 23 Oracle Corporation 16 GMO Internet Group, Inc. 14 IONOS SE 14 FranTech Solutions 11 Hostinger International Limited 10 Nubes, LLC Half of fediverse servers are on networks that host 50 or fewer servers - that's pretty good for resiliency. There is even more diversity when it comes to BGP prefixes, which is good for resiliency: for example, the cloud providers that have multiple availability zones will generally have them on different prefixes, so this gets closer to giving us a picture of the specific bits of infrastructure the fediverse relies on. The top BGP prefixes: 55 104.21.48.0/20 50 104.21.16.0/20 48 104.21.64.0/20 41 104.21.32.0/20 41 104.21.0.0/20 38 104.21.80.0/20 32 172.67.128.0/20 31 172.67.144.0/20 28 172.67.208.0/20 28 162.43.0.0/17 27 104.26.0.0/20 26 172.67.192.0/20 26 172.67.176.0/20 23 172.67.160.0/20 19 116.203.0.0/16 17 172.67.64.0/20 17 159.69.0.0/16 16 65.109.0.0/16 14 88.99.0.0/16 14 49.13.0.0/16 13 78.46.0.0/15 13 167.235.0.0/16 13 138.201.0.0/16 11 95.217.0.0/16 11 95.216.0.0/16 11 49.12.0.0/16 11 135.181.0.0/16 10 37.27.0.0/16 10 157.90.0.0/16 75% of fediverse servers are behind BGP prefixes that host 10 or fewer servers, meaning that the fediverse is *very* resilient to large network outages. Top countries where fediverse servers are hosted: 871 United States 439 Germany 156 France 148 Japan 75 Finland 57 Canada 49 Netherlands 38 United Kingdom 26 Switzerland 26 South Korea 21 Spain 19 Sweden 18 Austria 17 Australia 15 Russia 12 Czech Republic 10 Singapore 10 Italy And finally, a map of the locations of fediverse servers: https://ipinfo.io/tools/map/91960023-e8c6-4bee-9b07-721f2c8febab
#compsci#distributed
Public
fediverse
How decentralized is the fediverse, really? Comments

https://discuss.systems/@ricci/114396317436420669

#distributed

Result Details
discuss.systemsRob Ricci (@ricci@discuss.systems)With bluesky (mostly) going down for a few hours today, I got to wondering about how decentralized the fediverse really is in terms of where its servers are hosted. I grabbed a server list from fedidb, with network information coming from ipinfo.io . [EDIT: I did a better analysis on a dataset of 10x as many servers, see https://discuss.systems/@ricci/114400324446169152 ] These stats are by the number of *servers* not the number of *users* (maybe I'll run those stats later). fedidb currently tracks 2,650 servers of various types (Mastodon, pixelfed, lemmy, misskey, peertube, etc) The fediverse is most vulnerable to disruptions at CloudFlare: 24% of Fediverse servers are behind it. Also note that this means that I don't have real data on where this 24% are located or hosted, since CloudFlare obscures this by design. Beyond CloudFlare, the fediverse is not too concentrated on any one network. The most popular host, Hertzner, only hosts 14% of fediverse servers, and it falls off fast from there. Here are the top networks where fediverse servers are hosted: 504 Cloudflare, Inc. 356 Hetzner Online GmbH 130 DigitalOcean, LLC 114 OVH SAS 56 netcup GmbH 55 Amazon.com, Inc. 55 Akamai Connected Cloud 36 Contabo GmbH 33 SAKURA Internet Inc. 32 The Constant Company, LLC 31 Xserver Inc. 28 SCALEWAY S.A.S. 24 Google LLC 23 Oracle Corporation 16 GMO Internet Group, Inc. 14 IONOS SE 14 FranTech Solutions 11 Hostinger International Limited 10 Nubes, LLC Half of fediverse servers are on networks that host 50 or fewer servers - that's pretty good for resiliency. There is even more diversity when it comes to BGP prefixes, which is good for resiliency: for example, the cloud providers that have multiple availability zones will generally have them on different prefixes, so this gets closer to giving us a picture of the specific bits of infrastructure the fediverse relies on. The top BGP prefixes: 55 104.21.48.0/20 50 104.21.16.0/20 48 104.21.64.0/20 41 104.21.32.0/20 41 104.21.0.0/20 38 104.21.80.0/20 32 172.67.128.0/20 31 172.67.144.0/20 28 172.67.208.0/20 28 162.43.0.0/17 27 104.26.0.0/20 26 172.67.192.0/20 26 172.67.176.0/20 23 172.67.160.0/20 19 116.203.0.0/16 17 172.67.64.0/20 17 159.69.0.0/16 16 65.109.0.0/16 14 88.99.0.0/16 14 49.13.0.0/16 13 78.46.0.0/15 13 167.235.0.0/16 13 138.201.0.0/16 11 95.217.0.0/16 11 95.216.0.0/16 11 49.12.0.0/16 11 135.181.0.0/16 10 37.27.0.0/16 10 157.90.0.0/16 75% of fediverse servers are behind BGP prefixes that host 10 or fewer servers, meaning that the fediverse is *very* resilient to large network outages. Top countries where fediverse servers are hosted: 871 United States 439 Germany 156 France 148 Japan 75 Finland 57 Canada 49 Netherlands 38 United Kingdom 26 Switzerland 26 South Korea 21 Spain 19 Sweden 18 Austria 17 Australia 15 Russia 12 Czech Republic 10 Singapore 10 Italy And finally, a map of the locations of fediverse servers: https://ipinfo.io/tools/map/91960023-e8c6-4bee-9b07-721f2c8febab
ExploreLive feeds
About