Mastodon

0 posts0 participants0 posts today

Doug Whitfield [Minneapolis]so, gonna write some stuff on <a href="https://mastodon.social/tags/HDFS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDFS</a> <a href="https://mastodon.social/tags/MapReduce" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MapReduce</a> <a href="https://mastodon.social/tags/yarn" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#yarn</a> and maybe clustering. Also, <a href="https://mastodon.social/tags/machinelearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machinelearning</a> was suggested but I think that may be too broad of a topic for this. I did cover Machine Learning in a blog back in 2023, but this time is for KB, not blog: <a href="https://www.openlogic.com/blog/using-cassandra-kafka-and-spark-ai" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.openlogic.com/blog/using-cassandra-kafka-and-spark-ai</a>Hmm, perhaps some sort of ML performance (as in disk io, etc not accuracy) document would be good but still, where to even start.If anyone has beginner resources, I'll likely be pointing folks to some resources

HabrМой опыт эксплуатации кластера TrinoTrino — высокопроизводительный распределённый SQL-движок, с возможностью объединения данных из разнородных источников, таких как: реляционные БД, файловые хранилища, шины данных, inmemory-хранилища, облачные сервисы и тд. Архитектура ориентирована на выполнение аналитических запросов с минимальной задержкой. Т.е. с его помощью можно отправлять SQL-запросы в MongoDB и Kafka, например. Благодаря скорости, развитию, и удобству захватывает популярность у инженеров и аналитиков, работающих с bigdata. Я познакомился с Trino 1 год назад, за это время настроил с нуля кластер на baremetal и помог с проблемами в нескольких других. В этой статье делюсь краткой выжимкой опыта эксплуатации, накопленным за это время. Большая часть информации будет актуальна и для российского форка Trino: CedrusData .<a href="https://habr.com/ru/articles/863854/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://habr.com/ru/articles/863854/</a><a href="https://zhub.link/tags/trino" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#trino</a> <a href="https://zhub.link/tags/trinosql" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#trinosql</a> <a href="https://zhub.link/tags/hdfs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hdfs</a> <a href="https://zhub.link/tags/iceberg" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#iceberg</a> <a href="https://zhub.link/tags/cedrusdata" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#cedrusdata</a> <a href="https://zhub.link/tags/presto" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#presto</a> <a href="https://zhub.link/tags/prestodb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#prestodb</a>

Python for Data ScienceThe page for supporting (remote) file systems in Python has become much more comprehensive: <a href="https://www.python4data.science/en/latest/data-processing/file-systems.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.python4data.science/en/latest/data-processing/file-systems.html</a> And you are welcome to let us know if we have forgotten anything. <a href="https://mastodon.social/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://mastodon.social/tags/HDFS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDFS</a> <a href="https://mastodon.social/tags/S3" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#S3</a> <a href="https://mastodon.social/tags/GoogleDrive" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GoogleDrive</a> <a href="https://mastodon.social/tags/HuggingFace" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HuggingFace</a> <a href="https://mastodon.social/tags/Git" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Git</a>

LINUXexpert.orgThe Hadoop ecosystem comprises various tools and frameworks designed to handle large-scale data processing and analytics. Let's discuss the core components, namely Hadoop, HBase, and Hive, along with other significant tools such as Pig, Sqoop, Flume, Oozie, and Zookeeper.<a href="https://linuxexpert.org/so-you-wanna-do-big-data/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://linuxexpert.org/so-you-wanna-do-big-data/</a><a href="https://mastodon.social/tags/Hadoop" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hadoop</a> <a href="https://mastodon.social/tags/HBase" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HBase</a> <a href="https://mastodon.social/tags/Hive" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hive</a> <a href="https://mastodon.social/tags/BigData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#BigData</a> <a href="https://mastodon.social/tags/HadoopEcosystem" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HadoopEcosystem</a> <a href="https://mastodon.social/tags/HDFS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDFS</a> <a href="https://mastodon.social/tags/MapReduce" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MapReduce</a> <a href="https://mastodon.social/tags/YARN" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#YARN</a> <a href="https://mastodon.social/tags/Pig" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Pig</a> <a href="https://mastodon.social/tags/Sqoop" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Sqoop</a> <a href="https://mastodon.social/tags/Flume" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Flume</a> <a href="https://mastodon.social/tags/Oozie" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Oozie</a> <a href="https://mastodon.social/tags/Zookeeper" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Zookeeper</a> <a href="https://mastodon.social/tags/DataProcessing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataProcessing</a> <a href="https://mastodon.social/tags/DataAnalytics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataAnalytics</a> <a href="https://mastodon.social/tags/DataWarehousing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataWarehousing</a> <a href="https://mastodon.social/tags/ETL" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ETL</a> <a href="https://mastodon.social/tags/DataIngestion" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataIngestion</a> <a href="https://mastodon.social/tags/Security" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Security</a>

Patryk KrawaczyńskiPrzyśpieszanie odczytu zajętości dysków na Hadoop < 2.8.X ( <a href="https://nfsec.pl/root/6233" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://nfsec.pl/root/6233</a> ) <a href="https://infosec.exchange/tags/hadoop" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hadoop</a> <a href="https://infosec.exchange/tags/hdfs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hdfs</a> <a href="https://infosec.exchange/tags/tuning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tuning</a> <a href="https://infosec.exchange/tags/twittermigration" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#twittermigration</a>

Peter CzanikI did my usual syslog-ng git snapshot compiles today. RPM platforms were all right. On <a href="https://fosstodon.org/tags/FreeBSD" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FreeBSD</a> I ran into a minor problem:<a href="https://github.com/syslog-ng/syslog-ng/issues/4515" rel="nofollow noopener noreferrer" target="_blank">https://github.com/syslog-ng/syslog-ng/issues/4515</a><a href="https://fosstodon.org/tags/Java" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Java</a> modules fail. My guess is that nobody is really affected, unless someone uses the <a href="https://fosstodon.org/tags/HDFS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDFS</a> destination of <a href="https://fosstodon.org/tags/syslog_ng" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#syslog_ng</a>.

Big Data analytics NewsHadoop admin interview questions <a href="https://buff.ly/2nB1KSM" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://buff.ly/2nB1KSM</a> <a href="https://mastodon.world/tags/hdfs" class="mention hashtag" rel="tag">#hdfs</a> <a href="https://mastodon.world/tags/hive" class="mention hashtag" rel="tag">#hive</a> <a href="https://mastodon.world/tags/hbase" class="mention hashtag" rel="tag">#hbase</a> <a href="https://mastodon.world/tags/mapr" class="mention hashtag" rel="tag">#mapr</a> <a href="https://mastodon.world/tags/database" class="mention hashtag" rel="tag">#database</a> <a href="https://mastodon.world/tags/nosql" class="mention hashtag" rel="tag">#nosql</a> <a href="https://mastodon.world/tags/bigdata" class="mention hashtag" rel="tag">#bigdata</a> <a href="https://mastodon.world/tags/ai" class="mention hashtag" rel="tag">#ai</a>

Big Data analytics News40 Best Free and Open Source <a href="https://mastodon.world/tags/NoSQL" class="mention hashtag" rel="tag">#NoSQL</a> Databases <a href="https://buff.ly/3b6LQag" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://buff.ly/3b6LQag</a> <a href="https://mastodon.world/tags/bigdata" class="mention hashtag" rel="tag">#bigdata</a> <a href="https://mastodon.world/tags/hadoop" class="mention hashtag" rel="tag">#hadoop</a> <a href="https://mastodon.world/tags/HDFS" class="mention hashtag" rel="tag">#HDFS</a> <a href="https://mastodon.world/tags/Hive" class="mention hashtag" rel="tag">#Hive</a> <a href="https://mastodon.world/tags/IoT" class="mention hashtag" rel="tag">#IoT</a> <a href="https://mastodon.world/tags/Fintech" class="mention hashtag" rel="tag">#Fintech</a> <a href="https://mastodon.world/tags/MapR" class="mention hashtag" rel="tag">#MapR</a> <a href="https://mastodon.world/tags/ArtificialIntelligence" class="mention hashtag" rel="tag">#ArtificialIntelligence</a> <a href="https://mastodon.world/tags/AI" class="mention hashtag" rel="tag">#AI</a> <a href="https://mastodon.world/tags/ML" class="mention hashtag" rel="tag">#ML</a> <a href="https://mastodon.world/tags/DataScience" class="mention hashtag" rel="tag">#DataScience</a> <a href="https://mastodon.world/tags/DataScientists" class="mention hashtag" rel="tag">#DataScientists</a> <a href="https://mastodon.world/tags/CodeNewbies" class="mention hashtag" rel="tag">#CodeNewbies</a> <a href="https://mastodon.world/tags/Tech" class="mention hashtag" rel="tag">#Tech</a> <a href="https://mastodon.world/tags/deeplearning" class="mention hashtag" rel="tag">#deeplearning</a> <a href="https://mastodon.world/tags/CyberSecurity" class="mention hashtag" rel="tag">#CyberSecurity</a> <a href="https://mastodon.world/tags/Python" class="mention hashtag" rel="tag">#Python</a> <a href="https://mastodon.world/tags/Coding" class="mention hashtag" rel="tag">#Coding</a> <a href="https://mastodon.world/tags/javascript" class="mention hashtag" rel="tag">#javascript</a> <a href="https://mastodon.world/tags/rstats" class="mention hashtag" rel="tag">#rstats</a> <a href="https://mastodon.world/tags/100DaysOfCode" class="mention hashtag" rel="tag">#100DaysOfCode</a> <a href="https://mastodon.world/tags/programming" class="mention hashtag" rel="tag">#programming</a> <a href="https://mastodon.world/tags/Linux" class="mention hashtag" rel="tag">#Linux</a> <a href="https://mastodon.world/tags/IoT" class="mention hashtag" rel="tag">#IoT</a> <a href="https://mastodon.world/tags/IIoT" class="mention hashtag" rel="tag">#IIoT</a> <a href="https://mastodon.world/tags/BigData" class="mention hashtag" rel="tag">#BigData</a>

Volkan Özçelik 🦄SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of file<a href="https://github.com/seaweedfs/seaweedfs" rel="nofollow noopener noreferrer" target="_blank">https://github.com/seaweedfs/seaweedfs</a><a href="https://hachyderm.io/tags/Kubernetes" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Kubernetes</a> <a href="https://hachyderm.io/tags/DistributedSystems" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DistributedSystems</a> <a href="https://hachyderm.io/tags/S3Compatible" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#S3Compatible</a> <a href="https://hachyderm.io/tags/ErasureCoding" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ErasureCoding</a> <a href="https://hachyderm.io/tags/HDFS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDFS</a> <a href="https://hachyderm.io/tags/Hadoop" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Hadoop</a> <a href="https://hachyderm.io/tags/Fuse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Fuse</a> <a href="https://hachyderm.io/tags/Posix" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Posix</a> <a href="https://hachyderm.io/tags/infra" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#infra</a> <a href="https://hachyderm.io/tags/SeaweedFS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SeaweedFS</a>

Recent searches

Search options

Administered by:

Server stats:

#hdfs