#zstd
Removing newlines in FASTA file increases ZSTD compression ratio by 10x
https://log.bede.im/2025/09/12/zstandard-long-range-genomes.html
Shalien 
Rilasciato #PeaZip 10.6: Il Gestore di Archivi #OpenSource per Tutti i Sistemi Operativi
The best technique available now is to start running a tor middle and the Chinese firewall will stop poorly programmed IPv4 LLM scrapers from China for you (just make sure you are also reachable over IPv6).
Another technique is to add bombs folder and link to it and make some GNUzip bomps;
dd if=/dev/zero bs=1G count=1 | gzip > 01GiB.gz
dd if=/dev/zero bs=1G count=1 | gzip > 10GiB.gz
Also make some zstandard bombs;
dd if=/dev/zero bs=1G count=1 | zstd > 01GiB.zst
dd if=/dev/zero bs=1G count=10 | zstd > 10GiB.zst
Also add a text file noting to right-click -> Save As to save the bombs instead of triggering them.
Also add to robots.txt;
User-agent: *
Disallow: /path/to/bombs/
Then add to the relevant server {} in nginx.conf;
#gzip bombs
location ~* /path/to/bombs/.*\.gz {
add_header Content-Encoding "gzip";
default_type "text/html";
}
#zstd bombs
location ~* /path/to/bombs/.*\.zst {
add_header Content-Encoding "zstd";
default_type "text/html";
}
Also 403 empty useragents; if ($http_user_agent = "") { return 403; }
There are also some LLM scrapers that will identify themselves;
if ($http_user_agent ~ (.*Amazonbot.*|.*Applebot.*|.*ClaudeBot.*|.*GPTBot.*)) { return 403; }
You also want ratelimiting;
In http {};
#10 requests per second on average
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
Then to each server{};
limit_req zone=perip burst=10 nodelay;
For bursty protocols like git, you'll need a larger burst amount (allows several git clone's in a row from an IP, but will block if you spam git clone);
limit_req zone=perip burst=1024 nodelay;
That is usually all that is needed to stop all aggressive LLM scrapers and should continue to work forever.
Sites manually targeted for scraping will need specifically targeted defenses (evil Anubis will be bypassed in such cases).
Как я делаю бекапы домашней системы Linux: простой пример инкрементального rsync + btrfs с zstd сжатием
Статья покажет простой rsync скрипт для инкрементального бекапа (с использованием хардлинков из предыдущего бекапа) и про использование btrfs сжатия в zstd.
Fedora 43 to Use Zstd Compression for Faster Boot and Smaller Initrd #fedora43 #zstd #initrd #initramfs #fastboot #linux #dracut
https://ostechnix.com/fedora-43-initrd-zstd-compression/
Introducing {zap} - a new serialization framework for #RStats
* uses type-specfic transformations + #zstd
* for 'diamonds' dataset, it is more compressed than 'saveRDS(xz)' and faster than 'saveRDS(compress=FALSE)
https://github.com/coolbutuseless/zap
https://coolbutuseless.r-universe.dev/zap
#Amazon анонсував новий алгоритм стискання (на жаль не такий як у #SiliconValley). Його можна спробувати використати для створення резервних копій заміть #gzip, бо він в 3-5 разів швидчий при тому самому рейті стискання
#backups #archiving #zstd
https://engineering.fb.com/2016/08/31/core-infra/smaller-and-faster-data-compression-with-zstandard/
The new alpm-package #crate offers integration with #ALPM based #package files. 
https://crates.io/crates/alpm-package/0.1.0
Currently it allows to create package files from prepared input directories and supports the #bzip2 #gzip #xz and #zstd #compression algorithms.
Zeekstd – Rust Implementation of the ZSTD Seekable Format
wow, if I'm reading the graphs right, 'zstd' long-range matching mode blows 'lrzip' on default settings out of the water in terms of performance
https://github.com/facebook/zstd/releases/tag/v1.3.2
(and that's for the 2017 release, who knows how well it performs nearly 8 years later!)
#libzip 1.11.4 has been released (#LZMA / #ZIP / #Zip64 / #zstd / #Zstandard / #bzip2 / #AES / #PKWARE / #FileArchiver / #DataCompression) https://libzip.org/
New #blog post: Unix Data Compression Shootout
https://rldane.space/unix-data-compression-shootout.html
881 words
cc: my wonderful #chorus: @joel @dm @sotolf @thedoctor @pixx @twizzay @orbitalmartian @adamsdesk @krafter @roguefoam @clayton @giantspacesquid
(I will happily add/remove you from the chorus upon request! :)
php -r 'for($i=256;$i-->0;)for($ii=256;$ii-->0;)print(chr($i).chr($ii));' | zstd | wc -c
131085 bytes (131KB). The #zstd "compression" only makes it larger at every compression level!
bzip2 compresses it fine, to be 5 times smaller. (It compresses best at the worst compression setting, -1)
I always wondered what circumstances make bzip2 occasionally do better than the much newer zstd. Finally found a pattern by accident :)
If you know how to make zstd handle this properly, let me know please
Today I discovered that there is also an excellent compression format, #Zstandard (ZSTD), is fast with an excellent compression ratio, developed by #Meta, and released as #opensource
I needed to backup my files because I have to wipe my PC and reinstall #Linux. Now I have a dilemma: choosing between #antiX and #Lubuntu. I have a fairly decent computer, so I could even install a more full-featured OS, but I prefer an OS that doesn’t use too many resources.
#zstd
https://www.youtube.com/watch?v=k5XsiuxHv_A
