Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
mastodon.world is one of the many independent Mastodon servers you can use to participate in the fediverse.
Generic Mastodon server for anyone to use.

Administered by:

Server stats:

8.9K
active users

mastodon.world: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.8

#FLOAT

1 post1 participant0 posts today
Public
Hacker News

70% Size, 100% Accuracy: Lossless LLM Compression via Dynamic-Length Float

arxiv.org/abs/2504.11651

#HackerNews #Lossless #LLM #Compression #Dynamic-Length #Float #AI #Research #Machine #Learning

arXiv logo
arXiv.org70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length FloatLarge Language Models (LLMs) have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original model. DFloat11 is motivated by the low entropy in the BFloat16 weight representation of LLMs, which reveals significant inefficiency in existing storage format. By applying entropy coding, DFloat11 assigns dynamic-length encodings to weights based on frequency, achieving near information-optimal compression without any loss of precision. To facilitate efficient inference with dynamic-length encodings, we develop a custom GPU kernel for fast online decompression. Our design incorporates the following: (i) decomposition of memory-intensive lookup tables (LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) transformer-block-level decompression to minimize latency. Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3, validates our hypothesis that DFloat11 achieves around 30% model size reduction while preserving bit-for-bit exact outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU to meet memory constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation. With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context lengths than uncompressed models. Notably, our method enables lossless inference of Llama-3.1-405B, an 810GB model, on a single node equipped with 8x80GB GPUs. Our code and models are available at https://github.com/LeanModels/DFloat11.
Public
Gea-Suan Lin

用「加法」來計算浮點數乘法的近似值

在「Why Does Integer Addition Approximate Float Multiplication? (probablydance.com)」這邊看到的，原文在「Why Does Integer Addition Approximate Float Mu

blog.gslin.org/archives/2025/0

#Murmuring #754 #addition #approximate #approximation #float #ieee #integer #multiplication

Gea-Suan Lin's BLOG · 用「加法」來計算浮點數乘法的近似值在「Why Does Integer Addition Approximate Float Multiplication? (probablydance.com)」這邊看到的，原文在「Why Does Integer Addition Approximate Float Multiplication?」。 這邊的浮點數主要指的是 IEEE 754-1985 (32 bits 的 single precision 與 64 bits 的 double precision)，因為表示方式的性質，剛好讓「加法」與乘法的操作有點像。 這邊刻意用引號括弧起來的「加法」是直接把 IEEE 754...
ExploreLive feeds
About