Mastodon

Anoncheg<a href="https://techhub.social/tags/dailyreport" class="mention hashtag" rel="nofollow noopener" target="_blank">#dailyreport</a> <a href="https://techhub.social/tags/tokenizers" class="mention hashtag" rel="nofollow noopener" target="_blank">#tokenizers</a> <a href="https://techhub.social/tags/huggingface" class="mention hashtag" rel="nofollow noopener" target="_blank">#huggingface</a> <a href="https://techhub.social/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#rust</a> <a href="https://techhub.social/tags/gentoo" class="mention hashtag" rel="nofollow noopener" target="_blank">#gentoo</a> <a href="https://techhub.social/tags/ebuild" class="mention hashtag" rel="nofollow noopener" target="_blank">#ebuild</a> <a href="https://techhub.social/tags/secops" class="mention hashtag" rel="nofollow noopener" target="_blank">#secops</a> <a href="https://techhub.social/tags/cargo" class="mention hashtag" rel="nofollow noopener" target="_blank">#cargo</a> I compiled the HF 🤗 tokenizers library from sources and enhanced Gentoo ebuild file to allow reproducible installation from sources. I removed optional dependencies and disabled HTTP requirements to enhance security.I wrote very simple tests for tokenizers, safetensors, transformers and integration test for them, because tokenizers require HF hub for testing, that I disabled.It was a hard but good experience with Cargo package manager of Rust. The main problems was due to strange cfg flags that Gentoo should have set automaticly, for ex. target_os=linux was not set. "cfg" is an abomination that you can't add change this safely.I didn't find a working solution to manage "cfg" and, so I just patched the Cargo.toml files of dependencies by commenting out lines. (∠・ω )⌒

Anoncheg<a href="https://techhub.social/tags/dailyreport" class="mention hashtag" rel="nofollow noopener" target="_blank">#dailyreport</a> <a href="https://techhub.social/tags/tokenizers" class="mention hashtag" rel="nofollow noopener" target="_blank">#tokenizers</a> <a href="https://techhub.social/tags/huggingface" class="mention hashtag" rel="nofollow noopener" target="_blank">#huggingface</a> <a href="https://techhub.social/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#rust</a> <a href="https://techhub.social/tags/gentoo" class="mention hashtag" rel="nofollow noopener" target="_blank">#gentoo</a> <a href="https://techhub.social/tags/ebuild" class="mention hashtag" rel="nofollow noopener" target="_blank">#ebuild</a> <a href="https://techhub.social/tags/secops" class="mention hashtag" rel="nofollow noopener" target="_blank">#secops</a> <a href="https://techhub.social/tags/cargo" class="mention hashtag" rel="nofollow noopener" target="_blank">#cargo</a> I compiled HF tokenizers library from sources and enhanced Gentoo ebuild file that allow reproducable installation from sources. I removed optional dependencies and disabled http requirements to enhance security.I wrote very simple tests for tokenizers, safetensors, transformers and integration test for them.It was hard but good experience with Cargo package manager of Rust. Main problems was because of strange cfg flags that Gentoo should set automaticly: target_os=linux was not set. "cfg" is an abomination you can't add change this safely.I didn't found working solution to manage "cfg" and just patched Cargo.toml files of dependencies. by commenting lines. (∠・ω )⌒

Arthur Hau, PhD🐶🐱🌱🎵🦣To most people the word <a href="https://tribe.net/tags/token" class="mention hashtag" rel="nofollow noopener" target="_blank">#token</a> is a black box. I am not using the <a href="https://tribe.net/tags/tokenizers" class="mention hashtag" rel="nofollow noopener" target="_blank">#tokenizers</a> that are commonly used in <a href="https://tribe.net/tags/DeepLearning" class="mention hashtag" rel="nofollow noopener" target="_blank">#DeepLearning</a> <a href="https://tribe.net/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a>. Instead I am using my own <a href="https://tribe.net/tags/WordCoding" class="mention hashtag" rel="nofollow noopener" target="_blank">#WordCoding</a> system that I will call yxxx+. I am using base 16 for coding 300 common ESL English words for my <a href="https://tribe.net/tags/SLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#SLM</a> project. y ranges from 0-F which denotes the <a href="https://tribe.net/tags/POS" class="mention hashtag" rel="nofollow noopener" target="_blank">#POS</a> (part of speech) of a word. xxx are 3 base 16 digits. Theoretically, I can expand my model to 4000 "base" words. + denotes an additional code which I will explain later. <a href="https://tribe.net/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a>

:rss: .NET BlogIntroducing the AI Dev Gallery: Your Gateway to Local AI Development with .NET <a href="https://devblogs.microsoft.com/dotnet/introducing-ai-dev-gallery-gateway-to-local-ai-development/" rel="nofollow noopener" translate="no" target="_blank">https://devblogs.microsoft.com/dotnet/introducing-ai-dev-gallery-gateway-to-local-ai-development/</a><a href="https://rss-mstdn.studiofreesia.com/tags/microsoft" class="mention hashtag" rel="nofollow noopener" target="_blank">#microsoft</a> <a href="https://rss-mstdn.studiofreesia.com/tags/NET" class="mention hashtag" rel="nofollow noopener" target="_blank">#NET</a> <a href="https://rss-mstdn.studiofreesia.com/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://rss-mstdn.studiofreesia.com/tags/NET_9" class="mention hashtag" rel="nofollow noopener" target="_blank">#NET_9</a> <a href="https://rss-mstdn.studiofreesia.com/tags/dev_tools" class="mention hashtag" rel="nofollow noopener" target="_blank">#dev_tools</a> <a href="https://rss-mstdn.studiofreesia.com/tags/generative_ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#generative_ai</a> <a href="https://rss-mstdn.studiofreesia.com/tags/Machine_Learning" class="mention hashtag" rel="nofollow noopener" target="_blank">#Machine_Learning</a> <a href="https://rss-mstdn.studiofreesia.com/tags/tokenizers" class="mention hashtag" rel="nofollow noopener" target="_blank">#tokenizers</a> <a href="https://rss-mstdn.studiofreesia.com/tags/vector_search" class="mention hashtag" rel="nofollow noopener" target="_blank">#vector_search</a>

michabbb🔧 <a href="https://social.vivaldi.net/tags/code2prompt" class="mention hashtag" rel="nofollow noopener" target="_blank">#code2prompt</a>: A command-line tool for converting codebases to <a href="https://social.vivaldi.net/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a> promptsKey features: • 📁 Generates well-formatted <a href="https://social.vivaldi.net/tags/Markdown" class="mention hashtag" rel="nofollow noopener" target="_blank">#Markdown</a> prompts with source tree structure • 🛠️ Customizable <a href="https://social.vivaldi.net/tags/Handlebars" class="mention hashtag" rel="nofollow noopener" target="_blank">#Handlebars</a> templates for versatile prompt generation • 🔍 Respects .gitignore and supports file filtering with glob patterns • 🔢 Displays token count using various <a href="https://social.vivaldi.net/tags/tokenizers" class="mention hashtag" rel="nofollow noopener" target="_blank">#tokenizers</a> (cl100k, p50k, r50k_base) • 📊 <a href="https://social.vivaldi.net/tags/Git" class="mention hashtag" rel="nofollow noopener" target="_blank">#Git</a> diff integration for commit messages and <a href="https://social.vivaldi.net/tags/PullRequest" class="mention hashtag" rel="nofollow noopener" target="_blank">#PullRequest</a> descriptions • 📋 Automatic clipboard copy and option to save output to fileAdditional capabilities: • 🔢 Line numbering for source code blocks • 🔀 JSON output option for structured data • 🚫 Exclusion of files/folders from source tree • 📝 Support for user-defined variables in templates<a href="https://social.vivaldi.net/tags/opensource" class="mention hashtag" rel="nofollow noopener" target="_blank">#opensource</a> project written in <a href="https://social.vivaldi.net/tags/Rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#Rust</a>, available on <a href="https://social.vivaldi.net/tags/crates_io" class="mention hashtag" rel="nofollow noopener" target="_blank">#crates_io</a> and <a href="https://social.vivaldi.net/tags/AUR" class="mention hashtag" rel="nofollow noopener" target="_blank">#AUR</a>Useful for: • Quick <a href="https://social.vivaldi.net/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a> prompt generation from codebases • Code documentation and analysis • Bug finding and security vulnerability assessment • Performance optimization suggestions<a href="https://github.com/mufeedvh/code2prompt" rel="nofollow noopener" translate="no" target="_blank">https://github.com/mufeedvh/code2prompt</a>

Recent searches

Search options

Administered by:

Server stats:

#tokenizers