Mastodon

➴➴➴Æ🜔Ɲ.Ƈꭚ⍴𝔥єɼ👩🏻‍💻Okay, Back of the napkin math: - There are probably 100 million sites and 1.5 billion pages worth indexing in a <a href="https://lgbtqia.space/tags/search" class="mention hashtag" rel="nofollow noopener" target="_blank">#search</a> engine - It takes about 1TB to <a href="https://lgbtqia.space/tags/index" class="mention hashtag" rel="nofollow noopener" target="_blank">#index</a> 30 million pages. - We only care about text on a page.I define a page as worth indexing if: - It is not a FAANG site - It has at least one referrer (no DD Web) - It's activeSo, this means we need 40TB of fast data to make a good index for the internet. That's not "runs locally" sized, but it is nonprofit sized.My size assumptions are basically as follows: - <a href="https://lgbtqia.space/tags/URL" class="mention hashtag" rel="nofollow noopener" target="_blank">#URL</a> - <a href="https://lgbtqia.space/tags/TFIDF" class="mention hashtag" rel="nofollow noopener" target="_blank">#TFIDF</a> information - Text <a href="https://lgbtqia.space/tags/Embeddings" class="mention hashtag" rel="nofollow noopener" target="_blank">#Embeddings</a> - Snippet We can store an index for 30kb. So, for 40TB we can store an full internet index. That's about $500 in storage.Access time becomes a problem. TFIDF for the whole internet can easily fit in ram. Even with <a href="https://lgbtqia.space/tags/quantized" class="mention hashtag" rel="nofollow noopener" target="_blank">#quantized</a> embeddings, you can only fit 2 million per GB in ram. Assuming you had enough RAM it could be fast: TF-IDF to get 100 million candidated, <a href="https://lgbtqia.space/tags/FAISS" class="mention hashtag" rel="nofollow noopener" target="_blank">#FAISS</a> to sort those, load snippets dynamically, potentially modify rank by referers etc.6 128 MG <a href="https://lgbtqia.space/tags/Framework" class="mention hashtag" rel="nofollow noopener" target="_blank">#Framework</a> <a href="https://lgbtqia.space/tags/desktops" class="mention hashtag" rel="nofollow noopener" target="_blank">#desktops</a> each with 5tb HDs (plus one raspberry pi to sort the final condidates from the six machines) is enough to replace <a href="https://lgbtqia.space/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#Google</a>. That's about $15k. In two to three years this will be doable on a single machine for around $3k.By the end of the decade it should be able to be run as an app on a powerful desktopThree years after that it can run on a <a href="https://lgbtqia.space/tags/laptop" class="mention hashtag" rel="nofollow noopener" target="_blank">#laptop</a>.Three years after that it can run on a <a href="https://lgbtqia.space/tags/cellphone" class="mention hashtag" rel="nofollow noopener" target="_blank">#cellphone</a>.By #2040 it's a background process on your cellphone.

JMLR'Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity', by Ali Kara, Naci Saldi, Serdar Yüksel.<a href="http://jmlr.org/papers/v24/21-1457.html" rel="nofollow noopener" target="_blank">http://jmlr.org/papers/v24/21-1457.html</a> <a href="https://sigmoid.social/tags/quantization" class="mention hashtag" rel="nofollow noopener" target="_blank">#quantization</a> <a href="https://sigmoid.social/tags/quantized" class="mention hashtag" rel="nofollow noopener" target="_blank">#quantized</a> <a href="https://sigmoid.social/tags/mdps" class="mention hashtag" rel="nofollow noopener" target="_blank">#mdps</a>

Recent searches

Search options

Administered by:

Server stats:

#quantized