The article suggests Common Crawl as a replacement which probably doesn’t make sense on its own due to low update frequency (monthly) and somewhat limited crawl scope.
(Looks like the sentence following the suggestion addresses this somewhat.)
Their API doesn't provide the mixed Kagi metasearch results though. Only results from their smallweb Teclis index, Curlie, and I think Marginalia results are included. Probably also other sources I'm not aware of.
Marginalia has a free-ish API for non-commercial purposes if you go direct:
ps. Modern AI is in reality talking web search engine imo.
janalsncm 1 days ago [-]
An LLM does not continuously index the Internet and is therefore stale the second it starts training. You need to use something like RAG. And the R in RAG needs an index.
hmokiguess 19 hours ago [-]
What other index is there that is both free, reliable, and accurate at web scale?
iberator 7 hours ago [-]
yuck duck go
mamman777 1 days ago [-]
I just see another small competitor becoming big by taking over
Marginalia: free-ish for non-commercial use https://about.marginalia-search.com/article/api/
Brave: free for first 1000/mth, 0.5c per search https://brave.com/search/api/
Mojeek: £0.001 - £0.003 (0.1p - 0.3p) per search https://www.mojeek.com/services/search/web-search-api/
Kagi: 2.5c per search https://help.kagi.com/kagi/api/search.html
https://programmablesearchengine.googleblog.com/2026/01/upda...
(Looks like the sentence following the suggestion addresses this somewhat.)
https://help.kagi.com/kagi/api/search.html
Their API doesn't provide the mixed Kagi metasearch results though. Only results from their smallweb Teclis index, Curlie, and I think Marginalia results are included. Probably also other sources I'm not aware of.
Marginalia has a free-ish API for non-commercial purposes if you go direct:
https://about.marginalia-search.com/article/api/
ps. Modern AI is in reality talking web search engine imo.