Friendica Social Network

Hister: self-hosted search engine for webpages and files with offline result preview

I'm working on a self-hosted search service called Hister with the goal to reduce my dependence on online search engines.

Hister is a full text indexer for websites which saves all the visited pages rendered by your browser. It provides a flexible web (and terminal) search interface & query language to explore previously visited content with ease or quickly fall back to traditional search engines.

I've been using it for a few months and as my local index is growing I can avoid opening google/duckduckgo/kagi more and more frequently.

The project is still heavily under development with a growing community, but the current version is in a fairly usable state in my opinion, so I wanted to share it here - perhaps some of you find it useful as well. (Or at least have some constructive criticism =])

The code is AGPLv3 licensed, available at github.com/asciimoo/hister
website: hister.org/
re

asciimoo - Overview

Author of Hister, Searx, Colly and a bunch of smaller projects - asciimoo

^GitHub

#privacy

in reply to asciimoo

utopiah

in reply to asciimoo • 2 weeks ago • •

Interesting, I didn't see it in the documentation so if you didn't document that already, you can have your local instance as search suggestion for Firefox on mobile and desktop. I use it for my own wiki, e.g. mastodon.pirateparty.be/@utopi…

Also how I would imagine it is default search there and if no hit then fallback to a default search engine, e.g. DDG.

Utopiah (Fabien Benetou) (@utopiah@mastodon.pirateparty.be)

Attached: 4 images So much hype about "agentic" this or that, meanwhile free software and open source already caught up on integration.

^{Utopiah (Fabien Benetou) (Mastodon - PiratesBE)}

in reply to utopiah

asciimoo

in reply to utopiah • 2 weeks ago • •

Also how I would imagine it is default search there and if no hit then fallback to a default search engine, e.g. DDG.

This is exactly how I use it. Hister has even a hotkey to quickly jump to your preferred online search engine with the current search query if you cannot find what you are looking for.

in reply to asciimoo

Samsy

in reply to asciimoo • 2 weeks ago • •

Uhh, neat. Will definitely try it out.

in reply to asciimoo

racoon

in reply to asciimoo • 2 weeks ago • •

I just want to be able to download and look up offline dictionaries

in reply to racoon

INeedMana

in reply to racoon • 2 weeks ago • •

For that I use f-droid.org/packages/com.akyla…
The slob dumps require a bit of hunting but other than that it works well for me

OSS-Dict | F-Droid - Free and Open Source Android App Repository

Offline Dictionaries

^f-droid.org

in reply to racoon

HawtP0tat0

in reply to racoon • 2 weeks ago • •

Not technically dictionaries but Kiwix does Wiki's offline. Unfortunately its only the wiki's they provide or you have to scrape yourself.

in reply to asciimoo

steel_for_humans

in reply to asciimoo • 2 weeks ago • •

I don't get it. It indexes pages which were already visited, right? So in order to find some website I need to first use another search engine. Afterwards, that website is in my browsing history and if I need it again, I don't need to search for it. So what's the use case for this project?

in reply to steel_for_humans

paris

in reply to steel_for_humans • 2 weeks ago • •

Your browsing history does not have full text search, so if you only remember the content of the page and not the title of it, you're SOL. Or if you browse across multiple devices, you have to check multiple places to hope to find it.

in reply to steel_for_humans

asciimoo

in reply to steel_for_humans • 2 weeks ago • •

It indexes pages which were already visited, right?

Yes, if you use the browser extension only, but Hister has an API and a crawler as well if you'd like to add content you have not visited yet. Also, Hister supports indexing local text files, not just websites.

Afterwards, that website is in my browsing history and if I need it again, I don’t need to search for it

Unfortunately browser history does not include the page's content only the URL + title combo at best.
Browser's can't show an offline preview (Having offline previews is a huge privacy - and productivity - win in my opinion, it completely eliminates the need of creating external network requests)

These are the biggest weaknesses of the browser history compared to Hister, but there are many more nuances where Hister can provide extra features and QoL improvements. I recommend checking the documentation & posts on the website if you are interested in the details.

in reply to asciimoo

ScoffingLizard

in reply to asciimoo • 2 weeks ago • •

So does that mean that the index starts off as empty? If so, is there a way to create a centralized (I know that's a bad word) starting repo such that the engine already knows some cool results? I have tabbed bookmarks for news that is not shitty, archives, video that isn't YouTube, privacy resources, etc. It would be cool if people could post indices focused on certain topics that they could add. Like indices for random stuff, like dog grooming, kayaking, or woodworking. It could be a hub like Docker Hub, but for cool results.

Sorry. Ha ha. You know you have a good idea when people start asking for features. I haven't even started it yet. Maybe I can try self hosting on my desktop.

This is exciting! I normally use Searxist on Android.

This entry was edited (2 weeks ago)

in reply to ScoffingLizard

asciimoo

in reply to ScoffingLizard • 2 weeks ago • •

I'd absolutely love to do this! It's already on my future plans list: hister.org/support :

Create infrastructure for importable, pre-indexed databases organized by topic, letting users quickly expand their local index with curated, relevant content.

It could be a hub like Docker Hub, but for cool results.

Exactly!

Sorry. Ha ha. You know you have a good idea when people start asking for features. I haven’t even started it yet.

❤
No need to apologize. I appreciate suggestions a lot (especially if those are well aligned with my ideas =] ).

Hister

Support Hister's mission to eliminate dependence on untrusted search services by contributing or sponsoring full-time development.

^hister.org

in reply to asciimoo

Free_Appalachia

in reply to asciimoo • 2 weeks ago • •

This looks really rad. I have been trying to build a leftist search engine using searxng and it has a lot of issue because of things getting blocked over tor due to the amount of background requests you have to be making to those sites. I have thought about building something like this to deal with that issue, but just had my hands full with other projects. Quite obviously (for those who like to come online to shit on really cool project people are building) the use case is people looking to build their own privately indexed search results without it having to get polluted by shit that comes up in other search engines. I feel like that was immediately apparent to me, because I have that use case and I immediately saw the utility here. For those who don't get, just because it doesn't fit the use case you have for internet related projects, doesn't mean it's not extremely useful to others. You literally don't have to come on here just to shit on what other people are doing simply because, "you don't get it." This project looks really cool and I am going to look into replacing my searxn... Show more...

This entry was edited (2 weeks ago)