GitHub - asciimoo/hister: Your own search engine
github.com
Your own search engine. Contribute to asciimoo/hister development by creating an account on GitHub.

Hi everyone,

I am the original author of Searx. I started Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control.

Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches that index entirely on your own infrastructure.

Hister can automatically index pages through its Firefox and Chrome extensions. It can also watch local directories, import browser history and bookmarks, index individual URLs, and crawl complete documentation sites.

The feature I find most useful is offline previews. Hister stores the readable content and HTML of indexed pages locally. You can open a result in a clean and sanitized preview beside the search results without visiting the original website again.

Some other features:

  1. Full text search across web pages, PDFs, docx files, Markdown, OrgMode and text files
  2. Phrase searches, field filters, date filters, wildcards, negation, aliases, labels, facets, and result priorities
  3. Optional semantic search using an embeddings endpoint you configure
  4. Persistent website crawls
  5. Imports from browser history, Linkwarden, Karakeep, Shaarli, Wallabag, and Linkding
  6. Web, terminal, command line, HTTP API, and MCP interfaces
  7. SQLite and PostgreSQL support, plus optional multiple user hosting

Hister cannot replace a global search engine (yet) for subjects you have never encountered because it only searches what you have indexed. My workflow is to search Hister first, then use its shortcut to fall back to traditional search when I need broader web results.

The project is free software under the AGPLv3+ license. It can be installed as a standalone binary or with Docker.

Project: https://github.com/asciimoo/hister

Website and documentation: https://hister.org/

Small read-only demo: https://demo.hister.org/

I’d appreciate feedback, questions, and suggestions as well as joining our growing community.

AI disclosure: AI assisted contributions are not strictly prohibited, but all contributions should be made by humans. More details: https://github.com/asciimoo/hister/blob/master/CONTRIBUTING.md#ai-policy

  • SuspiciousCarrot78@aussie.zoneEnglish
    39·
    vor 3 Stunden

    If anyone else is confused what this does, I pointed the clankers at it and asked. Way to bury the lede OP.

    See [6] for some missing features (?) that would be nice to clear up.

    [1]

    • You open a webpage normally.
    • The browser extension reads the page that your browser loaded.
    • It sends the page text, title, URL, and other details to your Hister server.
    • Hister stores and indexes that content.
    • Later, you search Hister for words from the page.
    • Hister finds the saved page, even when you cannot remember its title or URL.

    [2]

    so, like a Google search of visited sites and my own drive?

    Yes. That is actually a reasonable use case.

    If you pointed Hister at:

    C:\Documents

    it would:

    1. Walk the folder tree.

    2. Read supported files.

    3. Build a full-text index.

    4. Watch for changes.

    5. Re-index new or edited files automatically.

    You could then search for things like:

    tax invoice lenovo

    lumbar radiculopathy

    that python script with sqlite

    “Methylphenidate” (exact phrase)

    without caring which folder the file lives in.

    Is it like Google?

    Yes, but only for your own data.

    Imagine Google, except the only websites are:

    C:\Documents

    D:\Notes

    your browser history

    pages you crawled yourself

    You open a browser, type a search, and receive ranked results from your own files instead of the internet.

    [3]

    Can you access it from elsewhere on your LAN?

    Yes.

    By default it listens only on localhost:

    127.0.0.1:4433

    However, you can configure it to listen on all network interfaces:

    0.0.0.0:4433

    Then any device on your LAN can use the web interface or API, provided your firewall allows it. You should also enable an access token or user accounts if you expose it beyond your own PC.

    [4] What interests me more for your home setup is that Hister is not just a search engine.

    It already exposes:

    a web UI,

    a REST API,

    and an MCP endpoint for AI assistants.

    That means you could run one Hister instance on your server, index your documents once, and then:

    search them yourself from any device on the LAN,

    or let an AI search the same index through MCP.

    If I wanted a self-hosted “Google for my own stuff” with minimal setup, Hister is much closer to that goal than Meilisearch. Meilisearch is an indexing engine. Hister is already a complete document search application.

    [5]

    Yes. By default, Hister stores its data in SQLite:

    Search index and metadata: db.sqlite3

    Stored HTML previews: separate files on disk

    Configuration: YAML

    It can also use PostgreSQL instead of SQLite if you want.

    Regarding browser history, there are two separate things:

    1. Your browser’s history database (Chrome, Firefox, etc.).

    2. Hister’s own index.

    They are independent after import.

    For example:

    Monday: You visit example.com.

    The extension sends the page to Hister.

    Hister stores the text, title, URL, metadata, and optionally the HTML preview.

    Tuesday: You clear Chrome’s history.

    The Hister copy remains. Clearing your browser history does not delete Hister’s copy. Hister is designed to be a long-term knowledge store, not a mirror of your browser history.

    If you want Hister to forget something, you must delete it from Hister itself. The web UI and terminal both support deleting indexed entries.

    I actually like that design. If it deleted documents every time you cleared browser history, it would defeat one of its main purposes. It treats browser history as an ingestion source, not as the source of truth.

    [6]

    From the current documentation:

    Are sources timestamped?

    Yes.

    Each indexed document records at least:

    Added timestamp.

    Updated timestamp.

    For browser pages, the visit timestamp.

    URL, title, and metadata.

    Is there a TTL (auto-expiry)?

    No.

    I could not find any TTL, age-based expiry, or retention policy.

    Documents remain until you delete them or overwrite them with newer content.

    Can I set a maximum database size?

    Not that I can find.

    There is no option such as:

    10 GB maximum.

    Delete oldest first.

    Keep only the last 365 days.

    Limit to 1 million documents.

    You can reduce storage by:

    Disabling HTML previews.

    Limiting indexed file size.

    Restricting indexed file types.

    Excluding paths or domains.

    Can I cleanly delete by topic?

    Partly.

    You can:

    Delete individual documents.

    Delete crawl jobs (but not the indexed documents).

    Label imported content.

    Query by labels and metadata.

    Configure rules to stop future indexing.

    I did not find built-in support for bulk deletion such as:

    “Delete everything about Tesla.”

    “Delete all Reddit pages.”

    “Delete every page tagged AI.”

    The API may allow scripted deletion, but I did not see a first-class “delete by query” feature in the documentation.

    Is browser ingestion automatic?

    Yes.

    That is one of its main features.

    Once the browser extension is installed and pointed at your server:

    Every page you visit is captured automatically.

    The extension extracts the rendered page.

    It sends it to Hister.

    If the page later changes, the extension periodically checks it and updates the stored copy.

    Duplicate visits update the existing document rather than creating a new one.

    You do not need to press a button.

    My only concern

    This project feels like it is missing a lifecycle policy.

    For a personal knowledge base, I would want options such as:

    Keep browser pages for 90 days.

    Keep GitHub forever.

    Keep PDFs forever.

    Delete pages that nobody has opened for two years.

    Maximum database size: 20 GB.

    Delete by label or query.

    I could not find those features in the current documentation.

    The project is very good at collecting information. It appears less mature at managing its lifetime. That would probably become the first thing I wanted if I intended to leave it running for years.

    [7]

    Yes, on both counts.

    Can I exclude particular sites?

    Yes, and it is quite flexible.

    Hister has skip rules that apply before a page enters the index. You can exclude:

    Entire domains.

    Subdomains.

    URL patterns.

    Login pages.

    Admin pages.

    Cookie consent pages.

    Anything that matches a regular expression.

    For example:

    ^https://reddit//.com/ ^https://mail//.google/.com/ ^https://.*.bank.com/

    Those pages never enter the index.

    The nice part is that the rules apply both to new browser captures and to re-index operations, so they stay excluded.

    How does multi-tenancy work?

    It is better than I expected.

    Each user has:

    Their own login.

    Their own search index.

    Their own browser history.

    Their own skip rules.

    Their own search aliases.

    Their own API token.

    If you and your wife both use the same Hister server:

    You ├── browser history ├── indexed pages ├── rules └── API token

    Wife ├── browser history ├── indexed pages ├── rules └── API token

    Neither user can search the other’s documents.

    There is also a global area.

    You can configure directories such as:

    /shared/docs

    Those appear in everyone’s searches, while personal folders remain private.

    One feature I really like

    Directory indexing also supports user ownership.

    For example:

    alice -> /nextcloud/alice bob -> /nextcloud/bob global -> /shared/docs

    The server enforces that visibility automatically. Even administrators do not automatically see other users’ indexed files.

    Overall, I came into this expecting “browser history search”. After reading the documentation, it is closer to a lightweight, self-hosted personal search appliance with sensible multi-user support. The remaining gaps are lifecycle management features such as retention policies, TTL, storage quotas, and richer bulk deletion. Those do not appear to be built in.

    • asciimoo@lemmy.mlOPEnglish
      7·
      vor 5 Stunden

      The summary has numerous inaccuracies. Most importantly: it is pretty easy to delete content by topic or age. The hister delete command can accept a search query to remove only matched documents. The same is true on the web UI “actions -> remove all matching documents”. You can quickly filter by age, simply query updated:>365d. Combine it with URLs, labels, domains or phrases.

      • SuspiciousCarrot78@aussie.zoneEnglish
        22·
        vor 5 Stunden

        Excellent - thanks for clearing that up.

        Is there a TTL / max database size per user setting? Say I have 4 users using the server; can I allocate a hard limit of 10GB per user, with 180 day retention rules?

        Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?

        I would like to further recommend it but your documentation is somewhat confusing at first blush.

        • asciimoo@lemmy.mlOPEnglish
          32·
          vor 5 Stunden

          Is there a TTL / max database size per user setting?

          As I wrote, you can simply automate deletion by document age. Schedule a delete event on each day with the desired retention time defined as a filter expression. Database size limit isn’t available yet.

          Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?

          No, sorry, I don’t have time to correct a copy of a multiple screens long AI prompt.

          I would like to further recommend Hister but your documentation is somewhat confusing at first blush.

          Which parts are confusing?

          • SuspiciousCarrot78@aussie.zoneEnglish
            16·
            vor 4 Stunden

            I am happy to narrow it further.

            The point is that you came here specifically to invite questions and feedback.

            I took the time to read the documentation, ask ChatGPT to summarise what I found, and then reduced my follow-up to a simple request:

            «Which of points 1–7 are materially wrong?»

            That is not the same as asking you to audit “multiple screens” of AI output.

            If the answer is “2 and 5 are incorrect”, or even “I do not have time to review it”, that is perfectly fine.

            However, dismissing it as “a multiple screens long AI prompt” does not only not answer the question, it comes off as abrasive.

            As for the documentation, the confusing parts are exactly those I listed: retention, lifecycle management, browser ingestion, storage limits, deletion, multi-user behaviour, and, most importantly, what Hister actually is and who it is for.

            What’s disappointing is not that you disagreed with the AI summary. AIs are idiots.

            It that after inviting questions and feedback, your response to a genuine attempt to understand the project is curt dismissal.

            The inner workings of Hister may be obvious to you; they are not obvious to others.

            “TL;DR” does not encourage the sort of community engagement you ostensibly came here to seek.

            • asciimoo@lemmy.mlOPEnglish
              41·
              vor 4 Stunden

              However, dismissing it as “a multiple screens long AI prompt” does not only not answer the question, it comes off as abrasive.

              I think it is more abrasive to copy/paste a poorly formatted LLM output instead of taking the time and summarizing it to a few sentences just as you did in your previous post.

              • SuspiciousCarrot78@aussie.zoneEnglish
                15·
                vor 3 Stunden

                That is not what happened.

                I fed your GitHub repository to a clanker because the documentation did not answer my questions. I then shared its summary here.

                You replied afterwards and said the summary was wrong. Fair enough. I then asked which specific points were wrong.

                You could have answered, declined, or ignored the post.

                Instead, you deigned only to dismiss the effort, then blamed me for objecting.

                You also asked which parts were confusing, although my previous reply had already listed those issues.

                You did not address them then, either.

                A prospective user should not need ChatGPT, a cloned repository, and several follow-up questions to understand key functions.

                You invited feedback. Your documentation remains unclear on several points, including issues beyond those I listed.

                Your responses show that further feedback is not worth my time.