commit 606d2c1dd62bcf938ab7262187949bac79f063fa Author: Virรกg Gรกbor Date: Wed Nov 6 10:56:28 2024 +0100 first commit diff --git a/README.md b/README.md new file mode 100644 index 0000000..a774508 --- /dev/null +++ b/README.md @@ -0,0 +1,731 @@ +![Whoogle Search](docs/banner.png) + +[![Latest Release](https://img.shields.io/github/v/release/benbusby/whoogle-search)](https://github.com/benbusby/shoogle/releases) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +[![tests](https://github.com/benbusby/whoogle-search/actions/workflows/tests.yml/badge.svg)](https://github.com/benbusby/whoogle-search/actions/workflows/tests.yml) +[![buildx](https://github.com/benbusby/whoogle-search/actions/workflows/buildx.yml/badge.svg)](https://github.com/benbusby/whoogle-search/actions/workflows/buildx.yml) +[![codebeat badge](https://codebeat.co/badges/e96cada2-fb6f-4528-8285-7d72abd74e8d)](https://codebeat.co/projects/github-com-benbusby-shoogle-master) +[![Docker Pulls](https://img.shields.io/docker/pulls/benbusby/whoogle-search)](https://hub.docker.com/r/benbusby/whoogle-search) + + + + + + +
SourceHutGitHub
+ +Get Google search results, but without any ads, JavaScript, AMP links, cookies, or IP address tracking. Easily deployable in one click as a Docker app, and customizable with a single config file. Quick and simple to implement as a primary search engine replacement on both desktop and mobile. + +Contents +1. [Features](#features) +3. [Install/Deploy Options](#install) + 1. [Heroku Quick Deploy](#heroku-quick-deploy) + 1. [Render.com](#render) + 1. [Repl.it](#replit) + 1. [Fly.io](#flyio) + 1. [Koyeb](#koyeb) + 1. [pipx](#pipx) + 1. [pip](#pip) + 1. [Manual](#manual) + 1. [Docker](#manual-docker) + 1. [Arch/AUR](#arch-linux--arch-based-distributions) + 1. [Helm/Kubernetes](#helm-chart-for-kubernetes) +4. [Environment Variables and Configuration](#environment-variables) +5. [Usage](#usage) +6. [Extra Steps](#extra-steps) + 1. [Set Primary Search Engine](#set-whoogle-as-your-primary-search-engine) + 2. [Custom Redirecting](#custom-redirecting) + 2. [Custom Bangs](#custom-bangs) + 3. [Prevent Downtime (Heroku Only)](#prevent-downtime-heroku-only) + 4. [Manual HTTPS Enforcement](#https-enforcement) + 5. [Using with Firefox Containers](#using-with-firefox-containers) + 6. [Reverse Proxying](#reverse-proxying) + 1. [Nginx](#nginx) +7. [Contributing](#contributing) +8. [FAQ](#faq) +9. [Public Instances](#public-instances) +10. [Screenshots](#screenshots) + +## Features +- No ads or sponsored content +- No JavaScript\* +- No cookies\*\* +- No tracking/linking of your personal IP address\*\*\* +- No AMP links +- No URL tracking tags (i.e. utm=%s) +- No referrer header +- Tor and HTTP/SOCKS proxy support +- Autocomplete/search suggestions +- POST request search and suggestion queries (when possible) +- View images at full res without site redirect (currently mobile only) +- Light/Dark/System theme modes (with support for [custom CSS theming](https://github.com/benbusby/whoogle-search/wiki/User-Contributed-CSS-Themes)) +- Randomly generated User Agent +- Easy to install/deploy +- DDG-style bang (i.e. `! `) searches +- User-defined [custom bangs](#custom-bangs) +- Optional location-based searching (i.e. results near \) +- Optional NoJS mode to view search results in a separate window with JavaScript blocked + +*No third party JavaScript. Whoogle can be used with JavaScript disabled, but if enabled, uses JavaScript for things like presenting search suggestions. + +**No third party cookies. Whoogle uses server side cookies (sessions) to store non-sensitive configuration settings such as theme, language, etc. Just like with JavaScript, cookies can be disabled and not affect Whoogle's search functionality. + +***If deployed to a remote server, or configured to send requests through a VPN, Tor, proxy, etc. + +## Install +There are a few different ways to begin using the app, depending on your preferences: + +___ + +### [Heroku Quick Deploy](https://heroku.com/about) +[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/benbusby/whoogle-search/tree/main) + +Provides: +- Easy Deployment of App +- A HTTPS url (https://\.herokuapp.com) + +Notes: +- Requires a **PAID** Heroku Account. +- Sometimes has issues with auto-redirecting to `https`. Make sure to navigate to the `https` version of your app before adding as a default search engine. + +___ + +### [Render](https://render.com) + +Create an account on [render.com](https://render.com) and import the Whoogle repo with the following settings: + +- Runtime: `Python 3` +- Build Command: `pip install -r requirements.txt` +- Run Command: `./run` + +___ + +### [Repl.it](https://repl.it) +[![Run on Repl.it](https://repl.it/badge/github/benbusby/whoogle-search)](https://repl.it/github/benbusby/whoogle-search) + +*Note: Requires a (free) Replit account* + +Provides: +- Free deployment of app +- Free HTTPS url (https://\.\\.repl\.co) + - Supports custom domains +- Downtime after periods of inactivity ([solution](https://repl.it/talk/learn/How-to-use-and-setup-UptimeRobot/9003)\) + +___ + +### [Fly.io](https://fly.io) + +You will need a [Fly.io](https://fly.io) account to deploy Whoogle. + +#### Install the CLI: https://fly.io/docs/hands-on/installing/ + +#### Deploy the app + +```bash +flyctl auth login +flyctl launch --image benbusby/whoogle-search:latest +``` + +The first deploy won't succeed because the default `internal_port` is wrong. +To fix this, open the generated `fly.toml` file, set `services.internal_port` to `5000` and run `flyctl launch` again. + +Your app is now available at `https://.fly.dev`. + +Notes: +- Requires a [**PAID**](https://fly.io/docs/about/pricing/#free-allowances) Fly.io Account. + +___ + +### [Koyeb](https://www.koyeb.com) + +Use one of the following guides to install Whoogle on Koyeb: + +1. Using GitHub: https://www.koyeb.com/docs/quickstart/deploy-with-git +2. Using Docker: https://www.koyeb.com/docs/quickstart/deploy-a-docker-application + +___ + +### [pipx](https://github.com/pipxproject/pipx#install-pipx) +Persistent install: + +`pipx install https://github.com/benbusby/whoogle-search/archive/refs/heads/main.zip` + +Sandboxed temporary instance: + +`pipx run --spec git+https://github.com/benbusby/whoogle-search.git whoogle-search` + +___ + +### pip +`pip install whoogle-search` + +```bash +$ whoogle-search --help +usage: whoogle-search [-h] [--port ] [--host ] [--debug] [--https-only] [--userpass ] + [--proxyauth ] [--proxytype ] [--proxyloc ] + +Whoogle Search console runner + +optional arguments: + -h, --help Show this help message and exit + --port Specifies a port to run on (default 5000) + --host Specifies the host address to use (default 127.0.0.1) + --debug Activates debug mode for the server (default False) + --https-only Enforces HTTPS redirects for all requests + --userpass + Sets a username/password basic auth combo (default None) + --proxyauth + Sets a username/password for a HTTP/SOCKS proxy (default None) + --proxytype + Sets a proxy type for all connections (default None) + --proxyloc + Sets a proxy location for all connections (default None) +``` +See the [available environment variables](#environment-variables) for additional configuration. + +___ + +### Manual + +*Note: `Content-Security-Policy` headers can be sent by Whoogle if you set `WHOOGLE_CSP`.* + +#### Dependencies +- [Python3](https://www.python.org/downloads/) +- `libcurl4-openssl-dev` and `libssl-dev` + - macOS: `brew install openssl curl-openssl` + - Ubuntu: `sudo apt-get install -y libcurl4-openssl-dev libssl-dev` + - Arch: `pacman -S curl openssl` + +#### Install + +Clone the repo and run the following commands to start the app in a local-only environment: + +```bash +git clone https://github.com/benbusby/whoogle-search.git +cd whoogle-search +python3 -m venv venv +source venv/bin/activate +pip install -r requirements.txt +./run +``` +See the [available environment variables](#environment-variables) for additional configuration. + +#### systemd Configuration +After building the virtual environment, you can add something like the following to `/lib/systemd/system/whoogle.service` to set up a Whoogle Search systemd service: + +```ini +[Unit] +Description=Whoogle + +[Service] +# Basic auth configuration, uncomment to enable +#Environment=WHOOGLE_USER= +#Environment=WHOOGLE_PASS= +# Proxy configuration, uncomment to enable +#Environment=WHOOGLE_PROXY_USER= +#Environment=WHOOGLE_PROXY_PASS= +#Environment=WHOOGLE_PROXY_TYPE= +# Site alternative configurations, uncomment to enable +# Note: If not set, the feature will still be available +# with default values. +#Environment=WHOOGLE_ALT_TW=farside.link/nitter +#Environment=WHOOGLE_ALT_YT=farside.link/invidious +#Environment=WHOOGLE_ALT_RD=farside.link/libreddit +#Environment=WHOOGLE_ALT_MD=farside.link/scribe +#Environment=WHOOGLE_ALT_TL=farside.link/lingva +#Environment=WHOOGLE_ALT_IMG=farside.link/rimgo +#Environment=WHOOGLE_ALT_WIKI=farside.link/wikiless +#Environment=WHOOGLE_ALT_IMDB=farside.link/libremdb +#Environment=WHOOGLE_ALT_QUORA=farside.link/quetre +#Environment=WHOOGLE_ALT_SO=farside.link/anonymousoverflow +# Load values from dotenv only +#Environment=WHOOGLE_DOTENV=1 +# specify dotenv location if not in default location +#Environment=WHOOGLE_DOTENV_PATH=/whoogle.env +Type=simple +User= +# If installed as a package, add: +ExecStart=/python3 /whoogle-search --host 127.0.0.1 --port 5000 +# For example: +# ExecStart=/usr/bin/python3 /home/my_username/.local/bin/whoogle-search --host 127.0.0.1 --port 5000 +# Otherwise if running the app from source, add: +ExecStart=/run +# For example: +# ExecStart=/var/www/whoogle-search/run +WorkingDirectory= +ExecReload=/bin/kill -HUP $MAINPID +Restart=always +RestartSec=3 +SyslogIdentifier=whoogle + +[Install] +WantedBy=multi-user.target +``` +Then, +``` +sudo systemctl daemon-reload +sudo systemctl enable whoogle +sudo systemctl start whoogle +``` + +#### Tor Configuration *optional* +If routing your request through Tor you will need to make the following adjustments. +Due to the nature of interacting with Google through Tor we will need to be able to send signals to Tor and therefore authenticate with it. + +There are two authentication methods, password and cookie. You will need to make changes to your torrc: + * Cookie + 1. Uncomment or add the following lines in your torrc: + - `ControlPort 9051` + - `CookieAuthentication 1` + - `DataDirectoryGroupReadable 1` + - `CookieAuthFileGroupReadable 1` + + 2. Make the tor auth cookie readable: + - This is assuming that you are using a dedicated user to run whoogle. If you are using a different user replace `whoogle` with that user. + + 1. `chmod tor:whoogle /var/lib/tor` + 2. `chmod tor:whoogle /var/lib/tor/control_auth_cookie` + + 3. Restart the tor service: + - `systemctl restart tor` + + 4. Set the Tor environment variable to 1, `WHOOGLE_CONFIG_TOR`. Refer to the [Environment Variables](#environment-variables) section for more details. + - This may be added in the systemd unit file or env file `WHOOGLE_CONFIG_TOR=1` + + * Password + 1. Run this command: + - `tor --hash-password {Your Password Here}`; put your password in place of `{Your Password Here}`. + - Keep the output of this command, you will be placing it in your torrc. + - Keep the password input of this command, you will be using it later. + + 2. Uncomment or add the following lines in your torrc: + - `ControlPort 9051` + - `HashedControlPassword {Place output here}`; put the output of the previous command in place of `{Place output here}`. + + 3. Now take the password from the first step and place it in the control.conf file within the whoogle working directory, ie. [misc/tor/control.conf](misc/tor/control.conf) + - If you want to place your password file in a different location set this location with the `WHOOGLE_TOR_CONF` environment variable. Refer to the [Environment Variables](#environment-variables) section for more details. + + 4. Heavily restrict access to control.conf to only be readable by the user running whoogle: + - `chmod 400 control.conf` + + 5. Finally set the Tor environment variable and use password variable to 1, `WHOOGLE_CONFIG_TOR` and `WHOOGLE_TOR_USE_PASS`. Refer to the [Environment Variables](#environment-variables) section for more details. + - These may be added to the systemd unit file or env file: + - `WHOOGLE_CONFIG_TOR=1` + - `WHOOGLE_TOR_USE_PASS=1` + +___ + +### Manual (Docker) +1. Ensure the Docker daemon is running, and is accessible by your user account + - To add user permissions, you can execute `sudo usermod -aG docker yourusername` + - Running `docker ps` should return something besides an error. If you encounter an error saying the daemon isn't running, try `sudo systemctl start docker` (Linux) or ensure the docker tool is running (Windows/macOS). +2. Clone and deploy the docker app using a method below: + +#### Docker CLI + +Through Docker Hub: +```bash +docker pull benbusby/whoogle-search +docker run --publish 5000:5000 --detach --name whoogle-search benbusby/whoogle-search:latest +``` + +or with docker-compose: + +```bash +git clone https://github.com/benbusby/whoogle-search.git +cd whoogle-search +docker-compose up +``` + +or by building yourself: + +```bash +git clone https://github.com/benbusby/whoogle-search.git +cd whoogle-search +docker build --tag whoogle-search:1.0 . +docker run --publish 5000:5000 --detach --name whoogle-search whoogle-search:1.0 +``` + +Optionally, you can also enable some of the following environment variables to further customize your instance: + +```bash +docker run --publish 5000:5000 --detach --name whoogle-search \ + -e WHOOGLE_USER=username \ + -e WHOOGLE_PASS=password \ + -e WHOOGLE_PROXY_USER=username \ + -e WHOOGLE_PROXY_PASS=password \ + -e WHOOGLE_PROXY_TYPE=socks5 \ + -e WHOOGLE_PROXY_LOC=ip \ + whoogle-search:1.0 +``` + +And kill with: `docker rm --force whoogle-search` + +#### Using [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli) +```bash +heroku login +heroku container:login +git clone https://github.com/benbusby/whoogle-search.git +cd whoogle-search +heroku create +heroku container:push web +heroku container:release web +heroku open +``` + +This series of commands can take a while, but once you run it once, you shouldn't have to run it again. The final command, `heroku open` will launch a tab in your web browser, where you can test out Whoogle and even [set it as your primary search engine](https://github.com/benbusby/whoogle#set-whoogle-as-your-primary-search-engine). +You may also edit environment variables from your appโ€™s Settings tab in the Heroku Dashboard. + +___ + +### Arch Linux & Arch-based Distributions +There is an [AUR package available](https://aur.archlinux.org/packages/whoogle-git/), as well as a pre-built and daily updated package available at [Chaotic-AUR](https://chaotic.cx). + +___ + +### Helm chart for Kubernetes +To use the Kubernetes Helm Chart: +1. Ensure you have [Helm](https://helm.sh/docs/intro/install/) `>=3.0.0` installed +2. Clone this repository +3. Update [charts/whoogle/values.yaml](./charts/whoogle/values.yaml) as desired +4. Run `helm install whoogle ./charts/whoogle` + +___ + +#### Using your own server, or alternative container deployment +There are other methods for deploying docker containers that are well outlined in [this article](https://rollout.io/blog/the-shortlist-of-docker-hosting/), but there are too many to describe set up for each here. Generally it should be about the same amount of effort as the Heroku deployment. + +Depending on your preferences, you can also deploy the app yourself on your own infrastructure. This route would require a few extra steps: + - A server (I personally recommend [Digital Ocean](https://www.digitalocean.com/pricing/) or [Linode](https://www.linode.com/pricing/), their cheapest tiers will work fine) + - Your own URL (I suppose this is optional, but recommended) + - SSL certificates (free through [Let's Encrypt](https://letsencrypt.org/getting-started/)) + - A bit more experience or willingness to work through issues + +## Environment Variables +There are a few optional environment variables available for customizing a Whoogle instance. These can be set manually, or copied into `whoogle.env` and enabled for your preferred deployment method: + +- Local runs: Set `WHOOGLE_DOTENV=1` before running +- With `docker-compose`: Uncomment the `env_file` option +- With `docker build/run`: Add `--env-file ./whoogle.env` to your command + +| Variable | Description | +| -------------------- | ----------------------------------------------------------------------------------------- | +| WHOOGLE_URL_PREFIX | The URL prefix to use for the whoogle instance (i.e. "/whoogle") | +| WHOOGLE_DOTENV | Load environment variables in `whoogle.env` | +| WHOOGLE_DOTENV_PATH | The path to `whoogle.env` if not in default location | +| WHOOGLE_USER | The username for basic auth. WHOOGLE_PASS must also be set if used. | +| WHOOGLE_PASS | The password for basic auth. WHOOGLE_USER must also be set if used. | +| WHOOGLE_PROXY_USER | The username of the proxy server. | +| WHOOGLE_PROXY_PASS | The password of the proxy server. | +| WHOOGLE_PROXY_TYPE | The type of the proxy server. Can be "socks5", "socks4", or "http". | +| WHOOGLE_PROXY_LOC | The location of the proxy server (host or ip). | +| WHOOGLE_USER_AGENT | The desktop user agent to use. Defaults to a randomly generated one. | +| WHOOGLE_USER_AGENT_MOBILE | The mobile user agent to use. Defaults to a randomly generated one. | +| WHOOGLE_USE_CLIENT_USER_AGENT | Enable to use your own user agent for all requests. Defaults to false. | +| WHOOGLE_REDIRECTS | Specify sites that should be redirected elsewhere. See [custom redirecting](#custom-redirecting). | +| EXPOSE_PORT | The port where Whoogle will be exposed. | +| HTTPS_ONLY | Enforce HTTPS. (See [here](https://github.com/benbusby/whoogle-search#https-enforcement)) | +| WHOOGLE_ALT_TW | The twitter.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_YT | The youtube.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_RD | The reddit.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_TL | The Google Translate alternative to use. This is used for all "translate ____" searches. Set to "" to disable. | +| WHOOGLE_ALT_MD | The medium.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_IMG | The imgur.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_WIKI | The wikipedia.org alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_IMDB | The imdb.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_QUORA | The quora.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_ALT_SO | The stackoverflow.com alternative to use when site alternatives are enabled in the config. Set to "" to disable. | +| WHOOGLE_AUTOCOMPLETE | Controls visibility of autocomplete/search suggestions. Default on -- use '0' to disable. | +| WHOOGLE_MINIMAL | Remove everything except basic result cards from all search queries. | +| WHOOGLE_CSP | Sets a default set of 'Content-Security-Policy' headers | +| WHOOGLE_RESULTS_PER_PAGE | Set the number of results per page | +| WHOOGLE_TOR_SERVICE | Enable/disable the Tor service on startup. Default on -- use '0' to disable. | +| WHOOGLE_TOR_USE_PASS | Use password authentication for tor control port. | +| WHOOGLE_TOR_CONF | The absolute path to the config file containing the password for the tor control port. Default: ./misc/tor/control.conf WHOOGLE_TOR_PASS must be 1 for this to work.| +| WHOOGLE_SHOW_FAVICONS | Show/hide favicons next to search result URLs. Default on. | +| WHOOGLE_UPDATE_CHECK | Enable/disable the automatic daily check for new versions of Whoogle. Default on. | +| WHOOGLE_FALLBACK_ENGINE_URL | Set a fallback Search Engine URL when there is internal server error or instance is rate-limited. Search query is appended to the end of the URL (eg. https://duckduckgo.com/?k1=-1&q=). | + +### Config Environment Variables +These environment variables allow setting default config values, but can be overwritten manually by using the home page config menu. These allow a shortcut for destroying/rebuilding an instance to the same config state every time. + +| Variable | Description | +| ------------------------------------ | --------------------------------------------------------------- | +| WHOOGLE_CONFIG_DISABLE | Hide config from UI and disallow changes to config by client | +| WHOOGLE_CONFIG_COUNTRY | Filter results by hosting country | +| WHOOGLE_CONFIG_LANGUAGE | Set interface language | +| WHOOGLE_CONFIG_SEARCH_LANGUAGE | Set search result language | +| WHOOGLE_CONFIG_BLOCK | Block websites from search results (use comma-separated list) | +| WHOOGLE_CONFIG_BLOCK_TITLE | Block search result with a REGEX filter on title | +| WHOOGLE_CONFIG_BLOCK_URL | Block search result with a REGEX filter on URL | +| WHOOGLE_CONFIG_THEME | Set theme mode (light, dark, or system) | +| WHOOGLE_CONFIG_SAFE | Enable safe searches | +| WHOOGLE_CONFIG_ALTS | Use social media site alternatives (nitter, invidious, etc) | +| WHOOGLE_CONFIG_NEAR | Restrict results to only those near a particular city | +| WHOOGLE_CONFIG_TOR | Use Tor routing (if available) | +| WHOOGLE_CONFIG_NEW_TAB | Always open results in new tab | +| WHOOGLE_CONFIG_VIEW_IMAGE | Enable View Image option | +| WHOOGLE_CONFIG_GET_ONLY | Search using GET requests only | +| WHOOGLE_CONFIG_URL | The root url of the instance (`https:///`) | +| WHOOGLE_CONFIG_STYLE | The custom CSS to use for styling (should be single line) | +| WHOOGLE_CONFIG_PREFERENCES_ENCRYPTED | Encrypt preferences token, requires preferences key | +| WHOOGLE_CONFIG_PREFERENCES_KEY | Key to encrypt preferences in URL (REQUIRED to show url) | +| WHOOGLE_CONFIG_ANON_VIEW | Include the "anonymous view" option for each search result | + +## Usage +Same as most search engines, with the exception of filtering by time range. + +To filter by a range of time, append ":past