Setting Up a Search Engine

If you want to set up a search engine for a private network or personal website, Searx and YaCy can help you get started. Both open source packages are simple to install and offer several installation methods, including Docker images, which provide an especially easy way to become familiar with each search engine as well as how the search engine affects your system.

With both Searx and YaCy, you need to pay attention to your system load. With YaCy, you also need to be mindful of your bandwidth due to YaCy's distributed nature.

Both YaCy and Searx can be tested using public instances, and both offer detailed websites with information on how to change settings to match your system, as well as defaults for other settings.

Finally, Searx and YaCy are also available as source code and other binary packages. YaCy, which uses Java, is the simpler option; you run the ant binary to compile. However, you can also install the Searx package from your distribution's standard repository.

Searx as a Solution

If you like Docker, you can stick with it for installation. You do need to pull the Searx package and select which ports to use:

docker pull searx/searx

All Searx settings are available for both regular and Docker installations. There are advantages to using the locally compiled version with performance being one such reason. To get deeper into the installation and settings, see the Searx installation document [9].

If you want to use another installation method, download the source code from GitHub [10] and use the appropriate script to install. You'll find several scripts. The NGINX and Apache web servers are supported. If you want to add Searx to an existing site, follow the instructions on the NGINX and Apache websites.

You should also be able to find the Searx package in your distribution's repository using a general binary. This works fine unless you want to squeeze all the performance possible out of your system.

Searx Settings

Regardless of installation method, you will use the default settings file (settings.yml) to set your preferences. You can use the default options, which contains many well-known search engines and some lesser known ones.

You also need to set up a directory for settings.yml and then point to it. The files in this directory are the ones controlling Searx. Since the settings file is written in YAML, you have access to all of YAML's features for creating links (like adding a site) that work as searches for your service.

When you compile, you will need most expected libraries for handling networking, the build, etc. You should check out the uwsgi install, which is a minimal binary protocol for communicating between the nodes.

In the main settings, you have one absolutely vital task to perform: setting the secret_key value (Listing 1). To choose this value, your best option is to let OpenSLL create it for you, but you can also use a password manager. The other options under the main settings are useful but not necessary to get started.

01 server: port : 8888
02 bind_address : "127.0.0.1" # address to listen on
03 secret_key : "SuperSecretKey" # change this!
04 base_url : <http://localhost:/> # Set custom base_url.
05 Possible values: False or
06 "<https://your.custom.host/location/>"
07 image_proxy : False # Proxying image results
08 through searx http_protocol_version : "1.0" # 1.0 and
09 1.1 are supported
10 method: "POST" # POST queries are more secure as
11 they don't show up in history but may cause problems when using
12 Firefox containers
13 default_http_headers:
14 X-Content-Type-Options :nosniff
15 X-XSS-Protection : 1;
16 mode=block
17 X-Download-Options : noopen
18 X-Robots-Tag : noindex, nofollow
19 Referrer-Policy : no-referrer

If you look through the rest of the settings file, you will find the valid search providers for your instance. You can also use this file to limit your searches if you have concerns about a particular provider.

To add more providers (you can even add a single page), you need to add a section to the settings file. Copy the default file and edit what you need. As a simple example, you can edit the Wikipedia settings entry in the file:

- name : wikipedia engine : wikipedia shortcut : wp base_url :
'https://{language}.wikipedia.org/'

In addition to the language variable, you'll find the query, page, and params variables, which can be used to control your searches. You also can control the type of results that are returned. Result options include strings, images, and videos, as well as torrent files.