This installs and setup MetaGer on a fresh debian installation.
Apache will be used as a webserver.
Redis will be used as Queue for the fetch jobs.
Supervisor will be used to keep the workers running that will fetch results from the search engines.
You can adapt those configurations to your needs.
This manual is tested on the latest stable debian version (stretch) but should also work on other distributions.
Install Dependencies
sudo apt -y update
sudo apt install -y git curl unzip build-essential libpng-dev supervisor apache2 redis-server php php-common php-curl php-mbstring php-mysql php-sqlite3 php-xml php-zip php-redis php-gd libapache2-mod-php
=> build-essential will only be needed if npm has to build some packages from source
Install the latest Composer Version
Steps from https://getcomposer.org/
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php
php -r "unlink('composer-setup.php');"
sudo mv composer.phar /usr/bin/composer
Install the latest nodejs
Steps from https://github.com/nodesource/distributions/blob/master/README.md#debinstall
-
curl -sL https://deb.nodesource.com/setup_11.x | bash -
(as root) sudo apt install -y nodejs
Download and install dependencies of MetaGer
git clone https://gitlab.metager3.de/open-source/MetaGer.git
cd MetaGer
composer install
npm install
cp .env.example .env
- Edit
.env
and setQUEUE_DRIVER=redis
(If you need to change settings for database and redis this would be the place, too) php artisan key:generate
- Set the correct Directory Permissions (web-server should be able to write to two dirs)
sudo chown -R :www-data storage && sudo chown -R :www-data bootstrap/cache
sudo chmod -R g+w storage && sudo chmod -R g+w bootstrap/cache
npm run prod
-
cp config/sumas.xml.example config/sumas.xml
(Enables some example search engines)
=> We will refer to the path where you cloned MetaGer to as $metager_path (including the MetaGer folder)
Make sure redis is running
sudo service redis restart
Setup and configure supervisor
cd /etc/supervisor/conf.d/
- Create the File laravel-worker.conf and put the following content
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php $metager_path/artisan queue:work --sleep=1 --tries=1 --timeout=295 --daemon
autostart=true
autorestart=true
user=$username
numprocs=200
redirect_stderr=true
stdout_logfile=$metager_path/storage/logs/worker.log
=> Replace the $metager_path and set the user to the username you cloned MetaGer with
3. Restart/start the supervisor daemon sudo service supervisor start
or sudo service supervisor restart
=> If it worked correctly you will see 200 php processes with pgrep php
. You can change the number in the config to whatever you find reasonable. It defines how many requests can be made to your search engines in parallel.
Setup and configure Apache
cd /etc/apache2/sites-available/
- Edit 000-default.conf and put the following content
<VirtualHost *:80>
# If you have a DNS-Entry pointing to your machine you can comment out the following line and set the dns name there
#ServerName www.example.com
ServerAdmin webmaster@localhost
# Replace $metager_path with the actual path
DocumentRoot $metager_path/public
<Directory $metager_path/public>
Allow Override all
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
sudo a2enmod rewrite
- restart/start the apache server
sudo service apache2 restart
If you want to modify/add searchengines to your MetaGer instance you will have to modify the search engine definition in $metager_path/config/sumas.xml and you will have to add a parser class for that search engine to $metager_path/app/Models/parserSkripte/.
With the information from the sumas.xml
MetaGer will make a request to all search engines with type=web
. The result will be given to above mentioned parser class where you have to write the code that transforms the result from the search engine API to a php object that MetaGer can use later.