Skip to content
Snippets Groups Projects
Commit 8e9e85e8 authored by Janis Daniel Dähne's avatar Janis Daniel Dähne
Browse files

- improved install guide

parent 252072d9
Branches
No related tags found
No related merge requests found
......@@ -2,7 +2,7 @@
we will install all yapex parts/projects into `/opt/yapex/*`
tested on debian 9
tested on debian 9 / ubuntu 18
## some useful hints
......@@ -25,8 +25,6 @@ ssh [user]@[xyz] -L 8080:[xyz]:8002 -N -v
goto https://github.com/dotnet/core/blob/master/release-notes/1.1/1.1.11/1.1.11-download.md and download sdk binaries (or runtime, if you want to build the client server locally and only copy fields)
we use the sdk here (if sdk complains about some `libunwind` is missing install then install `sudo apt-get install libunwind-dev`)
then
```
......@@ -39,15 +37,37 @@ cd /opt/dotnet
sudo ln -s /opt/dotnet/dotnet /usr/local/bin/dotnet
dotnet --version
# should output 1.1.11
# should output 1.1.12
```
then also install everything at https://github.com/dotnet/core/blob/master/Documentation/linux-prereqs.md
then
```
# required by nuget to install project dependencies via dotnet restore...
sudo apt-get install libcurl3
```
we use the sdk here (if sdk complains about some `libunwind` is missing install then install `sudo apt-get install libunwind-dev`)
if that fails with `Failed to initialize CoreCLR, HRESULT: 0x80131500` you **maybe** need to install some more libs...
run
```
find /opt/dotnet -name '*.so' -type f -print | xargs ldd | grep 'not found'
```
was missing in ubuntu 18 (but worked anyway)
```
liblldb-3.9.so.1 => not found
liblttng-ust.so.0 => not found
```
## install postgres (tested with version 9.x.x)
## install postgres (tested with version 10.x.x)
see https://www.postgresql.org/download/linux/debian/
or just try `apt-get install postgresql`
or just try `sudo apt-get install postgresql`
we will create a new postgres user to interact with the db
......@@ -65,16 +85,19 @@ sudo -u postgres psql postgres
# prompt should be psql=# here
grant all privileges on database "yapexDb" to yapex ;
#then exit
\q
# in production you might give only insert, update, delete rights...
# but then you need another user to do database migrations!
```
if you specified another db name or user name you need to ajust the settings in the `appsettings.json` in the client server...
if you specified another db name or user name you need to adjust the settings in the `appsettings.json` in the client server...
make sure we have local access
```
sudo vi /etc/postgresql/9.6/main/pg_hba.conf
sudo vi /etc/postgresql/[version]/main/pg_hba.conf
```
scroll down to something with `socket connection` (table at the very bottom)
......@@ -90,12 +113,9 @@ to
local all all md5
```
TODO setup db (root pw, create user)
## install client server
you can download the client server code locally and only deploy the needed files to run the client server
you can download the client server code locally and only deploy the needed files to run the client server
OR directly clone the repo on the server
we will clone the repo onto the server for this documentation
......@@ -105,39 +125,57 @@ sudo apt-get install git
sudo mkdir /opt/yapex/
cd /opt/yapex
# we clone here only the current branch
sudo git clone https://gitlab.informatik.uni-halle.de/Syndrom/ClientServer.git --branch nextVersion
sudo git clone https://gitlab.informatik.uni-halle.de/Syndrom/ClientServer.git --branch develop
# i use chown to not need to sudo everything inside ClientServer e.g. install dependencies
sudo chown -R ahbxf ClientServer/
sudo chown -R [nutzer] ClientServer/
```
**make sure in `src/ClientServer/ClientServer.csproj`**
- the `RuntimeFrameworkVersion` is set to your installed version (1.1.11), you can see your installed version when going into `cd /opt/dotnet/shared/Microsoft.NETCore.App` and look for a folder with `X.X.X`
- `RuntimeIdentifiers` is set to `linux-x64` (for a list see https://docs.microsoft.com/de-de/dotnet/core/rid-catalog)
>1.1.12 is the dotnet sdk/cli version, 1.1.11 is the runtime version
```
# install dependencies
cd /opt/yapex/ClientServer
dotnet restore
```
**make sure in `src/ClientServer/ClientServer.csproj`**
- the `RuntimeFrameworkVersion` is set to your installed version (1.1.11), you can see your installed version when going into `cd /opt/dotnet/shared/Microsoft.NETCore.App` and look for a folder with `X.X.X`
- `RuntimeIdentifiers` is set to `debian.9-x64` (for a list see https://docs.microsoft.com/de-de/dotnet/core/rid-catalog)
if you changed something in ClientServer.csproj go back to `/opt/yapex/ClientServer` and run `dotnet restore` again!
after that you can build the client server
you can get an idea about the build process from the file `build/build.sh` it should contain something like
---
if you cahnged anything go back to `/opt/yapex/ClientServer` and run `dotnet restore` again!
**before** you build the server (dotnet publish) you should configure the client server... see section *configuring the client server* below
**for the first run** make sure in `appsettings.json` *IsDbBootstrapEnabled* is set to `true` else the db tables will no be created!
---
make sure in `/opt/yapex/ClientServer/src/ClientServer` only you (and the group) has access to the `appsettings.json` because there the db password is stored
```
cd /opt/yapex/ClientServer/src/ClientServer
chmod o-r appsettings.json
# dotnet publish copies this file with it's file permission
```
cd src/ClientServer
after that you can build the client server
you can get an idea about the build process from the file `/opt/yapex/ClientServer/src/ClientServer/build/build.sh` it should contain something like (run this in a terminal)
```
cd /opt/yapex/ClientServer/src/ClientServer
dotnet publish --configuration Release --framework netcoreapp1.0 -o output/ClientServer
#--runtime debian.9-x64 should not be specified... i don't know why else the error 'libhostpolicy.so' required to execute the application.. is shown
# copy some scripts needed
# copy needed script
cp dashboard.php output/ClientServer/dashboard.php
```
the output is found in `output/ClientServer`
in `output/ClientServer` there should be an dir `wwwroot`. This is where will past the frontend files
in `output/ClientServer` there should be an dir `wwwroot`. This is where we will past the frontend files
then try to run the server by
......@@ -160,10 +198,7 @@ liblttng-ust.so.0 => not found
liblldb-3.9.so.1 => not found
```
were missing but it works anyway
----------- TODO
chmod ... appsettings... & appsettings in output?
were missing but it works anyway on ubuntu
### configuring the client server
......@@ -175,6 +210,7 @@ you need to change the following keys: `DbConnectionString`
```json
{
"IsDebugMode": true,
"IsDbBootstrapEnabled": true,
"DbConnectionString": "Username=[USERNAME];Password=[PASSWORD];Host=localhost;Database=[DBNAME];Pooling=true",
"SystemSalt": "NeverChangeThisAfterFirstStartup",
"ExternalLoginUrl": "https://uebungsportal.informatik.uni-halle.de/ostepu/DB/DBTransaction/transaction/authentication/redirect/transaction/",
......@@ -186,6 +222,7 @@ you need to change the following keys: `DbConnectionString`
}
```
`IsDbBootstrapEnabled` true to enable create db tables and apply any pending migrations/changes to the database
`MainEntrySite` needs to be the url where the site is running (e.g. via nginx)
`LoginExternalFirstTimeSite` is the url to redirect to after the user needs to set the initial password
`ExternalLoginUrl` is the login via ostepu
......@@ -200,13 +237,11 @@ chmod o-r /opt/yapex/ClientServer/src/ClientServer/appsettings.json
chmod o-r /opt/yapex/ClientServer/src/ClientServer/output/ClientServer/appsettings.json
```
### creating start script for systemd
### creating start script for systemd (optional)
```
cd /etc/systemd/system
touch runYapex.service
sudo touch runYapex.service
```
past the following into the file `runYapex.service`
......@@ -226,7 +261,7 @@ SyslogIdentifier=dotnet-yapex
WantedBy=multi-user.target
```
make sure the file can be executed `chmod +x runYapex.service`
make sure the file can be executed `sudo chmod +x runYapex.service`
then run to reload the daemon because we created/changed a uni
......@@ -234,18 +269,19 @@ then run to reload the daemon because we created/changed a uni
sudo systemctl daemon-reload
```
then you can `systemctl status unYapex.service` to get the status
`systemctl start unYapex.service` to start
`systemctl stop unYapex.service` to stop
`systemctl enable unYapex.service` to enable auto start when the server boots
`systemctl disable unYapex.service` to disable auto start when the server boots
then you can `systemctl status runYapex.service` to get the status
`systemctl start runYapex.service` to start
`systemctl stop runYapex.service` to stop
`systemctl enable runYapex.service` to enable auto start when the server boots
`systemctl disable runYapex.service` to disable auto start when the server boots
when you first start the server you might see `system settings not found, cannot run submissions scheduler.NextRun` in the logs
this is because the scheduler (which runs every X mins) tries to access the system settings (first row in table in the db) but we have no data yet
you can run `curl -i -X GET http://localhost:5000/api/init` to init the basic structure needed or import an old db dump
**on the first start** there are no data in the database. to fill in the default data open the browser at http://localhost:5000/api/init (some browsers display an json error, this can be ignored) OR run
`curl -i -X GET http://localhost:5000/api/init`
*this will only fill in the default data (first user, default group, default role, ...) when there are no (yapex) users in the database!*
## build frontend
......@@ -267,7 +303,8 @@ sudo apt-get install -y nodejs
sudo apt-get install gcc g++ make
# we want nodejs to be called node (to not change the scripts)
sudo ln -s /usr/bin/nodejs /usr/local/bin/node #or look where nodejs is installed via 'which nodejs'
#or look where nodejs is installed via 'which nodejs'
sudo ln -s /usr/bin/nodejs /usr/local/bin/node
```
also install yarn (package manager) (npm cannot install some dependencies...)
......@@ -281,10 +318,10 @@ sudo apt-get update && sudo apt-get install yarn
```
cd /opt/yapex
# we clone here only the current branch
sudo git clone https://gitlab.informatik.uni-halle.de/Syndrom/frontend-react.git --branch nextVersion
sudo git clone https://gitlab.informatik.uni-halle.de/Syndrom/frontend-react.git --branch develop
# i use chown to not need to sudo everything inside ClientServer e.g. install dependencies
sudo chown -R ahbxf frontend-react/
sudo chown -R [user] frontend-react/
#then install dependencies
cd /opt/yapex/frontend-react
......@@ -303,28 +340,38 @@ you need to copy the output into the `wwwroot` dir from the client server
cp -R /opt/yapex/frontend-react/dist/* /opt/yapex/ClientServer/src/ClientServer/output/ClientServer/wwwroot
# make sure the nocopy dir is not stored...
cd wwwroot
cd /opt/yapex/ClientServer/src/ClientServer/output/ClientServer/wwwroot
rm -R nocopy
```
after this you can start (if it is not running) the client server and open the browser at http://localhost:5000/index_deploy.html (assuming you didn't change the port)
## test server
when the app is running (backend & frontend) you may need to change the test server settting (if you changed the port).
When you open http://localhost:5000 on the server you will see the yapex site then login with
the default admin login is
>id: j
>pw: j
and goto menu > system settings and change `Test server*` settings to match the port you want to use.
later we will setup nginx and use it as the webserver and run our site on port 8002 so we would need to set
## test server
start the client server and open http://localhost:5000/index_deploy.html and login
and goto menu > `system settings` and change `Test server (urls)` settings to match the port you want to use.
later we will setup nginx and use it as a (reverse proxy) webserver and run our site on port 8002 and the test server on port 8003 so we would need to set
- `Test server url` to `http://localhost:8003/api.php`
- `Submit test server url` to `http://localhost:8003/api.php`
- `Test server stats url` to `http://localhost:8003/api.php`
- `Test server stats url` to `http://localhost:8003/stats/`
- `Test server config ui url` to `http://localhost:8003` (should point to phpmyadmin to manage the test server settings)
### install mysql
(https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/ OR https://dbahire.com/how-to-install-mysql-server-on-debian-stretch/)
```
sudo apt-get install mysql-server
#then setup
sudo mysql_secure_installation
```
if some post processing fails you may need to first clear all running mysql processes with
```
......@@ -372,8 +419,28 @@ sudo phpenmod mbstring
>Warning: When the prompt appears, “apache2” is highlighted, but not selected. If you do not hit SPACE to select Apache, the installer will not move the necessary files during installation. Hit SPACE, TAB, and then ENTER to select Apache.
then there should be a config for phpmyadmin in `/etc/apache2/conf-enabled/phpmyadmin.conf`
create a user called `yapex` you need to set the later in the `/opt/yapex/TestServer/config.json`
if not you probably no selected `apache2` in the configuration...
to rerun the configuration you can uninstall and reinstall phpmyadmin
```
sudo apt-get remove --purge phpmyadmin
sudo apt-get install phpmyadmin
```
you probably want to create a new user to manage the mysql database
```
sudo mysql --user=root mysql
#mysql>
CREATE USER '[DB user]'@'localhost' IDENTIFIED BY '[PASSWORD]';
GRANT ALL PRIVILEGES ON *.* TO '[DB user]'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;
\q
```
### install test server
......@@ -381,8 +448,8 @@ clone the test server
```
cd /opt/yapex
git clone https://gitlab.informatik.uni-halle.de/Syndrom/TestServer.git
sudo chown -R ahbxf TestServer
sudo git clone https://gitlab.informatik.uni-halle.de/Syndrom/TestServer.git
sudo chown -R [user] TestServer
```
then we need to edit the settings in `/opt/yapex/TestServer/config.json`
......@@ -395,12 +462,12 @@ e.g.
"dbName": "yapexTestDb",
"dbUser": "yapex",
"dbPw": "[PW]",
"dbTableName": "plangs",
"dbPLangTableName": "plangs",
"dbTransactionTableName": "transactions",
"useConfigFromDb": true,
"dbConfigServer": "localhost",
"dbConfigName": "yapexTestDb",
"dbConfigUser": "yapex",
"dbConfigUser": "[DB user]",
"dbConfigPw": "[PW]",
"dbTableNameConfig": "config",
......@@ -408,14 +475,30 @@ e.g.
}
```
the user `yapextester` is created via
```
sudo adduser yapextester
#pw: tester
```
you can read the `readme.md` for more information about the settings
now make sure only the correct users can access `/opt/yapex/TestServer/config.json`
```
sudo chmod o-r config.json
sudo chown yapextester config.json
#then try http://localhost:8003/config.json in thr browser should given 403 forbidden (because it has .json as file extension and we only forward .php)
```
then open phpmyadmin and create a new db e.g. `yapexTestDb`
then open phpmyadmin (http://localhost/phpmyadmin/) and create a new db e.g. `yapexTestDb`
you can create the tables as described in the `readme.md` in the test server or import the table from `TestServer/backup/*_server.sql`
the above config specifies that we use the table `config` over the local `config.json` file.
the above config specifies that we use the table `config` instead of the local `config.json` file.
You probably need to change the value for `environmentVars` to match your system/needs.
you will need to edit the table `config` and change `workingDirFullPath` to point the the dir used to run tests (the user files will be created there...)
......@@ -437,7 +520,7 @@ then you only need to start a *server* in the directory `/opt/yapex/TestServer/`
```
cd /opt/yapex/
sudo git clone https://gitlab.informatik.uni-halle.de/Syndrom/LinuxSafeTestRunner.git
sudo chown -R ahbxf LinuxSafeTestRunner
sudo chown -R [user] LinuxSafeTestRunner
```
you can also use the `DefaultTestRunnerFullThreaded` (https://gitlab.informatik.uni-halle.de/Syndrom/DefaultTestRunnerFullThreaded.git) but this is only for local testing because it does not run sandboxing commands
......@@ -458,7 +541,7 @@ cd /opt/yapex/LinuxSafeTestRunner/src
javac -encoding "UTF-8" Main.java
```
output will be `.class files`
output will be `.class` files
then we can set this in the configuration of the `TestServer`
open the table `config` in the `yapexTestDb` and edit the column `runner`
......@@ -488,10 +571,12 @@ then comment out all lines in `/etc/nginx/sites-available/default`
create a new entry for our site/app in `/etc/nginx/sites-enabled` (or in /etc/nginx/sites-available/ and then create a symlink)
```
sudo touch /etc/nginx/sites-enabled/yapex
sudo touch /etc/nginx/sites-available/yapex
ls /var/run/php/
# there should be a file phpX.Y-fpm.sock
```
and configure the site e.g.
and configure the site (at `/etc/nginx/sites-available/yapex`) e.g.
```
server {
......@@ -521,14 +606,20 @@ server {
location ~* \.php {
fastcgi_split_path_info ^(.+\.php)(/.+)$;
root /opt/yapex/TestServer;
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
fastcgi_pass unix:/var/run/php/php[X].[Y]-fpm-yapex.sock;
include fastcgi.conf;
}
}
```
restart nginx
then make the site available
```
sudo ln -s /etc/nginx/sites-available/yapex /etc/nginx/sites-enabled/
```
then restart nginx
```
sudo systemctl restart nginx.service
......@@ -537,15 +628,28 @@ sudo systemctl status nginx.service
then you can open http://localhost:8002 and should see the yapex site (you can use e.g. a ssh tunnel for that)
To get the real user ip (e.g. for exams) we need to add the X-Forwarded-For header via nginx (because else we ould get 127.0.0.1 because nginx is our reverse proxy).
Open `/etc/nginx/nginx.conf` and add to the `http` section
```
...
proxy_set_header X-Forwarded-For $remote_addr;
...
```
this will replace or set the X-Forwarded-For with the real remote addr... (the user might spoof the address badly, see https://serverfault.com/questions/314574/nginx-real-ip-header-and-x-forwarded-for-seems-wrong [ngx_http_realip_module] to prevent this)
you can also check the test-server is running properly when you open `http://localhost:8003/[a test-server.php file]` in the browser e.g. `http://localhost:8003/index.php`
### we switch from apach2 to nginx as webserver for phpMyAdmin
```
cd /etc/nginx/sites-enabled
sudo cp yapex phpMyAdmin
sudo vi phpMyAdmin
cd /etc/nginx/sites-available
ls /var/run/php/
# there should be a file phpX.Y-fpm.sock
sudo touch phpMyAdmin
```
open `/etc/nginx/sites-available/phpMyAdmin`
and insert
```
......@@ -557,13 +661,19 @@ server {
location ~* \.php {
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
fastcgi_pass unix:/var/run/php/php[X].[Y]-fpm.sock;
include fastcgi.conf;
}
}
```
then make the site available
```
sudo ln -s /etc/nginx/sites-available/phpMyAdmin /etc/nginx/sites-enabled/
```
then stop apache2 and restart nginx
```
......@@ -572,6 +682,7 @@ sudo systemctl disable apache2 # if we don't need apacahe anymore
sudo systemctl restart nginx
```
then you can open the browser at http://localhost:8000 and should see the phpmyadmin site
### configure php-fpm
......@@ -581,7 +692,7 @@ we will use apache php-fpm plugin to control the php processes
sudo apt install php-fpm
```
we already configured nginx to process php requests in the file `/etc/nginx/sites-enabled` to everything should work now
we already configured nginx to process php requests in the file `/etc/nginx/sites-available` to everything should work now
restart nginx via
......@@ -594,33 +705,36 @@ now almost everything should run. when you create an exercise in the ui and try
This is because the executing user (for the fpm) has no permission in the working dir `/opt/yapex/_tests`
so we need to change the executing user...
for security reasons we will create a new user
```
sudo adduser yapextester
#pw: tester
```
for security reasons we created the yapextester user
then change the fpm config
```
cd /etc/php/7.0/fpm/pool.d
sudo cp www.conf www.conf.backup
sudo vi www.config
cd /etc/php/[version]/fpm/pool.d
sudo cp www.conf www.conf.backup # just to be safe
sudo cp www.conf yapex_runner.conf
sudo vi www.conf
#change the lines
#; pool name ('www' here) to
#; pool name ('www' here) toy
[yapex]
...
# user = www-data
user = yapextester
group = yapextester
...
#listen = /run/php/php7.2-fpm.sock
listen = /run/php/php[X].[Y]-fpm-yapex.sock
```
then restart php-fcm
```
systemctl restart php7.0-fpm.service
systemctl restart php[X].[Y]-fpm.service
```
*we could let the default www-data user but we got for a custom user here*
......@@ -635,12 +749,12 @@ sudo chown -R yapextester _tests
if you used the `LinuxSafeTestRunner` you still need to setup firejal (sandbox program)
in order to run tests
remember that the new user syndromtest needs write access to `TestServer` and `LinuxSafeTestRunner`
remember that the new user `yapextester` needs write access to `TestServer` and `LinuxSafeTestRunner`
*[TODO-dev maybe we should run the runner with a different user that does not have permission to read the config from the `TestServer`]*
*[TODO-dev maybe we should run the runner with a different user that does not have permission to read the config from the `TestServer`??]*
### firejail
### firejail (required for LinuxSafeTestRunner)
the `LinuxSafeTestRunner` includes a very basic firejail config in `firejail/run.profile`
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment