Sbatch-service

This Page describes the management of dynamic web applications from the users view. That means running PHP, Python or generic CGI scripts.

Get the permission

For an improved security only static web pages are allowed in the users public_html folder. The run dynamic web applications a special permission is required and only intended for skilled users. Running faulty php can break major security hols and compromise the full account.

Why

For better scaling and isolation of the content, the users scripts are executed in a dedicated Apache instance. The program is executed in special Slurm partition and accessed by a reverse proxy. To simplify the handling we have implemented the command 'sbatch-service'.

Overview

This wrapper must be invoked as

 sbatch-service {init|deploy|start|stop|restart|reload|status|purge} [SERVICE ...]

to initialize/deploy/start/stop/restart/reload or check status of a specific persistent per-user sbatch-based one or more services listed by name as SERVICE (and so on), or alternatively to start/stop/restart/reload or check status of all sbatch-based services of the user (w/o SERVICE arguments), or to purge canonical SERVICE files (or all known services and central files of sbatch-service if no SERVICE given to 'purge').

The effects of 'init' are supposed to be idempotent for that service if nothing else changed. If a job is active on 'init', it is implicitly stopped before and *not* restarted after initialization, ie 'init' implies 'stop'.

On the other hand, 'deploy' implies 'stop' on any running instance before performing the actual ('init'-like) deployment, followed by 'start' (which 'init' does not perform). Unlike 'init', 'deploy' is also supported without a SERVICE argument (applying deployment to all services). This action is meant for easy re-deployment (updates/upgrades) of a service.

Note that 'init', 'deploy' must be used with an explicit 'SERVICE' argument if the corresponding service has never been set up before (or was purged).

When sbatch-service successfully completes 'start' or 'restart', the job is guaranteed to have been around at the time (though it may vanish for other reasons, including error conditions within the job itself).

When sbatch-service successfully completes 'stop', any previously present job is guaranteed to be gone.

By default, only users in group 'service-pool' are allowed to submit jobs on the equally named partition. Each job is called 'SERVICE-service', so that 'SERVICE' can be kept short while the job name remains significant regarding its service job nature.

Note that we rely on singleton nature of service-pool jobs (or jobs of some other suitable partition if configured), but cannot rely on sbatch option '-d singleton' for that (see slurmctld-prolog for details).

Reliable job control depends on coordinated invocation of sbatch-service. In particular, sbatch-service commands should only be invoked one at a time. Concurrent execution of multiple commands may produce unreliable results (both regarding sbatch-service operation and service script execution).

Defining custom service jobs

Service configuration and processes always apply $HOME as working directory. This is automatically taken care of by sbatch-service for consistency.

To define a custom service job, create

  .sbatch-service/SERVICE

which must be an executable (script) performing the actual service work when invoked with argument 'main' (within its sbatch job).

The process running this executable is also sent a potential reload signal, so should act accordingly (on SIGHUP by default, but see below). Note that this executable must not fork to daemonize, but stay in the job's foreground. Both stdout and stderr are sent to the log file.

This script is also invoked *locally* on other occasions, with a single argument defining the stage (ie not 'main'):

'init'

perform initialization of persistent files when first installing

the service (init or deploy) or later amending it to perform updates and additions (possibly init, but typically deploy); either will happen after sbatch-service itself has completed its generic part of the initialization (or idempotent augmenting).

No service instance will be around at this stage (temporarily stopped if one existed when sbatch-service was invoked for this action).

'start'

prepare start of service (explicitly requested, implied by

restart, or implied by deploy after temporary stop), optionally providing sbatch options on stdout, where each line must start with '#SBATCH', followed by zero or more sbatch options to apply after (and possibly overriding) the default options and before mandatory options added by sbatch-service; any other lines cause a fatal error. The current sbatch default option is '-p service-pool'.

For an explicit start, 'start' only happens if there is no job yet. For a start implied by restart or deploy, 'start' only happens after completely stopping the service. In other words, the 'start' stage implies that the job is not currently running/pending/lingering.

'stop', 'reload'

prepare stop (explicit or due to init/deploy/restart)

or reload of service, respectively, optionally providing the name of the signal to send (for stop or reload) on stdout. The default signal is 'SIGTERM' for 'stop' and 'SIGHUP' for reload. The signal is always sent to the main process only.

These stages only occur if the job is actually present. In other words, the 'stop' and 'reload' stages imply that there is a job at the time the script is invoked (though the job may even disappear then due to other factors, including it having been pending and the failing, or lingering).

Note also that some mandatory options cannot be overridden. Currently,

• sbatch options -J, -o, -e, --open-mode, --export,
• scancel options -u, -n, and
• squeue options -h, -a -u, -n, -t, -o

are always forcibly added by sbatch-service.

Log handling

The generic sbatch log file for each service is located at

  .sbatch-service/log/SERVICE.log

and automatically rotated by a logrotate configuration symlink

  .logrotate/conf.d/sbatch-service

auto-generated when the service is first invoked by sbatch-service.

Pre-defined service jobs

A pre-defined service type 'FOO' is taken from the system-wide template

  /etc/sbatch-service/FOO

by creating an equally named symlink in per-user .sbatch-service when FOO is initialized via sbatch-service by the given user for the first time.

The symlink FOO may later be manually replaced by a wrapper that invokes the corresponding /etc/sbatch-service/FOO target (so that you may add special sbatch/scancel options or pre/post processing) as needed. IMPORTANT: When installing a wrapper, at least when the argument is 'main', be sure to use 'exec' to run /etc/sbatch-service/FOO, unless you intend to explicitly perform reload/stop signal handling within the wrapper.

Pre-defined jobs are expected to perform their own initialization as needed. If they need their own directory set-up, they conventionally create it in

 .FOO-service

in the user's home directory, including a README for further details. If they need their own log rotation (typically in '.FOO-service/log') they conventionally auto-create

 .logrotate/conf.d/FOO-service

for that.

Above conventions are also expected/followed by 'purge', which is performed by sbatch-service itself (no service-specific script called).

Apache Service Type

Running Apache is the most common use case for using sbatch-Service. This is enabled with:

 sbatch-service init apache

Configuration Options

The default configuration only does static HTML. To enable active elements create or edit the file options.conf in the .apache-service directory. This file contains Apache Syntax and is include on top of the apache2.conf file.

In ITP all user which run apache-server are serving active pages (commonly PHP). At least you will need an ~/.apache-service/options.conf with PHP.

 Define EnablePHP

You have the following configuration options.

EnableAutoIndex

enables automatic index generation for directories that are accessible but do not have an index.* file (autoindex disabled by default for security reasons);

EnablePHP

enables PHP support (everywhere by convention; concerns all files *.php, *.php[3457], *.pht, *.phtml, *.phps; automatically treats index.php as a directory index);

EnableExplicitCGI

enables CGI support (explicitly set via .htaccess; this is implied by any of the below Enable...CGI tags); EnableContainedCGI enables CGI support (executable files in subdirectory cgi-bin only; implies EnableExplicitCGI)

EnableContainedCGI

enables CGI support (executable files in subdirectory cgi-bin only; implies EnableExplicitCGI);

EnableUbiquitousCGI

enables CGI support (everywhere; concerns all executable files *.cgi; implies EnableExplicitCGI; automatically treats index.cgi as a directory index);

EnableUbiquitousPerl

enables Perl support (everywhere; concerns all executable files *.pl; implies EnableExplicitCGI; consider using EnableContainedCGI instead and having executable files in cgi-bin only, *.pl or other, or using just EnableExplicitCGI with .htaccess; automatically treats index.pl as a directory index);

EnableExplicitSSI

enables SSI support (explicitly set via .htaccess; this is implied by the below Enable...SSI tag);

EnableUbiquitousSSI

enables SSI support (everywhere; concerns all files *.shtml; implies EnableExplicitSSI; note that this only enables IncludesNoExec, not full Includes, XBitHack, or support for legacy SSI parsing; all of that needs to be done via explicit options in .htaccess instead; automatically treats index.shtml as a directory index);

EnableStatus

enables status module (URL path /server-status);

DisableHTTPS

disables the default assumption of 'https' as frontend scheme, expecting it to be 'http' instead ;

DebugRewriting

sets rewrite log level to trace4 (which reports steps of RewriteCond and RewriteRules evaluation).

Cutting Path elements

Depending on your prefix URL some path elements may be cut away. The system is pre-configured for the default ~USERNAME URLs, which cuts the USERNAME from the request URI. If your service runs a full domain, you must disable this. Add

 Define PrefixParts 0

to your options.conf.

uWSGI

The uWSGI service is the second available interface for hosting web applications written in different laguages (like python, perl or ruby/rack)

When initilazing uWSGI a symlink

  .uwsgi-service/uwsgi.ini

is created automatically if not present yet; configuration via

 .uwsgi-service/app.ini
 .uwsgi-service/ports.ini

is required and created as needed on init/deploy. For app.ini, if not present yet, this asks for a request plugin (eg 'python') and location (eg 'wsgi-file') and creates a file from it; for ports.ini, this allocates a port as needed, heuristically deriving it from the UID of the invokding user, modulo 8192, adding 8192 (which may still fail to provide a suitable port number) and recording the port as a 'socket = :PORT' line.

This wrapper includes support for 'rack' applications, optionally with rvm. For the latter to work, make sure it is active in the current shell and enabled via ~/.profile in future shells (including the sbatch service job), by using something like

  gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
  curl -sSL https://get.rvm.io | bash -s -- --ignore-dotfiles stable
  . ~/.rvm/scripts/rvm
  { echo >>~/.profile; echo '# Activate rvm context.'; echo '. ~/.rvm/scripts/rvm'; } >>~/.profile

if rvm is not available as a system package, or by using

  { echo >>~/.profile; echo '# Activate rvm context.'; echo 'rvm use default'; } >>~/.profile

if rvm is available as a system package, and installing a desired ruby as eg

  rvm autolibs disable        # (no 'sudo apt-get update')
  rvm install 2.1             # (change version as needed)
  rvm use 2.1 --default       # (implied on first install, but be safe)
  sbatch-service init uwsgi   # (provides required plugin support) .

Note that the default environment is selected outside of uwsgi, allowing the latter to inherit it, so there is no need to have 'gemset = ...' in *.ini).

One may enable non-default options of the standard uwsgi configuration by creating/modifying a file

  .uwsgi-service/options.ini

holding one or more lines of the form

  [uwsgi]
  OPTION1 = VALUE1
  OPTION2 = VALUE2
  ...