Machine Learning Primer

Claudius Gros, WS 2024/25

Institut für theoretische Physik
Goethe-University Frankfurt a.M.

Linux - Basics

what is Linux

history

Unix $\to$ Linux
- 1960s/70s: Unix (AT&T) secure, multi-user
- 1991: Linus kernel (Linus Torval) open source
- 1990s Gnu libaries and untilities added
  (Free Software Foundation)
typing, not clicking
:: you know what you do (hopefully)
scripts
...

online utilities

one-click installation for Windows
Ubuntu Windows installer
online linux shells
- GoCalc
- copy.sh
- ...

Linux directories

Linux directory structure

cd directory : change to directory
cd / : change to root
cd : change to home
cd .. : one up
cd . : to current directory
ls : list directory content

user@pc:~$ cd /
user@pc:/$ cd var
user@pc:/var$

mkdir - create directory

mkdir directory : create directory
mkdir -p foo/test/abc/123/bar :
create directory foo/test/abc/123/bar and all parents if necessary

user@pc:~$ mkdir test
user@pc:~$ ls
test

rmdir – remove directory

rmdir directory : delete directory
only works on empty directories

user@pc:~$ rmdir test
user@pc:~$ rmdir downloads
rmdir: failed to remove `downloads': Directory not empty
user@pc:~$

recursive removal

rm -r directory/
delete directory and all subdirectories

Linux files

more – show file on screen

more file : show file on screen
move with ↑ and ↓
exit with Q
more /proc/cpuinfo : show processor informations

Linux file names

Hallo.dat and hallo.dat are distinct:
capital sensitive
data.really-important-stuff : arbitrary file extentions
avoid empty spaces

mv – move/rename files

mv file1 file2 : rename file1 to file2
mv file directory/ : move file to directory/
mv myImages/*.jpg /home/user/otherUser/yourImages/ :
move all jpg files in the directory myImages/
to the directory /home/user/otherUser/yourImages/

user@pc:~$ mv foo.txt bar.txt
user@pc:~$ mv bar.txt downloads
user@pc:~$

rm – remove files or directories

rm file : delete file
rm directory : delete directory
rm -r directory : recursive delete directory and all containing files
rm image* : delete all files starting with image
rm -r * : remove all data you ever created

user@pc:~$ rm downloads/bar.txt 
user@pc:~$ rm -r downloads/
user@pc:~$

cat – concatenate files and print to standard output

cat file : print contents of file
cat file1 file2 > file3 : concatenate the files file1 and file2
and print output to file3
cat aaa* > aaa.all : concatenate all files starting with aaa

user@pc:~$ cat file1
first line of file1
second line of file1

user@pc:~$ cat file2
first line of file2
second line of file2

user@pc:~$ cat file* > all
user@pc:~$ cat all
first line of file1
second line of file1
first line of file2
second line of file2

hidden files:
files starting with a `.` are hidden

used for setting user preferences/configuration for applications
.vimrc settings for vi/vim editor
.eclipse settings for the Eclipse development environment
...

user@pc:~/TEST$ ls -l
total 32
-rw-rw-r-- 1 user user  1775 2012-03-09 17:58 HashMapMapDemo.class
-rw-rw-r-- 1 user user  2019 2012-03-09 17:59 HashMapMapDemo.java
-rw-rw-r-- 1 user user 20961 2012-03-09 15:45 hashMap.svg
user@pc:~/TEST$ ls -la
total 56
drwxrwxr-x   2 user user  4096 2012-03-12 09:57 .
drwxr-xr-x 153 user user 12288 2012-03-12 09:56 ..
-rw-rw-r--   1 user user  1775 2012-03-09 17:58 HashMapMapDemo.class
-rw-rw-r--   1 user user  2019 2012-03-09 17:59 HashMapMapDemo.java
-rw-rw-r--   1 user user 20961 2012-03-09 15:45 hashMap.svg
-rw-rw-r--   1 user user     3 2012-03-12 09:56 .I-am-a-hidden-file

Linux command options

ls – list directory contents

ls -l : long listing format
ls -a : list all files
ls -h : human readable
ls *dat : all files ending with 'dat':
use wildcards

convert – audio, video and text formats

man convert : for options
convert -delay 50 frame*.gif -loop 0 animated.gif
converts frame01.gif, frame02.gif, .., into an animated gif
convert test.gif frame%04d.gif
extract frames from animated gif as size-4 integers
the basis of all crawlers: wget
:: downloading files and entire webpages
image viewer: eog ('eye of gnome')

wget https://upload.wikimedia.org/wikipedia/commons/3/30/Jud-morote-gari.gif
eog Jud-morote-gari.gif
mv Jud* test.gif
convert test.gif frame%04d.gif
convert -delay 50 frame*.gif -loop 0 animated.gif
eog animated.gif

Linux process control

ps – report a snapshot of the current processes

ps : list current processes
ps aux : list all processes

user@pc:~$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   4020   476 ?        Ss   Feb20   1:08 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Feb20   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Feb20   0:02 [migration/0]
root         4  0.0  0.0      0     0 ?        S    Feb20   0:23 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    Feb20   0:00 [watchdog/0]
root         6  0.0  0.0      0     0 ?        S    Feb20   0:04 [migration/1]
root         7  0.0  0.0      0     0 ?        S    Feb20   1:04 [ksoftirqd/1]
1635      4650  0.0  0.0  92248  2752 ?        S    Apr21   0:00 /usr/lib/eclipse/eclipse -vm /usr/lib/jvm/java-gcj/bin/java -install /usr/lib/eclipse
1635      4656  0.7  9.4 1021416 381456 ?      Sl   Apr21  29:27 /usr/lib/jvm/java-gcj/bin/java -Djava.library.path=/usr/lib/jni -Dgnu.gcj.precompiled
daemon    4715  0.0  0.0   8084   524 ?        Ss   Feb20   0:00 /sbin/portmap
root      4776  0.0  0.0    740    36 ?        Ss   Feb20   0:37 /usr/local/bin/netbeep eth0
statd     4813  0.0  0.0  14416   816 ?        Ss   Feb20   0:00 /sbin/rpc.statd -p 2051 -o 2052
1635     17246  0.0  0.1  17052  7332 pts/1    Ss   09:13   0:00 bash
user@pc:~$

kill – send a signal to a process

kill pid : send signal TERM to process with id pid
kill -9 pid : kill process with id pid
killall firefox : kill all processes with process name firefox

user@pc:~$ ps -l
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1467  8100  8095  0  80   0 -  8641 wait   pts/0    00:00:02 bash
0 S  1467 16732  8100  1  80   0 - 133912 poll_s pts/0   00:00:00 kwrite
0 R  1467 16797  8100  0  80   0 -  5796 -      pts/0    00:00:00 ps
0 S  1467 26446     1  0  80   0 - 221014 futex_ pts/0   00:02:11 soffice.bin
user@pc:~$ kill 16732

piping of Linux commands

| - output of one command as input of next

sort filename : sort the file filename
grep expression : extract all lines containing the expression from output
create files with content with echo

echo 'Ralph Junghans
Anna Rosenkranz
Joachim Neumayer
Anna Furlger' > names.txt

user@pc:~$ more names.txt 
Ralph Junghans
Anna Rosenkranz
Joachim Neumayer
Anna Furlger

user@pc:~$ more names.txt | grep "Anna" | sort > names.anna

user@pc:~$ more names.anna 
Anna Furlger
Anna Rosenkranz

wc - counting lines/words/characters

user@pc:~$ more names.txt | grep Anna
Anna Rosenkranz
Anna Furlger

user@pc:~ more names.txt | grep Anna | wc
      2       4      29

user@pc:~$ wc names.txt 
 4  8 61 names.txt

Linux editors

use the editor you feel at ease with

emacs, kwrite, .. possible editors
vi/vim : standard editor
keyboard controlled - no need to use the mouse

`i`		start inserting modus
`R`		start replacing modus
`o`		start insert in next line
`a`		start inserting at end of current line
`esc`		exit editing modi
`u`		undo last insertion or change

`r`		replace current char
`x`		delete char
`7x`		delete seven chars
`dd`		delete line
`8dd`		delete eight line
`y`		copy (yank) current line to buffer
`9y`		copy (yank) nine lines to buffer
`p`		insert (paste) buffer

`j`		move cursor a line down
`k`		move cursor a line up
`h`		move cursor to the left
`l`		move cursor to the right

`:w`		save (write) file
`:q`		exit (quit) `vi`
`:wq`		write and exit

`:111`		goto line 111
`:1,11d`		delete lines 1 through 11
`:1,$s/error/correct/`		substitute `error` by `correct` throught the entire file
`:10,20s/yes/no/g`		substitution from line 10 to twenty, also multiple times per line (`g`)

connection to remote hosts

using university computers from home

ssh remote-host : establish a connection to remote host
itp.uni-frankfurt.de
th.physik.uni-frankfurt.de
scp remote-host-path . : transfer data from remote host to current directory
scp ./filename remote-host-path : transfer data from current directory to remote host
g++ : compile a C++ program
time... : measuring the time needed to excute a program
nohup .a/out & : starting a program which runs in the background (&)
and which continues to be executed after logging-out (nohup)

user@pc:~$ ssh -l gros karlstad.th.physik.uni-frankfurt.de
gros@karlstad.th.physik.uni-frankfurt.de's password: 

gros@karlstad:~$ cd TEST/
gros@karlstad:~$ javac Test.java 
gros@karlstad:~$ nohup time java Test test.old test.new &
[1] 2176
gros@karlstad:~$ nohup: ignoring input and appending output to `nohup.out'

gros@karlstad:~$ exit
logout
Connection to karlstad.th.physik.uni-frankfurt.de closed.

user@pc:~$ scp gros@karlstad.th.physik.uni-frankfurt.de:TEST/nohup.out .
gros@karlstad.th.physik.uni-frankfurt.de's password: 
nohup.out                                                 100%  326     0.3KB/s   00:00    

user@pc:~$ more nohup.out 
# starting Test.main()
# scanning the file  test.old
# scanning the file  test.new
     1000    lines in    test.old
     1000    lines in    test.new
       84    lines in    test.output
0.15user 0.02system 0:00.21elapsed 77%CPU (0avgtext+0avgdata 77248maxresident)k
888inputs+80outputs (6major+5261minor)pagefaults 0swaps

ping remote-host : checking availability and connection quality

user@pc:~$ ping 141.2.247.163
PING 141.2.247.163 (141.2.247.163) 56(84) bytes of data.
64 bytes from 141.2.247.163: icmp_req=1 ttl=64 time=4.99 ms
64 bytes from 141.2.247.163: icmp_req=2 ttl=64 time=0.132 ms
64 bytes from 141.2.247.163: icmp_req=3 ttl=64 time=0.263 ms
64 bytes from 141.2.247.163: icmp_req=4 ttl=64 time=0.299 ms

Linux commands - overview

Directory Operations

cd		Change into the given directory (`"."` present directory, `".."` one level up, `"/"` top (root) directory)
ls		Show (list) the content of the given directory (`-a` all, `-l` long, `-h` human readable)
mkdir		Create (make) a new directory
pwd		Show path to the present directory
rmdir		Delete (remove) given directory when empty

Editors

emacs		Windowed editor with a large palette of functions
vi / vim		Standard in-console Unix-editors with full keyboard control

File Operations

chmod		For changing access rights for `-r` reading, `-w` writing and `-x` executing
cp		Copy files and directory (`-r` recursively)
file		Shows file type and information
more		Prints a text (file) on screen consecutively
mv		For moving (name change) of files and directories
rm		Deletes (removes) a given file or directory (`-r` recursively)
tail		Prints the last lines of a file to screen

File Manipulation

cat		Combines (concatenates) several files into a single file
grep		Search in a output for a string or expression (`more test.txt \| grep "information"`)
kprinter		Sending file to a printer
okular		Universal viewer for documents, images and graphics
sort		Sorting an output or a file (`-r` reverse, `-u` unique, `-g` numerical)
split		Splitting a given file into parts
wc		Counting lines, words and chars in an output or file (`-l` only lines)

Info and Documentation

apropos		Gives a list of man-pages for a given subject
man		Prints man pages (documentation) for a given command or program to screen

Data Compression

gzip / bzip2		For compression of files and data
tar		For compression and decompression (with `-x`) of archives and multiple files
zip / unzip		Compression and decompression WinZip (DOS/Win) files

Miscellaneous

`↑` / `↓`		Tabbing through previous commands
date		Prints current data to screen
env		Prints a list of all environment variables
expr		Evaluation of symbolic mathatical expresions, mostly for scripts
find		Searching for files, staring from current directory.
history		History of previous shell commands
hostname		Name of hosting computer
passwd		Password change
`tab`		Autocompletion of commands

Process Management

`&`		Process running in the background (used at the end of a command line)
`^C`		Terminates (kills) the current process (running in current console)
kill		Terminates (kills) a process with a given ID (`PID`)
nice		Setting the priority of a given process, mostly in order not to disturb other users
nohup		Starting a program which will continue to run after the user logs out (no hangup)
ps		Lists processes started by the user in present console (`-a` all, `-l` long)
time		Measuring the time needed to excute a given process or program
top		Shows all running processe with real-time updating of active processe and their respective CPU and memory consumption

System Infos

df		Disc use of all partitions
du		Disc use of current and child directories
uptime		Time since last reboot

Remote Connections

ftp		Data transfer to and from a remote host with line commands (file transfer protocol)
ping		Sending pings to remote host
scp		Copy data to and from remote host (secure copy)
ssh		Connect to remote host and log in

Linux - shell scripts

automated execution of commands

chmod +x : make file executable (x)

user@pc:~$ vi test.sh
user@pc:~$ chmod -x test.sh

echo : print message to screen
sleep 8 : sleep for 8 seconds

#! /bin/bash 
i=0
N=2
while test $i -le $N
 do
 a=`expr 50 \\* $i`
 b=`expr $a + 1`
 more "inputFile_$i.dat" | grep "Anna" > "outFile_$b.out"
 echo "ich gehe jetzt für $i Sekunden schlafen " $i
 sleep $i;
 i=`expr $i + 1`
done 
cat outFile_*.out > outFile.all

create input files

echo 'Ralph Junghans
Anna Rosenkranz
Joachim Neumayer
Anna Furlger' > inputFile_0.dat

echo 'Keine Anna hier,
oder etwa doch?' > inputFile_1.dat

echo 'Hier ist nun wirklich
keine anna zu finden.' > inputFile_2.dat

run script

user@pc:~$ ls
inputFile_0.dat  inputFile_1.dat  inputFile_2.dat  test.sh

user@pc:~$ ./test.sh 
ich gehe jetzt für 0 Sekunden schlafen  0
ich gehe jetzt für 1 Sekunden schlafen  1
ich gehe jetzt für 2 Sekunden schlafen  2

user@pc:~$ ls
inputFile_0.dat  inputFile_2.dat  outFile_1.out   outFile.all
inputFile_1.dat  outFile_101.out  outFile_51.out  test.sh

check output files

user@pc:~$ more inputFile_*.dat
::::::::::::::
inputFile_0.dat
::::::::::::::
Ralph Junghans
Anna Rosenkranz
Joachim Neumayer
Anna Furlger
::::::::::::::
inputFile_1.dat
::::::::::::::
Keine Anna hier,
oder etwa doch?
::::::::::::::
inputFile_2.dat
::::::::::::::
Hier ist nun wirklich 
keine anna zu finden.

user@pc:~$ more outFile.all 
Anna Rosenkranz
Anna Furlger
Keine Anna hier,

Linux open-source utilities

a very large selection of free utilities

gnuplot datafile
`fast and dirty' visualization of xy, xyz or multicolumn data files
xmgrace datafile
more elaborate plotting of xy, xyz or multicolumn data files
gimp
advanced editor for bitmap images
inkscape
advanced editor for scalable vector graphics
alpine
console mail program
libreoffice
OpenOffice (LibreOffice) package
dot / neato / circo
Graphviz graph ploting utilities
awk
awk is a lightweight text manipulation utility

Machine Learning Primer

Linux - Basics

what is Linux

history

online utilities

Linux directories

Linux directory structure

mkdir - create directory

rmdir – remove directory

recursive removal

Linux files

more – show file on screen

Linux file names

mv – move/rename files

rm – remove files or directories

cat – concatenate files and print to standard output

hidden files: files starting with a . are hidden

Linux command options

ls – list directory contents

convert – audio, video and text formats

Linux process control

ps – report a snapshot of the current processes

kill – send a signal to a process

piping of Linux commands

| - output of one command as input of next

wc - counting lines/words/characters

Linux editors

use the editor you feel at ease with

connection to remote hosts

using university computers from home

Linux commands - overview

Directory Operations

Editors

File Operations

File Manipulation

Info and Documentation

Data Compression

Miscellaneous

Process Management

System Infos

Remote Connections

Linux - shell scripts

automated execution of commands

Linux open-source utilities

a very large selection of free utilities

hidden files:
files starting with a `.` are hidden