Difference between revisions of "Storage and Backup"

From IT Service Wiki
Jump to: navigation, search
 
(17 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
The storage servers and importand hosts (like the login server) are connected with infiniband. Therefor prefer this host for doing larger file I/O as it will be much faster.
 +
 
== Storage ==
 
== Storage ==
  
−
There are different file systems with different lifetime and redundancy which you can use to store your data.  
+
There are different file systems with different lifetimes and redundancies which you can use to store your data.  
  
 
;your home directory
 
;your home directory
−
:This is your primary directory where all settings, your Mail and other important data is saved. It is mounted on every host to /home/YOUR_LOGIN. A backup is done nightly. This file system has an individual limit which you can check with the command 'df'. In change to earlier setups it has no soft limit which you can exceed for a few days.
+
:This is your primary directory where all settings, your Mail and other important data is saved. It is mounted on every host to /home/YOUR_LOGIN. A backup is done nightly. This file system has an individual limit which you can check with the command 'df'. In change to earlier setups it has no soft limit which you can exceed for a few days. If you have problems with graphical logins or reading mails, first check your disk space.
  
 
;/tmp 
 
;/tmp 
−
:This is the counterpart to your home. It is located on the local disk and should be used for temporary data. Unused files are delete after 14 days.
+
:This is the counterpart to your home. It is located on the local disk and should be used for temporary data. Unused files are deleted after 14 days.
 +
 
 +
;/mnt/ceph
 +
:This directory is globally accessible and can be used by everyone for minor important data, which is not worth to be backed up or simply to large for home. This files are not deleted, but if the disk is full we tell you to delete some files. It runs directly as cephfs on our CephCluster. This means a disk or even a host failure will not result in a service interruption. The Ceph Cluster is connected with Infiniband and you may get the best performance when using the login server or the compute nodes. It is currently limited to 30 TB.
 +
 
 +
:If you want to use this space please create a subdirectory with you login name. All other files and directories will be delete without notice.
  
−
;/home/work-itp<br>/home/work-itp2
+
;/home/work-itp (deprecated and replaced by /mnt/ceph)
−
:This directories are globally accessible and can be used by everyone for minor important data which are not worth to be backed up or simply to large for home. This files are not deleted, but if the disk is full we tell you to delete some files. It runs on an raid5 system, that means a disk failure will not result in data loss. It is also served by a dedicated machine (the old tantalos) and heavy load does not affect the home directories. If you want to use this space please create a sub directory with you login name.
+
:This directory is globally accessible and can be used by everyone for minor important data which are not worth to be backed up or simply to large for your home. These files are not deleted. But, if the disk is full, we tell you to delete some files. It runs on a raid6 system, which means that a disk failure will not result in data loss. It is also served by a dedicated machine (scratch1) and heavy load does not affect the home directories. It is connected with Infiniband and you may get the best performance when using the login server or the compute nodes. It is currently limited to 40 TB.  
  
−
:The new work-itp2 is powered by two Raid-Boxes, kindly provided the HRZ. They internally form a raid5 and are merged under the host to a raid0 (striping), which can be called a raid 50 and results in a disk throughput of near 2Gb/s. To give this performance to the network the host is connected with two Gigabit uplinks using channel bonding. Benchmarks with NFS has shown that we do not relay get much more performance than a 1Gb link. The reason is the the cpu's in the host are two weak. If we can get a stronger machine, this will be upgraded in the future.
+
;/data (some hosts only)
 +
:This is the local disk, but not available on all machines. It has the same level like /tmp except that it isn't deleted automatically. Data on this partition is silently deleted when reinstalling or upgrading the machine.
  
 
== Backup ==
 
== Backup ==

Latest revision as of 13:11, 10 March 2022

The storage servers and importand hosts (like the login server) are connected with infiniband. Therefor prefer this host for doing larger file I/O as it will be much faster.

Storage

There are different file systems with different lifetimes and redundancies which you can use to store your data.

your home directory
This is your primary directory where all settings, your Mail and other important data is saved. It is mounted on every host to /home/YOUR_LOGIN. A backup is done nightly. This file system has an individual limit which you can check with the command 'df'. In change to earlier setups it has no soft limit which you can exceed for a few days. If you have problems with graphical logins or reading mails, first check your disk space.
/tmp 
This is the counterpart to your home. It is located on the local disk and should be used for temporary data. Unused files are deleted after 14 days.
/mnt/ceph
This directory is globally accessible and can be used by everyone for minor important data, which is not worth to be backed up or simply to large for home. This files are not deleted, but if the disk is full we tell you to delete some files. It runs directly as cephfs on our CephCluster. This means a disk or even a host failure will not result in a service interruption. The Ceph Cluster is connected with Infiniband and you may get the best performance when using the login server or the compute nodes. It is currently limited to 30 TB.
If you want to use this space please create a subdirectory with you login name. All other files and directories will be delete without notice.
/home/work-itp (deprecated and replaced by /mnt/ceph)
This directory is globally accessible and can be used by everyone for minor important data which are not worth to be backed up or simply to large for your home. These files are not deleted. But, if the disk is full, we tell you to delete some files. It runs on a raid6 system, which means that a disk failure will not result in data loss. It is also served by a dedicated machine (scratch1) and heavy load does not affect the home directories. It is connected with Infiniband and you may get the best performance when using the login server or the compute nodes. It is currently limited to 40 TB.
/data (some hosts only)
This is the local disk, but not available on all machines. It has the same level like /tmp except that it isn't deleted automatically. Data on this partition is silently deleted when reinstalling or upgrading the machine.

Backup

Backups of the home directory and several other important system directories is done nightly.

  • There are held 3 generations of a file.
  • Overwritten files are stored for 90 days
  • A deleted file is stored for 90 days

If you have delete some files and want them restored, please contact the system administrator.

Retrieved from "https://itp.uni-frankfurt.de/wiki-it/index.php?title=Storage_and_Backup&oldid=1020"