|
Size: 253
Comment:
|
Size: 3677
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Some words on how to use MIB servers = | = Some words on how to work with MIB servers = |
| Line 4: | Line 4: |
| [[Computers description]] | 1. [[Computers description]] |
| Line 6: | Line 6: |
| [[Request an account]] | 2. [[Request an account]] |
| Line 8: | Line 8: |
| [[storage description]] | 3. [[storage description]] |
| Line 10: | Line 10: |
| [[First login]] | 4. [[First login]] |
| Line 12: | Line 12: |
| [[transfering data]] | 5. [[Receiving sequencing data]] |
| Line 14: | Line 14: |
| [[Submitting jobs]] | 6. [[transfering data]] |
| Line 16: | Line 16: |
| [[Check jobs]] | 7. [[Submitting jobs]] |
| Line 18: | Line 18: |
| [[Install tools]] | 8. [[Monitor jobs]] 9. [[Install tools]] = Space management = == Space usage == Every users should bear in mind that space management is tricky, on one hand it is hard to know how much is needed and on the other hand, it is never enough. The amount of space cannot be infinite (for the time being). However, with a good management it is possible to extend the life span of the available storage and make sure everyone can work. We distinguish 2 types of storage, the '''administrative''' storage and the '''working''' storage. === Administrative storage === The '''seq''' and '''home''' folders are administrative partitions and their access is limited. '''seq''' is where all the sequencing runs are saved. Only admins can modify the folders in this partition. Which means that only admins can save the sequencing run there. User can see all sequencing runs unless decided otherwise. In the '''home''' partition, users have access to their personal folders. '''These folders should not be used to store any sequencing runs or any output files related to sequencing run analysis or big files in general.''' Another administrative partition is '''tools''', this folder is only meant to host tools, no data and no databases. Users can install tools via 'conda'. Please address a message to admins to install a tool which is not present on conda. === Working storage === The working storage are '''projects''' and '''work''', these two folders are meant to store users projects files. To start working in the '''work''' partition, make a personal folder with the same name as your login name. By default, other users can see in that folder but not modify it, feel free to adjust the permissions on your folders. That personal folder will host your analyses. After running your analyses, all users are expected to clean their work space, remove any files/folders that are not useful for the future steps of analyses (intermediate files). After being done with all your analyses, delete any file/folder that is not worth keeping for publications and request a project folder to bioinfo.mib@wur.nl. For those who wants to keep intermediate output files until publication several options are available, * [[https://www.wur.nl/en/value-creation-cooperation/collaborating-with-wur-1/wdcc/research-data-management-wdcc/doing/yoda-at-wur.htm|WUR YODA]] offers to have some significant amount of space for free. * You can buy an extra hard drive of 2TB or 4TB * It is also possible to rent space from many providers, WUR, amazon,... '''projects''' partition is meant to host after publication files, the files you have to keep for 10 years and those folder should be kept clean. Make sure to compress '''all''' text files using either '''gzip''' or '''bzip2'''. Gzip is faster and more suitable for non redundant files, you can also use the multithreaded version [[https://linux.die.net/man/1/pigz|'''pigz''']] to speed up the compression. '''bzip2''' offers a better compression level and takes a bit more time, compression can be accelerated using the multithreaded version [[https://linux.die.net/man/1/pbzip2|'''pbzip2''']]. Use '''bzip2''' by default. DO NOT STORE RAW SEQUENCING FILES HERE! === Working storage === To ensure that everyone can How to work with databases = Working with sequence files = = Student files = = Leaving MIB = |
Some words on how to work with MIB servers
Contents
Space management
Space usage
Every users should bear in mind that space management is tricky, on one hand it is hard to know how much is needed and on the other hand, it is never enough. The amount of space cannot be infinite (for the time being). However, with a good management it is possible to extend the life span of the available storage and make sure everyone can work. We distinguish 2 types of storage, the administrative storage and the working storage.
Administrative storage
The seq and home folders are administrative partitions and their access is limited. seq is where all the sequencing runs are saved. Only admins can modify the folders in this partition. Which means that only admins can save the sequencing run there. User can see all sequencing runs unless decided otherwise. In the home partition, users have access to their personal folders. These folders should not be used to store any sequencing runs or any output files related to sequencing run analysis or big files in general. Another administrative partition is tools, this folder is only meant to host tools, no data and no databases. Users can install tools via 'conda'. Please address a message to admins to install a tool which is not present on conda.
Working storage
The working storage are projects and work, these two folders are meant to store users projects files. To start working in the work partition, make a personal folder with the same name as your login name. By default, other users can see in that folder but not modify it, feel free to adjust the permissions on your folders. That personal folder will host your analyses. After running your analyses, all users are expected to clean their work space, remove any files/folders that are not useful for the future steps of analyses (intermediate files). After being done with all your analyses, delete any file/folder that is not worth keeping for publications and request a project folder to bioinfo.mib@wur.nl. For those who wants to keep intermediate output files until publication several options are available,
WUR YODA offers to have some significant amount of space for free.
- You can buy an extra hard drive of 2TB or 4TB
- It is also possible to rent space from many providers, WUR, amazon,...
projects partition is meant to host after publication files, the files you have to keep for 10 years and those folder should be kept clean. Make sure to compress all text files using either gzip or bzip2. Gzip is faster and more suitable for non redundant files, you can also use the multithreaded version '''pigz''' to speed up the compression. bzip2 offers a better compression level and takes a bit more time, compression can be accelerated using the multithreaded version '''pbzip2'''. Use bzip2 by default. DO NOT STORE RAW SEQUENCING FILES HERE!
Working storage
To ensure that everyone can How to work with databases
Working with sequence files
Student files