Loading...
 
Print

Unix Utils FAQ

How to checkout a project from CVS

Well first you need to have a login on the computer hostign the CVS repository. Say the computer is muscle and your account is girardot. Then do:

  • Set CVSROOT environment variable

setenv CVSROOT girardot@muscle:/home/cvsrep OR export CVSROOT=girardot@muscle:/home/cvsrep

  • Might also need to set CVS_RSH=ssh
  • Checkout a project by typing

cvs checkout project_name

  • You'll be prompted for your pwd and that's it

How do I redirect display from a remote server to my machine?

Before doing ssh -X to the remote server, make sure that:

  • DISPLAY is set (ie export DISPLAY=localhost:0.0)
  • access to to your machine is granted to the remote server, or more easily do a xhost +
  • then ssh -X ...

Note: you can also try ssh -Y in case ssh -X complains...

How do I use SVN?

  • To checkout a project (for example the furlongChIP project hosted on pc-furlong11):

> svn co svn:/pc-furlong11/furlongChIP This will prompt for a pwd that you must have on this repository...

  • To update a project

> svn up possibly followed by a path

  • To commit code

> svn ci -m vi possibly followed by a path

How do I dump and reload a mysql database?

  • To dump (with compression on the fly; bash syntax):
mysqldump --opt -u USER -p dbname | bzip2 -c 1>/path/to/dbname.sql.bz2 2>dump_db.log

This will prompt for your pwd. Two files are created: dbname.sql.bz2 and dump_db.log , the compressed dump and a log file to check errors out

Note : uncompress with bunzip2 dbname.sql.bz2

Note : to dump all database use --all-databases instead of dbname

  • To reload
bunzip2 dbname.sql.bz2 | mysql -u USER -p DBNAME or mysql -u USER -p DBNAME < dbname.sql

How can I create a encrypted password file?

By using the htpasswd tool.

htpasswd -c /path/to/passwd username password (to create the file and add user) htpasswd /path/to/passwd username password (to add user)

How to start a software at linux startup?

Use the tool 'chkconfig' (/sbin/chkconfig)

  • Create the startup file in '/etc/init.d' for the software that should be started
  • Make this file executable (e.g. chmod 755 ...)
  • Add this file to chkconfig with ''chkconfig --add (as root)
  • Set the levels, e.g. 'chkconfig --level 2345 on' to set the runtime level 2, 3, 4 and 5 to 'on'. Get a more detailed documentation on the levels here

How do I bulk change file extension of many files at once?

Use:

ls *.fa |awk ' {s=substr($0, 1, length($0) - 3); system("mv " s ".fa " s ".pssm")}'

In this example, we changed all .fa files to .pssm files.

IMPORTANT: the value "3" in ... length($0) - 3) ... MUST be adapted to your case i.e.this number is extension length + 1 e.g. 'fa' => 2+1=3

How do I bulk rename some files in bash?

You can use several tools to accomplish the same. Here is an example using sed, but you can probably easily replace "sed -re" with "perl -pe" (and \1 with $1)

 

for BAM in `ls Filtered_BAM_*_H3K27me*.bam`; do 
   NEW=$(echo "$BAM" |  sed -re 's@Filtered_BAM_(.+)\.bam@\1-nodups-xs-filtered-sorted.bam@g'); 
   mv $BAM $NEW; 
   samtools index $NEW; 
   samtools flagstat $NEW > $NEW".stats"; 
done

# The main part is this:
echo "$BAM" |  sed -re 's@Filtered_BAM_(.+)\.bam@\1-nodups-xs-filtered-sorted.bam@g'
# Which captures parts of the string by (.+) and then put back in the string using \1 .
# This uses the alternative separator '@' instead of '/', this can be benificial if you try to replace for example part of a directory structure that contains '/'s

 

How to setup a linux machine with nfs, nis and automounter?

Well I don't know either, but you can use this tutorial made by smart people from Russel group:

http://www.russelllab.org/private/wiki/index.php/Desktop_Setup

How to add user to sudoers list in Ubuntu?

Every user in the group admin is also a sudoer. This means you just add the user to the admin group by saying:

sudo adduser admin

Where is all the memory on the server?

You are wondering why the free memory is close to 0 on your 60 Gb RAM server while nobody is logged in, no big processes seems to run according to top?

Well, check the cached column (using top or free -m), if all memory is in there, it is simply that the server is using it for IO caching (for better performance). No worries, this cached memory is considered as free i.e. it is given back to jobs if needed.

How to install a software package in SEPP

See here.

How to check network IO on a slow server?

A reply from M. Wahlers about how to find processes causing huge peak of IO traffic.

  • run 'sudo iftop'

this will (hopefully) show the hosts to which all the network bandwidth goes

  • Then, simultaneously run 'netstat -tap' and search for the hosts causing most

of the network use found above. This will give you the PID of the rogue process.

This is all assuming that there is only a few processes running wild...

Also, IO monitoring can be done with : 'dstat --top-io' and 'nmon'

How to use passwordless SSH login?

This simple tutorial explains you how to save be able to login to an ssh session with a passwordless ssh key using one simple alias. Especially useful for automated tasks that need to login to other servers. But, as this has some security implications, you might want to limit this usage to internal use only (see also this post on serverfault). Another option is to make use of keychains that manages your password protected keys - see for example, the help on github.com.

- Open the terminal and edit with your favourite text editor (vim in this case) the file .bashrc:

vim ~/.bashrc

- Add the new alias to the file (obviously change “user” with your SSH username and “host.com” with your SSH address host):

alias s='ssh -2 -p 22 user@host.com'

- This means that every time you will type “s” in your terminal, that line will be executed. You can change the letter “s” to another one if you prefer.

The “-2″ option is used to force SSH2 connections instead of simple SSH.

The “-p 22″ option is used to connect via SSH to the port 22 (usually the default port).

Run these commands one time only (you can copy and paste each one) in the following order; remember to change “user” and “host” with your data:

generate pub and priv keys, leave the passphrase empty
(simply press ENTER when asked for it)

ssh-keygen

copy the pub key to the remote computer
(change port number if different from the usual 22)
change "user" to your user name
change "host" to your domain name

scp -P 22 ~/.ssh/id_rsa.pub user@host:~/

log on to the remote computer

ssh -p 22 user@host

create the .ssh directory in the root login directory, if it doesn't already exist

mkdir .ssh

append key to file

cat id_rsa.pub >> ~/.ssh/authorized_keys

delete the public key file, no longer needed

rm -f id_rsa.pub

log off the remote server

exit

logon to the remote server, without password prompt

ssh -2 -p 22 user@host

- That’s it! Simply type “s” in your terminal and this will instantly launch the ssh connection without asking your password anymore!

Taken from http://www.micheledallatorre.it/blog/2009/01/10/how-to-save-ssh-passwords-in-terminal-on-linux-ubuntu/

How to pass C FLAGS to R CMD INSTALL?

like in this example: MAKEFLAGS='LDFLAGS=-L/tmp/boost_1_52_0/stage/lib CPPFLAGS=-I/tmp/boost_1_52_0/' R-2.15.2 CMD INSTALL spp_1.11.tar.gz

Note that simple quotes must be used and spaces within a flag definition would need to be escaped using a single backslash

Mysql gives me an 'Table already exists' error, when in fact it does not exist. What can I do?

You have a broken table here, try:

sudo mysqladmin flush-tables

How to remount a file system?

To remount a fileserver e.g. /g/steinmetz you need to

  • umount /g/steinmetz
  • mount /g/steinmetz

If the share cannot be unmounted, you can check who is still connecting with

  • lsof | grep '/g/steinmetz'

If some people use the problematic share, you can also try this :

fuser -mkv /g/steinmetz loop until all processes gone… umount -l /g/steinmetz

Mysql gives me a warning when using a password in command line. How to avoid that?

Since mysql 5.6 we can use mysql_config_editor to add configs for mysql.

mysql_config_editor set --login-path=gcbridge --host=localhost --user=galaxy --password

This will prompt you for the password.

To see all available configs, do: mysql_config_editor print --all

This will print something like:

gcbridge

user = galaxy

password = *****

host = localhost

Then in your script instead of: mysql -u galaxy -p pass -e "sql statement"

you can use: mysql --login-path=gcbridge -e "sql statement"

Can I use gcc 4.7.* or 4.8.* on schroedinger/spinoza?

Yes, these versions have been installed under /opt and can be activated using :

> scl enable devtoolset-1.1 bash ==> will activate gcc (GCC) 4.7.2 20121015 (Red Hat 4.7.2-5)

> scl enable devtoolset-2 bash ==> will activate gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)

Note that these were installed following this post

> cd /etc/yum.repos.d

> wget http://people.centos.org/tru/devtools-1.1/devtools-1.1.repo

> yum enablerepo=testing-1.1-devtools-6 install devtoolset-1.1-gcc devtoolset-1.1-gcc-c++

and

> wget http://people.centos.org/tru/devtools-2/devtools-2.repo

> yum enablerepo=testing-devtools-2-centos-6 install devtoolset-2-gcc devtoolset-2-gcc-c++

How to start a RStudio Server session with environment variables globally set

If you modify the file "/usr/lib/rstudio-server/R/ServerOptions.R" and add e.g.

> Sys.setenv("PATH"="/usr/local/texlive/2014/bin/x86_64-linux:/sbin:/usr/sbin:/bin:/usr/bin")

as we did it for spinoza and schroedinger to load the 2014 version of texlive.

Input/Output error? Check your quota

If you are working in your home e.g. /home/ and you get random input/output errors it might be that you exceed your quota.

To see if this is the case run "quota -v -s -f /home", you can ofcourse easily make this into an alias like "alias myquota='quota -v -s -f /home'"

 

Rename many files using a regex

Using sed is very powerful e.g.

remove all numbers and first underscore from file name
ls | sed -rn "s/^[0-9]+_(.*)/mv '&' '\1'/ p" | sh

-n : quiet mode
-r : extended regular expressions
^0-9+_ : start with some digits, followed by an underscore
(.*) : what we will keep
mv ‘&’ ‘\1’ : what we will do : move the matching files (‘&’) to what we will keep (‘\1’)
p : print the command
|sh : execute the command

  •  first execute without |sh to first see what would happen  
  • mix in text in the  ‘\1’  like 'prefix\1suffix' or use more regex captures