OSDI 2008

People in bay area -- see you there

Compiling kernel > 2.6.13

If you see anything like this --

modprobe: FATAL: could not load /lib/modules.dep " NO such file or directory
mount:unknown filesystem type 'devfs'
FATAL : could not load /lib/modules/2.6.14/modules.dep:No such file or
directory
umount: devfs: not mounted
/scripts/ext3-add-journal.sh:27:arith:syntax error : "0X"
modprobe:FATAL : could not load /lib/modules/2.6.14/modules.dep:No such
file or directory
mount: unknown filesystem type 'devfs'
/sbin/init:426:arith:syntax error : "0X"
kernel panic - not syncing:Attempted to kill init!


You are using mkinitrd for making initrd file. Use mkinitramfs

It works ! Make sure to * cramfs

Installing Hadoop on Ubuntu

Have a cluster of 4 machines and am set to install hadoop on it.
Machines:
ds16: master
ds11: slave
ds04: slave
ds14: slave


Am loosely following sangmi's blog and the following
http://sangpall.blogspot.com/2008/09/installing-hadoop.html
http://wiki.apache.org/hadoop/GettingStartedWithHadoop
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)

Step 1: chk if you have ssh, rsync and java on all machines
For java
sudo apt-get install sun-java5-jdk

Step 2:
Download hadoop and in conf/hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun

Step 3:
now you can just run your standalone operation as it is.
$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
$ cat output/*

[One problem faced was - when I did ssh to my machines, I get output

id: cannot find name for group ID 521

So I added a group with id 521 that solved the problem
sudo addgroup -gid 521 test

Also disabled stricthostchecking in /etc.ssh/ssh_config -- causing problem since ~ is mounted on nfs]
Till now we have run hadoop on single node. Lets move to a cluster now.

Cluster Operations:
Step 4:
Configure conf/hadoop-site.xml on the namenode as follows

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://ds16:54310</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://ds16:54311</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>8</value>
</property>
<property>
  <name>dfs.name.dir</name>
  <value>/export/pathaka/hadoop/dfs</value>
</property>
<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx512m</value>
</property>
</configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/export/pathaka/hadoop/tmp/</value>
  <description>A base for other temporary directories.</description>
</property>

Edit conf/masters and conf/slaves on ds16 as shown

$ cat conf/masters
ds16

$ cat conf/slaves 
ds16
ds04
ds11
ds14

[Used http://www.palfrader.org/code2html/code2html.html for converting code to html .. its nice]

[Make sure there are no leading spaces in the file -- gives a weird XML exception
[Fatal Error] hadoop-site.xml:2:6: The processing instruction target matching "[xX][mM][lL]" is not allowed.]

Step 5:

$mkdir /export/pathaka/hadoop/dfs
# This is the dfs file tree

$ mkdir /export/pathaka/hadoop/tmp

On master - ds16 run
$ bin/hadoop namenode -format
# This formats the dfs folder
#Do it only the first time and on master only


On ds16 - master, run
$ bin/start-dfs.sh
$ bin/start-mapred.sh
# master and jobtracker are the same machine

Stopping
$ bin/stop-mapred.sh
$ bin/stop-dfs.sh

PAM presentation at Cleveland OH

Presented a paper at cleveland OH last wee. Was fun driving up there and down here. Had a flat tire while return which was more like an adventure - trucks blazing at high speed past you and you are stranded on grass.

Slides would be posted soon.

http://pam2008.cs.wpi.edu/program.html

How travel grants make fun of poor grad students

We thank you again for applying to the XXXXXXXX student travel
grant program. This year the selection committee is formed from the
Sponsors. Due to the large number of applications and a limited budget, we
were able to fund only a fraction of the applicants.

We are pleased to inform you that after careful consideration of your
application, you have been selected by the committee to receive a
financial award from the XXXXXXXX'08 student travel grant program. Your
award will be in the form of:

* a free registration for the main conference + $100 of travel expense
reimbursement.

Paper in LEET 2008

Paper got in LEET 2008 -- Finally all the data collection was appreciated.

Check out the program here

See you at San Francisco ..

What to do

when your machine in lab is pinging, but not responding to ssh, you are at home and temperature is -5 Deg outside?