NameNode1 and NameNode1 expose the same port？

[Buttonwood]

Hi, Jurriaan:

I'm trying to use your hadoop-ha-docker for building a hadoop cluster.

I first start 3 journalnodes: hadoop-jn1-3

VM_NAME=$1
HD_NAME="hd01-"${VM_NAME}
docker rm -f $HD_NAME
docker create --name $HD_NAME -v /mnt/hadoop/journal/data jurmous/hadoop:2.5 /bin/true

docker run -it -d \
--name=$VM_NAME \
--hostname=$VM_NAME \
--net=net04 \
--volumes-from $HD_NAME \
-e "NNODE1_IP=hadoop-nn1" \
-e "NNODE2_IP=hadoop-nn2" \
-e "JN_IPS=hadoop-jn1:8485,hadoop-jn2:8485,hadoop-jn3:8485" \
-e "ZK_IPS=zk01:2181,zk02:2181,zk03:2181,zk04:2181,zk05:2181" \
jurmous/hadoop:2.5 \
/etc/bootstrap.sh -d journalnode

then 3 datanodes: hadoop-nn1-3

VM_NAME=$1
HD_NAME="hd01-"${VM_NAME}
docker create --name $HD_NAME -v /mnt/hadoop/dfs/data  jurmous/hadoop:2.5 /bin/true

docker run -it -d \
--name=$VM_NAME \
--hostname=$VM_NAME \
--net=net04 \
--volumes-from $HD_NAME \
-e "NNODE1_IP=hadoop-nn1" \
-e "NNODE2_IP=hadoop-nn2" \
-e "JN_IPS=hadoop-jn1:8485,hadoop-jn2:8485,hadoop-jn3:8485" \
-e "ZK_IPS=zk01:2181,zk02:2181,zk03:2181,zk04:2181,zk05:2181" \
 jurmous/hadoop:2.5 \
/etc/bootstrap.sh -d datanode

then start hadoop-nn1

VM_NAME="hadoop-nn1"
HD_NAME="hd01-"${VM_NAME}
docker rm -f $HD_NAME
docker create --name $HD_NAME -v /mnt/hadoop/dfs/name  jurmous/hadoop:2.5 /bin/true

docker run -it -d \
--name=$VM_NAME \
--hostname=$VM_NAME \
--net=net04 \
--volumes-from $HD_NAME \
-p 50470:50470 -p 8020:8020 -p 50070:50070 \
-e "NNODE1_IP=hadoop-nn1" \
-e "NNODE2_IP=hadoop-nn2" \
-e "JN_IPS=hadoop-jn1:8485,hadoop-jn2:8485,hadoop-jn3:8485" \
-e "ZK_IPS=zk01:2181,zk02:2181,zk03:2181,zk04:2181,zk05:2181" \
 jurmous/hadoop:2.5 \
/etc/bootstrap.sh -d format

and hadoop-nn2

VM_NAME="hadoop-nn2"
HD_NAME="hd01-"${VM_NAME}
docker create --name $HD_NAME -v /mnt/hadoop/dfs/name  jurmous/hadoop:2.5 /bin/true

docker run -it -d \
--name=$VM_NAME \
--hostname=$VM_NAME \
--net=net04 \
--volumes-from $HD_NAME \
-p 50470:50470 -p 8020:8020 -p 50070:50070 \
-e "NNODE1_IP=hadoop-nn1" \
-e "NNODE2_IP=hadoop-nn2" \
-e "JN_IPS=hadoop-jn1:8485,hadoop-jn2:8485,hadoop-jn3:8485" \
-e "ZK_IPS=zk01:2181,zk02:2181,zk03:2181,zk04:2181,zk05:2181" \
 jurmous/hadoop:2.5 \
/etc/bootstrap.sh -d format

But here I got something wrong.

docker: Error response from daemon: container 462e9702e2687a71afdc433c6515787eedcce10be6669114feb1469b65d61366: endpoint create on GW Network failed: failed to create endpoint gateway_462e9702e268 on network docker_gwbridge: Bind for 0.0.0.0:50470 failed: port is already allocated.

It seems port 50470/8020/50070 is alreay being used by hadoop-nn1. Need I change them with hadoop-nn2?

[jurmous]

Well it seems you are trying them out on the same machine which is of course not how it should run in the end for reliability. I only tested the namenodes running on different machines.
If you want to make it run on one machine the hdfs-site.xml.template should be edited with different ports for namenode 2. It seems there are already different ports exposed for namenode 2 (50090 50495) although not yet for another rpc port. I am not currently running a setup and thus cannot test any changes. You are welcome to push a change to make it easier to run on a single machine.