| hadoop-master | hadoop-slave1 | hadoop-slave2 | |
|---|---|---|---|
| HDFS | NameNode SecondaryNameNode |
DataNode | DataNode |
| YARN | ResourceManager LogServer |
NodeManager | NodeManager |
-
Docker(my version is 20.10.17)
just make sure
docker composecan work -
Download hadoop-3.2.4.tar.gz and Put the file in the same level directory of Dockerfile
-
Clone the repository
-
Build your own images
docker compose build
-
Start docker
docker compose up -d
docker exec -it hadoop-master bash
./start-hadoop.sh
./run-wordcount.shIf successful, you will see
MapReduce UI: http://<HOST IP>:8080
HDFS UI: http://:9870
Log UI: http://:19888
-
Can I use it offline?
NO, the docker base image is ubuntu:18.04 that pulled from Internet，and use apt to install openssh, java8.
In addition, at Dockerfile line-9，you can replace default sources.list to speed up apt installation.
-
How to increase the number of DataNode/NodeManager?
-
edit config/works file. Each line represents a Node.
-
edit docker-compose.yml.
-
go to Quick start step-2 rebuild your images.
-