Elasticity providing to Hadoop Data Node using LVM and Increase and Decrease size of Static Partition

Objective:-
🔅Integrating LVM with Hadoop and
providing Elasticity to DataNode Storage
🔅Increase or Decrease the Size of Static
Partition in Linux.

First of all Lets take a look of some new words that we are going to use in this blog.

1. PV:- It is physical volume , which is regular storage device for os.
2. VG:- It is Volume group ,It is created when we combine multiple phsical volumes .
3. LVM:- It is most important thing in this blog .It is some type of mapping of storage . Within a volume group we can slice so many logical volumes and LVM is manager of all the logical volumes.

So lets start. As we want to create physical volume and other partitions so we need to add some extra hard disk to our data node . so lets do it

Select VM, click on settings , click on controller SATA then second + icon(Adds Hard disk)

Click on create and select VDI -> Next

Click on Ok and start VM

After starting of VM go to command line and check about added HD by command lsblk

Now create a Physical volume drive to add in VG. Use command

pvcreate /dev/sdb

and then check the created PV by Command

pvdisplay

Created PV and displayed

Now create a volume group and add created pv to it and display it

vgcreate amanHadoop /dev/sdb
vgdisplay

Now create Logical Volume as much you want and add with VG by command and display details

lvcreate — size 5G — name myamanHadoop_lvm amanHadoop

Now what you want , You need to mount this with out DataNode but before that we need to format this partition. So Format It and then Mount On /dn1

mkfs.ext4 /dev/amanHadoop/myamanHadoop_lvm
mount /dev/amanHadoop/myamanHadoop_lvm /dn1
df -l

Here it is shown That New partition has been created and mounted to Datanode

Now Go to either Namenode or here itself check cluster report but before that do remember to restart DataNode .to restart it use commands

hadoop-daemon.sh stop datanode
hadoop-daemon.sh start datanode

now check cluster report

here one 5GB partition is added

That’s How we add LVM to Hadoop Datanode

Now lets increase and decrease partition size in linux

Here is one command in linux using lvm by which we can directly increase or decrease size of partition and make that permanent .

so here is the command to extend partition size ,and make that permanent as we can’t use format operation as it will delete all the data here in the partition so we use resize2fs command

lvextend — size +2G /dev/amanHadoop/myamanHadoop_lvm
resize2fs /dev/amanHadoop/myamanHadoop_lvm

Now again go and check the status of hadoop cluster it will be increasing by 2Gb

To reduce size of partition we use command
lvreduce — size -2G /dev/amanHadoop/myamanHadoop_lvm

That’s all with this article. I hope you found the post Informative, if something was missing or you think some more things could have been added, feel free to provide suggestions in the comments section or on LinkedIn .

You can check out my LinkedIn profile.

Aman Goyal - Student Intern and Technical volunteer - LinuxWorld Informatics Pvt Ltd | LinkedIn

Thanks for giving your precious time to this article✌ Hope you like it .