1. Integrating LVM with Hadoop to provide Elasticity
2. Increase/Decrease Linux Static Partition
3. Automate LVM with python Script
4. Start Httpd and run Python script on Container

LVM:

Logical Volume Manager is a manager in linux based systems that manages Logical Volume which are dynamic volumes (partition)that can be extended or reduced according to the requirement.

Many use cases can be solved with lv’s.Logical volumes are created from volume groups which consists of physical volumes.

creating lv

• Mounting LV on datanode to achieve elasticity in hadoop:

here we just created the logical volume just like any other partition.Now to use it format and mount .And then use it as distributed dir for Hadoop

• Increase/Decrease Static Partition in Linux:

First the disk which we want to change the size should be attached to the OS.So check whether it is attached or not

Here the OS have a disk attached with 2 GB and having 3 partitions.The third partiton is mounted on a directory.

Decreasing the static partition size:

First unmount the disk partition and then check whether the partitiion is clean or not.Here the total /dev/sdc disk size is 2GB and no space is free to make another new partition.

So extending the /dev/sdc3 partition beyond the available size won’t possible except decreasing it.

umount <partition name>

e2fsck -f <partition name>

resize2fsck <partition name> <size>

mount <partition name> <directory>

Increasing static partition size:

Now after decreasing the /dev/sdc3 partition some space is available.Again it can be increased to previous size 500M (although some space is lost as compared to previous 499M to 478M)

• automating LVM with python script:

Requirements:

Subprocess module

Procedure: For listing the partitions subprocess module’s Popen object is used.The command used for listing ,creating, deleting the partitions is fdisk .It is an interactive command.A process is created with Popen object and all the options which we give to fdisk command normally ,are given to input stream.

x=sp.run([‘‘fdisk’’,’disk’],shell=True)

we can also use run method with the argument shell=True to run interactive cmnds in a terminal.

x=sp.Popen([‘fdisk’,’disk’],stdin=sp.PIPE,stdout=sp.PIPE,stderr=sp.STDOUT) x.stdin.write(b’p\n’)

x.stdin.flush()

y=x.stdout.readlines()

In the above lines,we are giving the input ,output stream to the process for taking fdisk options.

a typical output of fdisk<diskname>

The above shown command output may vary depending on the num of partitions already created.Here To print available partitions we can print from 13th line.To create new partition the inputs(options to fdisk ) we want to give were decided and stored in a list as
inp=[b’n\n’,b’\n’,bytes(size.encode()),b’w\n’]
The input list varies according to number of partitions already created.
And every input is given to byte stream input(stdin) hence represented as byte string.

creating an extended partition.

Here in case of 5th partition the fdisk command doesn’t prompt user to choose primary or extended.So the inputs to the process created by Popen object are decided based on the partitions created.

deleting a partition

LVM:

For creating lv,first create physical volume,volume group.These processes need not be interactive.So getoutput() method can be used with normal commands as arguments.
For reducing logical volume we have to first scan the file system which need an interactive terminal.
sp.call([‘e2fsck’,’-f’,lv])
can be used.

physical volume can be created as

pvcreate <disk or partition name>

creating a physical volume

creating a volume group:

extending Volume Group

Finally creating a logical volume which can be extended or reduced .

creating lv

Formatting and mounting a partition either it can be logical partition or static partition can be done in the same as

mkfs.<fstype> <partition>

mount <partition> <directory>

Above commands are not interactive except if we want to format a partition again .Formatting second time asks whether to wipe the prevoius fstype or not.In this case the getoutput() or Popen object is not asking the input.It directly wipes and formats with provided filesystem type.

here directory will be created automatically if it doesn’t created previously.

formating,mounting a partition and then extending the lv

Shrinking the Logical volume: For shrinking the volume ,first it should be unmounted .To help the user for entering the mounted directory, all the mounted vols are shown.And then the volume should be scanned for any bad blocks or bad mappings.This scanning needs interactive terminal.so here instead of getoutput or getstausoutput methods call () method is being used.It is like Popen() object.

sp.call([‘e2fsck’,’-f’,lv])

sp.getoutput(‘resize2fs ‘+lv+’ ‘+(actualsize-sizetoreduce))

x=sp.Popen([‘lvreduce’,lv,’ — size’,d],stdin=sp.PIPE,stdout=sp.PIPE)

x.communicate(input=b’y\n’) #input option yes.

And then the vol is resized to desired size before actual reducing.The lvreduce command is interactive.

• Configure HTTPD server on Container and Run Python script:

On centos container to install Httpd type

yum[dnf] install httpd

yum install python3

To start the Httpd service ,we normally use systemctl command.But in containers it is not available.Systemctl behind the scenes starts /usr/sbin/httpd process.So manually typing /usr/sbin/httpd will start the server.

To run python script first install python with yum/dnf

Source Code:

Tirumalaparise/arth-tasks

Thank you for reading :)