Tuesday, 19 August 2014

Storage pools

Storage pools and volumes are not required for the proper operation of guest virtual machines.
Pools and volumes provide a way for libvirt to ensure that a particular piece of storage will be available for a guest virtual machine, but some administrators will prefer to manage their own storage and guest virtual machines will operate properly without any pools or volumes defined.

NFS storage pool
Suppose a storage administrator responsible for an NFS server creates a share to store guest virtual machines' data. T he system administrator defines a pool on the host physical machine with the details of the share (nfs.example.com:/path/to/share should be mounted on /vm _data). When
the pool is started, libvirt mounts the share on the specified directory, just as if the system administrator logged in and executed m ount nfs.exam ple.com :/path/to/share /vm data. If the pool is configured to autostart, libvirt ensures that the NFS share is mounted on the directory specified when libvirt is started.
Once the pool starts, the files that the NFS share, are reported as volumes, and the storage volumes' paths are then queried using the libvirt APIs. T he volumes' paths can then be copied into the section of a guest virtual machine's XML definition file describing the source storage for the guest virtual machine's block devices. With NFS, applications using the libvirt APIs can create and delete volumes in the pool (files within the NFS share) up to the limit of the size of the pool (the maximum storage capacity of the share).

Storage Pools

A storage pool is a file, directory, or storage device managed by libvirt for the purpose of providing storage to guest virtual machines. T he storage pool can be local or it can be shared over a network. A storage pool is a quantity of storage set aside by an administrator, often a dedicated storage administrator, for use by guest virtual machines. Storage pools are divided into storage volumes either
by the storage administrator or the system administrator, and the volumes are assigned to guest virtual machines as block devices. In short storage volumes are to partitions what storage pools are to disks.
Although the storage pool is a virtual container it is limited by two factors: maximum size allowed to it by qemu-kvm and the size of the disk on the host physical machine. Storage pools may not exceed the size of the disk on the host physical machine. T he maximum sizes are as follows:
virtio-blk = 2^63 bytes or 8 Exabytes(using raw files or disk)
Ext4 = ~ 16 T B (using 4 KB block size)
XFS = ~8 Exabytes
qcow2 and host file systems keep their own metadata and scalability should be evaluated/tuned when trying very large image sizes. Using raw disks means fewer layers that could affect scalability or max size.
libvirt uses a directory-based storage pool, the /var/lib/libvirt/images/ directory, as the default
storage pool. T he default storage pool can be changed to another storage pool.
Local storage pools - Local storage pools are directly attached to the host physical machine server. Local storage pools include: local directories, directly attached disks, physical partitions, and LVM volume groups. T hese storage volumes store guest virtual machine images or are attached to
guest virtual machines as additional storage. As local storage pools are directly attached to the host physical machine server, they are useful for development, testing and small deployments that do not  require migration or large numbers of guest virtual machines. Local storage pools are not suitable for many production environments as local storage pools do not support live migration.
Networked (shared) storage pools - Networked storage pools include storage devices shared over a network using standard protocols. Networked storage is required when migrating virtual machines between host physical machines with virt-manager, but is optional when migrating with virsh. Networked storage pools are managed by libvirt. Supported protocols for networked storage
pools include:
Fibre Channel-based LUNs
iSCSI
NFS
GFS2
SCSI RDMA protocols (SCSI RCP), the block export protocol used in InfiniBand and 10GbE iWARP adapters.

Memory overcommiting process

Guest virtual machines running on a KVM hypervisor do not have dedicated blocks of physical RAM assigned to them. Instead, each guest virtual machine functions as a Linux process where the host physical machine's Linux kernel allocates memory only when requested. In addition the host physical
machine's memory manager can move the guest virtual machine's memory between its own physical memory and swap space. T his is why overcommitting requires allotting sufficient swap space on the host physical machine to accommodate all guest virtual machines as well as enought memory for the
host physical machine's processes. As a basic rule, the host physical machine's operating system requires a maximum of 4GB of memory along with a minimum of 4GB of swap space.




T his example demonstrates how to calculate swap space for overcommitting. Although it may appear to be simple in nature, the ramifications of overcommitting should not be ignored. Refer to Important before proceeding.

ExampleServer1 has 32GB of physical RAM. T he system is being configured to run 50 guest virtual machines, each requiring 1GB of virtualized memory. As mentioned above, the host physical machine's system itself needs a maximum of 4GB (apart from the guest virtual machines) as well as an additional 4GB as a swap space minimum.
The swap space is calculated as follows:
Calculate the amount of memory needed for the sum of all the guest virtual machines - In this example: (50 guest virtual machines * 1GB of memory per guest virtual machine) = 50GB
Add the guest virtual machine's memory amount to the amount needed for the host physical machine's OS and for the host physical machine's minimum swap space - In this example: 50GB
guest virtual machine memory + 4GB host physical machine's OS + 4GB minimal swap = 58GB
Subtract this amount from the amount of physical RAM there is on the system - In this example
58GB - 32GB = 26GB
T he answer is the amount of swap space that needs to be allocated in Host.

Live migration Backend process

In a live migration, the guest virtual machine continues to run on the source host physical machine while its memory pages are transferred, in order, to the destination host physical machine. During migration,KVM monitors the source for any changes in pages it has already transferred, and begins to transfer
these changes when all of the initial pages have been transferred. KVM also estimates transfer speed during migration, so when the remaining amount of data to transfer will take a certain configurable period of time (10ms by default), KVM suspends the original guest virtual machine, transfers the remaining
data, and resumes the same guest virtual machine on the destination host physical machine.

Wednesday, 13 August 2014

Metadata

what the metadata is for. Your single file is split up into a bunch of small pieces and spread out of geographic location, servers, and hard drives. These small pieces also contain more data, they contain parity information for the other pieces of data, or maybe even outright duplication.
The metadata is used to locate every piece of data for that file over different geographic locations, data centres, servers and hard drives as well as being used to restore any destroyed pieces from hardware failure. It does this automatically. It will even fluidly move these pieces around to have a better spread. It will even recreate a piece that is gone and store it on a new good hard drive.

Friday, 8 August 2014

filesystem making process

mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2) -->Default block size 4KB.
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3276800 inodes, 13107200 blocks --> no of inodes & blocks created under that partition.
655360 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Wednesday, 6 August 2014

Cloud Computing

Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort.

Today, it is more or less accepted that there are three Cloud Computing models depending on the type of service provided, IaaS, Infrastructure as a Service, PaaS, Platform as a Service, and SaaS, Software as a Service.


IaaS – Infrastructure as a Service

 Infrastructure as a Service provides infrastructure capabilities like processing, storage, networking, security, and other resources which allow consumers to deploy their applications and data. This is the lowest level provided by the Cloud Computing paradigm. Some examples of IaaS are: Amazon S3/EC2, Microsoft Windows Azure, and VMWare vCloud.

  PaaS – Platform as a Service

 Platform as a Service provides application infrastructure such as programming languages, database management systems, web servers, applications servers, etc. that allow applications to run. The consumer does not manage the underlying platform including, networking, operating system, storage, etc. Some examples of PaaS are: Google App Engine, Microsoft Azure Services Platform, and ORACLE/AWS.

  SaaS – Software as a Service

 Software as a Service is the most sophisticated model hiding all the underlying details of networking, storage, operating system, database management systems, application servers, etc. from the consumer. It provides the consumers end-user software applications most commonly through a web browser (but could also be though a rich client). Some examples of SaaS are: Salesforce CRM, Oracle CRM On Demand, Microsoft Online Services, and Google Apps.


  PaaS – Platform as a Service have:


AIaaS – Application Infrastructure as a Service
Some analysts consider this model to provide application middleware, including applications servers, ESB, and BPM (Business Process Management).
 APaaS – Application Platform as a Service
Provides application servers with added multitenant elasticity as a service. The model PaaS (Platform as a Service) mentioned before includes AIaaS and APaaS.

 SaaS – Software as a Service have:

BPaaS – Business Process as a Service
Provides business processes such as billing, contract management, payroll, HR, advertising, etc. as a service.

 IaaS – Infrastructure as a Service have:

DaaS – Desktop as a Service
Based on application streaming and virtualization technology, provides desktop standardization, pay-per-use, management, and security.
 CaaS – Communications as a Service
Management of hardware and software required for delivering voice over IP, instant messaging, video conferencing, for both fixed and mobile devices.
 NaaS – Network as a Service
It allows telecommunication operators to provide network communications, billing, and intelligent features as services to consumers.

Reference:
------------------
http://itechthoughts.wordpress.com/category/cloud-computing/
http://itechthoughts.wordpress.com/category/virtualization/


Tuesday, 5 August 2014

Linux-Filesystem

A filesystem is the methods and data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk. The word is also used to refer to a partition or disk that is used to store the files or the type of the filesystem. Thus, one might say I have two filesystems meaning one has two partitions on which one stores files, or that one is using the extended filesystem, meaning the type of the filesystem.

Before a partition or disk can be used as a filesystem, it needs to be initialized, and the bookkeeping data structures need to be written to the disk. This process is called making a filesystem.

Most UNIX filesystem types have a similar general structure, although the exact details vary quite a bit. The central concepts are superblock, inode, data block, directory block, and indirection block. The superblock contains information about the filesystem as a whole, such as its size (the exact information here depends on the filesystem). An inode contains all information about a file, except its name. The name is stored in the directory, together with the number of the inode. A directory entry consists of a filename and the number of the inode which represents the file. The inode contains the numbers of several data blocks, which are used to store the data in the file. There is space only for a few data block numbers in the inode, however, and if more are needed, more space for pointers to the data blocks is allocated dynamically. These dynamically allocated blocks are indirect blocks; the name indicates that in order to find the data block, one has to find its number in the indirect block first.

 Like UNIX, Linux chooses to have a single hierarchical directory structure. Everything starts from the root directory, represented by /, and then expands into sub-directories instead of having so-called 'drives'.

 In general, 'block devices' are devices that store or hold data, 'character devices' can be thought of as devices that transmit or transfer data. For example, diskette drives, hard drives and CD-ROM drives are all block devices while serial ports, mice and parallel printer ports are all character devices.

A few terms to understand:
tty: Video console terminal (abbreviation for “Teletype”)
ttyS: Serial console terminal
pts: Virtual console terminal (pseudo-tty or pty but stands for Pseudo-Terminal Slave (PTS))
[root@host ~]# w
16:58:50 up 19:22,  3 users,  load average: 0.81, 0.90, 0.71
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    client.domain. 16:57    0.00s  0.01s  0.00s w
root     ttyS0    –                16:58    3.00s  0.00s  0.00s -bash
root     tty1     –                16:56    1:24   0.02s  0.02s -bash
As you can see from this, we have 1 user logged in via a virtual console (such as SSH), defined as pts/0. Another user logged in via serial console, defined as ttyS0. And another user logged in via the video console port, defined as tty1.

/lost+found directory:
 ---------------------------------

 Linux should always go through a proper shutdown. Sometimes your system might crash or a power failure might take the machine down. Either way, at the next boot, a lengthy filesystem check (the speed of this check is dependent on the type of filesystem that you actually use. ie. ext3 is faster than ext2 because it is a journalled filesystem) using fsck will be done. Fsck will go through the system and try to recover any corrupt files that it finds. The result of this recovery operation will be placed in this directory. The files recovered are not likely to be complete or make much sense but there always is a chance that something worthwhile is recovered. Each partition has its own lost+found directory. If you find files in there, try to move them back to their original location.

Mounting:
----------------

 Before one can use a filesystem, it has to be mounted. The operating system then does various bookkeeping things to make sure that everything works. Since all files in UNIX are in a single directory tree, the mount operation will make it look like the contents of the new filesystem are the contents of an existing subdirectory in some already mounted filesystem.

/Proc:
----------

/proc is very special in that it is also a virtual filesystem. It's sometimes referred to as a process information pseudo-file system. It doesn't contain 'real' files but runtime system information (e.g. system memory, devices mounted, hardware configuration, etc).


/sbin should contain only binaries essential for booting, restoring,
  recovering, and/or repairing the system in addition to the binaries
  in /bin. 
 
 

Reference:

 http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/glossary.html