Change default Image and Container location in Docker [CentOS 7]

When you start trying out docker it’s very normal that we don’t care about the default storage directory that docker will use to store images and containers. You may have to hit the panic button when Docker starts to occupy an enormous amount of space as you do more experiments with Docker. So now is the time to put troubleshooting cap on to figure out how to change the default location of docker. After wading through a number of sites and forums, I couldn’t figure out the steps to change the default directory of Docker images and containers on CentOS 7 host. This post provides instructions specific to CentOS 7. (It should work on RHEL 7 as well). For Debian, I could find a lot of documentation and “How to” articles posted on the Internet but there was not many for CentOS 7.  

Before attempting to change the default storage location of Docker, we must obtain some important information,

  • Default storage location used by Docker
  • Storage driver used by Docker
  • New storage space where the containers and images are going to reside

The default location of Docker is /var/lib/docker all existing images and containers are stored here. If you have any containers running, stop all and make sure no containers are running and then run the following command to determine the storage driver used by Docker. (You will find out later in the post why you want to make a note of it)

# docker info

In the output, look for Storage Driver and make a note of it. In my host it is devicemapper. Next step is to stop Docker service.

# sudo systemctl stop docker

Creating a Drop-In file

Next step is to create a Drop-In file “docker.conf” at  /etc/systemd/system/docker.service.d by default, docker.service.d folder will not be present. So you will have to create it. 

# sudo mkdir /etc/systemd/system/docker.service.d
# sudo touch /etc/systemd/system/docker.service.d/docker.conf 

The reason to create Drop-In file is that we want Docker service to override specific parameters mentioned in docker.conf file with that of the default service file located at /lib/systemd/system/docker.service. If you want to dive deep into Drop-In, read system.unit documentation.

Define the new storage location

Now open docker.conf and add the following, 

# sudo vi /etc/systemd/system/docker.service.d/docker.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --graph="/mnt/new_volume" --storage-driver=devicemapper 
Save and exit VI editor, new_volume is the new storage location and devicemapper is the storage driver. If your storage driver is different, type the value you have noted down earlier. More detailed information on various storage drives is available in Docker official documentationNow you can reload service daemon and start docker service. This will change the storage location for new images and containers.  
# sudo systemctl daemon-reload 
# sudo systemctl start docker

To confirm if all went well,  run # docker info to check the Docker root directory. It will be changed to /mnt/new_volume


What to do if you have existing containers and images?

If you want the existing containers and images to be migrated to the new location, don’t reload the service daemon and start the docker service, right after modifying docker.conf, move the existing data in /var/lib/docker to the new location. And create a symlink. 

Continue reading “Change default Image and Container location in Docker [CentOS 7]”

Beginners guide to OpenStack

For IT, cloud is a magic because cloud computing transforms how IT has been done for decades. If cloud is magic then what is the magic wand which transforms the data center to something magical?

Resources such as compute, storage and network is usually managed separately. When need arises resources are added manually. This is how computing had been done for decades. We humans can ever settle with anything. Computing doesn’t escape from that notion of human beings. We always try to enhance something which is taking a lot of manual tasks or create something new as the need grows. The need that we are talking about here is the growth of not just mobile apps, the reason for the growth of cloud computing is; people think it is efficient and not so time consuming to do stuff. Cloud computing enables rapid development and deployment model. It reduces the time that is required in each stage of development. Most importantly its self service model attracts many IT organizations to transform their IT.

OpenStack is that magic wand.

Over the past few years OpenStack had gained popularity and it is being widely adopted. Companies like Intel, Walmart and many others are transforming their entire IT infrastructure using OpenStack platform.

What is OpenStack?

OpenStack is a set of tools and services that can be used to build a cloud computing platform. A common myth is that, people think of it as a replacement to popular hypervisor’s. This is not true. OpenStack uses the hardware resources found in data centers such as Storage, Compute and Networking to create a service model. Once the hardware resources are abstracted, OpenStack presents these resources to users as services in various forms – like Infrastructure as a Service (Iaas), Platform as a Service (PaaS), Software as a Service (SaaS). OpenStack platform is suitable for any deployment model (Public, Private, Hybrid and Community Cloud). Consider reading Introduction to Cloud Computing if you want to understand different cloud computing service and deployment model.

openstack-software-diagram
High level architecture of OpenStack

Each year, OpenStack foundation – which is the control body for OpenStack development – releases two major versions of OpenStack. Based on these versions organizations like Red Hat, SUSE, Oracle, SwiftStack, Ubuntu, Rackspace, Mirantis, VMware etc. create their own distribution by customizing services and packaging it in different ways. Some of these distributions eliminate the complexity of deployment. Like wise each of them have their own pros and cons.

Building blocks of OpenStack

OpenStack is not packaged as a single software which can be deployed within few clicks. It is a collection of services which are inter connected together. Within OpenStack development community these service are developed as a project. Each service has its own API, using which they communicate with one another. For example, Compute service (Nova) creates instances (VMs) and manage the instance resources that are allocated to it. Storage services such as Cinder (Block) and Swift (Object) provides storage access to instances. Similarly there are different services available. Following are some of important services in OpenStack,

Service Project name
Dashboard Horizon
Compute Nova
Networking Neutron
Object Storage Swift
Block Storage Cinder
Identity Service Keystone
Image Service Glance
Telemetry Ceilometer
Orchestration Heat

Dashboard

The dashboard service provides centralized view of cloud environment to user as well as cloud administrator. Using dashboard a tenant (User) can self-provision resources, create/destroy Instances; modify networking for instances etc. Cloud administrator also interacts with dashboard and has more control over entire cloud environment.

Compute

Compute services manage Instances life cycle meaning, compute services takes care of Instances form the time it is created until it is destroyed. Compute services does not function alone it needs a Hypervisor to run instances therefore Instance related tasks such as CPU, Memory allocation is taken care by Hypervisor in its own way. Compute services just manages these instances and monitors them. When an instance creation command is received it’s passed on to Hypervisor to execute the task.

Networking

As the name denotes networking service provides network connectivity for OpenStack services. It allows uses to define network connectivity for instances that they own via dashboard. It also allows other network plugins such as VMware NSX, Open vSwitch, for better functionality. In any cloud environment Networking is the most complex part.

Object Storage

Object storage is provided by this service. Objects are stored and retrieved via REST API (HTTP based). Because this API access it can be directly accessed by an application.

Block Storage

This service creates block storage devices that can be directly provisioned to an instance. This block volume can be used for database or any high speed data access needs.

Identity Service

Different services of OpenStack make use of Identity service to communicate with each other. Identity service is an authentication and authorization service.

Image Service

Image service functionality is to store and retrieve virtual machine disk images. Snapshot of an instance can be taken and it can be used as a template for new instances. Virtual machine disk image is a file in which the operating system is installed. Popular formats are VMDK, VDI, VHD, OVF, qcow2 etc.

Telemetry

This service monitors OpenStack cloud for usage information for metering and performance information for statistical purpose. However this services is not an out of the box billing solution.

Orchestration

This service provides template based orchestration for a cloud application. This service executes appropriate API calls to create/modify OpenStack resources.

OpenStack architecture

In OpenStack cloud, the physical machines are represented as Controller nodes, Compute nodes, Network nodes, Block storage nodes, and Object storage nodes. There can be one or many physical machines (clustered). Each type of node cluster has its own set of services running on them. Following figure is a simplistic view of OpenStack infrastructure; the arrow represents its scale-out nature.

Types of NodesController Nodes

Controller nodes runs core services such as Dashboard, Image, identity service and also supporting services like SQL Database service, Message queue, Network time protocol, Compute management service, Networking ML2 Plugin, etc.

Compute Nodes

Compute nodes runs Compute service. Another important component of compute node is the Hypervisor. KVM is the default hypervisor support for OpenStack but there are a number of other hypervisors such as ESXi, XenServer, Hyper-V, Docker, etc. Importantly, compute node also runs networking modular layer 2 (ML2) plugin. This is for virtual network support for Instances.

Please note that what we discuss here applies to KVM as hypervisor. Implementation method varies for a few other hypervisors, i.e. ESXi. If OpenStack is implemented on top of ESXi infrastructure we normally interact directly with vCenter.

The compute service (Nova) running in compute nodes interacts with KVM and acts as a control element. KVM takes jobs from Nova service for instance creation/deletion/modification.

Network Nodes

Networking node runs tenant networking services which provides functionalities such as switching, routing, network address translation (NAT), and Dynamic Host Configuaration Protocol (DHCP). OpenStack networking is called “Neutron”.

When compared with its predecessor – Nova networking – Neutron release supports three tier architecture and provides functionalities such as load balancing, VPN, and firewall. These services are provided to tenant and can be individually charged. It also enhances security. Unlike Nova networking, Neutron allows usage of plugins. Open vSwitch, VMware NSX plugin, and many other plugins can be used with Neutron. Internet connectivity for tenant virtual machines is provided by network nodes.

Block and Object storage nodes

These nodes provide block and object storage. They are standard x86 servers with a bunch of drives from which storage space for instances are carved out. The storage space from these nodes is used for various other purposes such as backup, block volumes, etc.

Variations in architecture

When implementing OpenStack cloud, one does not necessarily need to follow the discussed architecture. It is possible to run storage-related services on controller and compute nodes. However, it is not the recommended way. Various vendors have released their own distribution of OpenStack which is production ready. There are also appliances available to kick start cloud as fast as you can.

Conclusion

We merely scratched the surface of OpenStack on what it is. For detailed documentation head over to OpenStack.org documentation center. I hope the information presented here helped you to understand what OpenStack is and how it used. Please feel free to comment if you have any questions.

Introduction to Cloud Computing

Due to the increase of mobile phones and other Internet connected devices cloud based services are emerging. Cloud Computing is not going to replace the data center. So what really is the need for cloud computing and why should a IT organization take the road to cloud. This post explains these questions.

Cloud computing is not simply a connection to something over the internet. Instead, it is re-imagination of a traditional data center. Data center components like Servers, Storage and Networking are managed separately. Cloud Computing enables efficient use of resources in a data center by sharing it with different instances of compute. It also enables users to provision the resources themselves. They will be able to easily scale up or scale down the resources which they need on demand. Cloud resources are elastic; depending on the load to a particular instance, compute resources like RAM, CPU can be added on demand so that the instance does not crash. National Institutes of Standards and Technology (NIST) defines what cloud computing is and what are the cloud computing models.

NIST Definition of Cloud Computing

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models1.

Image source: NIST

Any cloud environment should exhibit essential characteristics because that is the key distinguishing factor from a traditional IT environment where most of the tasks are done manually. This post explains essential characteristics of cloud, service models and deployment models.

Essential characteristics of Cloud

The essential characteristics are On-demand self-service, broad network access, resource pooling, rapid elasticity and measured service.

On-demand self-service – A user can provision resources for his/her needs such as more compute power, additional network bandwidth, more storage etc.
Broad network access – Cloud resources of a user should be accessible over network through standard mechanisms that may allow usage of heterogeneous equipment’s such as laptops, mobile, tablets etc.
Resource Pooling – The user will not have any control over where his/her compute resources reside at a physical scale. Instead in a cloud model the underlying physical resources are pooled. When a user creates an instance the resource is automatically taken from pool. When user destroys instance the resources used by it will be released back to pool.
Rapid Elasticity – This means on demand resource allocation which can scale out and also inward.
Measured service – In cloud environment all the resources usage can be metered. This enables user to pay for what is used or a user can prepay for resource usage.

Service Models

Service models must be well known to everyone. They are Software as a Service (SaaS), Platform as a Service (PaaS) and Infrastructure as a Service (IaaS).

Software as a Service (SaaS) – User can use the application which is preconfigured with little or no configuration change. User may have ability to configure the application according to one’s need, but cannot modify/control underlying components such as database, operating system etc.
Platform as a Service (PaaS) – In this service model user has the ability to deploy self-written application or licensed application on to cloud. User may control the environment which powers the application such as databases, but user does not have control over operating system, network, storage etc.
Infrastructure as a Service (IaaS) – User has access to fundamental computing resources. By making use of resources a user can create fully functional Instance. A user can provision required storage, network bandwidth, processing power etc. This allows users to run operating system and tools of their own choice.

Deployment Models

The deployment models are Private cloud, Community cloud, Public cloud, Hybrid cloud and Public cloud.

Private Cloud2 The cloud infrastructure is provisioned for exclusive use by a single organization comprising multiple consumers (e.g., business units). It may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises.
Community Cloud2 The cloud infrastructure is provisioned for exclusive use by a specific community of consumers from organizations that have shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be owned, managed, and operated by one or more of the organizations in the community, a third party, or some combination of them, and it may exist on or off premises.
Public Cloud2 The cloud infrastructure is provisioned for open use by the general public. It may be owned, managed, and operated by a business, academic, or government organization, or some combination of them. It exists on the premises of the cloud provider.
Hybrid Cloud2  The cloud infrastructure is a composition of two or more distinct cloud infrastructures (private, community, or public) that remain unique entities, but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load balancing between clouds).

The above discussed are the cloud deployment models and its fundamental characteristics. When a deployment model is selected the IT organization may decide on how to approach in case of private and hybrid cloud. There are two types of deployment options; Greenfield and Brownfield.

Greenfield deployment option

Greenfield deployment is nothing but building everything from scratch. The IT organization must procure all IT equipment in case of a Greenfield approach. Consider someone moving from public cloud to private. In this case, the IT organization must set up their data center.

Brownfield deployment option

If an IT organization decides to reuse their existing hardware (IT equipments in general) to transform into cloud infrastructure, it means their approach is Brownfield.

The Need

Road_To_CloudAmazon Web Services is a public cloud service provider. The services it provides can be accessed over the internet by a subscriber of AWS. IT organization will use private cloud for their own internal purpose. For example, Intel has built their private cloud using OpenStack; using which their developers and other users subscribe for resources which they need on demand. Cloud Computing bridges the gap between various departments in IT. Usually, each resource i.e. servers, networks, storage are managed separately by different teams. When transforming the data center to a cloud-based approach; silos that are usually present in any data center is removed. Therefore all the resources can be managed by a single governing body. When an IT organization decides to hit the road which takes them to the cloud, several questions must be asked. Following are few,

  • Will Cloud Computing suit their applications and services?
  • Will the new approach enhance their internal customers experience?
  • What impact is it going to make on the product or service which the company is ultimately selling? i.e. will it be helpful for the developers or the users to opt for self-service based approach?
  • Should they opt for public cloud instead of building their own?
  • Total cost involved in public cloud Vs private cloud
  • Total cost of ownership
  • Return of investment
  • How can they address security without any compromise?
  • Greenfield or Brownfield?
  • Does the IT organization require any organizational changes, such as hiring new talent?
  • What contracts and service-level agreements can be made with cloud service provider?
  • Does the move to the cloud include any purchase of new software? If so, how is it licensed?
  • How can the existing applications be migrated to the new cloud?

etc.

Conclusion

Most things in today’s world moves because of the need which arises

Organizations will see the true potential of cloud computing only if their need suits with what cloud computing may offer. Otherwise, there is no need for the organization to adopt cloud. Instead, they can focus on making their existing data center to a more converged and software-defined approach. Consider reading Top 5 storage and data center tech prediction for 2016 to understand why cloud is not the best approach for some IT organizations.

1NIST definition of Cloud Computing

2NIST definition of private, public, community, hybrid cloud

Neturon communication failure – Unable to establish connection to http://controller-node:9696/v2.0/extensions.json

After configuring neutron service i tried to verify its operation using command #neutron ext-list the following error encountered. This is not just specific to neutron alone. This error can occur while configuring services like cinder. This post discusses what caused this error and how it can be rectified.

Error : Unable to establish connection to http://controller-node:9696/v2.0/extensions.json

In my environment “controlle-node” is the host name of OpenStack controller node. The first place to look at is neutron server log located at /var/log/neutron/server.log It may help you to identify where it is failing. I verified recently configured /etc/neutron/neutron.conf file for any typos or wrong configuration. Nothing seems to be wrong. I also verified nova.conf and ml2_conf.ini files. All the configuration is apt and done as per the installation manual.

The cloud that I am building is a very small setup so i tried to uninstall neutron completely and reconfigure it from scratch to save some time. But it was not really a good choice to make. After the purge operation nova.conf file is lost. I quickly re configured nova (i had a backup of nova.conf) and verified its successful operation. And then I re configured neutron. But this time while verifying neutron service a different error interrupts the operation.

Error : Unauthorized (HTTP 401) (Request-ID: req-b92da632-9e08-4e7a-a8cf-51df8ed9ec28)

Checked /var/log/neutron/server.log following are the events that happened after executing #neutron ext-list

2015-02-15 10:46:32.945 20602 INFO neutron.wsgi [-] (20602) accepted ('x.x.x.x', 54745)
2015-02-15 10:46:32.947 20602 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): controller-node
2015-02-15 10:46:32.973 20602 WARNING keystonemiddleware.auth_token [-] Unexpected response from keystone service: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2015-02-15 10:46:32.973 20602 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-02-15 10:46:32.973 20602 INFO keystonemiddleware.auth_token [-] Invalid user token - rejecting request
2015-02-15 10:46:32.978 20602 INFO neutron.wsgi [-] 10.30.109.148 - - [15/Feb/2015 10:46:32] "GET /v2.0/extensions.json HTTP/1.1" 401 268 0.032020

Now its very evident that the keystone service is not accepting the credentials. If its an authentication error verify the password that is set for the neutron user account residing in keystone database. The next option is to verify neutron.conf in network node. In my case while re-configuring neutron password for neutron user in /etc/neutron/neutron.conf is incorrect. After changing the password keystone accepts authentication and neutron service is running successfully.

Please feel free to comment if the error still persists.