Archives

Categories

Used Car Quote

Used Car Buying Tips – info on getting your best deal Buying new cars, used car tips and resources for a great deal…

Uncategorized

How to enable Apache Hadoop in Cloud

0

Combining Big Data and Cloud Computing is a hot trend. Hadoop is another name to Big Data. It’s the right moment to understand how Apache Hadoop is enabled in Cloud Computing.
Hadoop is the foundation for many other big-data technologies and tools. Hadoop supports many types of workloads, which is crucial for industries that require large-scale scalability. With the exponential growth in big data, storage costs and maintainability are a major concern when considering the budgets of companies.
Are you a newbie looking to get a job in Big Data Hadoop? Check out our previous blog to learn how to start Learning Hadoop for Beginners.
Cloud computing is the latest trend in industry. Big data processing on cloud platforms can be especially useful for situations where an enterprise wants lower costs.
This blog will explain how to enable Apache Hadoop in a Cloud computing instance.
Important considerations to enable Apache Hadoop in Cloud Computing Environment
These points should be taken into consideration before you create an Apache Hadoop environment in a cloud:
Apache Hadoop cloud deployment is concerned about security in the public cloud. Every enterprise should evaluate security criteria before moving Hadoop cluster information, as Hadoop offers very limited security.
Apache Hadoop’s main purpose is data analysis. Apache Hadoop must be used in cloud computing deployments to support the Hadoop ecosystem’s tools, including data visualization and analytics.
Data transmission to the cloud is charged. It is important to consider the location from which data is being transferred to the cloud. The cost of loading data from an internal system that is not on the cloud, or if it is already in the cloud, will vary.
How to Configure Apache Hadoop Environments in the Cloud
You must have Linux platform installed on the cloud to begin Apache Hadoop cloud configuration. We will be discussing how to configure Apache Hadoop cloud computing using pseudo mode. Here, we have used AWS EC2 to represent the cloud environment.
Prerequisites for configuring Apache Hadoop environment on the cloud
AWS account in active status
Public and private keys available for the EC2 instance
Running Linux instance.
PuTTy was installed and set-up on the Linux
You want to improve your enterprise capabilities? Combine Big Data and Cloud Computing to create a powerful combination.
Next, you will need to execute two steps to enable Apache Hadoop within a Cloud computing environment.
Phase1: Connect to EC2 instance via PuTTy
Step 1: Create the private key for PuTTy
To connect EC2 instances to PuTTY in order to configure Apache Hadoop environment you will need a private key to PuTTy that supports AWS private key format (.pem). PuTTYyGen allows us to convert.pem files into the.ppk format, which is supported by PuTTY. Once the PuTTy private keys have been generated, we can connect to the EC2 instance using SSH client.
Step 2: Start PuTTy session to create an EC2 instance
After you have started your PuTTy session to connect to EC2, you will need to authenticate the connection. In the category panel, select connection and SSH. Then expand it. Next, choose Auth. Browse for the.ppk file and open it.
Step 3: Give the permission
It will ask permission for the first user of the instance. Enter your login name as ec2-user. After pressing enter, the session will begin.
Phase 2: Configuring Apache Hadoop for cloud computing
Before you start configuring Apache Hadoop environment in EC2 instance, make sure that you have downloaded the following software:
Java Package
Hadoop Package
Next, follow these steps:
Step 1: Create a Hadoop user on an EC2 instance
You will need to add a new Hadoop User to your EC2 instance. You will need root access to do this. You can get root access by using the following command

You can create a new user once you have root access.

Now, add a password for the additional user

Next, create a sudo user for the new user by typing visudo. Make the entry to visudo. File for the user whizlabs.

Step 2: Leave the root
Log in to the new user using the following commands.

Step 3: Transfer the Hadoop/Java dump to the EC2 instance
Before installing Hadoop on EC2 instances, Java must be installed. You will need to copy the zip files of Hadoop and Java downloaded from Windows to EC2 instance via file transfer tools such as WinSCP or FileZilla. These tools can be used to copy the files.
Start the tool.
Enter the hostname, username, and port number of the EC2 instance. This is the default port

Ferdinand
Related Posts
Docker Integration with AWS, Azure Clouds Software container firm Docker Inc. announced a beta program to Docker for AWS as well as Docker for Azure. The new offering integrates with the respective infrastructures of the Amazon Web Services Inc. (AWS), and Microsoft Azure cloud. It is described by Docker Inc. as the best way to install Docker and then configure and maintain the deployment. The new initiative includes Docker 1.12 with swarm-mode enabled. The infrastructure integration means users can use a pre-existing SSH key associated with their Infrastructure-as-a-Service (IaaS) account for access control, Docker said, while configuring security groups and virtual networks for simplified Docker setups and operations. The beta program is the same as Docker for Mac or Windows, according to the company.
Uncategorized
Docker Integration with AWS, Azure Clouds Software container firm Docker Inc. announced a beta program to Docker for AWS as well as Docker for Azure. The new offering integrates with the respective infrastructures of the Amazon Web Services Inc. (AWS), and Microsoft Azure cloud. It is described by Docker Inc. as the best way to install Docker and then configure and maintain the deployment. The new initiative includes Docker 1.12 with swarm-mode enabled. The infrastructure integration means users can use a pre-existing SSH key associated with their Infrastructure-as-a-Service (IaaS) account for access control, Docker said, while configuring security groups and virtual networks for simplified Docker setups and operations. The beta program is the same as Docker for Mac or Windows, according to the company.
AWS Brings ‘Intelligent’ Storage to Elastic File System Amazon Web Services (AWS) has added its “Intelligent-Tiering” feature to the Elastic File System service, letting users take better advantage of lower-priced storage classes in EFS. Intelligent-Tiering automates the transfer of items from a regular-priced storage level to a lower-priced tier if there is a change in the frequency that the item is accessed. Regular storage tiers are best for objects that are frequently accessed and used regularly. For objects that are not accessed often or infrequently, lower-priced storage tiers are preferred. Intelligent-Tiering automates the process of moving objects from one tier into another. According to a blog post written by Channy Yun (principal developer advocate at AWS), Intelligent-Tiering was made available to all EFS users on Thursday. He wrote that Amazon EFS Intelligent-Tiering is now available to all EFS users. It automatically optimizes storage costs when data access patterns change. This feature does not require any operational overhead. Intelligent-Tiering will allow objects to be moved across all four levels of EFS storage.
Uncategorized
AWS Brings ‘Intelligent’ Storage to Elastic File System Amazon Web Services (AWS) has added its “Intelligent-Tiering” feature to the Elastic File System service, letting users take better advantage of lower-priced storage classes in EFS. Intelligent-Tiering automates the transfer of items from a regular-priced storage level to a lower-priced tier if there is a change in the frequency that the item is accessed. Regular storage tiers are best for objects that are frequently accessed and used regularly. For objects that are not accessed often or infrequently, lower-priced storage tiers are preferred. Intelligent-Tiering automates the process of moving objects from one tier into another. According to a blog post written by Channy Yun (principal developer advocate at AWS), Intelligent-Tiering was made available to all EFS users on Thursday. He wrote that Amazon EFS Intelligent-Tiering is now available to all EFS users. It automatically optimizes storage costs when data access patterns change. This feature does not require any operational overhead. Intelligent-Tiering will allow objects to be moved across all four levels of EFS storage.
Cyberwarfare: The Origins, Motivations, and What You Can Do In Response Abstract This cybersecurity whitepaper explores the motivations and origins of cyberwarfare and offers suggestions for actions that you can take to counter them. Cyberwarfare: Examples of elements These can be viewed as primarily offensive, but some may also serve a defensive purpose. For a nation to be able perform an act of cyber warfare, these weapons must be developed and maintained. Cyberwarfare tools can be used to attack any nation using many of the concepts that are commonly associated with hacking and penetration testing, such as scanning, system identification, vulnerability scanning, scanning, scanning, and scanning for vulnerabilities. These tools and techniques are used to aid the attacker in gaining greater knowledge about the target’s IT infrastructure. Once the target’s IT infrastructure is known, the next step is to exploit that vulnerability. This is commonly known as gaining access. However, it could also be used to cause damage or destruction rather than remote control or intrusion. Download
Uncategorized
Cyberwarfare: The Origins, Motivations, and What You Can Do In Response Abstract This cybersecurity whitepaper explores the motivations and origins of cyberwarfare and offers suggestions for actions that you can take to counter them. Cyberwarfare: Examples of elements These can be viewed as primarily offensive, but some may also serve a defensive purpose. For a nation to be able perform an act of cyber warfare, these weapons must be developed and maintained. Cyberwarfare tools can be used to attack any nation using many of the concepts that are commonly associated with hacking and penetration testing, such as scanning, system identification, vulnerability scanning, scanning, scanning, and scanning for vulnerabilities. These tools and techniques are used to aid the attacker in gaining greater knowledge about the target’s IT infrastructure. Once the target’s IT infrastructure is known, the next step is to exploit that vulnerability. This is commonly known as gaining access. However, it could also be used to cause damage or destruction rather than remote control or intrusion. Download
CloudWatch Logs is a new AWS feature. This new extension service was introduced at the AWS Summit in New York. CloudWatch used to only monitor resource utilization. To monitor application-level logs, we need to use third-party tools. With CloudWatch Log service, one can upload and monitor various kinds of log files and even filter the logs for particular pattern which could help resolve various production issues like an invalid user trying to login to your application, a 404 page not found error or a bot attempting a denial-of-service-attack. You can also monitor other AWS services such as EBS, EC2, RDS, and others. CloudWatch can store and monitor application logs, system logs as well as webserver logs. These metrics can be set as alarms so that you are notified of any app/webserver issues and can take the necessary actions quickly. CloudWatch Logs: Loggly, Splunk and Logstash already monitor logs and provide detailed reports. CloudWatch Logs is quite basic at the moment, but it would not surprise if Amazon adds additional features in the future. What makes CloudWatch Logs better than other third-party tools? CloudWatch Logs is the only platform that can monitor resource usage and logs. CloudWatch Logs pricing works on a pay-as-you-use model, which can be more affordable than third party tools that use a per node license model. You will pay for log storage and bandwidth to upload files. CloudWatch Logs pricing – $0.50 per gigabyte ingested, $0.03 per gigabyte archived per month Ingested data : This is the data (log file) that CloudWatch is uploading to. Archived Data – All data (log events) uploaded to CloudWatch are retained. You can choose how long you want to keep the data. This data will be archived using gzip Level 6 compression and stored. Storage space for archived data is charged. Let’s review the basic terminologies used in CloudWatch Logs. Log Agent – A python script that directs logs to CloudWatch. Log events: A Log Event is a log file that contains an activity and a timestamp. Log events are only supported in text format. Other kinds of formats will be reported as error in the agent’s log file (located at /var/logs/awslogs.log). Log Stream: A log stream is a collection of log events that are reported by one source. Consider the apache server’s access log. It contains multiple events from one source, i.e. apache web server. Log Group – Log Group is a collection of Log Streams from multiple resources. A WebServerAccessLog, for example, reports Apache access logs from three identical instances. Log Group level is used for Metric filters and retention policy. Metric Filter: The Metric filters tell CloudWatch how to extract metrics from ingested Log events, and turn them into CloudWatch metrics. We can create a Metric filter called “404_Error” that will filter log events to find 404 access issues. To monitor 404 errors on different servers/instances, an alarm can be set up. Retention Policy: The retention policies determine how long events will be retained. Log Groups are assigned policies that can be applied to all Log Streams within the group. You can set the retention time from 1 day to 10 year, or you can choose to have logs never expire. How to install and configure Log Agent on Linux machine: Steps: SSH into the instance and switch to root. Run the following command in the terminal. Shellwget https://s3.amazonaws.com/aws-cloudwatch/downloads/awslogs-agent-setup-v1.0.py1wget https://s3.amazonaws.com/aws-cloudwatch/downloads/awslogs-agent-setup-v1.0.py 3.
Uncategorized
CloudWatch Logs is a new AWS feature. This new extension service was introduced at the AWS Summit in New York. CloudWatch used to only monitor resource utilization. To monitor application-level logs, we need to use third-party tools. With CloudWatch Log service, one can upload and monitor various kinds of log files and even filter the logs for particular pattern which could help resolve various production issues like an invalid user trying to login to your application, a 404 page not found error or a bot attempting a denial-of-service-attack. You can also monitor other AWS services such as EBS, EC2, RDS, and others. CloudWatch can store and monitor application logs, system logs as well as webserver logs. These metrics can be set as alarms so that you are notified of any app/webserver issues and can take the necessary actions quickly. CloudWatch Logs: Loggly, Splunk and Logstash already monitor logs and provide detailed reports. CloudWatch Logs is quite basic at the moment, but it would not surprise if Amazon adds additional features in the future. What makes CloudWatch Logs better than other third-party tools? CloudWatch Logs is the only platform that can monitor resource usage and logs. CloudWatch Logs pricing works on a pay-as-you-use model, which can be more affordable than third party tools that use a per node license model. You will pay for log storage and bandwidth to upload files. CloudWatch Logs pricing – $0.50 per gigabyte ingested, $0.03 per gigabyte archived per month Ingested data : This is the data (log file) that CloudWatch is uploading to. Archived Data – All data (log events) uploaded to CloudWatch are retained. You can choose how long you want to keep the data. This data will be archived using gzip Level 6 compression and stored. Storage space for archived data is charged. Let’s review the basic terminologies used in CloudWatch Logs. Log Agent – A python script that directs logs to CloudWatch. Log events: A Log Event is a log file that contains an activity and a timestamp. Log events are only supported in text format. Other kinds of formats will be reported as error in the agent’s log file (located at /var/logs/awslogs.log). Log Stream: A log stream is a collection of log events that are reported by one source. Consider the apache server’s access log. It contains multiple events from one source, i.e. apache web server. Log Group – Log Group is a collection of Log Streams from multiple resources. A WebServerAccessLog, for example, reports Apache access logs from three identical instances. Log Group level is used for Metric filters and retention policy. Metric Filter: The Metric filters tell CloudWatch how to extract metrics from ingested Log events, and turn them into CloudWatch metrics. We can create a Metric filter called “404_Error” that will filter log events to find 404 access issues. To monitor 404 errors on different servers/instances, an alarm can be set up. Retention Policy: The retention policies determine how long events will be retained. Log Groups are assigned policies that can be applied to all Log Streams within the group. You can set the retention time from 1 day to 10 year, or you can choose to have logs never expire. How to install and configure Log Agent on Linux machine: Steps: SSH into the instance and switch to root. Run the following command in the terminal. Shellwget https://s3.amazonaws.com/aws-cloudwatch/downloads/awslogs-agent-setup-v1.0.py1wget https://s3.amazonaws.com/aws-cloudwatch/downloads/awslogs-agent-setup-v1.0.py 3.
What is a WBS? Bill asked a great question about a WBS and how it can be used to plan project scope. Josh, please clarify. I am only at chapter 7, so if you have any questions, please forgive me. So far, it seems like the WBS is different than the schedule. Is this a point you’re making? Or am I misunderstanding it? I thought the WBS was the basis of the schedule. RegardsBill Bill, you are correct. The WBS is separate from the schedule. You should be able to focus only on the deliverables when creating the WBS. This is true regardless of the time. (I hesitate to recommend phase-based deliverables in general but I recognize that some programs may use this structure) I also recommend that you do not use staff roles to organize your work breakdown structure. Next, you break down the deliverables into tasks. The tasks will be what you use to create your schedule. Your schedule will most likely be influenced at least partially by the WBS structure. But not always. My version of a WBS is called a Product Breakdown Structure (PBS) in the PRINCE2 world. When I speak of task decomposition in a Base of Estimate (BOE), they use the same terminology for what they call a WBS. Although we do the same thing, the terminology can be confusing if you try to translate across the pond. I recommend that you never open a scheduling tool without first completing the necessary steps.
Uncategorized
What is a WBS? Bill asked a great question about a WBS and how it can be used to plan project scope. Josh, please clarify. I am only at chapter 7, so if you have any questions, please forgive me. So far, it seems like the WBS is different than the schedule. Is this a point you’re making? Or am I misunderstanding it? I thought the WBS was the basis of the schedule. RegardsBill Bill, you are correct. The WBS is separate from the schedule. You should be able to focus only on the deliverables when creating the WBS. This is true regardless of the time. (I hesitate to recommend phase-based deliverables in general but I recognize that some programs may use this structure) I also recommend that you do not use staff roles to organize your work breakdown structure. Next, you break down the deliverables into tasks. The tasks will be what you use to create your schedule. Your schedule will most likely be influenced at least partially by the WBS structure. But not always. My version of a WBS is called a Product Breakdown Structure (PBS) in the PRINCE2 world. When I speak of task decomposition in a Base of Estimate (BOE), they use the same terminology for what they call a WBS. Although we do the same thing, the terminology can be confusing if you try to translate across the pond. I recommend that you never open a scheduling tool without first completing the necessary steps.
What is a Project Coordinator? Scott was thinking about the role of Project Coordinator recently. He replied to one my Project Management Career Coaching questions with this: Josh, I am currently attending school for Project Management and have really enjoyed your daily emails. I am currently a Manufacturing Supervisor in a small company. After graduation, my goal is to be a full-fledged Project Manager. I was interested in your recent post about changing industries and how you were successful in landing a job as Project Coordinator. I have been looking for a company for some time, but I don’t have any way to get to know the workers. It is a different industry than mine and I don’t have any connections to the company. I discovered that they have two positions available for Project Coordinators. I am trying decide if it’s worth taking the risk and making the move now rather than after I graduate. It is difficult to gain experience in managing small projects in the position I am currently in. It’s a small company, so I have to accept the fact that I can’t advance in my career unless I leave or wait for someone else to retire. I am asking you this question: Do you think that a job as a Project Coordinator is a good starting place for an aspiring Project Manager. Or would you prefer to stay at the same place for another year and gain more experience in employee management? I know you are busy, so I appreciate any advice you can offer me. Scott, thank you for your email! What is the role of a Project Coordinator? Project coordinator roles are not the same in all organizations. This is why I don’t believe it fits into all strategic career paths. It depends on the industry and the organization, but as a general rule, a role in project coordination is a good way to get into project management. It indicates that the organization is mature in managing projects. It will also give you an opportunity to experience the ‘world of managing projects’ in that organization. If my experience is any indication, you’ll be right there. Building Relationships – I discuss how to get connected to people in the industry or company. If you approach it with respect and tact, you can use LinkedIn to reach out to these people. LinkedIn is a great networking tool that can bring you face-to-face connections and allow you to make friends and develop relationships over time. It’s not a quick process, but it’s possible to expect great results. It will take some time. It won’t take forever, but it won’t stop. It’s ongoing. Some people find themselves in small companies and want to go into a larger company. Once you have the experience and knowledge from a larger company, it is sometimes best to stay there. However, if you want to return to a smaller company, you can bring your expertise and help grow it. If I was you, I would interview for the position of Project Coordinator, even if it wasn’t something I wanted to do. This will give you valuable experience in interviewing and allow you to ask great questions about the company’s project management. It can make a huge difference in your chances of getting hired.
Uncategorized
What is a Project Coordinator? Scott was thinking about the role of Project Coordinator recently. He replied to one my Project Management Career Coaching questions with this: Josh, I am currently attending school for Project Management and have really enjoyed your daily emails. I am currently a Manufacturing Supervisor in a small company. After graduation, my goal is to be a full-fledged Project Manager. I was interested in your recent post about changing industries and how you were successful in landing a job as Project Coordinator. I have been looking for a company for some time, but I don’t have any way to get to know the workers. It is a different industry than mine and I don’t have any connections to the company. I discovered that they have two positions available for Project Coordinators. I am trying decide if it’s worth taking the risk and making the move now rather than after I graduate. It is difficult to gain experience in managing small projects in the position I am currently in. It’s a small company, so I have to accept the fact that I can’t advance in my career unless I leave or wait for someone else to retire. I am asking you this question: Do you think that a job as a Project Coordinator is a good starting place for an aspiring Project Manager. Or would you prefer to stay at the same place for another year and gain more experience in employee management? I know you are busy, so I appreciate any advice you can offer me. Scott, thank you for your email! What is the role of a Project Coordinator? Project coordinator roles are not the same in all organizations. This is why I don’t believe it fits into all strategic career paths. It depends on the industry and the organization, but as a general rule, a role in project coordination is a good way to get into project management. It indicates that the organization is mature in managing projects. It will also give you an opportunity to experience the ‘world of managing projects’ in that organization. If my experience is any indication, you’ll be right there. Building Relationships – I discuss how to get connected to people in the industry or company. If you approach it with respect and tact, you can use LinkedIn to reach out to these people. LinkedIn is a great networking tool that can bring you face-to-face connections and allow you to make friends and develop relationships over time. It’s not a quick process, but it’s possible to expect great results. It will take some time. It won’t take forever, but it won’t stop. It’s ongoing. Some people find themselves in small companies and want to go into a larger company. Once you have the experience and knowledge from a larger company, it is sometimes best to stay there. However, if you want to return to a smaller company, you can bring your expertise and help grow it. If I was you, I would interview for the position of Project Coordinator, even if it wasn’t something I wanted to do. This will give you valuable experience in interviewing and allow you to ask great questions about the company’s project management. It can make a huge difference in your chances of getting hired.
Project Failure: We are at It Again Serendipity happens. I replied to a student Inside pmStudent eLearning with what turned to be an article about project success and failure. Shim wrote, “Projects failure rates?” a few days later. The conventional wisdom is wrong! His blog is amazing. I began leaving comments, and it quickly became one of those comments that should be its own post. Chaos I mean the reports that various organizations have on project failure rates. Shim outlined the following in his post:
Uncategorized
Project Failure: We are at It Again Serendipity happens. I replied to a student Inside pmStudent eLearning with what turned to be an article about project success and failure. Shim wrote, “Projects failure rates?” a few days later. The conventional wisdom is wrong! His blog is amazing. I began leaving comments, and it quickly became one of those comments that should be its own post. Chaos I mean the reports that various organizations have on project failure rates. Shim outlined the following in his post:
Back to top