For the first part of the assignment, you will use patent data from the United States (US) and abroad  to analyze the total patent applications applied and granted each year.

Assessment Details and Submission Guidelines

Trimester 

T2 2024

Course Name 

Master of Data Analytics (MDA)

Unit Code 

MDA621 

Unit Title 

Software Practice for Big Data Analytics

Assessment 

Type

Assignment 2 (Group Assessment)

Assessment  

Title

Perform Data Analytics on Real-World Problems Using Amazon Web Services.

Purpose of the  assessment  

(with ULO  

Mapping)

This assignment assesses the following Unit Learning Outcomes; students should be able to demonstrate their achievements in them. 

c. Select the tools in the chosen software stack to design and program the big data  analytics platform; 

d. Relate the concept and use of visualization to big data analytics; e. Develop and appraise big data platforms for predictive analytics in complex real world domains.

Weight 

25%

Total Marks 

100 Marks

Word limit 

700-1200 Words

Due Date 

29 Sep 2024, 23:59 – Week 11 Sunday

Submission  

Guidelines

GENERATIVE AI TOOLS MAY BE USED WITH PRIOR PERMISSION  Students are permitted to use advanced automated tools for this formative assessment only  for understanding, learning and research purposes. Using Gen AI tools to write  assignments for you, will be considered as Academic Misconduct, and it will be penalized. If  students are using any of the information from Gen AI, then you must cite or attribute the  use of the Gen AI in their assessment. 

Group Submit (only the group leader): 

• Name your final report “MDA621_T2_2024_Assigment2_GroupID.docx” • Results, including codes and commands, in a MS Word file. 

• A maximum 10-minute recorded video (mp4 format) with all members participating  in presenting your analytics solutions. It is recommended that the recorded video be  uploaded to YouTube, and then include the URL link towards the end of the final  report. Please do not upload video directly onto Moodle. 

Individual Submit (every student): 

• Complete the “Peer Evaluation” table by commenting on contributions and  participation by your team members. Please include your team member’s name in the  evaluation. 

• The assignment must be in MS Word format, 1.5 spacing, 11-pt Calibri (Body) font  and 2.5 cm margins on all four sides of your page with appropriate section headings. • Reference sources must be cited in the text of the report and listed appropriately at the  end in a reference list using either the IEEE or the APA referencing style.

Extension 

• If an extension of time to submit work is required, a Special Consideration Application  must be submitted directly on AMS. You must submit this application three working days prior to the due date of the assignment. Further information is available at

Academic  

Misconduct

Academic Misconduct is a serious offense. Depending on the seriousness of the case, penalties can vary from a written warning or zero marks to exclusion from the course or rescinding the degree. Students should make themselves familiar with the full policy and  procedure available at:.

For further information, please refer to the Academic Integrity Section  in your Unit Description.

Description 

In this group assignment, you will explore various aspects of big data analysis and manipulation  using the Hadoop ecosystem, specifically focusing on Pig Latin and Hive Query Language  (Hive QL). The primary objective of this assignment is to gain practical experience in processing  large-scale data sets and understanding the differences in data trends over time. You will be working  with two distinct data sets: patent data and sale data related to patent applications and sale transactions. 

For the first part of the assignment, you will use patent data from the United States (US) and abroad  to analyze the total patent applications applied and granted each year. This will involve executing  patent files, creating directories on a Hadoop cluster, and running the Pig Latin program to count the  total number of patent applications applied and granted yearly. This analysis will help you  understand the development of analytics reports on patent data over a decade. 

In the second part of the assignment, you will work with sales data using Hive QL commands. The  primary goal here is to analyze product sale transactions by performing various data operations such  as uploading files to HDFS, joining multiple data sets, grouping data, and performing calculations  such as listing customers, products and supplier details. You will also filter and display the expensive  products and the average price of all products.  

By the end of this assignment, you will have acquired hands-on experience in leveraging the Hadoop  ecosystem to analyze and manipulate large-scale data sets efficiently. This will serve as a foundation  for further exploration in the field of big data analysis and processing. 

Note: As a starting point, it is essential to apply the knowledge gained from previous lessons to  create a cluster on Amazon Web Services (AWS) and set it up properly to execute all required files. Including, setting up the EC2 and EMR cluster on AWS, SSH to connect to the cluster, and setting  up the File Transfer Protocol. These initial steps lay the foundation for successfully managing and  processing large-scale data sets using the Hadoop ecosystem on a cloud-based platform.

Your Tasks 

To complete Assignment 2, which comprises two main parts, your team will follow the steps outlined  in the two questions below to perform data processing and analysis tasks using the Hadoop  ecosystem, Pig Latin and Hive QL. The primary focus will be working with datasets related to patent data and sale data, allowing for hands-on experience in managing and processing large-scale  information efficiently. 

Part I: Download the US_patent.csv data from the Assignment 2 folder on Moodle. It is a comma separated value (CSV) file. The file includes a list of US patents applied and granted between  1965 and 2020. The column “Total Patent Applications” in the file presents the total applied  applications, and “Total Patent Grants” contains information about total granted applications.  

[40 Marks] 

For Part I, using Pig Latin commands and Tableau to perform the following tasks: 

1. Upload the files to HDFS. 

2. Create directories on the cluster and name the directory Patent.  

3. Load the file US_patent to the new directory. 

4. Write a Pig program to find the total number of patents applied each year. 

5. Write a Pig program to find the total number of patents granted each year. 

6. Observe and compare the applied rate and granted rate from 1965 and 2020. 

7. Using Tableau Software, visualize the results in a suitable manner. Choose the format that you find  most appropriate. 

8. In 350 words, summarize your understanding of the changing trends and accepted rates of patent applications.  

Part II: Download the saledata.zip data from Assignment 2 folder on Moodle. This file is  compressed, and once you unzip it, you will find five tab-delimited text formats files: 

[50 Marks] 

• customer (which contains information about customer ID, name and zip code) 

• product (which contains information about product ID, name, price, supplier ID and product  category) 

• product sales(which contain information about product id, corresponding sale transaction id and  no of items sold) 

• sales (which contain information about sale ID, customer ID, store ID and transaction year) • supplier (which contains information about supplier ID and supplier name) 

           For Part II, using Hive QL commands to perform the following operations: 

1. Upload the five files to HDFS 

2. Create a directory on the cluster and name the directory Sales.  

3. Create a database named sales_db and create tables where the above files can be loaded.  

4. Display the supplier id and supplier name for all suppliers. 

5. Display the customer’s name and customer zip for all customers. 

6. Display the product id, product name, product price, and supplier name for all products. 

7. Filter and display the product id, product name, and product price for products with a product  price of $500 or lower. 

8. Filter and display the customer’s name and sale year for sales involving a customer buying more  than two products. 

9. Filter the top product id and product name based on product price. 

10. Calculate the average price of all products. 

11. In 350 words, summarize your findings. 

Submission 

The group must submit the followings (only group leader submits on behalf of the whole group): • A final report should detail how your team has carried out each step of the data analytics process  explained above, any assumptions made in the analysis, any limitations in your models, and comparison  of your findings. The names of team members must be included on the cover page.  • Include all Python code required for processing data analysis. 

• A maximum 10-minute recorded video (.mp4 format) with all members participating in presenting  this group Assignment 2. You can use PowerPoint slides in your presentation.  Each student must additionally submit: 

• “Peer Evaluation” table by commenting on contributions and participation by your team members.  Please include your team member’s name in the evaluation.

Marking Criteria for the Assignment 2 

Sections 

Description of the section 

Marks

Part I 

• You will use US patent data from 1965 and 2020 to analyze the applied and  granted applications over the decade. You will use a Pig Latin program to count  the applications for each year. It will involve creating directories on a Hadoop  cluster and running the Pig Latin program. Finally, visualize the results using  Tableau Software. 

40

Part II 

• You will work with sales data using HiveQL commands. The primary goal here  is to analyze sale transactions by performing various data operations such as  uploading files to HDFS, joining two data sets, grouping data, and performing  calculations such as total sales and customer buying patterns. You will also filter  and display the top product with the highest price.

50

Video  

Presentation

• “Peer Evaluation” table by commenting on contributions and participation by  your team members. 

10

Total Marks 

100

Example Marking Rubric for Assignment

Grade  

Mark 

Part I: 

Part II: 

Video  

Presentation

HD 

80% 

+

D 

70%-9%

CR 

60%- 9%

P 

50%- 9%

Fail 

<50% 

Unsatisfactory 

Argument is  confused and  

disjointed 

Argument is  confused and  

disjointed 

None or poor  

presentation  

of results.

Excellent 

Very Good 

Good 

Satisfactory 

Detailed data  Analysis,  

Interpretation of Results in  

written final  

report.

Consistency  

logical and  

convincing

Mostly  

consistent  

logical and  

convincing

Adequate cohesion and conviction

Detailed data  Analysis,  

Interpretation of Results in  

written final  

report.

Consistency  

logical and  

convincing

Mostly  

consistent  

logical and  

convincing

Adequate cohesion and conviction

Excellent  

presentation of results.

Good  

presentation 

n of results.

Average  

presentation  

of results.

Acceptable  

presentation of  results.

      For More Assignment Samples:-