代写MongoDB的作业，作业类型其实偏向数据分析而不是MongoDB的用法。数据集很大，实现要求的查询需求即可。
Introduction
In this assignment, you will show that you can work with different NoSQL
systems for data persistence and understand the strength and weakness of each
system with respect to certain workload features. You are asked to work with a
music data set and a list of target queries. You will design data schema based
on the data set feature and the given query workload for MongoDB, Neo4j and
HBase respectively. You will show that your design can support the target
queries by load the data in each system following your schema and run queries
against the data.
Data set
The data that you will use is the Last.fm ( http://www.last.fm
)data set released in Het-Rec2011 (
http://ir.ii.uam.es/hetrec2011/ ). You can
view the details of the data set and download it from
http://grouplens.org/datasets/hetrec-2011/
.
The data set contains information about “social networking, tagging, and music
artist listening information from a set of 2k users from Last.fm online music
system”. The data is organized as relational tables and is stored in several
text files, each corresponding to a table. All files are of tab separated
format. Unique IDs are assigned to artists, users and tags for easy
referencing across files.
Basic information about artists is stored in file artists.dat. Each line
contains four columns: id, name of the artist, an url pointing to the
description of the artist and another pictureURL pointing to a picture of the
artist. The tag data is stored in another file tags.dat with only two columns:
the tagID and the actual tagValue. The data set does not contain any personal
information about user, hence there is no separate file for users. All other
files in the data set contain information about some relationships.
The user artists.dat stores information about listening count per user per
artist. The user friends.dat stores the friend relations between users. The
tag assignments of artists by users are stored in two files. The only
difference between the files is the timestamp format. The file user
taggedartists.dat stores the day, month and year in separate columns while
user taggedartists-timestamps.dat stores the unix timestamp in a single
column. You only need to use one of the files for tag information in this
assignment.
Target Queries
Simple query
- given a user id, find all artists the user’s friends listen.
- given an artist name, find the most recent 10 tags that have been assigned to it.
- given an artist name, find the top 10 users based on their respective listening counts of this artist. Display both the user id and the listening count
- given a user id, find the most recent 10 artists the user has assigned tag to.
Complex queries
- find the top 5 artists ranked by the number of users listening to it
- given an artist name, find the top 20 tags assigned to it. The tags are ranked by the number of times it has been assigned to this artist
- given a user id, find the top 5 artists listened by his friends but not him. We rank artists by the sum of friends’ listening counts of the artist.
- given an artist name, find the top 5 similar artists. Here similarity between a pair of artists is defined by the number of unique users that have listened both. The higher the number, the more similar the two artists are.
Tasks
Your tasks include following.
Schema Design for each system
You should provide three schema design versions. For MongoDB and Neo4j, your
schema should support both the simple queries and the complex queries. For
HBase, your schema only needs to support the simple queries. For each schema
version, make sure you utilize features provided by the storage system such as
indexing, aggregation, ordering, filtering and so on. Please note that your
schema may deviate a lot from the relational structure of the original data
set. You can discard IDs if you find they are not useful. You can duplicate
data if you find that helps with certain queries. You will not get any mark if
you present a schema that is an exact copy of the relational structure in the
original data set.
Query Design and Implementation
For MongoDB, load the complete data on MongoDB and set up proper indexes that
will be used by the target queries. Design and implement all target queries.
You may implement a query using shell command, a combination of JavaScript and
shell command or as Python/Java program. For each query (or sub query), report
execution statistics such as: which index is used, how many documents are
examined to answer this query.
For Neo4j, load the complete data on Neo4j and set up proper indexes that will
be used by the target queries. Design and implement all target queries. You
may implement a query using cypher command or as Pyton/Java program. For each
query, report execution statics such as which index is used, how many records
are examined, whether or not a full scan is involved.
For HBase, load a small subset of data that can demonstrate the simple
queries. Design and implement only the simple queries. You may implement a
query using shell command, a combination of Ruby script and shell command, or
as Python/Java program. You can use filter as well. For each query, describe
the number of rows, or subset of columns are examined. Especially, highlight
if a full table scan is involved in answering this query.
Deliverable and Submission Guideline
This is a group project, each group can have up to 3 students. Each group
needs to produce the following.
A Written Report
The report should contain five sections. The first section is a brief
introduction of the project. Section two to four should cover a system each.
Section five should provide a comparison and summary.
There is no mark on section one. It is included to make the report complete.
So please keep it really short.
Section two to four should contain the following two sub sections
Schema Design
In this section, describe the schema with respect to the particular system.
Your description should include information at “table” and “column” level as
well as possible primary keys/row keys and secondary indexes. You should show
sample data based on schema. For instance, you may show sample documents of
each MongoDB collection, a sample property graph involving all node types and
relationship types for Neo4j, a sample row in each HBase table. If certain
data are duplicated in different collection/tables, highlight the duplication
and briefly justify your decision.
Query Design and Execution
In this section, describe implementation of each target query. You may include
the entire shell query command, or show pseudo code. You should also run
sample queries and describe the execution statistics for each sample query as
described in the Tasks section.
In section five, compare the three systems with respect to ease of use, query
performance and schema differences. You can also describe problems encountered
in schema design or query design.
Submit a hard copy of your report together with a signed group assignment
sheet in Week 10 lab.
System Demo
Each group will demo their implementation in week 10 lab. The required data
need to be loaded in respective systems for the demo. You can run demo in your
own machine, on lab machine or on some cloud servers. The tutor will ask you
to run a few randomly selected queries to test the implementation.
