Python programming - IT Assignment

Assignment Task

Data Engineering

Capstone project I would like to work on big data streaming using apache spark and twitter api
Background : in this project we create a sprak application “twitter streaming” that reads online streaming from Twitter using python then process the stream using Apache spark streaming to identify hashtags and finally,returns to top trending hashtags and represents this data on deal time dashboard There are two modules in this dashboard twitter stream and spark streaming In this project, we will be a Spark Application “Twitter Streaming” that reads online streams from Twitter using Python, then processes the tweets using Apache Spark Streaming to identify hashtags and, finally, returns top trending hashtags and represents this data on a real-time dashboard.

Project

1. The first module Twitter Stream is responsible for connecting with the twitter API and fetch the tweets using tweepy library then send it to the spark client as a DataStream. This module behaves as the server listening on localhost port 3456 and waits for a client request. Once the client request is accepted by the server it tries to connect to twitter API using bearer token and retrieves the tweets which are then filtered to get only the text content of the tweets and send it to the client which in our case is the spark application.

2. The Second module Spark Stream is our main application. This module starts the pySpark streaming context and accepts data via socketTextStream. The next step is to clear tweets and extract the tags using some lambda functions. The count of each tag is then calculated for a sliding window of 60 seconds, and saved to a temporary table called "tweets". The top 10 hashtags are pulled from this temporary table using the SQL context of the Spark application, and we present them as a bar graph using Matplotlib.

This IT Assignment Help has been solved by our IT Experts at My Uni Papers.Our Assignment Writing Experts are efficient to provide a fresh solution to this question. We are serving more than 10000+ Students in Australia, UK & US by helping them to score HD in their academics. Our Experts are well trained to follow all marking rubrics & referencing style.