![]() We serve billions of streams in 61 different markets and add thousands of new tracks to our catalogue every day. Big Data at SpotifyĪt Spotify we process a lot of data for various reasons, including business reporting, music recommendation, ad serving and artist insights. In the second post we will look at the basics of Scio, its unique features, and some concrete use cases at Spotify. In this first post we will take a look at the history of big data at Spotify, the Beam unified batch and streaming model, and how Scio + Beam + Dataflow compares to the other tools we’ve been using. It is now the preferred data processing framework within Spotify and has gained many external users and open source contributors. We announced Scio at GCPNEXT16 last March and it’s been gaining traction ever since. We run Scio mainly on the Google Cloud Dataflow runner, a fully managed service, and process data stored in various systems including most Google Cloud products, HDFS, Cassandra, Elasticsearch, PostgreSQL and more. Scio is a high level Scala API for the Beam Java SDK created by Spotify to run both batch and streaming pipelines at scale. With Beam, an end user can build a pipeline using one of the SDKs (currently Java and Python), which gets executed by a runner for one of the supported distributed systems, including Apache Apex, Apache Flink, Apache Spark and Google Cloud Dataflow. Dataflow introduced a unified model to batch and streaming that consolidates ideas from these previous systems, and the Google later donated the model and SDK code to the Apache Software Foundation as Apache Beam. Google released Cloud Dataflow in early 2015 ( VLDB paper), as a cloud product based on FlumeJava and MillWheel, two Google internal systems for batch and streaming data processing. One key consideration was Google’s unique offerings of high quality big data products, including Dataflow, BigQuery, Bigtable, Pub/Sub and many more. ![]() ![]() ![]() Over the past couple of years, Spotify has been migrating our infrastructure from on premise to Google Cloud. > Verb: I can, know, understand, have knowledge. In this series we will talk about Scio, a Scala API for Apache Beam and Google Cloud Dataflow, and how we built the majority of our new data pipelines on Google Cloud with Scio. This is the first part of a 2 part blog series. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |