{"cells":[{"cell_type":"markdown","metadata":{"id":"kPt5q27L5557"},"source":["# CS246 - Colab 4\n","## Collaborative Filtering"]},{"cell_type":"code","source":["from IPython.display import Image\n","print(\"Colab 4 Mascot\")\n","Image(url='https://media4.giphy.com/media/wtdVYmaRWJ1PyPcc8e/giphy.gif?cid=ecf05e478jdbzo92b89f2b5ud5184xda9cen1xce4bwmjeyu&ep=v1_stickers_search&rid=giphy.gif&ct=s',width=150)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":189},"id":"qcTbH0d9e0Nf","executionInfo":{"status":"ok","timestamp":1769669329867,"user_tz":480,"elapsed":11,"user":{"displayName":"Ayush Agrawal","userId":"18383753956890992007"}},"outputId":"2b1e5c27-0f7a-453f-9dfc-2496f241a7f2"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Colab 4 Mascot\n"]},{"output_type":"execute_result","data":{"text/html":["<img src=\"https://media4.giphy.com/media/wtdVYmaRWJ1PyPcc8e/giphy.gif?cid=ecf05e478jdbzo92b89f2b5ud5184xda9cen1xce4bwmjeyu&ep=v1_stickers_search&rid=giphy.gif&ct=s\" width=\"150\"/>"],"text/plain":["<IPython.core.display.Image object>"]},"metadata":{},"execution_count":1}]},{"cell_type":"markdown","metadata":{"id":"p0-YhEpP_Ds-"},"source":["### Setup"]},{"cell_type":"markdown","metadata":{"id":"Zsj5WYpR9QId"},"source":["Let's set up Spark on your Colab environment.  Run the cell below!"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"k-qHai2252mI","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1769669375671,"user_tz":480,"elapsed":10378,"user":{"displayName":"Ayush Agrawal","userId":"18383753956890992007"}},"outputId":"b8be8979-306a-46bb-b571-3879e1777083"},"outputs":[{"output_type":"stream","name":"stdout","text":["Requirement already satisfied: pyspark in /usr/local/lib/python3.12/dist-packages (4.0.1)\n","Requirement already satisfied: py4j==0.10.9.9 in /usr/local/lib/python3.12/dist-packages (from pyspark) (0.10.9.9)\n","openjdk version \"17.0.17\" 2025-10-21\n","OpenJDK Runtime Environment (build 17.0.17+10-Ubuntu-122.04)\n","OpenJDK 64-Bit Server VM (build 17.0.17+10-Ubuntu-122.04, mixed mode, sharing)\n"]}],"source":["!pip install pyspark\n","!pip install -U -q PyDrive2\n","\n","import os\n","os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-17-openjdk-amd64\"\n","os.environ[\"PATH\"] = \"/usr/lib/jvm/java-17-openjdk-amd64/bin:\" + os.environ[\"PATH\"]\n","\n","!java -version  # Should show Java 17"]},{"cell_type":"markdown","metadata":{"id":"PUUjUvXe3Sjk"},"source":["Now we authenticate a Google Drive client to download the files we will be processing in our Spark job.\n","\n","**Make sure to follow the interactive instructions.**"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lRElWs_x2mGh"},"outputs":[],"source":["from pydrive2.auth import GoogleAuth\n","from pydrive2.drive import GoogleDrive\n","from google.colab import auth\n","from oauth2client.client import GoogleCredentials\n","\n","# Authenticate and create the PyDrive client\n","auth.authenticate_user()\n","gauth = GoogleAuth()\n","gauth.credentials = GoogleCredentials.get_application_default()\n","drive = GoogleDrive(gauth)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QHsFTGUy2n1c"},"outputs":[],"source":["id='1QtPy_HuIMSzhtYllT3-WeM3Sqg55wK_D'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('MovieLens.training')\n","\n","id='1ePqnsQTJRRvQcBoF2EhoPU8CU1i5byHK'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('MovieLens.test')\n","\n","id='1ncUBWdI5AIt3FDUJokbMqpHD2knd5ebp'\n","downloaded = drive.CreateFile({'id': id})\n","downloaded.GetContentFile('MovieLens.item')"]},{"cell_type":"markdown","metadata":{"id":"qwtlO4_m_LbQ"},"source":["If you executed the cells above, you should be able to see the dataset we will use for this Colab under the \"Files\" tab on the left panel.\n","\n","Next, we import some of the common libraries needed for our task."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"twk-K-jilWK7"},"outputs":[],"source":["import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","import pyspark\n","from pyspark.sql import *\n","from pyspark.sql.types import *\n","from pyspark.sql.functions import *\n","from pyspark import SparkContext, SparkConf"]},{"cell_type":"markdown","metadata":{"id":"BtrJlMBt1Ela"},"source":["Let's initialize the Spark context."]},{"cell_type":"code","source":["!pip show pyspark | grep Version"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"HOKrLLuPgOsK","executionInfo":{"status":"ok","timestamp":1769669392967,"user_tz":480,"elapsed":1913,"user":{"displayName":"Ayush Agrawal","userId":"18383753956890992007"}},"outputId":"58b4960c-feb6-406e-9098-158940732c82"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Version: 4.0.1\n"]}]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Vm3sAVeK1EDZ"},"outputs":[],"source":["# create the session\n","conf = SparkConf().set(\"spark.ui.port\", \"4050\")\n","\n","# create the context\n","sc = pyspark.SparkContext(conf=conf)\n","spark = SparkSession.builder.getOrCreate()"]},{"cell_type":"markdown","metadata":{"id":"kAYRX2PMm0L6"},"source":["### Data Loading"]},{"cell_type":"markdown","metadata":{"id":"7hXdMR6wnEIM"},"source":["In this Colab, we will be using the [MovieLens dataset](https://grouplens.org/datasets/movielens/), specifically the 100K dataset (which contains in total 100,000 ratings from 1000 users on ~1700 movies).\n","\n","We load the ratings data in a 80%-20% ```training```/```test``` split, while the ```items``` dataframe contains the movie titles associated to the item identifiers."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5K93ABEy9Zlo"},"outputs":[],"source":["schema_ratings = StructType([\n","    StructField(\"user_id\", IntegerType(), False),\n","    StructField(\"item_id\", IntegerType(), False),\n","    StructField(\"rating\", IntegerType(), False),\n","    StructField(\"timestamp\", IntegerType(), False)])\n","\n","schema_items = StructType([\n","    StructField(\"item_id\", IntegerType(), False),\n","    StructField(\"movie\", StringType(), False)])\n","\n","training = spark.read.option(\"sep\", \"\\t\").csv(\"MovieLens.training\", header=False, schema=schema_ratings)\n","test = spark.read.option(\"sep\", \"\\t\").csv(\"MovieLens.test\", header=False, schema=schema_ratings)\n","items = spark.read.option(\"sep\", \"|\").csv(\"MovieLens.item\", header=False, schema=schema_items)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MC_m1oygCoEm","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1769669425342,"user_tz":480,"elapsed":38,"user":{"displayName":"Ayush Agrawal","userId":"18383753956890992007"}},"outputId":"6ce67430-c69a-43e0-e923-690a34e76cf6"},"outputs":[{"output_type":"stream","name":"stdout","text":["root\n"," |-- user_id: integer (nullable = true)\n"," |-- item_id: integer (nullable = true)\n"," |-- rating: integer (nullable = true)\n"," |-- timestamp: integer (nullable = true)\n","\n","root\n"," |-- user_id: integer (nullable = true)\n"," |-- item_id: integer (nullable = true)\n"," |-- rating: integer (nullable = true)\n"," |-- timestamp: integer (nullable = true)\n","\n"]}],"source":["training.printSchema()\n","test.printSchema()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"81Vgo4ovCqtQ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1769669428005,"user_tz":480,"elapsed":9,"user":{"displayName":"Ayush Agrawal","userId":"18383753956890992007"}},"outputId":"c0dc1abb-17db-4c87-c684-01446ed2dfd0"},"outputs":[{"output_type":"stream","name":"stdout","text":["root\n"," |-- item_id: integer (nullable = true)\n"," |-- movie: string (nullable = true)\n","\n"]}],"source":["items.printSchema()"]},{"cell_type":"markdown","metadata":{"id":"CRaF2A_j_nC7"},"source":["### Your task"]},{"cell_type":"markdown","metadata":{"id":"zM9w2aUvJ7KN"},"source":["Let's compute some stats!  What is the number of ratings in the training and test dataset? How many movies are in our dataset?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"8XZaH16t_CIw"},"outputs":[],"source":["''' 3 lines of code in total expected.\n","For sub-parts of the question (if any), creating different cells of code would be recommended.'''\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"wpsaYOqRxar2"},"source":["Using the training set, train a model with the Alternating Least Squares method available in the Spark MLlib: [https://spark.apache.org/docs/latest/ml-collaborative-filtering.html](https://spark.apache.org/docs/latest/ml-collaborative-filtering.html)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Oitav_xhQD9w"},"outputs":[],"source":["''' 5-8 lines of code in total expected but can differ based on your style.\n","For sub-parts of the question (if any), creating different cells of code would be recommended.'''\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"TtR1xRvonxiO"},"source":["Now compute the RMSE on the test dataset. **Please refer to Gradescope for the hyperparameters**.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"GP23Xkgwi0SD"},"outputs":[],"source":["''' 4-5 lines of code in total expected but can differ based on your style.\n","For sub-parts of the question (if any), creating different cells of code would be recommended.'''\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"lBvSaWGEMHXI"},"source":["At this point, you can use the trained model to produce the top-K recommendations for each user."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"KbMlWL5_UfSc"},"outputs":[],"source":["''' 9-10 lines of code in total expected but can differ based on your style.\n","For sub-parts of the question (if any), creating different cells of code would be recommended.'''\n","# YOUR CODE HERE\n"]},{"cell_type":"markdown","metadata":{"id":"SIrXJyVNP2AI"},"source":["Once you have working code for each cell above, **head over to Gradescope, carefully read the questions, and submit your solution for this Colab**! As a sanity check, the 4th most recommended movie under the setting of Q2.6 should be `Star Wars (1977)`."]},{"cell_type":"code","source":[],"metadata":{"id":"LhubtaWbqkOp"},"execution_count":null,"outputs":[]}],"metadata":{"colab":{"provenance":[{"file_id":"11NN10Zg670yFNLXImwGYl_6zgPMDlwij","timestamp":1769732737080}]},"kernelspec":{"display_name":"Python 3","name":"python3"}},"nbformat":4,"nbformat_minor":0}