{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H5FdE_BKEVVq"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/4.topics/HW4_ExploreTopics.ipynb)\n",
    "\n",
    "**N.B.** Once it's open on Colab, remember to save a copy (by e.g. clicking `Copy to Drive` above).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JzmdE4_yHpHG"
   },
   "source": [
    "# HW4: Exploring topics\n",
    "In this homework, you have an open-ended task: tell us something **interesting** about a collection of text data using topic modeling. You can choose between the following datasets:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7PU0NSxn-bZR"
   },
   "source": [
    "In `acl.all.tsv` you'll find 7,188 papers published at major NLP venues (ACL, EMNLP, NAACL, TACL, etc.) between 2013 and 2020.  Here is a sample of that data:\n",
    "\n",
    "|id|year of publication|title|abstract|\n",
    "|---|---|---|---|\n",
    "|pimentel-etal-2020-phonotactic|2020|Phonotactic Complexity and Its Trade-offs|We present methods for calculating a measure of phonotactic complexity ...|\n",
    "|wang-etal-2020-amr|2020|AMR-To-Text Generation with Graph Transformer|Abstract meaning representation (AMR)-to-text generation is the challenging task  ...|\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "I75gEBBz-4AR"
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/acl.all.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Abup6G02FAXU"
   },
   "source": [
    "In `gutenberg.genre.tsv` you'll find 1,250 passages of English-language fiction from Project Gutenberg, categorized by genre (adventure, detective, love stories, science fiction, westerns).  Here's a sample:\n",
    "\n",
    "|id|genre|author|passage|\n",
    "|---|---|---|---|\n",
    "|66390_1796\t|love stories\t|Prime-Stevenson, Edward\t|Only a few days absence. I shall think of you. Imre. P. S. Please write me.\" I was amused,  ...|\n",
    "|50157_2780\t|detective and mystery stories|\tWheeler, Janet D.\t|Edina shook her head. “They think I’ve lied to them. They think I’ve cheated them. They want their money, and you can’t rightly blame them...|\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S8oqE9GqFR5N"
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/gutenberg.genre.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MVjaykL2I_50"
   },
   "source": [
    "In `convote.train.tsv`, you'll find 2,723 dialogue turns from the Convote dataset on congressional speeches, paired with the party affiliation of the speaker (Democrat/Republican). Here's a sample:\n",
    "\n",
    "|party|passage|\n",
    "|---|---|\n",
    "|R\t|mr. speaker , i claim the time in opposition to the motion to recommit .|\n",
    "|D\t|mr. speaker , on that i demand the yeas and nays .|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "86UzkCXtLBF8"
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/convote.train.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7NqJX9w1H1cp"
   },
   "source": [
    "Choose one of these datasets and use topic modeling as exploratory data analysis to find some interesting structure in that data.  There are only two constraints:\n",
    "\n",
    "* You must use topic modeling to find topics in the data\n",
    "* You must relate those **topics** to some aspect of the metadata. Examples of this could be: charting the rise and fall of **topics** over time (i.e., with year of publication as metadata) or associating **topics** with genre to find which topics are more aligned one one genre over another (perhaps using some of the measures of association we talked about earlier this semester)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OHp0qGRvIc24"
   },
   "source": [
    "## Part 1: topic modeling\n",
    "\n",
    "Q1: Try creating several topic models with different parameters for the _number_ of topics. Compare the outputs, and decide what number of topics is most reasonable for your dataset. **In a couple of sentences, justify your selection.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q2: Using the topic model with the number of topics that you selected, identify a couple of topics that are coherent. **In a paragraph, interpret these topics (i.e., by giving them a name / description). Support your answer with evidence from word and/or document distributions.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: analysis\n",
    "\n",
    "Q3: Relate the topics to some aspect of the metadata. These don't have to be the same topics that you described in question 2. **Write a paragraph synthesizing your findings, and provide quantitative evidence.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SHkbbMbLbKR5"
   },
   "source": [
    "---\n",
    "\n",
    "## To submit\n",
    "\n",
    "Congratulations on finishing this homework!\n",
    "Please follow the instructions below to download the notebook file (`.ipynb`) and its printed version (`.pdf`) for submission on bCourses -- remember **all cells must be executed**.\n",
    "\n",
    "1.  Download a copy of the notebook file: `File > Download > Download .ipynb`.\n",
    "\n",
    "2.  Print the notebook as PDF (via your browser, or tools like [nbconvert](https://nbconvert.readthedocs.io/en/latest/))."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}