{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# SST-dev revalidation analysis" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "__author__ = 'Christopher Potts'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set-up" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "import pandas as pd\n", "\n", "import dynasent_utils as utils" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "src_filename = os.path.join(\n", " \"dynasent-v1.1\", \"sst-dev-validated.jsonl\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load the dataset" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "sst = utils.load_dataset(src_filename)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'hit_ids': ['s20533'],\n", " 'sentence': '-LRB- A -RRB- n utterly charming and hilarious film that reminded me of the best of the Disney comedies from the 60s.',\n", " 'tree': '(4 (2 (1 -LRB-) (2 (2 A) (3 -RRB-))) (4 (4 (2 n) (4 (3 (2 utterly) (4 (3 (4 charming) (2 and)) (4 hilarious))) (3 (2 film) (3 (2 that) (4 (4 (2 (2 reminded) (3 me)) (4 (2 of) (4 (4 (2 the) (4 best)) (2 (2 of) (3 (2 the) (3 (3 Disney) (2 comedies))))))) (2 (2 from) (2 (2 the) (2 60s)))))))) (2 .)))',\n", " 'text_id': 'sst-dev-validate-0000437',\n", " 'sst_label': '4',\n", " 'label_distribution': {'positive': ['w207', 'w3', 'w840', 'w135', 'w26'],\n", " 'negative': [],\n", " 'neutral': [],\n", " 'mixed': []},\n", " 'gold_label': 'positive'}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sst[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Validation label distributions" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "positive 2129\n", "negative 2052\n", "mixed 712\n", "neutral 612\n", "Total 5505\n", "dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "utils.get_label_distribution(sst, dist_labels=True)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "positive 433\n", "negative 421\n", "mixed 108\n", "neutral 75\n", "No Majority 64\n", "Total 1101\n", "Name: gold_label, dtype: int64" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "utils.get_label_distribution(sst, dist_labels=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Relationship between validation labels and the original SST labels" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame(sst)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "df.gold_label = df.gold_label.fillna(\"no majority\")" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "cols = ['positive', 'negative', 'neutral', 'mixed', 'no majority']" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sst_label01234
gold_label
positive0264207160
negative1322275750
neutral0844221
mixed62939322
no majority12325132
\n", "
" ], "text/plain": [ "sst_label 0 1 2 3 4\n", "gold_label \n", "positive 0 2 64 207 160\n", "negative 132 227 57 5 0\n", "neutral 0 8 44 22 1\n", "mixed 6 29 39 32 2\n", "no majority 1 23 25 13 2" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xtab5 = pd.crosstab(df.gold_label, df.sst_label)\n", "\n", "xtab5.loc[cols]" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "sst_map = {\n", " '0': 'negative', '1': 'negative',\n", " '2': 'neutral',\n", " '3': 'positive', '4': 'positive'}\n", "\n", "df['SST-3'] = df.sst_label.apply(lambda x: sst_map[x])" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SST-3positivenegativeneutral
gold_label
positive367264
negative535957
neutral23844
mixed343539
no majority152425
\n", "
" ], "text/plain": [ "SST-3 positive negative neutral\n", "gold_label \n", "positive 367 2 64\n", "negative 5 359 57\n", "neutral 23 8 44\n", "mixed 34 35 39\n", "no majority 15 24 25" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "xtab = pd.crosstab(df.gold_label, df['SST-3'])\n", "\n", "cols = ['positive', 'negative', 'neutral', 'mixed', 'no majority']\n", "\n", "xtab[cols[: 3]].loc[cols]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Distribution of validation response distributions" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "5 528\n", "4 278\n", "3 231\n", "dtype: int64" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "utils.get_dist_of_majority_dists(sst)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Worker distribution" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(103,)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "worker_dist = utils.get_global_worker_dist(sst)\n", "\n", "worker_dist.shape" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "w135 280\n", "w55 241\n", "w3 220\n", "w352 172\n", "w26 171\n", "dtype: int64" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "worker_dist.head()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "31.0" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "worker_dist.median()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 10\n", "dtype: int64" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "worker_dist.mode()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe4AAAE/CAYAAACAdSDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZlUlEQVR4nO3de7RkZX3m8e8jeCE0ogx6huDlGOM1tDcaYsSJ3d4VozgBhGVUxkubLDVq2slC1NEkk6x2JjqZSTIqRISZSex4AUVAEC8N6ijYrWijDNEx7Qj0EpWLtCGaxt/8Ufvo4VCnT0HXrur3nO9nrbNO1d679v69VX3q6XfvqvdNVSFJktpwl2kXIEmSRmdwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDektuJPcP8lnklyZ5OtJXtstf1uSa5Jc3v08u68aJElabtLX97iTHAIcUlVfTnIAsBU4Bjge2FlVf97LgSVJWsb27WvHVbUD2NHdvjnJlcChd2ZfBx98cM3Ozo6xun79+Mc/Zv/99592GRNlm1cG27wy2Obp27p16w+q6j7D1vUW3PMlmQUeC1wKHAW8OsmLgS3Ahqq6YXePn52dZcuWLX2XOTabN29m7dq10y5jomzzymCbVwbbPH1JvrPour6HPE2yCrgY+NOqOivJDPADoIA/YXA6/aVDHrceWA8wMzNz+KZNm3qtc5x27tzJqlWrpl3GRNnmlcE2rwy2efrWrVu3tarWDFvXa3AnuStwLnBhVb1zyPpZ4NyqOmx3+1mzZk3Z49672eaVwTavDLZ5+pIsGtx9fqo8wHuBK+eHdvehtTnPB67oqwZJkpabPq9xHwW8CNiW5PJu2SnAiUkew+BU+XbglT3WIEnSstLnp8o/B2TIqvP7OqYkScudI6dJktQQg1uSpIYY3JIkNcTgliSpIQa3JEkNMbglSWrIRMYq39vMnnxer/vfsHoXJ+3hMbZvPHpM1UiSlhN73JIkNcTgliSpIQa3JEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDTG4JUlqiMEtSVJDDG5JkhpicEuS1BCDW5KkhhjckiQ1xOCWJKkhBrckSQ0xuCVJaojBLUlSQwxuSZIaYnBLktQQg1uSpIYY3JIkNcTgliSpIQa3JEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDTG4JUlqiMEtSVJDDG5JkhrSW3AnuX+SzyS5MsnXk7y2W35QkouSfLP7fe++apAkabnps8e9C9hQVY8AHg+8KskjgZOBT1XVQ4BPdfclSdIIegvuqtpRVV/ubt8MXAkcCjwPOLPb7EzgmL5qkCRpuZnINe4ks8BjgUuBmaraAYNwB+47iRokSVoOUlX9HiBZBVwM/GlVnZXkxqq617z1N1TV7a5zJ1kPrAeYmZk5fNOmTWOrads1N41tX8PM7Affu2XP9rH60APHU8yE7Ny5k1WrVk27jImyzSuDbV4Z9rY2r1u3bmtVrRm2rtfgTnJX4Fzgwqp6Z7fsKmBtVe1Icgiwuaoetrv9rFmzprZs2TK2umZPPm9s+xpmw+pdvGPbvnu0j+0bjx5TNZOxefNm1q5dO+0yJso2rwy2eWXY29qcZNHg7vNT5QHeC1w5F9qdc4CXdLdfAny0rxokSVpu9qxbuHtHAS8CtiW5vFt2CrAR+ECSlwH/DziuxxokSVpWegvuqvockEVWP6Wv40qStJw5cpokSQ0xuCVJaojBLUlSQwxuSZIaYnBLktQQg1uSpIYY3JIkNcTgliSpIQa3JEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDdl32gWoTbMnn3eb+xtW7+KkBcumbfvGo6ddgiSNnT1uSZIaYnBLktQQg1uSpIYY3JIkNcTgliSpIQa3JEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDTG4JUlqiMEtSVJDDG5JkhpicEuS1BCDW5KkhhjckiQ1ZN9pFyCtZLMnnzftEnZr+8ajp12CpAXscUuS1BCDW5KkhhjckiQ1xOCWJKkhBrckSQ0xuCVJakhvwZ3k9CTXJbli3rK3JbkmyeXdz7P7Or4kSctRnz3uM4BnDln+X6rqMd3P+T0eX5KkZae34K6qS4Dr+9q/JEkr0TSucb86yde6U+n3nsLxJUlqVqqqv50ns8C5VXVYd38G+AFQwJ8Ah1TVSxd57HpgPcDMzMzhmzZtGltd2665aWz7GmZmP/jeLXu2j9WHHjieYnqy8DkcR5vHre/ncOfOnaxatWqP9tH3v8U9tfA5HEebW2ObV4a9rc3r1q3bWlVrhq2baHCPum6hNWvW1JYtW8ZWV9/jQ29YvYt3bNuzYeD39jGiFz6H42jzuPX9HG7evJm1a9fu0T5aG6t8HG1ujW1eGfa2NidZNLgneqo8ySHz7j4fuGKxbSVJ0u311kVK8n5gLXBwkquBtwJrkzyGwany7cAr+zq+JEnLUW/BXVUnDln83r6OJ0nSSuDIaZIkNcTgliSpIQa3JEkNGSm4kxyX5IDu9puTnJXkcf2WJkmSFhq1x/2Wqro5yROBZwBnAu/qryxJkjTMqMF9a/f7aOBdVfVR4G79lCRJkhYzanBfk+Q9wPHA+UnufgceK0mSxmTU8D0euBB4ZlXdCBwE/PveqpIkSUMtOQBLkrsAl80fU7yqdgA7+ixMkiTd3pI97qr6GfDVJA+YQD2SJGk3Rh3y9BDg60kuA348t7CqnttLVZIkaahRg/uPeq1CkiSNZKTgrqqLkzwQeEhVfTLJLwH79FuaJElaaNSR014BfAh4T7foUOAjfRUlSZKGG/XrYK8CjgJ+BFBV3wTu21dRkiRpuFGD+ydV9dO5O0n2BaqfkiRJ0mJGDe6Lk5wC7JfkacAHgY/1V5YkSRpm1OA+Gfg+sA14JXB+Vb2pt6okSdJQo34d7LFVdRpw2tyCJL9VVfa6JUmaoFF73KclWT13J8mJwJv7KUmSJC1m1B73scCHkrwQeCLwYuDpvVUlSZKGGnUAlm8nOYHBd7e/Czy9qm7ptTJJknQ7uw3uJNu47de+DmIwYtqlSaiqR/VZnCRJuq2letzPmUgVkiRpJLsN7qr6Tjcf99fmz8ctSZKmw/m4JUlqiPNxS5LUEOfjliSpIXdkPu4Z4Ihu0WVVdV1/ZUmSpGFGnY/7eOAy4DjgeAZfBzu2z8IkSdLtjXqq/E3AEXO97CT3AT4JfKivwiRJ0u2NOlb5XRacGv/hHXisJEkak1F73BckuRB4f3f/BcD5/ZQkSZIWs9SQp68DPg+cAvwWgwlGApxaVWf3X54kSZpvqR73/YD/Cjwc+BrwvxkE+Rd6rkuSJA2x1JCnbwBIcjdgDfAE4KUM5ue+saoe2X+JkiRpzqjXuPcD7gkc2P1cC2zrqyhJkjTcUte4TwV+DbgZuJTBqfJ3VtUNE6hNkiQtsFSP+wHA3YFvAtcAVwM39l2UNA6zJ5/X6/43rN7FST0fY9oWPod7Y5u3bzx62iVIE7XUNe5nJgmDXvcTgA3AYUmuB75QVW+dQI2SJKmz5DXuqirgiiQ3Ajd1P88BjgQMbkmSJmipa9y/z6CnfRTwL/ziq2Cn44fTJEmauKV63LMMxiN/fVXt6L8cSZK0O0td4/6DSRUiSZKW5kQhkiQ1pLfgTnJ6kuuSXDFv2UFJLkryze73vfs6viRJy1GfPe4zgGcuWHYy8Kmqegjwqe6+JEkaUW/BXVWXANcvWPw84Mzu9pnAMX0dX5Kk5WjS17hn5j6d3v2+74SPL0lS0zIYX6WnnSezwLlVdVh3/8aqute89TdU1dDr3EnWA+sBZmZmDt+0adPY6tp2zU1j29cwM/vB927Zs32sPvTA8RTTk4XP4Tja3BrbvHfo+29l586drFq1qtdj7G1s8/StW7dua1WtGbZu1NnBxuV7SQ6pqh1JDgGuW2zDqjoVOBVgzZo1tXbt2rEV0fdYyxtW7+Id2/bsqd3+wrXjKaYnC5/DcbS5NbZ579D338rmzZsZ5/tPC2zz3m3Sp8rPAV7S3X4J8NEJH1+SpKb1+XWw9zMYHvVhSa5O8jJgI/C0JN8EntbdlyRJI+rtnFdVnbjIqqf0dUxJkpY7R06TJKkhBrckSQ0xuCVJaojBLUlSQwxuSZIaYnBLktQQg1uSpIYY3JIkNcTgliSpIQa3JEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSH7TrsASdoTsyef1+v+N6zexUk9H2Patm88etol6A6wxy1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDTG4JUlqiMEtSVJDDG5JkhpicEuS1BCDW5KkhhjckiQ1xOCWJKkhBrckSQ0xuCVJaojBLUlSQwxuSZIaYnBLktQQg1uSpIYY3JIkNcTgliSpIQa3JEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUkH2ncdAk24GbgVuBXVW1Zhp1SJLUmqkEd2ddVf1giseXJKk5niqXJKkh0wruAj6RZGuS9VOqQZKk5qSqJn/Q5Jer6tok9wUuAl5TVZcs2GY9sB5gZmbm8E2bNo3t+NuuuWls+xpmZj/43i17to/Vhx44nmJ6svA5HEebW2ObV4aV0OaF7zc7d+5k1apVU6pmOva2Nq9bt27rYp//mkpw36aA5G3Azqr688W2WbNmTW3ZsmVsx5w9+byx7WuYDat38Y5te/bxge0bjx5TNf1Y+ByOo82tsc0rw0po88L3m82bN7N27drpFDMle1ubkywa3BM/VZ5k/yQHzN0Gng5cMek6JElq0TT+GzkDnJ1k7vh/V1UXTKEOSZKaM/HgrqpvA4+e9HElSVoO/DqYJEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUEINbkqSGLO/hgCRJSxo2EuJJPY8weUfs7SNJTpo9bkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDTG4JUlqiMEtSVJDDG5JkhriyGmSpL3awpHd+rCno8VNcnQ3e9ySJDXE4JYkqSEGtyRJDTG4JUlqiMEtSVJDDG5JkhpicEuS1BCDW5KkhhjckiQ1xOCWJKkhBrckSQ0xuCVJaojBLUlSQwxuSZIaYnBLktQQg1uSpIYY3JIkNcTgliSpIQa3JEkNMbglSWqIwS1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSEGtyRJDTG4JUlqyFSCO8kzk1yV5FtJTp5GDZIktWjiwZ1kH+CvgWcBjwROTPLISdchSVKLptHjPhL4VlV9u6p+CmwCnjeFOiRJas40gvtQ4Lvz7l/dLZMkSUtIVU32gMlxwDOq6uXd/RcBR1bVaxZstx5Y3919GHDVRAvdMwcDP5h2ERNmm1cG27wy2Obpe2BV3WfYin0nXQmDHvb9592/H3Dtwo2q6lTg1EkVNU5JtlTVmmnXMUm2eWWwzSuDbd67TeNU+ZeAhyR5UJK7AScA50yhDkmSmjPxHndV7UryauBCYB/g9Kr6+qTrkCSpRdM4VU5VnQ+cP41jT0iTp/j3kG1eGWzzymCb92IT/3CaJEm68xzyVJKkhhjcd1KS+yf5TJIrk3w9yWuHbLM2yU1JLu9+/sM0ah2XJPdIclmSr3Zt/qMh29w9yd93w9lemmR28pWOz4htPinJ9+e9zi+fRq3jlmSfJF9Jcu6QdcvqdYYl27tcX+PtSbZ1bdoyZH2S/Lfudf5aksdNo85xGaG9TbxnT+Ua9zKxC9hQVV9OcgCwNclFVfWNBdt9tqqeM4X6+vAT4MlVtTPJXYHPJfl4VX1x3jYvA26oql9NcgLwduAF0yh2TEZpM8DfV9Wrp1Bfn14LXAncc8i65fY6w+7bC8vzNQZYV1WLfX/5WcBDup9fB97V/W7Z7toLDbxn2+O+k6pqR1V9ubt9M4M/+GU9AlwN7Ozu3rX7WfghiecBZ3a3PwQ8JUkmVOLYjdjmZSfJ/YCjgb9ZZJNl9TqP0N6V6nnA/+j+Dr4I3CvJIdMuaqUzuMegO034WODSIat/ozvN+vEkvzbRwnrQnU68HLgOuKiqFrb550PaVtUu4CbgX022yvEaoc0Av92dSvxQkvsPWd+avwD+EPjZIuuX2+u8VHth+b3GMPhP6CeSbO1Gq1xouQ1RvVR7oYH3bIN7DyVZBXwYeF1V/WjB6i8zGLbu0cBfAh+ZdH3jVlW3VtVjGIx4d2SSwxZsMqzX1XQPdYQ2fwyYrapHAZ/kFz3RJiV5DnBdVW3d3WZDljX5Oo/Y3mX1Gs9zVFU9jsEp8Vcl+c0F65fN69xZqr1NvGcb3Hugu+b5YeBvq+qsheur6kdzp1m7767fNcnBEy6zF1V1I7AZeOaCVT8f0jbJvsCBwPUTLa4ni7W5qn5YVT/p7p4GHD7h0sbtKOC5SbYzmL3vyUn+14JtltPrvGR7l+FrDEBVXdv9vg44m8HsjfONNER1K5Zqbyvv2Qb3ndRdz3svcGVVvXORbf713HW/JEcyeL5/OLkqxyvJfZLcq7u9H/BU4P8s2Owc4CXd7WOBT1fDgwWM0uYF1/yey+DzDs2qqjdW1f2qapbBkMSfrqrfWbDZsnmdR2nvcnuNAZLs332wliT7A08Hrliw2TnAi7tPlz8euKmqdky41LEYpb2tvGf7qfI77yjgRcC27vonwCnAAwCq6t0M3tB+L8ku4BbghFbf3DqHAGcm2YfBP+gPVNW5Sf4Y2FJV5zD4z8z/TPItBj2wE6ZX7liM0ubfT/JcBt80uB44aWrV9miZv863swJe4xng7C6n9gX+rqouSPK78PP3sPOBZwPfAv4J+HdTqnUcRmlvE+/ZjpwmSVJDPFUuSVJDDG5JkhpicEuS1BCDW5KkhhjckiQ1xODWipfk1nmzAV2e5OQp1bF9Twd7SPK2JG9YYpszkhx7B/Y5m2Th93uXesyix0jyF3MjViXZnGTNHdl3X0Z87l6dpOWvRGkZ8HvcEtzSDWmqniU5CHh8Vb1u2rXcSacDnwfeN+1CtHLZ45aGSHJgkquSPKy7//4kr+huvyvJliyYn7vrMf9Zki906x+X5MIk/3dukIcM5vu9JMnZSb6R5N1Jbvd3mOR3MpgH/PIk78lgopN9up7sFRnMKfz6JdrwiiRf6iZM+HCSX5q3+qlJPpvkH7qxuucmU/nP3WO+luSVQ/Y5dJtuZK2/6tp0HnDfRco6FrhgkXpP7Np1RZK3z1v+sq7OzUlOS/JXQx77pHlnTL4yb4SsP+z2+dUkG0d4Xub29+AkF2QwGcVnkzwcoKr+Cdiewaha0lQY3BLst+BU+Quq6ibg1cAZGcw3fe+qOq3b/k1VtQZ4FPCkJI+at6/vVtVvAJ8FzmAQVI8H/njeNkcCG4DVwIOBfzu/mCSPYDC39VHdmYBbgRcCjwEOrarDqmo1S/f6zqqqI7oJE65kMIf2nFngSQymsnx3knt062+qqiOAI4BXJHnQgn0uts3zgYd1bXoF8IRFajoKuN1kHkl+mcGc3k/u2nlEkmO65W9h8Bw+DXj4Ivt9A/Cq7vn6N8AtSZ4FHAP8evcc/KcRnpc5pwKvqarDu33/93nrtnTHkKbCU+XSIqfKq+qiJMcBfw08et6q4zOYEnBfBkOiPhL4WrfunO73NmBVN1f7zUn+Od2Y58BlVfVtGPTkgScymNN6zlMYTGLxpQyGZ9yPwZSiHwN+JclfAucBn1iiXYcl+Y/AvYBVwIXz1n2gqn4GfDPJtxkE4tOBR827Nn0g8BDgH+Y9brFtfhN4f1XdClyb5NOL1HQI8P0hy48ANlfV9wGS/G23T4CLq+r6bvkHgYcOefzngXd2jzurqq5O8lTgfV0vmbl9LPG8zM349wTgg/nFFON3n7fJdSz+Hwipdwa3tIjuFPYjGIxZfBBwdde7fANwRFXdkOQM4B7zHjY3g9TP5t2euz/397ZwnOGF9wOcWVVvHFLTo4FnAK8CjgdeupsmnAEcU1VfTXISsHY3x6zuuK+pqoVBNrugtmHbPHvIPoe5hds+X/P3O8xiy2+jqjZ2p+ifDXyxC+0sUtMZLP68wOBM5I27+dzDPRi0Q5oKT5VLi3s9g1OpJwKnZzCN6z2BHwM3JZlhMK/vHXVkkgd1/zF4AfC5Bes/BRyb5L4w+EBXkgdm8Inzu1TVhxmcPn7cEsc5ANjR1f3CBeuOS3KXJA8GfgW4ikHP8/e67Uny0AxmUZpvsW0uAU7oroEfAqxbpKYrgV8dsvxSBpcdDs5gQpcTgYuBy7rl985g+tDfHrbTJA+uqm1V9XYGp7IfzuCMxEvnrmFn8MG4pZ4XqupHwD92Z1vmrt/PP+PyUG4/i5Y0Mfa4pe4a97z7FzD49PDLgSOr6uYklwBvrqq3JvkK8HXg2wxO0d5RXwA2MrgefAmDeYF/rqq+keTNwCe6cP8XBj3sW4D35RcfZrtdj3yBtzAIxO8wOHV/wLx1VzEIxhngd6vqn5P8DYNr31/O4Bzx9xlcI55vsW3OZnB9ehuDU+sXL1LTecAru/3Mb/OOJG8EPsOgp3x+VX0UIMmfde24FvgGcNOQ/b4uyToGnwf4BvDxqvpJkscAW5L8lMFMV6cs8bzMeSHwru51uCuDebq/2q07CvijIY+RJsLZwaQJSrIWeENVPWfatUxLks8Bz6mqG0fcflVV7ex63GcDp1fV2Us9rg9JHgv8QVW9aBrHl8BT5ZImbwPdvPUjelt3RuQK4B+Bj/RS1WgOZtBjl6bGHrckSQ2xxy1JUkMMbkmSGmJwS5LUEINbkqSGGNySJDXE4JYkqSH/Hyo33COVdyi0AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "utils.plot_global_worker_dist(worker_dist)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Examples for the paper" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "def sample_sst_examples_by_group(df, sst_labels, gold_labels, sampsize=0, random_state=1):\n", " sf = df[(df.gold_label.isin(gold_labels)) & (df['SST-3'].isin(sst_labels))].copy()\n", " sf['responses'] = sf['label_distribution'].apply(utils._format_dist)\n", " if sampsize == 0:\n", " samp = sf\n", " else:\n", " samp = sf.sample(sampsize, random_state=random_state)\n", " samp = samp[['sentence', 'SST-3', 'responses']].reset_index(drop=True)\n", " return samp.sort_values('SST-3')" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentenceSST-3responses
0Moretti's compelling anatomy of grief and the difficult process of adapting to loss.negativeneu, pos, pos, pos, pos
1Nothing is sacred in this gut-buster.negativeneg, neg, pos, pos, pos
\n", "
" ], "text/plain": [ " sentence \\\n", "0 Moretti's compelling anatomy of grief and the difficult process of adapting to loss. \n", "1 Nothing is sacred in this gut-buster. \n", "\n", " SST-3 responses \n", "0 negative neu, pos, pos, pos, pos \n", "1 negative neg, neg, pos, pos, pos " ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_sst_examples_by_group(\n", " df,\n", " sst_labels=['negative'],\n", " gold_labels=['positive'],\n", " sampsize=0)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentenceSST-3responses
0... routine, harmless diversion and little else.positivemix, mix, neg, neg, neg
1Hilariously inept and ridiculous.positivemix, neg, neg, neg, neg
2Reign of Fire looks as if it was made without much thought--and is best watched that way.positivemix, neg, neg, neg, neg
3So much facile technique, such cute ideas, so little movie.positivemix, mix, neg, neg, neg
4While there's something intrinsically funny about Sir Anthony Hopkins saying ` Get in the car, bitch ,' this Jerry Bruckheimer production has little else to offerpositivemix, neg, neg, neg, neg
\n", "
" ], "text/plain": [ " sentence \\\n", "0 ... routine, harmless diversion and little else. \n", "1 Hilariously inept and ridiculous. \n", "2 Reign of Fire looks as if it was made without much thought--and is best watched that way. \n", "3 So much facile technique, such cute ideas, so little movie. \n", "4 While there's something intrinsically funny about Sir Anthony Hopkins saying ` Get in the car, bitch ,' this Jerry Bruckheimer production has little else to offer \n", "\n", " SST-3 responses \n", "0 positive mix, mix, neg, neg, neg \n", "1 positive mix, neg, neg, neg, neg \n", "2 positive mix, neg, neg, neg, neg \n", "3 positive mix, mix, neg, neg, neg \n", "4 positive mix, neg, neg, neg, neg " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_sst_examples_by_group(\n", " df,\n", " sst_labels=['positive'],\n", " gold_labels=['negative'],\n", " sampsize=0)" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentenceSST-3responses
0Returning aggressively to his formula of dimwitted comedy and even dimmer characters, Sandler, who also executive produces, has made a film that makes previous vehicles look smart and sassy.neutralneg, neg, neg, neg, neg
1should be seen at the very least for its spasms of absurdist humor.neutralpos, pos, pos, pos, pos
2A workshop mentality prevails.neutralneu, neu, neu, neu, neu
3Van Wilder brings a whole new meaning to the phrase ` comedy gag . 'neutralmix, neu, pos, pos, pos
4` They' begins and ends with scenes so terrifying I'm still stunned.neutralneu, neu, pos, pos, pos
5Barely gets off the ground.neutralneg, neg, neg, neg, neg
6As a tolerable diversion, the film suffices; a Triumph, however, it is not.neutralmix, mix, mix, mix, neg
7Christina Ricci comedy about sympathy, hypocrisy and love is a misfire.neutralneg, neg, neg, neg, neg
8Jacquot's rendering of Puccini's tale of devotion and double-cross is more than just a filmed opera.neutralneg, neu, pos, pos, pos
9Candid Camera on methamphetamines.neutralneg, neg, neg, neu, pos
\n", "
" ], "text/plain": [ " sentence \\\n", "0 Returning aggressively to his formula of dimwitted comedy and even dimmer characters, Sandler, who also executive produces, has made a film that makes previous vehicles look smart and sassy. \n", "1 should be seen at the very least for its spasms of absurdist humor. \n", "2 A workshop mentality prevails. \n", "3 Van Wilder brings a whole new meaning to the phrase ` comedy gag . ' \n", "4 ` They' begins and ends with scenes so terrifying I'm still stunned. \n", "5 Barely gets off the ground. \n", "6 As a tolerable diversion, the film suffices; a Triumph, however, it is not. \n", "7 Christina Ricci comedy about sympathy, hypocrisy and love is a misfire. \n", "8 Jacquot's rendering of Puccini's tale of devotion and double-cross is more than just a filmed opera. \n", "9 Candid Camera on methamphetamines. \n", "\n", " SST-3 responses \n", "0 neutral neg, neg, neg, neg, neg \n", "1 neutral pos, pos, pos, pos, pos \n", "2 neutral neu, neu, neu, neu, neu \n", "3 neutral mix, neu, pos, pos, pos \n", "4 neutral neu, neu, pos, pos, pos \n", "5 neutral neg, neg, neg, neg, neg \n", "6 neutral mix, mix, mix, mix, neg \n", "7 neutral neg, neg, neg, neg, neg \n", "8 neutral neg, neu, pos, pos, pos \n", "9 neutral neg, neg, neg, neu, pos " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_sst_examples_by_group(\n", " df,\n", " sst_labels=['neutral'],\n", " gold_labels=['positive', 'negative', 'mixed', 'neutral'],\n", " sampsize=10)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }