{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# LightTag API \n", "\n", "Welcome to the LightTag API tutorial. This tutorial consists of 3 parts\n", "1. Part 1 covers quickly defining schemas, datasets and tasks\n", "2. Part 2 covers retreiving metrics about your projects\n", "3. Part 3 covers updating configuratons and appending data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "LightTag allows you to annotate a **Dataset** with a **Schema**. A combination of the two, together with some paramaters is called a **Task Definition** (TD for short). Your Datasets,Schemas and Task Definitions live inside of a **Project**. \n", "\n", "The **Examples** you will be annotating live inside of the Dataset. During annotation, you will apply a **Tag** to parts of an Example, classify the overall Example with **ClassificationTypes** or both. Tags and Classification Types live inside of a Schema. \n", "With exception of a Task Definition, each of these objects has a name, through which you can access it. (Task Definitions only have ids). \n", "\n", "Thus, the URL structure follows a nested hierarchy \n", "- **/api/v1/projects/** All of the projects-\n", "- **/api/v1/projects/some-project/** Details about the project named \"some-project\"\n", "- **/api/v1/projects/some-project/datasets/** All of the Datasets that belong to \"some-project\"\n", "- **/api/v1/projects/some-project/datasets/dataset-1/** Details about the dataset \"dataset-1\" which belongs to \"some-project\"\n", "- **/api/v1/projects/some-project/schemas/** All of the Schemas that belong to \"some-project\"\n", "- **/api/v1/projects/some-project/schemas/schema-1/** Details about the dataset \"dataset-1\" which belongs to \"some-project\"\n", "\n", "etc.. (The detailed endpoints are enumerated in part 3\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A note on naming\n", "You are free to name your datasets,projects schemas and tags as you wish. To ensure a consistent URL, LightTag will calculate a [Slug](https://en.wikipedia.org/wiki/Clean_URL#Slug). \n", "\n", "For example, if we were to create a dataset called *I am a Dataset;;-100* it's slug would be *i-am-a-dataset-100* and it's url would be **/api/v1/projects/some-project/datasets/i-am-a-dataset-100/ **\n", "\n", "You can always find the Slug and the URL for a resource by querying it's parent, for example querying all datasets of a project will show you the slug and url for each dataset,as we'll see shortly\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A note on Projects\n", "At the time of writing, the LightTag API supports multiple projects but the built in UI supports a single project, the **default project**, whose url is /api/v1/projects/default/. \n", "\n", "As you use the API, DO NOT define a new project if you intend to consume the data via the UI, instead use the default project. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 0 - Authentication\n", "LightTag's authentication is via a token which you acquire with your username and password. \n", "Most endpoints can only be accessed by a manager role, so make sure you are using your manager role when following this example" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "import requests\n", "import pandas as pd\n", "\n", "LIGHTTAG_DOMAIN = 'demo' #should be your lighttag domain\n", "SERVER = 'https://{domain}.lighttag.io/api/'.format(domain=LIGHTTAG_DOMAIN) \n", "API_BASE = SERVER +'v1/'\n", "MY_USER='YOUR_USER_HERE'\n", "MY_PWD='MY_PWSED_HERE'" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "response = requests.post(SERVER+'auth/token/login/',\n", " json={\"username\":MY_USER,\"password\":MY_PWD})\n", "assert response.status_code ==200, \"Couldn't authenticate\"\n", "auth_details = response.json()\n", "token = auth_details['key']\n", "assert auth_details['is_manager'] ==1, \"not a manager\" # Check you are a manager\n" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'slug': 'default',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/',\n", " 'name': 'default'}]" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#It's convenient to set up a requests session instead of repeating tokens\n", "session = requests.session()\n", "session.headers.update({\"Authorization\":\"Token {token}\".format(token=token)})\n", "#Try it out\n", "session.get(API_BASE+'projects/').json()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Great! We are authenticted and can see the default project" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 1 - Quickly setting up a project\n", "Adhering to the UI onboarding process, LightTag's API allows you to define a Dataset, Schema and Tasks in bulk, instead of uploading individual tags or examples one by one. This section will go over how we do that. We'll follow the following hypothetical use case:\n", "\n", "We'd like to have a *team* of annotators label President Trump's tweets. We want to label both entities in the tweet (Person,Location...) as well as the sentiment of the tweet, e.g. a classification.\n", "\n", "We'd like to have a very high quality test set and a larger training set, for which we'll tolerate some noise. \n", "\n", "Before we go ahead and label the whole thing, we'd to do some experiments: First to see if our Schema makes sense to our labelers, and second to see if it is more convenient to label entities and sentiment together, or do it as individual tasks. \n", "\n", "To achieve this we'll follow the following steps:\n", "\n", "1. Split our dataset into a traing and test set\n", "\n", "2. Set aside part of the training set for experimentation\n", "\n", "3. Define and upload our Datasets\n", "\n", "4. Define and upload our Schema\n", "\n", "5. Defining Teams\n", "\n", "5. Define the tasks (small evaluation, training set labeling, test set labeling)\n", "\n", "In ourder to keep this section focused on quickly uploading, we will hold over evalutation till section 3. In a usual use case you will probably first upload a small evaluation set, review the results, iterate and then set out on labeling the full training/test sets\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1 Preparing our dataset\n", "We've conveniently prepared a collection of President Trump's tweets in a JSON file. These will be our examples. We'll load them and divide them into a training set, test set and further take a slice of the training set for exploration " ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'created_at': 'Mon Jan 01 13:37:52 +0000 2018',\n", " 'date': '2018-01-01',\n", " 'favorite_count': 54056,\n", " 'id_str': '947824196909961216',\n", " 'in_reply_to_user_id_str': None,\n", " 'is_retweet': False,\n", " 'retweet_count': 8656,\n", " 'source': 'Twitter for iPhone',\n", " 'text': 'Will be leaving Florida for Washington (D.C.) today at 4:00 P.M. '\n", " 'Much work to be done, but it will be a great New Year!',\n", " 'time': 1514813872000000000}\n", "'total of 2605 examples'\n" ] } ], "source": [ "import json\n", "from pprint import pprint\n", "\n", "all_data = json.load(open('./trump3.json'))\n", "pprint(all_data[0])\n", "pprint(\"total of {num} examples\".format(num=len(all_data)))\n" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "train,test = all_data[:2000], all_data[2000:] # 2000 train examples, 600 test examples\n", "exploratory = train[:50] # Take 50 examples from the training set for exploratory work" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Uploading our exploratory dataset\n", "To upload our dataset, we send over the whole JSON object. We need to tell LightTag which field contains the text to be annotated. Additionaly, we need to specify\n", "1. A name for the dataset\n", "2. An optional aggregation field (If specified, all examples with the value will be shown together)\n", "3. An optional sort field (If secified with an aggregation field, examples will be displayed ordered by that field)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "exploratory_dataset = {\n", " \"name\":\"Exploratory Dataset1\",\n", " \"content_field\":\"text\", # text is the field in the JSON we will be annotating\n", " \"examples\" :exploratory # Set the list of examples that are part of this dataset\n", "}" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "#POST the dataset definition and the examples \n", "resp= session.post(API_BASE+'projects/default/datasets/bulk/',json=exploratory_dataset)\n", "assert resp.status_code ==201" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'id': '2a6e0c06-017d-473c-b3d5-bbb24c28b71c',\n", " 'slug': 'bible',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/bible/',\n", " 'name': 'Bible',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': 'chapter',\n", " 'order_field': 'verse',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '1bf62f49-d88e-4bf0-9178-79012ce355c4',\n", " 'slug': 'drilling-comments-small',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/drilling-comments-small/',\n", " 'name': 'Drilling Comments Small',\n", " 'id_field': 'FIELD1',\n", " 'content_field': 'FIELD2',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '1709c71b-2092-4d6b-800c-1341cae915ec',\n", " 'slug': 'some-new-dataset',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/some-new-dataset/',\n", " 'name': 'Some new dataset',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'f620a31a-c907-43cb-9f66-6642736d4b4f',\n", " 'slug': 'jabberwocky',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/jabberwocky/',\n", " 'name': 'Jabberwocky',\n", " 'id_field': 'id',\n", " 'content_field': 'data',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'e04f9a51-acb4-44a4-b633-0d03fc6f7af4',\n", " 'slug': 'test',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/test/',\n", " 'name': 'Test',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': 'chapter',\n", " 'order_field': 'verse',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '24408e1f-dfe8-4d37-8f42-7192de1956e5',\n", " 'slug': 'hebrew-medical',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/hebrew-medical/',\n", " 'name': 'hebrew medical',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '045896b7-6e3e-427f-b920-7241da0088ae',\n", " 'slug': 'trump-tweets',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/trump-tweets/',\n", " 'name': 'Trump Tweets',\n", " 'id_field': None,\n", " 'content_field': 'text',\n", " 'aggregation_field': 'date',\n", " 'order_field': 'time',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'ff31041a-69ac-4641-8f7f-6cad8bd257b7',\n", " 'slug': 'oaa-test',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/oaa-test/',\n", " 'name': 'öäå test',\n", " 'id_field': None,\n", " 'content_field': 'Message',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '4ddb421d-28b0-42a5-bb5c-54509b35d27b',\n", " 'slug': 'format-test',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/format-test/',\n", " 'name': 'format test',\n", " 'id_field': None,\n", " 'content_field': 'Message',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'f44b27f2-e84b-441b-b6e9-ff7049111b6f',\n", " 'slug': 'zxzx',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/zxzx/',\n", " 'name': 'zxzx',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': 'chapter',\n", " 'order_field': 'verse',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '8e9246b1-ebf4-47df-8443-e81dc4a0dbb5',\n", " 'slug': 'enrontest',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/enrontest/',\n", " 'name': 'EnronTest',\n", " 'id_field': 'SortSubKey',\n", " 'content_field': 'Line',\n", " 'aggregation_field': 'FileName',\n", " 'order_field': 'SortSubKey',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '79575f45-a38f-4150-9dfe-79fe2f738896',\n", " 'slug': 'enron2',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/enron2/',\n", " 'name': 'Enron2',\n", " 'id_field': 'SortSubKey',\n", " 'content_field': 'Line',\n", " 'aggregation_field': 'FileName',\n", " 'order_field': 'SortSubKey',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'ca83c74d-a95c-4f6c-aeb4-a0eb53f1405e',\n", " 'slug': 'enron3',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/enron3/',\n", " 'name': 'Enron3',\n", " 'id_field': 'SortSubKey',\n", " 'content_field': 'Line',\n", " 'aggregation_field': 'FileName',\n", " 'order_field': 'SortSubKey',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'd70d7152-e339-42d3-9f52-b24f2dd3217d',\n", " 'slug': 'falafel1',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/falafel1/',\n", " 'name': 'Falafel1',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'c7b6e5d7-8ab4-4daa-a1d1-cd8674f730ac',\n", " 'slug': 'bibletest',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/bibletest/',\n", " 'name': 'bibletest',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': 'chapter',\n", " 'order_field': 'verse',\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '847b6cd8-aa4f-44d8-862d-44cd86a0e370',\n", " 'slug': 'hebrew2',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/hebrew2/',\n", " 'name': 'Hebrew2',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': 'verse',\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '9735d41b-c4d8-4db2-b70b-aaf984527d29',\n", " 'slug': 'deleteme',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/deleteme/',\n", " 'name': 'DeleteMe',\n", " 'id_field': None,\n", " 'content_field': 'title',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '61b75760-d1cb-4029-a74f-7c6a14365589',\n", " 'slug': 'demo-for-noa',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/demo-for-noa/',\n", " 'name': 'Demo For Noa',\n", " 'id_field': None,\n", " 'content_field': 'sentence',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '796bdfbc-ef73-4513-bb8b-08c5179ab3b1',\n", " 'slug': 'exploratory-dataset',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/exploratory-dataset/',\n", " 'name': 'Exploratory Dataset',\n", " 'id_field': None,\n", " 'content_field': 'text',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '841a5ae8-c3b5-4b51-945d-1fdcda2e9cee',\n", " 'slug': 'lighttag-sample-dataset',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/lighttag-sample-dataset/',\n", " 'name': 'LightTag Sample Dataset',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '066bc411-e5ea-4fe1-8d80-ced124d16363',\n", " 'slug': 'jeoperdy-async',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/jeoperdy-async/',\n", " 'name': 'Jeoperdy Async',\n", " 'id_field': None,\n", " 'content_field': 'question',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'starting',\n", " 'archived': False},\n", " {'id': '2d78f91d-a480-40e3-b77f-408981a46b5a',\n", " 'slug': 'lighttag-sample-datasetbible',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/lighttag-sample-datasetbible/',\n", " 'name': 'LightTag Sample DatasetBible',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'a47006a3-404c-457f-99c7-1f5133311bf5',\n", " 'slug': 'my-bible-dataset',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/my-bible-dataset/',\n", " 'name': 'My Bible Dataset',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'b31a6ed1-de5e-4346-9de9-a0f1abc10894',\n", " 'slug': 'ny_times_data',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/ny_times_data/',\n", " 'name': 'ny_times_data',\n", " 'id_field': None,\n", " 'content_field': 'tweet',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'ce522344-82c1-4209-a7a0-2b33baadbf74',\n", " 'slug': 'biblefdwe',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/biblefdwe/',\n", " 'name': 'biblefdwe',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': 'chapter',\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': '8f5bd425-ae8c-45f2-9cdd-f297ae6a5806',\n", " 'slug': 'tweets',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/tweets/',\n", " 'name': 'tweets',\n", " 'id_field': None,\n", " 'content_field': 'text',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'ba785877-2dcd-418b-ba54-ff3d55a72e99',\n", " 'slug': 'the-bible-english',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/the-bible-english/',\n", " 'name': 'The Bible (English)',\n", " 'id_field': None,\n", " 'content_field': 'content',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'done',\n", " 'archived': False},\n", " {'id': 'aa690b22-3f51-4a77-bb12-6b16e4e08e63',\n", " 'slug': 'exploratory-dataset1',\n", " 'url': 'https://demo.lighttag.io/api/v1/projects/default/datasets/exploratory-dataset1/',\n", " 'name': 'Exploratory Dataset1',\n", " 'id_field': None,\n", " 'content_field': 'text',\n", " 'aggregation_field': None,\n", " 'order_field': None,\n", " 'project_id': '2753ca38-69d9-4c96-9d31-df6d4069b027',\n", " 'upload_status': 'started',\n", " 'archived': False}]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "datasets = session.get(API_BASE+'projects/default/datasets/').json() # Get all of the datasets in our project\n", "datasets" ] }, { "cell_type": "code", "execution_count": 154, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'aggregation_value': None,\n", " 'content': 'RT @shawgerald4: @realDonaldTrump Thank you President TRUMP!! https://t.co/LKdkT0FL99',\n", " 'dataset': 'ee688ba1-b0d3-4fdb-9355-d7924e3875e4',\n", " 'id': '4f028792-9c84-42be-99e5-09db63524687',\n", " 'metadata': {'created_at': 'Sun Dec 24 12:31:48 +0000 2017',\n", " 'date': '2017-12-24',\n", " 'favorite_count': 0,\n", " 'id_str': '944908467499884544',\n", " 'in_reply_to_user_id_str': None,\n", " 'is_retweet': True,\n", " 'retweet_count': 11846,\n", " 'source': 'Twitter for iPhone',\n", " 'time': 1514118708000000000}},\n", " {'aggregation_value': None,\n", " 'content': 'The Fake News refuses to talk about how Big and how Strong our BASE is. They show Fake Polls just like they report Fake News. Despite only negative reporting, we are doing well - nobody is going to beat us. MAKE AMERICA GREAT AGAIN!',\n", " 'dataset': 'ee688ba1-b0d3-4fdb-9355-d7924e3875e4',\n", " 'id': '20e8dd31-5e45-44b5-94aa-bb0d6699609f',\n", " 'metadata': {'created_at': 'Sun Dec 24 13:48:11 +0000 2017',\n", " 'date': '2017-12-24',\n", " 'favorite_count': 140299,\n", " 'id_str': '944927689638662145',\n", " 'in_reply_to_user_id_str': None,\n", " 'is_retweet': False,\n", " 'retweet_count': 33220,\n", " 'source': 'Twitter for iPhone',\n", " 'time': 1514123291000000000}}]" ] }, "execution_count": 154, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#See the examples in our dataset\n", "session.get(datasets[0]['url']+'examples/').json()[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Uploading with an Aggregation field\n", "As mentioned, we can tell LightTag to show examples together, and in what order by specifying an aggregation key and order key. Let's do that for the test and train set" ] }, { "cell_type": "code", "execution_count": 155, "metadata": {}, "outputs": [], "source": [ "train_dataset = {\n", " \"name\":\"Training Set\",\n", " \"content_field\":\"text\", # text is the field in the JSON we will be annotating\n", " \"examples\" :train, # Set the list of examples that are part of this dataset\n", " \"aggregation_field\":\"date\", # Show all tweets from the same day together\n", " \"order_field\":\"time\" # And sort them by time\n", "}\n", "\n", "test_dataset = {\n", " \"name\":\"Test Set\",\n", " \"content_field\":\"text\", # text is the field in the JSON we will be annotating\n", " \"examples\" :test, # Set the list of examples that are part of this dataset\n", " \"aggregation_field\":\"date\", # Show all tweets from the same day together\n", " \"order_field\":\"time\" # And sort them by time\n", "}" ] }, { "cell_type": "code", "execution_count": 156, "metadata": {}, "outputs": [], "source": [ "# Upload the training set\n", "resp = session.post(API_BASE+'projects/default/datasets/bulk/',json=train_dataset)\n", "assert resp.status_code ==201\n", "# Upload the test set\n", "resp = session.post(API_BASE+'projects/default/datasets/bulk/',json=test_dataset)\n", "assert resp.status_code ==201\n" ] }, { "cell_type": "code", "execution_count": 157, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[{'aggregation_field': 'date',\n", " 'content_field': 'text',\n", " 'id': '92ea39c8-1d7d-47eb-b6d3-25ceebc6ba33',\n", " 'id_field': None,\n", " 'name': 'Test Set',\n", " 'order_field': 'time',\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'slug': 'test-set',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/datasets/test-set/'},\n", " {'aggregation_field': 'date',\n", " 'content_field': 'text',\n", " 'id': 'de7ed65d-e685-4ea9-bf3b-0c3c10f20b46',\n", " 'id_field': None,\n", " 'name': 'Training Set',\n", " 'order_field': 'time',\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'slug': 'training-set',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/datasets/training-set/'},\n", " {'aggregation_field': None,\n", " 'content_field': 'text',\n", " 'id': 'ee688ba1-b0d3-4fdb-9355-d7924e3875e4',\n", " 'id_field': None,\n", " 'name': 'Exploratory Dataset',\n", " 'order_field': None,\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'slug': 'exploratory-dataset',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/datasets/exploratory-dataset/'}]\n" ] } ], "source": [ "# Let's see all of the Datasets in our project\n", "datasets = session.get(API_BASE+'projects/default/datasets/').json() # Get all of the datasets in our project\n", "pprint(datasets)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Defining our Schemas\n", "The schema defines how we label our examples. A schema can either define classification types for classifying an example, tags for tagging parts of an example or both. We'll define all three options" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Defining Classification types\n", "We simply define a list of classification types, each item has a name (The class) and a description. The labelers will be able to see the description as they work. \n", "Remember to give your annotators an UNK or Confusing class. " ] }, { "cell_type": "code", "execution_count": 158, "metadata": {}, "outputs": [], "source": [ "\n", "classification_types=[\n", " {\n", " \"name\":\"Insult\", \n", " \"description\":\"This tweet contains an insult to a particular person\"\n", " },\n", " {\n", " \"name\":\"Praise\", \n", " \"description\":\"This tweet contains praise of a particular person\"\n", " },\n", " {\n", " \"name\":\"Confusing\", \n", " \"description\":\"This tweet is ambiguous\"\n", " },\n", " {\n", " \"name\":\"UNK\", \n", " \"description\":\"This tweet is niether an insult or praise\"\n", " },\n", "]\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Defining tags\n", "We simply define a list of tags, each item has a name (The tag) and a description. The labelers will be able to see the description as they work. \n", " " ] }, { "cell_type": "code", "execution_count": 159, "metadata": {}, "outputs": [], "source": [ "tags=[\n", " {\n", " \"name\":\"Person\", \n", " \"description\":\"The proper name of a person (John is a person. He is not)\"\n", " },\n", " {\n", " \"name\":\"Place\",\n", " \"description\": \"A physical place. For example the White House.\"\n", " },\n", " {\n", " \"name\":\"Issue\",\n", " \"description\": \"A political issue the President is discussing.\"\n", " },\n", " {\n", " \"name\":\"Insult\",\n", " \"description\": \"A word or phrase that is unsulting.\"\n", " },\n", " {\n", " \"name\":\"Media organization\",\n", " \"description\": \"CNN/Fox etc.\"\n", " },\n", " {\n", " \"name\":\"Politcal Group\",\n", " \"description\": \"Republicans/Democrates etc\"\n", " },\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Making a Schema\n", "To define a schema, we give it a name and specify classification types, tags or both. " ] }, { "cell_type": "code", "execution_count": 160, "metadata": {}, "outputs": [], "source": [ "classification_only_schema= {\n", " \"name\" :\"Trump Insult Classification\",\n", " \"classification_types\":classification_types,\n", "}\n", "\n", "tags_only_schema ={\n", " \"name\":\"Entity Tags only\",\n", " \"tags\":tags\n", "}\n", "\n", "tags_and_classifications_schema = {\n", " \"name\": \"Classifications and tags Schema\",\n", " \"tags\":tags,\n", " \"classification_types\":classification_types\n", "}\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Uploading the schema\n", "We simply post the schema definition to the projects /schemas/bulk/ endpoint" ] }, { "cell_type": "code", "execution_count": 161, "metadata": {}, "outputs": [], "source": [ "\n", "resp = session.post(API_BASE+'projects/default/schemas/bulk/',json=classification_only_schema)\n", "assert resp.status_code ==201\n", "\n", "resp = session.post(API_BASE+'projects/default/schemas/bulk/',json=tags_only_schema)\n", "assert resp.status_code ==201\n", "\n", "resp = session.post(API_BASE+'projects/default/schemas/bulk/',json=tags_and_classifications_schema)\n", "assert resp.status_code ==201" ] }, { "cell_type": "code", "execution_count": 162, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'id': '3b73c2b2-73aa-4c42-a265-a0c459abd295',\n", " 'name': 'Classifications and tags Schema',\n", " 'slug': 'classifications-and-tags-schema',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/schemas/classifications-and-tags-schema/'},\n", " {'id': 'fbe6fd1d-bb59-4909-8a88-81772cc0d996',\n", " 'name': 'Entity Tags only',\n", " 'slug': 'entity-tags-only',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/schemas/entity-tags-only/'},\n", " {'id': '9dcf0091-d82c-4292-a09d-eaff062fc1c4',\n", " 'name': 'Trump Insult Classification',\n", " 'slug': 'trump-insult-classification',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/schemas/trump-insult-classification/'}]" ] }, "execution_count": 162, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Indeed we now have three schemas\n", "(session.get(API_BASE+'projects/default/schemas/').json())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**WAIT**\n", "If you are following this tutorial in order to use your own suggestions, you might want to pause, go to the suggestions section and then come back" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Defining Teams\n", "LightTag allows us to assign specific work to one or more teams. Thus, we must specify at least one team that will work on a task. LightTag automatically provides a team that holds \"Everyone\", e.g. all of your annotators and you can define more teams.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Getting a list of existing teams\n", "We can query the teams end point of a project to see the availble teams" ] }, { "cell_type": "code", "execution_count": 163, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'description': '',\n", " 'id': 'dd40e50b-2ffa-4943-b5ca-6983dba36331',\n", " 'members': [{'id': 1, 'username': 'demo'}],\n", " 'name': 'Everyone',\n", " 'slug': 'everyone',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/everyone/'}]" ] }, "execution_count": 163, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(session.get(API_BASE+'projects/default/teams/').json())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating a new team\n", "Sometimes we want to have seperate teams. Two common use cases are\n", "1. Limiting access to certain datasets to specific teams\n", "2. Having \"experts\" label the test set and \"experts\" + \"layman\" label the training set\n", "\n", "To create a new team, we'll provide a list of annotator ids, a name and a description to the teams endpoint\n", "#### First we get a list of availble annotators" ] }, { "cell_type": "code", "execution_count": 166, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'id': 1, 'username': 'demo'},\n", " {'id': 2, 'username': 'Bob'},\n", " {'id': 3, 'username': 'Hanz'},\n", " {'id': 4, 'username': 'Franz'}]" ] }, "execution_count": 166, "metadata": {}, "output_type": "execute_result" } ], "source": [ "annotators = (session.get(API_BASE+'projects/default/annotators/').json())\n", "annotators" ] }, { "cell_type": "code", "execution_count": 167, "metadata": {}, "outputs": [], "source": [ "experts = annotators[:3]\n", "layman = annotators[3:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Now we define our teams\n" ] }, { "cell_type": "code", "execution_count": 168, "metadata": {}, "outputs": [], "source": [ "expertTeam = {\"name\":\"Experts\", \"description\": \"People who are very knowledgable about twitter\", \"members\":experts}\n", "layManTeam = {\"name\":\"Layman\", \"description\": \"People who know a little about twitter\", \"members\":layman}" ] }, { "cell_type": "code", "execution_count": 169, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[{'description': '',\n", " 'id': 'dd40e50b-2ffa-4943-b5ca-6983dba36331',\n", " 'members': [{'id': 1, 'username': 'demo'},\n", " {'id': 2, 'username': 'Bob'},\n", " {'id': 3, 'username': 'Hanz'},\n", " {'id': 4, 'username': 'Franz'}],\n", " 'name': 'Everyone',\n", " 'slug': 'everyone',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/everyone/'},\n", " {'description': 'People who are very knowledgable about twitter',\n", " 'id': 'd2648770-7886-4c41-ad7d-b1f6f713ffbd',\n", " 'members': [{'id': 1, 'username': 'demo'},\n", " {'id': 2, 'username': 'Bob'},\n", " {'id': 3, 'username': 'Hanz'}],\n", " 'name': 'Experts',\n", " 'slug': 'experts',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/experts/'},\n", " {'description': 'People who know a little about twitter',\n", " 'id': '044b04b5-7cdd-404a-8b00-1c7b61d50474',\n", " 'members': [{'id': 4, 'username': 'Franz'}],\n", " 'name': 'Layman',\n", " 'slug': 'layman',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/layman/'}]\n" ] } ], "source": [ "session.post(API_BASE+'projects/default/teams/',json=expertTeam)\n", "session.post(API_BASE+'projects/default/teams/',json=layManTeam)\n", "teams = session.get(API_BASE+'projects/default/teams/').json()\n", "pprint(teams)" ] }, { "cell_type": "code", "execution_count": 170, "metadata": {}, "outputs": [], "source": [ "teamIds = [x['id'] for x in teams]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Defining Tasks\n", "Now that we have Datasets and tasks we can define the work we want to do on them. \n", "As we said earlier, we want to do a quick exploratory job, and label the exploratory dataset with each of the schemas to see which works best. \n", "\n", "Then we want to define work on the training and test sets. \n", "For exploration, we'll use 2 labelers per example and suggestions. \n", "For the training set we'll use 1 labeler per example and suggestions. \n", "For the test set, we'll use 3 labelers per example and disable suggestions - so that we don't introduce any bias" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using Slugs\n", "To specify a task we must refer to a dataset and a schema. In the dataset definition we do this by referencing the **slug** of the schema/dataset. " ] }, { "cell_type": "code", "execution_count": 171, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "classifications-and-tags-schema \t\t Classifications and tags Schema\n", "entity-tags-only \t\t Entity Tags only\n", "trump-insult-classification \t\t Trump Insult Classification\n" ] } ], "source": [ "resp = session.get(API_BASE+'projects/default/schemas/') # fetch all of the schemas\n", "for schema in resp.json():\n", " print(schema['slug'],\"\\t\\t\",schema['name'])" ] }, { "cell_type": "code", "execution_count": 172, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "test-set \t\t Test Set\n", "training-set \t\t Training Set\n", "exploratory-dataset \t\t Exploratory Dataset\n" ] } ], "source": [ "resp = session.get(API_BASE+'projects/default/datasets/') # fetch all of the schemas\n", "for dataset in resp.json():\n", " print(dataset['slug'],'\\t\\t',dataset['name'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Now we can define our tasks\n", "For each task we want, we specify the schema, dataset, how many annotators and if suggestions are availble or not. \n", "We can also pass a Markdown text as guidelines for the annotators" ] }, { "cell_type": "code", "execution_count": 173, "metadata": {}, "outputs": [], "source": [ "exploratory_classifications_only_task = {\n", " \"name\":\"Explore classification only\",\n", " \"dataset_slug\":\"exploratory-dataset\",\n", " \"schema_slug\":\"trump-insult-classification\",\n", " \"annotators_per_example\":2,\n", " \"allow_suggestions\":True,\n", " \"guidelines\":\"### Example Guidelines \\n This is an example\",\n", " \"teams\":[teamIds[0]],\n", " \"models\":[] # we set models to a blank list. See the Documentation on suggestions for associating a model to a task\n", "}\n", "\n", "exploratory_tags_only_task = {\n", " \"name\": \"Explore tags only\",\n", " \"dataset_slug\":\"exploratory-dataset\",\n", " \"schema_slug\":\"entity-tags-only\", # Changed the schema id\n", " \"annotators_per_example\":2,\n", " \"allow_suggestions\":True,\n", " \"teams\":[teamIds[0]],\n", " \"models\":[]\n", "}\n", "\n", "exploratory_tags_and_classifications_task = {\n", " \"name\": \"Explore tags and classes\",\n", " \"dataset_slug\":\"exploratory-dataset\",\n", " \"schema_slug\":\"classifications-and-tags-schema\", # Changed the schema id\n", " \"annotators_per_example\":2,\n", " \"allow_suggestions\":True,\n", " \"teams\":[teamIds[1]],\n", " \"models\":[]\n", "}\n", "\n", "training_set_task = {\n", " \"name\":\"training set tags and classes\",\n", " \"dataset_slug\":\"training-set\",\n", " \"schema_slug\":\"classifications-and-tags-schema\", # Changed the schema id\n", " \"annotators_per_example\":1, # Only one labeler per example\n", " \"allow_suggestions\":True,\n", " \"teams\":[teamIds[2]],\n", " \"models\":[]\n", "}\n", "\n", "test_set_task = {\n", " \"name\":\"test set tags and classes\",\n", " \"dataset_slug\":\"test-set\",\n", " \"schema_slug\":\"classifications-and-tags-schema\", # Changed the schema id\n", " \"annotators_per_example\":3, # Three labelers per example\n", " \"allow_suggestions\":True,\n", " \"teams\":[teamIds[2],teamIds[1]],\n", " \"models\":[]\n", "}\n", "\n", "tasks = [test_set_task,training_set_task,exploratory_classifications_only_task, exploratory_tags_only_task,\n", " exploratory_tags_and_classifications_task]" ] }, { "cell_type": "code", "execution_count": 174, "metadata": {}, "outputs": [], "source": [ "for task_definition in tasks:\n", " resp =session.post(API_BASE+'projects/default/task_definitions/',json=task_definition)\n", " assert resp.status_code==201, (task_definition)" ] }, { "cell_type": "code", "execution_count": 175, "metadata": {}, "outputs": [], "source": [ "# And we can see our task definitions on the server" ] }, { "cell_type": "code", "execution_count": 176, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'active': True,\n", " 'allow_suggestions': True,\n", " 'annotators_per_example': 2,\n", " 'dataset_id': 'ee688ba1-b0d3-4fdb-9355-d7924e3875e4',\n", " 'dataset_slug': 'exploratory-dataset',\n", " 'guidelines': None,\n", " 'id': '9bab798d-c8db-43e8-9d4e-bf893a2f83f0',\n", " 'name': 'Explore tags and classes',\n", " 'priority': 5,\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'schema_id': '3b73c2b2-73aa-4c42-a265-a0c459abd295',\n", " 'schema_slug': 'classifications-and-tags-schema',\n", " 'slug': 'explore-tags-and-classes',\n", " 'teams': [{'description': 'People who are very knowledgable about twitter',\n", " 'id': 'd2648770-7886-4c41-ad7d-b1f6f713ffbd',\n", " 'members': [{'id': 1, 'username': 'demo'},\n", " {'id': 2, 'username': 'Bob'},\n", " {'id': 3, 'username': 'Hanz'}],\n", " 'name': 'Experts',\n", " 'slug': 'experts',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/experts/'}],\n", " 'url': 'http://localhost:8000/api/v1/projects/default/task_definitions/explore-tags-and-classes/'},\n", " {'active': True,\n", " 'allow_suggestions': True,\n", " 'annotators_per_example': 2,\n", " 'dataset_id': 'ee688ba1-b0d3-4fdb-9355-d7924e3875e4',\n", " 'dataset_slug': 'exploratory-dataset',\n", " 'guidelines': None,\n", " 'id': 'f5fe2bbb-e41b-4cc5-9cd2-8a28f6b37ab4',\n", " 'name': 'Explore tags only',\n", " 'priority': 4,\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'schema_id': 'fbe6fd1d-bb59-4909-8a88-81772cc0d996',\n", " 'schema_slug': 'entity-tags-only',\n", " 'slug': 'explore-tags-only',\n", " 'teams': [{'description': '',\n", " 'id': 'dd40e50b-2ffa-4943-b5ca-6983dba36331',\n", " 'members': [{'id': 1, 'username': 'demo'},\n", " {'id': 2, 'username': 'Bob'},\n", " {'id': 3, 'username': 'Hanz'},\n", " {'id': 4, 'username': 'Franz'}],\n", " 'name': 'Everyone',\n", " 'slug': 'everyone',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/everyone/'}],\n", " 'url': 'http://localhost:8000/api/v1/projects/default/task_definitions/explore-tags-only/'},\n", " {'active': True,\n", " 'allow_suggestions': True,\n", " 'annotators_per_example': 2,\n", " 'dataset_id': 'ee688ba1-b0d3-4fdb-9355-d7924e3875e4',\n", " 'dataset_slug': 'exploratory-dataset',\n", " 'guidelines': '### Example Guidelines \\n This is an example',\n", " 'id': '4bec1ef3-a34d-4ac6-a484-2ee0ec60fc85',\n", " 'name': 'Explore classification only',\n", " 'priority': 3,\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'schema_id': '9dcf0091-d82c-4292-a09d-eaff062fc1c4',\n", " 'schema_slug': 'trump-insult-classification',\n", " 'slug': 'explore-classification-only',\n", " 'teams': [{'description': '',\n", " 'id': 'dd40e50b-2ffa-4943-b5ca-6983dba36331',\n", " 'members': [{'id': 1, 'username': 'demo'},\n", " {'id': 2, 'username': 'Bob'},\n", " {'id': 3, 'username': 'Hanz'},\n", " {'id': 4, 'username': 'Franz'}],\n", " 'name': 'Everyone',\n", " 'slug': 'everyone',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/everyone/'}],\n", " 'url': 'http://localhost:8000/api/v1/projects/default/task_definitions/explore-classification-only/'},\n", " {'active': True,\n", " 'allow_suggestions': True,\n", " 'annotators_per_example': 1,\n", " 'dataset_id': 'de7ed65d-e685-4ea9-bf3b-0c3c10f20b46',\n", " 'dataset_slug': 'training-set',\n", " 'guidelines': None,\n", " 'id': 'ace018ea-e8ee-4e9e-8b2d-1a0418f30878',\n", " 'name': 'training set tags and classes',\n", " 'priority': 2,\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'schema_id': '3b73c2b2-73aa-4c42-a265-a0c459abd295',\n", " 'schema_slug': 'classifications-and-tags-schema',\n", " 'slug': 'training-set-tags-and-classes',\n", " 'teams': [{'description': 'People who know a little about twitter',\n", " 'id': '044b04b5-7cdd-404a-8b00-1c7b61d50474',\n", " 'members': [{'id': 4, 'username': 'Franz'}],\n", " 'name': 'Layman',\n", " 'slug': 'layman',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/layman/'}],\n", " 'url': 'http://localhost:8000/api/v1/projects/default/task_definitions/training-set-tags-and-classes/'},\n", " {'active': True,\n", " 'allow_suggestions': True,\n", " 'annotators_per_example': 3,\n", " 'dataset_id': '92ea39c8-1d7d-47eb-b6d3-25ceebc6ba33',\n", " 'dataset_slug': 'test-set',\n", " 'guidelines': None,\n", " 'id': 'f91a95a4-61e5-40f8-87b4-63103e1f00f0',\n", " 'name': 'test set tags and classes',\n", " 'priority': 1,\n", " 'project_id': 'e488e45a-564a-4b93-8beb-f7aa4c73ea97',\n", " 'schema_id': '3b73c2b2-73aa-4c42-a265-a0c459abd295',\n", " 'schema_slug': 'classifications-and-tags-schema',\n", " 'slug': 'test-set-tags-and-classes',\n", " 'teams': [{'description': 'People who are very knowledgable about twitter',\n", " 'id': 'd2648770-7886-4c41-ad7d-b1f6f713ffbd',\n", " 'members': [{'id': 1, 'username': 'demo'},\n", " {'id': 2, 'username': 'Bob'},\n", " {'id': 3, 'username': 'Hanz'}],\n", " 'name': 'Experts',\n", " 'slug': 'experts',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/experts/'},\n", " {'description': 'People who know a little about twitter',\n", " 'id': '044b04b5-7cdd-404a-8b00-1c7b61d50474',\n", " 'members': [{'id': 4, 'username': 'Franz'}],\n", " 'name': 'Layman',\n", " 'slug': 'layman',\n", " 'url': 'http://localhost:8000/api/v1/projects/default/teams/layman/'}],\n", " 'url': 'http://localhost:8000/api/v1/projects/default/task_definitions/test-set-tags-and-classes/'}]" ] }, "execution_count": 176, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(session.get(API_BASE+'projects/default/task_definitions/').json())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Setting priorities on work\n", "When our annotators log in to LightTag they will be show work to do based on how we prioritize our tasks. Of course we'd like to set that priority. In this section we will set the priorities such that the exploratory tasks will go first, then the test set, then the training set\n", "\n", "***\n", "Note, using the same endpoint we can also update the guidelines for a task definiiton or set it to inactive" ] }, { "cell_type": "code", "execution_count": 177, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Setting explore-tags-and-classes to priority 1\n", "Setting explore-tags-only to priority 2\n", "Setting explore-classification-only to priority 3\n", "Setting training-set-tags-and-classes to priority 4\n", "Setting test-set-tags-and-classes to priority 5\n" ] } ], "source": [ "# First pull all of our tasks\n", "task_definitions = (session.get(API_BASE+'projects/default/task_definitions/').json())\n", "#Now we'll sort them by slug, which will end up being the exact order we wanted \n", "sorted(task_definitions,key=lambda x:x['slug'])\n", "#And now we update their priority\n", "for num,td in enumerate(task_definitions):\n", " print (\"Setting\", td['slug'], \"to priority\", num+1),\n", " resp= session.put(td['url'],json={\"priority\":num+1})\n", " assert resp.status_code == 200, resp.status_code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2 - Metrics and analytics\n", "This section will show how to pull various metrics from LightTag to get a sense of where your project stands.\n", "We'll continue with the work defined in the previous section, though to illustrate the metrics we've randomly annotated the data.\n", "\n", "The current version of the API allows you to see aggregate metrics about your projects, that is, what work has been done, how much of it and what level of agreement has been reached.\n", "\n", "**To make the results prettier, we'll use the pandas library. This is a nice to have but you can easily consume results as raw JSONs **" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How metrics are organized\n", "LightTag's metrics are organized by task, that is, when we calculate how many annotations were made, or present a confusion matrix of tags, we always retreive the data from the same task. In part 1 we defined 5 tasks, 3 on exploratory data, 1 for the training set and 1 for the test set. \n", "\n", "Each of these tasks will have their own isolated metrics, despite sharing an underlying dataset or schema. \n", "\n", "To that end, all metrics live under the task_definitions endpoint as we'll see\n", "\n", "Their are three kinds of metrics we currently display\n", "1. **Task progress metrics** These answer the question \"How much work has been done\" \n", "2. **Annotator metrics** These answer the question \"How much and how good is the work each of my labelers is doing\"\n", "3. **Annotation Metrics** These answer the question \"What do the results we are getting look like\" and can be further divided into\n", " 1. **Classification metrics** These answer the question \"What do our classsification results look like?\"\n", " 2. **Tag metrics** These anser the question \"What do our tagging results look like?\"\n", "\n", "Their is some interaction between these metrics, and some times you might wish to slice and dice the data differently to answer a particular question. In that case, you can always download and calculate a particular metric yourself. If you do feel something is missing, write us at support@lighttag.io and we will try to add it" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Task Progress metrics\n", "You can quickly see the high level results for all of your tasks at the *projects/default/task_definitions/metrics/* endpoint\n", "This is most useful to see how much work in progress you currently have. And how many annotations and classifications were collected" ] }, { "cell_type": "code", "execution_count": 178, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
annotationscomplete_tasksdefinitionidpartial_tasksparticipating_annotatorspercent_completepercent_partial_completetotal_tasks
0050{'url': 'http://localhost:8000/api/v1/projects...4bec1ef3-a34d-4ac6-a484-2ee0ec60fc855031.01.050
180450{'url': 'http://localhost:8000/api/v1/projects...9bab798d-c8db-43e8-9d4e-bf893a2f83f05031.01.050
211560254{'url': 'http://localhost:8000/api/v1/projects...ace018ea-e8ee-4e9e-8b2d-1a0418f3087825411.01.0254
381850{'url': 'http://localhost:8000/api/v1/projects...f5fe2bbb-e41b-4cc5-9cd2-8a28f6b37ab45031.01.050
49341111{'url': 'http://localhost:8000/api/v1/projects...f91a95a4-61e5-40f8-87b4-63103e1f00f011141.01.0111
\n", "
" ], "text/plain": [ " annotations complete_tasks \\\n", "0 0 50 \n", "1 804 50 \n", "2 11560 254 \n", "3 818 50 \n", "4 9341 111 \n", "\n", " definition \\\n", "0 {'url': 'http://localhost:8000/api/v1/projects... \n", "1 {'url': 'http://localhost:8000/api/v1/projects... \n", "2 {'url': 'http://localhost:8000/api/v1/projects... \n", "3 {'url': 'http://localhost:8000/api/v1/projects... \n", "4 {'url': 'http://localhost:8000/api/v1/projects... \n", "\n", " id partial_tasks \\\n", "0 4bec1ef3-a34d-4ac6-a484-2ee0ec60fc85 50 \n", "1 9bab798d-c8db-43e8-9d4e-bf893a2f83f0 50 \n", "2 ace018ea-e8ee-4e9e-8b2d-1a0418f30878 254 \n", "3 f5fe2bbb-e41b-4cc5-9cd2-8a28f6b37ab4 50 \n", "4 f91a95a4-61e5-40f8-87b4-63103e1f00f0 111 \n", "\n", " participating_annotators percent_complete percent_partial_complete \\\n", "0 3 1.0 1.0 \n", "1 3 1.0 1.0 \n", "2 1 1.0 1.0 \n", "3 3 1.0 1.0 \n", "4 4 1.0 1.0 \n", "\n", " total_tasks \n", "0 50 \n", "1 50 \n", "2 254 \n", "3 50 \n", "4 111 " ] }, "execution_count": 178, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = session.get(API_BASE+'projects/default/task_definitions/metrics/').json()\n", "metrics = pd.DataFrame(data)\n", "metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Annotator metrics\n", "It is often useful to see how each of our annotators is performing, both individually and compared to their peers. The *labelers* endpoint of an individual task definition will give us that information.\n", "\n" ] }, { "cell_type": "code", "execution_count": 179, "metadata": {}, "outputs": [], "source": [ "test_td = task_definitions[4]['url'] # Get the url for the test set taskdefinition" ] }, { "cell_type": "code", "execution_count": 181, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
annotator_idfull_agreementfull_agreement_from_meanno_agreementno_agreement_from_meanotherssome_agreementsome_agreement_from_meantotal_annotationstotal_annotations_from_mean
01.00.0009610.0000050.9385210.000271[{'agreement_pct': 0.06241872561768531, 'examp...0.060519-0.0002752082.00.0
12.00.0011490.0001930.9383380.000088[{'agreement_pct': 0.060453400503778336, 'exam...0.060513-0.0002812611.00.0
23.00.0012810.0003250.933817-0.004433[{'agreement_pct': 0.060906515580736544, 'exam...0.0649020.0041082342.00.0
34.00.000434-0.0005220.9423240.004074[{'agreement_pct': 0.06018518518518518, 'examp...0.057242-0.0035522306.00.0
\n", "
" ], "text/plain": [ " annotator_id full_agreement full_agreement_from_mean no_agreement \\\n", "0 1.0 0.000961 0.000005 0.938521 \n", "1 2.0 0.001149 0.000193 0.938338 \n", "2 3.0 0.001281 0.000325 0.933817 \n", "3 4.0 0.000434 -0.000522 0.942324 \n", "\n", " no_agreement_from_mean others \\\n", "0 0.000271 [{'agreement_pct': 0.06241872561768531, 'examp... \n", "1 0.000088 [{'agreement_pct': 0.060453400503778336, 'exam... \n", "2 -0.004433 [{'agreement_pct': 0.060906515580736544, 'exam... \n", "3 0.004074 [{'agreement_pct': 0.06018518518518518, 'examp... \n", "\n", " some_agreement some_agreement_from_mean total_annotations \\\n", "0 0.060519 -0.000275 2082.0 \n", "1 0.060513 -0.000281 2611.0 \n", "2 0.064902 0.004108 2342.0 \n", "3 0.057242 -0.003552 2306.0 \n", "\n", " total_annotations_from_mean \n", "0 0.0 \n", "1 0.0 \n", "2 0.0 \n", "3 0.0 " ] }, "execution_count": 181, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#Or get details for all of our tasks at once\n", "data = session.get(test_td+'labelers/').json()\n", "pd.DataFrame(data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classification Metrics\n", "It is often useful to monitor the distribution of class labels as they are coming in, the level of agreement among them and which classes are frequently confused. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fetching the distribution of classes\n", "The first metric we're usually interested in is the distribution of classes, simply how frequently does each class appear. Some common patterns to look out for\n", " 1. One very prominent class\n", " 2. One very infrequent class \n", " 3. Prominence of the UNK class if you specified an UNK(Unkown) class and it dominates the distribution, you may have confusing instructions or lazy labelers. Look into it" ] }, { "cell_type": "code", "execution_count": 182, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'http://localhost:8000/api/v1/projects/default/task_definitions/test-set-tags-and-classes/'" ] }, "execution_count": 182, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_td" ] }, { "cell_type": "code", "execution_count": 183, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 183, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdUAAAE+CAYAAADbOoq2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNX1//H3YZN9ERCVEQaNCswwM+xGBSGiomKAqLhg\nBGQR0ai/xN1o1K9GEo0YNRGXqGDUiIhKFEVcEHFlcUBcWQ2g7Mq+zMD5/VE1bc8wG0Mx3QOf1/P0\nM9W3bt06dbu6T9etmmpzd0RERGTvVUp0ACIiIvsLJVUREZGIKKmKiIhERElVREQkIkqqIiIiEVFS\nFRERiYiSqhTJzEab2S1lXPYLM+sWTt9mZv+OMK6bzOzxqNrbg/X2NbOlZrbJzNruwXKRbv++YmaX\nmdnKcPsaJjqePGbWzcyWJToOKRszW2JmPcqw3FNmdue+iGlfUlI9QIU7+lYz22hmP5nZh2Y23Mxi\n+4S7D3f3/ytL++6e5u5TI4hztw9Ud/+zuw/Z27bL4F7gCnev7e6fJWD9+4yZVQXuA04Nt29tBG2W\n6cN0L9c50MymR9TWPovfzNzMfpGo9cu+o6R6YDvL3esAzYGRwPXAv/amQTOrEkVgSao58EWig9hH\nmgDVKcP2WUCfJSIoqQrg7uvdfSJwHjDAzNIh//CLmTUys1fDo9p1ZvZ+3gdp+I36ejObC2w2syqF\nfMuubmbPh0fGs80sM29GwW/tees1s1rA68Dh4ZDkJjM7vOBwqpn9Ohxu/snMpppZq7h5S8zsGjOb\na2brwxiqF9YPZlbJzP5oZt+Z2SozG2tm9czsIDPbBFQG5pjZwiKWTzOzKWH/rDSzm4qo94KZrQjj\nmWZmaXHzzjCzL8N+Wm5m15Si/w83sxfNbLWZLTazK+Pa62RmM81sQxjTfYXEcwzwTfj0JzN7Jyw/\n3sxmhHHOMLPj45aZamZ3mdkHwBbgyAJtPg00A/4bvm7XlXXbC4n3yrBeSoHyVsBo4JfhOn8Kyw8y\ns3vN7H9hH4w2sxrF9WtR8RcSy0AzWxTGvNjM+sfNu8TMvjKzH81sspk1D8unhVXmhG2fV0i7Zem/\nhmb23/C1nhG+h6aH88zMRoX79QYz+9zC93kh624Q9snqMPZX4/s6fO3/z8w+CLf7TTNrFDf/txa8\nh9aa2c2FraPA+k60YKTsJwtOrwwsQ0yFvg5m9gszey/srzVm9nzcMi3t5/frN2bWL25eqfbFQrm7\nHgfgA1gC9Cik/H/AZeH0U8Cd4fTdBB9YVcNHF8Di2soGjgBqFGwfuA3IAc4Jl70GWAxUDec78Iu4\nGOLX2w1YViDG24B/h9PHAJuBU8K2rwMWANXi4vgUOBw4GPgKGF5En1wSLnskUBuYADwdNz9fnAWW\nrQP8APyB4IivDtC5YLxx66kDHATcD2THzfsB6BJONwDaFdf/BF+MZwG3AtXC2BcBp4XLfQT8Npyu\nDRxXRPyp4fZVCZ8fDPwI/BaoAlwQPm8Yzp9KsK+khfOrlmYfK+O2x/aBcDtnA42L2I6BwPQCZaOA\nieE21QH+C9xdyv16t/dIXLu1gA3AseHzw4C0cLp3uC+1Cvvnj8CHpdmX9qL//hM+agKtgaV5fQGc\nFu4n9cP9phVwWBHrbQicHbZTB3gBeDlu/lRgIcF7r0b4fGQ4rzWwCegaxngfkFtUPxKM/mwM96+q\n4bqzCvkcKDKmEl6H54CbCd4n1YET45ZZCgwKX5+2wBqgdXH7YmkeOlKVgr4n+PApKIdgZ23u7jnu\n/r6He1zoAXdf6u5bi2h3lruPd/ccgjdadeC4COI9D3jN3aeEbd9L8EY/Pq7OA+7+vbuvI/hAzSqi\nrf7Afe6+yN03ATcC51vphrR7ASvc/W/uvs3dN7r7J4VVdPcnwvnbCRJuppnVC2fnAK3NrK67/+ju\ns+PKC+v/jgQJ5g533+Hui4DHgPPjlvuFmTVy903u/nEptgXgTGC+uz/t7rnu/hzwNXBWXJ2n3P2L\ncH5OaRot47ZDcLB1H3Aq0N3dV5dmfWZmwDDg/7n7OnffCPyZ/P1T3H5dkl1AupnVcPcf3D1v+Hw4\nQeL+yt1zw3Vm5R2tllVR/WdmlQmSzp/cfYu7fwmMiVs0hyAZtST40vCVu/9QxDrWuvuLYTsbgbuA\nkwpUe9Ldvw3f7+P4+T11DvCqu08LY7wl7KOiXAi85e7Phf2/1t2zyxBTUa9DDkHiPjx8X+adb+8F\nLHH3J8P99zPgReDcuOWK2heLpaQqBTUF1hVSfg/BN+83w2GWGwrMX1pCu7H57r4LWEZw9Li3Dge+\nK9D2UoLtyLMibnoLwRFbiW2F01UIzjeW5AiCb+/FMrPKZjbSzBaa2QaCoxGAvOGzs4EzgO/CYatf\nhuVF9X9zguHxn/IewE1xMQ8mOKL4OhwS7FWKbYHd+4LweXy/lvSa57MX2w7BEdYwgkS1fg9W25jg\n6GZWXP+8EZZDyft1fPyj7efTEDe5+2aCL3XDgR/M7DUzaxlWbw78PW6d6wiOEJsW0fbrcW33L6JO\ncf3XmGBfjX9N4t9z7wAPAf8AVpnZo2ZW18yaxa13U7iemmb2SDiEuwGYBtQPE3eeot5ThxdY72Yg\ndtFb/LrMrBmlf98UGVMJr8N1BP3+qQWniC4Jy5sDnQu8b/oDh4bzi9sXi6WkKjFm1pHgTb/b1ZPh\nt+M/uPuRwK+B35vZyfFVSmj+iLj1VAJSCI6KIXhT1oyre2jcdEntfk/wBslr28J1LS9huRLbIjin\nlQusLMWySylwXrEIFxIMDfYA6hEMu0LwxsfdZ7h7b+AQ4GWCo4Di+n8psNjd68c96rj7GeFy8939\ngrC9vwDjLThXXZKCfQFBf8T3a0mvTcH5Zdr20I8ERxdPmtkJe7DONcBWguHAvP6p5+61w3UWt1/n\na8uDq+Frh48/h2WT3f0UgqPdrwlGCSB4XS4t8LrUcPcPCw3a/fS4tp8pYluK67/VBPtq/HnmI+IX\ndvcH3L09wRDtMcC17v6/uPXmJcY/AMcSnL6oSzCUm7eekvxA/vd6TYKh27wYasc9/kfQT0eVot1i\nYyrqdXD3Fe4+1N0PBy4F/mnB9RtLgfcKvD613f2ycLni9sViKakK4TfWXgTnY/7t7p8XUqdXeNLf\ngPXAToof1imovZn9JhxKvRrYDuQNRWYDF4bfxHuSf1hnJdAwboiwoHHAmWZ2sgX/FvKHsO1CP7xK\n8Bzw/8yshZnVJhiyez4cvivJq8BhZna1BRfG1DGzzoXUqxPGt5bgi8Sf82aYWTUz629m9cLh1A2E\nfVxM/38KbLTgQrEaYR+mh1+QMLOLzKxxeAT/U7iq0rxuk4BjzOxCCy48O4/gw/jVUiybZyX5v2iU\nadvzePAvWv2BCWbWqZh1pphZtXCZXQQfsKPM7JBwXU3N7LRwurj9umD8+ZhZEzPrHX5J2U5wLjFv\n2dHAjRZeSGTBEO25cYsX23YRdYrsP3ffSXANwG3hUV1L4OK4WDuaWefwPbIZ2EbR+0Edgi8iP5nZ\nwcCfSogz3niglwUXH1UD7qD4PPMM0MPM+oX7WUMzK+z0TJExFfc6mNm59vMFTT8SfFHZRbAfH2PB\nRVVVw0dHM2tVmn2xOEqqB7b/mtlGgm9tNxOc6xxURN2jgbcIdtiPgH+6+7t7sK5XCIZo8i5++Y3/\nfB7uKoJzdXlDMC/nLeTuXxMku0XhME2+IWN3/wa4CHiQ4KjkLIJ/FdqxB7HleQJ4mmBoaTHBB8/v\nSrNgeJ7nlHD9K4D5QPdCqo4lGEZdDnzJz18s8vwWWBIOcQ0n6A8oov/DD9NeBOe0FhP0weMERzIA\nPYEvwqG9vwPne9HnveO3Z23Y7h8IPsSvA3q5+5qSlo1zN/DH8HW7Zi+2PT6uKQQX6/zXzNoVss53\nCP4taIWZ5cV6PcEQ78dh228RHPVA8ft1wfgLqgT8nuCofh3Bl8G8I52XCEYG/hOucx5wetyytwFj\nwrb7Ubg97b8rCF73FQT78XMESQagLsGXix/DNtYSDH0X5n6C6xLWhOt4o4h6uwnPZV4OPEtw1Poj\nwameour/j2CY9Q8EfZgNZBZStbiYinwdCK45+CTc/ycCV3lwzcRGgvPz54fLrSB4vQ4KlytxXyxK\n3lVuIiKyHzGzvwCHuvuARMdyINGRqojIfsCC/7vMsEAngovUXkp0XAea/fnuNyIiB5I6BEO+hxOc\nj/0bwWkXKUca/hUREYmIhn9FREQiouHfA0yjRo08NTU10WGIiFQos2bNWuPujUuqp6R6gElNTWXm\nzJmJDkNEpEIxs4J3GCuUhn9FREQioqQqIiISESVVERGRiOicqpCTk8OyZcvYtm1bokORJFG9enVS\nUlKoWrVqokMRqVCUVIVly5ZRp04dUlNTCe4rLgcyd2ft2rUsW7aMFi1aJDockQpFw7/Ctm3baNiw\noRKqAGBmNGzYUCMXImWgpCoASqiSj/YHkbJRUhUREYmIzqnKblJveC3S9paMPDPS9kREkpWSqgiw\nZMkSevXqxbx58yJp79Zbb6Vr16706NGD999/n+HDh1O1alVee+01rrrqKsaPH7/HbT711FOceuqp\nHH548DvtQ4YM4fe//z2tW7eOJGaR0oj6S/e+kMgv8kqqIvvAHXfcEZt+5plnuPHGG7nooosAypRQ\nIUiq6enpsaT6+OOP732gIhIpnVOVpLBkyRJatWrF0KFDSUtL49RTT2Xr1q1069Ytdq/iNWvWkPdj\nAE899RR9+vThlFNOITU1lYceeoj77ruPtm3bctxxx7Fu3boi17VgwQJ69OhBZmYm7dq1Y+HChbvF\n0qVLF9q1a0e7du348MMPAfjhhx/o2rUrWVlZpKen8/7777Nz504GDhxIeno6bdq0YdSoUQAMHDiQ\n8ePH8/jjjzNu3DhuueUW+vfvz5IlS0hPTwdg586dXHPNNaSnp5ORkcGDDz4IBAm5Y8eOpKenM2zY\nMNyd8ePHM3PmTPr3709WVtZuffPcc8/Rpk0b0tPTuf7662PbUrt2bW6++WYyMzM57rjjWLlyJQAv\nvPAC6enpZGZm0rVr1719+UQkpKQqSWP+/PlcfvnlfPHFF9SvX58XX3yx2Prz5s1jwoQJzJgxg5tv\nvpmaNWvy2Wef8ctf/pKxY8cWuVz//v25/PLLmTNnDh9++CGHHXZYvvmHHHIIU6ZMYfbs2Tz//PNc\neeWVADz77LOcdtppZGdnM2fOHLKyssjOzmb58uXMmzePzz//nEGDBuVra8iQIfz617/mnnvu4Zln\nnsk379FHH2XJkiVkZ2czd+5c+vfvD8AVV1zBjBkzmDdvHlu3buXVV1/lnHPOoUOHDjzzzDNkZ2dT\no0aNWDvff/89119/Pe+88w7Z2dnMmDGDl19+GYDNmzdz3HHHMWfOHLp27cpjjz0GBIl78uTJzJkz\nh4kTJxbbzyJSekqqkjRatGhBVlYWAO3bt2fJkiXF1u/evTt16tShcePG1KtXj7POOguANm3aFLns\nxo0bWb58OX379gWCOwfVrFkzX52cnByGDh1KmzZtOPfcc/nyyy8B6NixI08++SS33XYbn3/+OXXq\n1OHII49k0aJF/O53v+ONN96gbt26pd7et956i0svvZQqVYKzMAcffDAA7777Lp07d6ZNmza88847\nfPHFF8W2M2PGDLp160bjxo2pUqUK/fv3Z9q0aQBUq1aNXr16Afn79IQTTmDgwIE89thj7Ny5s9Qx\ni0jxlFQlaRx00EGx6cqVK5Obm0uVKlXYtWsXwG43I4ivX6lSpdjzSpUqkZubW+Y4Ro0aRZMmTZgz\nZw4zZ85kx44dAHTt2pVp06bRtGlTBg4cyNixY2nQoAFz5syhW7dujB49miFDhpR5vRBs44gRIxg/\nfjyff/45Q4cO3aubMFStWjX2P6d5fQowevRo7rzzTpYuXUr79u1Zu3btXsUtIgFdqCS7SaZ/gUlN\nTWXWrFl06tSpzBf4xKtTpw4pKSm8/PLL9OnTh+3bt+92pLZ+/XpSUlKoVKkSY8aMic3/7rvvSElJ\nYejQoWzfvp3Zs2dzxhlnUK1aNc4++2yOPfbY2MVIpXHKKafwyCOP0L17d6pUqcK6deuoVCn4ntuo\nUSM2bdrE+PHjOeecc2Kxb9y4cbd2OnXqxJVXXsmaNWto0KABzz33HL/73e+KXffChQvp3LkznTt3\n5vXXX2fp0qU0bNiw1LGLSOF0pJpEzOxYM8uOe2wws6vN7GAzm2Jm88O/DcL6ZmYPmNkCM5trZu0S\nvQ1Ru+aaa3j44Ydp27Yta9asiaTNp59+mgceeICMjAyOP/54VqxYkW/+iBEjGDNmDJmZmXz99dfU\nqlULgKlTp5KZmUnbtm15/vnnueqqq1i+fDndunUjKyuLiy66iLvvvrvUcQwZMoRmzZqRkZFBZmYm\nzz77LPXr12fo0KGkp6dz2mmn0bFjx1j9gQMHMnz48NiFSnkOO+wwRo4cSffu3cnMzKR9+/b07t27\n2HVfe+21sQubjj/+eDIzM0sdt4gUzdw90TFIIcysMrAc6AxcDqxz95FmdgPQwN2vN7MzgN8BZ4T1\n/u7unYtrt0OHDp53xWier776ilatWu2LzZAKTPuFFOZA/T9VM5vl7h1Kqqcj1eR1MrDQ3b8DegNj\nwvIxQJ9wujcw1gMfA/XN7LDdmxIRkfKgc6rJ63zguXC6ibv/EE6vAJqE002BpXHLLAvLfogrw8yG\nAcMAmjVrtq/iTTqXX345H3zwQb6yq666ard/exERiYqSahIys2rAr4EbC85zdzezPRqzd/dHgUch\nGP4tos5+98sk//jHPxIdQoWl00IiZaPh3+R0OjDb3VeGz1fmDeuGf1eF5cuBI+KWSwnL9kj16tVZ\nu3atPkgF+PlHyqtXr57oUEQqHB2pJqcL+HnoF2AiMAAYGf59Ja78CjP7D8GFSuvjholLLSUlhWXL\nlrF69eq9i1r2G9WrVyclJSXRYYhUOEqqScbMagGnAJfGFY8ExpnZYOA7oF9YPongyt8FwBagTCcL\nq1atSosWLcocs4iIBJRUk4y7bwYaFihbS3A1cMG6TvDvNiIikgR0TlVERCQiSqoiIiIRUVIVERGJ\niJKqiIhIRJRURUREIqKkKiIiEhElVRERkYgoqYqIiERESVVERCQiSqoiIiIRUVIVERGJiJKqiIhI\nRJRURUREIqKkKiIiEhElVRERkYgoqYqIiERESVVERCQiSqoiIiIRUVIVERGJiJJqkjGz+mY23sy+\nNrOvzOyXZnawmU0xs/nh3wZhXTOzB8xsgZnNNbN2iY5fRORApqSafP4OvOHuLYFM4CvgBuBtdz8a\neDt8DnA6cHT4GAY8XP7hiohIHiXVJGJm9YCuwL8A3H2Hu/8E9AbGhNXGAH3C6d7AWA98DNQ3s8PK\nOWwREQkpqSaXFsBq4Ekz+8zMHjezWkATd/8hrLMCaBJONwWWxi2/LCzLx8yGmdlMM5u5evXqfRi+\niMiBTUk1uVQB2gEPu3tbYDM/D/UC4O4O+J406u6PunsHd+/QuHHjyIIVEZH8lFSTyzJgmbt/Ej4f\nT5BkV+YN64Z/V4XzlwNHxC2fEpaJiEgCKKkmEXdfASw1s2PDopOBL4GJwICwbADwSjg9Ebg4vAr4\nOGB93DCxiIiUsyqJDkB28zvgGTOrBiwCBhF8+RlnZoOB74B+Yd1JwBnAAmBLWFdERBJESTXJuHs2\n0KGQWScXUteBy/d5UCIiUioa/hUREYmIkqqIiEhElFRFREQioqQqIiISESVVERGRiCipioiIRERJ\nVUREJCJKqiIiIhFRUhUREYmIkqqIiEhEdJtC2WupN7yW6BBKZcnIMxMdgojs53SkKiIiEhElVRER\nkYgoqYqIiERESVVERCQiSqoiIiIRUVIVERGJiJKqiIhIRJRURUREIqKkmmTMbImZfW5m2WY2Myw7\n2MymmNn88G+DsNzM7AEzW2Bmc82sXWKjFxE5sCmpJqfu7p7l7h3C5zcAb7v70cDb4XOA04Gjw8cw\n4OFyj1RERGKUVCuG3sCYcHoM0CeufKwHPgbqm9lhiQhQRESUVJORA2+a2SwzGxaWNXH3H8LpFUCT\ncLopsDRu2WVhWT5mNszMZprZzNWrV++ruEVEDni6oX7yOdHdl5vZIcAUM/s6fqa7u5n5njTo7o8C\njwJ06NBhj5YVEZHS05FqknH35eHfVcBLQCdgZd6wbvh3VVh9OXBE3OIpYZmIiCSAkmoSMbNaZlYn\nbxo4FZgHTAQGhNUGAK+E0xOBi8OrgI8D1scNE4uISDnT8G9yaQK8ZGYQvDbPuvsbZjYDGGdmg4Hv\ngH5h/UnAGcACYAswqPxDFhGRPEqqScTdFwGZhZSvBU4upNyBy8shNBERKQUN/4qIiERESVVERCQi\nSqoiIiIRUVIVERGJiJKqiIhIRJRURUREIqKkKiIiEhElVRERkYgoqYqIiERESVVERCQiSqoiIiIR\nUVIVERGJiJKqiIhIRJRURUREIqKkKiIiEhElVRERkYgoqYqIiERESVVERCQiSqoiIiIRUVJNQmZW\n2cw+M7NXw+ctzOwTM1tgZs+bWbWw/KDw+YJwfmoi4xYROdApqSanq4Cv4p7/BRjl7r8AfgQGh+WD\ngR/D8lFhPRERSRAl1SRjZinAmcDj4XMDfgWMD6uMAfqE073D54TzTw7ri4hIAiipJp/7geuAXeHz\nhsBP7p4bPl8GNA2nmwJLAcL568P6+ZjZMDObaWYzV69evS9jFxE5oCmpJhEz6wWscvdZUbbr7o+6\newd379C4ceMomxYRkThVEh2A5HMC8GszOwOoDtQF/g7UN7Mq4dFoCrA8rL8cOAJYZmZVgHrA2vIP\nW0REQEeqScXdb3T3FHdPBc4H3nH3/sC7wDlhtQHAK+H0xPA54fx33N3LMWQREYmjpFoxXA/83swW\nEJwz/VdY/i+gYVj+e+CGBMUnIiJo+DdpuftUYGo4vQjoVEidbcC55RqYiIgUSUeqIiIiEVFSFRER\niYiSqoiISESUVEVERCKiC5VEZL+WesNriQ6hVJaMPDPRIUgEdKQqIiISESVVERGRiCipioiIRERJ\nVUREJCJKqiIiIhFRUhUREYmIkqqIiEhElFRFREQioqQqIiISESVVERGRiCipioiIRERJVUREJCJK\nqiIiIhFRUhUREYmIkmoSMbPqZvapmc0xsy/M7PawvIWZfWJmC8zseTOrFpYfFD5fEM5PTWT8IiIH\nOiXV5LId+JW7ZwJZQE8zOw74CzDK3X8B/AgMDusPBn4My0eF9UREJEGUVJOIBzaFT6uGDwd+BYwP\ny8cAfcLp3uFzwvknm5mVU7giIlKAkmqSMbPKZpYNrAKmAAuBn9w9N6yyDGgaTjcFlgKE89cDDQtp\nc5iZzTSzmatXr97XmyAicsBSUk0y7r7T3bOAFKAT0DKCNh919w7u3qFx48Z7HaOIiBROSTVJuftP\nwLvAL4H6ZlYlnJUCLA+nlwNHAITz6wFryzlUEREJKakmETNrbGb1w+kawCnAVwTJ9Zyw2gDglXB6\nYviccP477u7lF7GIiMSrUnIVKUeHAWPMrDLBF55x7v6qmX0J/MfM7gQ+A/4V1v8X8LSZLQDWAecn\nImgREQkoqSYRd58LtC2kfBHB+dWC5duAc8shNBERKQUN/4qIiERESVVERCQiSqoiIiIRUVIVERGJ\niJKqiIhIRJRURUREIqKkKiIiEhElVRERkYgoqYqIiERESVVERCQiSqoiIiIRUVIVERGJiJKqiIhI\nRJRURUREIqKkKiIiEhElVRERkYgoqYqIiERESVVERCQiSqoiIiIRUVIVERGJiJJqEjGzI8zsXTP7\n0sy+MLOrwvKDzWyKmc0P/zYIy83MHjCzBWY218zaJXYLREQObEqqySUX+IO7twaOAy43s9bADcDb\n7n408Hb4HOB04OjwMQx4uPxDFhGRPEqqScTdf3D32eH0RuAroCnQGxgTVhsD9AmnewNjPfAxUN/M\nDivnsEVEJKSkmqTMLBVoC3wCNHH3H8JZK4Am4XRTYGncYsvCsoJtDTOzmWY2c/Xq1fssZhGRA52S\nahIys9rAi8DV7r4hfp67O+B70p67P+ruHdy9Q+PGjSOMVERE4impJhkzq0qQUJ9x9wlh8cq8Yd3w\n76qwfDlwRNziKWGZiIgkgJJqEjEzA/4FfOXu98XNmggMCKcHAK/ElV8cXgV8HLA+bphYRETKWZVE\nByD5nAD8FvjczLLDspuAkcA4MxsMfAf0C+dNAs4AFgBbgEHlG66IiMRTUk0i7j4dsCJmn1xIfQcu\n36dBiYhIqWn4V0REJCJKqiIiIhFRUhUREYmIzqmKJJnUG15LdAilsmTkmYkOQSTp6EhVREQkIkqq\nIiIiEVFSFRERiYiSqoiISESUVEVERCKipCoiIhIRJVUREZGIKKmKiIhERElVREQkIkqqIiIiEVFS\nFRERiYiSqoiISESUVEVERCKipCoiIhIRJVUREZGIKKkmGTN7wsxWmdm8uLKDzWyKmc0P/zYIy83M\nHjCzBWY218zaJS5yERFRUk0+TwE9C5TdALzt7kcDb4fPAU4Hjg4fw4CHyylGEREphJJqknH3acC6\nAsW9gTHh9BigT1z5WA98DNQ3s8PKJ1IRESlISbViaOLuP4TTK4Am4XRTYGlcvWVhWT5mNszMZprZ\nzNWrV+/bSEVEDmBKqhWMuzvge7jMo+7ewd07NG7ceB9FJiIiSqoVw8q8Yd3w76qwfDlwRFy9lLBM\nREQSQEm1YpgIDAinBwCvxJVfHF4FfBywPm6YWEREylmVRAcg+ZnZc0A3oJGZLQP+BIwExpnZYOA7\noF9YfRJwBrAA2AIMKveARUQkRkk1ybj7BUXMOrmQug5cvm8jEhGR0tLwr4iISESUVEVERCKipCoi\nIhIRJVUREZGIKKmKiIhERElVREQkIkqqIiIiEVFSFRERiYiSqoiISESUVEVERCKipCoiIhIRJVUR\nEZGIKKmKiIhERElVREQkIkqqIiIiEVFSFRERiYiSqoiISESUVEVERCKipCoiIhIRJdX9gJn1NLNv\nzGyBmd0qgaZpAAAdaklEQVSQ6HhERA5USqoVnJlVBv4BnA60Bi4ws9aJjUpE5MCkpFrxdQIWuPsi\nd98B/AfoneCYREQOSObuiY5B9oKZnQP0dPch4fPfAp3d/Yq4OsOAYeHTdoCVe6D7KTND76HoqD+j\npf6MlLt7iQeiVcojEkksd38UeBTAzLwivMk6dOjAzJkzEx1GiSrKh5b6M1rqz+hUoL4s1cGIhn8r\nvuXAEXHPU8KypFC5cmWysrJIT0/n3HPPZcuWLXu0/JAhQ/jyyy/3UXQV35IlS0hPT89Xdtttt3Hv\nvfcycOBAmjZtyvbt2wFYs2YNqamphS732GOP0b59e3788cdyi31fWrFiBeeffz5HHXUU7du354wz\nzuDbb7/d43bef/990tLSyMrKYuvWrXu07OjRoxk7duwer7Oiq127dqTtxe+r2dnZTJo0KdL2o6ak\nWvHNAI42sxZmVg04H5iY4JhiatSoQXZ2NvPmzaNatWqMHj0633x3Z9euXUUu//jjj9O6ta67KqvK\nlSvzxBNPFFvn6aef5sEHH2Ty5Mk0aNCgnCLbd9ydvn370q1bNxYuXMisWbO4++67Wbly5R639cwz\nz3DjjTeSnZ1NjRo19mjZ4cOHc/HFF+/xOqVoSqqyz7l7LnAFMBn4Chjn7l8kNqrCdenShQULFrBk\nyRKOPfZYLr74YtLT01m6dCmXXXYZHTp0IC0tjT/96U8MGxacAu7WrRszZ85k586dDBw4kPT0dNq0\nacOoUaMAWLhwIT179qR9+/Z06dKFr7/+OpGbmHSuvvpqRo0axeDBgwudP27cOEaOHMmbb75Jo0aN\nyjm6fePdd9+latWqDB8+PFaWmZnJiSeeyLXXXhvbh55//nkApk6dSrdu3TjnnHNo2bIl/fv3x915\n/PHHGTduHLfccgv9+/dn6tSp9OrVC4Bhw4ZxxRVX8NRTTwFwww030Lp1azIyMrjmmmuAn0cMINiP\nr7/+ejp16sQxxxzD+++/D8CWLVvo168frVu3pm/fvnTu3LlCDIWWRlH9Cvn76+CDDwZg4MCBjB8/\nPrZ8wSPeHTt2cOutt/L888+TlZUVe/2Sjc6p7gfcfRKQ1F/fcnNzef311+nZsycA8+fPZ8yYMRx3\n3HEA3HXXXRx88MHs3LmTk08+mQceeCDf8tnZ2Sxfvpx58+YB8NNPPwHBh9vo0aM5+uij+eSTTxgx\nYgTvvPNOOW5ZcmvWrBknnngi1atX323ed999xxVXXMFnn33GoYcemoDo9o158+bRvn373conTJhA\ndnY2c+bMYc2aNXTs2JGuXbsC8Nlnn/HFF19w+OGHc8IJJ/DBBx8wZMgQpk+fTq9evTjnnHOYOnVq\nrK28pAqwdu1aXnrpJb7++mvMLLZvFpSbm8unn37KpEmTuP3223nrrbf45z//SYMGDfjyyy+ZN28e\nWVlZ0XdIAhXWr61atSpVfxVUrVo17rjjDmbOnMlDDz20jyMvOx2pyj61detWsrKy6NChA82aNYsd\nMTVv3jyWUCE4YmrXrh1t27bliy++2O086pFHHsmiRYv43e9+xxtvvEHdunXZtGkTH374Ieeeey5Z\nWVlceuml/PDDD+W6fYlW1LUT8eU33ngj99xzz27D7I0bN6ZZs2aMGzdun8aYLKZPn84FF1xA5cqV\nadKkCSeddBIzZswAoFOnTqSkpFCpUiWysrJYsmRJqdutV68e1atXZ/DgwUyYMIGaNWsWWu83v/kN\nAO3bt4+1P336dM4//3wA0tPTycjIKPsGJqHC+rW0/VVR6UhV9qm8c6oF1apVKza9ePFi7r33XmbM\nmEGDBg0YOHAg27Zty1e/QYMGzJkzh8mTJzN69GjGjRvH/fffT/369Qtt/0DRsGHD3S4uWrduHS1a\ntIg9P/roo8nKytotedasWZNJkybRpUsXDjnkEPr3718uMe9raWlp+YYRS+Oggw6KTVeuXJnc3Nzd\n6lSpUiXfF5O8fbRKlSp8+umnvP3224wfP56HHnqo0NGSvHUU1f7+qLB+Laq/4vt3165d7NixI1Fh\n7xUdqUrCbdiwgVq1alGvXj1WrlzJ66+/vludNWvWsGvXLs4++2zuvPNOZs+eTd26dWnRogUvvPAC\nEFygMmfOnPIOP6Fq167NYYcdFvsQX7duHW+88QYnnnhivno333xz7PxevEMOOYQ33niDm266icmT\nJ5dLzPvar371K7Zv386jjz4aK5s7dy7169fn+eefZ+fOnaxevZpp06bRqVOnUrfbvHlzvvzyS7Zv\n385PP/3E22+/DcCmTZtYv349Z5xxBqNGjdqjffCEE06Ifdn58ssv+fzzz0u9bEVVVH+lpqYya9Ys\nACZOnEhOTs5uy9apU4eNGzeWa7x7SkeqknCZmZm0bduWli1bcsQRR3DCCSfsVmf58uUMGjQo9k32\n7rvvBoKrMy+77DLuvPNOcnJyOP/888nMzCzX+BNt7NixXH755fz+978H4Nprr+Woo47KVyctLY12\n7drx0Ucf7bZ8ixYtmDhxImeccQYvvfTSHiWaZGRmvPTSS1x99dX85S9/oXr16qSmpnL//fezadMm\nMjMzMTP++te/cuihh5b64rYjjjiCfv36kZ6eTosWLWjbti0AGzdupHfv3mzbtg1357777it1rCNG\njGDAgAG0bt2ali1bkpaWRr169cq03RXFxo0b6dKlCzVr1szXX02aNOHVV18lMzOTnj175hvNytO9\ne3dGjhxJVlYWN954I+edd155h18i3VHpAJPMN3/49ttvueyyy1i5ciXz5s1j7ty5TJw4kT/+8Y+J\nDq1IyfjP9R07duTNN9/c7d9j3nzzTQYPHszSpUsTFFnJkrE/AWbPnr1bWb169WjevDlVqpT92GTn\nzp3k5ORQvXp1Fi5cSI8ePfjmm2+oVq3a3oQbk6z92a5du936tLCyZBL2ZYk3gNCRqiSNoUOHcs89\n93DppZcCkJGRwYUXXpjUSTUZDRs2jO7duzNlyhQaN24MwLPPPsvNN9/Ma6+9luDoKqYRI0Ywe/Zs\nMjIycHfmzZtHWloa69ev5+GHH+bUU08tU7tbtmyhe/fu5OTk4O7885//jCyhJqPXX3+dSZMmsXz5\ncq688spY+YYNG/bqy0kyKfZItUaNGiu2bdvWpBzjkX2sevXqu10EJGWn/oyW+jNa6s/oVK9efdfW\nrVsrl1Sv2AuVtm3b1sTd0WP/eeSd90nGR8+ePVmwYAFt27bF3XnhhRfo2bNnwuOqqP05btw4mjRp\nQseOHVm9enXC46nI/ZmWllZkWWZmZsLjq2j9mXdkXpEe27ZtK9WFvfvd1b+F3Qt1b9x666289dZb\nQP77gC5fvpxzzjmnTG0+9dRTfP/997Hnur9t4B//+AeXXnopX3/9NU2bNuX+++/n4YcfLnN7qamp\ntGnTJvZ/svH+9re/YWasWbMGgFdeeYWMjIxY3enTp+/VtiRSmzZtyMjI4I477mDLli2sXbuWX/3q\nV7Hysrjkkks45JBDdntvPfjgg7ELbK677joAcnJyGDBgAG3atKFVq1axi8oqsrS0NC677DLee+89\n3nvvPUaMGEHr1q3Zvn07VatW3aO2tm3bRqdOncjMzIzdQQyCq9dvvvlmjjnmGFq1apXvBihTp04l\nKyuLtLQ0TjrppEi3rTzl7YPt2rUjIyNjt0dZfPPNN2RlZcUedevW5f777+eWW26JvadPPfXU2Gfu\n+vXrOeuss2L9/+STT0a5iRSbmYPZFcvixYs9LS1tn7R96aWX+tNPP73X7Zx00kk+Y8aMCCLacxXh\nNd20aZNv2LBhr9tp3ry5r169erfy//3vf37qqad6s2bNYvM3btzou3btcnf3OXPm+LHHHluqdSRj\nfy5ZsqTYR1m89957PmvWrHzvrXfeecdPPvlk37Ztm7u7r1y50t3dn3nmGT/vvPPc3X3z5s3evHlz\nX7x4canWk4z96e6+ZcsWv/fee71Pnz7ep08fv+eee3zz5s2+c+dO37hx4x61tWvXrtgyO3bs8E6d\nOvlHH33kTzzxhP/2t7/1nTt3uvvP/fnjjz96q1at/LvvvstXXhrJ1p/7Yt+Ml5ub602aNPElS5b4\n+vXrY+V///vf/dJLL3V397vuusuvu+46d3dftWqVN2jQwLdv315i22FflnhEG3lSXbx4sbds2dKH\nDBnirVu39lNOOcW3bNmSL5GsXr3amzdv7u7uTz75pPfu3dt79OjhzZs39wcffND/9re/eVZWlnfu\n3NnXrl1b5Lrmz5/vJ598smdkZHjbtm19wYIF+ZLq4sWL/cQTT/S2bdt627Zt/YMPPnB39++//967\ndOnimZmZnpaW5tOmTfPc3FwfMGCAp6WleXp6ut93333u7j5gwAB/4YUX/LHHHvMGDRp4amqqX3jh\nhfnWk5ub63/4wx88LS3N27Rp4w888IC7u99+++3eoUMHT0tL86FDh/quXbv8hRde8Fq1avkxxxzj\nmZmZu/XNs88+6+np6Z6WlhZ74d3da9Wq5TfddJNnZGR4586dfcWKFe7uPm7cOE9LS/OMjAzv0qVL\nia9Psr3J4t1///2+fv1637Vrlw8ePNjbtm3rkydPLnN7RSXVs88+27Ozs4uc/+GHH3rLli1LtY5k\n7s+oFfzCeu655/qUKVN2q/fss896r169PCcnx9esWeNHH310se/jeAdSf7oHXzratm3rH3/8sXfs\n2NHnz5+/W51//OMffvPNN5ep/QOtPydPnuzHH3/8buV//vOfffjw4bHpyy67zHft2uWLFi3yo446\nKvZFpjgJTaqVK1f2zz77zN2DN97TTz9dbFI96qijfMOGDb5q1SqvW7euP/zww+7ufvXVV/uoUaOK\nXFenTp18woQJ7u6+detW37x5c743/ubNm33r1q3u7v7tt996+/bt3d393nvv9TvvvNPdg4S4YcMG\nnzlzpvfo0SPW9o8//ujuPyfVgtPx6/nnP//pZ599tufk5Li7xz5A4j9ILrroIp84caK7736kmvd8\n+fLlfsQRR/iqVas8JyfHu3fv7i+99JK7By9o3vLXXnut/9///Z+7u6enp/uyZcvyxVycZH6TZWRk\nuLv7G2+84X369PF58+Z527Zty9xeamqqt23b1tu1a+ePPPKIu7u//PLLfuWVV7r77kl3woQJfuyx\nx3qDBg38ww8/LNU6krE/U1NTvUWLFrFH/PMjjzyyzO0WTKqZmZl+6623eqdOnbxr167+6aefuntw\n9HXeeed5o0aNvGbNmrG+L41k7E939+nTp3uPHj386KOPzte3ZZWbm+uZmZleq1at2Jfngw8+2O+8\n805v37699+zZ07/99lt3d7/qqqt8xIgRftJJJ3m7du18zJgxpV5PsvZn7dq1vU6dOl6nTh0/6KCD\nvFKlSl6nTp29bnfQoEH+4IMPxp7fdNNNnpKS4mlpab5q1Sp3d9+wYYN369bNDz30UK9Vq5a/+uqr\npWq7tEl1n1zD3KJFi9iNoePvc1mU7t27U6dOHerUqUO9evU466yzgGD8fe7cuYUus3HjRpYvX07f\nvn0BCr1heE5ODldccQXZ2dlUrlw59nuKHTt25JJLLiEnJ4c+ffqQlZWV796yZ5555h5dIv/WW28x\nfPjw2CXheb+68O677/LXv/6VLVu2sG7dOtLS0mLbVpgZM2bQrVu32L9B9O/fn2nTptGnTx+qVasW\n+4WM9u3bM2XKFCC4I8vAgQPp169f7N6iFVWw38KkSZO4+OKLYxeClNX06dNp2rQpq1at4pRTTqFl\ny5b8+c9/5s033yy0ft++fenbty/Tpk3jlltuiZ1Lr2gK/srJrl27GDduHPfee2/shgVRyM3NZd26\ndXz88cfMmDGDfv36sWjRIj799FMqV67M999/z48//kiXLl3o0aMHRx55ZGTrLm+DBw9m1KhRtG/f\nnsqVS7wAtESVK1cmOzubn376ib59+zJv3jy2b99O9erVmTlzJhMmTOCSSy7h/fffJzc3l1mzZvH2\n22+zdetWfvnLX3LcccdxzDHHRLBliRF/VyR355VXXuHjjz/eqzZ37NjBxIkT853Dv+uuu7jrrru4\n++67eeihh7j99tuZPHkyWVlZvPPOOyxcuJBTTjmFLl26ULdu3b1af559cqFSUfd7zLsbTsFLvOPr\nV6pUKfa8UqVKe3WPzFGjRtGkSRPmzJnDzJkzY/eS7Nq1K9OmTaNp06YMHDiQsWPHxu4t261bN0aP\nHs2QIUPKvF4ItnHEiBGMHz+ezz//nKFDh+7Vpe1Vq1aN3SQ9/t6ho0eP5s4772Tp0qW0b9+etWvX\n7lXcidS+fXtOPfVUJk2axGmnncbGjRupVKnsu2jTpk2B4FZ8ffv25b333mPx4sVkZmaSmprKsmXL\naNeuHStWrMi3XNeuXVm0aFHsIqaKpmHDhjRs2JAGDRrw6quv0r17dz766CNee+01XnzxxcjWk5KS\nwm9+8xvMjE6dOlGpUiXWrFnDs88+S8+ePalatSqHHHIIJ5xwQoX/ObN69epx+umnc8ghh8T6t2HD\nhnvdbv369enevTtvvPFGrD8h+IKXd0CRkpLCaaedRq1atWjUqBFdu3bdr27HaWb06dNnr2+T+frr\nr9OuXTuaNNn9v0D79+8f2/effPLJ2H77i1/8ghYtWkT6k5HldvVv/H0d9/Rm14WpU6cOKSkpvPzy\nywBs376dLVu25Kuzfv16DjvsMCpVqsTTTz/Nzp07geAnr5o0acLQoUMZMmQIs2fPLvTesqV1yimn\n8Mgjj8QS3bp162IJtFGjRmzatCnfNhd1/8pOnTrx3nvvsWbNGnbu3Mlzzz1X4pV+CxcupHPnztxx\nxx00btw4qe+WU5J//etfjBw5khkzZlCzZk127NhR5ivzNm/eHOvjzZs38+abb9KxY0dWrVrFkiVL\nWLJkCSkpKcyePZtDDz2UBQsWxI6KZ8+ezfbt2yP50EyEnJwcHnnkEVq3bs3777/Pyy+/zL///e/I\nf+y9T58+vPvuu0BwN6wdO3bQqFEjmjVrFrsX8ebNm/n4449p2bJlpOsub927d+faa6/lo48+Yvbs\n2bFHWaxevTr2c2dbt25lypQptGzZMl9/vvfee7Ej0d69ezN9+nRyc3PZsmULn3zyCa1atYpmwxJk\nwoQJscf48eO54YYbCh1t3BPPPfccF1xwQez5/PnzY9OvvPJKbB9s1qxZ7L7NK1eu5Jtvvol0FKXc\nbmFxzTXX0K9fPx599FHOPPPMSNp8+umnufTSS7n11lupWrUqL7zwQr4jmxEjRnD22WczduzYfPeS\nnDp1Kvfccw9Vq1aldu3ajB07tsh7y5bGkCFD+Pbbb8nIyKBq1aoMHTqUK664gqFDh5Kens6hhx5K\nx44dY/UHDhzI8OHDqVGjRr57sR522GGMHDmS7t274+6ceeaZ9O7du9h1X3vttcyfPx935+STT66Q\n9739+uuvadmyZezXZhYtWrTXba5cuTJ2aiA3N5cLL7ww9luuhXnxxRcZO3YsVatWpUaNGjz//PNF\n/qxasmvRogVVqlTh6quvplmzZsydOzffaZSynCa44IILmDp1KmvWrCElJYXbb7+dSy65hEsuuYT0\n9HSqVavGmDFjMDMuv/xyBg0aFBu+HzRoUIX/SbNPPvkEyD+0bmZl+u3eH374gQEDBrBz50527dpF\nv3796NWrFyeeeCL9+/dn1KhR1K5dm8cffxyAVq1a0bNnTzIyMqhUqRJDhgyJ9N8GE+G///1vbLpK\nlSqkpqbyyiuvlLm9zZs3M2XKFB555JFY2Q033MA333xDpUqVaN68OaNHjwbglltuYeDAgbRp0wZ3\n5y9/+QuNGjUq+8YUUOwdlZL5PrFSNsl4L9Bhw4bx6KOP0r17993mlfWDq7wkY38OGjQIyP+bqnkx\nmhlPPPFEQuIqjWTsz4pM/Rmd0t77V0n1AKM3WbSSsT/zbmwRn0gbN27MiSeemO93VpNRsvXnv//9\nby666KIif3km75eBklWy9Wee6667jj/+8Y/UqFGDnj17MnfuXEaNGsVFF12U6NCKFElS1b1/9z/V\nq1ffVdrbbUnJ1J/RUn9GS/0ZnerVq6/cunXroSXV00+/SdIwsz8B3YDWwCTgdGC6u5ftfpCSj5kd\nDLzl7u0SHYsc2Mxsnrunm9njwHh3f8PM5rh7xbsopAB9g5Fkcg5wMrDC3QcBmcD+/YvN5cjd1wEV\n8+qrBDOzv5pZXTOramZvm9lqM0vescrk96qZfQ20B942s8bAfvFzOkqqkky2uvsuINfM6gKrgCMS\nHNN+w8y6Az8mOo4K6lR33wD0ApYAvwCuTWhEFZi73wAcD3Rw9xxgM1D8vzpUEPvHr8LK/mKmmdUH\nHgNmAZuAj4pfRAoys8+Bgud1Dga+By4u/4j2C3mflWcCL7j7+or6L1dJpCWQambxeWhsooKJis6p\nSlKw4BMqxd2Xhs9TgbruXvh9KqVIZta8QJEDa919cyLi2R+Y2UigD7AV6ATUB151984JDayCMrOn\ngaOAbGBnWOzufmXiooqGkqokDTP73N3bJDoOkcKEF3qtd/edZlaT4EvfipKWk92Z2VdA6/3xfzY1\n/CvJZLaZdXT3GYkORCSemV0cNx0/q8IPVybIPOBQ4IdEBxI1JVVJJp2Bi8xsCcGFC0YwJFSx73En\n+4OOcdPVCa5Sn42Salk1Ar40s0+B7XmF7v7rxIUUDQ3/StIo5FwgAO7+XXnHIlKc8IK6/7h70TeU\nliKZWaG/FOLu75V3LFFTUpWEM7PqwHCCf1P4HPiXu5f9N/9E9jEzqwrMc/djEx2LJBcN/0oyGAPk\nAO8T3EWpNXBVQiMSiWNm/+Xnf1OqRLCPjktcRBWTmW1k93/3gp9P9UTzS+EJpCNVSbj4q37D/1n7\nVLfSk2RgZge5+/YCw5W5wHfuvixRcUny0pGqJIOcvAl3z9U/1UsS+QhoBwxx998mOhhJfkqqkgwy\nzWxDOG1AjfD5fjMkJBVWNTO7EDjezHb7dXd3n5CAmCSJKalKwrl75UTHIFKE4UB/gjsonVVgngNK\nqpKPzqmKiJTAzAa7+78SHYckPyVVEZFSMLPjgVTiRvjcXTd/kHw0/CsiUoKibgCP7qgkBehIVUSk\nBPvzDeAlWvqRchGRkuXdAF6kWBr+FREp2X57A3iJlpKqiEjJbkt0AFIx6JyqiEgpmFkTfv4JuE/d\nfVUi45HkpHOqIiIlMLN+wKfAuUA/4BMzOyexUUky0pGqiEgJzGwOcEre0amZNQbecvfMxEYmyUZH\nqiIiJatUYLh3Lfr8lELoQiURkZK9YWaTgefC5+cBkxIYjyQpDf+KiBTBzH4BNHH3D8JfqTkxnPUT\n8Iy7L0xcdJKMlFRFRIpgZq8CN7r75wXK2wB/dveCv1wjBzidExARKVqTggkVICxLLf9wJNkpqYqI\nFK1+MfNqlFsUUmEoqYqIFG2mmQ0tWGhmQ4BZCYhHkpzOqYqIFCG8i9JLwA5+TqIdgGpAX3dfkajY\nJDkpqYqIlMDMugPp4dMv3P2dRMYjyUtJVUREJCI6pyoiIhIRJVUREZGIKKmKiIhERElVREQkIkqq\nIrJPmFmqmX1lZo+Z2Rdm9qaZ1TCzoWY2w8zmmNmLZlYzrP+UmT1sZh+b2SIz62ZmT4RtPBXX7qlm\n9pGZzTazF8ysdsI2UqQAJVUR2ZeOBv7h7mkEN6E/G5jg7h3D3yL9ChgcV78B8Evg/wETgVFAGtDG\nzLLMrBHwR6CHu7cDZgK/L7etESmBfvpNRPalxe6eHU7PIrhfbrqZ3UlwC8DawOS4+v91dzezz4GV\neffdNbMvwmVTgNbAB2YGwU0YPiqH7RApFSVVEdmXtsdN7yS4X+5TQB93n2NmA4FuhdTfVWDZXQSf\nVzuBKe5+wT6KV2SvaPhXRMpbHeAHM6sK9N/DZT8GTgh/5xQzq2Vmx0QdoEhZKamKSHm7BfgE+AD4\nek8WdPfVwEDgOTObSzD02zLqAEXKSrcpFBERiYiOVEVERCKipCoiIhIRJVUREZGIKKmKiIhERElV\nREQkIkqqIiIiEVFSFRERicj/B4nGkYNSRVn8AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "data = session.get(test_td+'classification_count/').json()\n", "pd.DataFrame(data).set_index('name').plot.bar(table=True,\n", " title=\"Disribution of classes for task {name}\".format(name=\"test-set-tags-and-classes\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Analyzing the agreement of classes\n", "When labeling with more than one annotator per example, we should also look at the agreement level. LightTags *classification_agg* endpoint, which sits under a particular task definition, will give you those results. That is, you will see, per class, how many of its instances were agreed upon by 1,2..n labelers who saw that example" ] }, { "cell_type": "code", "execution_count": 184, "metadata": {}, "outputs": [], "source": [ "data = session.get(test_td+'classification_agg/').json() # Get the data\n", "class_agg = pd.DataFrame(data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is often instructive to pivot and plot this metric" ] }, { "cell_type": "code", "execution_count": 185, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
agrees123
name
Confusing246488
Insult27418134
Praise28210517
UNK14492
\n", "
" ], "text/plain": [ "agrees 1 2 3\n", "name \n", "Confusing 246 48 8\n", "Insult 274 181 34\n", "Praise 282 105 17\n", "UNK 144 9 2" ] }, "execution_count": 185, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class_agg_p = class_agg.pivot_table(index='name',columns='agrees',values='count')\n", "class_agg_p" ] }, { "cell_type": "code", "execution_count": 186, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 186, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc0AAAE+CAYAAADvQ0uqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8FmW99/HPV0ARQRElMlBhe8ITEiJamac8607MQ5kl\nhIdtT7062K7M6kk72lEf221Tt25PaVlqmZZ5Ck9pnhUNK1IUDBVJCFAM8Pf8cV23a1ise60BF2tm\nrfV9v17rteae4+++5pr5zVwz94wiAjMzM+vYWlUHYGZm1l04aZqZmZXkpGlmZlaSk6aZmVlJTppm\nZmYlOWmamZmV5KT5JkiaKWnfLlrWZEl3lRx3pKSQ1HdNx7U6JG0j6RFJCyV9YhWm20vS7DUY148l\nfbnw+aOSXpC0SNJG+f+/rYHlPiFpr86ebwfL/LqklyQ935XL7ciq1HOrn7zf2XI1ppsq6YQ1EVNn\nK71TlTQV2Al4a0S8tsYiqglJpwNbRsSHqo6lB/oc8PuIGFt1IEURcXKjW1I/4AfAbhHxaO498M0u\nQ9LFwOyI+FJhudu/2fmuYgybAZ8BNo+IFztpngFsFREzOmN+JZd5Op20ja6p+CWNBJ4G+kXEsq5e\nvnW+UmeaecW/Gwjgvau7sLqe+XSF3vzd27A58ETVQXRgGNCf+se5OjYD5q1OwnQ9tl4vIjr8A/4v\ncDfpyPv6VsM2An4N/BO4H/g6cFdheAAfA/4KPJ37jQZuBv4B/Bk4ujD+OsD3gGeBF4AfA+vmYXsB\ns0lnKi8Cc4CJwMHAX/L8TivMay3gVOBvwDzgKmBIHjYyxzYpL+sl4It52IHAv4ClwCLg0SblMhP4\nAvAn4GXgf4H+rWL9PPA8cFnufyjwCDAf+AMwpjC/RqwL8zwPLwyb3KpcvwvcBWwA9Mll9hLwVC7v\nAPrmcd8GXJfLZwZwYu7fH3gV2Dh//iKwDFg/f/4acHbuvhj4EXBDju+PwBbt1Jn3khLOfGAqsG3u\nfxuwHFiSy3brNqYdksvy77lcf1ks05LltSVwO7Agl8vPcn8BZ5Hqzz+BacAOhe/4dWBrYHEuw0XA\nbYW6vGXuXhf4PvBMXsZdtNTTn+d1vgC4A9g+9z+JVKf+lef760I92rdQ/8/O3/3vuXudVnXqM7TU\n/48UvvPBuRwWAs8B/9lG2e6b1/nrOYaL21tfhfg+DzwGvEauV4Xhd+SyWZzn+X5gQ+B6YG5eh9cD\nI1rV56dyrE8Dx3ZUz1sts81tlLQ9XJjL5rm8Pvt0UCdWir9Jnf58nudC0n7rPSX2M8/SUo8WAe9o\nY76rU36j8nQLgVtI2+blhe368hzLfNJ+eViT77QFaZucl8vkJ8DgVuv+P/O6XwD8jLyPy8M/m8v6\n78AUCttIk+UdRtr//TOX14G5/1TghJIxNVsPE4AH8rxfAH5QmGY30v52PvAosFdHdbHpd2hvYGGm\nM4D/A+xMqqTDCsN+mv8GANsBs1g5ad5M2hGuC6yXx/kIqXn47blgtsvjn0XawQ8BBpES8rcKO41l\npCTeDzgxV6or8rjbk3YIo/L4nwTuBUaQdkbnAVfmYSNzbBfkuHYi7RAaO/fTyZWwnXKZCTwObJrj\nvRv4eqtYv52XvW7+ri8Cu5IS3aQ8j8ZO8ShSgluLtOEsBjYp7kzysAuA3wED8rCTgScLcfyeFZPm\nHcB/kzamsbnM9ikMOyJ330SqyAcVhh1eSCjzSBWzL6ki/7RJuTSSzn55PX2OVIfWbr2BNJn+BtLG\nuWGefs9i0iiM1155XUk6CFgrf+/dc/8DgAeBwaQEum1hmosL629ksQwLdbmRNH+Uv8fwvC7fWViP\nU0j1sZEAHynM441ltKpHjaT5VVKdfQswlLShf61VnfpqLpeDgVeADfPwOcC7c/eGwLgm5du6HDta\nXzNJO7pNyQcGbcxzhZ0l6WD6CNJ+YRDpQKJx8LMeace2Tf68CS0HFpNpUs/bWObptNpGgWtJ2/l6\nuQzvA/6jvTrRVvxtLGsb0n7rbYX6scUq7Gf6Npv3qpZfHn4P6UB5bWD3XJ6NpPkfpP3mAFLd3Jl8\nINzGcrfM630dUn27g3ygXFj395G2syHAdODkPOxAUnLaIZf3Fe2VI2nfsSAvby3StjO69T6hvZg6\nWA/3AB/O3QNJl1bIy5lH2l7WyvOel+fdtC42XVftDcwz2Z2UKBtnI08Cn87dffKwbQrjt3WmuU/h\n8/uBO1st4zzgK6Sd2GIKZzDAO2g5Q92LlBQbR46D8vx3LYz/IDAxd08nH4UUCmQpaac/Mk9bPHq7\nD/hAsw2yjbKZ2ahA+fPBwN8Ksf6LFY/KziXvAAv9/kxOCm3M/xHgsMLO5I+kZHI1eYeWh93WKo79\n83frS9rRLQcGFYZ/i5YzjK8B5+RxnyftAM6k5Sx0ozzexcD/tPquTzaJ+8vAVYXPa5GODPdqvYG0\nMe0mpLOgDdsYtheFnX0H5XUpcH5x/eb++5BaJXYD1mo17GJKJM38fV4Fdiqx/QzO023Qehmt6lEj\naf4NOLgw7ABgZqv6X4zpRVp2Ds+Sdpht7iCblWOJ9TUTmNLBPDtKOmOBl3P3eqQj/iNolYRpp563\nMc/TKWyjpCb114rzBI4hXT9vWidKxr9lLut9Sdcni8PK7GdWKWl2UH6bkQ6eBhSGX05L0pxCq1as\nsn+klruHW9XNDxU+fwf4ce6+CDizMGzr9r4HaT9/VpNhU2m+T3gjpg7Wwx3AGeRcVej/eXJLX6Hf\n70gnLU3rYrO/Mtc0JwE3RcRL+fMVuR+kTN2XlPkbit1t9dsc2FXS/MYfcCzw1jy/AcCDhWE35v4N\n8yJiee5+Nf9/oTD8VVpu2NgcuLYwr+mkBDKsMH7x7sFXWPWbPYrf7RnSEVnD3IhYUvi8OfCZVt99\n08Y0ko7Ld5U2hu0AbFyYfktS88YZEfGvQv+3tRFHcdg/ImJhq+HDc/ftpJ3oOFJT5c3AnqSkMiMi\n5hWmK1tWbyvGEBGv5/iGNxm/aNMc78sdjdhBeX2OdBB2X747dUqO5Tbgv0hnii9KOl/S+iXiKtqY\ndFDxtzZi6iPpTEl/k/RP0k6nMU0ZK5QdK9epebHiDSXF9XAE6WDmGUm3S3rH6iyzyfpqa7tuStIA\nSedJeiaXwx3AYEl9ImIx6eD5ZGCOpBskjS5M3qyed2Rz0pnynEKdOI90xglN6kST+H+b75ZeJOnY\nSDfofIqUqF+U9FNJjfVSZj9TnPcThXm/u8k4TcuPlm36lcIkxfVzGSkp/FTS3yV9R1I/Se8uLPeJ\nvJxh+bs8l5dzOSvX1WbbfdP9jqTNCstalHtvShvbTBvfvWlMHayH40mJ+0lJ90s6NPffHDiq1X53\nd1ILU0d1cSXtJk1J6wJHA3tKej7fnv5pYCdJO5Ga+ZaRmiUaNm1jVlHongXcHhGDC38DI+KjpGba\nV0mnx41hG0TE6t61OIvU1FhcVv+IeK7EtNHxKMCK33czUtt+s3nMAr7RKp4BEXGlpM1JzVEfJ53d\nDSY1/aow/XRSs/ZvJW1T6D+njTga/g4MkTSo1fBGGfyB1ORxOGm9/CkPP5iUUFfH30kVFQBJyvGV\nKfdZOd7B7Y3UUXlFxPMRcWJEvI109vXfjVvhI+KciNiZdDlha9J1mVXxEuma7BZtDPsgaYe/L+n6\n2shGyPl/R/VqhbJj5TrVVETcHxGHkZLEL0nX1soos77Kbg8NnyHVq10jYn1gj8bsc6y/i4j9SGdl\nT5LWZUOzet5aW9vXa6Qzjcb2tX7ku5PbqxMrzTjioLxfGhgRP8n9roiI3UllFaRLL43lNtvPrFRu\nEbF9Yd53Nvlu7ZXfHNI2MqAw/hvbf0QsjYgzImI70mWDQ4HjIuLOwnIbd2x/M8e4Y17Oh1hxn9Oe\npvudiHi2sKzG/nsWbW8zrbUbU7P1EBF/jYhjSPX/28AvJDUuB17Wav2sFxFn5unaq4sr6ehMcyLp\niGk7UvPAWNI1oDtJK2E5cA1wej4yGg0c18E8rwe2lvThfPTTT9IukrbNR7gXAGdJeguApOGSDuhg\nns38GPhG3sEiaaikw0pO+wIwUlJHZfQxSSMkDSFdL/lZO+NeAJwsaVcl60k6JCe09UgVYG6O9SOk\nM6cVRMSVwGnALZIaFfAq4BM5jg1JNyU0xp9FSozfktRf0hjSEdnlefgrpCbtj9GSJP9AOvJa3aR5\nFXCIpPco/XTjM6Sd2R86mjAi5gC/Je3QNsz1Y482Rm23vCQdJalxMPdyHvf1XNd2zXEtJiW/11fl\ny+V6ehHwA0lvy2eX75C0DumSwWukayYDSDuAoheA9n7reSXwpVxXNyZdv7+8o5gkrS3pWEkbRMRS\n0nWast9rtddXQevvNYh0ADw/bxtfKcQ6TNJheYf2GunmlxVibVLP21rmG9torjs3Ad+XtL6ktSRt\nIWnPvNw260ST+Feg9NviffI6XkLLzVTQ/n5mbh6vo9/3li6/iHiGdMPL6Xm9vwP490Kse0vaMZ+V\n/pPUVNysLgwilf8CScNZtQPIq4DJkrbLCfwrHYx/IfCRXM/Wyvv2ts7qmsbU3nqQ9CFJQ/P2OT9P\n8jpp+/l3SQfkbbW/0m++R5Spiytpr+2W1DT6/Tb6H006Ze9Lajq9gZa7Z78N3FoYd6U2btIR1A2k\nCjWPdE1ubB7Wn7SjeSrPczrwiTxsL1a8FtM3z39kod9d5DZ40kHBKaTrhgtJTQPfzMNGsvI1q6m0\nXIzeKM/rZeChJuUzk5a7Z+cDl9Byc84KsRamOTCX03zSkdrPydcbgW+Q7nB9iXSn8u2FeCaz4rXi\nE0nNISNzOZyVy/JpVr57dgTpYOUfuQxObhXTt0iVr3Ejy8fz9MUbvi6mcC2u2fcrDD88l8uC/D22\nb6ucm0w7JJflC7n8r2my/tsrr++QzpQW5e98Uu7/HtKdgItouTNvYOvv2KR+vFGXSTd2nZ2X0bhL\ndl1S09WvSPXtGdJBZHG6rWi5e7pxY8xMWq5p9iddY56T/86h1R3ZbdTBfUk3hNyYy6uxLe7epHzb\nmk976+uN+NpZZyfneOeT9g9vy+t5Eeka8n80ypN0RN+4i7Vxt27jRsDJNKnnbSxzpW2UdHZ/Luku\n4wXAw7Tcp9BmnWgr/jaWNYZ0z8NCUp27npabUZruZ/Lwr5L2dfPJ15/fTPnl8bcgnbwsBG4lXau9\nMA87JseymLQNnUOTa6qkmycfzMt5hHTAVNzGVlj3rHwd+VRSLih79+zhpO1vIelmswPa2Pc2jamD\n9XA56XrnItKd4BMLy92VVOf+kdfFDaQz46Z1sdmf8gw7jaRvkx6AMKnDkc3M7E2T9DPSjXkdne3Z\nm/SmH6MnabSkMbm5cQKp6e/aNx+amZm1JV9m2CI3cx5Iuo7+y6rj6g064+keg0jXYd5Gagr4Pql5\nyszM1oy3ku4n2YjUFP3RiHi42pB6h05vnjUzM+up/JYTMzOzknr9w5c33njjGDlyZNVhmJl1Gw8+\n+OBLETG04zF7nl6fNEeOHMkDDzxQdRhmZt2GpGc6HqtncvOsmZlZSbVOmpJmSpqm9HzRB3K/IZJu\nlvTX/H/D3F+SzpE0Q9JjksZVG72ZmfU0tU6a2d4RMTYixufPp5KeOLQV6UkYjUfGHUR62spWpPcW\nntvlkZqZWY/WHa9pHkZ6DBikR61NJb365TDg0ki/oblX0mBJm0R6HqWZWa0sXbqU2bNns2TJko5H\nrkj//v0ZMWIE/fr1qzqU2qh70gzgJkkBnBcR55Oeh9pIhM/T8vqd4az4mprZud9KSVPSSaSzUTbb\nbLPWg83M1rjZs2czaNAgRo4ciVT2xSJdJyKYN28es2fPZtSoUVWHUxt1b57dPSLGkZpeP9b6bRf5\nrHKVn84QEedHxPiIGD90aK+8a9rMKrZkyRI22mijWiZMAElstNFGtT4TrkKtk2bk915GxIuk59lO\nAF6QtAlA/v9iHv05Vny32wjKvb/RzKwSdU2YDXWPrwq1TZr5XZODGt3A/qSXDF8HNN6gMomW59xe\nBxyX76LdDVjg65lmZtaZ6nxNcxhwbT7S6QtcERE3SrofuErS8aT37B2dx/8NcDDpHW2vkN78bmZm\n1mlqmzQj4ilgpzb6zyO9SLh1/yC9fNl6kB0v2bHqEEqZNmla1SFYL7ds2TL69q3tLr3HqG3zrJlZ\nbzRx4kR23nlntt9+e84//3wALrzwQrbeemsmTJjAiSeeyMc//nEAJk+ezMknn8yuu+7K5z73ORYv\nXsyUKVOYMGECb3/72/nVr9LVq+XLl/PZz36WXXbZhTFjxnDeeecBMGfOHPbYYw/Gjh3LDjvswJ13\n3lnNl+5GfFhiZlYjF110EUOGDOHVV19ll1124ZBDDuFrX/saDz30EIMGDWKfffZhp51aGuFmz57N\nH/7wB/r06cNpp53GPvvsw0UXXcT8+fOZMGEC++67Lz/5yU/YYIMNuP/++3nttdd417vexf777881\n11zDAQccwBe/+EWWL1/OK6+8UuE37x6cNM3MauScc87h2muvBWDWrFlcdtll7LnnngwZMgSAo446\nir/85S9vjH/UUUfRp08fAG666Sauu+46vve97wHpZy3PPvssN910E4899hi/+MUvAFiwYAF//etf\n2WWXXZgyZQpLly5l4sSJjB07tiu/arfkpGlmVhNTp07llltu4Z577mHAgAHstddejB49munTpzed\nZr311nujOyK4+uqr2WabbVYYJyL44Q9/yAEHHLDS9HfccQc33HADkydP5pRTTuG4447rvC/UA/ma\npplZTSxYsIANN9yQAQMG8OSTT3LvvfeyePFibr/9dl5++WWWLVvG1Vdf3XT6Aw44gB/+8Iek+yLh\n4YcffqP/ueeey9KlSwH4y1/+wuLFi3nmmWcYNmwYJ554IieccAIPPfTQmv+S3ZzPNM3MauLAAw/k\nxz/+Mdtuuy3bbLMNu+22G8OHD+e0005jwoQJDBkyhNGjR7PBBhu0Of2Xv/xlPvWpTzFmzBhef/11\nRo0axfXXX88JJ5zAzJkzGTduHBHB0KFD+eUvf8nUqVP57ne/S79+/Rg4cCCXXnppF3/j7keNI5Le\navz48eGXUNeXf3JiPdX06dPZdtttS427aNEiBg4cyLJlyzj88MOZMmUKhx9++BqOMGkrTkkPFt48\n1au4edbMrOZOP/30N34WMmrUKCZOnFh1SL2Wm2fNzGqucTesVc9nmmZmZiU5aZqZmZXkpGlmZlaS\nk6aZmVlJvhHIam3a089WHYJZlxh56g2dOr+ZZx7S4ThTpkzh+uuv5y1veQuPP/54py6/p/KZpplZ\nLzV58mRuvPHGqsPoVpw0zcx6qT322OONB8FbOU6aZmZmJTlpmpmZleSkaWZmVpKTppmZWUn+yYmZ\nWQ2U+YlIZzvmmGOYOnUqL730EiNGjOCMM87g+OOP7/I4uhMnTTOzXurKK6+sOoRux82zZmZmJTlp\nmpmZleSkaWZmVpKTppmZWUlOmmZmZiU5aZqZmZXkn5yYmdXB6Rt08vwWdDjKrFmzOO6443jhhReQ\nxEknncQnP/nJzo2jh3HSNDPrpfr27cv3v/99xo0bx8KFC9l5553Zb7/92G677aoOrbbcPGtm1ktt\nsskmjBs3DoBBgwax7bbb8txzz1UcVb05aZqZGTNnzuThhx9m1113rTqUWnPSNDPr5RYtWsQRRxzB\n2Wefzfrrr191OLVW+6QpqY+khyVdnz+PkvRHSTMk/UzS2rn/OvnzjDx8ZJVxm5l1B0uXLuWII47g\n2GOP5X3ve1/V4dRe7ZMm8ElgeuHzt4GzImJL4GWg8Uj+44GXc/+z8nhmZtZERHD88cez7bbbcsop\np1QdTrdQ67tnJY0ADgG+AZwiScA+wAfzKJcApwPnAoflboBfAP8lSRERXRmzWZ3teMmOVYdQyrRJ\n06oOoeuV+IlIZ7v77ru57LLL2HHHHRk7diwA3/zmNzn44IO7PJbuotZJEzgb+BwwKH/eCJgfEcvy\n59nA8Nw9HJgFEBHLJC3I47/UeqaSTgJOAthss83WWPBmZnW2++674/OKVVPb5llJhwIvRsSDnT3v\niDg/IsZHxPihQ4d29uzNzKyHqvOZ5ruA90o6GOgPrA/8P2CwpL75bHME0PhR0XPApsBsSX2BDYB5\nXR+2mZn1VLU904yIL0TEiIgYCXwAuC0ijgV+DxyZR5sE/Cp3X5c/k4ff5uuZZmbWmWqbNNvxedJN\nQTNI1ywvzP0vBDbK/U8BTq0oPjMz66Hq3Dz7hoiYCkzN3U8BE9oYZwlwVJcGZmZmvUq3SJpm1jmm\nPf1s1SGYdWtOmmZmNdDZv6Et81vXJUuWsMcee/Daa6+xbNkyjjzySM4444xOjaOncdI0M+ul1lln\nHW677TYGDhzI0qVL2X333TnooIPYbbfdqg6ttrrjjUBmZtYJJDFw4EAgPYN26dKlpAevWTNOmmZm\nvdjy5csZO3Ysb3nLW9hvv/38arAOOGmamfViffr04ZFHHmH27Nncd999PP7441WHVGtOmmZmxuDB\ng9l777258cYbqw6l1pw0zcx6qblz5zJ//nwAXn31VW6++WZGjx5dcVT15rtnzcxqoIrXoc2ZM4dJ\nkyaxfPlyXn/9dY4++mgOPfTQLo+jO3HSNDPrpcaMGcPDDz9cdRjdiptnzczMSnLSNDMzK8lJ08ys\nInV/e2Hd46uCk6aZWQX69+/PvHnzapuYIoJ58+bRv3//qkOpFd8IZGZWgREjRjB79mzmzp1bdShN\n9e/fnxEjRlQdRq04aZqZVaBfv36MGjWq6jBsFbl51szMrCQnTTMzs5KcNM3MzEpy0jQzMyvJSdPM\nzKwkJ00zM7OSnDTNzMxKctI0MzMryUnTzMysJCdNMzOzkpw0zczMSnLSNDMzK8lJ08zMrCQnTTMz\ns5KcNM3MzEpy0jQzMyvJSdPMzKwkJ00zM7OSap00JfWXdJ+kRyU9IemM3H+UpD9KmiHpZ5LWzv3X\nyZ9n5OEjq4zfzMx6llonTeA1YJ+I2AkYCxwoaTfg28BZEbEl8DJwfB7/eODl3P+sPJ6ZmVmn6Ft1\nAO2JiAAW5Y/98l8A+wAfzP0vAU4HzgUOy90AvwD+S5LyfLrEjpfs2FWLelOmTZpWdQhmZt1O3c80\nkdRH0iPAi8DNwN+A+RGxLI8yGxieu4cDswDy8AXARm3M8yRJD0h6YO7cuWv6K5iZWQ9R+6QZEcsj\nYiwwApgAjO6EeZ4fEeMjYvzQoUPfdIxmZtY71Lp5tigi5kv6PfAOYLCkvvlscgTwXB7tOWBTYLak\nvsAGwLyujHPa08925eLMzKwL1fpMU9JQSYNz97rAfsB04PfAkXm0ScCvcvd1+TN5+G1deT3TzMx6\ntrqfaW4CXCKpDynBXxUR10v6E/BTSV8HHgYuzONfCFwmaQbwD+ADVQRtZmY9U62TZkQ8Bry9jf5P\nka5vtu6/BDiqC0IzM7NeqNbNs2ZmZnVS6zNNM+tcI5dcUXUIpcysOgCzJnymaWZmVpKTppmZWUlO\nmmZmZiX5mqbVmq/BmVmd+EzTzMysJCdNMzOzkpw0zczMSnLSNDMzK8lJ08zMrCQnTTMzs5KcNM3M\nzEpy0jQzMyvJSdPMzKwkJ00zM7OS/Bi9TubHvpmZ9Vw+0zQzMyvJSdPMzKwkJ00zM7OSnDTNzMxK\nctI0MzMryUnTzMysJCdNMzOzkpw0zczMSnLSNDMzK8lJ08zMrCQnTTMzs5KcNM3MzEpy0jQzMyvJ\nSdPMzKwkJ00zM7OSnDTNzMxKqm3SlLSppN9L+pOkJyR9MvcfIulmSX/N/zfM/SXpHEkzJD0maVy1\n38DMzHqa2iZNYBnwmYjYDtgN+Jik7YBTgVsjYivg1vwZ4CBgq/x3EnBu14dsZmY9WW2TZkTMiYiH\ncvdCYDowHDgMuCSPdgkwMXcfBlwayb3AYEmbdHHYZmbWg9U2aRZJGgm8HfgjMCwi5uRBzwPDcvdw\nYFZhstm5X1vzO0nSA5IemDt37hqJ2czMep7aJ01JA4GrgU9FxD+LwyIigFjVeUbE+RExPiLGDx06\ntJMiNTOznq7WSVNSP1LC/ElEXJN7v9Bods3/X8z9nwM2LUw+IvczMzPrFLVNmpIEXAhMj4gfFAZd\nB0zK3ZOAXxX6H5fvot0NWFBoxjUzM3vT+lYdQDveBXwYmCbpkdzvNOBM4CpJxwPPAEfnYb8BDgZm\nAK8AH+nacM3MrKerbdKMiLsANRn8njbGD+BjazQoMzPr1WrbPGtmZlY3TppmZmYlOWmamZmV5KRp\nZmZWkpOmmZlZSU6aZmZmJTlpmpmZleSkaWZmVpKTppmZWUlOmmZmZiU5aZqZmZXkpGlmZlaSk6aZ\nmVlJtX3LiZlZ3e14yY5Vh9ChaZOmVR1Cj+KkaWa2mqY9/WzVIVgXc/OsmZlZSU6aZmZmJTlpmpmZ\nleRrmmZmq2nkkiuqDqFDM6sOoIfxmaaZmVlJTppmZmYlOWmamZmV5KRpZmZWkpOmmZlZSU6aZmZm\nJTlpmpmZleSkaWZmVpKTppmZWUlOmmZmZiU5aZqZmZXkpGlmZlaSk6aZmVlJTppmZmYl1TppSrpI\n0ouSHi/0GyLpZkl/zf83zP0l6RxJMyQ9JmlcdZGbmVlPVOukCVwMHNiq36nArRGxFXBr/gxwELBV\n/jsJOLeLYjQzs16i1kkzIu4A/tGq92HAJbn7EmBiof+lkdwLDJa0SddEamZmvUGtk2YTwyJiTu5+\nHhiWu4cDswrjzc79ViLpJEkPSHpg7ty5ay5SMzPrUbpj0nxDRAQQqzHd+RExPiLGDx06dA1EZmZm\nPVF3TJovNJpd8/8Xc//ngE0L443I/czMzDpFd0ya1wGTcvck4FeF/sflu2h3AxYUmnHNzMzetL5V\nB9AeSVcCewEbS5oNfAU4E7hK0vHAM8DRefTfAAcDM4BXgI90ecBmZtaj1TppRsQxTQa9p41xA/jY\nmo3IzMx6s+7YPGtmZlYJJ00zM7OSnDTNzMxKctI0MzMryUnTzMysJCdNMzOzkpw0zczMSnLSNDMz\nK8lJ08z+versAAAKsElEQVTMrCQnTTMzs5KcNM3MzEpy0jQzMyvJSdPMzKwkJ00zM7OSnDTNzMxK\nctI0MzMryUnTzMysJCdNMzOzkpw0zczMSnLSNDMzK8lJ08zMrCQnTTMzs5KcNM3MzEpy0jQzMyvJ\nSdPMzKwkJ00zM7OSnDTNzMxKctI0MzMryUnTzMysJCdNMzOzkpw0zczMSnLSNDMzK8lJ08zMrCQn\nTTMzs5J6XNKUdKCkP0uaIenUquMxM7Oeo0clTUl9gB8BBwHbAcdI2q7aqMzMrKfoUUkTmADMiIin\nIuJfwE+BwyqOyczMeghFRNUxdBpJRwIHRsQJ+fOHgV0j4uOtxjsJOCl/3Ab4c5cGuuo2Bl6qOoge\nxOXZuVyenas7lOfmETG06iCq0LfqAKoQEecD51cdR1mSHoiI8VXH0VO4PDuXy7NzuTzrrac1zz4H\nbFr4PCL3MzMze9N6WtK8H9hK0ihJawMfAK6rOCYzM+shelTzbEQsk/Rx4HdAH+CiiHii4rA6Q7dp\nSu4mXJ6dy+XZuVyeNdajbgQyMzNbk3pa86yZmdka46RpZmZWkpOmmZlZSU6a1itIOqpMP7OuImmD\ndob5d5o15aRZU5LGtfG3haQedcdzF/pCyX5WgqStJd0q6fH8eYykL1UdVzdzi6QNW/eUtD9wbQXx\nWAneAdfXfwPjgMcAATsATwAbSPpoRNxUZXDdhaSDgIOB4ZLOKQxaH1hWTVQ9wgXAZ4HzACLiMUlX\nAF+vNKru5Xzg95L2i4i5AJI+CHwDOKTSyKwpJ836+jtwfON3pvltLV8FPgdcAzhplvN34EHgvfl/\nw0Lg05VE1DMMiIj7JBX7+SBkFUTEBZKWALfls8v3AycDe0fEzEqDs6acNOtr6+KDGSLiT5JGR8RT\nrXZU1o6IeBR4VNLlEeGdeud5SdIWQMAbL0uYU21I3U9EXJYT58PAs8DuEVH3h7X3ak6a9fWEpHNJ\nrzeDdBT6J0nrAEurC6t7kTSNlh37SsMjYkxXx9RDfIzUvDha0nPA08CHqg2peynUTQEDgI1IZ50C\nwnWznvxEoJqStC7wf4Ddc6+7Sdc5l5CaxhZVFVt3Imnz9oZHxDNdFUtPJGk9YK2IWFh1LN2N62b3\n5KRpZqtM0ieB/yVdG76AdNPaqb5BzXo6J82akvQu4HRgcwrN6BHxb1XF1J1JWkhupgXWBvoBiyNi\n/eqi6r4kPRoRO0k6gHTzypeAyyJiXMWhdRuSnqalTkJqpm18jojYouujso74mmZ9XUi6u/NBYHnF\nsXR7ETGo0Z2vGR0G7FZdRN1e4wLxwcClEfGEfIfaqmr9AIO1gKOB/yTdGGQ15DPNmpL0x4jYteo4\nejJJD0fE26uOozuS9L/AcGAUsBPpVXxTI2LnSgPrhiStBXyY9LvXR4BvRsSfqo3KmnHSrClJZ5J2\nRNcArzX6R8RDlQXVjUl6X+HjWqSj/D0j4h0VhdSt5R39WOCpiJgvaSNgeEQ8VnFo3YakfsAUUovS\nXcCZETGj2qisI06aNSXp9230jojYp8uD6QHymVHDMmAmcEFEvFhNRN1T/q3wk5LavHbpg7ryJM0m\n1cWzSb/RXEFEXNPlQVmHnDTNrDRJ50fEST6oe/MKB3KtbwaCVJZTujgkK8FJs2YkfSgiLpd0SlvD\nI+IHXR1TTyDpO6Tnor4K3AiMAT4dEZdXGpj1WpI+Q8vDDcjdc4G7IuLpygKzdvktJ/WzXv4/qMmf\nrZ79I+KfwKGkptktSTde2GqStIOkoyUd1/irOqZuZiBpmx5Y6B4P/FbSB6oMzJrzmab1CpIej4gd\nJP0P8IuIuLHxW8OqY+uOJH0F2AvYDvgNcBDpDOnIKuPqCSQNAW7xb17ryWeaNSXpO5LWl9Qvv7dw\nriQ/23P1XS/pSWBn4FZJQ0mPJLTVcyTwHuD5iPgI6WcnTV+qbOVFxD9oabK1mnHSrC83J3aiiDgV\neCcwPiKWAotJDziw1fNqRLwOLJO0PvAisGnFMfUIkvYGXq46DmubnwhUX411cwjw84hY4AeuvGmj\ngZGSivX+0qqC6eYekDSY9NzZB4FFwD3VhtS9FN/AUzCE9A5YXx+uKV/TrKn8cIOJpLs9JwCDgev9\nlKDVI+kyYAvSE1cajyWMiPhEdVF1T/lxeSMiYlb+PBJY3w82WDVtvOUkgHkRsbiKeKwcJ80ayzcE\nLIiI5ZIGkHZMz1cdV3ckaTqwXbjCdwpJ0yJix6rjMOtqbp6tqeLt+62aZd2cuHoeB94KzKk6kB7i\nIUm7RMT9VQdi1pWcNOtrl0J3f9Kdig/hpLm6Ngb+JOk+VnyW73urC6lb2xX4kKSZpJuqRGruHlNp\nVGZrmJtnu4l808VPI+LAqmPpjiTt2Vb/iLi9q2PpCdq4HgdARDzT1bGYdSUnzW4ivxHh8YjYpupY\nrPeS1J/00uktgWnAhRGxrNqozLqOm2drStKvabkdfS3Sk1euqi6i7knSQla+rR9amhPX7+KQurtL\ngKXAnaSnAG0HfLLSiMy6kM80a0bSOhHxWqvmxGXAMxExu6q4zGDFu2bz713v8+PerDfxmWb93AOM\nA06IiA9XHYxZK0sbHRGxzA/csN7GSbN+1pb0QeCdkt7XeqBfTGsV20nSP3O3gHXzZzd3W6/gpFk/\nJwPHkp4A9O+thgXgpGmViYg+VcdgViVf06wpScdHxIVVx2FmZi2cNGtM0juBkRRaBCLCDzcwM6uI\nm2drqtkDxvETgczMKuMzzZryA8bNzOrHL6Gur8YDxs3MrCbcPFtffsC4mVnNOGnW1+lVB2BmZivy\nNc0akzSMlleE3RcRL1YZj5lZb+drmjUl6WjgPuAo4Gjgj5KOrDYqM7PezWeaNSXpUWC/xtmlpKHA\nLRGxU7WRmZn1Xj7TrK+1WjXHzsPry8ysUr4RqL5ulPQ74Mr8+f3AbyqMx8ys13PzbM1I2hIYFhF3\n57ec7J4HzQd+EhF/qy46M7PezUmzZiRdD3whIqa16r8j8M2IaP3mEzMz6yK+RlY/w1onTIDcb2TX\nh2NmZg1OmvUzuJ1h63ZZFGZmthInzfp5QNKJrXtKOgF4sIJ4zMws8zXNmslPAboW+BctSXI8sDZw\neEQ8X1VsZma9nZNmTUnaG9ghf3wiIm6rMh4zM3PSNDMzK83XNM3MzEpy0jQzMyvJSdPMzKwkJ00z\nM7OSnDTNuoCkkZKmS7pA0hOSbpK0rqQTJd0v6VFJV0sakMe/WNK5ku6V9JSkvSRdlOdxcWG++0u6\nR9JDkn4uaWBlX9KsF3DSNOs6WwE/iojtSQ/gPwK4JiJ2ye9JnQ4cXxh/Q+AdwKeB64CzgO2BHSWN\nlbQx8CVg34gYBzwAnNJl38asF/Krwcy6ztMR8UjufpD0LOEdJH2d9PjEgcDvCuP/OiJC0jTghcYz\niSU9kacdAWwH3C0J0gMw7umC72HWazlpmnWd1wrdy0nPEr4YmBgRj0qaDOzVxvivt5r2ddK2uxy4\nOSKOWUPxmlkrbp41q9YgYI6kfsCxqzjtvcC78jtYkbSepK07O0Aza+GkaVatLwN/BO4GnlyVCSNi\nLjAZuFLSY6Sm2dGdHaCZtfBj9MzMzErymaaZmVlJTppmZmYlOWmamZmV5KRpZmZWkpOmmZlZSU6a\nZmZmJTlpmpmZlfT/AT482+/QgcgsAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "class_agg_p.plot.bar(\n", " stacked=True,\n", " title=\"Agreement breakdown of classifications for task {name}\".format(name=\"test-set-tags-and-classes\")\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And transposing the pivot table can also be indicative, in this example it makes clear that only the Insult class gets significnt \"full agreement\"" ] }, { "cell_type": "code", "execution_count": 187, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 187, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc0AAAETCAYAAABUcV1CAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FfW9//HXh0Uji4CIqGyhbuxEQYELKlWrFmnRulz9\n0QqulWpbaxe1m7TaVlsr3t5eS3G5qLUKtWq97lXEvQIqyGpFjQIiAgKCrIHP74/vN2GSnJNM4CQn\nJu/n45FH5sx8Z+Yz3/nOfGY7Z8zdERERkeo1yXcAIiIinxdKmiIiIikpaYqIiKSkpCkiIpKSkqaI\niEhKSpoiIiIpKWnuBjMrNrMT6mheY83sxZRlC83MzaxZbce1K8zsMDObbWbrzew7NRhvuJktrcW4\nJprZzxKfx5nZCjPbYGbt4/8v1MJ855vZ8FxPt5p5Xmdmq8zso7qcb3Vq0s6l/on7nYN3YbzpZnZh\nbcSUa6l3qmY2HegP7O/uW2otonrCzMYDB7v71/MdSwP0I+BZdy/KdyBJ7n5JabeZNQduAga7+5zY\nu9XuzsPMJgNL3f2nifn23t3p1jCGrsD3gW7u/nGOpunAIe6+OBfTSznP8eRoG62t+M2sEHgPaO7u\nJXU9f8m9VGeaccUfDTjw1V2dWX0986kLjXnZM+gGzM93ENXoCBRQ/+PcFV2B1buSMNWOpdFz92r/\ngJ8DLxGOvB+pMKw98H/Ap8BM4DrgxcRwBy4F3gbei/16AP8EPgHeAs5KlN8TuBH4AFgBTAT2isOG\nA0sJZyofA8uBU4ERwL/j9H6cmFYT4CrgHWA1MBXYJw4rjLGNifNaBfwkDjsZ2ApsAzYAc7LUSzFw\nNbAAWAP8L1BQIdYrgY+Au2P/kcBsYC3wMtAvMb3SWNfHaZ6WGDa2Qr3+DngRaAM0jXW2Cng31rcD\nzWLZA4GHY/0sBi6K/QuATcC+8fNPgBJg7/j5WuDm2D0Z+B/g0Rjfq8BBVbSZrxISzlpgOtAz9p8G\nbAc2x7o9NMO4+8S6/DDW60PJOk1ZXwcDzwHrYr1Mif0NmEBoP58Cc4E+iWW8DjgU+CzW4QZgWqIt\nHxy79wJ+D7wf5/EiO9vp3+I6Xwc8D/SO/S8mtKmtcbr/l2hHJyTa/81x2T+M3XtWaFPfZ2f7Py+x\nzCNiPawHlgE/yFC3J8R1viPGMLmq9ZWI70rgTWALsV0lhj8f6+azOM3/BNoBjwAr4zp8BOhcoT2/\nG2N9DxhdXTuvMM+M2yhhe7g91s2yuD6bVtMmKsWfpU1fGae5nrDfOj7FfuYDdrajDcCQDNPdlfrr\nHsdbDzxN2Db/ktiu/xJjWUvYL3fMskwHEbbJ1bFO7gHaVlj3P4jrfh0whbiPi8N/GOv6Q+B8EttI\nlvmNIuz/Po31dXLsPx24MGVM2dbDUcCsOO0VwE2JcQYT9rdrgTnA8OraYtZlqGpgYqKLgW8BAwiN\ntGNi2H3xrwXQC1hC5aT5T8KOcC+gZSxzHuHy8OGxYnrF8hMIO/h9gNaEhPybxE6jhJDEmwMXxUb1\n11i2N2GH0D2W/y7wL6AzYWf0Z+DeOKwwxnZrjKs/YYdQunMfT2yEVdRLMTAP6BLjfQm4rkKsN8R5\n7xWX9WNgECHRjYnTKN0pnklIcE0IG85nwAHJnUkcdivwJNAiDrsEWJSI41nKJ83ngVsIG1NRrLPj\nEsNOj91PERrylxPDTksklNWEhtmM0JDvy1IvpUnnS3E9/YjQhvaouIFkGf9RwsbZLo5/bDJpJMpV\nVV/3Eg4CmsTlHhb7nwS8BrQlJNCeiXEmJ9ZfYbIOE225NGn+T1yOTnFd/kdiPZ5PaI+lCXB2Yhpl\n86jQjkqT5i8JbXY/oANhQ7+2Qpv6ZayXEcBGoF0cvhw4Ona3A47IUr8V67G69VVM2NF1IR4YZJhm\nuZ0l4WD6dMJ+oTXhQKL04KclYcd2WPx8ADsPLMaSpZ1nmOd4KmyjwIOE7bxlrMMZwDerahOZ4s8w\nr8MI+60DE+3joBrsZ5plm3ZN6y8Of4VwoLwHMCzWZ2nS/CZhv9mC0DYHEA+EM8z34Lje9yS0t+eJ\nB8qJdT+DsJ3tAywELonDTiYkpz6xvv9aVT0S9h3r4vyaELadHhX3CVXFVM16eAX4RuxuRbi1QpzP\nasL20iROe3Wcdta2mHVdVTUwTmQYIVGWno0sAr4Xu5vGYYclymc60zwu8fk/gRcqzOPPwDWEndhn\nJM5ggCHsPEMdTkiKpUeOreP0ByXKvwacGrsXEo9CEhWyjbDTL4zjJo/eZgBnZ9sgM9RNcWkDip9H\nAO8kYt1K+aOyPxF3gIl+bxGTQobpzwZGJXYmrxKSyd+JO7Q4bFqFOE6My9aMsKPbDrRODP8NO88w\nrgX+EMt+RNgBXM/Os9D2sdxk4LYKy7ooS9w/A6YmPjchHBkOr7iBZBj3AMJZULsMw4aT2NlXU193\nAZOS6zf2P45wVWIw0KTCsMmkSJpxeTYB/VNsP23jeG0qzqNCOypNmu8AIxLDTgKKK7T/ZEwfs3Pn\n8AFhh5lxB5mtHlOsr2Lg/GqmWV3SKQLWxO6WhCP+06mQhKminWeY5ngS2yjhkvqW5DSBcwj3z7O2\niZTxHxzr+gTC/cnksDT7mRolzWrqryvh4KlFYvhf2Jk0z6fCVay0f4Qrd29UaJtfT3z+LTAxdt8B\nXJ8YdmhVy0HYz0/IMmw62fcJZTFVsx6eB35BzFWJ/lcSr/Ql+j1JOGnJ2haz/aW5pzkGeMrdV8XP\nf439IGTqZoTMXyrZnalfN2CQma0t/QNGA/vH6bUAXksMeyL2L7Xa3bfH7k3x/4rE8E3sfGCjG/Bg\nYloLCQmkY6J88unBjdT8YY/ksr1POCIrtdLdNyc+dwO+X2HZu5SOY2bnxqdKS4f1AfZNjH8w4fLG\nL9x9a6L/gRniSA77xN3XVxjeKXY/R9iJHkG4VPlP4FhCUlns7qsT46WtqwOTMbj7jhhfpyzlk7rE\neNdUV7Ca+voR4SBsRnw69fwYyzTgj4QzxY/NbJKZ7Z0irqR9CQcV72SIqamZXW9m75jZp4SdTuk4\naZSrOyq3qdVe/oGS5Ho4nXAw876ZPWdmQ3ZlnlnWV6btOisza2Fmfzaz92M9PA+0NbOm7v4Z4eD5\nEmC5mT1qZj0So2dr59XpRjhTXp5oE38mnHFCljaRJf7H49PSG8xstIcHdC4nJOqPzew+MytdL2n2\nM8lpz09M++gsZbLWHzu36Y2JUZLr525CUrjPzD40s9+aWXMzOzox3/lxPh3jsiyL8/kLldtqtu0+\n637HzLom5rUh9u5Chm0mw7Jnjama9XABIXEvMrOZZjYy9u8GnFlhvzuMcIWpurZYSZVJ08z2As4C\njjWzj+Lj6d8D+ptZf8JlvhLCZYlSXTJMyhPdS4Dn3L1t4q+Vu48jXKbdRDg9Lh3Wxt139anFJYRL\njcl5Fbj7shTjevVFgPLL25VwbT/bNJYAv6oQTwt3v9fMuhEuR11GOLtrS7j0a4nxFxIuaz9uZocl\n+i/PEEepD4F9zKx1heGldfAy4ZLHaYT1siAOH0FIqLviQ0JDBcDMLMaXpt6XxHjbVlWouvpy94/c\n/SJ3P5Bw9nVL6aPw7v4Hdx9AuJ1wKOG+TE2sItyTPSjDsP9H2OGfQLi/VlgacvxfXbsqV3dUblNZ\nuftMdx9FSBIPEe6tpZFmfaXdHkp9n9CuBrn73sAxpZOPsT7p7l8inJUtIqzLUtnaeUWZtq8thDON\n0u1rb49PJ1fVJipN2P3Lcb/Uyt3vif3+6u7DCHXlhFsvpfPNtp+pVG/u3jsx7ReyLFtV9becsI20\nSJQv2/7dfZu7/8LdexFuG4wEznX3FxLzLX1i+9cxxr5xPl+n/D6nKln3O+7+QWJepfvvJWTeZiqq\nMqZs68Hd33b3cwjt/wbgfjMrvR14d4X109Ldr4/jVdUWK6nuTPNUwhFTL8LlgSLCPaAXCCthO/AA\nMD4eGfUAzq1mmo8Ah5rZN+LRT3MzO9LMesYj3FuBCWa2H4CZdTKzk6qZZjYTgV/FHSxm1sHMRqUc\ndwVQaGbV1dGlZtbZzPYh3C+ZUkXZW4FLzGyQBS3N7JSY0FoSGsDKGOt5hDOnctz9XuDHwNNmVtoA\npwLfiXG0IzyUUFp+CSEx/sbMCsysH+GI7C9x+EbCJe1L2ZkkXyYcee1q0pwKnGJmx1v46sb3CTuz\nl6sb0d2XA48TdmjtYvs4JkPRKuvLzM40s9KDuTWx7I7Y1gbFuD4jJL8dNVm42E7vAG4yswPj2eUQ\nM9uTcMtgC+GeSQvCDiBpBVDVdz3vBX4a2+q+hPv3f6kuJjPbw8xGm1kbd99GuE+Tdrl2eX0lVFyu\n1oQD4LVx27gmEWtHMxsVd2hbCA+/lIs1SzvPNM+ybTS2naeA35vZ3mbWxMwOMrNj43wztoks8Zdj\n4bvFx8V1vJmdD1NB1fuZlbFcdd/vTV1/7v4+4YGX8XG9DwG+koj1i2bWN56Vfkq4VJytLbQm1P86\nM+tEzQ4gpwJjzaxXTODXVFP+duC82M6axH17prO6rDFVtR7M7Otm1iFun2vjKDsI289XzOykuK0W\nWPjOd+c0bbGSqq7dEi6N/j5D/7MIp+zNCJdOH2Xn07M3AM8kyla6xk04gnqU0KBWE+7JFcVhBYQd\nzbtxmguB78Rhwyl/L6ZZnH5hot+LxGvwhIOCKwj3DdcTLg38Og4rpPI9q+nsvBndPk5rDfB6lvop\nZufTs2uBO9n5cE65WBPjnBzraS3hSO1vxPuNwK8IT7iuIjyp/FwinrGUv1d8EeFySGGshwmxLt+j\n8tOznQkHK5/EOrikQky/ITS+0gdZLovjJx/4mkziXly25UsMPy3Wy7q4HL0z1XOWcfeJdbki1v8D\nWdZ/VfX1W8KZ0oa4zBfH/scTngTcwM4n81pVXMYs7aOsLRMe7Lo5zqP0Kdm9CJeu/kFob+8TDiKT\n4x3CzqenSx+MKWbnPc0Cwj3m5fHvD1R4IjtDGzyB8EDIE7G+SrfFYVnqN9N0qlpfZfFVsc4uifGu\nJewfDozreQPhHvI3S+uTcERf+hRr6dO6pQ8CjiVLO88wz0rbKOHs/k+Ep4zXAW+w8zmFjG0iU/wZ\n5tWP8MzDekKbe4SdD6Nk3c/E4b8k7OvWEu8/7079xfIHEU5e1gPPEO7V3h6HnRNj+YywDf2BLPdU\nCQ9PvhbnM5twwJTcxsqteyrfR76KkAvSPj17GmH7W0942OykDPverDFVsx7+QrjfuYHwJPipifkO\nIrS5T+K6eJRwZpy1LWb7szjBnDGzGwg/gDCm2sIiIrLbzGwK4cG86s72ZDft9s/omVkPM+sXLzce\nRbj09+DuhyYiIpnE2wwHxcucJxPuoz+U77gag1z8ukdrwn2YAwmXAn5PuDwlIiK1Y3/C8yTtCZei\nx7n7G/kNqXHI+eVZERGRhkpvOREREUlJSVNERCSlRv/Ggn333dcLCwvzHYaIyOfGa6+9tsrdO1Rf\nsuFp9EmzsLCQWbNm5TsMEZHPDTN7v/pSDZMuz4qIiKSkpCkiIpKSkqaIiEhKjf6epog0DNu2bWPp\n0qVs3ry5+sKSSkFBAZ07d6Z58+b5DqXeUNIUkQZh6dKltG7dmsLCQsLbzWR3uDurV69m6dKldO/e\nPd/h1Bu6PCsiDcLmzZtp3769EmaOmBnt27fXmXsFSpoi0mAoYeaW6rMyJU0REZGUdE+zDi3s0TPf\nIdSqnosW5jsEEZFapTNNEZHdUFxcTM+ePbnooovo3bs3J554Ips2beLWW2/lyCOPpH///px++uls\n3LgRgLFjxzJu3DgGDx7MF77wBaZPn875559Pz549GTt2bNl0n3rqKYYMGcIRRxzBmWeeyYYNG/K0\nhJKkpCkispvefvttLr30UubPn0/btm35+9//zte+9jVmzpzJnDlz6NmzJ7fffntZ+TVr1vDKK68w\nYcIEvvrVr/K9732P+fPnM3fuXGbPns2qVau47rrrePrpp3n99dcZOHAgN910Ux6XUErp8qyIyG7q\n3r07RUVFAAwYMIDi4mLmzZvHT3/6U9auXcuGDRs46aSTysp/5Stfwczo27cvHTt2pG/fvgD07t2b\n4uJili5dyoIFCxg6dCgAW7duZciQIXW/YFKJkqaIyG7ac889y7qbNm3Kpk2bGDt2LA899BD9+/dn\n8uTJTJ8+vVL5Jk2alBu3SZMmlJSU0LRpU770pS9x77331tkySDq6PCsiUgvWr1/PAQccwLZt27jn\nnntqNO7gwYN56aWXWLx4MQCfffYZ//73v2sjTKkhJU0RkVpw7bXXMmjQIIYOHUqPHj1qNG6HDh2Y\nPHky55xzDv369WPIkCEsWrSoliKVmjB3z3cMeTVw4ECvq/dp6isnIrVn4cKF9OzZsLexfMhUr2b2\nmrsPzFNIeaUzTRERkZSUNEVERFJS0hQREUlJSVNERCQlJU0REZGUlDRFRERSUtIUEcmhjz76iLPP\nPpuDDjqIAQMGMGLEiF36YYIXXniB3r17U1RUxKZNm2o07sSJE7nrrrtqPE+pnn5GT0QapMKrHs3p\n9IqvP6XaMu7OaaedxpgxY7jvvvsAmDNnDitWrODQQw+t0fzuuecerr76ar7+9a/XONZLLrmkxuNI\nOjrTFBHJkWeffZbmzZuXS1r9+/dn2LBh/PCHP6RPnz707duXKVOmADB9+nSGDx/OGWecQY8ePRg9\nejTuzm233cbUqVP52c9+xujRo5k+fTojR44sm+Zll13G5MmTAbjqqqvo1asX/fr14wc/+AEA48eP\n58YbbwRg+PDhXHnllRx11FEceuihvPDCCwBs3LiRs846i169enHaaacxaNAg6uqHXj7PdKYpIpIj\n8+bNY8CAAZX6P/DAA8yePZs5c+awatUqjjzySI455hgA3njjDebPn8+BBx7I0KFDeemll7jwwgt5\n8cUXGTlyJGeccUa5H3tPWr16NQ8++CCLFi3CzFi7dm3GciUlJcyYMYPHHnuMX/ziFzz99NPccsst\ntGvXjgULFjBv3ryyt7RI1XSmKSJSy1588UXOOeccmjZtSseOHTn22GOZOXMmAEcddRSdO3emSZMm\nFBUVUVxcnHq6bdq0oaCggAsuuIAHHniAFi1aZCz3ta99Ddj52rLSmM4++2wA+vTpQ79+/XZ9ARsR\nJU0RkRzp3bs3r732Wo3GqfhasZKSkkplmjVrxo4dO8o+b968uaz/jBkzOOOMM3jkkUc4+eSTq5xH\ntulLekqaIiI5ctxxx7FlyxYmTZpU1u/NN9+kbdu2TJkyhe3bt7Ny5Uqef/55jjrqqNTT7datGwsW\nLGDLli2sXbuWZ555BoANGzawbt06RowYwYQJE5gzZ07qaQ4dOpSpU6cCsGDBAubOnZt63MZM9zRF\nRHLEzHjwwQe5/PLLueGGGygoKKCwsJCbb76ZDRs20L9/f8yM3/72t+y///6pX/fVpUsXzjrrLPr0\n6UP37t05/PDDgfDOzlGjRrF582bcnZtuuil1rN/61rcYM2YMvXr1okePHvTu3Zs2bdrs0nI3Jno1\nmF4NljN6NZjkk14NVjPbt29n27ZtFBQU8M4773DCCSfw1ltvsccee5Qrp1eDlaczTRGRRmjjxo18\n8YtfZNu2bbg7t9xyS6WEKZUpaYqINEKtW7fW9zJ3gR4EEhERSUlJU0REJCUlTRERkZSUNEVERFKq\n10nTzL5nZvPNbJ6Z3WtmBWbW3cxeNbPFZjbFzPaIZfeMnxfH4YX5jV5EGptWrVrldHrFxcX06dMH\ngNmzZ/PYY4/ldPpSc/X26Vkz6wR8B+jl7pvMbCpwNjACmODu95nZROAC4E/x/xp3P9jMzgZuAP4z\nT+GLSL6Nz/EX9cevy+30amj27NnMmjWLESNG5DWOxq5en2kSkvpeZtYMaAEsB44D7o/D7wROjd2j\n4mfi8OPNzOowVhERIPsrvyDzq7zGjh3L/fffXzZ+xTPWrVu38vOf/5wpU6ZQVFRU9moxqXv19kzT\n3ZeZ2Y3AB8Am4CngNWCtu5f+4vBSoFPs7gQsieOWmNk6oD2wquK0zexi4GKArl271uZilHPW1fW2\nunNCv1wpslOmV3717Nkz1au8Ktpjjz345S9/yaxZs/jjH/9Yy5FLVertmaaZtSOcPXYHDgRaApl/\nwr+G3H2Suw9094EdOnTIxSRFRMrJ9MqvtK/ykvqr3iZN4ATgPXdf6e7bgAeAoUDbeLkWoDOwLHYv\nA7oAxOFtgNV1G7KISJDplV/ZXuWVfPXXjh072Lp1a15ilurV56T5ATDYzFrEe5PHAwuAZ4EzYpkx\nwD9i98PxM3H4NG/sv0YvIvVKtld5FRYWlr2H8+GHH2bbtm2Vxm3dujXr16+v03ilsnqbNN39VcID\nPa8Tbpc1ASYBVwJXmNliwj3L2+MotwPtY/8rgKvqPGgRkSqsX7+ekSNH0q9fP4YNG1b2Kq+LLrqI\n5557jv79+/PKK6/QsmXLSuN+8YtfZMGCBXoQKM/0arA6fDVY3zv71sl88mXuGD0KJPmjV4PVDr0a\nrLx6e6YpIiJS3yhpioiIpKSkKSIikpKSpoiISEpKmiIiIikpaYqIiKSkpCkikiNNmzalqKiIPn36\ncOaZZ7Jx48YajX/hhReyYMGCWopOcqFh/4K4iDRauf5edJrvIe+1117Mnj0bgNGjRzNx4kSuuOKK\nsuHujrvTpEnm85XbbrstN8FKrdGZpohILTj66KNZvHgxxcXFHHbYYZx77rn06dOHJUuWMG7cOAYO\nHEjv3r255pprysYZPnw4s2bNYvv27YwdO5Y+ffrQt29fJkyYAMA777zDySefzIABAzj66KNZtGhR\nvhav0dKZpohIjpWUlPD444+X/SD722+/zZ133sngwYMB+NWvfsU+++zD9u3bOf7443nzzTfp169f\n2fizZ89m2bJlzJs3D6DsFWIXX3wxEydO5JBDDuHVV1/lW9/6FtOmTavjpWvclDRFRHJk06ZNFBUV\nAeFM84ILLuDDDz+kW7duZQkTYOrUqUyaNImSkhKWL1/OggULyiXNL3zhC7z77rt8+9vf5pRTTuHE\nE09kw4YNvPzyy5x55pll5bZs2VJ3CyeAkqaISM4k72kmJX+A/b333uPGG29k5syZtGvXjrFjx7J5\n8+Zy5du1a8ecOXN48sknmThxIlOnTuXmm2+mbdu2GacvdUf3NEVE6tCnn35Ky5YtadOmDStWrODx\nxx+vVGbVqlXs2LGD008/neuuu47XX3+dvffem+7du/O3v/0NCA8Vlb5aTOqOzjRFROpQ//79Ofzw\nw+nRowddunRh6NChlcosW7aM8847r+zF1L/5zW8AuOeeexg3bhzXXXcd27Zt4+yzz6Z///51Gn9j\np1eD6dVgOaNXg0k+6dVgtUOvBitPl2dFRERSUtIUERFJSUlTREQkJSVNERGRlJQ0RUREUlLSFBER\nSUlJU0QkR4qLi+nTp0+5fuPHj+fGG29k7NixdOrUqeyn71atWkVhYWHG8W699VYGDBjAmjVr6ix2\nSUc/biAiDdLCHrn9zmbPRQt3expNmzbljjvuYNy4cVnL3H333fz3f/8306ZNo127drs9T8ktnWmK\niNSRyy+/nAkTJlBSUpJx+NSpU7n++ut56qmn2Hfffes4OklDSVNEpI507dqVYcOGcffdd1ca9v77\n73PZZZfx1FNPsf/+++chOklDSVNEJEfMrNr+V199Nb/73e/Kfle2VIcOHejatStTp06t1Rhl9+ie\npohIjrRv377SwzuffPIJ3bt3L/t8yCGHUFRUVCk5tmjRgscee4yjjz6a/fbbj9GjR9dJzFIzOtMU\nEcmRVq1accABBzBt2jQgJMwnnniCYcOGlSv3k5/8hBtvvLHS+Pvttx9PPPEEP/7xj3nyySfrJGap\nGSVNEZEcuuuuu7j22mspKiriuOOO45prruGggw4qV6Z3794cccQRGcfv3r07Dz/8MOeffz4zZsyo\ni5ClBvRqML0aLGf0ajDJJ70arHbo1WDl6UxTREQkJSVNERGRlJQ0RUREUlLSFJEGo7E/o5Frqs/K\n6nXSNLO2Zna/mS0ys4VmNsTM9jGzf5rZ2/F/u1jWzOwPZrbYzN40s8yPpolIg1RQUMDq1au1o88R\nd2f16tUUFBTkO5R6pb7/uMF/AU+4+xlmtgfQAvgx8Iy7X29mVwFXAVcCXwYOiX+DgD/F/yLSCHTu\n3JmlS5eycuXKfIfSYBQUFNC5c+d8h1Gv1NukaWZtgGOAsQDuvhXYamajgOGx2J3AdELSHAXc5eEw\n81/xLPUAd19ex6GLSB40b9683C/viNSG+nx5tjuwEvhfM3vDzG4zs5ZAx0Qi/AjoGLs7AUsS4y+N\n/URERHKiPifNZsARwJ/c/XDgM8Kl2DLxrLLGNzDM7GIzm2Vms3QpR0RE0qrPSXMpsNTdX42f7yck\n0RVmdgBA/P9xHL4M6JIYv3PsV4m7T3L3ge4+sEOHDrUSvIiINDz1Nmm6+0fAEjM7LPY6HlgAPAyM\nif3GAP+I3Q8D58anaAcD63Q/U0REcqnePggUfRu4Jz45+y5wHiHRTzWzC4D3gbNi2ceAEcBiYGMs\nKyIikjP1Omm6+2wg048CH5+hrAOX1npQIiLSaNXby7MiIiL1jZKmiIhISkqaIiIiKSlpioiIpKSk\nKSIikpKSpoiISEpKmiIiIikpaYqIiKSkpCkiIpKSkqaIiEhKSpoiIiIpKWmKiIikpKQpIiKSUr1+\ny0lDM/e9D/IdgoiI7AadaYqIiKSkpCkiIpKSkqaIiEhKSpoiIiIpKWmKiIikpKQpIiKSkpKmiIhI\nSkqaIiIiKSlpioiIpKSkKSIikpKSpoiISEpKmiIiIikpaYqIiKSkpCkiIpKSkqaIiEhKSpoiIiIp\nKWmKiIikpKQpIiKSkpKmiIhISkqaIiIiKSlpioiIpFTvk6aZNTWzN8zskfi5u5m9amaLzWyKme0R\n++8ZPy+OwwvzGbeIiDQ89T5pAt8FFiY+3wBMcPeDgTXABbH/BcCa2H9CLCciIpIz9Tppmlln4BTg\ntvjZgOPvLU2AAAAI/ElEQVSA+2ORO4FTY/eo+Jk4/PhYXkREJCfqddIEbgZ+BOyIn9sDa929JH5e\nCnSK3Z2AJQBx+LpYXkREJCfqbdI0s5HAx+7+Wi1M+2Izm2Vms1auXJnryYuISANVb5MmMBT4qpkV\nA/cRLsv+F9DWzJrFMp2BZbF7GdAFIA5vA6zONGF3n+TuA919YIcOHWpvCUREpEGpt0nT3a92987u\nXgicDUxz99HAs8AZsdgY4B+x++H4mTh8mrt7HYYsIiINXL1NmlW4ErjCzBYT7lneHvvfDrSP/a8A\nrspTfCIi0kA1q75I/rn7dGB67H4XOCpDmc3AmXUamIiINCqfxzNNERGRvFDSFBERSUlJU0REJCUl\nTRERkZSUNEVERFJS0hQREUlJSVNERCSlz8X3NEXqg4U9euY7hFrVc9HC6guJNHI60xQREUlJZ5oi\nKZ11dcPeXObmOwCRzwGdaYqIiKSkpCkiIpKSkqaIiEhKSpoiIiIpKWmKiIikpKQpIiKSkpKmiIhI\nSkqaIiIiKSlpioiIpKSkKSIikpKSpoiISEpKmiIiIikpaYqIiKSkpCkiIpKSkqaIiEhKSpoiIiIp\nKWmKiIikpKQpIiKSkpKmiIhISs3yHUBjUrj5r/kOoVYV5zsAEZFapjNNERGRlJQ0RUREUtLlWZGU\n5r73Qb5DEJE805mmiIhISkqaIiIiKdXbpGlmXczsWTNbYGbzzey7sf8+ZvZPM3s7/m8X+5uZ/cHM\nFpvZm2Z2RH6XQEREGpp6mzSBEuD77t4LGAxcama9gKuAZ9z9EOCZ+Bngy8Ah8e9i4E91H7KIiDRk\n9TZpuvtyd389dq8HFgKdgFHAnbHYncCpsXsUcJcH/wLamtkBdRy2iIg0YPU2aSaZWSFwOPAq0NHd\nl8dBHwEdY3cnYElitKWxn4iISE7U+6RpZq2AvwOXu/unyWHu7oDvwjQvNrNZZjZr5cqVOYpUREQa\nunqdNM2sOSFh3uPuD8TeK0ovu8b/H8f+y4AuidE7x36VuPskdx/o7gM7dOhQO8GLiEiDU2+TppkZ\ncDuw0N1vSgx6GBgTu8cA/0j0Pzc+RTsYWJe4jCsiIrLb6vMvAg0FvgHMNbPZsd+PgeuBqWZ2AfA+\ncFYc9hgwAlgMbATOq9twRUSkoau3SdPdXwQsy+DjM5R34NJaDUpERBq1ent5VkREpL5R0hQREUlJ\nSVNERCQlJU0REZGUlDRFRERSUtIUERFJSUlTREQkJSVNERGRlJQ0RUREUlLSFBERSUlJU0REJCUl\nTRERkZSUNEVERFJS0hQREUlJSVNERCQlJU0REZGUlDRFRERSUtIUERFJSUlTREQkJSVNERGRlJQ0\nRUREUlLSFBERSUlJU0REJCUlTRERkZSUNEVERFJqlu8ARETqwsIePfMdQq3puWhhvkNoNJQ0RaRR\nOOvqhru7m5vvABoRXZ4VERFJqeEeeonkWOHmv+Y7hFpVnO8ARD4HlDRFpFGY+94H+Q5BGgBdnhUR\nEUlJSVNERCQlJU0REZGUlDRFRERSUtIUERFJqcElTTM72czeMrPFZnZVvuMREZGGo0ElTTNrCvwP\n8GWgF3COmfXKb1QiItJQNLTvaR4FLHb3dwHM7D5gFLAgr1GJSN415B+nKM53AI1IQ0uanYAlic9L\ngUEVC5nZxcDF8eMGM3urDmLLh32BVXU1M7uhrubUaGj9fb7V2frLw7rrVudzrCcaWtJMxd0nAZPy\nHUdtM7NZ7j4w33HIrtH6+3zT+muYGtQ9TWAZ0CXxuXPsJyIistsaWtKcCRxiZt3NbA/gbODhPMck\nIiINRIO6POvuJWZ2GfAk0BS4w93n5zmsfGrwl6AbOK2/zzetvwbI3D3fMYiIiHwuNLTLsyIiIrVG\nSVNERCQlJU0REZGUlDRF6gkz62Fmx5tZqwr9T85XTJKOmR1lZkfG7l5mdoWZjch3XJJ7ehCoETCz\n89z9f/Mdh2RnZt8BLgUWAkXAd939H3HY6+5+RD7jk+zM7BrC7103A/5J+BWyZ4EvAU+6+6/yGJ7k\nmJJmI2BmH7h713zHIdmZ2VxgiLtvMLNC4H7gbnf/LzN7w90Pz2uAklVcd0XAnsBHQGd3/9TM9gJe\ndfd+eQ1QcqpBfU+zMTOzN7MNAjrWZSyyS5q4+wYAdy82s+HA/WbWjbAOpf4qcfftwEYze8fdPwVw\n901mtiPPsUmOKWk2HB2Bk4A1Ffob8HLdhyM1tMLMitx9NkA84xwJ3AH0zW9oUo2tZtbC3TcCA0p7\nmlkbQEmzgVHSbDgeAVqV7nSTzGx63YcjNXQuUJLs4e4lwLlm9uf8hCQpHePuWwDcPZkkmwNj8hOS\n1Bbd0xQREUlJXzkRERFJSUlTREQkJSVNERGRlJQ0ReohM9NDeiL1kJKmSC0ys4fM7DUzm29mF8d+\nF5jZv81shpndamZ/jP0nm9lEM3sV+K2ZtTSzO2K5N8xsVCzX1Mx+Z2YzzexNM/tm7H+AmT1vZrPN\nbJ6ZHZ23BRdpoHQ0K1K7znf3T+Kvw8w0s0eBnwFHAOuBacCcRPnOwH+4+3Yz+zUwzd3PN7O2wAwz\nexoYDaxz9yPNbE/gJTN7Cvga8WfbzKwp0KLuFlOkcVDSFKld3zGz02J3F+AbwHPu/gmAmf0NODRR\n/m/x12UATgS+amY/iJ8LgK6xfz8zOyP2bwMcAswE7jCz5sBDmb6zKyK7R0lTpJbEn8I7gfCbshvj\nj0wsAnpWMdpnyUkAp7v7WxWma8C33f3JDPM8BjgFmGxmN7n7Xbu3FCKSpHuaIrWnDbAmJswewGCg\nJXCsmbWLD/ucXsX4TwLfjkkSMzs80X9cPKPEzA6N9z+7ASvc/VbgNsIlYBHJIZ1pitSeJ4BLzGwh\n8BbwL2AZ8GtgBvAJ4cxzXZbxrwVuBt40sybAe8BIQkIsBF6PCXUlcCowHPihmW0DNhB+mk9Eckg/\noydSx8ysVfxB9mbAg8Ad7v5gvuMSkerp8qxI3RtvZrOBeYSzx4fyHI+IpKQzTRERkZR0pikiIpKS\nkqaIiEhKSpoiIiIpKWmKiIikpKQpIiKSkpKmiIhISv8feMyRXCByKCcAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "class_agg_p.T.plot.bar(\n", " stacked=True,\n", " title=\"Agreement breakdown of classifications for task {name}\".format(name=\"test-set-tags-and-classes\")\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Classsiffication Confusion Matrix\n", "As we see above, often times our labelers do not agree. In the example above (on synthetic data) we see that most of the time our labelers do not agree at all, regardless of the class. In these instances, it is often convenient to construct a confusion matrix and see which classes our labelers confusing the most. In turn, this lets us update our instructions or Schema to ensure quality data\n", "LightTag provides a confusion matrix for classifications under the **classification_confusion** endpoint of a taskdefinition" ] }, { "cell_type": "code", "execution_count": 188, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
class_aclass_bcount
0ConfusingConfusing72
1ConfusingInsult175
2ConfusingPraise105
3ConfusingUNK31
4InsultConfusing130
5InsultInsult283
6InsultPraise208
7InsultUNK76
8PraiseConfusing117
9PraiseInsult248
10PraisePraise156
11PraiseUNK49
12UNKConfusing30
13UNKInsult73
14UNKPraise47
15UNKUNK15
\n", "
" ], "text/plain": [ " class_a class_b count\n", "0 Confusing Confusing 72\n", "1 Confusing Insult 175\n", "2 Confusing Praise 105\n", "3 Confusing UNK 31\n", "4 Insult Confusing 130\n", "5 Insult Insult 283\n", "6 Insult Praise 208\n", "7 Insult UNK 76\n", "8 Praise Confusing 117\n", "9 Praise Insult 248\n", "10 Praise Praise 156\n", "11 Praise UNK 49\n", "12 UNK Confusing 30\n", "13 UNK Insult 73\n", "14 UNK Praise 47\n", "15 UNK UNK 15" ] }, "execution_count": 188, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = session.get(test_td+'classification_confusion/').json() #Fetch the confusion matrix\n", "class_confusion = pd.DataFrame(data)\n", "class_confusion" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, it is often convenient to pivot the data" ] }, { "cell_type": "code", "execution_count": 189, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
class_bConfusingInsultPraiseUNK
class_a
Confusing7217510531
Insult13028320876
Praise11724815649
UNK30734715
\n", "
" ], "text/plain": [ "class_b Confusing Insult Praise UNK\n", "class_a \n", "Confusing 72 175 105 31\n", "Insult 130 283 208 76\n", "Praise 117 248 156 49\n", "UNK 30 73 47 15" ] }, "execution_count": 189, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class_confusion_p= (class_confusion.pivot_table(index='class_a',columns='class_b',values='count'))\n", "class_confusion_p\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And visualize it as a heatmap\n" ] }, { "cell_type": "code", "execution_count": 190, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
class_bConfusingInsultPraiseUNK
class_a
Confusing0.1879900.6449090.9190601.0
Insult0.1865140.5925390.8909611.0
Praise0.2052630.6403510.9140351.0
UNK0.1818180.6242420.9090911.0
\n", "
" ], "text/plain": [ "class_b Confusing Insult Praise UNK\n", "class_a \n", "Confusing 0.187990 0.644909 0.919060 1.0\n", "Insult 0.186514 0.592539 0.890961 1.0\n", "Praise 0.205263 0.640351 0.914035 1.0\n", "UNK 0.181818 0.624242 0.909091 1.0" ] }, "execution_count": 190, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class_confusion_p.div(class_confusion_p.sum(1),axis=0).cumsum(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tag Metrics\n", "The metrics we are interested in with tags are similar to classifications with some subtle differences. In LightTag, a single example has only one classification but can have many tags. Additionaly, a tag has a location as well as a type. For example in \"President Donald Trump\" Donald Trump has type \"Person\", starts at 11 and ends at 23.\n", "Thus when we calculate agreement things get a little more subtle" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fetching counts and agreements for tags\n", "Exactly as we did for classsifications, we can fetch the distribution of each tag from a taskdefinitions **tag_count** endpoint" ] }, { "cell_type": "code", "execution_count": 191, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 191, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAFsCAYAAADFQW5GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8XfOd//HXOxcJEuKSKoKTVorEJdFE9KKVaalQdWsN\nVRVFysTQTjtT2pkfvzFa7bRqlGGoa6lbqSpRNX7uLZIIibiMlJiEiBAiRMjl8/tjfXeyc3pO9rns\nc9ZZe72fj8d5nL2/e10+a+2112d9v991UURgZmbl0yvvAMzMLB9OAGZmJeUEYGZWUk4AZmYl5QRg\nZlZSTgBmZiVVmgQg6WJJ/9LBcWdJ2ju9PlPSNXWM6/uSflmv6bVjvodImivpHUmjunv+rZG0g6Qn\nJC2RdEre8VSTNEfS5/OOw9qvo79bSXtLmtcVMfUEffIOoB4kzQG2AFYAK4GngauBSyJiFUBEnNjR\n6UfEiDqESUoi10TEkKpp/7Ae0+6AnwInR8TvWvpQUgDDImJ294bFPwH3RsTIekxM0pnA9hHxtXpM\nrx3zrcv668r4JV0JzIuIf85j/pa/RqoBHBgRA4HtgHOA7wGXdWaCkhoiQbZiO2BW3kG0oMNxNfj3\nZVZ/EVH4P2AO8PlmZXsAq4Cd0/srgX9LrzcHbgfeAhYBDwK9qqb1PWAG8D5ZLWn19IEzgd8ANwBL\ngMeB3armG2RHTFTPF9gQeC/F9E762ypN75qq4b9EtgN8C7gP2KnZcn43xbY4xdC/lXXSC/hn4CXg\nNbIa0cZAvzTvAN4F/tLCuA9Uff4O8LfAJmmdLQTeTK+HVI0zNI23BPhv4MLKcgH9gWuAN9JyTQG2\naGG+/4+sBrcszfdjKear03xfSstU+a4mAA8DP0/T/rdm09sP+ABYnqb3ZCo/FngmxfoC8M2qcWpt\nG5XtYCfgReDItqy/VP5F4Ik07T8Bu1aN8z3g5RTTc8DnWou/hfltD9yftonXgRuqPtsRuDsty3PA\n4al8YpruB2nav29huu1ef+nzfwLmA68Ax1P1mwD2J6uhL0nL+911/K5PA/6Shn0aOKTqswnAQ2Q1\n2TfTdzG+2fZ4fxr3buACqn5nLcxrU+CKFPObwK2pfG+yWlJbYmrxewBEto2+BrwNzGTNfqlfWob/\nBRYAFwPr19oW67bv7Iodcnf/0UICSOX/C5yUXl/JmgTwo7Si+6a/vQBVTesJYJuqL2L19Ml22MuB\nL6dxv5s2vr7p8xYTQEsbU9X0KjvKj5HtNPZJ0/4nYDawXlUcj5Eljk3JfoQntrJOvpHG/QgwALgF\n+FXV52vF2cL4zZdjM+AwYANgIHBT5UeSPv9z2pDXAz6dNvTKcn0T+H0atzfwcWCjVuZ7H3B81fur\ngd+leTYB/wMcV7UTWAH8PVmiXr+F6a1ev1VlBwAfJfthfhZYCuzexm3j88DuZNvWF9ux/kaR7QDG\npnVwTJpeP2AHYC6wVRq2Cfhoa/G3MK/rgB+QJf3+wKdT+YZpusem9TOKbMc0vPm2uY5pt3f97Qe8\nCoxI3/c1rJ0A5gN7pdebVMZrZd5fIdvWe5EdhLwLbFn13S8HTkjr8ySynXflu/ozcG5av58h22Gv\nKwHcQXZAtUn63j/b0m+2RkytfQ9fAKYBg9I626lqnJ8Dt5H9ngeS/U5+VGtbrNdfIzUBteQVshXb\n3HJgS2C7iFgeEQ9GWuPJ+RExNyLea2W60yLiNxGxnGwj6w/sWYd4/xa4IyLuTtP+KbA+8Mlmsb0S\nEYvINpbW2sqPAs6NiBci4h3gdOCIjjaTRMQbEXFzRCyNiCXA2WQ/fiRtC4wB/k9EfBARD5Ft1BXL\nyRLI9hGxMiKmRcTbteYpqTdwBHB6RCyJiDnAz4CjqwZ7JSJ+EREr1vF9NV+WOyLiL5G5H/gj2Y+r\nEuu6to290rJ9PSJub8v8konAf0XEo2kdXEVWw9yTrNbTDxguqW9EzImIv7Rj2svJms62iohlaf1D\nVuOYExFXpPUzHbiZbCfWYTXW3+HAFRExKyKWkiWQ5rEOl7RRRLwZEY+vYz43pW19VUTcADxPVrOv\neCkiLo2IlcBVZN/bFlXb479ExPsR8QDZb6VFkrYExpMdTL2Zvvf7OxBTa9/DcrKd+45kO/BnImK+\nJJFtF9+OiEXpd/VDsm2+Mt66tsVOa/QEsDVZ1am5fyc7Ov6jpBckndbs87k1prv688g6meeRHRV0\n1lZkzRzV055LthwVr1a9Xkp2dF9zWul1H7LO8naTtIGk/5L0kqS3yZo5BqWd9FbAovSDr6heh78C\n7gKul/SKpJ9I6tuG2W5OduTTfDmq10et76qlZRkv6RFJiyS9RdYssXn6uNa2cSLwp4i4r52z3Q74\njqS3Kn9ktcytIuso/hbZzvI1SddLanF7krRXOnPrHUmVvpJ/IjuyfCydsfaNqnmObTbPo4APtzLt\no6qmfWdrC1Jj/W3F2t9J8+/nsDT8S5Lul/SJNM07q+Z9VCr7ejojrBL7zlXzgarfQtW2NyDF8GZE\nvFs17OptKJ0RWJnX98m+h0UR8WZry1w17rpiavF7iIj/R9YEdSHZ93uJpI2AwWS1pGlV0/tDKofa\n22KnNWwCkDSGbEfxUPPP0tHkdyLiI2Rt7v8g6XPVg9SY/DZV8+kFDCGrbUC2U96gatjqH1ut6b5C\n9qOtTFtpXi/XGK/mtIBtyZpLFnRgWgDfIWuqGBsRG5FVqyHb4OcDm0qqXu7V6ygdvfzfiBhOVpv5\nIvD1NszzddYcVVUvR/X6qLVO1/pcUj+yo+CfkvVDDAImp+Voy7ZxIrCtpJ+3If5qc4GzI2JQ1d8G\nEXFdmu+vI+LTaVkD+HFL8aejwAHpb0QqezUiToiIrcia2/5T0vZpnvc3m+eAiDiplWlfWzXt8R1Z\nf2TbwpCqUbapHj8ipkTEQcCHgFuBG1P5+Kp5XytpO+BS4GRgszSfp6rmsy7zgU0kbVhVtm1VDCdW\nzeuHaT1tKmnQuiZaK6Z1fA9ExPkR8XFgOFlT7z+Sbd/vASOqvp+NI2JAGqfWtthpDZcAJG0k6YvA\n9WRtfjNbGOaLkrZPO9jFZFXwVe2YzcclHZqaU75FVpV/JH32BPBVSb0l7UdqJkkWAJtJ2riV6d4I\nHCDpc+kI+Ttp2n9qR2wV1wHfljRU0gCyquUNEbGijeMvIOs/qBhItrG+JWlT4IzKBxHxEjAVOFPS\neumo7sDK55LGSdol1RbeJtup11zfqWp/I3C2pIHpB/gPZO3KbbUAaEqJGrI+in5kncorJI0H9q2K\ntda2sYSsnfszks6pMd/q9XcpcKKkscpsKOmAtFw7SPqbtHNdxpqTBVqK/69I+oqkyk73TbKd9iqy\nDsSPSTpaUt/0N0bSTq3E2NpytHn9kX1fx0raKR0QrL72Jm0bR0naOLImzrdpfTvYMC3HwjTusWRH\n2zVVbY//N83z01Rtjy0MPx+4k2yHvUlaT59pYdB1xtTa95DW+dj0m36X7DtelWr4lwI/l/ShNI2t\nJX0hve7sfqqmRkoAv5e0hCyb/4Csbf7YVoYdRnamyjtknUX/GRH3tmNevyNrr3+TrD360LRBA5xK\ntrFVqtu3VkaKiGfJdswvpCrfWtX8iHgO+BrwC7KjgwPJTm/9oB2xVVxO1vTyAFkn9TKyztK2OhO4\nKsV5OHAeWX/E62TJ7g/Nhj8K+ATpbByyDrX302cfJjtz6m2yjuv7U2xt8fdkP5oXyGpzv07L1lY3\npf9vSHo8tbOeQrajehP4Kmv3V9TcNiLiLbKO+vGSzmplvmdStf4iYipZh+UFab6zyToyIduhnkO2\nbl8lOzo+vaX4W5nXGOBRSe+kZTk1sr6fJWQ75yPIaoSvktUs+qXxLiNrj39L0q0tTPev5l9r/UXE\nncD5wL1pGSsHRpVt4WhgjrJmxBPJtpu/EhFPk/X3/JksCe1CdsZXW32VrMN9EdnBytU1hj+a7MDk\nWbLO+m91IKYWvwdgI7Id/ZtkTVFvkDXvQHb212zgkbRO/puspg2d30/VVOkxN6srSTcAz0bEGTUH\ntoaVahtPAf3aUfu0btJINQDLUarmflRSr9T0dRBVtR8rD2W3GeknaROyGsfvvfPvmZwArF4+THYO\n/ztkTQAnRXbaoZXPN8maUf5C1m590roHt7y4CcjMrKRcAzAzKyknADOzkurRd0/cfPPNo6mpKe8w\nzMwKZdq0aa9HxOBaw/XoBNDU1MTUqVPzDsPMrFAkvVR7KDcBmZmVlhOAmVlJOQGYmZVUj+4DaMny\n5cuZN28ey5YtyzsU60L9+/dnyJAh9O3blrtGm1lHFC4BzJs3j4EDB9LU1ER2kzxrNBHBG2+8wbx5\n8xg6dGje4Zg1rMI1AS1btozNNtvMO/8GJonNNtvMtTyzLla4BAB4518C/o7Nul4hE4CZmXVe4foA\nmms67Y66Tm/OOQfUdXpFcN555zFx4kQ22GCDdg23//778+tf/5pBg9b5JD2zhlHv/U0tXb0/cg3A\nOO+881i6dGm7h5s8ebJ3/mYF5gTQAXPmzGGnnXbihBNOYMSIEey7776899577L333qtvXfH6669T\nuY/RlVdeycEHH8w+++xDU1MTF1xwAeeeey6jRo1izz33ZNGiRa3O69JLL2XMmDHstttuHHbYYat3\nwBMmTOCUU07hk5/8JB/5yEf4zW9+A8B9993H3nvvzZe//GV23HFHjjrqKCq3/L7nnnsYNWoUu+yy\nC9/4xjd4//33Of/883nllVcYN24c48aNA+Ckk05i9OjRjBgxgjPOyB7o1dJwTU1NvP766wCce+65\n7Lzzzuy8886cd95561xPlekNHz6cXXfdlSOOOKJu342ZtV3NBCBpG0n3Snpa0ixJp6byMyW9LOmJ\n9Ld/1TinS5ot6bnKA45T+X6pbLak07pmkbrH888/z6RJk5g1axaDBg3i5ptvXufwTz31FLfccgtT\npkzhBz/4ARtssAHTp0/nE5/4BFdf3frjSg899FCmTJnCk08+yU477cRll122+rP58+fz0EMPcfvt\nt3PaaWtW5/Tp0znvvPN4+umneeGFF3j44YdZtmwZEyZM4IYbbmDmzJmsWLGCiy66iFNOOYWtttqK\ne++9l3vvzR43evbZZzN16lRmzJjB/fffz4wZM1ocrmLatGlcccUVPProozzyyCNceumlTJ8+fZ3r\n6ZxzzmH69OnMmDGDiy++uH0r38zqoi01gBXAdyJiOLAnMEnS8PTZzyNiZPqbDJA+OwIYAewH/Kek\n3pJ6AxcC44HhwJFV0ymcoUOHMnLkSAA+/vGPM2fOnHUOP27cOAYOHMjgwYPZeOONOfDAAwHYZZdd\n1jnuU089xV577cUuu+zCtddey6xZs1Z/dvDBB9OrVy+GDx/OggULVpfvscceDBkyhF69ejFy5Ejm\nzJnDc889x9ChQ/nYxz4GwDHHHMMDDzzQ4jxvvPFGdt99d0aNGsWsWbN4+umn17lsDz30EIcccggb\nbrghAwYM4NBDD+XBBx9c53raddddOeqoo7jmmmvo06fwXVFmhVQzAUTE/Ih4PL1eAjwDbL2OUQ4C\nro+I9yPiRbIn3u+R/mZHxAsR8QFwfRq2kPr167f6de/evVmxYgV9+vRh1apVAH91Dnv18L169Vr9\nvlevXqxY0frjUidMmMAFF1zAzJkzOeOMM9aabvU0q5/s1lJsbfXiiy/y05/+lHvuuYcZM2ZwwAEH\ndOp8/NZiueOOO5g0aRKPP/44Y8aMaVeMZlYf7eoDkNQEjAIeTUUnS5oh6fL0AGjIksPcqtHmpbLW\nyhtGU1MT06ZNA1jdJt9ZS5YsYcstt2T58uVce+21HZ7ODjvswJw5c5g9ezYAv/rVr/jsZz8LwMCB\nA1myZAkAb7/9NhtuuCEbb7wxCxYs4M4771w9jerhqu21117ceuutLF26lHfffZff/va37LXXXq3G\nsmrVKubOncu4ceP48Y9/zOLFi3nnnXc6vGxm1jFtrntLGgDcDHwrIt6WdBFwFhDp/8+Ab3Q2IEkT\ngYkA2267bc3he9Jpm9/97nc5/PDDueSSSzjggPrEddZZZzF27FgGDx7M2LFjW9wBt0X//v254oor\n+MpXvsKKFSsYM2YMJ554IgATJ05kv/32W93GP2rUKHbccUe22WYbPvWpT62eRvPhKnbffXcmTJjA\nHnvsAcDxxx/PqFGjWm3aWrlyJV/72tdYvHgxEcEpp5zis4nMctCmh8JL6gvcDtwVEee28HkTcHtE\n7CzpdICI+FH67C7gzDTomRHxhVS+1nAtGT16dDR/IMwzzzzDTjvtVDNmKz5/19bTFOU6AEnTImJ0\nreHachaQgMuAZ6p3/pK2rBrsEOCp9Po24AhJ/SQNBYYBjwFTgGGShkpaj6yj+La2LpCZmdVXW5qA\nPgUcDcyU9EQq+z7ZWTwjyZqA5gDfBIiIWZJuBJ4mO4NoUkSsBJB0MnAX0Bu4PCJmYQBMmjSJhx9+\neK2yU089lWOPPTaniMys0dVMABHxENDSnbkmr2Ocs4GzWyifvK7xyuzCCy/MOwQzK5lCXgncln4L\nKzZ/x2Zdr3AJoH///rzxxhveQTSwygNh+vfvn3coZg2tcJdgDhkyhHnz5rFw4cK8Q7EuVHkkpJl1\nncIlgL59+/oxgWZmdVC4JiAzM6sPJwAzs5JyAjAzKyknADOzknICMDMrKScAM7OScgIwMyspJwAz\ns5Iq3IVgVpx7kptZz+YagJlZSTkBmJmVlBOAmVlJOQGYmZWUE4CZWUk5AZiZlZQTgJlZSTkBmJmV\nlBOAmVlJOQGYmZWUE4CZWUk5AZiZlZQTgJlZSTkBmJmVlBOAmVlJOQGYmZWUE4CZWUk5AZiZlVTN\nBCBpG0n3Snpa0ixJp6byTSXdLen59H+TVC5J50uaLWmGpN2rpnVMGv55Scd03WKZmVktbakBrAC+\nExHDgT2BSZKGA6cB90TEMOCe9B5gPDAs/U0ELoIsYQBnAGOBPYAzKknDzMy6X80EEBHzI+Lx9HoJ\n8AywNXAQcFUa7Crg4PT6IODqyDwCDJK0JfAF4O6IWBQRbwJ3A/vVdWnMzKzN2tUHIKkJGAU8CmwR\nEfPTR68CW6TXWwNzq0abl8paKzczsxy0OQFIGgDcDHwrIt6u/iwiAoh6BCRpoqSpkqYuXLiwHpM0\nM7MWtCkBSOpLtvO/NiJuScULUtMO6f9rqfxlYJuq0YekstbK1xIRl0TE6IgYPXjw4PYsi5mZtUNb\nzgIScBnwTEScW/XRbUDlTJ5jgN9VlX89nQ20J7A4NRXdBewraZPU+btvKjMzsxz0acMwnwKOBmZK\neiKVfR84B7hR0nHAS8Dh6bPJwP7AbGApcCxARCySdBYwJQ33rxGxqC5LYWZm7VYzAUTEQ4Ba+fhz\nLQwfwKRWpnU5cHl7AjQzs67hK4HNzErKCcDMrKScAMzMSsoJwMyspJwAzMxKygnAzKyknADMzErK\nCcDMrKScAMzMSqott4IwM2uzptPu6Nb5zTnngG6dXyNxDcDMrKScAMzMSsoJwMyspJwAzMxKygnA\nzKyknADMzErKCcDMrKScAMzMSsoJwMyspJwAzMxKygnAzKyknADMzErKCcDMrKScAMzMSsoJwMys\npJwAzMxKygnAzKyknADMzErKCcDMrKScAMzMSsoJwMyspGomAEmXS3pN0lNVZWdKelnSE+lv/6rP\nTpc0W9Jzkr5QVb5fKpst6bT6L4qZmbVHW2oAVwL7tVD+84gYmf4mA0gaDhwBjEjj/Kek3pJ6AxcC\n44HhwJFpWDMzy0mfWgNExAOSmto4vYOA6yPifeBFSbOBPdJnsyPiBQBJ16dhn253xGZmVhed6QM4\nWdKM1ES0SSrbGphbNcy8VNZauZmZ5aSjCeAi4KPASGA+8LN6BSRpoqSpkqYuXLiwXpM1M7NmOpQA\nImJBRKyMiFXApaxp5nkZ2KZq0CGprLXylqZ9SUSMjojRgwcP7kh4ZmbWBh1KAJK2rHp7CFA5Q+g2\n4AhJ/SQNBYYBjwFTgGGShkpaj6yj+LaOh21mZp1VsxNY0nXA3sDmkuYBZwB7SxoJBDAH+CZARMyS\ndCNZ5+4KYFJErEzTORm4C+gNXB4Rs+q+NGZm1mZtOQvoyBaKL1vH8GcDZ7dQPhmY3K7ozMysy/hK\nYDOzknICMDMrKScAM7OScgIwMyspJwAzs5JyAjAzKyknADOzknICMDMrqZoXghVR02l3dOv85pxz\nQLfOz8ysHlwDMDMrKScAM7OScgIwMyspJwAzs5JyAjAzKyknADOzknICMDMrKScAM7OScgIwMysp\nJwAzs5JyAjAzKyknADOzknICMDMrKScAM7OScgIwMyspJwAzs5JyAjAzKyknADOzknICMDMrKScA\nM7OScgIwMyspJwAzs5JyAjAzK6maCUDS5ZJek/RUVdmmku6W9Hz6v0kql6TzJc2WNEPS7lXjHJOG\nf17SMV2zOGZm1lZtqQFcCezXrOw04J6IGAbck94DjAeGpb+JwEWQJQzgDGAssAdwRiVpmJlZPmom\ngIh4AFjUrPgg4Kr0+irg4KryqyPzCDBI0pbAF4C7I2JRRLwJ3M1fJxUzM+tGHe0D2CIi5qfXrwJb\npNdbA3OrhpuXylor/yuSJkqaKmnqwoULOxiemZnV0ulO4IgIIOoQS2V6l0TE6IgYPXjw4HpN1szM\nmuloAliQmnZI/19L5S8D21QNNySVtVZuZmY56WgCuA2onMlzDPC7qvKvp7OB9gQWp6aiu4B9JW2S\nOn/3TWVmZpaTPrUGkHQdsDewuaR5ZGfznAPcKOk44CXg8DT4ZGB/YDawFDgWICIWSToLmJKG+9eI\naN6xbGZm3ahmAoiII1v56HMtDBvApFamczlwebuiMzOzLuMrgc3MSsoJwMyspJwAzMxKygnAzKyk\nanYCm3W3ptPu6Nb5zTnngG6dn1lP4RqAmVlJOQGYmZWUE4CZWUk5AZiZlZQTgJlZSTkBmJmVlBOA\nmVlJOQGYmZWUE4CZWUk5AZiZlZQTgJlZSTkBmJmVlBOAmVlJOQGYmZWUE4CZWUk5AZiZlZQTgJlZ\nSTkBmJmVlBOAmVlJOQGYmZWUE4CZWUk5AZiZlZQTgJlZSTkBmJmVlBOAmVlJOQGYmZVUpxKApDmS\nZkp6QtLUVLappLslPZ/+b5LKJel8SbMlzZC0ez0WwMzMOqYeNYBxETEyIkan96cB90TEMOCe9B5g\nPDAs/U0ELqrDvM3MrIO6ognoIOCq9Poq4OCq8qsj8wgwSNKWXTB/MzNrg84mgAD+KGmapImpbIuI\nmJ9evwpskV5vDcytGndeKluLpImSpkqaunDhwk6GZ2ZmrenTyfE/HREvS/oQcLekZ6s/jIiQFO2Z\nYERcAlwCMHr06HaNa2ZmbdepGkBEvJz+vwb8FtgDWFBp2kn/X0uDvwxsUzX6kFRmZmY56HACkLSh\npIGV18C+wFPAbcAxabBjgN+l17cBX09nA+0JLK5qKjIzs27WmSagLYDfSqpM59cR8QdJU4AbJR0H\nvAQcnoafDOwPzAaWAsd2Yt5mZtZJHU4AEfECsFsL5W8An2uhPIBJHZ2fmZnVl68ENjMrqc6eBWRm\n7dR02h3dOr855xzQrfOz4nANwMyspJwAzMxKygnAzKyknADMzErKCcDMrKScAMzMSsoJwMyspJwA\nzMxKygnAzKyknADMzErKCcDMrKScAMzMSsoJwMyspJwAzMxKygnAzKyknADMzErKCcDMrKScAMzM\nSsoJwMyspJwAzMxKygnAzKyknADMzErKCcDMrKScAMzMSsoJwMyspJwAzMxKygnAzKyknADMzErK\nCcDMrKS6PQFI2k/Sc5JmSzqtu+dvZmaZbk0AknoDFwLjgeHAkZKGd2cMZmaW6e4awB7A7Ih4ISI+\nAK4HDurmGMzMDFBEdN/MpC8D+0XE8en90cDYiDi5apiJwMT0dgfguW4LEDYHXu/G+XU3L1+xefmK\nq7uXbbuIGFxroD7dEUl7RMQlwCV5zFvS1IgYnce8u4OXr9i8fMXVU5etu5uAXga2qXo/JJWZmVk3\n6+4EMAUYJmmopPWAI4DbujkGMzOjm5uAImKFpJOBu4DewOURMas7Y6ghl6anbuTlKzYvX3H1yGXr\n1k5gMzPrOXwlsJlZSTkBmJmVVI87DdTM1kgnS+wIBPBcuoDSrC5K3wcg6SsRcVOtsqKTtEFELM07\njnqSNBg4AWii6mAmIr6RV0z1JOkA4GLgL4CAocA3I+LOXAOzhuEEID0eEbvXKisqSZ8EfgkMiIht\nJe1GthP5u5xD6zRJfwIeBKYBKyvlEXFzbkHVkaRngS9GxOz0/qPAHRGxY76R1Zek7YBhEfHfktYH\n+kTEkrzjKoPSNgFJGg/sD2wt6fyqjzYCVuQTVZf4OfAF0vUWEfGkpM/kG1LdbBAR38s7iC60pLLz\nT14AGmrHKOkEslu/bAp8lOzi0IuBz+UZVz1IOhT4MfAhshqcgIiIjXINrEppEwDwCtmR45fS/4ol\nwLdziaiLRMRcSdVFK1sbtmBul7R/REzOO5AuMlXSZOBGsj6ArwBT0o6FiLglz+DqZBLZTSIfBYiI\n5yV9KN+Q6uYnwIER8UzegbSmtAkgIp4EnpR0TUQ00hF/c3NTM1BI6gucCvTYDbKdTgW+L+kDYHkq\n61FHWJ3UH1gAfDa9XwisDxxIlhAaIQG8HxEfVA5QJPUhW7ZGsKAn7/yhxAlA0kzShtbs6BiAiNi1\nu2PqIicC/wFsTXbfpT+SHXUVXkQMzDuGrhQRx+YdQze4X9L3gfUl7QP8HfD7nGOql6mSbgBuBd6v\nFPakmltpO4FTx1OrIuKl7orFOk7Sl4BKn8Z9EXF7nvHUk6QraOFouFHOcgKQ1As4DtiXrI38LuCX\n0QA7pvT9NRc96fsrbQIoC0lXAadGxFvp/SbAz3rSRthRks4BxgDXpqIjgakRcXp+UdWPpMOq3vYH\nDgFeiYhTcgrJGkzpE4CkJaw5yloP6Au82yjtyJKmR8SoWmVFJGkGMDIiVqX3vYHpDdR8t5Z0tPxQ\nRHwy71hhDzJIAAAODUlEQVTqRdKLtFzL+UgO4dSVpCHAL4BPpaIHyQ7G5uUX1dpK2wdQUd2OrKwz\n4CBgz/wiqrtekjaJiDcBJG1KY33vg4BF6fXGeQbSDYaRnVLYSKofktKf7EynTXOKpd6uAH5NtkwA\nX0tl++QWUTOlrwG0pFGOkAEkfR34PnATWRvrl4GzI+JXuQZWB5KOBM4B7iVbts8Ap0XEDbkGVidV\ntVOl/68CpzfKhW6tkTQtIj6edxydJemJiBhZqyxPjXQk2CGVc6qTXmRHJMtyCqfuIuJqSdOAcano\n0Ih4Os+Y6iUirpN0H1k/AMD3IuLVHEOqq0Y/ywlAUvUV95XfX6Psl96Q9DXguvT+SOCNHOP5K6Wv\nATTrqV8BzAEujYjX8omoa6SLa/pX3kfE/+YYTqdI2jEinm2281gtIh7v7pi6SiOf5QQg6d6qt5Xf\n308j4rl8IqqfdKbhL4BPpKKHgVN60m+v9Amg0aUdyM+ArYDXgO2AZyJiRK6BdYKkSyJiYrOdR0VE\nxN90e1BdoJWznKZExPfzi8oaSemfByDpJ5I2ktRX0j2SFqZqW6M4i6xT+38iYijZPVYeyTekzomI\nienl+IgYV/1Hdn+nRrE/sE9EXB4RlwP7AV/MOaa6knRq+v1J0i8lPS5p37zjqoci7FtKnwCAfSPi\nbbIf1hxge+Afc42ovpZHxBtkZwP1ioh7WfvMiyL7UxvLimxQ1etGPMvpG+n3ty+wGXA0Wcd+I+jx\n+5ZG6WzpjMo6OAC4KSIWt3RriAJ7S9IA4AHgWkmvAe/mHFOnSPow2a0t1pc0iuwsGcju5LpBboHV\n34+A6ampa/VZTvmGVHeV725/4OqImKXG+QH2Tf977L7FCSC7o+SzwHvASekhIw1zFhDZdQ3LyO5w\nehTZUeS/5hpR530BmEB26+Bzq8qXkJ3yWnhpJ/gQWfNdQ57llEyT9Eeyh92cLmkgsCrnmOrltp6+\nb3EnMKsvjlocESslbQBs1IA/tIYj6bBGPide0syI2CXvOLpSurp5JPBCRLyVfotDImJGzqF1Slqu\nPYFnWbNv2RAY2JP2LU4ArH5qVhNrP1bw6twCqoNmFxHBmsvte9xDKTojPTZxBGuf4lr0Gg6w+j5O\nF0TElLxj6SqSPgU8ERHvpg7S3YH/aISbMRbhgtLSJwBJvyJ7EtETrHlQSviGWz2fpIvJ2vzHkT32\n8svAYxFxXK6B1UlqPtgeeIms36aSvBvmXkfpfk67AbsCV5J9j4dHxGfXNV4RSPop8Gfglp56d1Mn\nAOkZYHhP/YI6SlJ/smcBbA/MAC5vtAffSJoREbtW/R8A3BkRe+UdWz20dsvyRjg6rlB6/rak/wO8\nHBGXqUGeyZ1q4RuSHVi+Rw+sfbsTGJ4CPgzMzzuQOruK7ClZD5KdYTGC7AlajeS99H+ppK3ILrPf\nMsd46qqyo5e0NdA7Fb+SX0RdYomk08lulPaZ1Hbet8Y4hVCEW3k4AcDmwNOSHmPtp/Z8Kb+Q6mJ4\npQNR0mXAYznH0xVulzQI+HfgcbJ+jl/mG1LnpR1i36q+jD8Db5HdrvwqstNDG8XfAl8FjouIVyVt\nS/Z9Fl46k+soYGhEnCVpG2DLiOgxv0U3AUkttjVGxP3dHUs9Na9GN0q1ujWS+gH9I2Jx3rF0lqTH\ngb0i4t30fnpEjErPO7g/Ij6db4TWFpIuIjul9W8iYqf0MKY/RsSYGqN2m9LXAIq+o1+H3SS9nV6L\n7KKpt+mB7ZAdJekvwL9HxMUR8T7wvqTbI6Lwt0uo7PyT/0hlKyWtn1NIdaW1H8S01kc0yPYJjE39\nG9MBIuJNSevlHVS10iaARt8AI6J37aEKbzkwTtJY4JsR8QHZFcJFN0BS34hYDhARV8LqWk6ht8uK\nIrSP18HyVGsLgHQhWI+6yK209wKKiIERsVELfwOLvvMvkaUR8bfAM8CDqf24Edo0fwP8V7ooEYB0\nEdHF6TMrhvOB3wIfknQ22ZXdP8w3pLWVvg/Aiqv6QhtJnwcuADaNiEI/NjEdNZ4NHE92DQDAtsBl\nwD832um8jUzSjmR34BVwT0Q8k3NIa3ECsMKSdGBE/L7q/XbAMQ10JfD6ZNdxAMyOiPfWNbz1DM2u\nwZkJXNZTk7YTgBVOmZ4IZsUj6QbWXIMzHpgTEd/KN6qWOQFY4ZTliWBWTNU38ZPUh+z2JD3yFOzS\nngVkxVV5Ilh6AphZT7O88iIiVvS0ZwBUcw3ACq1B7+S6zqNFN3H1bJJWsuahSwLWB5bSA08xdwKw\nwmrUO7m20rRV4SYuqxsnACusRr2Tq1l3cR+AFVmj3sl1NUk7A8NZ+4E3hW7isp7DCcCKrFHv5AqA\npDOAvckSwGSyUwofApwArC7cBGSF1ah3cq2QNJPsaVnTI2I3SVsA10TEPjmHZg3CNQArrEbZ0a/D\nexGxStIKSRsBrwHb5B2UNY7S3gzOik/SnpKmSHpH0geSVlbdArsRTE0PvLkUmEb20Js/5xuSNRI3\nAVlhSZoKHAHcBIwGvg58LCJOzzWwLiCpCdgoImbkHIo1ENcArNAiYjbQOyJWRsQVwH55x1Qvkg6R\ntDFARMwB/lfSwflGZY3ECcCKbGl6wtITkn4i6ds01jZ9RvUjLiPiLeCMHOOxBtNIPxYrn6OB3sDJ\nZJfebwMclmtE9dXS79MnbljduA/ArIeSdDnwFnBhKppE9sCbCbkFZQ3FCcAKK50n33wDXgxMBf4t\nIt7o/qjqJz0G8l+Az5Mt591ky7U018CsYTgBWGFJ+gnZTeB+nYqOADYAXgU+HREH5hVbPUj6SkTc\nVKvMrKOcAKywJD3e/EEblbLqh3IU1bqWL6+YrLG4Q8mKrLekPSLiMQBJY8g6hQF65DNY20LSeGB/\nYGtJ51d9tBEFXi7reZwArMiOA66QNCC9XwIcl9rOf5RfWJ32Clk/xpfIrgCuWAJ8O5eIrCG5CcgK\nSVIv4MsRcWPVxVKLa4xWKJL6RISP+K3LOAFYYUmaGhGj846j3iTdGBGHt3KWExGxaw5hWQNyArDC\nknQO8DpwA2uewUpELMotqDqQtGVEzJe0XUufR8RL3R2TNSYnACssSS+2UBwR8ZFuD8asgJwAzHoY\nSUtYu+lH6b3IEtxGuQRmDccJwApLUl/gJOAzqeg+4L8iYnluQZkViBOAFZakXwJ9gatS0dHAyog4\nPr+o6kvSbsBe6e0Dfh6A1ZMTgBWWpCcjYrdaZUUl6VTgBOCWVHQIcElE/CK/qKyR+EIwK7KVkj4a\nEX8BkPQRsnsDNYrjgLER8S6ApB+TPRLSCcDqwgnAiuwfgXslvUDWQbodcGy+IdWVWDuhrUxlZnXh\nBGCFFRH3SBoG7JCKnouI9/OMqc6uAB6V9Nv0/mDgshzjsQbjPgCzHkzS7sCn09sHI2J6nvFYY3EC\nMOthJPUHTgS2B2YCl/meQNYVnADMehhJNwDLgQeB8cCciPhWvlFZI3ICsEKTtAkwDOhfKYuIB/KL\nqPOqH2YjqQ/wmB8CY13BncBWWJKOB04FhgBPAHuSnSb5N3nGVQerr2SOiBWST/yxruEagBVWul3y\nGOCRiBgpaUfghxFxaM6hdYqklay5u6mA9YGl+F5AVmeuAViRLYuIZZKQ1C8inpW0Q+3ReraI6F17\nKLPOcwKwIpsnaRBwK3C3pDcB3yvfrI3cBGQNQdJngY2BP0TEB3nHY1YETgBWOJI2ioi3JW3a0udF\nfyKYWXdxArDCkXR7RHwxPRGs8qCUCj8RzKyNnADMzErKncBWOOn+OK2KiMe7KxazInMNwApH0r3p\nZX9gNPAkWTPQrsDUiPhEXrGZFUmvvAMwa6+IGBcR44D5wO4RMToiPg6MAl7ONzqz4nACsCLbISJm\nVt5ExFPATjnGY1Yo7gOwIpuRHgx/TXp/FOCHppu1kfsArLDSffNPAj6Tih4ALoqIZflFZVYcTgBW\naJLWB7aNiOfyjsWsaNwHYIUl6Utkt4H+Q3o/UtJt+UZlVhxOAFZkZwB7AG8BRMQTwNBcIzIrECcA\nK7LlEbG4WZnbNM3ayGcBWZHNkvRVoLekYcApwJ9yjsmsMFwDsCL7e2AE8D5wHfA24Ienm7WRzwIy\nMyspNwFZ4dQ60ycivtRdsZgVmROAFdEngLlkzT6PsvbzAMysjdwEZIUjqTewD3Ak2R1A7wCui4hZ\nuQZmVjDuBLbCiYiVEfGHiDgG2BOYDdwn6eScQzMrFDcBWSFJ6gccQFYLaALOB36bZ0xmReMmICsc\nSVcDOwOTgevTbaDNrJ2cAKxwJK0C3k1vqzdgkT0UfqPuj8qseJwAzMxKyp3AZmYl5QRgZlZSTgBm\nZiXlBGBmVlJOAGZmJeUEYAZIapL0jKRLJc2S9EdJ60s6QdIUSU9KulnSBmn4KyVdJOkRSS9I2lvS\n5WkaV1ZNd19Jf5b0uKSbJA3IbSHNmnECMFtjGHBhRIwge8zkYcAtETEmInYDngGOqxp+E7Ib030b\nuA34OdnzCXZJzyfeHPhn4PMRsTswFfiHblsasxp8KwizNV5MzxUGmEZ2i4mdJf0bMAgYANxVNfzv\nIyIkzQQWRMRMAEmz0rhDgOHAw5IA1gP+3A3LYdYmTgBma7xf9XolsD5wJXBwRDwpaQKwdwvDr2o2\n7iqy39ZK4O6IOLKL4jXrFDcBma3bQGC+pL7AUe0c9xHgU5K2B5C0oaSP1TtAs45yAjBbt38he+jM\nw8Cz7RkxIhYCE4DrJM0ga/7Zsd4BmnWU7wVkZlZSrgGYmZWUE4CZWUk5AZiZlZQTgJlZSTkBmJmV\nlBOAmVlJOQGYmZWUE4CZWUn9f/rbtn6lYRd1AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "data = session.get(test_td+'tag_count/').json()\n", "pd.DataFrame(data).set_index('name').plot.bar(\n", " title=\"Disribution of tags for task {name}\".format(name=\"test-set-tags-and-classes\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similarly we can fetch the agreements from the **tag_agg** endpoint" ] }, { "cell_type": "code", "execution_count": 192, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 192, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAFsCAYAAADxOK9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8VXW9//HXW0ARGRRFr4oJ5YQaEiLqzZzKuZuY6c3r\nTc0pu3qz4VeZ1dWywebSupomOeSQXbVMrSRzKjVFNIdwIMPAUAkFEcUAP78/vt/NWRzPBOy119n7\nvJ+Px3mcvb9r+qzxs9Z3ffdaigjMzMzqbY2qAzAzs9bkBGNmZqVwgjEzs1I4wZiZWSmcYMzMrBRO\nMGZmVgonmDqTNFPSuxo0rWMk/b6H/Y6SFJL6lx3XqpC0taQHJS2U9JGq46mRtLakX0paIOlnVcdT\nJOliSV+qOg5beZL2lDR7FYcNSVvUO6YyrFaCkXSbpBclrVWvgHozSWdK+knVcbSoTwG3RsSQiDin\nfce8rR1fQVzvAzYC1o+Iw1Z3ZKtzYFnN6dZl+ZUZf09OmKpafrZqVjnBSBoFvAMI4D2rMZ5eeUbd\nCH153juwOfBo1UF0YHPgiYhYurIDev1anxcRq/QH/A/wB+DbwA3tuq0P/BJ4CbgP+BLw+0L3AE4G\nngT+msu2AaYALwCPA4cX+l8L+CbwN+A54Hxg7dxtT2A26Qz4eWAOMAk4EHgij+/0wrjWAE4D/gLM\nA64Ghuduo3JsR+dp/QP4bO62P/BPYAnwMvCnTpbLTOAzwJ+BF4EfAwPbxfpp4Fngslz+buBBYD5w\nFzC2ML5arAvzOA8pdDum3XL9BvB7YBjQLy+zfwBP5eUdQP/c7ybA9Xn5zABOyOUDgVeBDfL3zwJL\ngaH5+1nAd/Pni4EfADfm+P4IvKWLbeY9pCQyH7gNGJPLfwcsAxbnZbtVu+G+3K7793P594BZpO3s\nfuAdhWHWBi7J62B63j5mF7p/Gngmx/048M4O4v1Cu3V+HGn7+RzwNGl7uxQY1m77OY60/dzRbnzr\n5GX7eh7fy3k9TATuzstlDvB9YM08jIDv5Gm9BDwMbF9Y/l/Kn4cAtwLnAOrh8utqnzuQtL0tzMvp\n/3UWfwfLbSDwE9L+NZ90DNgodxsGXJTn8xnSsaEfMCbHtyyPd34H413p5ZeH2zfP3wLgf4HbgeNz\nty3y9wWkfeWnXWy/BwEP5PUwCziz0K227t9w7ChsjxeTtsc/A5+ksD12MK1+wOm07fv3A5sVjp9b\n9CCmrtbDMaTjwkLgr8CRheGOJe0zLwK/ATbvblvsdD5WI8HMAP4L2JG0A25U6HZV/hsEbJtnvH2C\nmQIMzwt+ndzPB4H+wNvyCto29/8d0sFwOGlH+iXw1cJBeykp4Q0ATgDmAlfkfrcjbZSjc/+nAvcA\nI0mJ64fAle02kgtzXDsAr9F2IDwT+Ek3y2Um8AiwWY73D7QdBGqxfi1Pe+08r88DO+eN6ug8jrXy\nMIeRdqI1gH8HFgEbFxNM7nZh3hgG5W4nAY8V4riVFRPMHaSdbSAwLi+zvQvdDs2fbyZt5AcUuh1S\nOMDNI+3g/YHLgas6WS5b5dj3yevpU6RtqHYgvY2803cy/Bu6A/9JOpnpD3yClLRryfxs0oFjvbyu\nHyLv0MDWpO1tk8J67zAxtl/npJ1vBvBmYDBwLW0nCqPyMr6UtE2v3cH49qTdgYW0D+2S52MUaef+\naO62H+ngsi5pBx9TWP8Xkw7Q6wP3krezniw/ut/n5pATdl6G4zuLv4NpfYi0jw4ibdM70naCch1p\nn1sH2DDH/aHi9tzNuFd2+W1AOhi+N3c/lXS8qiWYK0knUWuQ9oXdupn2W3O/Y0knu5N6eOw4G7iT\ntC9uRjpGdJVgPkk6gG+d1/sOpGpaWDHBdBVTh+shL/uXgK1zfxsD2+XPB5O27zF5eX0OuKu7bbHT\n+ehJMulg5nfLK6l2lvsY8LH8uV/utnWh/46uYPYufP934M520/ghcEaekUUUDgDArrRd+exJSiD9\n8vchefw7F/q/v7DQp1M4W80LdwltG2cAIwvd7wXe39HBppNlMxM4qfD9QOAvhVj/ST4I5rLzgLPa\njeNxYI9Oxv8gcHBhh/wj8FPgGlY8a/tduzj2zfPWn7SBLwOGFLp/Fbg4fz6LdCbcn3TQPpW0g9Su\nbmob+sXAj9rN62OdxP154OrC9zVIZ7B7RgcHwA6G77J77udFYIf8+Slgv0K342lLMFuQkvq7gAHd\njHOFdQ7cAvxX4fvWHWw/b+5ifHvS/QH6o8B1+fPepCvxXYA12vV3MTCZdLD6ZDfjXGH50cU+lz//\njXSAGroK8R9LuyvxXL4R6aC7dqHsCNK9t9r2vNIJppvldxRwd6GbSIm1lmAuBS6gsM/39A/4LvCd\n/Lm27js7djwF7F/odmJX80E6BhzcSbflCaabmDpbD+uQrmgOpd1JEPAr4LjC9zWAV0hVxZ1ui539\nreo9mKOBmyPiH/n7FbkMYARpZ5tV6L/4uaOyzYGdJc2v/QFHAv+SxzcIuL/Q7de5vGZeRCzLn1/N\n/58rdH+VdLZZm9Z1hXFNJx1sNyr0/2zh8yuFYXuqOG9Pk65AauZGxOLC982BT7Sb981qw0g6Kreu\nqnXbnnRWVrMF6azjCxHxz0L5Jh3EUez2QkQsbNd90/z5dtKOPJ50FjUF2IO0Yc2IiHmF4Xq6rDYp\nxhARr+f4Nu2k/25J+n+SpucWXvNJ1S+1ZdN+/pd/jogZpIPQmcDzkq6SVFxHXVlhPvLn/qy4/XS0\nvXc1H1tJukHSs5JeAr5Sm4+I+B2pyucHOdYLJA0tDH4Q6Yz5/JWZJl3vc5AOPgcCT0u6XdKuXcT/\ncuHvTcBlpKvpqyT9XdLXJQ3I0xwAzClM84ekK5mOxvum4ri7mH6ny49220Gko2axkcCnSEnnXkmP\nSjo2j/P0wrTPz2U7S7pV0lxJC0i1BMV9ETrfHzrdHyUdWZjWr3LxZqSagy51E1OH6yEiFpFOME4i\nrYsbJW2Th9kc+F5h/byQl8+mPdgW32ClE4yktYHDgT3yCn0W+Biwg6QdSFUtS0nVEjWbdTCqKHye\nBdweEesW/gZHxIdJl+2vki7hat2GRcTKHvSL0zqg3bQGRsQzPRg2uu8FWHF+3wT8vYtxzAK+3C6e\nQRFxpaTNSZfcp5CuGtYlna2qMPx0UjXHryRtXSif00EcNX8Hhksa0q57bRncRTozP4S0Xv6cux9I\nSj6r4u+kjRcAScrx9WS5Q7vlJukdpIPD4cB6edksoG3ZzKGLbTAiroiI3XJMQaq2XOn5IC2Xpax4\nQtPVdtJRt/NItQBbRsRQUt378nUcEedExI6k6uatSNUnNReSTrhukrTOSky3q32OiLgvIg4mHfx/\nTrpX2WH8ebja398iYklEfCEitgX+lXSP8ag8zddINR+1aQ6NiO06Gnce1/JxdzZ9ul5+K2wHebtb\n/j0ino2IEyJiE9IV2/9K2iIivlKY9km59ytIVfWbRcQwUlIv7otd6XR/jIjLC9M6IBfPAt7Sg/F2\nGlMX64GI+E1E7EOqwXmMtB3VpvuhdtvF2hFxVx6uq23xDVblCmYS6Yx/W1Ld/ThSXdydwFH5SuJa\n4ExJg3JmPKqbcd4AbCXpA5IG5L+dJI3JZ7oXAt+RtCGApE0l7bcKsUNaAV/OB28kjZB0cA+HfQ4Y\nJam75XaypJGShpPqd3/aRb8XAiflMxFJWkfSQfngvw5ph5qbY/0g6QpmBRFxJWmn+q2k2kZ5NfCR\nHMd6pMYCtf5nkZLIVyUNlDSWdGP6J7n7K6RqxZNpSyh3kc54VjXBXA0cJOmd+Wz2E6SDzV09HP45\n0n2PmiGkA/tcoL+k/yHVLxen9xlJ60nalJSkgeW/udlbqXn9YtpuHPfElcDHJI2WNJh0tvzT6Hkr\ns+eA9SUNazcvLwEv5/3lw4VYd8rbxgBSVfHiDmI9hVSl8st8AtjZdIvLr9N9TtKa+ax6WEQsybG9\nXhhP+/hXIGkvSW+V1C8PuwR4PSLmkO7pfUvSUElrSHqLpD0K4x4pac3Oxt3J9DtdfqQGKG+VNEmp\nVd/JtF2lIekwSbWE8yJpf+tsWxhCuvJfLGki8B9dxNlecXscCfx3N/3/CDhL0pb5uDBW0vorE1Nn\n60HSRpIOzickr5EaS9Tm+fwc53Z5HMMkHZY/92RbXFFP6tHa1dH9GvhWB+WHky4P+5Oqr26krRXZ\n14BbCv2+oQ6RdMZ8I+mAMY90D2Fc7jaQtCM/lcc5HfhIdFAnm6cfwKhC2e+B/4y2OsWPk3bIhaTL\n0K/EivWo/QvD3kZbfe36eVwvAtM6WT4zaWtFNp/UkmlQR7EWhtk/L6daK5ifke+PkFoAvUC6kvs2\nK7aAOYYV722dQLr0HpWXw3fysvwrb2xFNpJ0kHkhL4OT2sX0VdKBt9bY4JQ8fLExx8UUbix3Nn+F\n7ofk5bIgz8d2HS3nTobdlVT/+yLp/lA/0v2Hl/Iy+1Re9u+Ktnrmy/IynU66WVm7FzaWVD++MM//\nDXTQGir3eyYr3oNZg9SgZBZpW/0J6Qqqw+2nk3FOpq1lzybA7qSzyJdJJ2pfrK1X4J2kBgov523g\ncmBw++Wf47qUdAAf2N3y62qfA9Yk7ecv0rYP79ZZ/B1M6wjS/rWIlBDOoW27G0a64pidt4MHaLtP\nsWaO5wXgH/VYfoX96wnaWpHdDXwgd/s66Sr6ZdJ+cGIX030faf9amLeZ79e2jY7WPSseOwbl9TOf\nnrci+xxp312Y18HI9sfPbmLqcD2QrlpqLedqLTq3LUz7A6Sq8VrLtMndbYud/SkPWCpJXwP+JSKO\n7rZnsxJI+jDpQLZHtz1by8q1D7NJzXJvrTqeVlfKo2IkbZMv6ZQv244jNU80awhJG0t6e66G2ZpU\nJedtsA+StJ+kdXOVaO3+zD0Vh9UnlPVL4yGkuupNSJdm3wJ+UdK0zDqyJqmF0mhSNcBVpOoR63t2\nJd0MX5NUPTUpIl7tehCrh4ZUkZmZWd/jpymbmVkpnGDMzKwUvfpprxtssEGMGjWq6jDMzJrK/fff\n/4+IGNF9n+Xq1Qlm1KhRTJ06teowzMyaiqSnu++rfK4iMzOzUjjBmJlZKbpNMJI2U3pa55+VnjZ6\nai4/U9IzSk/6fVDSgYVhPiNphqTHi88Mk7R/Lpsh6bSOpmdmZq2hJ/dglgKfiIhp+QGM90uakrt9\nJyK+WexZ0rbA+0kv+tqE9ADGrXLnH5BeODUbuE/S9ZGe1Gtm1qssWbKE2bNns3jx4u57rsjAgQMZ\nOXIkAwYMqDqUDnWbYCI9AXVO/rxQ0nS6fofHwaS3Gr4G/FXSDNIbDyG9S+QpAElX5X6dYMys15k9\nezZDhgxh1KhRSD19Kn/jRATz5s1j9uzZjB49uupwOrRS92AkjSK9WvWPuegUSQ9JmpwfCQ8p+RRf\nrDM7l3VWbmbW6yxevJj111+/VyYXAEmsv/76vfoKq8cJJr/74hrSu65fIj1y+y2kx3vPIT1vbLVJ\nOlHSVElT586dW49Rmpmtkt6aXGp6e3w9SjD5BTPXAJdHxLUAEfFcRCyLtheC1arBnmHFN7eNzGWd\nla8gIi6IiAkRMWHEiMp/J2RmZquo23swSinyImB6RHy7UL5xvj8D6UVSj+TP1wNXSPo26Sb/lqSX\nOwnYUtJoUmJ5Pyv3Rjiz1nBmpy+DLGl6Cxo7PevQ0qVL6d+/V/+2ve56cgXzdtIbzvZu1yT565Ie\nlvQQsBfwMYCIeJT0etA/k96Kd3K+0llKeivib0hvGLw692tm1jQmTZrEjjvuyHbbbccFF1wAwEUX\nXcRWW23FxIkTOeGEEzjllPSG7mOOOYaTTjqJnXfemU996lMsWrSIY489lokTJ/K2t72NX/wivcVk\n2bJlfPKTn2SnnXZi7Nix/PCHPwRgzpw57L777owbN47tt9+eO++8s5qZXkU9aUX2e9LVR3s3dTHM\nl0mv+m1fflNXw5mZ9XaTJ09m+PDhvPrqq+y0004cdNBBnHXWWUybNo0hQ4aw9957s8MOOyzvf/bs\n2dx1113069eP008/nb333pvJkyczf/58Jk6cyLve9S4uv/xyhg0bxn333cdrr73G29/+dvbdd1+u\nvfZa9ttvPz772c+ybNkyXnnllQrnfOX1res1M7PVdM4553DddenlqLNmzeKyyy5jjz32YPjw4QAc\ndthhPPHEE8v7P+yww+jXrx8AN998M9dffz3f/Gb6+eDixYv529/+xs0338xDDz3E//3f/wGwYMEC\nnnzySXbaaSeOPfZYlixZwqRJkxg3blwjZ3W1OcGYmfXQbbfdxm9/+1vuvvtuBg0axJ577sk222zD\n9OnTOx1mnXXWWf45IrjmmmvYeuutV+gnIjj33HPZb7/92g/OHXfcwY033sgxxxzDxz/+cY466qj6\nzVDJ/CwyM7MeWrBgAeuttx6DBg3iscce45577mHRokXcfvvtvPjiiyxdupRrrrmm0+H3228/zj33\nXGpvEn7ggQeWl5933nksWbIEgCeeeIJFixbx9NNPs9FGG3HCCSdw/PHHM23atPJnso58BWNm1kP7\n778/559/PmPGjGHrrbdml112YdNNN+X0009n4sSJDB8+nG222YZhwzpuKfj5z3+ej370o4wdO5bX\nX3+d0aNHc8MNN3D88cczc+ZMxo8fT0QwYsQIfv7zn3PbbbfxjW98gwEDBjB48GAuvfTSBs/x6lEt\nk/ZGEyZMCL8PxlqOmyk3henTpzNmzJge9fvyyy8zePBgli5dyiGHHMKxxx7LIYccUnKESUdxSro/\nIiY0JIAuuIrMzGw1nXnmmcubEo8ePZpJkyZVHVKv4CoyM7PVVGsVZitygjFrsFGLr2jo9GY2dGpm\nbVxFZmZmpXCCMTOzUjjBmJlZKZxgzMx6qWOPPZYNN9yQ7bffvupQVolv8puZ9cCo026s6/hmnn1Q\nt/0cc8wxnHLKKU31eJgiX8GYmfVSu++++/KHaDYjJxgzMyuFE4yZmZXCCcbMzErhBGNmZqVwgjEz\n66WOOOIIdt11Vx5//HFGjhzJRRddVHVIK8XNlM3MeqAnzYrr7corr2z4NOvJCcbM6svvu7HMVWRm\nZlYKJxgzMyuFE4yZmZXCCcbMzErhBGNmZqVwKzIzqyu/Erp+Zs2axVFHHcVzzz2HJE488UROPfXU\nqsPqMScYM7OeqHfz6x40r+7fvz/f+ta3GD9+PAsXLmTHHXdkn332Ydttt61vLCVxFZmZWS+18cYb\nM378eACGDBnCmDFjeOaZZyqOquecYMzMmsDMmTN54IEH2HnnnasOpcecYMzMermXX36ZQw89lO9+\n97sMHTq06nB6zAnGzKwXW7JkCYceeihHHnkk733ve6sOZ6U4wZiZ9VIRwXHHHceYMWP4+Mc/XnU4\nK63bBCNpM0m3SvqzpEclnZrLh0uaIunJ/H+9XC5J50iaIekhSeML4zo69/+kpKPLmy0zs+b3hz/8\ngcsuu4zf/e53jBs3jnHjxnHTTTdVHVaP9aSZ8lLgExExTdIQ4H5JU4BjgFsi4mxJpwGnAZ8GDgC2\nzH87A+cBO0saDpwBTAAij+f6iHix3jNlZlZ3FTy1ebfddiMiGj7deun2CiYi5kTEtPx5ITAd2BQ4\nGLgk93YJMCl/Phi4NJJ7gHUlbQzsB0yJiBdyUpkC7F/XuTEzs15jpe7BSBoFvA34I7BRRMzJnZ4F\nNsqfNwVmFQabncs6KzczsxbU4wQjaTBwDfDRiHip2C3SNVxdruMknShpqqSpc+fOrccozcysAj1K\nMJIGkJLL5RFxbS5+Lld9kf8/n8ufATYrDD4yl3VWvoKIuCAiJkTEhBEjRqzMvJiZWS/Sk1ZkAi4C\npkfEtwudrgdqLcGOBn5RKD8qtybbBViQq9J+A+wrab3c4mzfXGZmZi2oJ63I3g58AHhY0oO57HTg\nbOBqSccBTwOH5243AQcCM4BXgA8CRMQLks4C7sv9fTEiXqjLXJiZWa/TbYKJiN8D6qTzOzvoP4CT\nOxnXZGDyygRoZtZXLV68mN13353XXnuNpUuX8r73vY8vfOELVYfVY35cv5lZD7z1krfWdXwPH/1w\nt/2stdZa/O53v2Pw4MEsWbKE3XbbjQMOOIBddtmlrrGUxY+KMTPrpSQxePBgID2TbMmSJaTb4s3B\nCcbMrBdbtmwZ48aNY8MNN2Sfffbx4/rNzKw++vXrx4MPPsjs2bO59957eeSRR6oOqcecYMzMmsC6\n667LXnvtxa9//euqQ+kxJxgzs15q7ty5zJ8/H4BXX32VKVOmsM0221QcVc+5FZmZWS81Z84cjj76\naJYtW8brr7/O4Ycfzrvf/e6qw+oxJxgzsx7oSbPiehs7diwPPPBAw6dbL64iMzOzUjjBmJlZKZxg\nzMysFE4wZmad6O2vK+7t8TnBmJl1YODAgcybN6/XHsQjgnnz5jFw4MCqQ+mUW5E1ozOHNXh6Cxo7\nPbNeYOTIkcyePZve/GbdgQMHMnLkyKrD6JQTjJlZBwYMGMDo0aOrDqOpuYrMzMxK4SuYJjRq8RUN\nnd7Mhk7NzFqFr2DMzKwUTjBmZlYKJxgzMyuFE4yZmZXCCcbMzErhBGNmZqVwgjEzs1I4wZiZWSmc\nYMzMrBROMGZmVgonGDMzK4UTjJmZlcIJxszMSuEEY2ZmpXCCMTOzUjjBmJlZKbpNMJImS3pe0iOF\nsjMlPSPpwfx3YKHbZyTNkPS4pP0K5fvnshmSTqv/rJiZWW/SkyuYi4H9Oyj/TkSMy383AUjaFng/\nsF0e5n8l9ZPUD/gBcACwLXBE7tfMzFpUt69Mjog7JI3q4fgOBq6KiNeAv0qaAUzM3WZExFMAkq7K\n/f55pSM2M7OmsDr3YE6R9FCuQlsvl20KzCr0MzuXdVb+BpJOlDRV0tS5c+euRnhmZlalVU0w5wFv\nAcYBc4Bv1SugiLggIiZExIQRI0bUa7RmZtZg3VaRdSQinqt9lnQhcEP++gywWaHXkbmMLsrNzKwF\nrdIVjKSNC18PAWotzK4H3i9pLUmjgS2Be4H7gC0ljZa0JqkhwPWrHraZmfV23V7BSLoS2BPYQNJs\n4AxgT0njgABmAh8CiIhHJV1Nunm/FDg5Ipbl8ZwC/AboB0yOiEfrPjdmZtZr9KQV2REdFF/URf9f\nBr7cQflNwE0rFZ2ZmTUt/5LfzMxK4QRjZmalcIIxM7NSOMGYmVkpnGDMzKwUTjBmZlYKJxgzMyuF\nE4yZmZXCCcbMzErhBGNmZqVwgjEzs1Ks0uP6e70zhzV4egsaOz0zsybgKxgzMyuFE4yZmZWiJavI\nRi2+oqHTm9nQqZmZNQdfwZiZWSmcYMzMrBROMGZmVgonGDMzK4UTjJmZlcIJxszMSuEEY2ZmpXCC\nMTOzUjjBmJlZKZxgzMysFE4wZmZWCicYMzMrhROMmZmVwgnGzMxK4QRjZmalcIIxM7NSOMGYmVkp\nnGDMzKwU3SYYSZMlPS/pkULZcElTJD2Z/6+XyyXpHEkzJD0kaXxhmKNz/09KOrqc2TEzs96iJ1cw\nFwP7tys7DbglIrYEbsnfAQ4Atsx/JwLnQUpIwBnAzsBE4IxaUjIzs9bUbYKJiDuAF9oVHwxckj9f\nAkwqlF8ayT3AupI2BvYDpkTECxHxIjCFNyYtMzNrIat6D2ajiJiTPz8LbJQ/bwrMKvQ3O5d1Vv4G\nkk6UNFXS1Llz565ieGZmVrXVvskfEQFEHWKpje+CiJgQERNGjBhRr9GamVmDrWqCeS5XfZH/P5/L\nnwE2K/Q3Mpd1Vm5mZi1qVRPM9UCtJdjRwC8K5Ufl1mS7AAtyVdpvgH0lrZdv7u+by8zMrEX1764H\nSVcCewIbSJpNag12NnC1pOOAp4HDc+83AQcCM4BXgA8CRMQLks4C7sv9fTEi2jccMDOzFtJtgomI\nIzrp9M4O+g3g5E7GMxmYvFLRmZlZ0/Iv+c3MrBROMGZmVgonGDMzK4UTjJmZlcIJxszMStFtKzKz\nhjtzWIOnt6Cx0zPrI5xgrNcZtfiKhk5vZkOnZtZ3uIrMzMxK4QRjZmalcIIxM7NSOMGYmVkpnGDM\nzKwUTjBmZlYKJxgzMyuFE4yZmZXCCcbMzErhBGNmZqVwgjEzs1I4wZiZWSmcYMzMrBROMGZmVgon\nGDMzK4UTjJmZlcIJxszMSuEEY2ZmpXCCMTOzUjjBmJlZKZxgzMysFE4wZmZWCicYMzMrhROMmZmV\nwgnGzMxK4QRjZmalWK0EI2mmpIclPShpai4bLmmKpCfz//VyuSSdI2mGpIckja/HDJiZWe9UjyuY\nvSJiXERMyN9PA26JiC2BW/J3gAOALfPficB5dZi2mZn1UmVUkR0MXJI/XwJMKpRfGsk9wLqSNi5h\n+mZm1gusboIJ4GZJ90s6MZdtFBFz8udngY3y502BWYVhZ+eyFUg6UdJUSVPnzp27muGZmVlV+q/m\n8LtFxDOSNgSmSHqs2DEiQlKszAgj4gLgAoAJEyas1LBmZtZ7rNYVTEQ8k/8/D1wHTASeq1V95f/P\n596fATYrDD4yl5mZWQta5QQjaR1JQ2qfgX2BR4DrgaNzb0cDv8ifrweOyq3JdgEWFKrSzMysxaxO\nFdlGwHWSauO5IiJ+Lek+4GpJxwFPA4fn/m8CDgRmAK8AH1yNaZuZWS+3ygkmIp4CduigfB7wzg7K\nAzh5VadnZmbNxb/kNzOzUqxuKzIzsz5j1Gk3NnR6M88+qKHTqzdfwZiZWSmcYMzMrBROMGZmVgon\nGDMzK4UTjJmZlcIJxszMSuEEY2ZmpXCCMTOzUjjBmJlZKZxgzMysFE4wZmZWCicYMzMrhROMmZmV\nwgnGzMxK4QRjZmalcIIxM7NSOMGYmVkpnGDMzKwUTjBmZlYKJxgzMyuFE4yZmZXCCcbMzErhBGNm\nZqVwgjEzs1I4wZiZWSmcYMzMrBROMGZmVgonGDMzK4UTjJmZlcIJxszMSuEEY2ZmpWh4gpG0v6TH\nJc2QdFqjp29mZo3R0AQjqR/wA+AAYFvgCEnbNjIGMzNrjEZfwUwEZkTEUxHxT+Aq4OAGx2BmZg2g\niGjcxKSLy8hPAAARc0lEQVT3AftHxPH5+weAnSPilEI/JwIn5q9bA483LEDYAPhHA6fXaJ6/5ub5\na16NnrfNI2JEA6fXof5VB9BeRFwAXFDFtCVNjYgJVUy7ETx/zc3z17xaed660ugqsmeAzQrfR+Yy\nMzNrMY1OMPcBW0oaLWlN4P3A9Q2OwczMGqChVWQRsVTSKcBvgH7A5Ih4tJExdKOSqrkG8vw1N89f\n82rleetUQ2/ym5lZ3+Ff8puZWSmcYMzMrBS9rpmymVkryA2ZtgECeDz/uLxP6fP3YCQdFhE/667M\neh9JI4ATgFEUTpYi4tiqYrJVI2lQRLxSdRz1Iukg4HzgL4CA0cCHIuJXlQbWYE4w0rSIGN9dWTOT\ntDmwZUT8VtLaQP+IWFh1XKtL0l3AncD9wLJaeURcU1lQtlIk/SvwI2BwRLxJ0g6kA/F/VRzaapH0\nGPDuiJiRv78FuDEitqk2ssbqs1Vkkg4ADgQ2lXROodNQYGk1UdWfpBNIj94ZDryF9OPW84F3VhlX\nnQyKiE9XHUSZJL0X+BqwIelMWEBExNBKA6uf7wD7kX8PFxF/krR7tSHVxcJacsmeApr+pG5l9dkE\nA/yddOb7nvy/ZiHwsUoiKsfJpIeM/hEgIp6UtGG1IdXNDZIOjIibqg6kRF8H/i0iplcdSFkiYpak\nYtGyzvptIlMl3QRcTboHcxhwXz5hICKurTK4RumzCSYi/gT8SdJPIqJlrlg68FpE/LO2A0vqT9rg\nW8GpwOmS/gksyWWtdHYP8FwrJxdgVq4mC0kDSOu0FeZ3IPAcsEf+PhdYG/g30v7nBNPKJD1MPtC2\nO3sCICLGNjqmktwu6XRgbUn7AP8F/LLimOoiIoZUHUMDTJX0U+DnwGu1whY6Az4J+B6wKem5hDeT\nrrqbWkR8sOoYeoM+e5M/3/juVEQ83ahYyiRpDeA4YF9S/f1vgB9Fi6x4Se8BanX2t0XEDVXGU2+S\nftxBcbilXO+W19sb9rG+tt76bIKx5ifpbGAn4PJcdAQwNSI+U11UtjIkXQKcGhHz8/f1gG81+4FY\n0qGFrwOBQ4C/R8RHKgqpEn0+wUhaSNuZxprAAGBRq9TjS/orHZ9JvbmCcOpK0kPAuIh4PX/vBzzQ\nQtWbSBoJnAu8PRfdSTogz64uqvqR9EBEvK27smaXaxJ+HxH/WnUsjdRn78HUFOvxlW7GHAzsUl1E\ndVd8ydFAUmuW4RXFUoZ1gRfy52FVBlKSHwNXkNYbwH/msn0qi6i+1pC0XkS8CCBpOK15XNqS1NS8\nT+nzVzAdacUzqCJJ90fEjlXHsbokHQGcDdxKur+0O3BaRPy00sDqSNKDETGuu7JmJeko4HTgZ6R1\n+D7gyxFxWaWBraZCzYjy/2eBz/S1HwG34pnCSqm1S8/WIJ3xL64onLqTVHwiQW3+WmK9R8SVkm4j\n3YcB+HREPFthSGWYJ+k/gSvz9yOAeRXGU1cRcamk+4G9ctF7I+LPVcZUD32khWO3+vwVTLtWOkuB\nmcCFEfF8NRHVl6RbC19r8/fNiHi8mohWn6RtIuKxdslzuYiY1uiYypJbO54L7JqL/gB8JCL+Vl1U\n9Zd//Duw9r0V5q/VWzj2RJ9PMNZ8JF0QESe2S541ERF7NzwoWyX5IPwtYBPgeWBzYHpEbFdpYKup\nkxaO90XE6dVF1Xh9/n0wkr4uaaikAZJukTQ3V0m0BEmn5vmTpB9JmiZp36rjWh0RcWL+eEBE7FX8\nIz1frmW0+vYJnEVqVPNERIwmPSPvnmpDqosDgX0iYnJETAb2B95dcUwN1+cTDLBvRLxEWvkzgS2A\nT1YaUX0dm+dvX2B94AOkG+Ot4K4eljWzVt8+l0TEPFJrsjUi4lZWbPnYzNYtfG7FFo7daombvaup\ntgwOAn4WEQs6enRME6vNzIHApRHxqJp8BiX9C+nRImtLehtt8zgUGFRZYOUYkP+36vY5X9Jg4A7g\ncknPA4sqjqkevgo8kKtxl7dwrDakxnOCSU/kfQx4FfhwfolVy7QiA+6XdDPphUefkTQEeL3imFbX\nfsAxpFcPfLtQvpDU5LWVXN/i2+fBpPn5GHAk6Uz/i5VGtJryCdzvSVV/rdzCsVu+yc/yH3ctiIhl\nkgYBQ1tlY8i/IB4HPBUR8/O8joyIhyoObbVJOrSVf1eQ190uwGO0bZ/rAENaZftsVZIejoi3Vh1H\n1ZxgWP5WvVGs+NrdSysLqI4kvR14MCIW5ZvD44HvtdDDPA8CtmPFJq5NfQZc1Ko/+m33Q0Roe5xR\nS7xQLT9j7fsRcV/VsVSpzycYSZeR3vT4IG0vOopWeShdfl7XDsBY4GLS62kPj4g9uhquGUg6n3TP\nZS/SfL0PuDcijqs0sDqS9E3gbuDaVnkCdl+QqzW3AJ4m3VOqJc6WeU5eTzjBSNOBbVt155U0LSLG\nS/of4JmIuKhWVnVsq0vSQxExtvB/MPCriHhH1bHVSz7TX4d08vMqrXOGP5D0LpgtgIeAya304r/O\nXgfSKjUHPeWb/PAI8C/AnKoDKclCSZ8hPSRx91yvP6CbYZrFq/n/K5I2IT1CZeMK46m7Fn7kyCWk\nt5DeSWrhuB3pbZYtoZZIJG0K9MvFf68uomo4wcAGwJ8l3cuKbwx8T3Uh1dW/A/8BHBcRz0p6E/CN\nimOqlxskrUuan2mkevwfVRtSfeUWSUcCoyPiLEmbARtHxL0Vh7a6tq3dBJd0EdDs8wNAPpkbULgP\neDcwn/QqkEtIzZf7DFeRSR3ei4iI2xsdi606SWsBAyNiQdWx1JOk80jNyveOiDH5hVw3R8RO3Qza\nq7Wvpm2hattpwDsiYlH+/kBEvC2/q+j2iNit2ggbq89fwbRqImn3IrUVOtECdfgAkv4CfCMizo+I\n14DXJN0QEa30SI6d8z20BwAi4kVJa1YdVB3sIOml/FmkH82+RAtsn7Xkkn0vly2TtHZFIVWmzyaY\nVj8At3DdfdESYC9JOwMfioh/kn7h30qW5LPfAMg/tGz2H8oSEf2676spDZY0ICKWAETExbD8Crup\njymros8+iywihkTE0A7+hjR7culDXomIfwemA3fm+0utVud7DnAdsKGkL5N+If6VakOyLvwf8MP8\ng20A8o9jz8/d+pQ+fw/GmlfxR4iS3gV8HxgeES31alpJ25CeMizgloiYXnFI1ol8tfll4HjSb2AA\n3gRcBHyulZpi94QTjDUtSf8WEb8sfN8cOLoVfsnf7nciDwMX9bWDUzPL91u2yF9nRMSrXfXfqpxg\nrOn0hTdaSvopbb8TOQCYGREfrTYqs5XjBGNNpy+80bL4sERJ/UmPwGn6ZrzWt/TZVmTWvGpvtMxv\nsGxVS2ofImJpi70DxvoIX8FYU2vVJ2FLWkbbi7cErA28Qos0o29VnVXb1rRC9e3KcIKxptXqT8K2\n5tNJtW1NS1TfrgwnGGtarf4kbLNm53sw1sxa/UnY1sQkbQ9sy4ovw2v66tuV4QRjzazVn4RtTUrS\nGcCepARzE6mp+e+BPpVgXEVmTctPwrbeStLDpDfJPhARO0jaCPhJROxTcWgN5SsYa1pOJNaLvRoR\nr0taKmko8DywWdVBNVqffdilNT9Ju0i6T9LLkv4paVnhEfBmVZqaX4Z3IXA/6YV4d1cbUuO5isya\nlqSpwPuBnwETgKOArSLiM5UGZlYgaRQwNCIeqjiUhvMVjDW1iJgB9IuIZRHxY2D/qmMyk3SIpGEA\nETET+JukSdVG1XhOMNbMXslvd3xQ0tclfQxv09Y7nFF8fXdEzAfOqDCeSnhntGb2AaAfcArpsSqb\nAYdWGpFZ0tGxtc81qvI9GDOzOpM0GZgP/CAXnUx6Gd4xlQVVAScYa1r5twbtN+AFwFTgSxExr/FR\nmS1/TfLngXeRttEppG3ylUoDazAnGGtakr5OesjlFbno/cAg4Flgt4j4t6pis75N0mER8bPuylqd\nE4w1LUnT2r+Eq1ZWfGGXWaN1tW1WFVMV+txNJ2sp/SRNjIh7ASTtRLrpD+D311vDSToAOBDYVNI5\nhU5D6YPbpBOMNbPjgB9LGpy/LwSOy/XfX60uLOvD/k66B/ge0i/4axYCH6skogq5isyakqQ1gPdF\nxNWFH7Qt6GYws4aQ1D8i+twVS3tOMNa0JE2NiAlVx2FWI+nqiDi8kxaORMTYCsKqjBOMNS1JZwP/\nAH5K2/vriYgXKgvK+jRJG0fEHEmbd9Q9Ip5udExVcoKxpiXprx0UR0S8ueHBmNkbOMGYmdWJpIWs\nWDWm/F2kk5+hlQRWEScYa1qSBgAfBnbPRbcBP4yIJZUFZWbLOcFY05L0I2AAcEku+gCwLCKOry4q\ns0TSDsA78tc7+uL7YJxgrGlJ+lNE7NBdmVmjSToVOAG4NhcdAlwQEedWF1Xj+YeW1syWSXpLRPwF\nQNKbSc8mM6vaccDOEbEIQNLXSK9MdoIxaxKfBG6V9BTpJurmwAerDckMSNtj8WRnWS7rU5xgrGlF\nxC2StgS2zkWPR8RrVcZklv0Y+KOk6/L3ScBFFcZTCd+DMTMrgaTxwG75650R8UCV8VTBCcbMrE4k\nDQROArYAHgYu6svPJHOCMTOrE0k/BZYAdwIHADMj4qPVRlUdJxhrapLWA7YEBtbKIuKO6iKyvqz4\nojtJ/YF7+9pLxop8k9+alqTjgVOBkcCDwC6kpqB7VxmX9WnLnyIREUulPtdwbAW+grGmlR+JvhNw\nT0SMk7QN8JWIeG/FoVkfJWkZbU/2FrA28Ap99FlkvoKxZrY4IhZLQtJaEfGYpK27H8ysHBHRr/u+\n+g4nGGtmsyWtC/wcmCLpRaBPvW/DrDdzFZm1BEl7AMOAX0fEP6uOx8ycYKwJSRoaES9JGt5Rd7/R\n0qx3cIKxpiPphoh4d36jZe1lTjV+o6VZL+EEY2ZmpfBNfms6+RlPnYqIaY2Kxcw65ysYazqSbs0f\nBwITgD+RqsnGAlMjYteqYjOzNmtUHYDZyoqIvSJiL2AOMD4iJkTEjsDbgGeqjc7MapxgrJltHREP\n175ExCPAmArjMbMC34OxZvaQpB8BP8nfjwQeqjAeMyvwPRhrWvndGx8Gds9FdwDnRcTi6qIysxon\nGGtqktYG3hQRj1cdi5mtyPdgrGlJeg/pMf2/zt/HSbq+2qjMrMYJxprZGcBEYD5ARDwIjK40IjNb\nzgnGmtmSiFjQrsx1vma9hFuRWTN7VNJ/AP0kbQl8BLir4pjMLPMVjDWz/wa2A14DrgReAj5aaURm\ntpxbkZmZWSlcRWZNp7uWYhHxnkbFYmadc4KxZrQrMItULfZHVnwfjJn1Eq4is6YjqR+wD3AE6QnK\nNwJXRsSjlQZmZivwTX5rOhGxLCJ+HRFHA7sAM4DbJJ1ScWhmVuAqMmtKktYCDiJdxYwCzgGuqzIm\nM1uRq8is6Ui6FNgeuAm4Kj+m38x6GScYazqSXgcW5a/FDVhARMTQxkdlZu05wZiZWSl8k9/MzErh\nBGNmZqVwgjEzs1I4wZiZWSmcYMzMrBROMGaApFGSpku6UNKjkm6WtLakEyTdJ+lPkq6RNCj3f7Gk\n8yTdI+kpSXtKmpzHcXFhvPtKulvSNEk/kzS4spk0azAnGLM2WwI/iIjtSK9hPhS4NiJ2iogdgOnA\ncYX+1yM9ePNjwPXAd0jvp3mrpHGSNgA+B7wrIsYDU4GPN2xuzCrmR8WYtflrRDyYP99PegTN9pK+\nBKwLDAZ+U+j/lxERkh4GnouIhwEkPZqHHQlsC/xBEsCawN0NmA+zXsEJxqzNa4XPy4C1gYuBSRHx\nJ0nHAHt20P/r7YZ9nbRvLQOmRMQRJcVr1qu5isysa0OAOZIGAEeu5LD3AG+XtAWApHUkbVXvAM16\nKycYs659nvRSsz8Aj63MgBExFzgGuFLSQ6TqsW3qHaBZb+VnkZmZWSl8BWNmZqVwgjEzs1I4wZiZ\nWSmcYMzMrBROMGZmVgonGDMzK4UTjJmZlcIJxszMSvH/AWGYfYa673ucAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "data = session.get(test_td+'tag_agg/').json()\n", "pd.DataFrame(data).pivot_table(index='name',columns='agrees',values='count').plot.bar(\n", " stacked=True,\n", " \n", " title=\"Agreement breakdown of tags for task {name}\".format(name=\"test-set-tags-and-classes\")\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tag level error analysis and confusion matrix\n", "Similar to classifications, we'd like to understand why are labelers are disagreeing. As mentioned before, tags have a location in the example which means their are a number of possible ways to disagree. The endpoint **tag_confusion** on a task definition will return the error type; it's possible values are\n", "### Types of tag errors\n", "1. **agree** The location and type of the annotation were equal (this is ideal)\n", "2. **singular** Annotator A annotated but annotator B did not\n", "3. **tag_mismatch** Annotator A and B agreed on the location of an annotation but not it's tag. For example, \"White House\" might have been labeled by A as location and B as Organization\n", "4. **span_mismatch** Annotator A and B made an overlapping annotation with the same type, but the they were not equal. For example, \"President Donald Trump\", A and B will both label \"Person\" but A selected \"President Donald Trump\" and B ommited President, labeling only \"Donald Trump\"\n", "5. **tag_span_mismatch** A and B made an overlapping annotation, but the location was not equal and the tag was different. For example, \"The White House\", A labeled the entire phrase as Location while B labeled just \"White House\" as an organisation" ] }, { "cell_type": "code", "execution_count": 193, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
counterrortag_atag_b
031agreeInsultInsult
146agreeIssueIssue
211agreeMedia organizationMedia organization
3118agreePersonPerson
484agreePlacePlace
53agreePolitcal GroupPolitcal Group
6880singularInsult
71228singularIssue
8615singularMedia organization
91707singularPerson
\n", "
" ], "text/plain": [ " count error tag_a tag_b\n", "0 31 agree Insult Insult\n", "1 46 agree Issue Issue\n", "2 11 agree Media organization Media organization\n", "3 118 agree Person Person\n", "4 84 agree Place Place\n", "5 3 agree Politcal Group Politcal Group\n", "6 880 singular Insult \n", "7 1228 singular Issue \n", "8 615 singular Media organization \n", "9 1707 singular Person " ] }, "execution_count": 193, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import seaborn as sns\n", "data = session.get(test_td+'tag_confusion/').json() # Fetch the tag level \n", "D = pd.DataFrame(data).fillna('') #Fill null entriess with a blank string\n", "D.head(10)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As always, it is convenient to pivot the data, this time using a multi index" ] }, { "cell_type": "code", "execution_count": 194, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
erroragreesingularspan_mismatchspan_tag_mismatchtag_mismatch
tag_atag_b
Insult0.0880.00.00.00.0
Insult31.00.049.00.00.0
Issue0.00.00.064.041.0
Media organization0.00.00.031.014.0
Person0.00.00.0112.047.0
Place0.00.00.088.044.0
Politcal Group0.00.00.022.013.0
Issue0.01228.00.00.00.0
Insult0.00.00.053.041.0
Issue46.00.098.00.00.0
Media organization0.00.00.044.033.0
Person0.00.00.0110.071.0
Place0.00.00.0129.054.0
Politcal Group0.00.00.023.015.0
Media organization0.0615.00.00.00.0
Insult0.00.00.018.023.0
Issue0.00.00.038.024.0
Media organization11.00.031.00.00.0
Person0.00.00.050.042.0
Place0.00.00.051.029.0
Politcal Group0.00.00.04.07.0
Person0.01707.00.00.00.0
Insult0.00.00.099.045.0
Issue0.00.00.0150.077.0
Media organization0.00.00.074.026.0
Person118.00.0198.00.00.0
Place0.00.00.0174.086.0
Politcal Group0.00.00.034.021.0
Place0.01509.00.00.00.0
Insult0.00.00.077.045.0
Issue0.00.00.0116.060.0
Media organization0.00.00.063.038.0
Person0.00.00.0179.095.0
Place84.00.0150.00.00.0
Politcal Group0.00.00.027.016.0
Politcal Group0.0253.00.00.00.0
Insult0.00.00.020.010.0
Issue0.00.00.025.016.0
Media organization0.00.00.011.07.0
Person0.00.00.035.024.0
Place0.00.00.032.018.0
Politcal Group3.00.04.00.00.0
\n", "
" ], "text/plain": [ "error agree singular span_mismatch \\\n", "tag_a tag_b \n", "Insult 0.0 880.0 0.0 \n", " Insult 31.0 0.0 49.0 \n", " Issue 0.0 0.0 0.0 \n", " Media organization 0.0 0.0 0.0 \n", " Person 0.0 0.0 0.0 \n", " Place 0.0 0.0 0.0 \n", " Politcal Group 0.0 0.0 0.0 \n", "Issue 0.0 1228.0 0.0 \n", " Insult 0.0 0.0 0.0 \n", " Issue 46.0 0.0 98.0 \n", " Media organization 0.0 0.0 0.0 \n", " Person 0.0 0.0 0.0 \n", " Place 0.0 0.0 0.0 \n", " Politcal Group 0.0 0.0 0.0 \n", "Media organization 0.0 615.0 0.0 \n", " Insult 0.0 0.0 0.0 \n", " Issue 0.0 0.0 0.0 \n", " Media organization 11.0 0.0 31.0 \n", " Person 0.0 0.0 0.0 \n", " Place 0.0 0.0 0.0 \n", " Politcal Group 0.0 0.0 0.0 \n", "Person 0.0 1707.0 0.0 \n", " Insult 0.0 0.0 0.0 \n", " Issue 0.0 0.0 0.0 \n", " Media organization 0.0 0.0 0.0 \n", " Person 118.0 0.0 198.0 \n", " Place 0.0 0.0 0.0 \n", " Politcal Group 0.0 0.0 0.0 \n", "Place 0.0 1509.0 0.0 \n", " Insult 0.0 0.0 0.0 \n", " Issue 0.0 0.0 0.0 \n", " Media organization 0.0 0.0 0.0 \n", " Person 0.0 0.0 0.0 \n", " Place 84.0 0.0 150.0 \n", " Politcal Group 0.0 0.0 0.0 \n", "Politcal Group 0.0 253.0 0.0 \n", " Insult 0.0 0.0 0.0 \n", " Issue 0.0 0.0 0.0 \n", " Media organization 0.0 0.0 0.0 \n", " Person 0.0 0.0 0.0 \n", " Place 0.0 0.0 0.0 \n", " Politcal Group 3.0 0.0 4.0 \n", "\n", "error span_tag_mismatch tag_mismatch \n", "tag_a tag_b \n", "Insult 0.0 0.0 \n", " Insult 0.0 0.0 \n", " Issue 64.0 41.0 \n", " Media organization 31.0 14.0 \n", " Person 112.0 47.0 \n", " Place 88.0 44.0 \n", " Politcal Group 22.0 13.0 \n", "Issue 0.0 0.0 \n", " Insult 53.0 41.0 \n", " Issue 0.0 0.0 \n", " Media organization 44.0 33.0 \n", " Person 110.0 71.0 \n", " Place 129.0 54.0 \n", " Politcal Group 23.0 15.0 \n", "Media organization 0.0 0.0 \n", " Insult 18.0 23.0 \n", " Issue 38.0 24.0 \n", " Media organization 0.0 0.0 \n", " Person 50.0 42.0 \n", " Place 51.0 29.0 \n", " Politcal Group 4.0 7.0 \n", "Person 0.0 0.0 \n", " Insult 99.0 45.0 \n", " Issue 150.0 77.0 \n", " Media organization 74.0 26.0 \n", " Person 0.0 0.0 \n", " Place 174.0 86.0 \n", " Politcal Group 34.0 21.0 \n", "Place 0.0 0.0 \n", " Insult 77.0 45.0 \n", " Issue 116.0 60.0 \n", " Media organization 63.0 38.0 \n", " Person 179.0 95.0 \n", " Place 0.0 0.0 \n", " Politcal Group 27.0 16.0 \n", "Politcal Group 0.0 0.0 \n", " Insult 20.0 10.0 \n", " Issue 25.0 16.0 \n", " Media organization 11.0 7.0 \n", " Person 35.0 24.0 \n", " Place 32.0 18.0 \n", " Politcal Group 0.0 0.0 " ] }, "execution_count": 194, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Res =D.pivot_table(index=['tag_a','tag_b'],columns=['error'],values='count',)\n", "Res = Res.fillna(0)\n", "Res" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Downloading Results\n", "Of course, sometimes we just want to results\n", "These are easily attained from a task_definitions **download** endpoint\n", "The download view returns \"everything\", that is it returns the entire schema definition, dataset definition, all of the examples and the annotations and classifications. We'll quickly explore the structure of the results here" ] }, { "cell_type": "code", "execution_count": 198, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'http://localhost:8000/api/v1/projects/default/task_definitions/test-set-tags-and-classes/'" ] }, "execution_count": 198, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_td" ] }, { "cell_type": "code", "execution_count": 200, "metadata": {}, "outputs": [], "source": [ "data = session.get(test_td+'download/').json() \n" ] }, { "cell_type": "code", "execution_count": 201, "metadata": {}, "outputs": [], "source": [ "res= data['result']\n", "x = next(filter(lambda x:x['results']['annotations'],res))" ] }, { "cell_type": "code", "execution_count": 202, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'example': {'content': \"Great listening session with CEO's of the Retail Industry Leaders Association this morning! https://t.co/sy6xJcWfcF\",\n", " 'definition_id': 'f91a95a4-61e5-40f8-87b4-63103e1f00f0',\n", " 'example_id': '008eb3c3-ac50-474e-b58c-04f755987ca0',\n", " 'metadata': {'created_at': 'Wed Feb 15 16:34:27 +0000 2017',\n", " 'date': '2017-02-15',\n", " 'favorite_count': 66081,\n", " 'id_str': '831904516316479489',\n", " 'in_reply_to_user_id_str': None,\n", " 'is_retweet': False,\n", " 'retweet_count': 10692,\n", " 'source': 'Twitter for iPhone',\n", " 'time': 1487176467000000000}},\n", " 'results': {'annotations': [{'annotator_id': 2,\n", " 'end': 65,\n", " 'from_suggestion': False,\n", " 'start': 58,\n", " 'tag': 'Issue',\n", " 'value': 'Leaders'},\n", " {'annotator_id': 4,\n", " 'end': 5,\n", " 'from_suggestion': False,\n", " 'start': 0,\n", " 'tag': 'Issue',\n", " 'value': 'Great'},\n", " {'annotator_id': 4,\n", " 'end': 48,\n", " 'from_suggestion': False,\n", " 'start': 38,\n", " 'tag': 'Person',\n", " 'value': 'the Retail'},\n", " {'annotator_id': 4,\n", " 'end': 115,\n", " 'from_suggestion': False,\n", " 'start': 105,\n", " 'tag': 'Person',\n", " 'value': 'sy6xJcWfcF'},\n", " {'annotator_id': 3,\n", " 'end': 23,\n", " 'from_suggestion': False,\n", " 'start': 16,\n", " 'tag': 'Person',\n", " 'value': 'session'},\n", " {'annotator_id': 3,\n", " 'end': 57,\n", " 'from_suggestion': False,\n", " 'start': 49,\n", " 'tag': 'Insult',\n", " 'value': 'Industry'},\n", " {'annotator_id': 3,\n", " 'end': 65,\n", " 'from_suggestion': False,\n", " 'start': 58,\n", " 'tag': 'Insult',\n", " 'value': 'Leaders'},\n", " {'annotator_id': 3,\n", " 'end': 90,\n", " 'from_suggestion': False,\n", " 'start': 83,\n", " 'tag': 'Media organization',\n", " 'value': 'morning'},\n", " {'annotator_id': 2,\n", " 'end': 115,\n", " 'from_suggestion': False,\n", " 'start': 105,\n", " 'tag': 'Insult',\n", " 'value': 'sy6xJcWfcF'},\n", " {'annotator_id': 4,\n", " 'end': 77,\n", " 'from_suggestion': False,\n", " 'start': 66,\n", " 'tag': 'Place',\n", " 'value': 'Association'},\n", " {'annotator_id': 4,\n", " 'end': 37,\n", " 'from_suggestion': False,\n", " 'start': 35,\n", " 'tag': 'Issue',\n", " 'value': 'of'},\n", " {'annotator_id': 2,\n", " 'end': 28,\n", " 'from_suggestion': False,\n", " 'start': 16,\n", " 'tag': 'Media organization',\n", " 'value': 'session with'}],\n", " 'classifications': [{'annotator_id': 4, 'classname': 'Praise'},\n", " {'annotator_id': 3, 'classname': 'Praise'},\n", " {'annotator_id': 2, 'classname': 'Insult'}]}}" ] }, "execution_count": 202, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x" ] }, { "cell_type": "code", "execution_count": 203, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['schema', 'dataset', 'id', 'result'])" ] }, "execution_count": 203, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The dataset and schema fields are the same we've seen before when we defined our schema and dataset. LightTag provides them so you don't need to do further lookups.\n", "\n", "The actual results live in the result key, which itself contains a list.\n", "Each item in that list contains the **example** that was annotated and the results of the annotation " ] }, { "cell_type": "code", "execution_count": 204, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['results', 'example'])" ] }, "execution_count": 204, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['result'][0].keys()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The results field further divides into annotations and classifications" ] }, { "cell_type": "code", "execution_count": 205, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['annotations', 'classifications'])" ] }, "execution_count": 205, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['result'][0]['results'].keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An annotation result contains the \n", "1. id of the annotator, \n", "2. the start and end of the annotation, \n", "3. the tag applied, \n", "4. the value, that is the string that was captured" ] }, { "cell_type": "code", "execution_count": 206, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'annotator_id': 2,\n", " 'end': 65,\n", " 'from_suggestion': False,\n", " 'start': 58,\n", " 'tag': 'Issue',\n", " 'value': 'Leaders'}" ] }, "execution_count": 206, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['result'][0]['results']['annotations'][0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A classification result contains the \n", "1. id of the annotator, \n", "3. the classname applied, \n" ] }, { "cell_type": "code", "execution_count": 207, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'annotator_id': 4, 'classname': 'Praise'}" ] }, "execution_count": 207, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['result'][0]['results']['classifications'][0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The example field contains the text we labeled" ] }, { "cell_type": "code", "execution_count": 208, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\"Great listening session with CEO's of the Retail Industry Leaders Association this morning! https://t.co/sy6xJcWfcF\"" ] }, "execution_count": 208, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['result'][0]['example']['content']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As well as any additional metadata we provided" ] }, { "cell_type": "code", "execution_count": 209, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'created_at': 'Wed Feb 15 16:34:27 +0000 2017',\n", " 'date': '2017-02-15',\n", " 'favorite_count': 66081,\n", " 'id_str': '831904516316479489',\n", " 'in_reply_to_user_id_str': None,\n", " 'is_retweet': False,\n", " 'retweet_count': 10692,\n", " 'source': 'Twitter for iPhone',\n", " 'time': 1487176467000000000}" ] }, "execution_count": 209, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data['result'][0]['example']['metadata']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running custom analytics\n", "Sometimes we want to do some analysis on our data that LightTag does not yet provide. In that case we are down to simple JSON manipilation. \n", "Let's get the top 5 words for each tag as an example" ] }, { "cell_type": "code", "execution_count": 210, "metadata": {}, "outputs": [], "source": [ "#Aggregate all of the results\n", "annotations = []\n", "for result in data['result']:\n", " annotations +=result['results']['annotations']" ] }, { "cell_type": "code", "execution_count": 211, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
tagInsultIssueMedia organizationPersonPlacePolitcal Group
tagInsultIssueMedia organizationPersonPlacePolitcal Group
value
INaNNaNNaNNaNNaN7.0
andNaN27.013.041.038.07.0
coNaN27.0NaNNaN36.0NaN
forNaNNaN11.0NaNNaNNaN
https20.0NaNNaNNaNNaNNaN
is20.0NaNNaNNaNNaNNaN
of16.0NaN14.034.0NaNNaN
tNaN26.0NaN46.041.07.0
the35.047.019.060.064.07.0
to23.030.023.054.038.09.0
\n", "
" ], "text/plain": [ "tag Insult Issue Media organization Person Place Politcal Group\n", "tag Insult Issue Media organization Person Place Politcal Group\n", "value \n", "I NaN NaN NaN NaN NaN 7.0\n", "and NaN 27.0 13.0 41.0 38.0 7.0\n", "co NaN 27.0 NaN NaN 36.0 NaN\n", "for NaN NaN 11.0 NaN NaN NaN\n", "https 20.0 NaN NaN NaN NaN NaN\n", "is 20.0 NaN NaN NaN NaN NaN\n", "of 16.0 NaN 14.0 34.0 NaN NaN\n", "t NaN 26.0 NaN 46.0 41.0 7.0\n", "the 35.0 47.0 19.0 60.0 64.0 7.0\n", "to 23.0 30.0 23.0 54.0 38.0 9.0" ] }, "execution_count": 211, "metadata": {}, "output_type": "execute_result" } ], "source": [ "annotations_df = pd.DataFrame(annotations)\n", "z = annotations_df.groupby('tag').value.value_counts()\n", "z.groupby(level=0).nlargest(5).unstack().T" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }