{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Setting up\n", "You can copy and paste the code here to an R script. If you have installed R and Jupyter (via anaconda) you can download the .ipynb source file [here](http://research.bowdoin.edu/galileos-library/ariosto-and-tasso/galileos-ariosto/). For Mac users, if you have not already done so, please follow the [instructions](https://irkernel.github.io/installation/) to link your current version of R with Jupyter. All output will appear in the folder where this file is saved. It is presumed that your data is in a subfolder within this folder.\n", "\n", "Please cite this code if you find it useful for your research:\n", "Hall, Crystal. \"Galileo's Ariosto - Code Tutorial.\" *Galileo's Library.* http://www.bowdoin.edu/~chall/GalileosAriostoCodeTutorial.html August 31, 2017. \n", "\n", "## Getting Started\n", "The code below loads R packages with functions necessary for the processes below. Included are also functions that I have written to streamline processing of files." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Loading required package: projectoR\n" ] } ], "source": [ "#install.packages(\"plyr\") #Run these lines if you have not already installed these packages.\n", "#install.packages(\"tau\")\n", "#install.pacakges(\"devtools\")\n", "#devtools::install_github(“mdlincoln/projectoR\")\n", "\n", "library(plyr)\n", "library(tau)\n", "require(projectoR)\n", "\n", "clean <- function(filename){ # a function to read and process text files\n", " scanned <- scan(filename, what = \"character\", sep = \"\\n\", quote = \"\")\n", " bp <- grep(\"Ogni uso commerciale è vietato\", scanned) # Removes boiler-plate language from Biblioteca Italiana\n", " if(length(bp) > 0){\n", " scanned <- scanned[(bp+1):length(scanned)]\n", " }\n", " string <- paste(scanned, collapse = \" \") # See Matt Jockers' Macroanalysis for an explanation of these lines\n", " cleaned1 <- gsub(\"[[:digit:]]+\", \"\", string)\n", " cleaned2 <- noaccents(cleaned1)\n", " lowered <- tolower(unlist(strsplit(cleaned2, \"\\\\W\")))\n", " cleaned3 <- gsub(\"v\", \"u\", lowered) #Standardizes orthography based on early modern print\n", " cleaned4 <- gsub(\"j\", \"i\", cleaned3)#Standardizes orthography based on early modern print\n", " noblanks <- which(cleaned4 !=\"\")\n", " words <- cleaned4[noblanks] # vector of words in order of text\n", " uniquewords <- unique(words) # vector of unique words in text\n", " table <- sort(table(words), decreasing = TRUE) # table of word frequencies\n", " bigrams <- textcnt(words, n = 2, method = \"string\")\n", " bigrams_m <- as.matrix(bigrams[order(bigrams, decreasing = TRUE)])\n", " bg_l <- list(bigrams, bigrams_m)\n", " trigrams <- textcnt(words, n = 3, method = \"string\")\n", " trigrams_m <- as.matrix(trigrams[order(trigrams, decreasing = TRUE)])\n", " tg_l <- list(trigrams, trigrams_m)\n", " fourgrams <- textcnt(words, n = 4, method = \"string\")\n", " fourgrams_m <- as.matrix(fourgrams[order(fourgrams, decreasing = TRUE)])\n", " frg_l <- list(fourgrams, fourgrams_m)\n", " fivegrams <- textcnt(words, n = 5, method = \"string\")\n", " fivegrams_m <- as.matrix(fivegrams[order(fivegrams, decreasing = TRUE)])\n", " fvg_l <- list(fivegrams, fivegrams_m)\n", " data <- list(filename, words, uniquewords, table, bigrams = bg_l, trigrams = tg_l, fourgrams = frg_l, fivegrams = fvg_l)\n", " return(data)\n", "}\n", "\n", "noaccents <- function(string){ # a function to standardize orthography due to inconsistencies in early modern print\n", " string <- gsub(\"à\", \"a\", string)\n", " string <- gsub(\"á\", \"a\", string)\n", " string <- gsub(\"è\", \"e\", string)\n", " string <- gsub(\"é\", \"e\", string)\n", " string <- gsub(\"í\", \"i\", string)\n", " string <- gsub(\"ì\", \"i\", string)\n", " string <- gsub(\"ò\", \"o\", string)\n", " string <- gsub(\"ó\", \"o\", string)\n", " string <- gsub(\"ù\", \"u\", string)\n", " string <- gsub(\"ú\", \"u\", string)\n", " return(string)\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Input of Files\n", "The code below requires input from you: the path to the folder where your plain text (UTF-8) files are saved. Change the definition of *path* and run the code cell. You may notice that the article only talks about 47 documents, but there are 49 in the example. The \"extra\" two texts are the earlier versions of the *Orlando furioso*, included as controls for this project." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1] \"Your folder contains 49 files.\"\n" ] } ], "source": [ "path <- \"Data/OriginalEditions\" # Change the information between quotation marks to match your file structure\n", "filenames <- dir(path) # Do not edit the lines below\n", "docs <- list()\n", "for(i in 1:length(filenames)){\n", " docs[[i]] <- clean(paste(path, filenames[i], sep = \"/\"))\n", "}\n", "print(paste(\"Your folder contains \", length(docs), \" files.\", sep = \"\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The objects *docs* (created above) is like a filing cabinet, in this case with 49 drawers, one for each text. Each drawer contains 8 hanging file folders with different information:\n", "1. The filename.\n", "2. A vector of all words in the text, in the order they occur.\n", "3. A vector of the word types in the text.\n", "4. A table of word frequencies.\n", "5. Bigram information: an alphabetized table of counts and a matrix of the same.\n", "6. Trigram information: an alphabetized table of counts and a matrix of the same.\n", "7. Fourgram information: an alphabetized table of counts and a matrix of the same.\n", "8. Fivegram information: an alphabetized table of counts and a matrix of the same.\n", "\n", "## Selecting Reference and Focus Files\n", "\n", "In the article, the reference file is Ariosto's *Orlando furioso*, the text against which all others will be compared. The focus file is Galileo's *Saggiatore*, the text whose style is of most interest.\n", "\n", "Run the code cell below to determine the number associated with your reference and focus texts. In this example the *Orlando furioso* is 3 and the *Saggiatore* is 44." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " [1] \"1516_OF_Ariosto.txt\" \n", " [2] \"1521_OF_Ariosto.txt\" \n", " [3] \"1525_FaceteDiLuciano_daLonigo.txt\" \n", " [4] \"1532_OF_Ariosto.txt\" \n", " [5] \"1536_PetrarcaSpirituale_Malipiero.txt\" \n", " [6] \"1538_Rime_Brocardo.txt\" \n", " [7] \"1539_DialogoDonne_Piccolomini.txt\" \n", " [8] \"1542_Rime_Sandoval.txt\" \n", " [9] \"1545_Egle_GiraldiCinzio.txt\" \n", "[10] \"1547_ItaliaGoti_Trissino.txt\" \n", "[11] \"1548_Fiamme_GiraldiCinzio.txt\" \n", "[12] \"1551_PublicheOrationi_Vergerio_EEBO.txt\"\n", "[13] \"1555_Sacrificio_Beccari.txt\" \n", "[14] \"1557_Ercole_GiraldiCinzio.txt\" \n", "[15] \"1557_Leonora_Betussi.txt\" \n", "[16] \"1559_Amore_Minturno.txt\" \n", "[17] \"1559_IstituzioneUomo_Piccolomini.txt\" \n", "[18] \"1561_TreLibriAmore_Cattani.txt\" \n", "[19] \"1563_Metamorfosi_Ovid_Anguillara.txt\" \n", "[20] \"1570_Rinaldo_TTasso.txt\" \n", "[21] \"1571_Gentilhuomo_Muzio.txt\" \n", "[22] \"1574_Ragionamento_Contile.txt\" \n", "[23] \"1577_Aminta_TTasso.txt\" \n", "[24] \"1580_Discorso_Bocchi.txt\" \n", "[25] \"1581_GerusalemmeLiberata_TTasso.txt\" \n", "[26] \"1581_VitaCarloMagno_Ubaldini.txt\" \n", "[27] \"1584_EducationeChristiana_Antoniano.txt\"\n", "[28] \"1589_Ragion_Botero.txt\" \n", "[29] \"1590_Viandante_Zinano.txt\" \n", "[30] \"1594_StatoTreCorti_Ubaldini_EEBO.txt\" \n", "[31] \"1595_SceltaAttioni_Ubaldini_EEBO.txt\" \n", "[32] \"1596_Addizione_Betussi.txt\" \n", "[33] \"1596_Rime_Ubaldini_EEBO.txt\" \n", "[34] \"1598_RagioneStato_Botero.txt\" \n", "[35] \"15xx_Satire_Bolognetti.txt\" \n", "[36] \"1600_AggiunteRagionStato_Botero.txt\" \n", "[37] \"1607_Monarchia_Botero.txt\" \n", "[38] \"1607_Nobilta_Botero.txt\" \n", "[39] \"1607_RelazioneNizza_Botero.txt\" \n", "[40] \"1607_RelazionePiamonte_Botero.txt\" \n", "[41] \"1607_RelazioneSpagna_Botero.txt\" \n", "[42] \"1616_GliOcchi_Peretto_EEBO.txt\" \n", "[43] \"1622_Rime_Guarini.txt\" \n", "[44] \"1623_Saggiatore_Galileo.txt\" \n", "[45] \"1639_Alfea_Cascina.txt\" \n", "[46] \"1675_DifesaLingua_DeLuca.txt\" \n", "[47] \"16xx_Damone_Baldini.txt\" \n", "[48] \"1757_Rime_Burchiello.txt\" \n", "[49] \"1864_Raverta_Betussi.txt\" \n" ] } ], "source": [ "print(filenames)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Running the Comparison\n", "\n", "The code below adds a hanging file folder to each drawer with comparison information to your reference text. This is a great moment to define your reference and focus texts. Then run the code." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [], "source": [ "reference <- 3 # Change to the number that matches your reference text filename\n", "focus <- 44 # Change to the number that matches your focus text filename\n", "\n", "compareToReference <- function(text1_l, text2_l){ # a function to facilitate comparison (can be scaled up for multiple reference documents)\n", " comparison <- list(intersect(text1_l[[2]], text2_l[[2]]))\n", " return(comparison)\n", "}\n", " \n", "for(i in 1:length(docs)){\n", " docs[[i]][[9]] <- compareToReference(docs[[i]], docs[[reference]])\n", "} " ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Comparing the Comparisons\n", "\n", "Now that we have compared every text to the reference text (the *Orlando furioso* in the article), we want to evaluate similarities and differences between these texts' relationships to that reference file.\n", "\n", "The code below is a bit repetitive (ie: not designed by a software engineer), but it is hopefully transparent. It creates a new filing cabinet, still with 49 drawers (for this example). Each drawer now has 49 hanging file folders, one for every text to which it is compared. Each hanging file folder will have five manila folders with three pieces of information in each:\n", "\n", "**For example:**\n", "Drawer 44 [[Galileo's *Saggiatore*]]:\n", " > Hanging File Folder 3 [[Ariosto' *Orlando furioso* 1532 ed.]]:\n", " > Manila Folder 1: unigram comparison information \n", " > percentage of all words in *Saggiatore* that are in the *Furioso*\n", " > percentage of word types in *Saggiatore* that are in the *Furioso*\n", " > vector of which words in *Saggiatore* are in the *Furioso*\n", " > Manila Folder 2: bigram comparison information \n", " > vector of which bigrams in *Saggiatore* are in the *Furioso*\n", " > table of frequencies of bigrams in *Saggiatore* that are in the *Furioso*\n", " > percentage of all bigrams in *Saggiatore* that are in the *Furioso* \n", " > Manila Folder 3: trigram comparison information \n", " > vector of which trigrams in *Saggiatore* are in the *Furioso*\n", " > table of frequencies of trigrams in *Saggiatore* that are in the *Furioso*\n", " > percentage of all trigrams in *Saggiatore* that are in the *Furioso*\n", " > Manila Folder 4: fourgram comparison information \n", " > vector of which fourgrams in *Saggiatore* are in the *Furioso*\n", " > table of frequencies of fourgrams in *Saggiatore* that are in the *Furioso*\n", " > percentage of all fourgrams in *Saggiatore* that are in the *Furioso*\n", " > Manila Folder 5: fivegram comparison information \n", " > vector of which fivegrams in *Saggiatore* are in the *Furioso*\n", " > table of frequencies of fivegrams in *Saggiatore* that are in the *Furioso*\n", " > percentage of all fivegrams in *Saggiatore* that are in the *Furioso*\n", "\n", "For the situation of a text being compared to itself, the information is populated with 100% matches." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Just run this cell. No need to edit.\n", "similarities <- list()\n", "for(i in 1:length(docs)){\n", " vec <- docs[[i]][[2]]\n", " file1 <- docs[[i]][[1]]\n", " bigr <- row.names(docs[[i]][[5]][[2]])\n", " trigr <- row.names(docs[[i]][[6]][[2]])\n", " frgr <- row.names(docs[[i]][[7]][[2]])\n", " fvgr <- row.names(docs[[i]][[8]][[2]])\n", " sim <- list()\n", " for(x in 1:length(docs)){\n", " file2 <- docs[[x]][[1]]\n", " mini_sim <- list()\n", " if(i != x){\n", " comp <- intersect(vec, docs[[x]][[2]])\n", " token_perc <- 100*length(comp)/length(docs[[i]][[3]])\n", " type_perc <- 100*length(which(vec %in% comp))/length(vec)\n", " mini_sim[[1]] <- list(comp, token_perc, type_perc)\n", " bi_comp <- intersect(bigr, row.names(docs[[x]][[5]][[2]]))\n", " bi_comp_m <- docs[[i]][[5]][[2]][which(bigr %in% bi_comp),]\n", " bi_comp_p <- 100*length(bi_comp)/length(bigr)\n", " mini_sim[[2]] <- list(bi_comp, bi_comp_m, bi_comp_p)\n", " tri_comp <- intersect(trigr, row.names(docs[[x]][[6]][[2]]))\n", " tri_comp_m <- docs[[i]][[6]][[2]][which(trigr %in% tri_comp),]\n", " tri_comp_p <- 100*length(tri_comp)/length(trigr)\n", " mini_sim[[3]] <- list(tri_comp, tri_comp_m, tri_comp_p)\n", " fr_comp <- intersect(frgr, row.names(docs[[x]][[7]][[2]]))\n", " fr_comp_m <- docs[[i]][[7]][[2]][which(frgr %in% fr_comp),]\n", " fr_comp_p <- 100*length(fr_comp)/length(frgr)\n", " mini_sim[[4]] <- list(fr_comp, fr_comp_m, fr_comp_p)\n", " fv_comp <- intersect(fvgr, row.names(docs[[x]][[8]][[2]]))\n", " fv_comp_m <- docs[[i]][[8]][[2]][which(fvgr %in% fv_comp),]\n", " fv_comp_p <- 100*length(fv_comp)/length(fvgr)\n", " mini_sim[[5]] <- list(fv_comp, fv_comp_m, fv_comp_p)\n", " }\n", " else { # to account for a text being compared to itself\n", " mini_sim[[1]] <- list(100, 100, rep(\"same\", length(vec)))\n", " mini_sim[[2]] <- list(rep(\"same\", length(bigr)), \"Same\", 100)\n", " mini_sim[[3]] <- list(rep(\"same\", length(trigr)), \"Same\", 100)\n", " mini_sim[[4]] <- list(rep(\"same\", length(frgr)), \"Same\", 100)\n", " mini_sim[[5]] <- list(rep(\"same\", length(fvgr)), \"Same\", 100)\n", " }\n", " sim[[x]] <- list(file2, mini_sim)\n", " }\n", " similarities[[i]] <- list(file1, sim)\n", "}" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Evaluating the Comparisons\n", "\n", "The object created above, *similarities*, is a powerful resource. Each drawer has full comparison information - to every other text in the corpus, not just the reference text. This makes it easier to follow leads and respond to surprises in the results below.\n", "\n", "The code cell below creates 5 matrices with as many rows and columns as there are texts. In the case of our example, 49 x 49. Each matrix shows the percentage similarity that one text shares with another (without considering the reference text). After running the code, a snippet of a matrix will be displayed to show how to interpret the results." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\n", "
1516_OF_Ariosto.txt1521_OF_Ariosto.txt1525_FaceteDiLuciano_daLonigo.txt
1516_OF_Ariosto.txt100.0000000 59.6625511 0.09214062
1521_OF_Ariosto.txt 62.5361046 100.0000000 0.07726274
1525_FaceteDiLuciano_daLonigo.txt 0.1664757 0.1331805 100.00000000
1532_OF_Ariosto.txt 48.3087755 42.7574944 0.07331059
1536_PetrarcaSpirituale_Malipiero.txt 0.3908724 0.2153417 0.07057418
1538_Rime_Brocardo.txt 0.3715745 0.1857873 0.04644682
1539_DialogoDonne_Piccolomini.txt 0.5093520 0.3381764 0.43837675
1542_Rime_Sandoval.txt 0.5229420 0.2699055 0.10121457
1545_Egle_GiraldiCinzio.txt 0.4020678 0.3331419 0.22400919
1547_ItaliaGoti_Trissino.txt 0.5053917 0.3064270 0.11544211
\n" ], "text/latex": [ "\\begin{tabular}{r|lll}\n", " & 1516\\_OF\\_Ariosto.txt & 1521\\_OF\\_Ariosto.txt & 1525\\_FaceteDiLuciano\\_daLonigo.txt\\\\\n", "\\hline\n", "\t1516\\_OF\\_Ariosto.txt & 100.0000000 & 59.6625511 & 0.09214062\\\\\n", "\t1521\\_OF\\_Ariosto.txt & 62.5361046 & 100.0000000 & 0.07726274\\\\\n", "\t1525\\_FaceteDiLuciano\\_daLonigo.txt & 0.1664757 & 0.1331805 & 100.00000000\\\\\n", "\t1532\\_OF\\_Ariosto.txt & 48.3087755 & 42.7574944 & 0.07331059\\\\\n", "\t1536\\_PetrarcaSpirituale\\_Malipiero.txt & 0.3908724 & 0.2153417 & 0.07057418\\\\\n", "\t1538\\_Rime\\_Brocardo.txt & 0.3715745 & 0.1857873 & 0.04644682\\\\\n", "\t1539\\_DialogoDonne\\_Piccolomini.txt & 0.5093520 & 0.3381764 & 0.43837675\\\\\n", "\t1542\\_Rime\\_Sandoval.txt & 0.5229420 & 0.2699055 & 0.10121457\\\\\n", "\t1545\\_Egle\\_GiraldiCinzio.txt & 0.4020678 & 0.3331419 & 0.22400919\\\\\n", "\t1547\\_ItaliaGoti\\_Trissino.txt & 0.5053917 & 0.3064270 & 0.11544211\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "| | 1516_OF_Ariosto.txt | 1521_OF_Ariosto.txt | 1525_FaceteDiLuciano_daLonigo.txt | \n", "|---|---|---|---|---|---|---|---|---|---|\n", "| 1516_OF_Ariosto.txt | 100.0000000 | 59.6625511 | 0.09214062 | \n", "| 1521_OF_Ariosto.txt | 62.5361046 | 100.0000000 | 0.07726274 | \n", "| 1525_FaceteDiLuciano_daLonigo.txt | 0.1664757 | 0.1331805 | 100.00000000 | \n", "| 1532_OF_Ariosto.txt | 48.3087755 | 42.7574944 | 0.07331059 | \n", "| 1536_PetrarcaSpirituale_Malipiero.txt | 0.3908724 | 0.2153417 | 0.07057418 | \n", "| 1538_Rime_Brocardo.txt | 0.3715745 | 0.1857873 | 0.04644682 | \n", "| 1539_DialogoDonne_Piccolomini.txt | 0.5093520 | 0.3381764 | 0.43837675 | \n", "| 1542_Rime_Sandoval.txt | 0.5229420 | 0.2699055 | 0.10121457 | \n", "| 1545_Egle_GiraldiCinzio.txt | 0.4020678 | 0.3331419 | 0.22400919 | \n", "| 1547_ItaliaGoti_Trissino.txt | 0.5053917 | 0.3064270 | 0.11544211 | \n", "\n", "\n" ], "text/plain": [ " 1516_OF_Ariosto.txt 1521_OF_Ariosto.txt\n", "1516_OF_Ariosto.txt 100.0000000 59.6625511 \n", "1521_OF_Ariosto.txt 62.5361046 100.0000000 \n", "1525_FaceteDiLuciano_daLonigo.txt 0.1664757 0.1331805 \n", "1532_OF_Ariosto.txt 48.3087755 42.7574944 \n", "1536_PetrarcaSpirituale_Malipiero.txt 0.3908724 0.2153417 \n", "1538_Rime_Brocardo.txt 0.3715745 0.1857873 \n", "1539_DialogoDonne_Piccolomini.txt 0.5093520 0.3381764 \n", "1542_Rime_Sandoval.txt 0.5229420 0.2699055 \n", "1545_Egle_GiraldiCinzio.txt 0.4020678 0.3331419 \n", "1547_ItaliaGoti_Trissino.txt 0.5053917 0.3064270 \n", " 1525_FaceteDiLuciano_daLonigo.txt\n", "1516_OF_Ariosto.txt 0.09214062 \n", "1521_OF_Ariosto.txt 0.07726274 \n", "1525_FaceteDiLuciano_daLonigo.txt 100.00000000 \n", "1532_OF_Ariosto.txt 0.07331059 \n", "1536_PetrarcaSpirituale_Malipiero.txt 0.07057418 \n", "1538_Rime_Brocardo.txt 0.04644682 \n", "1539_DialogoDonne_Piccolomini.txt 0.43837675 \n", "1542_Rime_Sandoval.txt 0.10121457 \n", "1545_Egle_GiraldiCinzio.txt 0.22400919 \n", "1547_ItaliaGoti_Trissino.txt 0.11544211 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "tokens <- list()\n", "types <- list()\n", "bigrams <- list()\n", "trigrams <- list()\n", "fourgrams <- list()\n", "fivegrams <- list()\n", "\n", "for (i in 1:length(similarities)){\n", " examine <- similarities[[i]][[2]]\n", " token_percentage <- as.numeric()\n", " type_percentage <- as.numeric()\n", " bigram_count <- as.numeric()\n", " trigram_count <- as.numeric()\n", " fourgram_count <- as.numeric()\n", " fivegram_count <- as.numeric()\n", " for(x in 1:length(examine)){\n", " token_percentage[x] <- examine[[x]][[2]][[1]][[2]]\n", " type_percentage[x] <- examine[[x]][[2]][[1]][3]\n", " bigram_count[x] <- examine[[x]][[2]][[2]][[3]]\n", " trigram_count[x] <- examine[[x]][[2]][[3]][[3]]\n", " fourgram_count[x] <- examine[[x]][[2]][[4]][[3]]\n", " fivegram_count[x] <- examine[[x]][[2]][[5]][[3]]\n", " }\n", " tokens[[i]] <- token_percentage\n", " types[[i]] <- type_percentage\n", " bigrams[[i]] <- bigram_count\n", " trigrams[[i]] <- trigram_count\n", " fourgrams[[i]] <- fourgram_count\n", " fivegrams[[i]] <- fivegram_count\n", "}\n", "\n", "tokens_df <- do.call(rbind, tokens)\n", "colnames(tokens_df) <- filenames\n", "row.names(tokens_df) <- filenames\n", "#min(tokens_df) # To check the range of similarities - quite large!\n", "#max(tokens_df[which(tokens_df !=100)])\n", "\n", "types_df <- do.call(rbind, types)\n", "colnames(types_df) <- filenames\n", "row.names(types_df) <- filenames\n", "\n", "bigrams_df <- do.call(rbind, bigrams)\n", "colnames(bigrams_df) <- filenames\n", "row.names(bigrams_df) <- filenames\n", "\n", "trigrams_df <- do.call(rbind, trigrams)\n", "colnames(trigrams_df) <- filenames\n", "row.names(trigrams_df) <- filenames\n", "\n", "fourgrams_df <- do.call(rbind, fourgrams)\n", "colnames(fourgrams_df) <- filenames\n", "row.names(fourgrams_df) <- filenames\n", "\n", "fivegrams_df <- do.call(rbind, fivegrams)\n", "colnames(fivegrams_df) <- filenames\n", "row.names(fivegrams_df) <- filenames\n", "\n", "ngrams <- list(tokens_df, types_df, bigrams_df, trigrams_df, fourgrams_df, fivegrams_df)\n", "\n", "fourgrams_df[1:10, 1:3]" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "The matrix is read by row, from left column to right. We can already see the 100 (%) that shows that a text has 100% fourgrams in common with itself. For the other values, they can be read as follows:\n", " - Of the fourgrams in the 1521 edition of the *Orlando furioso* (row 2), 62.5% match the fourgrams in the 1516 edition (column 1).\n", " - Of the fourgrams in the 1547 edition of *Italia liberata dai Goti* (row 10), only 0.11% match the fourgrams in the 1525 translation of Lucian's witticisms (column 3).\n", "\n", "Even with this small snippet from just one of the matrices, we see the range of similarities between texts. If you want to save each matrix as a spreadsheet, run the code cell below." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "write.csv(types_df, \"CorpusWordTypeSimilarity.csv\", row.names = TRUE)\n", "write.csv(bigrams_df, \"CorpusBigramSimilarity.csv\", row.names = TRUE)\n", "write.csv(trigrams_df, \"CorpusTrigramSimilarity.csv\", row.names = TRUE)\n", "write.csv(fourgrams_df, \"CorpusFourgramSimilarity.csv\", row.names = TRUE)\n", "write.csv(fivegrams_df, \"CorpusFivegramSimilarity.csv\", row.names = TRUE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How the Focus Text Compares to the Corpus \n", "\n", "Now we are ready to examine the focus text, in this case Galileo's *Saggiatore*. We can extract the *Saggiatore* row from each matrix. That row number is the same as the number of the filename, stored as the variable *focus*. Your results will be saved in the folder where this .ipynb file is saved. Look for the \"SimilarityToCorpus.csv\" filename. You can open it with Excel." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": true }, "outputs": [], "source": [ "focusToCorpus <- cbind(tokens_df[focus,], types_df[focus,], bigrams_df[focus,], trigrams_df[focus,], fourgrams_df[focus,], fivegrams_df[focus,])\n", "metadata <- unlist(strsplit(filenames[focus], \"_\"))\n", "fn <- paste(metadata[2], \"SimilarityToCorpus.csv\", sep = \"\")\n", "write.csv(focusToCorpus, file = fn, row.names = TRUE)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## How the Focus Text Compares to the Corpus in Relationship to the Reference Text\n", "\n", "Now we can examine how much the relationship between the reference text (*Orlando furioso*) and the focus text (*Saggiatore*) have in common with the relationship between the reference text and the rest of the corpus. We will do this by looking at the ngrams in our focus text that are also in the reference text and checking to see if those ngrams are present in any other texts. The output will again be five matrices (one for each type of ngram). The rows will show an ngram, while the columns indicate texts. A 0 means that the ngram is not present in the text (but is present in the focus and reference texts). A 1 means that the ngram is also in the text (as well as the focus and reference texts). \n", "\n", "Depending on your interest in spreadsheets or different ngrams, you can customize the code below by changing the number to correspond to the length of the ngram you would like (as long as it is 1, 2, 3, 4, or 5). You can change the number as many times as you want; the output will have a filename that reflects the ngram you choice you made and will not overwrite the other ngram outputs." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ngram <- 1\n", "focusNgrams <- similarities[[focus]][[2]][[reference]][[2]][[ngram]][[1]]\n", "ngrams_l <- list()\n", "for(i in 1:length(similarities)){\n", " hits <- which(focusNgrams %in% similarities[[i]][[2]][[reference]][[2]][[ngram]][[1]])\n", " vessel <- rep(0, length(focusNgrams))\n", " vessel[hits] <- 1\n", " ngrams_l[[i]] <- vessel\n", "}\n", "ngrams_df <- do.call(cbind, ngrams_l)\n", "row.names(ngrams_df) <- focusNgrams\n", "colnames(ngrams_df) <- filenames\n", "fn2 <- paste(metadata[2], ngram, \"GramComparisonToCorpus.csv\", sep = \"\")\n", "write.csv(ngrams_df, file = fn2, row.names = TRUE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can quickly examine how to interpret the output:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\n", "
1547_ItaliaGoti_Trissino.txt1548_Fiamme_GiraldiCinzio.txt1551_PublicheOrationi_Vergerio_EEBO.txt
falsa110
moltitudine100
ualore110
atteso000
diletta111
restar110
\n" ], "text/latex": [ "\\begin{tabular}{r|lll}\n", " & 1547\\_ItaliaGoti\\_Trissino.txt & 1548\\_Fiamme\\_GiraldiCinzio.txt & 1551\\_PublicheOrationi\\_Vergerio\\_EEBO.txt\\\\\n", "\\hline\n", "\tfalsa & 1 & 1 & 0\\\\\n", "\tmoltitudine & 1 & 0 & 0\\\\\n", "\tualore & 1 & 1 & 0\\\\\n", "\tatteso & 0 & 0 & 0\\\\\n", "\tdiletta & 1 & 1 & 1\\\\\n", "\trestar & 1 & 1 & 0\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "| | 1547_ItaliaGoti_Trissino.txt | 1548_Fiamme_GiraldiCinzio.txt | 1551_PublicheOrationi_Vergerio_EEBO.txt | \n", "|---|---|---|---|---|---|\n", "| falsa | 1 | 1 | 0 | \n", "| moltitudine | 1 | 0 | 0 | \n", "| ualore | 1 | 1 | 0 | \n", "| atteso | 0 | 0 | 0 | \n", "| diletta | 1 | 1 | 1 | \n", "| restar | 1 | 1 | 0 | \n", "\n", "\n" ], "text/plain": [ " 1547_ItaliaGoti_Trissino.txt 1548_Fiamme_GiraldiCinzio.txt\n", "falsa 1 1 \n", "moltitudine 1 0 \n", "ualore 1 1 \n", "atteso 0 0 \n", "diletta 1 1 \n", "restar 1 1 \n", " 1551_PublicheOrationi_Vergerio_EEBO.txt\n", "falsa 0 \n", "moltitudine 0 \n", "ualore 0 \n", "atteso 0 \n", "diletta 1 \n", "restar 0 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ngrams_df[900:905, 10:12]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The word *falsa* appears in the *Orlando furioso*, the *Saggiatore*, and (according to this snippet) Trissino's *Italia liberata dai Goti* and Giraldi Cinzio's *Fiamme*. It does not apear in Vergerio's *Orationi*." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How the Reference Text Compares to the Corpus\n", "\n", "We can expand our vision of the reference text (the *Furioso*) to look at all of the matches, not just those that occur in the focus text (the *Saggiatore*). This information will be the stepping stone for then identifying the networks of ngrams and texts that use them.\n", "\n", "Just like above, you can choose your ngram value. Depending on the lexical richness of your reference text and the size of your corpus, this could produce an enormous file. Beware!" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ngram <- 4\n", "really_long <- as.character()\n", "for(i in 1:length(similarities)){\n", " if (i != reference){ # this focuses only on the ngrams that have a match in the corpus\n", " really_long <- c(really_long, similarities[[i]][[2]][[reference]][[2]][[ngram]][[1]])\n", " }\n", " \n", "}\n", "long <- unique(really_long)\n", "all_common <- list()\n", "for (i in 1:length(similarities)){\n", " blank <- rep(0, length(long))\n", " ones <- which(long %in% similarities[[i]][[2]][[reference]][[2]][[ngram]][[1]])\n", " blank[ones] <- 1\n", " all_common[[i]] <- blank\n", "}\n", "all_common_df <- do.call(cbind, all_common)\n", "row.names(all_common_df) <- long\n", "colnames(all_common_df) <- filenames\n", "all_common_df[,reference] <- 1 # the reference text is withheld from generating the full list of ngrams (since over 80% of the text might not have a single match other than itself)\n", "fn3 <- paste(\"All\", ngram, \"GramsInCommonWith\", metadata[2], \".csv\", sep = \"\")\n", "write.csv(all_common_df, file = fn3, row.names = TRUE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The interpretation of this spreadsheet is very similar to the previous example, except the filter of the focus text has been removed:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\t\n", "\t\n", "\t\n", "\t\n", "\t\n", "\n", "
1539_DialogoDonne_Piccolomini.txt1542_Rime_Sandoval.txt1545_Egle_GiraldiCinzio.txt
che di lui non000
da solo a solo000
de la terra che000
di qua et di000
di tutto il mondo001
\n" ], "text/latex": [ "\\begin{tabular}{r|lll}\n", " & 1539\\_DialogoDonne\\_Piccolomini.txt & 1542\\_Rime\\_Sandoval.txt & 1545\\_Egle\\_GiraldiCinzio.txt\\\\\n", "\\hline\n", "\tche di lui non & 0 & 0 & 0\\\\\n", "\tda solo a solo & 0 & 0 & 0\\\\\n", "\tde la terra che & 0 & 0 & 0\\\\\n", "\tdi qua et di & 0 & 0 & 0\\\\\n", "\tdi tutto il mondo & 0 & 0 & 1\\\\\n", "\\end{tabular}\n" ], "text/markdown": [ "\n", "| | 1539_DialogoDonne_Piccolomini.txt | 1542_Rime_Sandoval.txt | 1545_Egle_GiraldiCinzio.txt | \n", "|---|---|---|---|---|\n", "| che di lui non | 0 | 0 | 0 | \n", "| da solo a solo | 0 | 0 | 0 | \n", "| de la terra che | 0 | 0 | 0 | \n", "| di qua et di | 0 | 0 | 0 | \n", "| di tutto il mondo | 0 | 0 | 1 | \n", "\n", "\n" ], "text/plain": [ " 1539_DialogoDonne_Piccolomini.txt 1542_Rime_Sandoval.txt\n", "che di lui non 0 0 \n", "da solo a solo 0 0 \n", "de la terra che 0 0 \n", "di qua et di 0 0 \n", "di tutto il mondo 0 0 \n", " 1545_Egle_GiraldiCinzio.txt\n", "che di lui non 0 \n", "da solo a solo 0 \n", "de la terra che 0 \n", "di qua et di 0 \n", "di tutto il mondo 1 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "all_common_df[14:18, 7:9]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Both the reference text (*Orlando furioso*) and Trissino's *Italia liberata dai Goti* contain the fourgrams *che di lui non, da solo a solo,* and *de la terra che*, phrases which are not found in Sandoval's *Rime* or Giraldi Cinzio's *Egle*." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Identifying Communites of Ngram Use\n", "\n", "Because of the processing capacity required, the following code was run on a High Performance Computing (HPC). The matrix above is treated like a bimodal adjacency matrix: ngrams are connected to a text and are connected to each other by occuring together in a text. Matthew Lincoln's package \"projectoR\" contains a function to convert the bimodal edge information into a unimodal network of phrases that are interconnected. Please contact the author if this approach is of interest to you. Visualizing the results requires the software Gephi, for which another tutorial in a different medium is necessary." ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "R", "language": "R", "name": "ir" }, "language_info": { "codemirror_mode": "r", "file_extension": ".r", "mimetype": "text/x-r-source", "name": "R", "pygments_lexer": "r", "version": "3.4.0" } }, "nbformat": 4, "nbformat_minor": 0 }