You can copy and paste the code here to an R script. If you have installed R and Jupyter (via anaconda) you can download the .ipynb source file here. For Mac users, if you have not already done so, please follow the instructions to link your current version of R with Jupyter. All output will appear in the folder where this file is saved. It is presumed that your data is in a subfolder within this folder.
Please cite this code if you find it useful for your research: Hall, Crystal. "Galileo's Ariosto - Code Tutorial." Galileo's Library. http://www.bowdoin.edu/~chall/GalileosAriostoCodeTutorial.html August 31, 2017.
The code below loads R packages with functions necessary for the processes below. Included are also functions that I have written to streamline processing of files.
#install.packages("plyr") #Run these lines if you have not already installed these packages.
#install.packages("tau")
#install.pacakges("devtools")
#devtools::install_github(“mdlincoln/projectoR")
library(plyr)
library(tau)
require(projectoR)
clean <- function(filename){ # a function to read and process text files
scanned <- scan(filename, what = "character", sep = "\n", quote = "")
bp <- grep("Ogni uso commerciale è vietato", scanned) # Removes boiler-plate language from Biblioteca Italiana
if(length(bp) > 0){
scanned <- scanned[(bp+1):length(scanned)]
}
string <- paste(scanned, collapse = " ") # See Matt Jockers' Macroanalysis for an explanation of these lines
cleaned1 <- gsub("[[:digit:]]+", "", string)
cleaned2 <- noaccents(cleaned1)
lowered <- tolower(unlist(strsplit(cleaned2, "\\W")))
cleaned3 <- gsub("v", "u", lowered) #Standardizes orthography based on early modern print
cleaned4 <- gsub("j", "i", cleaned3)#Standardizes orthography based on early modern print
noblanks <- which(cleaned4 !="")
words <- cleaned4[noblanks] # vector of words in order of text
uniquewords <- unique(words) # vector of unique words in text
table <- sort(table(words), decreasing = TRUE) # table of word frequencies
bigrams <- textcnt(words, n = 2, method = "string")
bigrams_m <- as.matrix(bigrams[order(bigrams, decreasing = TRUE)])
bg_l <- list(bigrams, bigrams_m)
trigrams <- textcnt(words, n = 3, method = "string")
trigrams_m <- as.matrix(trigrams[order(trigrams, decreasing = TRUE)])
tg_l <- list(trigrams, trigrams_m)
fourgrams <- textcnt(words, n = 4, method = "string")
fourgrams_m <- as.matrix(fourgrams[order(fourgrams, decreasing = TRUE)])
frg_l <- list(fourgrams, fourgrams_m)
fivegrams <- textcnt(words, n = 5, method = "string")
fivegrams_m <- as.matrix(fivegrams[order(fivegrams, decreasing = TRUE)])
fvg_l <- list(fivegrams, fivegrams_m)
data <- list(filename, words, uniquewords, table, bigrams = bg_l, trigrams = tg_l, fourgrams = frg_l, fivegrams = fvg_l)
return(data)
}
noaccents <- function(string){ # a function to standardize orthography due to inconsistencies in early modern print
string <- gsub("à", "a", string)
string <- gsub("á", "a", string)
string <- gsub("è", "e", string)
string <- gsub("é", "e", string)
string <- gsub("í", "i", string)
string <- gsub("ì", "i", string)
string <- gsub("ò", "o", string)
string <- gsub("ó", "o", string)
string <- gsub("ù", "u", string)
string <- gsub("ú", "u", string)
return(string)
}
The code below requires input from you: the path to the folder where your plain text (UTF-8) files are saved. Change the definition of path and run the code cell. You may notice that the article only talks about 47 documents, but there are 49 in the example. The "extra" two texts are the earlier versions of the Orlando furioso, included as controls for this project.
path <- "Data/OriginalEditions" # Change the information between quotation marks to match your file structure
filenames <- dir(path) # Do not edit the lines below
docs <- list()
for(i in 1:length(filenames)){
docs[[i]] <- clean(paste(path, filenames[i], sep = "/"))
}
print(paste("Your folder contains ", length(docs), " files.", sep = ""))
The objects docs (created above) is like a filing cabinet, in this case with 49 drawers, one for each text. Each drawer contains 8 hanging file folders with different information:
In the article, the reference file is Ariosto's Orlando furioso, the text against which all others will be compared. The focus file is Galileo's Saggiatore, the text whose style is of most interest.
Run the code cell below to determine the number associated with your reference and focus texts. In this example the Orlando furioso is 3 and the Saggiatore is 44.
print(filenames)
The code below adds a hanging file folder to each drawer with comparison information to your reference text. This is a great moment to define your reference and focus texts. Then run the code.
reference <- 3 # Change to the number that matches your reference text filename
focus <- 44 # Change to the number that matches your focus text filename
compareToReference <- function(text1_l, text2_l){ # a function to facilitate comparison (can be scaled up for multiple reference documents)
comparison <- list(intersect(text1_l[[2]], text2_l[[2]]))
return(comparison)
}
for(i in 1:length(docs)){
docs[[i]][[9]] <- compareToReference(docs[[i]], docs[[reference]])
}
Now that we have compared every text to the reference text (the Orlando furioso in the article), we want to evaluate similarities and differences between these texts' relationships to that reference file.
The code below is a bit repetitive (ie: not designed by a software engineer), but it is hopefully transparent. It creates a new filing cabinet, still with 49 drawers (for this example). Each drawer now has 49 hanging file folders, one for every text to which it is compared. Each hanging file folder will have five manila folders with three pieces of information in each:
For example: Drawer 44 [[Galileo's Saggiatore]]:
> Hanging File Folder 3 [[Ariosto' *Orlando furioso* 1532 ed.]]:
> Manila Folder 1: unigram comparison information
> percentage of all words in *Saggiatore* that are in the *Furioso*
> percentage of word types in *Saggiatore* that are in the *Furioso*
> vector of which words in *Saggiatore* are in the *Furioso*
> Manila Folder 2: bigram comparison information
> vector of which bigrams in *Saggiatore* are in the *Furioso*
> table of frequencies of bigrams in *Saggiatore* that are in the *Furioso*
> percentage of all bigrams in *Saggiatore* that are in the *Furioso*
> Manila Folder 3: trigram comparison information
> vector of which trigrams in *Saggiatore* are in the *Furioso*
> table of frequencies of trigrams in *Saggiatore* that are in the *Furioso*
> percentage of all trigrams in *Saggiatore* that are in the *Furioso*
> Manila Folder 4: fourgram comparison information
> vector of which fourgrams in *Saggiatore* are in the *Furioso*
> table of frequencies of fourgrams in *Saggiatore* that are in the *Furioso*
> percentage of all fourgrams in *Saggiatore* that are in the *Furioso*
> Manila Folder 5: fivegram comparison information
> vector of which fivegrams in *Saggiatore* are in the *Furioso*
> table of frequencies of fivegrams in *Saggiatore* that are in the *Furioso*
> percentage of all fivegrams in *Saggiatore* that are in the *Furioso*
For the situation of a text being compared to itself, the information is populated with 100% matches.
# Just run this cell. No need to edit.
similarities <- list()
for(i in 1:length(docs)){
vec <- docs[[i]][[2]]
file1 <- docs[[i]][[1]]
bigr <- row.names(docs[[i]][[5]][[2]])
trigr <- row.names(docs[[i]][[6]][[2]])
frgr <- row.names(docs[[i]][[7]][[2]])
fvgr <- row.names(docs[[i]][[8]][[2]])
sim <- list()
for(x in 1:length(docs)){
file2 <- docs[[x]][[1]]
mini_sim <- list()
if(i != x){
comp <- intersect(vec, docs[[x]][[2]])
token_perc <- 100*length(comp)/length(docs[[i]][[3]])
type_perc <- 100*length(which(vec %in% comp))/length(vec)
mini_sim[[1]] <- list(comp, token_perc, type_perc)
bi_comp <- intersect(bigr, row.names(docs[[x]][[5]][[2]]))
bi_comp_m <- docs[[i]][[5]][[2]][which(bigr %in% bi_comp),]
bi_comp_p <- 100*length(bi_comp)/length(bigr)
mini_sim[[2]] <- list(bi_comp, bi_comp_m, bi_comp_p)
tri_comp <- intersect(trigr, row.names(docs[[x]][[6]][[2]]))
tri_comp_m <- docs[[i]][[6]][[2]][which(trigr %in% tri_comp),]
tri_comp_p <- 100*length(tri_comp)/length(trigr)
mini_sim[[3]] <- list(tri_comp, tri_comp_m, tri_comp_p)
fr_comp <- intersect(frgr, row.names(docs[[x]][[7]][[2]]))
fr_comp_m <- docs[[i]][[7]][[2]][which(frgr %in% fr_comp),]
fr_comp_p <- 100*length(fr_comp)/length(frgr)
mini_sim[[4]] <- list(fr_comp, fr_comp_m, fr_comp_p)
fv_comp <- intersect(fvgr, row.names(docs[[x]][[8]][[2]]))
fv_comp_m <- docs[[i]][[8]][[2]][which(fvgr %in% fv_comp),]
fv_comp_p <- 100*length(fv_comp)/length(fvgr)
mini_sim[[5]] <- list(fv_comp, fv_comp_m, fv_comp_p)
}
else { # to account for a text being compared to itself
mini_sim[[1]] <- list(100, 100, rep("same", length(vec)))
mini_sim[[2]] <- list(rep("same", length(bigr)), "Same", 100)
mini_sim[[3]] <- list(rep("same", length(trigr)), "Same", 100)
mini_sim[[4]] <- list(rep("same", length(frgr)), "Same", 100)
mini_sim[[5]] <- list(rep("same", length(fvgr)), "Same", 100)
}
sim[[x]] <- list(file2, mini_sim)
}
similarities[[i]] <- list(file1, sim)
}
The object created above, similarities, is a powerful resource. Each drawer has full comparison information - to every other text in the corpus, not just the reference text. This makes it easier to follow leads and respond to surprises in the results below.
The code cell below creates 5 matrices with as many rows and columns as there are texts. In the case of our example, 49 x 49. Each matrix shows the percentage similarity that one text shares with another (without considering the reference text). After running the code, a snippet of a matrix will be displayed to show how to interpret the results.
tokens <- list()
types <- list()
bigrams <- list()
trigrams <- list()
fourgrams <- list()
fivegrams <- list()
for (i in 1:length(similarities)){
examine <- similarities[[i]][[2]]
token_percentage <- as.numeric()
type_percentage <- as.numeric()
bigram_count <- as.numeric()
trigram_count <- as.numeric()
fourgram_count <- as.numeric()
fivegram_count <- as.numeric()
for(x in 1:length(examine)){
token_percentage[x] <- examine[[x]][[2]][[1]][[2]]
type_percentage[x] <- examine[[x]][[2]][[1]][3]
bigram_count[x] <- examine[[x]][[2]][[2]][[3]]
trigram_count[x] <- examine[[x]][[2]][[3]][[3]]
fourgram_count[x] <- examine[[x]][[2]][[4]][[3]]
fivegram_count[x] <- examine[[x]][[2]][[5]][[3]]
}
tokens[[i]] <- token_percentage
types[[i]] <- type_percentage
bigrams[[i]] <- bigram_count
trigrams[[i]] <- trigram_count
fourgrams[[i]] <- fourgram_count
fivegrams[[i]] <- fivegram_count
}
tokens_df <- do.call(rbind, tokens)
colnames(tokens_df) <- filenames
row.names(tokens_df) <- filenames
#min(tokens_df) # To check the range of similarities - quite large!
#max(tokens_df[which(tokens_df !=100)])
types_df <- do.call(rbind, types)
colnames(types_df) <- filenames
row.names(types_df) <- filenames
bigrams_df <- do.call(rbind, bigrams)
colnames(bigrams_df) <- filenames
row.names(bigrams_df) <- filenames
trigrams_df <- do.call(rbind, trigrams)
colnames(trigrams_df) <- filenames
row.names(trigrams_df) <- filenames
fourgrams_df <- do.call(rbind, fourgrams)
colnames(fourgrams_df) <- filenames
row.names(fourgrams_df) <- filenames
fivegrams_df <- do.call(rbind, fivegrams)
colnames(fivegrams_df) <- filenames
row.names(fivegrams_df) <- filenames
ngrams <- list(tokens_df, types_df, bigrams_df, trigrams_df, fourgrams_df, fivegrams_df)
fourgrams_df[1:10, 1:3]
The matrix is read by row, from left column to right. We can already see the 100 (%) that shows that a text has 100% fourgrams in common with itself. For the other values, they can be read as follows:
Even with this small snippet from just one of the matrices, we see the range of similarities between texts. If you want to save each matrix as a spreadsheet, run the code cell below.
write.csv(types_df, "CorpusWordTypeSimilarity.csv", row.names = TRUE)
write.csv(bigrams_df, "CorpusBigramSimilarity.csv", row.names = TRUE)
write.csv(trigrams_df, "CorpusTrigramSimilarity.csv", row.names = TRUE)
write.csv(fourgrams_df, "CorpusFourgramSimilarity.csv", row.names = TRUE)
write.csv(fivegrams_df, "CorpusFivegramSimilarity.csv", row.names = TRUE)
Now we are ready to examine the focus text, in this case Galileo's Saggiatore. We can extract the Saggiatore row from each matrix. That row number is the same as the number of the filename, stored as the variable focus. Your results will be saved in the folder where this .ipynb file is saved. Look for the "SimilarityToCorpus.csv" filename. You can open it with Excel.
focusToCorpus <- cbind(tokens_df[focus,], types_df[focus,], bigrams_df[focus,], trigrams_df[focus,], fourgrams_df[focus,], fivegrams_df[focus,])
metadata <- unlist(strsplit(filenames[focus], "_"))
fn <- paste(metadata[2], "SimilarityToCorpus.csv", sep = "")
write.csv(focusToCorpus, file = fn, row.names = TRUE)
Now we can examine how much the relationship between the reference text (Orlando furioso) and the focus text (Saggiatore) have in common with the relationship between the reference text and the rest of the corpus. We will do this by looking at the ngrams in our focus text that are also in the reference text and checking to see if those ngrams are present in any other texts. The output will again be five matrices (one for each type of ngram). The rows will show an ngram, while the columns indicate texts. A 0 means that the ngram is not present in the text (but is present in the focus and reference texts). A 1 means that the ngram is also in the text (as well as the focus and reference texts).
Depending on your interest in spreadsheets or different ngrams, you can customize the code below by changing the number to correspond to the length of the ngram you would like (as long as it is 1, 2, 3, 4, or 5). You can change the number as many times as you want; the output will have a filename that reflects the ngram you choice you made and will not overwrite the other ngram outputs.
ngram <- 1
focusNgrams <- similarities[[focus]][[2]][[reference]][[2]][[ngram]][[1]]
ngrams_l <- list()
for(i in 1:length(similarities)){
hits <- which(focusNgrams %in% similarities[[i]][[2]][[reference]][[2]][[ngram]][[1]])
vessel <- rep(0, length(focusNgrams))
vessel[hits] <- 1
ngrams_l[[i]] <- vessel
}
ngrams_df <- do.call(cbind, ngrams_l)
row.names(ngrams_df) <- focusNgrams
colnames(ngrams_df) <- filenames
fn2 <- paste(metadata[2], ngram, "GramComparisonToCorpus.csv", sep = "")
write.csv(ngrams_df, file = fn2, row.names = TRUE)
We can quickly examine how to interpret the output:
ngrams_df[900:905, 10:12]
The word falsa appears in the Orlando furioso, the Saggiatore, and (according to this snippet) Trissino's Italia liberata dai Goti and Giraldi Cinzio's Fiamme. It does not apear in Vergerio's Orationi.
We can expand our vision of the reference text (the Furioso) to look at all of the matches, not just those that occur in the focus text (the Saggiatore). This information will be the stepping stone for then identifying the networks of ngrams and texts that use them.
Just like above, you can choose your ngram value. Depending on the lexical richness of your reference text and the size of your corpus, this could produce an enormous file. Beware!
ngram <- 4
really_long <- as.character()
for(i in 1:length(similarities)){
if (i != reference){ # this focuses only on the ngrams that have a match in the corpus
really_long <- c(really_long, similarities[[i]][[2]][[reference]][[2]][[ngram]][[1]])
}
}
long <- unique(really_long)
all_common <- list()
for (i in 1:length(similarities)){
blank <- rep(0, length(long))
ones <- which(long %in% similarities[[i]][[2]][[reference]][[2]][[ngram]][[1]])
blank[ones] <- 1
all_common[[i]] <- blank
}
all_common_df <- do.call(cbind, all_common)
row.names(all_common_df) <- long
colnames(all_common_df) <- filenames
all_common_df[,reference] <- 1 # the reference text is withheld from generating the full list of ngrams (since over 80% of the text might not have a single match other than itself)
fn3 <- paste("All", ngram, "GramsInCommonWith", metadata[2], ".csv", sep = "")
write.csv(all_common_df, file = fn3, row.names = TRUE)
The interpretation of this spreadsheet is very similar to the previous example, except the filter of the focus text has been removed:
all_common_df[14:18, 7:9]
Both the reference text (Orlando furioso) and Trissino's Italia liberata dai Goti contain the fourgrams che di lui non, da solo a solo, and de la terra che, phrases which are not found in Sandoval's Rime or Giraldi Cinzio's Egle.
Because of the processing capacity required, the following code was run on a High Performance Computing (HPC). The matrix above is treated like a bimodal adjacency matrix: ngrams are connected to a text and are connected to each other by occuring together in a text. Matthew Lincoln's package "projectoR" contains a function to convert the bimodal edge information into a unimodal network of phrases that are interconnected. Please contact the author if this approach is of interest to you. Visualizing the results requires the software Gephi, for which another tutorial in a different medium is necessary.