My kid and I were listening to a podcast about Jack the Ripper. We started talking, and I mentioned how newspapers would reprint stories from each other. This led to us developing an interesting question: did our local newspaper print anything about the Whitechapel murders, and if so, what would that have meant to the people of this community? I filed the thoughts away for future reference, but then later saw some posts about Anastasia Salter’s session at the MLA on agentic coding for the humanities (January 2026). I looked up her course materials, and thought I would follow along with them, using their guidance for prompting Claude Code with our initial question.
Claude was great and making a nice one-page html visualization from the eventual analysis of the OCR. Anastasia’s directions were clear and we had fun. But the tricky bit – ’twas ever thus – was the bloody OCR. Our version used pytesseract. Vision models from Google etc do extremely good OCR, if you’ve got an api key and are paying for it. I wanted to keep things on my local machine though. So I futzed with pytesseract, paddleocr, and surya. Pytesseract is fast but all over the map; Surya is pretty good but can get flumoxed; paddleocr just freezes my damn machine up. But the hardest bit was just getting the newspapers chopped up into small enough bits so that I could process them.
The Shawville Equity was scanned some time ago by the provincial archives, the BANQ; here’s the very first issue from 1883. There is a text layer in the pdf from the BANQ, but it looks to have been done through an automatic process without human intervention, so images are sometimes askew and the underlying text is often very very poor indeed (look up the work of Ian Milligan on the consequences for research of bad newspaper OCR).
The process that ended up working best involved counting on the Equity to maintain its 5-column layout and horizontal spacers. The Equity, as befits a publication with over 140 years of history, has gone through some layout changes over the years.
That slight skew drives me nuts.
Anyway, we preprocessed the paper by trying to identify those vertical lines and horizontal spacers then chopping the image up accordingly. Because the OCR stuff memory-wise works better on smaller images, we also chopped up anything longer than 2000 px. The coordinates are all mapped out in the output json so that everything can be stitched back together again.
Oh, and yeah: The Shawville Equity did publish stories about the Ripper. And at the same time, they printed a bunch of stuff about the Burke and Hare murders too, for good measure. They published a few stories shortly after the murders in Whitechapel started, and then returned a few years later for good measure. So still an interesting question to explore…
I need to workshop my titles more. But anyway: this post reflects on teaching the history of the internet to a class of 160 first year students in a world where generative AI shabbiness is pushed on them and a perfectly rationale way to deal with the myriad pressures and bad choices of being a student is to go ahead and use it. What’s a prof to do?
The first thing I tried to do was use the metaphor of going to the gym: you go to exercise your body and get stronger. If there was a machine that lifted weights for you, could you go, turn it on, point to it and say, ‘look, weights have been lifted! I have therefore exercised!’ No, you cannot. But – the same error is made in university classrooms all the time. Look! An essay has been written! Give me my grade! And I don’t need to spill any more photons, bits, or ink over the instrumentalization of higher education that has led us here. Instead, here’s how I tried to deal with it this term. And no, I didn’t put any trojan horse prompts into my assignments.
Instead, I chose to focus on reading and notemaking.
By hand.
I asked all students to get a little paper notebook. I showed them the readings; I showed them hypothes.is; we talked about how to read and what to pay attention to (“don’t read it through like a novel! Read like a predator! Go to where the game is!” etc etc). Then in class, I asked them to do two things for a given reading: write a rhetorical précis (using a model developed by historian Chad Black) and a research memo-to-self that pulls together one’s observations and annotations. They had to do this cold. In my lecture hall. No computer. No phone. No notes. (Students with accommodations: I made accommodations.)
I also told them: we’re doing this multiple times throughout the semester. You’re going to have off days. I’ll take your best 3 of 4 examples for grading. And we graded at first for the format, for the shape of what we were after, and then started pushing them towards deeper engagement with the content. They were always encouraged to filter these ideas through my lectures too. A pretty good example (though not perfect) of what we were after is this composite of a couple of student’s responses, after reading a longer blog post by Doctorow on Enshittification:
PRECIS MAJOR CLAIM: Doctorow argues in his McLuhan lecture on enshittification (2024) that platforms degrade through a three-stage process of user exploitation, business exploitation, and shareholder extraction leading to a world of digital decay known as the enshittocene. HOW: Doctorow develops this argument through a detailed case study of Facebook, tracing the three stages of enshittification (from user surplus to business surplus to shareholder surplus) while systematically dismantling the historical constraints that once prevented such decay, and showing how the erosion of competition, regulation, self-help, and labor power enabled the collapse of digital trust. PURPOSE: The author’s apparent purpose is to diagnose the systemic decay of digital platforms and show how it spreads across industries in order to empower users, workers, and policymakers to reverse the trend and build a more equitable, open digital world.
MEMO INITIAL OBSERVATION: WHAT IF there’s a connection with the Bory piece; what if tech ceos believe themselves to be the hero of the journey? This’d create a cultural narrative in which enshittification is not a failure, but a necessary stage of progress. THEN this mythos might normalize the extraction of surplus from users, workers, and business partners, treating exploitation as a form of “service” or “evolution”? #to-investigate #possible-thesis KEY: The reading matters because it reframes enshittification not as a technical process, but as a cultural one. #cultural-processes MY CONTRIBUTION: Doctorow’s framework shows how platforms collapse through a three-stage exploitation process: user → business → shareholder. There’s a connection here with Bory’s critique, which reveals that this process is culturally enabled by a narrative in which the founder is the hero, and the platform is the vehicle of a moral mission. When founders say, “I created this to serve humanity,” they are not just describing a product; they are enacting a myth. And when that myth is accepted, enshittification becomes not just a crisis, but a natural consequence of leadership.
These for the most part got better as the term went on. However, it took us longer to grade them than I would’ve liked. I transformed the final exercise from another round of precis/memo combos (we’d do 2 per session) to one last class workshop on ‘how to write with these things’ (where grading was pass/fail did-you-do-the-thing?-full-points).
The idea is, a student would look at these precis/memos and think to themselves, ‘what’s the story here? How do these observations speak to one another?’ How you look at things – ie, historical theory – guides your attention to some ideas rather than others. It being the last week of term, I wanted to do something fun first to get them in the mood, so today we did a kind of team debate-cum-tournament style sort of thing, where suggestions for the most important people/ideas/technologies of the history of the internet were gathered. These were arranged into a bracket. For each round in the bracket, I suggested a different lens through which the disputants were to make their argument for the greater importance of their person/idea/technology. Winners were chosen through applause from the class (y’know, I forget the winner? But I think it was between the ENIAC women and Vannevar Bush). And do you know, students were drawing some pretty nifty arguments from their precis/memos to do this, bouncing ideas off one another. It was neat to see! And difficult: the power went off during class and we did this via the blackboard and cellphone flashlights (internal lecture theatre without windows).
On wednesday this week, the idea is the students will have their precis/memo combos ready to hand. I’ll say, ‘let’s assume we’re looking at the history of the internet through a social history lens. What have you got that speaks to that or could be informed from that?’ The idea is, they’ll make a list (with page & paragraph numbers, since they’ll have numbered the pages in their booklets) of these interesting observations. We’ll do some think-pare-share: show your neighbour what you’ve got. Then, I’ll have them create an outline with each element they have, beginning with: where’s the question here? They’ll reorder their useful observations such that there appears to be an emergent story or argument. At that point, I’ll ask them to think about ‘what is missing? What pieces of connective tissue do you have to write?’ … and they’ll then make quick notes about what they’d need to look into or write to make the tissue of observations whole.
This will be what they need to do for the final exam, so I’ll give them the exam question on Friday (in the exam room: no aide-memoire. They’ll have had to work through their materials before going in). I’m feeling pretty good about this.
And that’s how I’ve moved through reading -> note making -> thinking -> writing in an age of generative AI.
Yes, this was a lot of work. And I find language models interesting to explore. But that doesn’t mean I think they have any business in a first year class.
Post script: Because I’m interested in code, and in the way generative AI as an average machine spit out things that work (for a given value of work) I also like to try cooking up small one-page html tools since I know precious little about javascript, react, etc. I built a little outliner tool that I will use in class on wednesday to explain the concept that I am after, on the big screen. And maybe some of my gang will find it useful. You can give it a whirl here: https://shawngraham.github.io/outliner/ and you can grab the html from here: https://github.com/shawngraham/outliner/.
…in which I try to retrain/fine-tune a spaCy model on Latin inscriptions
There is a lot of Roman epigraphic data online; the EDH is a great source for this. But none of the databases (at least, the ones that I have looked at) seem to provide a version with structured demographic or onomastic or whatever data derived from the inscriptions. Presumably that data is out there – epidoc formatted xml would have what I’m after, I should think – but I thought, what the hell: how hard could it be to train a spaCy model to read Roman inscriptions, which are after all famously formulaic? They don’t call it ‘the epigraphic habit‘ for nothing, right? If the average Roman could read them and understand – with their relatively low level of functional literacy – then a machine should be able to do this? …Right? Reader, it was harder than I thought.
Be Warned: My epigraphic experience is limited to the scintillating world of stamped Roman bricks. And it’s been over twenty years since I really futzed in any meaningful way with Latin. Caveat lector.
The idea is therefore:
download real data
annotate the data with the start and end positions of the different kinds of structured data that I am after
enhance this data with synthetic examples so that I get enough coverage of the different kinds of elements (the origin of the deceased in a funerary inscription is not as common as listing their cognomen, right? So training on exclusively real data would overfit on some things and miss others, right? That was my logic).
harmonize the synethic data with the real data so that any annotation label glitches in step 2 get sorted out
fix alignments so that annotations do not overlap
train.
(as an aside, how the f*n hell do you get the Gutenberg editor to give you a numbered list? THIS kind of shit is why I don’t blog very much any more: it’s such a bloody pain in the ass!)
Yes, I had help from Claude Haiku 4.5 and Gemini 3 Pro Preview for the fiddly bits. I downloaded data in two tranches. The first batch I tried downloading via the API and so got the inscriptions but didn’t realize I was leaving a lot of useful metadata behind – the second tranche I got from the EDH data dump website itself, where some of the metadata was provided by virtue of the column headings. I dropped the first tranche through a local LLM (Qwen 3) with instructions on returning jsonl data with annotations… that was an enormous pain in the arse and ultimately largely a waste of time. But I did get around 450 lines of stuff that was annotated sufficiently I could use it. The second tranche was easier- I downloaded the dump, filtered columns using Excel so that I got around 750 rows where an inscription had metadata for each column of interest. That was a reduction of tens of thousands of rows of data to just under a thousand (!). I converted each row to jsonl.
This next bit was where I had the most help from the big-ass LLMs. I devised the logic for an inscription generator that would use the Roman’s own epigraphic habits as rules for generation. It is probabilistic and is pretty good, for the most part, at creating legible inscriptions (I am reminded of John Clarke’s 19th century Eureka machine for generating Latin hexametre verse). Then, looking at what kinds of things my real data contained, I tweaked the generator so that it would produce examples to fill the gaps. I ended up with a ratio of about 2 synthetic examples for every 1 real example.
After that, it was just a matter of training.
LABEL
PREC
REC
F1
AGE
0.96
0.96
0.96
COGNOMEN
0.88
0.78
0.83
FORMULA
0.94
0.68
0.79
MILITARY_UNIT
0.88
0.73
0.80
NOMEN
0.75
0.76
0.75
OCCUPATION
0.84
0.63
0.72
ORIGO
0.73
0.46
0.57
PRAENOMEN
0.78
0.84
0.80
RELATIONSHIP
0.97
0.79
0.87
TRIBE
0.75
0.73
0.74
If you look at the ‘train your model’ jupyternotebook (in the repo), you’ll see where I ran the model against the testing split (dev.jsonl); these were the metrics:
Total Predictions: 4426
Total Gold Labels: 5034
Correct (Exact Match): 3847
Precision: 0.869
Recall: 0.764
F1 Score: 0.813
Now – more real well-annotated examples, complemented by synthetic ones to fill the gap, might lead to higher scores, but the real proof is in the pudding, not in these test-case scenarios. It might be that I’ve got a model here that’s really good at… my bespoke admixture of read/synthetic. It might (probably will?) fall down when thrown against your data. But that’s what makes this fun. So… give it a whirl on your own epigraphic data, see what percolates out? Feel free to modify, make better.
One day, I decided that my story-loving kid and I would play an rpg together. We settled on Ken Lowery’s ‘Lighthouse at the End of the World‘. This is a solo journaling RPG: you read the backstory, you throw the die, you pull some cards and consult the game for what those cards will prompt you to think about. And then you write. My kid and I enjoyed a quiet morning, pulling the cards and writing two vastly different stories that somehow seemed like a shattered crystal reflecting some deeper reality. We’d pull a card, write for a bit, then read each other what we’d written. We’re nerdy like that.
But! What a powerful way to think about writing, constraints, and world-building! Colleen Morgan has an excellent piece on world-building for archaeologists. We’re not storytellers; we build worlds, and if we build worlds well, others can tell their stories, other stories can be told. I went exploring, and discovered there is a myriad of systems out there to build these solo journaling rpgs. I’m attracted to the ‘solo’ part, because I teach, and I want students doing things on their own. At least at first. I’m particularly attracted to the ‘Wretched & Alone’ system for crafting these experiences, by Mat Sanders in consultation with Chris Bissette ( who wrote the first ‘Wretched’ game. The system is explained here). The main thing about this system is that they are “about struggling in the face of insurmountable odds to survive, or to achieve something important.” I’m also coming to these games with a back ground in agent based simulation, and some faffing about with board games and video games. I like the idea of changing the rules to better capture some truth about the past that you’re trying to communicate.
I set out to write my own Wretched-style game. My first attempt was based on the Franklin Expedition, and how the demise of those men is captured in Inuit oral tradition. I wrote my card prompts, I thought about how the game might end, I imagined what their colonial mindset when confronted with survival and the obvious ability of the Inuit to live in the North might do, and I used canva to produce a kind of ‘zine of my rules (I recognize now that there’s a flub with my ‘salvation’ mechanism which I need to get around to fixing some day). I so much enjoyed setting up the pieces this way that I spent quite a lot of time during my sabbatical writing a few more such games – one set at the interface of antiquarianism/nascent professionalization of archaeology in late Victorian/Edwardian England, another featuring the Teutoberg Forest disaster, and one about being a new grad student in a DH program.
One feature of ‘Wretched’ style games is the use of a block tower and the pulling of blocks to ramp up the tension and dread. It’s not the sort of thing one always has handy. I started mucking about building a writing pad in an html page where that kind of tower simulation could be accessed by mouse-click. Turns out, there’s a lot of discussion online about how to do this with dice rolls so I figured out how to implement that. At around this time I was also doing a lot of ‘homecooked meals‘, building little things using some coding help, for my students and with my students. I sketched out what I was after, got the html and css skeleton working, and eventually surfaced with an online gamepad for solo journaling rpgs.
There I left things for a while, until this week, when I grew unhappy with having to flip between the pdf with card prompts and the gamepad. Why not merge them, and have a unique html page for each game? After much faffing, I emerged with The Cave and Whoever Finds This Paper. You can see that the second one is very much the first one, but reskinned; both have arrays of json data in them to keep the cards straight and the actions and consequences and so on. So why not make that modular?
This is where I turned to Claude Code’s new sandbox feature. I gave it the original pdfs, my merged one-page gamepad/prompt html, and instructed it to turn the html into a template and move the game and prompts and styling to separate yaml, the idea being to have a static site generator just for Wretched & Alone style games. It took a while, with Claude suggesting/making changes, me tweaking, rolling things slightly differently, but… I now have something that I can use with my students. They can focus on the writing, the thinking through of how the prompts leave some things unspoken while directing attention elsewhere (what, in Terry Pratchett’s Discworld, might be called ‘storytelling with hole’), and use the generator to create the html. Students more familiar with css and js can extend things just by mucking about with the theme.yaml.
and you’re ready to go. There’s an example game included (not a great game, but just one to show you the ropes) that is derived from The Cave. Run the generator with:
python cli/wretched.py build example-game
and it will stitch together the necessary files in the example-game folder and output a single html page that you can then put online, or run locally. There’s even a mobile theme! Copy the theme you want from the ‘themes’ folder into your example-game folder, and rename it ‘theme.yaml’. If you don’t like typing python cli/wretched.py every time, there is also a bash script wretched that you can use (once you give permission for it to run; on a mac that’d be something like chmod +x wretched and then you can run it ./wretched build example-game.
I don’t think I’ll be doing much more development with this for a while; it’s been a lot of fun, but there are so many other fires I ought to be putting out right now. So take a copy of it, expand it, enhance it, play with it, use it, write great stories with hole.
I wrote a short, incomplete, opinionated introduction to generative AI for history and archaeology students. The work is what I wish my students had before they came to my class last September, and is built from my teaching notes and remarks that I gave at a few public venues last year. Here’s what the publisher has to say about it:
“The buzz surrounding AI these days is nearly deafening and hardly a week goes by without some breathless utterance about the future of AI. One day AI is eliminating the need for teachers, the next it is streamlining our entire consumer economy, revolutionizing warfare, and turning us all into mindless drones.
Shawn Graham’s new book will not help us predict the future, but it will cut through some of the hype and show how AI technology can help us understand the past in new ways. Practical Necromancy starts with a thoughtful guide to the history and inner workings of AI. The second part of the book offers some concrete exercises well-suited for students and curious faculty alike. The final section offers some wisdom to administrators who like the rest of us struggle with how to use AI most effectively on their campuses.
The observations and exercises offered in this book continue in tradition of Graham’s 2019 book Failing Gloriously and Other Essays. Failing Gloriously has become one of the most downloaded books ever published by The Digital Press with nearly 5000 downloads since it first appeared. Practical Necromancy continues to advocate for the fearless experimentation central to Failing Gloriously. It is only through a fearlessness approach to AI that we can “break them, push them, prod them, make them give up the ghosts in their data.”
I don’t think I’ve seen anyone do this yet: drop a Harris Matrix through Retrieval Augmented Generation. That is to say, I think I have a web toy here with a nifty feature. The ability to ask questions of our data using our everyday language is the StarTrek dream, isn’t it. ‘Computer, when and how was this site abandoned?’ ‘Computer, identify the logical inconsistencies in this stratigraphy’. ‘Computer, what is the relationship between phase 4 and the area B group?’ Things like that.
I’m a long way removed from day to day field work or analysis. But I’m teaching a course in the fall (asynchronous, online) that aims to introduce history students (nb, not archaeology: we don’t have a program or department) to some of the digital work involved in making sense of archaeological materials. I wanted to give them the experience of trying to understand stratigraphy from a section drawing, but I also didn’t want to pay for a license for existing Harris Matrix software, or hit them over the head with the full complexity of the exercise. I set about to create the individual pieces I would need, wired up in a single html page (see previous exercises in ‘homecook history‘). I shared the result on Mastodon, and had a few feature requests, and now I think I have a tool/toy that’s actually quite good. You can try it out here: https://shawngraham.github.io/homecooked-history/hm-generator-site/enhanced.html . Click to create a context. Drag and drop to set up stratigraphic relationships. Click to edit and add context metadata. There’s some validation going on under the hood for chronology etc. It still has its kinks, but it will serve my purpose. It also exports the Harris Matrix you build as csv (and a nice svg too if you want).
In my class, I am also addressing various kinds of machine learning things that archaeologists do. In order not to overwhelm everyone, this is mostly related to image processing. But we do consider image similarity through vectors and embeddings. So… why not express the information about each context as an embedding? And then, having done that, let’s do some retrieval-augmented generation. Then we can use an LLM to express our query in the same embedding space, find the contexts that are closest to it, and then constrain the LLM to generate a response using only the information from those contexts.
This notebook does just that https://gist.github.com/shawngraham/bd845937d8f9789dd282ea80b0a03e4e . It uses Simon Willison’s LLM to express the contexts in the embedding space and to handle the generation. Load it into Google Colab to give it a whirl. It has some demo data, but you can load your own CSV in there too (and if you want, you can use my Harris Matrix toy/tool to create a matrix and export it to the csv format the notebook is expecting).
QUERY: What evidence exists for domestic activities across different phases?
==========================================================
The evidence from Context 1 (ID: C006) and Context 2 (ID: C007) demonstrates that domestic activities, specifically those related to hearth use and food preparation, were present during the Medieval phase. The stratigraphic relationships and dating evidence support the interpretation that these activities were part of the site's use during this period. The transition into the Post-Medieval phase, marked by Context 3 (ID: C008), indicates changes at the site, which may reflect alterations in domestic activities or the site's purpose, but direct evidence for domestic activities during this later phase is not provided within the given archaeological contexts.
SOURCES: C006, C007, C008
RETRIEVED CONTEXTS:
1. Context C006 (Similarity: 0.246)
Type: Feature
Description: Stone-lined hearth with evidence of burning and ash deposits...
Dating: 1250.0 AD to 1350.0 AD
Phase: Medieval
Relationship: built into C005
2. Context C007 (Similarity: 0.243)
Type: Fill
Description: Ash and charcoal fill of hearth C006, rich in pottery and animal bone...
Dating: 1250.0 AD to 1350.0 AD
Phase: Medieval
Relationship: fills C006
3. Context C008 (Similarity: 0.218)
Type: Layer
Description: Post-medieval demolition layer with brick and tile rubble...
Dating: 1600.0 AD to 1700.0 AD
Phase: Post-Medieval
Relationship: overlies C005
I’ve made some bug fixes and some enhancements to my personal knowledge management plugin for JupyterLab (and thus, JupyterLab Desktop). The goal is to enable personal note making around Jupyter notebooks, with bi-directional linking, discovery, and all the standard #pkm things we’ve come to expect. While my semantic versioning skills are a bit suspect, there’s now a version available through pypi that works pretty darned well and will be what I develop one of my autumn asynchronous courses around. The extension handles the functionality; I create a folder with markdown and ipynb files for students to use and build their personal notemaking around. I’ll share the course website & the ‘workbench’ files in due course. The extension is only one part; that ‘workbench’ is the other. Together, I think this will make for an excellent asynchronous learning experience.
Latest Changes
interface as it looks with the default Jupyter light theme
solarized PKM theme
markdown preview/source toggle can be hidden or made visible
a bug where infinite regression through files searching for wikilinks has been fixed (*fingers crossed*); this only happened when a folder (workspace) did not have a start.md file in it
start.md file is made on first run now if it doesn’t already exist, and contains helpful info about the extension (also, a much more expansive ‘pkm guide’ is also written to the folder on first run, detailing all the features)
Contextual menu for opening/closing backlinks panel
Markdown files with embedded content (eg, code or code output cells from ipynb files), when previewed, could not be printed formerly (they’d print, but the embed markdown code would show, not the content). Now, there’s a context menu that uses the print -> save as pdf function to save a rendered markdown file and its embeds as pdf (or I suppose, even print!)
Word export. A contextual menu allows for export of a rendered markdown file with its embedded content showing properly to Word. It’s glorified hyptertext, but sufficient that you could then tidy things up for eg reports and so on.
The backlinks panel
The solarized pkm theme, just because I like it. The automatically generated Guide note that gets written upon first run is shown, along with its table of contents.
Screenshots showing how the PKM: Print Markdown Preview correctly displays embedded content rather than embedded code. Also, you can use the browser’s print dialogue to save it to PDF.
Close up showing the contextual menu via right-click on a markdown note preview. The Export to Word menu item is shown.
You can try this yourself by installing it into your environment where Jupyter lives, via
pip install jupyterlab-pkm
If you’re using JupyterLab Desktop, you can also get it through the Extension Manager; search for ‘pkm’.
And yes, I built this through a judicious use of Claude & Gemini as I built one feature at a time, drawing on the work and examples of Simon Willison and Harper Reed , pushing just a little bit further than my own comfort level and knowledge each time. It seemed to me that pushing the existing markdown functionality of JupyterLab was a safer endeavour than trying to push the code executing functionality of a note-making app. This emphatically is NOT vibe-coding, and if you dig through this blog, you’ll see that I’ve been experimenting in this space since the days of RNN, so I’m aware of the issues and what is at stake.
The TRMNL device & system, an e-ink dashboard that you can extend and hook into all sorts of data streams, is all sorts of cool. And they’ve also made a lot of stuff open source. So I thought I would bring my own server and bring my own device and see what I could do. These are my notes-to-self on getting everything up and running.
BYOS – there are a number of options for setting up your own server. I went with this one built on Laravel/PHP. I already had Docker installed on my machine, so after downloading the latest release from the github page, I started Docker and then, in the terminal in the server folder I ran docker compose up and I was away to the races. The server is at http://localhost:4567/dashboard . So far, so good.
BYOD – I have a Kobo Aura lying around. I connected that to my mac, and used Finder to get into its file system, making sure that dot files were visible. Then, I followed the instructions at this repo for using a Kobo with TRMNL to get things installed, turning the device off and rebooting as indicated (important!). Also, you need the device’s MAC address. This can be found under more -> settings -> device information. Write that down somewhere handy. Now, the config file:
{ "TrmnlId": "your TRMNL Mac Address", "TrmnlToken": "your TRMNL API Key", "TrmnlApiUrl": "https://usetrmnl.com/api", "DebugToScreen": 0, "LoopMaxIteration": 0, "ConnectedGracePeriod": 0 }
Before you copy this file to your Kobo, you need to replace your TRMNL Mac Address with the actual MAC address, between the quotations. For the TrmnlToken, leave this null: “”. Then for the TrmnlApiUrl, you need to find the address for your computer on your home network. Both the device and the computer you’re using as a server have to be on the same network. On Windows: Open Command Prompt and type ipconfig. Look for the “IPv4 Address” under your main network connection (e.g., Wi-Fi or Ethernet). It will look like 192.168.1.XX or 10.0.0.XX. On macOS/Linux: Open a terminal and type ifconfig. Look for the inet address under your main network interface (e.g., en0 or wlan0). Again, look for something starting with 192… Then you’ll slot that in to your config, eg::
"TrmnlApiUrl": "http://192.168.1.100:4567/api" Save your config.json file, then move it over as per the instructions. Disconnect and then turn your Kobo off and on. At your server webpage, flip the toggle for device auto-join. Then when your Kobo has finally finished starting up, click ‘NickelMenu’ -> TRMNL et voilà.
Digital signage is yours!
Quick Update some moments later
Under the ‘Recipes’ section of the server, you can add/make new things to push to your digital sign. The interface is fairly straightforward; you add a data source that you can GET data from, and then you define a template for the data to go into. I came across this: https://github.com/SnarfulSolutionsGroup/TRMNL-Plugins/blob/main/TRMNL_Comic.md and thought, let’s go with that! So the data is ‘polled’ and comes from https://xkcd.com/info.0.json . Then we just define the template. The problem is in the server I am using (which isn’t the official one, and which is why I should probably pay for a key and use the official server with my device) is a bit more fiddly to make the templating work. However, the solution is to remember that in this particular base (BYOS laraval/php server for TRMNL) your template needs to use blade php templating conventions. Thus, for the XKCD recipe, my template looks like this:
{{–
This is a Blade template. We use Blade comments and PHP variables.
The entire data payload is available in a PHP array variable called $data.
–}}
<div class=”view bg-white”>
<div class=”layout flex flex–center-xy”>
{{– To access properties, we use PHP’s array syntax: $data[‘key’] –}}
<img src=”{{ $data[‘img’] }}” alt=”{{ $data[‘alt’] }}” style=”max-width: 100%; max-height: 100%;” />
</div>
<div class=”title_bar”>
<span class=”title”>{{ $data[‘title’] }}</span>
<span class=”instance”>#{{ $data[‘num’] }}</span>
</div>
</div>
I’ve spend a lot of time rubber ducking and cobbling together an extension for the Jupyter ecosystem that extends its interface for markdown such that JupyterLab (and its Desktop and Lite versions) can be used as a personal knowledge management system, linking ipynb files into that ecosystem, enabling code block and code output blocks to be embedded in md files, backlinks, wikilinks, etc.
In its JupyterLite version, a person can create a static website from a github repo and via a github action have the pkm extension installed. The resulting site uses pyodide and r-wasm to enable limited versions of both kernels to be run in the browser. Since JupyterLite is storing everything in its cache, accessing/saving/loading .csv files etc can be a problem, so there’s another plugin (not by me) installed when the site builds to enable sensible file access. With it working you shouldn’t even notice – but only if you view the static site in Chrome or Edge. The demo content is the kind of thing I might use, not anything I am currently using. I’ve got a lot of writing to do, but I wanted to make sure there was something in this for testing and exploration that gives a sense of what’s possible.
(Backlinks: open and close the panel to refresh for a given note).
Jupyterlite in action
I picture jupyterlite with pkm being used in introductory teaching contexts where the idea is to get used to the interface and the basics of doing both note-making and coding. I imagine therefore that a person might take a copy of my repo (including the critical deploy.yml file) and modify the /content/ directory for their own uses and teaching. Just make sure to have start.md in there. And after class or the session is done, a student could download all their notes and code files and open them up in Obsidian or another pkm system no problem. (Well, maybe not the .ipynb files. At least, not yet: see below).
If you fancy extending, fixing, or improving the pkm extension for JupyterLite, you can find it at https://github.com/XLabCU/jupyterlite-pkm . You don’t need to send me a pull request, just knock yourself out, but if you do make something really cool, do let me know.
I don’t plan on making any changes to it for a while; I’ve got a bunch of teaching materials I need to write!
JupyterLab Desktop
The JupyterLab/Desktop version does the same thing, but a bit cleaner, and of course can use full Python, R or whatever kernels, saving materials directly onto your computer. Within JupyterLab Desktop you can find it via the extension manager by searching ‘pkm’. Install it, then restart JupyterLab Desktop (You might want to use JupyterLab Desktop’s environment management to create a new python environment to use, and then install the pkm extension into that).
Once you’ve restarted JupyterLab Desktop, start a session in a folder that has a start.md file in it. When the extension first runs, it’ll create a new markdown file with a guide to using its features in that folder, too. If you check the command palette, all the pkm commands are prefaced with PKM: .
If you’re running Jupyter Lab via cli/browser with localhost, you can install at the command line with pip install jupyterlab_pkm. You can find some more information about the package at pypi (although don’t try installing the earlier versions because I borked the packaging; you’ll want version 0.1.9): https://pypi.org/project/jupyterlab-pkm/
Then, when you start the session and point it at the folder with your start.md file, you’ll see something like this:
JupyterLab Desktop with PKM in action and preview turned on
The backlinks panel can be a little wiffy, and you might want to open/close that panel to force a refresh. It’s supposed to refresh whenever the file focus changes, but it can get a little out of sync.
You could also have the same folder opened in Obsidian, if you must, at the same time; files made by other programs in that folder will display in the file panel and can be modified, etc.
Typing [[ will trigger an autocomplete to exiting content:
wikilink autocomplete in action … those ‘checkpoint.ipynb’ files probably shouldn’t be visible. Hmm.
And if a wikilink is to a page that doesn’t exit, you’ll see that visually and clicking on it will create the new file:
creating a new file from a wikilink to nowhere
Anyway, what I like is that now by using Jupyter Lab, I can have my note-making and my coding all in the same place on my machine, and Jupyter Lab can handle managing the python environments. CRUCIALLY, for the students I work with, I can start them with JupyterLite and get them used to all of this BEFORE worrying about how to install software, how to troubleshoot python installations, environments, and paths. In years past, I have lost several hours of class time to those kinds of problems! And sometimes, some students never tell you that there is a problem (for they feel that the problem is with them, not the darned computer) and so they drop out or otherwise make poor choices. I don’t want that.
So this fall, I’m going to leave installing things locally until well later in the term, and I’ll get them to install Jupyterlab Desktop. I think this will build a lot of confidence. They’ll download their materials from the jupyterlite site, and we can continue, doing the more complicated things…
…at least, that’s the plan. I am also planning on a v2 of ODATE using all this, as well as the DHPrimer. Stay tuned! (And at this point, where I’m feeling pretty good about all this, is usually when some horrible horrible bug of some sort emerges to blow everything up; earlier today I couldn’t get the damned thing to install from pypi, which was because I’d screwed up the packaging and… sigh. Anywho. I think all this should work for you.)
When we built the Open Digital Archaeology Text Book Environment, we used R files and bookdown to create a static website with our teaching materials, and then jupyter notebooks through the mybinder service to create a computational environment for the students to use. This worked, but it wasn’t ideal. There was a lot of material – longer pieces on the main ODATE page setting the context and so on, while the notebooks had a lot of technical guidance surrounding the code that sometimes seemed very divorced from the materials that sent the student to the virtual machine via Binder. In a lot of ways, the vision that ODATE had was still a bit ahead of its time, but there were too many rough edges, too much friction. ODATE needs a revamp and an update. (A lovely piece on teaching for data resuse that considers ODATE and other approaches, by Kevin Garstki was published in the SAA Record and can be read here).
Since we released ODATE into the world, several markdown-first note making applications have emerged, all in one way or another inspired by the idea of zettlekasten, where one makes small notes and through a system of interlinking, new ideas can become discoverable as they emerge through the paths of linked thought. I like that vision. I like the idea of small experiments tied together through a cluster of notes. What I think I wanted, was a note making system with background materials and worked through code that a student could then examine, adjust, redeploy, and keep track of.
So why not just use something like Obsidian? We have to keep in mind the audience. In my classes, at least, students are anxious about doing anything that Apple or Microsoft do not permit. And when it is permitted, getting python or R installed and so on is so much work. For me. For them. I have lost several sessions just for basic PATH troubleshooting. So, Obsidian: it’s sleek, and I really like it, but it has grown so much in recent years that it too is very overwhelming for the student. And while there are plugins to enable code execution within notes, those depend on the system’s default python or whatever. Plugins go stale. Obsidian versions update frequently.
Therefore, whatever I come up with needs to be :
installable with minimal nonsense
reasonably intuitive in terms of how-to-use it
stable for a couple of years at least
a place where note making and code execution are nicely bound together.
Round One
So no Obsidian. I started by taking the open source Tangent note making app and extending it to bundle python within its exe or dmg. Its developer, Taylor Hadden, couldn’t have been more patient with me as I poked and prodded around and made mistakes, and it’s a lovely app that I enjoy using.
I set out to figure out how to extend it so code blocks could be run within a note. After much faffing and disaster, I ended up with something that worked (although I’m nervous about security and other issues where code is involved.) In that experiment, code can be executed in a note, it’s stateful, output from one block can be handed over to the next block, variables don’t bleed from one note into the next, etc. I was feeling good.
Executable code block in a markdown note. The code has been run; the output is below. At right, the console window with debug statements.
Problem is, Tangent was at version 0.8 alpha while I worked on it (with significant help from Claude.ai, because I am, have always been, and am quite open about it, crap at coding. But it worked. I am also well-informed enough to know that I’m playing with fire. I get burned so you don’t have to.) While I puttered away trying to make things work, and then make them work safely, I fell behind and Tangent is now several versions beyond what I was messing with, and I despair at trying to integrate the latest version with my RubeGoldbergCode.
In any event, I tested it out what I achieved with some grad students who match my target audience: very little coding/dh experience. And it was overwhelming for them, from the download and installation (“oh no! untrusted developer! you don’t want to open this!” which, ok, maybe, fair enough) to navigating the demo materials I cooked up. We’re still going down this path, and have a Kanban board open with things they’re encountering which often sit at the intersection between things-I-hadn’t-imagined-would-be-issues and things that are oh-my-god-this-is-terrible. So, y’know. Building things.
Welcome Back dialogue in Tangent, with some minor customization.
Round Two
But I started thinking maybe I’m going about this back to front. Maybe what I want is basic jupyter code notebooks but with better note making. Therefore I’ve been working on a pkm extension for jupyterlite.
My thinking here is: I point students at a website. I have excellent learning materials there that show them how to make basic interlinked notes. Code in my teaching notes/materials can be run in .ipynb files in the same environment; the code note and the thinking note are wikilinkable. Alt+b shows backlinks so a student knows which files point to the ipynb or note file they’re on (most of the time. Still a glitch there). Note blocks can be embedded in other note blocks. Wikilinks can be autocompleted when [[ is typed. Markdown/Preview can be toggled. Shift+click on a markdown wikilink can take you to the destination.
Basic PKM baby!
Anyway, I’ve got the basics all working; the only thing I think I’m going to try to add is to embed code or output blocks from ipynb into markdown. That’ll be next week’s job, maybe.
What I’m hoping to develop content-wise is a browser-based pkm system preloaded with teaching materials pitched at beginner dh, ok-media-literate, advanced student dh such that an instructor could point them at my site as part of a course. Or, an instructor could fork my jupyterlite static site, swap in their own custom content just by switching up the markdown, and then deploy their own version. The point of this wouldn’t be to be a system for heavy-duty research and computation. Rather, it’ll be a sandbox to get started from which the student would graduate to a more robust approach: I’m trying to get rid of the anxiety around dh and getting students to a point where they develop a habit of note making around their explorations.
First load with the pkm extension running
So, in this approach, the computational part and the documentation part are side by side and able to wikilink. I don’t have to worry about kernel management etc. There’s nothing to install. A student would be directed to the website, and they’d be off to the races.
I think this is a good idea.
I’m also going to rewrite ODATE to use this, if I don’t spontaneously combust first.
(crossposted from the XLab) We use PixPlot a lot around here, both in our teaching and in projects like the BoneTrade project. It is an excellent tool for visualizing image similarity over thousands of images. It achieves this by measuring how each image perturbs a trained image network (in this case, Inception3). We end up with a vectorized representation of each image, and the distance between vectors can be measured, and visualized, as an indication of similarity: the less distance, the greater the similarity.
However, PixPlot (source code repository here) uses Python 3.7 and a series of packages that themselves could be out of date, or depend on other dependencies that could be out of date… and so on. Suffice it to say, Python 3.7 isn’t handy any more (the usual ways of installing it on your machine don’t have copies of it), and the tool is generally not usable (we did figure out how to use it within Google Colab, however, and there are cookbooks for different machines and set ups that you should probably check out before going any further with our materials here!)
We decided to see if we could update PixPlot to work with Python 3.10. How hard could it be?
Reader, we were foolish.
We started by using ‘files-to-prompt‘ with ‘llm‘ to explore the code and its installation. Most of what PixPlot does is in a single script, but we’re learning how things go together, and being able to feed the script, the setup, the readme, and the web template code in all at once to Gemini/Claude 3.7 (Claude moreso than Gemini) helped us understand what was going on. This gave us a strategy for incremental things we could target, one bit at a time, seeing what happens if we do this… The upshot is, we now have Pixplot working with Python 3.10 on a mac mini m1. We haven’t fully tested it out on eg a windows machine, because we don’t have one handy at the moment. The most challenging part was diagnosing the dependencies and getting the right ones installed in the right order. There was also a difficult moment with rasterfairy, and we came up with a kludge that gets us around that. Proper coders will no doubt be horrified but… we had a problem, we were able to solve it. Isn’t that what universal computing was supposed to be about?
Then we wanted to go one further. Work by Arnold & Tilton on ‘distant viewing’ (see their book! it’s great!) showed that in some circumstances, measuring visual similarity also permits one to see how visual influences spread (in their case, in a collection of photos where the photographers; teacher/student relationships were known). Our repo therefore has a tool to take the output of PixPlot, and represent the images as a network according to n nearest neighbours. This can then be loaded into Gephi for further analysis; if you run PixPlot with its metadata flags, that metadata (and other metadata like thumbnail image names/locations) can be passed to the network too, which could then let you create networked image visualizations with things like Gephi’s sigma.js exporter plugin. If you put those images online, you could export from Gephi as gexf, where the path to the images is one of the attributes of the node, and then use the images themselves as nodes in something like Gephi-lite.
The National Library of Australia terminated Tim Sherratt’s API access to their Trove service, which effectively nukes the GLAM Workbench. You can read about it here. Tim’s work is hugely important to the GLAM Sector. It has directly had an impact on how I teach, what I teach, and has taught me so much. I’ve written to the Director General of the NLA and to the Minister for Arts. My letter is below. If you’ve used the GLAM Workbench, if you’ve been inspired by it, if its model is something you wish your own national institutions would employ (*cough* *cough* LAC? Anyone? You there?) you should maybe make your views known.
My letter:
The recent decision by the National Library of Australia to revoke Mr. Tim Sherratt’s access to the National Library of Australia’s Trove service, by cancelling the APIs used by his GLAM Workbench, is ill considered and ill advised. I write to ask you to reconsider. I am Full Professor of Digital Humanities in the Department of History at Carleton University in Ottawa Canada. I teach and research in the fields of Public History, Digital Cultural Heritage Informatics, and Digital Humanities.
Mr. Sherratt’s work is well known across the world in the galleries, libraries, archives, and museums sectors. His work developing the GLAM Workbench has promoted Australia’s cultural heritage world wide. Indeed, because of Mr. Sherratt’s work my own students in our Public History graduate program are more familiar with the National Library of Australia’s Trove service, and hence Australian culture, than with what our own Library and Archives Canada provides. By developing the GLAMWorbench with the Trove service, Mr. Sherratt has had a major impact in how cultural heritage materials are understood at scale, across the world. His work is complementary to your own, and enhances the prestige of the National Library of Australia
I am baffled therefore at the decision to revoke Mr. Sherratt’s API keys. The computational notebook referenced in the notice of termination as grounds for revocation doesn’t even use the API key. The GLAM Workbench service that Mr. Sherratt provides is a model of best practices for the rest of this field in how to make effective use of cultural heritage resources, and ought to be supported, valued, and promoted by your organization.
I implore you to restore Mr. Sherratt’s access, and to celebrate the accomplishment of the GLAM Workbench, Trove, and the NLA together.
Electric Archaeology 
You must be logged in to post a comment.