A few months ago I attended NERCOMP opens a new window for the first time and I got to connect with IT professionals in the higher ed space. This is the first article in a series collecting my takeaways from some of the most interesting AI-related sessions at the conference.

I tried to focus on real-world examples of how higher ed professionals are thinking through and applying AI and ML, with all their limitations, in day-to-day academic life.

Keep Your Data Close and Your AI Closer: Local AI in the Academy

In their presentation, Gerol Petruzella opens a new window & Trevor Murphy opens a new window (Williams College) presented 3 case studies that leveraged AI to solve real scenarios at their institution.

These were 3 privacy-first, cloud-free applications of local AI to support pedagogy, accessibility, and student experimentation.

Case 1: 📂 Private Archive Analysis

In this case study, the team needed to analyze 822 historical syllabi for teaching patterns.

“The Rice Center for Teaching sought to use a set of historical Williams syllabi (in a mixture of pdf and docx file formats) to discover patterns in pedagogical design and practice, to inform support for incoming Williams faculty. These documents are not publicly available, and so RCT sought a local (non-cloud) retrieval augmented generation (RAG) application to preserve the privacy of the source documents, while leveraging the power of a chat-based generative LLM to efficiently discover and retrieve useful patterns from a large corpus in a timely way.”

It was very important that the solution should keep sensitive internal documents entirely offline, due to copyright restrictions.

More technical details:

• Tech Stack: Ollama opens a new window + OpenWebUI opens a new window
• Key Feature: Retrieval-Augmented Generation (RAG) with embedded citations
• Embed Model: mxbai-embed-large-v1 opens a new window
• Generation Model: phi-3-medium-128k-instruct opens a new window

Both the project design and the proposed solution are appropriately crafted to respect Williams College’s Guidance for Working with Generative AI Tools opens a new window .

Case 2: 🎞️ Audio Transcription with OpenVINO

In this scenario, they needed to create accurate SRT files with no cloud tools, ensuring copyright compliance and data control. The source of the content was films in the Italian language.

“The OpenVINO toolkit expands the familiar open-source audio editing application Audacity with 100% local (not cloud-based) AI capacities.”

More technical details:

• Tech Stack: Audacity opens a new window with OpenVINO opens a new window plugins

Case 3: 🤖 EphBot – The Pocket-Sized GenAI Lab

In the last case, they wanted to allow students to explore LLM behavior offline. They designed and implemented a solution that was easily portable and not resource intensive.

Part of the features allowed people to measure electricity usage to teach sustainability and AI ethics.

One of the neat features was that you could specify the configuration you wanted to use with the LLM (e.g. temperature) and you could set it to a value that could make it produce more or less hallucinations.

“You can check out EphBot from the Equipment Loan Center in Sawyer Library. Plug it in, connect your phone to the “ephbot” Wi-Fi hotspot it creates, point your browser at ephbot.local:8080, and you’re ready to chat. EphBot’s interface is similar to other AI chatbots, but it gives you more controls to tinker under the hood: settings, prompts, and more.”

More technical details:

• Tech Stack: Raspberry Pi 5
• LLM: llamafile opens a new window + OLMo LLM opens a new window

Hardware and Software Requirements:

• 128GB Samsung EVO Plus micro-SD card (flashed with Raspberry Pi OS)
• CanaKit MicroSD reader/USB-A adapter
• Laptop (with Raspberry Pi Imager application)
• Ethernet connection (for initial setup and llamafile download)
• Standard Pi peripherals (power cord, micro-HDMI cable, keyboard, mouse)

Final Thoughts

This presentation was thought-provoking and enlightening. It shows that there are many freely available tools and models that can be used to host AI-powered solutions in your own infrastructure with basic knowledge of AI and ML.

Attendees learned that you don’t need to use third party services like Chat GPT to develop powerful AI solutions. This is crucial when working with copyrighted content that needs to be protected from improper use by third parties.