The Valley Project began with a proposal written by Edward L. Ayers in September 1991.

Originally conceived as a traditional book, Ayers wanted to deal with both the North and the South in a comparative story. Intending to examine two places close to the border between the North and the South to see how people in such proximity and similarity went to war, he studied maps and guides to military units and indexes of newspapers to find two areas centrally involved in the Civil War from start to finish.

But as Ayers’s research continued, it became clear that the Valley Project would be far more expansive than a single book.

The Valley became one of two founding projects that established the Institute for Advanced Technology in the Humanities (IATH), a center dedicated to using information technology as a tool for humanities research, at the University of Virginia. IBM donated a number of powerful workstations, a server, and a technical advisor. Just as important, the University of Virginia Library donated space to build the institute.

With this “advanced technology,” the Valley team began collecting, transcribing, and converting original-source material into computer-readable files.

In the summer of 1993, Anne Rubin, a graduate student who had arrived at UVA after her studies at Princeton, came on board and soon took the lead in converting the 19th-century newspapers into Standard General Markup Language (SGML) on an IBM workstation, as well as directing the efforts of other Valley Project graduate research-assistants.

In the fall of 1993, Thornton Staples, associate director of IATH and the person responsible for building the earliest prototypes, showed the team Mosaic, a new tool for viewing something its creator called the World Wide Web.

The team learned that the project need not wait for years to be disseminated but could be shared even as material was gathered. The archive could reach anywhere in the world people could tie into the internet, a network expanding exponentially. And, to the project’s good fortune, the World Wide Web was built around something called HTML, Hypertext Markup Language—a subset of the SGML already adopted as the basic structure of the work at IATH.

The first web version of the Valley Project appeared as a research report on the IATH website in 1993. The Woodrow Wilson Presidential Library & Museum in Staunton secured a grant of $6,000 from the Virginia Foundation for the Humanities to help build the project; the team, in turn, set up a workstation at the museum running the Valley Project, mimicking the web. This iteration—the Augusta Archive—became the true prototype of the Valley Project. The team held a “History Harvest” in Staunton and Chambersburg to digitize artifacts and documents people shared. The site began to look more coherent as the team gathered more sources from the census, newspapers, free Black register, maps, and letters.

In 1996, the team applied for a “Teaching with Technology” grant from the National Endowment for the Humanities and were awarded over $200,000. That same year, William G. Thomas III took Anne Rubin’s place at the head of the Valley team as she finished her dissertation. Forging ahead into the Civil War years of the project, Thomas helped develop the Valley CD-ROM with W.W. Norton; led teams to transcribe thousands of Compiled Service Records at the National Archives; scanned hundreds of photographs at the United States Army Military Institute library in Carlisle, Pennsylvania; and located diaries and letters in Franklin, Augusta, and in libraries around the country.

Although access to the web continued to improve, the great majority of people reached the Valley Project through a telephone line, and the site could not be burdened with large images or elaborate navigation. Thomas, with Michael Mullins, devised the octagon-shaped navigation tools that became the project’s trademark and that is echoed in the new version of the Valley. He also established the Virginia Center for Digital History at the University of Virginia, fostering an array of innovative projects. Thomas also collaborated with Ayers to write a digital article for the American Historical Review: “The Differences Slavery Made: A Close Analysis of Two American Communities.”

Will Thomas moved to the University of Nebraska for an endowed chair in 2002, when Andrew J. Torget became the project’s manager in 2002. He oversaw the project’s final redesign and overhaul, completed the construction of the Aftermath portion of the archive, and finished work on numerous other parts of the archive. The team added thousands of letters and diaries, newspapers and speeches, government and military records to the site while continuing to upgrade the technology that drove the project. Dedicated graduate and undergraduate students at the University of Virginia produced and processed the vast amounts of material.

All the Valley letters, diaries, and newspapers were converted into Extensible Markup Language (XML), which allowed full-text searching capability. Dozens of maps were also added, which used Geographic Information Systems technology (GIS), producing detailed images of Augusta and Franklin counties. Generous private gifts from supporters and help from provosts and deans also sustained the work as the team brought the Valley Project to its conclusion.

Preservation through the UVA Library

The University of Virginia Library began processing the Valley Project in 2007 for archiving as part of its permanent digital collections. Subject specialist librarians assessed the content of the Valley of the Shadow collections, and Library technical staff updated text and image files to the current technical standards for digital preservation.

As part of the migration, the Valley databases and associated search engines were re-engineered as a Lucene index, with Apache Solr and Cocoon providing a framework for new search and content-retrieval pages. The Library’s Digital Collections department also undertook the research and updating of provenance information on collection material, and Digitization Services staff supervised the re-digitizing of the images in the collection. The metadata for collection items was augmented and standardized. The entire site, consisting of some 12,000 files, was consolidated from three servers to a single project, which was migrated to the Library production web environment in the spring of 2009.

Reimagining the Valley of the Shadow

As the 30th anniversary of the Valley of the Shadow approached, Ayers and the team at New American History partnered with Journey Group to refresh the Valley’s visual design and empower its databases.

Collaborating with the historians and educators at New American History, the strategists, designers, and developers at Journey Group proposed a path forward that would update the Valley but maintain its original spirit of exploration and discovery.

Data Structure

The Valley’s initial method of organizing information adhered closely to the original design of the World Wide Web. The site used folders and files in a directory structure, like drawers in a filing cabinet.

Journey Group developers created a new relational database that separated the data into an independent storage mechanism using JavaScript Object Notation (JSON), rendered via an application programming interface (API). The team expanded and simplified access to the data by adding the API and MySQL on top of existing XML files, as well as updating the Cocoon back-end with Laravel to power robust searches.

In this way, the primary sources, such as the letters, diaries, and newspapers, could now be accessed as open-source information: publicly available data that could be turned into any other website or native application.

Information Architecture

The strategic and structural work was guided by the twin principles of inquiry and synthesis. The team wanted the site to preserve a small amount of friction to allow for exploration and the subsequent processing of those discoveries. Learning facts about the Civil War is an entry-level task that anyone can complete on thousands of other educational websites. But learning how to think about history and to investigate it independently was always the goal of the Valley Project.

The Valley, in other words, should not be Google, serving up algorithmically ordered results. Nor should it be Wikipedia, doing all of the explaining and contextualizing for the user. Instead, the site ought to preserve its unique ethos of discovery, which is an increasingly difficult thing to defend on the all-seeing, all-knowing internet.

With these goals in mind, Journey Group strategists proposed a new consistent navigational scheme and an updated site plan for the Valley. The new navigation reorders and renames some of the primary sections in each chapter (the Eve of War, the War, and the Aftermath) and pulls out and prioritizes others, such as the overarching timeline, contextual tables and maps, and the sources. Importantly, the search becomes much more prominent throughout the site, unifying and aggregating content from the various databases.

Interface Design

Redesigning the Valley’s interfaces was an exercise in restraint.

With current accessibility and readability standards in mind, the team prioritized readable typography throughout the site, paired with responsive web design for tables, images, and other associated media, so that content would appear in the most flattering and accessible light on a variety of devices.

The team used archival photographs from the Valley to add an appealing but subtle visual texture throughout the site, such as the tinted background on the chapters and the similarly styled headers for each chapter section.

The refreshed interface design also serves to increase the sense of trust in the project for present-day visitors, who are conditioned to expect a degree of visual design polish and sophistication from powerful websites and applications.