HuMetrics: Building Humane Metrics for the Humanities

This is the second in a series of posts about each of the teams that will be attending SCI 2016, and their projects. This one was submitted by Nicky Agate.

Stained glass window spiral

What are the goals of your project and how do they fit the theme of this year’s Institute?

As Martin Paul Eve writes of current metrics systems in “Metrics in the Arts and Humanities,” “They are a quantification of a symbolic capital that maps onto material capital.” We approach this year’s Institute with the goal of finding a way to measure a humane metrics, that is, in a way to expose a broader sense of value or worth that goes beyond quantitative measures that speak only to the market. We want to answer a number of questions that have left scholars serious about the promise of impact metrics for the humanities unsatisfied:

  • If we start from the premise that citation metrics are insufficient to understand academic impact in the humanities and that, at the same time, more extensive alternative metrics that measure the popularity, trendiness, or authorial stamp of a given piece of scholarship cannot alone indicate quality, where does that leave us?
  • What of the obscure scholarship that uncovers major new territory within its field when that field is extremely small?
  • How can we ensure that it is not only the importance of the loudly public that is measured?
  • How are we to make claims for the value of non-traditional scholarly communication in the humanities? How do we gather metrics that value the entire lifecycle of scholarly production rather than just the end results?
  • If existing altmetrics can only measure the impact of something published online — and if they require a Digital Object Identifier or other persistent identifier(s) — what alternatives might we envision?
  • How can we incentivize openness and public availability so that scholars working outside the framework of traditional scholarly publishing are rewarded for their contributions to advancing knowledge in their discipline?

Illustration of statistics and people

Who is on your team, and what are you hoping they will contribute to the project?

Our team offers extensive experience representing a broad array of stakeholders: scholarly societies, altmetrics organizations, university administration, open access repositories, funding agencies, and academic libraries. We believe it is essential to bring these people together to discuss not only how we should measure scholarly impact (and what we should be measuring), but how we can assure such metrics play a role in tenure and promotion cases, grantmaking, and library assessment in a systematic rather than an ad-hoc way.

Nicky Agate is project manager for digital initiatives at the Modern Language Association, where she manages MLA Commons, the CORE repository, and Humanities Commons.

Rebecca Kennison is a principal at the non-profit organization K|N Consultants and co-founder of the Open Access Network (OAN).

Stacy Konkiel is the Outreach and Engagement Manager at Altmetric, a data science company that uncovers the attention that research receives online.

Christopher Long is the dean of the College of Arts & Letters at Michigan State University.

Jason Rhody recently joined the Social Science Research Council (SSRC) as director of the Digital Culture program, which focuses on scholarly communication, digital methods, and transparency in social science research.

Simone Sacchi is the Research and Scholarship Initiatives Manager at the Center of Digital Research and Scholarship at Columbia University.

What do you look forward to most from SCI, and what do you hope to accomplish through the Institute?

We look forward to engaging in rich conversation about how we might develop more humane ways of documenting the impact and quality of humanities scholarship.

Ideally, our work will form the basis of an inclusive and open system that allows for evolving conceptions of scholarly communication and academic impact, one that can help funding agencies, research administrators, and individual researchers better understand the depth and breadth of the impact of humanities scholarship in public and private sectors alike.

Jumble of numbers

Building a Sustainable Digital Edition Ecosystem

This is the first in a series of posts about each of the teams that will be attending SCI 2016, and their projects. This one was submitted by R. Darrell Meadows.

Frederick Douglass portrait

Longstanding documentary editions like the Frederick Douglass Papers are beginning to find important pathways toward building a sustainable digital edition ecosystem. The NHPRC is proud to support the editing and publication of all of Douglass’s writings. To learn more, visit the Frederick Douglass Papers Edition at http://www.iupui.edu/~douglass/.

What are the goals of your working group and how do they fit the theme of this year’s Institute?

This TriangleSCI 2016 workgroup will explore the array of challenges and opportunities facing the digital edition. Over several days of intensive discussion, the group will identify immediate next steps and potential collaborations that can help to build a sustainable digital edition publishing ecosystem—one in which more and more projects are working in similar ways (leading to greater interoperability), in which a wider range of actors and institutions are participating (sharing skills, costs, and a shared commitment to the work), and in which the products of these efforts are discoverable, intelligible, usable, and freely accessible to researchers and the broader American public (in ways that will facilitate new research and learning at all levels).

By providing a forum for documentary editors, historians, archivists, publishers, and other digital humanities professionals to collaborate and discover together viable solutions and pathways toward a sustainable, open-access publishing ecosystem for digital editions, the working group also seeks to ensure that efforts are rooted in the practical realities of project design, work plans, management, costs, and related challenges facing all of these stakeholders. Our working group addresses, for an important sub-field within the scholarly communication ecosystem, the need for “scholars, universities, and funders” and others “to help re-align the incentives and economics” that currently guide the production of scholarly digital editions.

Who is on your team, and what are you hoping they will contribute to the project?

Participants in our working group are historians, digital humanists, editors, archivists, technologists, and university press/library publishers. As such, they represent some of the key professional groups whose knowledge, expertise and perspective have much to contribute to these discussions, and the new insights for building a sustainable future for the digital edition that we expect to emerge from them. Participants are:

  • Tenisha Hart Armstrong, Associate Editor and Associate Director, Papers of Martin Luther King, Jr., Stanford University
  • Hugh Cayless, Digital Humanities Senior Programmer, Duke University Libraries
  • Julia Flanders, Director, Digital Scholarship Group, Northeastern University Library
  • Ondine Le Blanc, Director of Publications, Massachusetts Historical Society
  • Darrell Meadows (organizer). Director for Publishing, National Historical Publications and Records Commission
  • Daniel Powell, Marie Skłodowska-Curie Fellow with the Digital Scholarly Editions Initial Training Network (DiXiT ITN)
  • Joshua Sternfeld, Independent scholar and Senior Program Officer, National Endowment for the Humanities, Division of Preservation and Access
  • Rebecca Welzenbach, Director of Strategic Integration and Partnerships, University of Michigan Press/Michigan Publishing
Photo of airplane flying over a wagon train

Digital technologies are driving significant changes in the digital edition publishing ecosystem. Our working group brings together a wide range of voices to learn more about and share how we can effectively collaborate and navigate, in practical ways, the shift toward a sustainable digital edition ecosystem.

What do you look forward to most from SCI, and what do you hope to accomplish through the Institute? What are your plans for next steps after the Institute this fall?

We are delighted at the opportunity and eager to engage in this wide-ranging exploration of the challenges and opportunities facing the digital edition. Intensive discussions will explore six core issues or challenges facing the digital edition: (1) users and uses of documentary editions; (2) ecosystem change; (3) editions, scholarship, and the historical profession; (4) work flows and common ways of working; (5) cost models and sustainability; and (6) best practices.

We also have a number of objectives for continued outreach, dissemination, and follow-on activities. First and foremost, the conversation will help the workgroup participants and other TriangleSCI 2016 participants to better understand the current state of the digital edition publishing ecosystem, and to identify immediate next steps and potential collaborations. The workgroup will outline a series of medium- and long-term goals, and map out a working agenda for subsequent convenings which the NHPRC will sponsor either individually or in cooperation with other entities and funders. We expect this conversation to result in at least one white paper, and to help inform the strategic direction of the NHPRC’s Publishing Historical Records in Documentary Editions program. Ideas emerging from this conversation will also be shared in participants’ own (or their institution’s) blogs, publications, talks, and other dissemination activities. Last but certainly not least, we expect that participation at TriangleSCI will shape our engagement with and across the full spectrum of professional groups that make up and will contribute to the emerging digital edition ecosystem.

Is there anything else you’d like to say about your project or participation in the Institute?

All of the participants in this working group feel the urgent need for what we believe will be an important, forward-looking conversation, and wish to extend our thanks to Triangle SCI for this opportunity. We are thrilled and honored that our working group was selected. It is only through collaborations like these–spanning across professional groups—that we will be able to build the kind of sustainable digital edition ecosystem we want and need.

Screen shot from ePADD interface

With the growing proliferation of born-digital collections, there will be ever-greater need for projects that ensure these research collections are not only discoverable and visually accessible, but appropriately intelligible and usable. Pictured here is the NHPRC-funded tool, ePADD, developed at Stanford University Special Collections & University Archives. Tools like ePADD will open new possibilities for collaboration across a variety of professional groups in the creation of born-digital editions. (Visit the EPADD site at https://library.stanford.edu/projects/epadd.)

SCI 2016 project teams

Illustration by Kyle BeanAll of the invited SCI 2016 teams have confirmed that they are indeed able to attend this year’s institute on Incentives, Economics, and Values: Changing the Political Economy of Scholarly Publishing, so we’re now ready to announce who they are and the titles of their projects:

  • Building a Sustainable Digital Edition Ecosystem
    Tenisha Hart Armstrong, Hugh Cayless, Julia Flanders, Ondine Le Blanc, R. Darrell Meadows, Daniel Powell, Joshua Sternfeld, Rebecca Welzenbach
  • Does One Size Fit All?: Small Societies, Humanities Journals, and the Risk and Promise of Open Access Conversion
    Patrick Alexander, Eric Bain-Selbo, Cheryl Ball, Meredith Goldsmith, John McLeod, Kristen Ratan
  • HuMetrics: Building Humane Metrics for the Humanities
    Nicky Agate, Rebecca Kennison, Stacy Konkiel, Christopher Long, Jason Rhody, Simone Sacchi
  • Global Voices in Developing a Sustainable, Equitable Open Access Future
    Kamal Bawa, Ada Emmett, Town Peterson, Rosario Rogel-Salazar, David Shulenburger, Tetiana Yaroshenko
  • Social Integration for the Distributed Commons
    Janneke Adema, Sherri Barnes, Eileen Joy, Donna Lanclos, Stuart Lawson, Sam Moore

Congratulations to all of these teams, and we look forward to seeing you in Chapel Hill in October!

Over the next few weeks we’ll be adding blog posts where each team will describe their project and introduce the team members, so stay tuned for more information soon…

[ Image by Kyle Bean, used under a CC license. ]

Discussion at SCI wrap-up

SCI 2016: submit your proposal to join the Institute in October, and help shape the political economy of scholarly publishing

After a successful second year of the Scholarly Communication Institute in the Research Triangle, we’re happy to announce the theme and dates for SCI 2016, and invite you to submit a proposal to participate!

The dates for SCI 2016 are October 9 to 13, and as in the last two years, it will be held at the Rizzo Center in Chapel Hill, NC.

This year’s theme is “Incentives, Economics, and Values: Changing the Political Economy of Scholarly Publishing”, which we describe on the theme page in this way:

“The scholarly publishing ecosystem is undergoing transformation, with new technologies creating both opportunities and challenges to traditional publication models. Underlying these models are economic and behavioral patterns developed over time in response to sets of incentives. What can scholars, universities, and funders do to help re-align the incentives and economics of scholarly publishing with the values of academia? How can we set conditions to enable the greatest benefit to be achieved at the lowest cost? How can we ensure that the system is not biased in favor of those who already have status, and does not inhibit a diversity of perspectives from being heard? What models could we put in place to help high quality publications to be financially sustainable, build stronger communities of stakeholders, surface the true costs, and fairly reward those who are doing the work?”

“This year’s Triangle Scholarly Communication Institute invites teams to explore these and other questions related to the political economy of scholarly publishing, to develop plans, to test processes, to come to agreements, and to launch initiatives. SCI is an ideal place to bring together perspectives and expertise that may not normally intersect, and to build understandings and new models based on them. We encourage pragmatic, proactive optimism, and hope stakeholders in the scholarly publishing ecosystem will use SCI as a platform to nurture positive change.”

kyle-bean-digitaleducation-3Please see the theme page for ideas of who you might bring together to form a team – we’re looking for a broad and diverse set of perspectives, and teams that will address both specific and general problems and opportunities. This is a great opportunity to launch a new project, have some concentrated time to develop an existing project with a broader set of collaborators, or just to begin to explore and experiment with ideas that are difficult to pursue in your usual work context.

To participate, form a team of 3 to 8 people, and submit a proposal along the lines of what’s described in our RFP (submission deadline is March 14, 2016). If your proposal is selected, the Institute will cover costs for your team to attend.

To learn more about what it was like in past years, see the Storify threads from SCI 2014 and SCI 2015, which contain tweets and photos from participants, and the web pages for 2014 and 2015, which have links to information about the teams that participated in those years, their projects, and other notes.

There’s also a lot more information in our FAQ. If you have any questions, contact us at scholcomm-institute@duke.edu

 

[ Featured photo of SCI 2015 discussion by Eric Dye used under CC license: https://www.flickr.com/photos/137077175@N05/sets/72157659949691661/ ]

[ Light bulb image by Kyle Bean, used under CC license. ]

Photo from SCI 2015

SCI 2015 has concluded, stay tuned for SCI 2016

The 2015 Scholarly Communication Institute concluded last Thursday and it was an exhilarating and exhausting week for everybody involved. As we hoped, participants started new collaborations, incubated new ideas, developed new plans, built new things, and made new friends. One team completed and submitted a grant proposal by the end of SCI in order to continue their project, and other teams are already working on their next steps. We got lucky with beautiful fall weather, ate well, took walks in the DuBose house gardens, and had tours of nearby universities and their host cities.

I won’t recount the whole week here, but will point you to some links where you can get a gist of what it was like to be at SCI.

We’ll soon begin work on planning SCI 2016, and will announce the RFP early in the new year. Check back on trianglesci.org or @TriangleSCI for the announcement, and we hope you will consider submitting a proposal for the next round.

A big thanks again to everybody who participated, and especially to the Andrew W. Mellon Foundation for making this possible and to our partners at Duke University, North Carolina Central University, North Carolina State University, The University of North Carolina at Chapel Hill, the Triangle Research Libraries Network, the National Humanities Center, and the American Council of Learned Societies for their work to make it a success.

This slideshow requires JavaScript.

[ Featured photo and slide show photos by Eric Dye used under CC license: https://www.flickr.com/photos/137077175@N05/sets/72157659949691661/ ]

SCI team time

Think, Do, Collaborate, Cross-Pollinate

We’re less than a month away from the beginning of the second Scholarly Communication Institute held in the Research Triangle area of North Carolina, and are looking forward to this year’s cohort coming to Chapel Hill for five days of thinking, planning, and doing in a collaborative and relaxed setting. Last year, the first year of SCI in it’s new home, most of the participants were from the local area, and we tested out a new model for how such an institute might work. By all accounts it was a great success, and this year participants are coming from far and wide – from all across the United States, and about one third from other countries too, including as far away as Perth, Australia.

What do we do during these Institutes?

About a third of the schedule is unstructured, what we call “team time”. Each of the invited participants is part of a team working on a project they proposed during the RFP process – they set their own goals, process, and deliverables. During the team times there are no rules – teams can brainstorm, research their topics, run a charrette, document plans, develop software, write a paper, test models – last year participants did all of these things and more during their unstructured team times.

Another third of the schedule involves the entire cohort in discussions together, what we call “plenary” meetings. During these sessions all the teams come together in conversations regarding issues of mutual interest to all of their projects. Some of these sessions start with a focus on a particular team’s project, allowing them to seek advice from the broader group on issues that are challenging them, or to seek feedback on ideas they are trying to advance. Other plenary sessions are conversations guided by several facilitators, who throughout each day have been engaging with each of the teams, and listening for and suggesting areas of intersection between the different projects.

DuBose House gardensThe final major piece of the schedule is social time. We know that often the best insights come when you’re not necessarily looking for them, but rather over a meal, or drinks, or when taking a walk someplace you haven’t been before. So we’ve built a lot of time for that into the schedule. Breakfast and lunch each day will be in the rooms of a historic house on the grounds the conference center, with ample time after lunch to take a walk in the nearby gardens. On one evening we’ll have a reception at the National Humanities Center, where SCI participants will have an opportunity to talk with fellows and staff of the Humanities Center as well as invited guests from nearby universities. And on other evenings there will be optional small group dinners at various restaurants in Durham and Chapel Hill, with visits to Duke University and the University of North Carolina along the way.

However, almost all of this is flexible. Last year, we adjusted the schedule along the way, based on suggestions from participants, and in response to observations about how useful different types of activities were at different times of the Institute. Mealtimes are fixed, but aside from that the schedule is fair game.

Slide: Notes for a retreat

What there won’t be at the Institute are PowerPoint slides. No reading a prepared talk, no deciding which conference track you’re going to attend, no vendor sales pitches. Alright, maybe there are a few presentations, but they’re brief and mostly about sparking ideas and setting tone. We’ll have some brief remarks at the receptions, and on the first day, Tom Scheinfeldt will be opening the Institute meetings with some observations on setting the conditions for a productive retreat. On the last day each team will practice their “elevator pitch” with SCI’s advisory board, answering these questions about their project: What? So what? and What next?

The participants will come together on October 11 mostly not having worked together or even met each other before, and will leave on October 15 having started new collaborations, incubated new ideas, developed new plans, and built new things. They will also have eaten well, relaxed away from their usual work, and, we hope, made new friends.

Over the next few weeks you’ll be able to follow the progress of the 2015 Triangle Scholarly Communication Institute via the #TriangleSCI hashtag on Twitter, and afterwards, via this blog. To get a sense of what it was like last year, see this Storify thread that collected a representative sample of tweets from SCI 2014, and these blog posts from SCI 2014. And if you’re interested in participating next year, keep an eye on this blog for the next RFP, to be announced in early 2016.

An Analytical Attribution Framework

This is the fifth and final in a series of posts about each of the teams that will be attending SCI 2015, and their projects. This one was submitted by Christopher Blackwell.

What are the goals of your project and how do they fit the theme of this year’s Institute?

Attributing and valuing scholarship was easy when scholarship was monographic and communities of scholars were small. It was easy to attach an author to a citation when the author was universally known (“Aristotle”, “Linnaeus”) and the citation pointed to a clearly defined, grossly granular publication, one of relatively few to have emerged in a given year. Scholarship has always emerged from collaboration, but in a rigid hierarchy it was easy to collapse a team of researchers to a single named authority.

When scholarship was largely monographic, attribution could remain monolithic: a name or one-dimensional list of names attached to a work. Just as scholarship supported one primary and straightforward method of interaction—reading—attribution likewise supported a relatively simple number of methods: credit for authorship, ranking in a list of authors, a multiplier based on the perceived value of the work.

Detail from the Venetus A manuscript, showing Iliad 3.1-9

The promise of digital scholarship lies in the potential for synthesis and analysis, a much richer body of operations that may extract more meaning and prompt more insight from a given body of data. We wonder if approaches to synthesis and analysis that have proven fruitful for our own research might also be fruitful ways of approaching how we credit and value contributions to that research. We have all encountered problems of attributing and valuing authorship in situations like:

  • many editors producing a single edition of a text,
  • a group of developers contributing to a single software project,
  • many editors indexing or commenting on a body of data (texts, images, &c.),
  • scholars producing complementary analyses of a given text, (that is, one scholar produces a syntactic analysis, and one a semantic analysis),
  • scholars producing exclusive analyses of the same data (that is, one scholar analyzes syntax one way, another analyzes it another way).

In each case, while it is possible to attach “authorship” to individual pieces of work—lines of code or XML, individual indexed relationships, individual analysis—it is extremely difficult to quantify the significance of each author’s contribution:

  • Editor A enters an initial OCR text of the Iliad to a GitHub repository, thus “contributing” 15,000 lines; Editor B meticulously documents variants, receiving credit for only 75 lines.
  • Author N writes a short algorithm that is called innumerable times throughout the execution of a piece of software; it is brilliant because it consists of only 12 lines of code.
  • Scholar A captures the syntax of a complex sentence in Thucydides; Scholar B builds on that initial analysis, making it better. It would be desireable to capture Scholar B’s debt to Scholar A, and the extent to which the two analyses differ.

Who is on your team, and what are you hoping they will contribute to the project?

The members of this proposed working group have extensive experience applying innovative approaches to analysis, for topics that require collaborative effort across diverse data, often under conditions of uncertainty:

  • documenting conflicting interpretations of damaged text-bearing artifacts,
  • integrating various kinds of image-data for recovering lost text,
  • exploring the intersection of syntactic and semantic graphs of texts,
  • associating metadata with texts and data-structures at differing levels of granularity,
  • capturing iterative analyses of corpora undergoing collaborative editing,
  • aligning diverse data across generic and chronological axes,
  • building learning portfolios to track specific performance in the acquisition of a foreign language.

Our team members are:

  • Bridget Almas, the lead software developer and architect for the Perseus Digital Library
  • Christopher W. Blackwell, Project Architect for the Homer Multitext and co-developer (with Neel Smith) of the Canonical Text Service Protocol
  • Francesco Mambrini, research fellow at the Digital Humanities Department (IT-Referat) of the German Archaeological Institute
  • Ségolène Tarte, senior research fellow at the University of Oxford’s e-Research Centre
  • Gabriel A. Weaver, a Research Scientist at the Coordinated Science Laboratory at the University of Illinois at Urbana-Champaign

What do you look forward to most from SCI, and what do you hope to accomplish through the Institute?

All the members of our team have talked about these issues, and possible technological solutions, for years, in various ad hoc conversations. The Institute will give us the most valuable opportunity of dedicated in corpore time and space to align our individual ongoing work and technologies with our shared goal of flexible, expressive, machine-actionable attribution and evaluation.

We are also looking forward to the opportunity to share ideas with, gather criticism from, and face the need of clear presentation to the larger group that will gather in October.

Venetus A

What are your plans for next steps after the Institute this fall?

We are all engaged in collaborative work that could immediately serve as test-beds for ideas about analytical attribution. Blackwell’s work on historical botany, for example, continues to engage undergraduate students from across disciplines, over relatively short periods of time, contributing diverse and very specific data to an evolving digital library: taxonomic indexing, medical commentary, historical essays, transcriptions of letters, compilation of geo-spatial data, photography. Almas, in her work on Perseids has an immediate need and audience for innovative approaches to citation of complex and evolving analyses. Mambrini’s research at the DAI is likewise focused on analysis of texts and meta-analysis of scholarly interpretation.

Weaver notes that an analytical framework for attribution would be immediately useful in the domain of computer science as a discipline and within industry. Currently, there is demand among practitioners to be able to search, retrieve, and measure the evolution of multiple versions of security policies and compliance reports over time.

The proposed discussion for this SCI workshop can have strong impacts on the current activities of the Deutsches Archäologisches Institut, and any conclusions that we reach would be welcomed contributions to any of the ongoing outreach initiatives of the Institute, e.g. the Digital Classicist Berlin series of symposia. Mambrini is a co-chair of the conference on “Corpus-Based Research in the Humanities” (next held in Warsaw in December 2015), whose participants would be specifically interested in this topic.

Is there anything else you’d like to say about your project or participation in the Institute?

The Scholarly Communication Institute represents an opportunity that is all too rare: a space for forward-looking conversation among scholars from different disciplines. We are excited at the prospect, and honored to have been invited.

[ Image credits: http://blogs.brandeis.edu/greek10/?p=22 and http://homermultitext.blogspot.com/2013/06/iliad-8-scholia-on-mythological.html ]