ASCL.net

On Tuesday, January 7, the AAS Working Group on Astronomical Software (WGAS) and the ASCL sponsored a special session on code sharing as a follow up to the splinter meeting “Astrophysics Code Sharing?” held at AAS 221. We continue the dialogue for ways to improve the transparency and efficiency of research by sharing codes and to mitigate the negative aspects of releasing them.

Before the session started, however, there were a few nerve-wracking moments;  weather- and Amtrak-related delays had one of the presenters arriving at AAS at 2:40 AM the day of the session rather than before lunch on Monday, and another getting to AAS after the session had started (!) but before his talk was to begin. So yes! There were minutes to spare!

The standing-room-only session was moderated by Peter Teuben of the University of Maryland and chairman of the ASCL Advisory Committee; Robert Hanisch, STScI, outgoing chair of the WGAS and also a member of the ASCL Advisory Committee, provided closing remarks. Those not in the room were not without news of what was being said in it, as there was much tweeting about the session (#aas223, #astroCodeShare).

Peter started the session by introducing the speakers (present or not) and explaining a bit how the session would work: code case studies would have 2-minute question periods for any clarifications or questions about the cases themselves, and other questions would be deferred until the open discussion period, which was approximately the latter half of the session.

Presentations
A very brief summary of some main points of the sessions, along with their titles, presenters, and links to slides where applicable, is given here.

• Occupy Hard Drives: Making your work more valuable by giving it away, Benjamin Weiner (University of Arizona)
  Ben pointed out that time spent writing software represents an enormous sunk cost that is, unfortunately, not viewed as doing real work, though writing software is part of doing science. He stated that widely-used software has enabled at least as much science as a new instrument would. He encouraged people to document their code for their own sake, release it without worrying about bugs or other potential issues in the software, and to write software methods papers for journals.
  slides (PDF)

• Maintaining A User Community For The Montage Image Mosaic Toolkit, Bruce Berriman (Caltech)
  In this case study of Montage, Bruce stated that releasing software comes with a cost, but that it is still worth doing. Montage was developed under contract, and was designed for ease of maintenance, modularity, and sustainability from the beginning. It is maintained primarily through volunteer effort, and in part through collaborations, e.g., with the LSST EPO team. He said the Caltech license under which Montage is licensed does not allow users to redistribute modified code, nor can Montage be included in other distributions such as Redhat. He suggests coders consider licensing carefully.
  slides (PDF)

• Cloudy – simulating the non-equilibrium microphysics of gas and dust, and its observed spectrum, Gary Ferland (University of Kentucky)
  Gary discussed Cloudy, which, with over three decades of use, is the most mature of the three codes covered in this session. The code is autonomous and self-aware, providing warnings about what might have gone wrong when things do go wrong. Though the user community is broad and participants in the summer schools that are held on the code have formed collaborations, a Yahoo! discussion forum for Cloudy has not been as successful as they had hoped. Cloudy was released as open access, with the most permissive license possible; Gary cited NSF as making this necessary since the code was developed with public grant funds. Students who work on the code get industry-standard programming experience, which is intended to help students gain employment after graduation.
  slides (PDF)

• NSF Policies on Software and Data Sharing and their Implementation, Daniel Katz (National Science Foundation)
  Dan covered the NSF policies that govern software funded by the agency. Though some NSF panels are much more rigorous than others, it is expected that PIs will publish all significant findings, included data and software; he stated quite firmly that data include software according to the Government. He also said that it is up to the community via peer review panels to enforce these policies, that many core research programs don't enforce this very well, and that the community determines what is and is not acceptable. This may be changing, however, as with an Office of Science and Technology Policy memo on open data, OMB policies are pushing harder on open access.
  slides (PDF)

• The Astropy Project’s Self-Herding Cats Development Model, Erik Tollerud (Yale University)
  The newest of the three code projects highlighted is Astropy. Erik described the grass-roots effort to self-organize the now ~60 code base contributors, and that this arose out of a common goal: to streamline astronomy tools written in Python, as having eight different packages to do the same thing means that 7/8s of the effort was wasted effort. He stated that technology now exists that provides good support for such an effort, including GitHub to manage the processes of many developers, Travis for testing code, and Sphinx for documentation, which is written as the code is written. He pointed out that agreement on the problem was the key in getting the effort to come together and that consensus, guidelines, and expectations make it work.
  slides (PDF)

• Costs and benefits of developing out in the open, David W. Hogg (New York University)
  David started out by saying that everything his group does is open --  all papers, grant proposals, comments, and codes -- and has been since 2005, and that this was a pragmatic, not an ethical decision. He stated that the negatives others give for not releasing code -- getting scooped, embarrassment, time, e-mail and support requests, licensing -- are overplayed, and that since the public is paying for this, we should return the products we develop to them. He doesn't know of a single case of someone's getting scooped because he/she shared code. Rather, the benefits that sharing openly provides, establishing priority, visibility and goodwill, re-use and citations, feedback and bug-catching, and having the moral high ground, outweigh the overplayed negatives.
  slides (PDF)

Discussion
After David's presentation, Peter opened the floor for questions and discussion, and Kelle Cruz from Hunter College was ready! Kelle said that AAS should require code release and then asked whether anyone from the AAS journals was present. There was not.

Kelle then suggested to Daniel Katz that the NSF should take stronger role in enforcement. Dan said he will see what he can do to get astronomy reviewers training for what to look for, and that he already does this for his area. David Hogg said there aren't any mechanisms for long-term stewardship of software and asked whether the NSF was looking at this. Dan said it is not at this time, and that the NSF generally avoids long-term commitment of funds.

Someone in the back of the room pointed out that protection of code can also lead to the protection of errors, told a sad anecdote to illustrate that point, and commented that code sharing fosters improvements in coding practice. In response to a question about whether it was worthwhile to share very specific code, David answered yes, just post it, that if it's not useful to others, so what? But it just might be! And Benjamin Weiner suggested the code be put in GitHub.

Two questions came from someone else in the back of the room, one on whether export control restrictions (ITAR) would be changing; the second question relayed that PhD students write a code for their thesis but then protect it because, in their perception, the code makes them employable, and did the panel have anything to say about that? Erik Tollerud made the point that people are hired for the skills that went into creating the code, not for a particular code. David replied that he has seen this with data, that proprietary data does sometimes give someone leverage for employment. Dan answered the ITAR question by saying that changes in ITAR were probably not going to happen soon.

Another attendee asked about the cost of making code shareable, of what that cost is, and felt that the panelists had swept it under the rug. Ben replied that it's a community problem, the community needs to reward it, and there needs to be a values change. In the meantime, put it out there anyway; clean it up if you can, but put it out. David agrees there are costs, but the benefits are more substantial than the costs. The cost is not very large and the upside is larger than the downside. Bruce thinks it is worth the effort to plan upfront; that will save time/money later on. This is harder if the code is not initially planned, but one should try to address this when possible.

Nuria Lorente, who was following the session from Australia through Twitter, tweeted that "NOT releasing code also comes at a price, which is often forgotten."

Andrej Prsa from Villanova made a strong appeal to post code to arXiv; he stated that astro-ph should be open to other things beside preprints. Someone else pointed out that arXiv doesn't necessarily agree. David said that he put the documentation for emcee, the MCMC hammer on arxiv and that gets cited. Erik made the point about additional contributors to a software development project such as Astropy don't get credit if they are not on the author list on the paper uploaded to arXiv. Alberto Accomazzi from ADS mentioned that updating the author list on arXiv was a way to fix that and give others credit, even though that will not be reflected on ADS.

Someone commented on the need for some sort of code sharing infrastructure to help with sharing. David commented that he wants all flowers to bloom, but some flowers are more valuable than others. Erik said that better search engines over time will help, that Astropy is more findable because of better search engines and because more people now link to it. It was mentioned that with more code sharing, finding useful codes may become more difficult as the signal to noise ratio goes down.

Alberto Accomazzi brought up the uncertain provenance of code, code that does not have a license, and sometimes no author, attached to it, and stated that it is hard to deal with because it cannot be shared. This was echoed by David, who pointed out that a lack of a license for a code can prevent release. Bruce suggested a licensing workshop would be a good idea, and this idea got traction among attendees. The recent re-licensing of yt was brought up. Dan Katz looks specifically for licensing information when looking at proposal, and it's clear to him that many people don't know what they are doing on this and could use guidance. David suggested that people use BSD or MIT licenses if they know nothing about licensing.

Peter Teuben then brought the discussion to an end and turned the podium over to Robert Hanisch for closing remarks.

Session wrapup
Robert Hanisch reiterated that software sharing is fundamental to the dissemination and validation of research results, and though there are carrots and sticks for software sharing, the sticks are not very strong. He also pointed out that nothing within the funding agencies offers support for software development and that there is a disconnect between national policy and implementation. Journals at best only encourage code release, too; they do not require it. A sociological change is necessary; in the meantime, he hopes those attending will just put codes out there!  The benefits outweigh the costs.

He talked also about opportunity for change; as of Sunday January 5, the Working Group on Astronomical Software has Frossie Economou as its new chair, and that over the weekend, the Council of AAS had suggested that the WGAS be elevated from a Working Group to a Division within AAS. He had requested that the Council have the WGAS offer a prize specifically for software, and though the Council did not accept the idea upon presentation, Bob noted that a Division can award prizes independently. Having a Division focused on software will also provide more visibility for it, and on this hopeful note, the session ended.

... though the discussion continues...

My thoughts (just a few)
This is the fourth discussion session the ASCL has arranged; previous sessions include one at AAS 221 and two at the previous two ADASS meetings. (Links to materials or discussion from previous sessions are below.)

I was glad to hear several of the presenters say the concerns people have about releasing their codes are overplayed. I was particularly happy when David said that if people would only go ahead and release their imperfect software, other people would see that released codes are also imperfect and thus feel more emboldened to release their own imperfect work. Yes! Lose the fear, gain the codes! It really doesn't need to be perfect; Nick Barnes, among others, have written eloquently, or amusingly, on this subject already. Astronomical software wants to be free; please release it, let it show!

It was hard for me to stay silent when the need for a code sharing infrastructure was mentioned, not because I disagree with the need -- I believe the need is very great! -- but because the ASCL is trying hard to help with that. I've looked at other similar efforts tried over the years, and either they have started, lived (usually briefly) and in one case, even flowered, and died, or they still exist but are mostly silent and their efforts in code sharing dormant. The ASCL has been around since 1999 and is indexed by ADS, and use of it has been increasing. It's not perfect, but it does work and is actively growing.

I believe that science should be as transparent as possible, that code release (absent ITAR and other truly compelling reasons) even if only for examination, not reuse, is part of this transparency, and that ultimately, code release is better for code authors, especially if the astronomy community works together to make it better for them. Code sharing can make astronomy more efficient, too, which is especially important in the current financial climate.

Finally, I want to thank Peter for moderating the session, Bob for offering closing remarks, and the most excellent Ben, Bruce, Gary, Erik, Dan, and David for presenting at this session and also for not protesting even one time about the innumerable emails they received from me from May on. I also have to thank our wonderful volunteer whose name I did not get, alas, for her great work and for counting the 149 (!) attendees, the AAS for accepting the proposal in the first place, and the amazing people who sent this session literally around the world through their tweets. Thank you!

AAS 221 Astronomy Code Sharing? links
Announcement
Omar Laurino joins Astronomy Code Sharing panel
Brief blog post
Astronomy Computing Today post
Slides used at meeting: Google Doc  PDF

ADASS XXIII (2013) links
Announcement
Our eight questions
The eight questions that were discussed/links to discussion notes
Pre-print of proceedings paper

ADASS XXII (2012) links
Birds of a Feather session
Resources used/linked to for ADASS
Pre-print of proceedings paper