Editor’s Note: This is the second in a three-part series exploring the opportunities and challenges facing the Trump administration’s changes to how the Pentagon buys software. Part 1 is available here.
The new rule book for buying software that Defense Secretary Pete Hegseth recently made mandatory has its origins in problems with an aircraft.
“The F-35 was always talked about as a computer wrapped in a jet,” said Bess Dopkeen, who served until January as a senior advisor to the Undersecretary of Defense for Research and Engineering Heidi Shyu. “And there were lots of wonderful people working on it. But the way it was purchased and developed, in reality, was that they built a jet, thinking, ‘Someday we’ll plug in that computer and it’ll all work fine.’ And of course it didn’t work fine, because you made the software an afterthought.”
The F-35 was only one of many major Pentagon programs that had problems with software, but the yearslong delays and budget overruns made it the poster child for widespread dissatisfaction in Congress and elsewhere with the way the Defense Department acquired software.
“There was a lot of frustration on the Hill,” said Dopkeen.
It was apparent, even in 2017-18, that the Pentagon was a generation behind in its thinking. “Software was definitely thought about as a secondary backroom thing in DOD. We buy a jet. We buy hardware. We put software in it later. Even seven, eight years ago, no serious commercial concern thought like that. They were all ‘Software First’” she said.
To drag DOD into the 21st century, Congress mandated the Defense Innovation Board, an advisory council of industry executives, venture capital founders, and academic luminaries, “to undertake a study on streamlining software development and acquisition regulations” for DOD and the military services.
The Software Acquisition and Practices (SWAP) Study was helmed by innovation board members CalTech scientist Richard Murray and former United Technologies executive J. Michael McQuade, and Dopkeen was the initial director. It published its final report in March 2019.
But, in the fashion of modern software development, it had already released several advance features— including an October 2018 concept paper titled “Detecting Agile BS” that skewered the practice of using contemporary agile terminology about traditional legacy programs in order to make them seem more modern and relevant.
The paper went viral on Reddit, revealing that the practice of dressing legacy mutton up as agile lamb was not restricted to the DOD.
“It turns out everyone was doing it,” said Dopkeen. “We were really on to something and speaking to people who were frustrated, even in the commercial sector, by people pretending to be doing agile, but really holding on to the old ways of doing things.”
Along with the report, the board also published a set of six two-or-three page case studies, based on visits and interviews with ongoing acquisition programs, where they dissected the ways current programs of record were succeeding or failing at buying software.
“We really wanted to avoid ‘Innovation Tourism,’” said Dopkeen, which she defined as just telling the Pentagon, “‘You suck at everything, including buying software.’”
“There are smart people working very hard on this stuff. It’s not bad because they suck, it’s bad because it’s an extremely complex, difficult problem with multiple underlying causes,” she added.
The vignettes were designed “to make sure everyone understood there aren’t any easy answers,” and to help come up with recommendations that could address real problems in a detailed way. “We wanted to move beyond simply: ‘Don’t buy software the way you used to,’” she said.
By the time the final report was published in March 2019, Dopkeen was working on the House Armed Services Committee and helped staff a bipartisan effort to put the recommendations of the study into law in the 2020 National Defense Authorization Act, including:
- Directing a new policy on talent management for digital and software professionals.
- Directing a Department-wide strategy for software science and technology
- Granting authorities for continuous integration and delivery of software applications and upgrades
- Directing software development and acquisition training and management programs
The Color of Money
It took one more year to get a legislative fix to a different but very much related problem: The color of money.
“We had to get the appropriators on board to deal with that,” said Dopkeen, referencing the congressional committees that actually allocate funding, as opposed to those that authorize policy. The NDAA is drafted by the Armed Services Committees in each chamber. But the appropriations bills that allocate spending are written by the appropriations committees.
DOD has several different “colors” of funding, Dopkeen explained, representing the different purposes for which money is appropriated by those committees. Research, development, test, and evaluation, or RDT&E, dollars can be used for prototyping and testing new capabilities, but buying them needs procurement dollars. Once they’ve been purchased, money that’s spent on sustaining them has to come from operations and maintenance (O&M) dollars.
That might work for buying a tank, said Dopkeen, but software is different. And the budget planning process means program managers have to predict two years out what proportion of their activities will be dedicated to which color of money. “Every time I push an update, I am fixing bugs, which is probably O&M, but I am also introducing new capabilities, which should count as procurement, and I might be beta testing the next round of new capabilities, which is arguably RDT&E. It just doesn’t make sense for software,” she said.
Worse, the problem meant that even software projects which were funded by a big program of record like F-35 might find themselves unfunded if they needed the wrong kind of money in the wrong year.
“If you’ve decided or been told that you are O&M, and the program manager only has RTD&E dollars that year, then you are out of luck,” she said. “It is literally paralyzing to the point where it slows things down for years. Because the manager might be, ‘OK, I can request O&M funds next year,’ but in the meantime all the work on software has just ground to halt.”
To get around the problem, reformers created a new funding stream under RTD&E. The seven existing budget activities under that color, Dopkeen explained, spanned the full developmental lifecycle from basic research to deployment. BA8 was a special activity for software which could be used at any stage of its development, she said, “because software is never done.”
Part 1 looked at how the Air Force is embracing the new Software Acquisition Pathway. Part 3 will look at questions about the pathway and the flexible acquisition tools that accompany it.