An Interview with Deloitte Analysts William Eggers and Steve Berman
The nonpartisan Government Accountability Office issues more than a thousand recommendations to Congress and executive agencies each year to improve operations and increase fiscal efficiency. According to the Comptroller General’s annual estimates, the Congressional watchdog agency routinely achieves a return-on-investment of more than $100 in financial benefits for each dollar that Congress spends on GAO’s budget.
In March, Comptroller General Gene Dodaro submitted to Congress a budget request of $744.3 million dollars. As Congress considers this request, lawmakers have an opportunity to evaluate the role that GAO plays in the policymaking and oversight process.
In 2015, Deloitte published a groundbreaking analysis of 26 years of GAO reports and recommendations using text analytics to evaluate GAO’s recommendations as data. The analysts studied more than 40,000 recommendations that GAO made between 1983 and 2014 and drew insights from 1.3 million pages of GAO reports.
William Eggers, the Executive Director of Deloitte’s Center for Government Insights, and Steve Berman, a data scientist at Deloitte, agreed to be interviewed about their 2015 study and discuss current opportunities for Congress to better leverage GAO’s work to improve government operations.
Lincoln Network: Can you provide some background on how you approached this study. How did you use text analytics to draw insights from all of those GAO reports?
Bill: We analyzed over 40,000 recommendations made by GAO to federal agencies from 1983 through 2014. Each of these recommendations was made on the basis of detailed, evidence-based GAO reports meant to spur specific actions within agencies. These recommendations span almost every issue in government, from information security to environmental resources. Using text analytics, we asked seven key questions to assess GAO’s effectiveness as a change agent in government and to understand what areas GAO tends to focus on. We also calculated the success rate of agencies implementing GAO’s recommendations.
If you are interested in learning more about our methodology, you can find it in the Appendix of our study.
Lincoln Network: Your analysis focused on answering seven key questions, including whether GAO’s recommendations were effective in driving change within agencies. What were your main findings?
Steve: We found that implementation of recommendations exceeded 80% over the period, and that this rate was very stable over time. In addition, the focus areas examined by GAO also remained steady. While the overall success rate was encouraging, there were some areas with greater rates and others that lagged. Procedural reforms due to legislation or administrative changes had limited impact, but transparency has improved over time.
Bill: We also found that the problems that cut across multiple agencies, impact numerous parts of an agency’s duties, or deal with high-profile issues were found to be difficult to address and repeated examination in any particular area didn’t improve implementation rates.
Another finding: agencies have the highest success rate in implementing GAO recommendations in four key areas: information security, education, information technology, and equal opportunity.
Lastly, the top four issue areas investigated by GAO (audit, government operations, law enforcement, and national defense) stayed constant throughout three decades.
Lincoln Network: Your report explains that GAO has a lower success rate when its recommendations involve certain issues, such as data or cross-agency challenges. You also found that agencies don’t improve their compliance when GAO makes the same recommendation multiple times. Do you have any thoughts about why these kinds of recommendations are less effective?
Bill: Whenever a recommendation cuts across multiple topic areas or multiple agencies, agency leaders should devote additional financial and political resources to it if they reasonably expect the recommendation to be implemented. To improve the chances of implementation, GAO could prioritize laying out more-specific roadmaps for issues relating to data, multiple agencies, or the involvement of higher-ranking officials.
Question: Your report mentions that agencies often take four years to implement GAO’s recommendations. Do you have advice for Congress or GAO could encourage agencies to answer GAO’s recommendations in a more timely manner?
Bill: GAO could address this issue by setting target completion dates for implementing each recommendation and then making real-time data available to the public showing how long it is taking each agency to implement GAO recommendations. This could motivate agencies to quickly address GAO recommendations and realize the benefits they deliver to the public. In addition to setting specific deadlines, GAO could also further motivate agencies by classifying both noncompliance and extreme tardiness as failures. If this is not done, it is easy to lose a sense of urgency, and recommendations can languish. GAO could consider giving recommendations a “criticality” score that allows its analysts to sort out whether an agency is struggling with major items or just “nice-to-have” items. The lack of such a score for assessing the importance of an individual recommendation is a weakness in GAO’s methodology, and the agency would likely benefit from adopting one.
Lincoln Network: Do you have recommendations for how Congress and/or GAO could improve agencies’ compliance when GAO makes repeated recommendations to an agency about an ongoing challenge or problem?
Bill: Given agencies’ generally high success rate in resolving individual GAO recommendations, Congress should view GAO as an effective scalpel but not a panacea for the federal government’s longstanding problems. GAO may sometimes succeed in helping agencies make meaningful changes, but problems often exist that are beyond GAO’s reach. Addressing the root causes of these problems may often require Congressional intervention, as well as a sustained focus on changing the culture within an organization.
Lincoln Network: Beyond helping Congress and the public to better understand GAO’s value, you described your 2015 report as providing an example of how agencies “can better structure their internal oversight activities to quantify accountability and drive results”? Can you describe what you were hoping federal agencies and Congress could learn from your analysis?
Bill: Agencies should look to mimic GAO by investing in “keeping score”—that is, tracking where their recommendations are succeeding or failing. They should plan on a one-time investment in converting reports into an electronic format that is friendly to text analytics. To facilitate that, agencies should establish a coding structure for reports that has a stable core but retains the ability to adapt to changes in management interest and the influence of current events. This will allow for analyses that go beyond filtering reports along the lines of preset categorizations and enable an agency to respond to specific leadership interest in how oversight has been changing over time. To drive speed of implementation and enhance accountability, agencies can make real-time dashboards that score recommendation completion rates and speed of implementation accessible to the appropriate audiences.
Question: Since you published your report, GAO has created a new Science, Technology Assessment, and Analytics team. One of the STAA’s missions is to advance GAO’s use of data science and analytics in its auditing. Given your 2015 study and your expertise in the use of data science, do you have recommendations about how GAO could use data analytics to increase its impact?
Steve: Cutting edge analytics techniques would help the GAO process and review increasingly large amounts of data. While the initial focus of the analytics teams would be descriptive in nature, developing predictive models would allow GAO to highlight potential areas of concern for review or remediation. Production models that scored risks across various services would allow for automated early warning that would help mitigate severe problems more efficiently than humans could. The analytics would ensure that domain experts could focus their time on areas more likely to be problematic, while permitting standard, low-risk processes to run automatically.
Question: One of the big problem areas that GAO has identified across agencies is improper payments. The federal government misspends about $175 billion per year (with about $75 billion reported as a loss). Could GAO use these new data analytics to help agencies curb improper payments?
Steve: Definitely. Many organizations, including some governmental, use a variety of methods to prevent or control fraud, waste and mismanagement. The simplest methods would code expert domain knowledge to identify common indicators of excess payments. Predictive models could find higher risk entities based on a large number of variables. Pattern detection methods can take standardized or unstructured data (like free-text expense information) and cluster into groups of transactions. Some of these groups might in themselves be of greater risk, and certain transactions on the fringes of these groups may indicate abnormalities from common expenses. Unsupervised methods like clustering could uncover new forms of fraud, which is important in an ever-evolving world. A robust defense to improper payments would involve a combination of all of these methods.
Lincoln Network: Thank you both for taking the time to answer these questions and share your insights.
