The critical need to create new applications almost in real-time

The critical need to create new applications almost in real-time

Michael Mestrovich, Principal Deputy Chief Information Officer, U.S. Department of State

Michael Mestrovich, Principal Deputy Chief Information Officer, U.S. Department of State

When people think of nimble and agile technology companies, their minds gravitate to Silicon Valley, Seattle, Washington, and maybe even Austin, Texas. But few would ever dare to consider the United States Department of State (DOS), or potentially any Federal Government agency for that matter, as spry, responsive, and able to turn on a dime.

Back in late January of 2020, as the true scope of the COVID-19 epidemic was beginning to unfold, the U.S. Department of State began to plan for the eventuality that the virus would spread and would disrupt all facets of life as we know it. As a precautionary measure, the Department leadership asked the Bureau of Information Resource Management (IRM) to develop information technology contingency plans that would allow for the over 100,000 members of the Department’s globally dispersed workforce to work remotely. The Department of State has a long (231 years), proud tradition of service to the American people, and operates in 195 countries via 275 foreign posts and throughout the United States at nearly 300 domestic facilities. The information technology infrastructure that supports all of those personnel in all of those locations is based primarily on PCs. While the Department has for many years enabled the workforce to utilize mobile computing technologies, e.g. tablets and mobile phones, these solutions were not well suited for the day to day conduct of diplomacy and for the many unique mission functions of the Department. It was one thing to be able to check e-mail remotely via mobile devices, but a wholly different proposition to enable the global workforce to create content, collaborate, publish reports, transact foreign aid and the myriad other special activities that are conducted by Department personnel every day.

Knowing that time was not on our side, IRM embarked on a three phased approach to enable telecommuting capabilities globally for the entire Department. The first strategy undertaken was to increase the capacity of the Virtual Desktop Infrastructure (VDI) that had been supporting remote access. The VDI capacity was initially scaled to support about 5,000 concurrent users and this was immediately expanded to support 8,000 users by scavenging and repurposing compute and storage infrastructure. At the same time, orders were placed for additional hardware that would ultimately expand that capacity to 15,000 concurrent users. Next, the team ordered several thousand new laptops and created an automated imaging and configuration process that was able to provision hundreds of new laptops each day. And finally, IRM enabled access to the Department’s Office365 instance via a web browser, coupled with multi-factor authentication (MFA). Through these three remote access technologies, the Department enabled telework to the entire global workforce in a matter of days.

“Giving people the capability to telework was one thing, providing them with the tools they needed to work effectively was another”

Giving people the capability to telework was one thing, providing them with the tools they needed to work effectively was another. Toward this end, IRM was able to provision video and audio collaboration systems across the Department almost instantaneously. Working with our solution providers, we scaled out these cloud-based collaboration systems to accommodate the global workforce and then enhanced network and firewall capacity and routing algorithms to better accommodate the increase in network traffic. Ultimately, by June, the Department was conducting about 1,000 multi-party meetings per day with over 15,000 unique internal and over 7,000 external (non-DOS) daily participants. It was quite normal for many of these meetings to host well over 500 personnel, as these technologies were being used to communicate to the entire staffs and ex-pat communities at many embassies and posts.

None of this would have been possible, however, had it not been for the hard work the Department had undertaken over the previous few years. The State Department had been moving to Office365 for over 18 months and finally transitioned the last users in the Fall of 2019. Additionally, the Department was consolidating on one Enterprise Mobility Management suite and had already transitioned the majority of the Government Furnished Equipment (GFE) and Bring Your Own Device (BYOD) user base over to that new solution. And perhaps most importantly, the Department enrolled its entire user base into an MFA solution in December of 2019 which became the lynchpin in enabling remote access via a web browser to the Office365 instance and allowing for expanded VDI access without the need to distribute Personal Identity Verification (PIV) readers and hard tokens.

While these efforts were underway, the Department also had a critical need to create new applications almost in realtime. As the crisis expanded, repatriating US citizens from abroad became one of the primary focuses of the Department. Luckily, the Department had just brought on-line a Software as a Service (SaaS) capability that allowed developers to quickly deploy new applications daily. This team successfully delivered 12 new, custom applications in the span of four weeks that did everything from tracking the repatriation of US citizens to mapping out the various travel restrictions between countries to monitoring the caseload and mortality rate of COVID-19 in various countries. What had previously been done with spreadsheets was now done in an automated platform that could correlate different data sets and create visualizations to provide new insights for the Department leadership to help focus the Department’s response. In the end, the Department successfully repatriated over 100,000 American citizens from abroad using a combination of charter, commercial and Government aircraft.

Now to be fair, we had our share of growing pains. Some of the systems introduced new workflows and there were some cumbersome periods of adoption. Some users tried multiple remote access solutions and found that for the application they used, only one or two were well suited. And we continue to iterate on these to improve their functionality based on user feedback. But the results of a Department-wide telework survey showed us that 80% of the workforce was satisfied with telework and hoped that the Department would continue to allow for more telework options once the COVID crisis dissipated. All in all, this was a true team effort that should make every taxpayer proud. The thousands of men and women across the Department worked tirelessly to make telework a reality which enabled the mission of the State Department to continue unabated.

Weekly Brief

Read Also

Change Management: Part 1: Don't Bump The Fish Bowl

Cory Godwin, Director of Jail Operations,Walton County Sheriff's Office

Cyber Mutual Aid

Dennis Tomlin, Chief Information Security Officer,Multnomah County

How Local Governments in Rural America are Combatting Cybersecurity

Shane McDaniel, Director of Information Technology,City of Seguin

EMS: An Alarming Situation in the Healthcare Landscape

Dave Edgar, Assistant Chief Emergency Medical Services,City of West Des Moines

Disaster Recovery in Emergency Management. No, Not That Kind!

Brent A. Olson, CEM, Director, Office of Homeland Security and Emergency Management, City of Phoenix

To Curb Climate Change and Design for Whole Life Carbon, the HVAC Industry Needs a More Transparent Roadmap

Rebecca Delaney, P.E., Associate Director and Operations Leader for Sustainable Engineering Studio, and Luke Leung P.E., ASHRAE Fellow, LEED Fellow, BEMP, P Eng, Director of Sustainable Engineering Studio, Skidmore, Owings and Merrill