"Aging Software"?--Tufekci Is Wrong?

NYTimes columnist Tufekci had a piece  discussing Southwest Airlines problems in managing their airplanes and personnel.  I just looked it up online on Thursday evening, finding to my surprise the Times has a featire in their comment system called NYTimes replies where she responded to some of the comment.

She puts the blame on "aging software" used to schedule pilots and crew.  I'm not sure why I immediately objected to the term, but here's my thoughts:

"Aging", which I am, means to me a deterioration of abilities, your body goes, your mind goes, you go. But software, once written (and debugged) remains the same, essentially forever. It's a set of ideas, of data, of information, which may be lost or destroyed, but will always do what it was capable of doing at its origin. 

Tufekci also uses the metaphor of building a structure, but using shortcuts, skimping on the foundation, etc.  Saving money now but setting the stage for problems later.  That's also wrong.   She mentions "technical debt", which is an interesting concept, but seems to me to conflate problems. 

I'd suggest the Southwest problem is a problem of "aging", but not in the sense I outlined above--deterioriation.  Consider the mature individual, the completed building, the proven software--each fits its environment, fulfills its function. There's a match of thing and context. Obviously the match isn't perfect; it may be flawed, corners were cut on the building or the software, the individual ends in the wrong occupation, with the wrong spouse, etc.)  As time passes, the building and the individual will deteriorate, they'll require maintenance to stay functional. But not the software, except as bugs appear. 

So the term "aging" has two sides: change for the worse in the entity discussed and change in the environment in which the entity operates, impairing the match between entity and environment.

For Southwest I suspect their software dates to the airline's early days, when it was doing point-to-point flights, basically within California.  It's expanded vastly over the years, getting lots of plaudits from customers.  The consensus seems to be they failed to spend enough on upgrading their software.  News media doesn't die into details, so we don't know whether the software ran into capacity limits, whether the system was never changed to use new technology (like generating text messages to personnel), whether the fundamental data model was flawed in light of the new environment and the impact of very bad weather, or whether all of the these factors were at work.

At the end there's a mismatch of capacity and environment. For humans the capacity declines and the environment changes. For software the capacity stays the same while the environment changes. 

The Lessons of Fairfax Scools

 As I understand it, the Fairfax school system had problems with remote learning back in the spring for two reasons--using older software (Blackboard, I believe) and running it on their own server instead of in the cloud.  The outcome was initially a fiasco, as the system couldn't handle the big load.

This problem, and my experience, suggests that the educators advising the Fairfax County School Board weren't paying enough attention to their infrastructure, likely because they regarded it as a distraction from the real job of educating students and running the system. 

As I used to say: "maintenance has no sex appeal".

Loren Becker Is Happy

As I've said, I've been lurking in the FSA Facebook group, watching the exchanges of hints, encouragements, etc. as the field offices struggle through CFAP (along with their regular work, all working from home or with restricted access to the offices].

One facet of the implementation effort is the use of Excel worksheets.  Back in the day, Loren Becker worked in KCMO. He became very proficient in Lotus 1-2-3, the dominant spreadsheet software of the day, and strongly urged us to use Lotus to develop test data, modeling what the results of System/36 software programs should be.  FSA isn't doing exactly that, but Loren would be happy, maybe is happy but I don't know, to see the extensive use of spreadsheets.

How To Do Big IT Projects

FCW has a post on how to do big IT projects, referring back to a study of 5 years ago.    There are four keys listed, but I can boil it down to one:

• Get the right bigshot personally involved from start to finish and be sure she has skin in the game, as in will lose her job if the project fails.

Early on I was involved in a project to bring computers to county administrative actions (payroll and related services).  The big shot then was the deputy administrator, management (Felber) who brought people together from DASCO and DAM to do the project.

In the middle of my career I was involved with implementing the Payment-in-Kind program in 1983.  The big shot then was Seeley Lodwick, who was the Under Secretary (following service in a previous administration as exec assistant to the Administrator, ASCS)  He pulled together lawyers and program people and kept on us until it was off the ground.  

By contrast other projects failed because either they lacked bigshot involvement and/or the bigshots moved on with a change of administration.

The Obama administration did one thing right--put Biden in charge of the stimulus package implementation and one thing wrong--ineffective leadership in rollout of Obamacare.

FSA Reorganization

I found two new notices from FSA interesting:

One was a reorganization into Safety Net and Program Delivery Divisions.  If I understand it correctly it splits program policy and automation into separate organizations.  The question of the best organization has been an issue ever since the original System/36 automation of county offices in the mid 1980's.  At different times and in different areas we've had policy and automation united in one person, or the responsibility in one section but with different people specializing in each, or in separate sections within the division.

When Jerry Sitter was division director in the mid 80's he split out a branch to handle automation under Mike McCann, with the policy in other branches.  In a way this followed the personnel--the policy types were mostly established DC specialists, people who'd come in from the field before the System 36 arrived.  The automation types were the early "SCOAPers", mostly program assistaants brought in under 2-year temporary appointments (which turned into permanent slots as time passed).  It also, IMHO, reflected an attitude among management that automation was a subject they didn't really understand or feel comfortable with, so it was best housed in its own shop.  There was a similar setup in the commodity loan area.

I always had my reservation with that setup--my argument was that a program specialist needed to know the whole span of operations.  Just as in the pre-automation days we'd work with MSD to get forms designed and printed, procedures written, cleared and distributed, regulations written  and published,  automation was just another area to learn and manage.  Looking back, I was reflecting my own belief in my abilities to do the whole scope of activities, and I was probably unrealistic.  But I still think there's a kernel of truth there--sometimes policy issues and automation issues become one and the same.

Which leads me to the second notice: on a workaround to handle multi-county producers, which seems to me to be an example.  Here the history of ASCS/FSA going back to New Deal days has been to work with producers on a county by county basis, unlike FmHA which tried to consider all of a producer's assets and liabilities when making a loan.  FSA has gradually been forced to move away from a county basis with need to enforce payment limitation.  My point is that a policy decision to apply rules on a producer basis, as with loans, and to allow producers one-stop shopping at one office, or at one web page, as with this notice, has big implications for automation, both in the design of the database and in the operation of the software.

USDA and FSA IT

The USDA CIO's office has a blog post touting their work towards "dashboards" consolidating access to data across the USDA.

Fedscoop notes in the second phase of the "lighthouse" project:

In this second phase, USDA plans to award contracts across the same five focus areas as Phase I — IT Infrastructure Optimization, Cloud Adoption, Customer Experience, Data Analytics and Contact Center — and an additional contract for support of its program management office.

 The same piece offers this quote:

"While the CoEs address a wide swath of IT modernization at USDA, the White House’s Matt Lira argued in June that what they all have in common is creating a better-functioning government.
“We are ultimately in the business of restoring the public’s faith in these institutions themselves,” Lira said.

I'm a little dubious of these efforts.  I do hope they are collecting metrics.  If I were feeling energetic, I'd file a FOIA request for available metrics of online usage. But then, if I were feeling energetic, I'd have better things to do than nitpick efforts.

USDA a "Lighthouse Agency"

That's from this FCW piece on some GSA IT contract awards:

"The awards support the first phase of work at five IT Modernization Centers of Excellence. Work will begin at the Department of Agriculture, which was selected as the government's "lighthouse" agency.
SIE Consulting Group will be working on cloud adoption, McKinsey & Company is tackling infrastructure optimization, ICF Inc. won two contracts for customer experience and service delivery analytics, while Kaiser Associates was awarded a contact center contract."

Don't know what "lighthouse" means--presumably a new bit of jargon that sounds good but turns out meaningless, like "tiger teams" back in the 90's. 

The Dream of Online Access to USDA Operations

In 1992 we had the dream of permitting farmers online access to ASCS, SCS, Farmers Home applications.  In the initial Infoshare pilot we found very limited adoption.  As I've observed from a distance the different embodiments of that dream over the years, I've always been curious how many farmers were really getting online and making use of the capabilities USDA provided. But despite my suggestions over the years, I'm not aware of any Federal site which publishes usage figures, so there's no way for a member of the public to see whether progress is being made.

Recently I found a clue, at least for FSA/NRCS/RD, thanks to the requirement for public notice on data requirements.  (The first time in my life I've really seen a value for that procedural requirement.)

Here is the Federal Register document from USDA on the information collection requirement for e-Auth.

"The USDA eAuthentication Service provides public and government businesses single sign-on capability for USDA applications, management of user credentials, and verification of identify, authorization, and electronic signatures. USDA eAuthentication obtains customer information through an electronic self-registration process provided through the eAuthentication Web site. The voluntary online self-registration process applies to USDA Agency customers, as well as employees who request access to protected USDA web applications and services via the Internet. Users can register directly from the eAuthentication Web site located at www.eauth.egov.usda.gov. The information collected through the online self-registration process will be used to provide an eAuthentication account that will enable the electronic authentication of users. The users will then have access to authorized resources without needing to reauthenticate within the context of a single Internet session."

 "Description of Respondents: Farms; Individuals or Households; Business or other for-profit; Not-for-profit institutions; Federal government; State, Local or Tribal Government.
Number of Respondents: 114,256.

 There's no breakdown given for how many of the respondents are actually farmers.  My guess would be about 80,000 to 100,000, which might be from 10 to 25 percent of potential users.

One Stop Shopping for Government Services

FCW has Steve Kelman's piece on a one-stop shop in China:

Some 20 different agencies are represented in the center. Lots of the work handled involves services for businesses, such as registration and approvals for establishing a new business, and various approvals related to construction. The center also provides a number of citizen services, such as applications for passports and work permits, and various transactions related to health insurance. Many, though not all, of the forms can be completed online. The in-person services are designed for people -- often the older and less-educated -- with questions or who need in-person assistance actually filling out a form.

Back in the 90's I had this sort of thing in the back of my mind.  InfoShare had that dream, and the Osage County office in Kansas was a step along the way.  I was ambivalent about the projects: moving to PC's and the Internet in county offices could only be justified by cost savings--good, which inevitably meant personnel cuts, but that meant a further decline in rural area jobs--not good.  One faint possibility would be a true consolidation of USDA services, where things like Skype (CU-SeeMe back then) could enable one employee to tap the expertise of others located in distant offices but then adding other services.  Problem was, government doesn't have that heavy of an impact on daily lives, particularly in rural areas.  Suppose the service center could handle social security--how many visits do the 2 or 3,000 residents of a rural county make to a distant social security office in a year?  And given the difficulty in getting USDA agencies working together, any further expansion at that time was a pipe dream.

FBI and Sentinel

I recall writing about the FBI's case management project back in the day.  Apparently they've learned some lessons on how to develop software, that is if one can trust this writeup.

SSNs and VA

FCW describes a bill to force the VA to stop using SSN's.  On this weekend I want to pat myself on my back--the SCIMs data design was intended to allow FSA to stop using them, and that was 20 years ago.  I hasten to add that I've no information or confidence that all FSA systems no longer use SSN's, or even that SCIMS doesn't.  The force of inertia and the interweaving of dependencies hard to overcome.

Why Don't IT Contractors Fail?

Just finished reading The Confidence Game,--anyone who enjoyed The Sting and the David Mamet film House of Games (which inspired the book) will enjoy it.  The author views con artists running con games as employing human traits, the desire to believe, the desire for meaning, the reluctance to cut one's losses, etc.we all share. With that perspective, I was struck by the question in my title.

How did I get there?  It's true, I believe, that most massive IT projects, possibly especially those in government, fail; the success rate is maybe 30 percent.  With that sort of track record, why do we in government keep creating and funding the projects and why can IT contractors get contracts to run them?  Surely if Elon Musk's space venture only got into orbit 30 percent of the time, he'd fail to attract venture capital.  But as far as I can tell (not very far), no big IT contractor has gone out of business because they can't get any more contracts.  So why?

Maybe they're running a con game?  After all that in the beginning there's lots of enthusiasm, enough to sweep agency employees, agency officials, even OMB and Congress into supporting the project.  A big project may paradoxically be easier to sell than a small one: a big project has meaning, it offers to change many things, to solve lots of problems, etc. etc. For IT projects it's likely that management and Congress don't really understand the nuts and bolts; they just know that people who should know, who seem to know, claim it can work, can succeed.   In the early stages it's easy to use the project focus to find more improvements to make, problems to solve, things to be folded into the project scope.  And once you're committed to a project, your reputation is involved, there's money been spent, meetings have been held, promises made.  And the problems surely are fixable, no need to abandon hope, just spend a little more money here, work some more hours there, move the schedule back just a little.

Finally there's a loss of confidence by those who should know, an increasing desperation, and Congress and management cut their losses, a process made much easier because there's been turnover in both areas so they aren't killing their own baby, it's some else's bastard child.  That can in turn make it easier for those who know (who haven't retired or moved to higher paying private jobs) to blame the big shots for not keeping the faith.

Meanwhile the IT contractors can move on to run another con.

Note: I don't necessarily think IT contractors are knowingly con artists; they may be conning themselves as much as their customers and they do have the occasional success.

The Paperless Office Redux

John Quiggin at Crooked Timber discusses a paper of his that says we've reached peak paper.

I remember in 1984 when the IT guys were trying to build their case for the System/36 in ASCS offices, one of them asked me about the paperless office.  Now we all know my memory is fading as rapidly as my age is increasing, so I don't really remember whether I was asked a leading question about it, and answered with a paean to the possibility that ASCS offices and farmers would get out from under the paperwork burden, or whether I was skeptical.  Most likely I wimped out and answered somewhere in between.

Anyhow, Quiggin thinks it's finally here, and that means the consumption of paper worldwide will increase.  It makes sense to me now: I'm typing on a 23 inch monitor with color and WYSIWYG; in 1984 it would have been a monochrome 14 inch monitor, etc. etc.   He points to the explosion in information (in 1984 it would have been an 8-inch floppy).  He goes on to discuss a parallel with Peak Oil, Peak Coal, and Peak Steel.

Request for Information on Acreage Reporting

USDA is:

investigating the use of a commercially available service to provide a reporting system to support Farm Service Agency (FSA) and Risk Management Agency's (RMA) common acreage and crop reporting compliance and other program needs. The objective of this RFI is to investigate whether commercial capability presently exists that can supply a viable alternative to the current Java based in-house system. 

The actual RFI is here.

Looking at the template for responding, I'm wondering whether there's been previous extensive meetings with potential respondents.  If there hasn't, the respondents may be at a loss to complete many of the items.  

Customer Self-Service Portal

FSA Notice CM-778 covers the release of the Customer Self-Service Portal, which permits FSA producers access to see (and print some) of their basic data.   This was mandated by Congress in the farm bill.  I guess they got tired of waiting for the different administrations to come through with their promises.  [/end snark]

Without access to the software my comments aren't worth much, but here goes, in no particular order:

•  I hope usage statistics are maintained so managers can get a feel of who finds this application useful, when, and how often.  My own guess is that it's not too useful, that's why in 1992 I resisted the idea of slapping together a similar application under Infoshare.  I think I realize now that was a poor decision.
• there's mention of the FSA-156EZ (that brings back memories)--I'd hope that they're moving towards covering all the data on the farm operating plan at some point
• the fact that the same data exists in different places, which is how I interpret the statement that SCIMS is not the system of record, says to me there's been deficiencies in management over the years.  A part of me wonders, though, whether my concern with storing data only once is a carryover from the old days of IT, and it isn't really that much of a problem with modern systems which replicate and synchronize data in different databases.  
• this portal seems to have been developed as a separate stand-alone app, simply added to the Online Services page.  Again I'm stuck in the past, but it would seem better to have an integrated approach with some logic behind it.

Why Didn't We Do a Joint Legacy Viewer

DOD and VA have had problems integrating their health care IT systems.  Now they've focused on a "Joint Legacy Viewer", which uses the principle of "write locally, read globally".

As is often the case, I think back to the 1990's (old geezers live in the past, you know) and the idea of integrating the USDA farmer  service agencies, at least in their IT.  At that time our (my) focus was always creating one database to serve the agencies.  In retrospect that was wrong. 

In 1992 we were demoing a mocked-up viewer of ASCS data.  Maybe we should have tried to build on that, rather than going for the big top-down solution.

Good Data Modeling Is Important--MP3s

I was usually ambivalent about the IT contractors working for ASCS/FSA. On the one hand I always thought I we could do the job better and faster ourselves, if management would only give us the dollars/people.  On the other hand I have to admit I did learn a lot, even if all the contractors didn't accomplish much (at least in my fallible memory).

One of the things I did learn was data modeling, normalizing data structures and the importance of having it right.  For example, the System/36 just used flat files, with multiple indexes to them.  While the IT people in IRMD and KCMO did a pretty good job in capturing the data needed for some of our activities, one big thing was missed: time, most notably time as reflected in crop/fiscal/program years, and the idea that we could be operating programs for different years at the same time. 

Of course you never get the thing totally right.  For example, our "name" structure assumed the standard WASP structure: first name, middle name, last name, and failed to allow for the naming structures present in other parts of the world or other cultures.

Mistakes in data modeling can have big consequences, as seen in the case of the metadata scheme for MP3 files, as discussed in this piece at The Atlantic.  Classical music is particularly affected.

What's Wrong With the Auditors?

The old question, from the Roman poet Juvenal, is:  "quis custodiet ipsos custodes?"

Earlier I posted about the new OIG report on FSA's MIDAS project. I've lost track of all the GAO and OIG reports critical of ASCS/FSA/USDA's automation efforts.  Juvenal's question doesn't quite fit--nowadays it implies some misconduct while my point is directed towards effectiveness.

In other words, given all those audit reports you'd think there would be some improvement over the years, but USDA and its agencies still seem to be ineffective in doing large IT projects.  I wonder why?

Some possibilities:

• IT procurement and development of IT systems keeps getting more complicated, so the bureaucracy's learning curve as embodied in the GAO/OIG reports doesn't gain on the difficulty curve of the projects.
• the USDA bureaucracy is incapable of learning, maybe because the policy officials turnover too fast, there's no insitutional memory, there's lack of ability or training, or something else.
• the auditors give bad advice, either misleading the bureaucracy on how to correct the problems or misidentifying the problems
• Congress fails to do good oversight--using the reports to hold the bureaucrats feet to the fire, or maybe they focus on the wrong issues.
• Congress fails to provide the money to do well
• the President and OMB fail to follow through on the reports
• the IT projects conflict with an outdated orgnaizational structure and culture.

I suspect all of these issues may apply.

OIG on MIDAS

OIG released an audit report on MIDAS last month.

Hat tip to this post from the Capital Press which represents what the media might make of the report.

"The federal government put a man on the moon, but 46 years later it can’t come up with a computer system for the USDA Farm Service Agency.
Such is the plight of the federal government in the 21st century. When it comes to computers, Uncle Sam is — how should we say it — a few bauds short of being online.

May have more thoughts when I read the report.

The Receipt for Service II

I've got a problem with the Receipt for Service implementation. Just in terms of bureaucracy and system design, county employees are asked to dual-task, do the work to support what the customer wants or needs plus as a separate operation record the history of the encounter. The extra work isnot likely to please the employee and the fact it's separate increases the likelihood it won't get done, undermining the validity of the statistic

A separate problem arises when it's the producer/farmer herself going online to do the work, as for example the new NRCS process.  How are those transactions going to be tracked?