Shadow IT vs Silo IT

IT departments in larger companies seem to have a mandate to find and stamp out Shadow IT whereever it can find it. First, a bit about Shadow IT. Imagine that your department actually does something for the company. Let's say that it's at headquarters, and manages sales. Let's say that there is a sales force spread across the country, and this department is responsible for getting the most from it. One thing it might attempt is to track how each sales person, each small group of sales people, and each region are doing. The idea is that a group manager writes up sales data for each sales person in the group. The data is tagged with what the group is and what the region is. Then, the data will be analysed back at the headquarters.

Doing this analysis by hand is going to be time consuming (expensive, remember, Einstein proved that time = money), tedious (which was never stops businesses) and error prone. So you hire a programmer, and tell them what you want. She says, we need a way for the managers to enter the data. We'll write a web application where they'll enter the data, and it will get stored back here in a database. There's no budget of any kind, so she finds a PC that's so old, slow and has so little memory that no one wants it. She loads Linux on it, installs an Apache web server, and a Postgres database server and configures them. Then she writes a form, and a Perl script that takes form data and stuffs it into the database. She announces that it's in production, send this URL by email to the managers, and let them at it. And despite using a four year old PC instead of a quarter million dollar server, the application is responsive and reliable.

Soon, data is coming in. By then, she's written a quick report that adds up sales by group and region for each month, in a table. The analysts say it's great. For one thing, they're no longer swamped in paper. While our intrepid programmer adds logins and real security, someone gets the idea that there should be a report that graphs the groups and regions over time so that one can pick out which ones are doing better and which ones are doing worse. She grabs some graphing routines from CPAN and puts something up.

What's wrong with this picture? Well, corporate says that having a server sit under someone's desk is bad. For one thing, there's no backup. She counters that she does have backup. There's a second PC with Linux, and it gets a copy of the primary's data every night. They also complain that Perl isn't supported, that Java is the corporate standard. Further, they say that they can save money by having the server administrated by the Unix group, the database administrated by the database group, the web server administrated by the Web group, security managed by the network security group.

Two years later, the application is rewritten in Java, it's hosted in the server room, and it follows all the company rules. Everything's great, right? Except that the original developer has been spending the past two years bringing the offshore people up to speed on the app, instead of adding the new features that the customer now knows they want, and the app still doesn't do everything that the original did, and it cost a million dollars, and nothing was saved, because the original developer is still on the payroll.

Why did the rewrite, with 30 developers take so long? For one, they had to use the Silo IT development model. What's Silo IT?

Corporate management has labeled the department model Shadow IT. It's a negative name. It casts the endeavor as murky, hard to follow, uncontrollable, and underhanded. So, it's only fair that the popular alternative also has a negative name. Silo IT refers to this idea that under the idea that pooled resources are efficient resources, the endeavor is split up into areas of responsibility. Typically, this is systems administration, web administration, database administration, network administration, architechture, web framework, mentors and application developers. Each of these groups forms their own sub-department.

If a developer wants to create an application, it needs to follow the rules laid down by architects, and it needs to fit in with the framework. Now, there's a considerable amount of information to go through, and typically there are documents, hundreds or thousands of pages of documents, that one needs to go through to absorb it. No problem - there are mentors that can help, right? Wrong. The mentor's job is to make sure that the rules are followed. The developer gets the application to work, then submits the code to the mentor, who comments one sections that don't follow the rules. It's like an exam. Mentors never consult with the developers to get them going. Their job is to slow the deployment process down by several days while they review the code.

What does the developer do while waiting? Well, perhaps they have another application to work on. If they do, they'll generally find the going slow and error prone, because each shift in focus takes several hours for their brains to catch up. So, as often as not, they do nothing.

So, let's say that the developer needs a new table in the database. They submit a ticket, and it enters the database administrator's queue. In ten days, an administrator takes the ticket out of the queue and creates the table. The developer then tests it to make sure it's what they needed, then submits a ticket to get the change made to the test servers. Another ten days. Then ten more days to get the change into production. Mind you, new tables can have no negative impact on production, so it could have been there at the start. And in Shadow IT, that's what happens. Except that it doesn't take thirty days. It takes maybe half an hour in Shadow IT. It's not hard to see where this is going. The developers are easily thirty times more productive in a Shadow IT organization than in a Silo IT system. So, it's not that the Shadow IT developer is brilliant, she just doesn't have the same ball and chain attached to her ankle while she swims the English Channel.

So what if you can't find a developer who knows how to install and administrate Linux, Apache, Postgres, and knows a web aware language? Well, according to Fred Brooks, the department should build a surgical team. Maybe it takes two or three people. But since they are a single team, they can still be responsive. In particular, the database administrator can spend time learning the current applications, and tune both the database and the queries to the real needs.