Sten Vesterli

Making IT Live up to its Promise

Author : Sten Vesterli

Which Snow do you Shovel?

,

Which snow should you shovel? We’ve just had a couple of inches of snow here in Denmark, which means that I will have to get out the snow shovel and clear the sidewalk. But I live on a small private road where the snowplough doesn’t go. Should I shovel the snow from the road as well? Should I clear the patio? There is always more snow I could shovel.

In any IT organization, there is an infinite amount of possible work. It is constantly snowing new tasks – security patches, new cloud services, new integrations, enhancement requests, bug reports. You can easily run out of space for more post-its on your Kanban board, but you will never run out of tasks. As Elton John sang in The Lion King: “There’s more to do than can ever be done.” As an IT leader, it is your job to decide what gets done. Do you have a policy for what gets done first? If you don’t, write one and distribute it to your team. That makes it easier for them to find and do the most important jobs first.

Hackers Almost Poisoned our Water Supply

What would be a truly scary computer intrusion? It would have to be something potentially lethal and something we weren’t expecting. Like hackers poisoning our water supply. But the water supply is highly secured, you say? Couldn’t happen, you say? Think again. It just did.

In a US city, hackers turned up the amount of sodium hydroxide that is added to the water. Adding a little is part of normal procedures, but the hackers turned it up to dangerous levels. Fortunately, operators immediately noticed, and countermanded the order.

Like in almost all disasters and near-disasters, there is a long chain of events that have to go wrong for the problem to occur. For example, you would have to be running an unsupported old Windows 7 installation. Check. You would need to keep remote access software running all the time. Check. You would need to have a widely shared common password. Check. You would need to have no firewall software in place. Check.

If you are a CIO, share the story of this almost-disaster. Security reviews are good, and would have caught most or all of these problems. But security awareness among users is better. Reminding people of the IT policy doesn’t work. But sharing a story of how it almost went wrong might change behavior.

Risk and Reward

,

Last week’s episode of my podcast Beneficial Intelligence was about risk and reward. Humans are very good at calculating risk and reward. That means we will do what is best for us, even if it is not the best for the company.

It is easy to create incentives for being fast and cheap, but hard to create good incentives for quality. That’s why we try to use incentives for speed and cost, but try to use QA procedures to ensure quality.

Incentives almost always win over procedures. As CIO, you need to make sure there are also incentives for quality. If not, you can be sure that your procedures will be circumvented, and corners will be cut.

Risk and Reward

, , ,

This week’s episode of my podcast Beneficial Intelligence is about risks and rewards. Humans are a successful species because we are good at calculating risks and rewards. Similarly, organizations are successful if they are good at calculating the risks they face and the rewards they can gain.

Different people have different risk profiles, and companies also have different appetite for risk. Industries like aerospace and pharmaceuticals face large consequences if something goes wrong and have a low risk tolerance. Hedge funds, on the other hand, takes big risks to reap large rewards.

It is easy to create incentives for building things fast and cheap, but it is harder to create incentives that reward quality. Most organizations don’t bother with quality incentives and try to ensure quality through QA processes instead. As Boeing found out, even a strong safety culture does not protect against misaligned incentives.

As an IT leader at any level, it is your job to consider the impact of your incentive structure. If you can figure out a way to incentivize user friendliness, robustness and other quality metrics, you can create a successful IT organization. If you depend on QA processes to counterbalance powerful incentives to ship software, corners will be cut.

Listen here or find “Beneficial Intelligence” wherever you get your podcasts.

Another Avoidable Disaster

,

Today’s totally avoidable IT disaster is found in the Slack app for Android. It turns out the app stored the user password in unencrypted plain text. That means that every other app on your phone had access to it, and it might now lurk in various log files on your device. Slack is red-facedly asking users to update their app and change their password.

This is an example of what happens when developers operate under tight deadlines and without adult supervision. Any competent IT development organization has code review procedures. If you are a large, high-profile organization that release apps to millions of user, any new release should have a separate security review performed by a security professional. But Slack insisted on letting their team operate without any guardrails. That means it was a matter of time before they ran off the track.

If you are a CIO, take a look at your systems list. For every non-trivial or externally facing system, there should be a link to the latest security review with a date and a name of a real person – outside the development team – who performed the security audit.

Avoidable Disasters

,

Humans keep causing avoidable disasters. I’m a pilot qualified to fly under Visual Flight Rules (VFR), and I am acutely aware that the number one cause of deadly crashes for pilots like me is to fly into clouds or fog. It turns out that it takes only 45 seconds for an untrained pilot  to become completely disoriented in clouds. Professionals train long hours to learn to override their intuitive feeling of what is up and down and trust their instruments.

Nevertheless, a professional helicopter pilot who had only VFR training flew his helicopter into the ground after getting disoriented in a cloud, killing himself, basketball icon Kobe Bryant, and seven others.

In IT, we also know how to do things. As an industry, we have decades of experience building solid, user-friendly systems and running IT projects. But we mysteriously insist on doing it wrong, causing one IT disaster after another. We think we can take a shortcut in order to meet our deadline, just like the helicopter pilot taking the shortcut through a cloud. As the CIO, you need to make sure you have a process in place to prevent people working on critical systems from taking shortcuts.

Convenience vs Security

, , , ,

The convenience of Microsoft Azure come with some serious problems. It seemed like a good idea at the time to store your cloud service credentials in your on-premise identity management solution. With Microsoft Active Directory and Microsoft Azure, you got exactly that convenience.

The only problem is that when hackers get into your on-premise system, they own your cloud instances too. The US Cybersecurity and Infrastructure Security Agency (CISA) has issued an alert about SolarWinds hackers using privilege escalation to gain access to the Microsoft Active Directory Federated Services (ADFS) and then producing OAuth tokens to move laterally to your cloud instances.

The SolarWinds hack shows that having intruders in your system is the new normal. You need to compartmentalize access, and storing all your access rights in one central place is a very dangerous convenience.

Looking into the Future

, ,

After 9/11, the US vowed never to be caught by surprise again. They created the Department of Homeland Security (DHS) to coordinate the intelligence gathering of a quarter of a million employees. On January 5th, the DHS intelligence summary said “Nothing significant to report.” On January 6th, a mob overwhelmed unprepared police at the Capitol and went rampaging through the building.

Companies are also regularly blindsided by events that in hindsight were obvious. You are gathering a lot of information, but it can be very hard to sift out actionable knowledge. Throwing a huge pile of data to a team of data scientists asking them to find the hidden patterns have rarely been successful.

For you as CIO to be able to peer into the future, you need to disengage from the daily running of the organization. It takes several days away from screens, news, email, and social media before the intuition you have can manifest itself, and generate new insights. Can you step away from your organization for several days? If not, your organization and procedures need some work.

https://www.wsj.com/articles/capitol-riot-warnings-werent-acted-on-as-system-failed-11612787596

Doing the Right Thing

,

Last week’s episode of my podcast Beneficial Intelligence was about doing the right thing. Google used to say “Don’t be evil,” but now they are struggling with their employees who want them to do the right thing. Amazon is unpopular for squeezing warehouse workers, and McKinsey paid $600 million for the role their advice played in the opioid epidemic in the U.S. They could have done the right thing, but didn’t.

As CIO, you also constantly have opportunities to cut corners and squeeze employees to work a little harder. But if you want to attract and retain top talent, you need to do the right thing. 

Doing the Right Thing

,

This week’s episode of my podcast Beneficial Intelligence is about doing the right thing. Google started out with a motto of “Don’t be evil” but that has fallen by the wayside. Occasionally, employees can enforce a change as when they stopped working on military AI. But Google doesn’t seem terribly committed, and their Ethical AI Team is falling apart after they fired the head researcher.

Amazon never promised not to be evil, and they are forcing their delivery drivers to do 10-hour graveyard shifts starting before sunrise and going until mid-day. They are trying to avoid tired drivers causing accidents by installing cameras and AI in the vans so the computer can detect when the worker is falling asleep behind the wheel and can wake him up.

As a CIO, you’re engaged in a war for talent. But you also need to meet your budget, implement hot new technologies like AI and maintain IT security. There is always an opportunity to cut a corner, roll out inadequately tested technology or squeeze employees so you can hit your goals this quarter. But if you want to be able to attract and keep top IT talent, you need to do the right thing.

Listen here or find “Beneficial Intelligence” wherever you get your podcasts.