Serious Fusterclcuk by UK Bank in IT migration scheme

Kick Back and Relax in the Cheers! Forum. Thoughts on life or want advice or thoughts from other pca members. Or just plain "chill". Originator of da Babe threads.
Post Reply
User avatar
wvjohn
Posts: 9238
Joined: Wed Nov 22, 2000 7:09 am
Contact:

Serious Fusterclcuk by UK Bank in IT migration scheme

Post by wvjohn »

Long story short - moving 1 billion+ records from system OLD to System NEW and the whole thing goes sideways. It may be there was no provision for fallback to old system if major fail on new one. People can't access accounts, wrong info, other people's info shown, etc. Lots of twitter screens in original
https://www.nakedcapitalism.com/2018/04 ... -data.html

TSB Train Wreck: Massive Bank IT Failure Going into Fifth Day; Customers Locked Out of Accounts, Getting Into Other People’s Accounts, Getting Bogus Data
Posted on April 24, 2018 by Yves Smith
From time to time, we’ve written about how bank IT is a systemic risk waiting to happen. Major financial firms have legacy code at the core of their systems that they can’t migrate off at acceptable costs and risk (numerous banks have had a go at this issue, and projects wind up being shelved; at best, they can port only some products or customers off the aging systems). Readers, even ones who are in IT but not in banking, sometimes scoff at what we have said.

The disaster at TSB should serve as a big wake up call. The very short version is that a UK bank, TSB, which had been merged into and then many years later was spun out of Lloyds Bank, was bought by the Spanish bank Banco Sabadell in 2015. Lloyds had continued to run the TSB systems and was to transfer them over to Sabadell over the weekend. It’s turned out to be an epic failure, and it’s not clear if and when this can be straightened out.

It is bad enough that bank IT problem had been so severe and protracted a major newspaper, The Guardian, created a live blog for it that has now been running for two days.

The more serious issue is the fact that customers still can’t access online accounts and even more disconcerting, are sometimes being allowed into other people’s accounts, says there are massive problems with data integrity. That’s a nightmare to sort out.

Even worse, the fact that this situation has persisted strongly suggests that Lloyds went ahead with the migration without allowing for a rollback. If true, this is a colossal failure, particularly in combination with the other probable planning failure, that of not remotely adequate debugging (while there was a pilot, it is inconceivable that it could have been deemed to be a success if the testing had been adequate).

Let’s turn the mike over to the Telegraph:

Customers of TSB continued to complain of being unable to access their accounts on Tuesday morning as the bank’s IT fiasco dragged on into its fifth day.

TSB confirmed its 1.9m customers were still facing “intermittent” problems when attempting to log in to online services after a bungled switch over from a system the bank had been renting from its old owner Lloyd’s Banking Group.

Customers had been warned the transfer of 1.3 billion customer records to a new system could affect services from 4pm on Friday to 6pm on Sunday – but the disruption continued overnight and into Monday and Tuesday.

A look at Twitter suggests that “intermittent” is not exactly accurate. These tweets are from this morning:


Lola Moreno LaVey
@Lucky_Lola_
@TSB Finally get the option to log in and it says my username/password are incorrect! They definitely aren’t Is anyone else having this problem or have I been hacked as well as messed about? #TSB #TSBdown


Martin
@apphancer
Are @TSB developers seriously coding live changes in production? I've found loads of ugly debugging logs in the code which are being spat out in the browser console. I don't dare to imagine what mess they have in the back-end. #TSB #tsbdown


The overview is that Lloyds Bank, one of the former four British “clearing banks,” made a series of acquisitions. Its first large deal was for TSB. As a result of the crisis, it acquired HBOS, which itself was a merger of Halifax Building Society and the Bank of Scotland.

European banking regulators deemed the combined banks to be too big and concentrated. Rather than sell some products or some regional operations, Lloyds TSB decided to demerge TSB. That was already a bit, erm, gutsy, since TSB would presumably be well integrated from an IT perspective by now and therefore not necessarily so easy to hive off. From a 2013 Guardian story:

Twenty years after disappearing from the high street, the TSB bank will reappear in towns across the UK on Monday when more than 630 branches that were Lloyds units on Friday reopen with a new identity…

Lloyds has been forced to split off and rebrand the TSB branches by the EU as a result of the £20bn of taxpayer money pumped into the bank during the 2008 bailouts. It has pledged to turn TSB back to its heritage as a “local” bank. The 631 branches were scheduled to be sold to the Co-operative Bank but the collapse of that deal earlier this year means it is likely the TSB network will be floated on the stock market as a separate bank.

The TSB is being unveiled a week before an industry-wide current account switching service is launched to reduce the hassle of moving a bank account.

One would also assume that any buyer would make sure that the buyer’s and seller’s systems were sufficiently compatible. In the US going back to the 1990s, many a promising-seeming banking deal was scuppered because the integration issues looked too hairy. So as much as it’s easy to point fingers at Lloyds, Sabadell is the one that should have made the call as to whether they could successfully port the data and needed routines from the Lloyds/TSB systems. As indicated, that has long been a major due diligence issue in US bank mergers.

Sabadell appears to have decided to move the TSB customers onto an entirely new system.

As one of our house IT expert Richard Smith pointed out, based on a Computerworld interview with the Sabadell CIO:

If I read this right (and it’s not just hype) the target system is “brand new”, which is to say, untested (or nearly untested). That’s a whole other dimension of extra risk.

“We are in the process of cutting the rope, and the first step is having a core platform up and running. We are now working with Lloyds Bank on the data migration,” Abarca told Computer Weekly. “We have built a new technology platform for TSB, but this is not just a technology refresh or upgrade of an existing core banking system. It is a brand new core banking system.”

They *did* do a pilot but clearly it wasn’t effective, for reasons we are yet to understand:

Proteo4UK, as the new platform in the UK is known, was rolled out to the bank’s staff in November 2017 with a full range of banking services. It will move to a full roll-out in the first quarter of 2018.

That suggests that the Sabadell knew its IT systems were not compatible with TSB’s and rather than do the sensible and normal thing, which would be to nix the deal, it went ahead based on the naive belief that it could build something new that would work. It’s hard enough to shake down a “new” retail bank IT system work, let alone roll it out by porting a ton of data from an old banking system into it.

We may have a better sense soon, but there are indications not just of data mapping problems, which potentially may be hard to isolate but not necessarily hard to fix once found, but of data corruptions, such as wildly incorrect account information (zero balances, incorrect currencies, massively inflated mortgage amounts, and e-mails saying that there are no records of recent direct debits). If there are indeed problems on the books and records level, and as we suspect, the changes can’t be rolled back, this could produce a world of hurt for customers.

Needless to say, we’ll be revisiting this topic once the press has more intel on the nature and severity of the IT mess. And as always, reader sightings and observations are of great help. But at least so far, this looks as if this might not be a gunshot wound, as bad as that would be, but gangrene.
User avatar
FlyingPenguin
Flightless Bird
Posts: 32773
Joined: Wed Nov 22, 2000 11:13 am
Location: Central Florida
Contact:

Re: Serious Fusterclcuk by UK Bank in IT migration scheme

Post by FlyingPenguin »

That's ALWAYS been my worst nightmare. I had a client where every couple of years I had to upgrade Quickbooks Enterprise on a server and 10 workstations.

I NEVER did this without first making a complete image of the hard drive on the server, AND also backing up the company data (even before QB does it's own backup during the conversion). If anything went wrong I'd roll it all back. Took extra time, but it was worth the peace of mind.

Same thing for replacing file servers. Keep the old one on standby for a month or two. Anything goes wrong with the new one, throw the old one back in there with the latest data backup.
"Turns out I’m 'woke.' All along, I thought I was just compassionate, kind, and good at history. "

Image
User avatar
Err
Life Member
Posts: 5842
Joined: Thu Nov 22, 2007 11:54 am

Re: Serious Fusterclcuk by UK Bank in IT migration scheme

Post by Err »

Well FP, that's was a competent IT person would do. You'd think the banking industry would have regulations that govern this type of move.

This would be a nightmare for the customers. I surprised there hasn't been a run on the bank. I know I'd sure move my money out.
User avatar
wvjohn
Posts: 9238
Joined: Wed Nov 22, 2000 7:09 am
Contact:

Re: Serious Fusterclcuk by UK Bank in IT migration scheme

Post by wvjohn »

REading a couple of articles on this it appears that a rollback on a system like this won't work because of the daily transaction level - once it reaches a certain # of new entries, money in/money out , it can't be unwound.
User avatar
wvjohn
Posts: 9238
Joined: Wed Nov 22, 2000 7:09 am
Contact:

Re: Serious Fusterclcuk by UK Bank in IT migration scheme

Post by wvjohn »

The beat goes on with this one. People on holiday abroad - screwed. Autopay mortgages - screwed. Some people are even getting their balances displayed in $USD.

https://www.theguardian.com/business/20 ... -it-crisis
User avatar
wvjohn
Posts: 9238
Joined: Wed Nov 22, 2000 7:09 am
Contact:

Still a mess - here's an update

Post by wvjohn »

Post Reply