Payroll software provider Apex Human Capital Management suffered a ransomware attack this week that severed payroll management services for hundreds of the company’s customers for nearly three days. Faced with the threat of an extended outage, Apex chose to pay the ransom demand and begin the process of restoring service to customers.

Roswell, Ga. based Apex HCM is a cloud-based payroll software company that serves some 350 payroll service bureaus that in turn provide payroll services to small and mid-sized businesses. At 4 a.m. on Tuesday, Feb. 19, Apex was alerted that its systems had been infected with a destructive strain of ransomware that encrypts computer files and demands payment for a digital key needed to unscramble the data.

The company quickly took all of its systems offline, and began notifying customers that it was trying to remediate a security threat. Over a series of bi-hourly updates, Apex kept estimating that it expected to restore service in a few hours, only to have to walk back those estimates almost every other time a new customer update went out.

Contacted Wednesday by an Apex client who was nervous about being unable to make this week’s payroll for his clients, KrebsOnSecurity reached out to Apex for comment. Ian Oxman, the company’s chief marketing officer, said the ransomware never touched customer data, but instead encrypted and disrupted everything in the company’s computer systems and at its off-site disaster recovery systems.

“We had just recently completed a pretty state-of-the-art disaster recovery plan off-site out and of state that was mirroring our live system,” Oxman said. “But when the ransomware bomb went off, not only did it go through and infect our own network, it was then immediately picked up in our disaster recovery site, which made switching over to that site unusable.”

Oxman said Apex hired two outside security firms, and by Feb. 20 the consensus among all three was that paying the ransom was the fastest way to get back online. The company declined to specify how much was paid or what strain of ransomware was responsible for the attack.

“We paid the ransom, and it sucked,” Oxman said. “In respect for our clients who needed to get their businesses up and running that was going to be obviously the quicker path.”

Unfortunately for Apex, paying up didn’t completely solve its problems. For one thing, Oxman said, the decryption key they were given after paying the ransom didn’t work exactly as promised. Instead of restoring all files and folders to their pre-encrypted state, the decryption process broke countless file directories and rendered many executable files inoperable — causing even more delays.

“When they encrypt the data, that happens really fast,” he said. “When they gave us the keys to decrypt it, things didn’t go quite as cleanly.”

One of Apex’s older business units — ACA OnDemand — is still offline, but the company is now offering to move customers on that platform over to newer (and more expensive) software-as-a-service systems, and to train those customers on how to use them.

Experts say attacks like the one against Apex HCM are playing out across the world every day, and have turned into a billion-dollar business for cyber thieves. The biggest group of victims are professional services firms, according to a study by NTT Security.

Ransomware victims perhaps in the toughest spot include those offering cloud data hosting and software-as-service, as these businesses are completely unable to serve their customers while a ransomware infestation is active.

The FBI and multiple security firms have advised victims not to pay any ransom demands, as doing so just encourages the attackers and in any case may not result in actually regaining access to encrypted files.

In practice, however, many cybersecurity consulting firms are quietly urging their customers that paying up is the fastest route back to business-as-usual. It’s not hard to see why: Having customer data ransomed or stolen can spell the end of cloud-based business, but just being down for more than a few days can often be just as devastating. As a result, the temptation to simply pay up may become stronger with each passing day — even if the only thing being ransomed is a bunch of desktops and servers.

On Christmas Eve 2018, cloud data hosting firm Dataresolution.net was hit with the Ryuk strain of ransomware. More than a week later on Jan. 2, 2019, this blog reported that the company — which had chosen not to pay the ransom and instead restore everything from backups — was still struggling to bring its systems back online.

One dataresolution.net client said the company didn’t succeed in rebuilding its server or turning over his company’s database stored there until Jan. 9 — 16 days after the ransomware outbreak.

“From my understanding it was another two weeks until all of the clients were rebuilt,” said the customer, who works as an IT manager at a benefits management firm that used dataresolution.net and its now transitioning away from the company. “The vendor never provided any analysis on how it occurred and how they would prevent it from occurring again.  Other than different antivirus and not allowing RDP connections to the internet they don’t seem to have put any additional safeguards in place. They did not proactively offer any compensation for the outage. I am in the process of documenting the business financial impact to request a ‘credit’ at the same time as planning on bringing the system in house.”

For its part, Apex is still trying to determine how the ransomware got into its systems.

“That’s where this forensic analysis is still going on,” Oxman said. “For us, the emergency response team literally worked 48 hours straight getting our systems back up, and secondary to that is now trying to figure out what the hell happened and how do we prevent this from happening again. We had just completed a security audit and we were feeling pretty good. Obviously, these cyber hackers found a way in, but I’m sure that’s how every company feels that gets hit.”

Here are a few tips for preventing and dealing with ransomware attacks:

-Patch, early and often: Many ransomware attacks leverage known security flaws in servers and desktops.

-Disable RDP: Short for Remote Desktop Protocol, this feature of Windows allows a system to be remotely administered over the Internet. A ridiculous number of businesses — particularly healthcare providers — get hit with ransomware because they leave RDP open to the Internet and secured with easy-to-guess passwords. And there are a number of criminal services that sell access to brute-forced RDP installations.

-Filter all email: Invest in security systems that can block executable files at the email gateway.

-Isolate mission-critical systems and data: This can be harder than it sounds. It may be worth hiring a competent security firm to make sure this is done right.

-Backup key files and databases: Bear in mind that ransomware can encrypt any network or cloud-based files or folders that are mapped and have been assigned a drive letter. Backing up to a secondary system that is not assigned a drive letter or is disconnected when it’s not backing up data is key. The old “3-2-1” backup rule comes into play here: Wherever possible, keep three backups of your data, on two different storage types, with at least one backup offsite.

-Disable macros in Microsoft Office: Block external content in Office files. Educate users that ransomware very often succeeds only when a user opens Office file attachment sent via email and manually enables Macros.

-Enable controlled folder access: Create rules to disallow the running of executable files in Windows from local user profile folders (App Data, Local App Data, ProgramData, Temp, etc.)

Sites like nomoreransom.org distribute free tools and tutorials that can help some ransomware victims recover files without paying a ransom demand, but those tools often only work with specific versions of a particular ransomware strain.