Kakao needs to get back to basics after ‘wake-up call’: CEO

Kakao co-CEOs Namkoong Whon, left, and Hong Eun-taek bow in apology after the company’s services crashed over the weekend at a press conference held Wednesday morning at the company’s office in Pangyo, Gyeonggi. [WOO SANG-JO]

 
PANGYO, Gyeonggi — One Kakao CEO resigned Wednesday and another CEO, who will remain on the job, promised to get the company to focus on basics again after a dramatic “wake-up call” over the weekend, in which almost all the company’s offerings failed due to the lack of effective and robust backup.
 
The resignation of co-CEO Namkoong Whon is effective Oct. 19, according to a Kakao disclosure. Co-CEO Hong Eun-taek will remain at the top post as the sole CEO.  
 
The resigning co-CEO made the announcement at a press conference Wednesday at the company’s office in Pangyo – about 1.8 kilometers (1.1 miles) away from the server farm that crashed and took the company’s services offline for many hours and in some cases days.
 
“I am aware that it will take a painstaking effort to restore the trust lost,” Namkoong said. “Following the event, we will overhaul and revamp the entire Kakao system.”  
 
Namkoong, who was named co-CEO in January, added that he will continue in an advisory capacity at Kakao and lead an emergency committee formed in the wake of the outage.    
 
Kakao servers were knocked out as a result of a Saturday fire at an SK C&C data center. Messenger, banking, transportation, payment and e-commerce services were unavailable, degraded or intermittent for more than 10 hours as backup systems failed to kick in fast enough.  
 
Five days in, not all services have been fully restored, although major features of KakaoTalk, Daum, Kakao Pay and Kakao Bank were online. SK C&C issued an apology Wednesday, adding that its power system is back to normal.
 
At around 3:20 p.m. on Saturday, a fire broke out in a basement room of batteries that supplied power to the data center. The room is managed by SK C&C, although the company could not confirm whether the batteries were configured as an energy storage system. It was not able to immediately identify the manufacturer of the batteries.
 
Firemen requested electricity be cut for the entire building at 4:52 p.m. so water could be used to fight the fire. The outage extended to servers used by other companies, such as Naver and IBM, Hong explained.  
 
Kakao executives said the company was not properly prepared for a disruption on such a large scale — where all 32,000 servers housed in a data center were cut off from power. Kakao runs a total of 90,000 servers in four different data centers including the Pangyo center, which acts as a main server center for Kakao’s family of apps, according to Hong.  
 
At the Pangyo site, SK C&C runs the building and Kakao owns and operates the servers themselves.  
 
“This accident was a wake-up call for us to take care of the basics,” Hong said, “Since KakaoTalk is used by the vast majority of Koreans, the service is deeply related to public interest. We will be more keen in fulfilling our responsibilities.”    
 
Hong confirmed that the backup system failed to activate and sought to clarify the key issue: whether the company’s disaster recovery architecture was up to standard.  
 
“The slow recovery can be attributed to the fact that major tools and systems used by developers were not dualized, although we built a dual backup system for data storage and software,” the co-CEO said, adding that it will take around two months to correct the faulty system.
 
When asked about the type of a backup plan that the company employs at the Pangyo center, Hong said that the backup was at another site but that manual intervention was required to activate the system.
 
It was not clear whether Kakao’s backups were “hot,” in industry jargon.  
 
A hot-site backup plan has mirrored servers standing by at the ready and can be brought online in a short period of time. With cold-sites or warm-sites, recovery takes time.
 
Hot sites are considered essential for the continuity planning of corporations that maintain data. They are expensive to build and maintain as storage and bandwidth costs are high and considerable expertise must be deployed to develop and configure code to keep the sites in sync and available on a real-time basis.
 
Kakao will compensate users, both those that paid and those that used free services, like KakaoTalk.
 
“We will consider providing compensation for users, businesses and different parties affected by the disruption as well as subscribers of paid services,” Hong said.  
 
The company promised to spend more on server infrastructure to prevent a recurrence. It aims to complete its own data center in Ansan, Gyeonggi, by 2024 and another one in Siheung, Gyeonggi.  
 
Shares of Kakao rose 0.81 percent to close at 49,800 won ($34.9) Wednesday and those of SK that owns SK C&C slid 0.24 percent to close at 204,500 won. 

BY PARK EUN-JEE [park.eunjee@joongang.co.kr]