Solana Goes Dark for 7 Hours as NFT Demand Surges

After an "insane amount of data" flooded the proof-of-stake chain, pushing validators out of consensus and grinding still block production, Solana stakeholders hurried to put the network back together Saturday night.

Bots stormed Candy Machine, a popular NFT minting tool, earlier Saturday, causing an unprecedented flood of inbound traffic: four million transaction requests and 100 gigabits of data per second — a network record, according to one source at the Solana Foundation.

This swarm pushed validators out of consensus for unknown reasons. At 4:32 p.m. EST, block production became impossible, and the network fell black. Validators rebooted the cluster at slot 131973970 by 11:00 p.m. EST, coordinating using Solana's Discord channels and a Google doc generated by one of the validators.

Co-founder Anatoly Yakovenko commended the validator community for spearheading mainnet recovery, despite the fact that he was traveling for much of the squabble. On Saturday, he was criticized on Twitter for being "missing in action" amid a network outage.

Unlike the 17-hour outage in September, the hard fork restart on Saturday did not result in new-and-improved code filling throughout the validators. They just resumed where the network had left off seven hours before.

Validators debated whether to write code that would temporarily halt Candy Machine transactions as they prepared for the restart. Some in the Discord questioned whether such a move amounted to censorship. Regardless, it would only work if two-thirds of validators agreed to participate. On Saturday night, it appeared that few did.

The interruption caused a bloodbath in SOL markets, which lasted only a few minutes. According to CoinGecko, Solana's native token hit a 24-hour low of $83.13 roughly three hours into the outage before rebounding back above $89 again.

According to a member of the foundation, Solana core developers have yet to address what went wrong Saturday or how the alleged botting attack bypassed established measures to stymie consensus.