Hedge-fund billionaire William Ackman is dropping plans to use his SPAC to invest in Universal Music Group, saying the Securities and Exchange Commission wasn’t convinced the deal met the rules for such vehicles.

Mr. Ackman said his investment firm, Pershing Square Holdings Ltd., would instead take a stake in Universal and become a long-term investor in the company. The U-turn is a setback for Mr. Ackman, who crafted a first-of-its-kind pact that set it apart from a wave of other deals orchestrated recently by special-purpose acquisition companies.

In June, Mr. Ackmann said his SPAC had agreed to buy a 10% stake in Universal from French media conglomerate Vivendi SE for about $4 billion. The deal valued Universal at some $40 billion. Typically, such deals involve a previously listed SPAC, or blank-check company, merging with an unlisted business, taking it public.

Mr. Ackman’s deal was different: New York Stock Exchange-listed Pershing Square Tontine Holdings Ltd., the SPAC, didn’t intend to merge with Universal but instead become a shareholder ahead of an already-planned listing by Universal in the Netherlands. People familiar with the matter said it was structured that way because of tax and legal implications for Vivendi, The Wall Street Journal reported.

The structure was hailed by some as a feat of financial engineering that also freed Mr. Ackman from some of the usual constraints of SPACs. In another departure from the typical SPAC structure, investors weren’t slated to vote on the deal.

Some observers, though, saw the structure as a concession to the reality that in an increasingly crowded SPAC market, and because of the relatively large size of the vehicle, Mr. Ackman wasn’t able to pull off a more conventional deal, as had been expected. In a sign of waning investor enthusiasm, shares in the SPAC have fallen 18% since the original transaction was announced on June 4.

Mr. Ackman acknowledged that Monday, saying the deal didn’t work for many shareholders. “We underestimated the reaction that some of our shareholders would have to the transaction’s complexity and structure,” he wrote in a letter to shareholders. “We also underestimated the transaction’s potential impact on investors who are unable to hold foreign securities, who margin their shares, or who own call options on our stock.”

He said Tontine was dropping the SPAC plan because of issues raised by the SEC with several elements of the proposed transaction. The SEC wasn’t convinced the structure qualified under NYSE rules for such vehicles, Mr. Ackman said.

The SEC scrutiny comes amid a broader review by the U.S. regulator of the SPAC structure. In May, the SEC said it was weighing new protections and guidelines amid worries that the blank-check structure appropriately protects small investors.

Universal is home to stars including Taylor Swift, Queen and the Beatles. It commands a market share of about 40% of the U.S. recorded-music business and around 30% globally.

Mr. Ackman had triggered one of the biggest guessing games on Wall Street in his hunt for a SPAC target. The activist investor had made frequent public pronouncements about a mystery deal. After launching the $4 billion vehicle last summer, Mr. Ackman eyed some of the biggest private companies, including Airbnb Inc., which has since gone public, Stripe Inc. and Bloomberg LP. The SPAC’s shares had been trading at a roughly 25% premium to their initial public offering price, reflecting the high hopes for a deal, the Journal reported.

Vivendi said Monday it approved the request to transfer the share-purchase agreement with the SPAC to a number of funds tied to Mr. Ackman. It also said those funds would acquire between 5% and 10% of Universal. If the stake were less than 10%, Vivendi said it would sell shares up to that amount to other investors ahead of the planned September listing.

Mr. Ackman’s Pershing Square Holdings will sit as a large shareholder alongside a consortium led by Chinese internet company Tencent Holdings Ltd., which owns a 20% stake in Universal.

From the Archives

Private companies are flooding to SPACs to bypass the traditional IPO process and gain a public listing. WSJ explains why critics say investing in blank-check companies isn’t worth the risk. (Originally published Sept. 30, 2020) The Wall Street Journal Interactive Edition

Mr. Ackman said his blank-check company would now pursue a conventional SPAC merger. Based on rules governing SPACs, it has 18 months left to complete a transaction. Mr. Ackman now faces the same challenge he had initially: finding a large and attractive target when the pool of candidates is relatively small.

Write to Nick Kostov at Nick.Kostov@wsj.com and Ben Dummett at ben.dummett@wsj.com