Talks for Broadcom Inc. to buy SAS Institute Inc. have ended after the founders of the closely held software company changed their minds about a sale, people familiar with the matter said.

The Wall Street Journal reported Monday that the companies were discussing a deal that would value SAS in the range of $15 billion to $20 billion, including any debt. Following the report, Jim Goodnight and John Sall, who co-founded SAS decades ago and still run the company, had a change of heart and decided not to sell to Broadcom, the people said. Whether another suitor for SAS could emerge isn’t clear.

Some SAS employees saw the company as a strange fit for efficiency-focused Broadcom, some of the people familiar with the matter said. SAS is known for a tightknit culture and has a sprawling North Carolina campus with amenities including a yoga studio and a disc golf course.

Cary, N.C.-based SAS sells analytics-, business-intelligence and data-management software to enterprises. The company traces its roots back to the 1960s, when universities teamed up to analyze troves of agricultural data through a program called the Statistical Analysis System.

Broadcom, a semiconductor powerhouse built largely through acquisitions, has been on the hunt for a deal to beef up its presence in the corporate-software market. Its chief executive, Hock Tan, said earlier this year the company would look at buybacks and possibly debt repayment, if it didn’t make an acquisition by the end of the fiscal year. That typically ends in late October or early November.

Write to Cara Lombardo at cara.lombardo@wsj.com