One constant of life is that mores and morals change all the time, so novels that were racy and daring (such as Ulysses or even Howell's Indian Summer, where out-of-wedlock sexual relations do not automatically lead to the female's death, as in Sister Carrie, for instance) are now seen as fairly tame. This is definitely the case for homosexual characters. Whereas it was fairly daring to include homosexuals in Radclyffe Hall's The Well of Loneliness or Molly Keane's Devoted Ladies, it really isn't any longer, at least not in Western fiction.
Novels that feature abortion are still on the rare side. It was particularly ground-breaking for Molly Keane to tackle an attempted abortion in Taking Chances and for Tess Slessinger to write about an abortion that didn't kill off the woman in The Unpossessed. Still, today they generally wouldn't be considered transgressive novels, with all that implies.
I'd say there are probably only four bright red lines that still mark a novel today as really transgressive: incest (between immediate family, not between cousins), corrupting of minors (i.e. statutory rape), bestiality and necrophilia. Possibly some of the more extreme forms of S&M might still qualify, depending on the level of detail but also whether there was full consent or faked consent under duress. Maybe a novel that featured cannibalism, with or without a sexual element.
Based on this classification, I am wondering how many transgressive novels I have read. While there is a lot of sex in most of Bukowski's novels, especially Post Office, it probably doesn't count. Perhaps Leonard Cohen's Beautiful Losers still is on the transgressive side of the line and probably Han Kang's The Vegetarian. I actually dropped a novel fairly recently when it became clear it was about father-daughter incest, though I did finish Alice Denham's Secrets of San Miguel, which includes a short story partially told from the perspective of a child abuser, which makes for very uncomfortable reading. I definitely couldn't finish an entire novel from that point of view, which is the same rationale for not even attempting to start Ellis's American Psycho.
I have to admit, I have never made it through Burroughs's Naked Lunch or any of Henry Miller's novels or de Sade's Justine or Juliette. I'm fairly sure these would all still make the transgressive list. I don't have any immediate plans to pick any of these up, though Naked Lunch will probably be the first I do actually read.
On the other hand, I finally read The Satiricon by Petronius and have launched into Apuleius's The Golden Ass, which are certainly pretty racy and would easily be classified as transgressive today. It's kind of mind-boggling that they were written so long ago (and then weren't completely destroyed during the Middle Ages). To round out the collection, I'll be rereading Marian Engel's Bear soon, which along with The Golden Ass and a small section of Kosinski's Steps (and perhaps an incident from Jim Carroll's The Basketball Diaries) are the only literary explorations of bestiality that come to mind.
I'm not sure I've actually read a book about necrophilia, though it is possible that some of the events in Kathy Acker's works qualify, even if some of them occur in dream states. Boris Vian's Foam of the Daze (aka Froth on the Daydream) features a wife who sort of falls into a coma and this may be close enough for (dis)comfort. It was certainly a quirky book. There are probably at least a few short stories or even novels about old men whose hearts gave out during love-making, though none come to mind at the moment. Since it is such a squicky topic, it is usually played for laughs when it comes up at all.