I'm at my wits end regarding my 5-year old son. He doesn't follow rules at my house and his mother only makes it worse. : Parenting

I'm at my wits end regarding my 5-year old son. He doesn't follow rules at my house and his mother only makes it worse.

TLDR: 5-year old son doesn't follow rules at my house, frequently disciplined. Mother is no help. He's now declining visits. I'm about to give up.

I have a 5-year old son from a previous relationship. He comes every other weekend and every time he comes, we have issues. He does not follow rules. Frequently, he will be released from time out and break the same rule 2 minutes later. We (my wife and I) remind him of the rules every time he comes, yet he spends a ton of time in time out and frequently loses privileges like screen time.

On Thanksgiving, he was banging toys around and we told him to cut it out. He did it again and ended up breaking the TV. After time out, he resumed his rough play, so we had to ban him from those toys.

His mother has admitted multiple times that she provides no structure and few rules. She has even said that our rules are "dumb" in front of our son. So I get no help from her, if anything, she's making the issue worse. His grandparents spoiling him rotten is adding to the frustration. I tried speaking to them, but I got shut down with a "I raised 4 kids, don't tell me how to handle kids."

"You know your children are growing up when they stop asking you where they came from and refuse to tell you where they’re going." - P. J. O’Rourke

Our rules aren't extreme, either. No rough play inside, no running inside, don't touch things that don't belong to you without asking, etc. We may have to be on top of these rules more than most families, due to a small apartment and having up to 4 kids at a time, but the rules are similar to what he experiences in kindergarten. Speaking of school, I got a report from his teacher in the first month that he often disrupts the class and doesn't keep his hands to himself.

I just don't know what to do with him. His mother is no help, his grandparents are no help, the courts are no help. I just now messaged his mother about this weekend and I got "He said that he doesn't want to come." This makes the third time in a row that he has voiced that he doesn't want to come (I assume because he spends so much time in time out). I tempted to just let it go and wait for him to miss me. What can I do?