AITA for canceling my husband’s family Christmas plans?

My (F20) Husbands (M22) father invited us to his mom’s house about an hour away on Christmas Day. We already had plans to go visit my family four hours away for Christmas, especially since my sister is coming home from basic training. I decided to arrange our plans to leave the day after Christmas so as to spend Christmas with my husband‘s family.
Well as it turns out, my husband’s brother (M23) is refusing to go Christmas Day and his sister (F24)will not be in town in time. (she lives in another state and is traveling for the holidays)
My FIL made a group chat with all of us (his three children and their spouses) and proposed that we do breakfast/brunch on the 26th so as to accommodate all of the siblings. It’s been a very long time since all of the siblings and dad have been in the same place at the same time. Knowing this, I tried my best to be accommodating on my end, but it felt like everyone was making it difficult to be available and kept trying to push the time later and later, despite me and my husband saying that we needed to get on the road at some point on the 26th. 
BIL is now saying he’s working and the 26th will not work for him before 6 PM, but SIL is saying that she will no longer be in our area after 11 Ish.
BIL Is now saying he refuses to go because of my husband and most of the families political standing. (For reference my husband and most family leans more conservative while his brother leans more to the left.) He claims he doesn’t want to see us because of political views, but we already avoid politics around family in order to keep conversation happy and lighthearted. Most of us can have different views and still have discussions, but not everyone can handle these discussions appropriately so we avoid it all together.
I got fed up with the arguing and the back and forth, and sent a lengthy message about immaturity, last minute planning, etc. and said we will not be attending whatever may or may not get planned the 26th, but will be at Christmas and I’d love to see them all. My husband supported my decision, but I’m worried about being the a**hole in this situation. It’s my family but it’s also NOT my family, and I don’t want my husband to eventually resent me for cancelling his families plans.
This EXTRA sucks because my husband and I were both excited to see our nieces (4 and 2) and meet our newest nephew (3 mo.) We got them all sorts of gifts and were so excited to see them and love and them and give them all the presents, so we’re just super disappointed. AITA?