Assalamalikum brothers and sisters in Islam.

Please help me out with this doubt and clearify if possible with Hadith reference.

We read our prayers together at our house .
My brother is usually the Imam and I along with my nephews stsnd behind along with my other brother.
The Female members pray behind us.
So the incident was like this it was magrib time and after he gave the Athan we all gathered in the room to pray.
The ladies had made the Suf but I was the only ond in the front Suf behind the Imam. Now my nephews are small children 7 yrs , so he was coming in about 5 secs and so was my other brother . I started the Iqamah but then I got shouted upon that I need to go in front as no one else was there . But I said to them that untill I finish the Iqamah they will join us. But then I got shouted at and then when I went ahead I asked the same doubt. My brother got angry and said im not gonna make you read and so in the anger I also left the Jamaat and read seperately . I know i was wrong to leave the Jamaat but I wanted to know that if we are 100% sure someone is gonna join us in the next 5 secs that too before the Imam says Allah Hu Akbar then can one person be at the back.
Thank you for reading and answering.