Let me take a stab at this. I have been the one that had to step in for others who were struggling with IEM on monitors, not so much on wedges. some things to consider:
1) The #1 rule applies to non-vocal channels: ONLY put something in a wedge/IEM if it is asked by the band.
2) Once things are 'dialed in" you should walk the stage and hear what the band hears. If they are IEM then you will only hear what is acoustically loud at each part. THAT gives you a reference point and you can make suggestions based on this information, but only if necessary.
3) One possibility of a problem was there was a mic, like an audience mic, overhead, HH, spare or acoustic guitar to a DI that was ON but on the stand and adding hell to the mix when not in use, or ANY high-gain inputs were set with TOO much gain. That was piped into IEM and blurring everything else in the mix.
4) Ensure your gain structure is correct. If you still having a difficult time keep in mind you will get the best signal to noise by using lower pre-gain, if you must turn up the bus output to compensate. I am not talking severe here, perhaps 1/2-1 tick on the gain knob or 1-3 db and at the output add 1/2-1 tick of additional gain over unity.
5) sounds like you were overwhelmed at the soundcheck. Several things:
a) have paper/sharpie available, and learn/create shorthand symbols for sources, write things down as each person tells you what they want if you are not in front of the board. ALTHOUGH I would say it sounds like you didn't come off as confident/competent by taking the time to introduce yourself to the band, let them know "(I am a fellow musician / or NOT which will help so you each know how to communicate - like do you say "do you need more "SSSSS", highs, or 6k bump. )
b) Tell THEM how you will proceed, and work this out with A1. Normally the A2 will run through things, and the A1 will ride along, but you two should have this agreed upon upfront.
c) It sounds like you haven't given the FOH/A1 guy comfort that you are proficient because after the gig you two should ALWAYS do a debrief on how to improve. That goes with the band as well, you should have checked with them during the show and gotten a thumbs up. You should always check after the gig as well, apologize that they didn't have a comfortable gig that is what you take pride in and will absolutely do better next time, but only after you brainstorm with them how to fix it in the future. This is the ONLY way to get better at the job.
d) Prior to soundcheck, tell them "we will start with (vocals or kick..., move on to.. etc), Give me a moment to set gain structure while I am you need to raise your hand if you want that input in your IEM/wedge, keep it there until it is the right volume. If it is too loud (during festivals esp), then point at the floor. Repeat until it is right, then a thumbs up with eye contact when perfect, I will get each of you set when there are no more chances for me or FOH then I will call out the next input. If I miss anything or anyone SPEAK UP. If there is time for a song or a few bars, play one that involves all inputs, then I will call out each performer (names are best, instruments are fine) to check with each of you to make changes. During the set, if you need more of someone, make eye contact with me and walk toward them. For the drummer or others who can't move, I will come to you at the appropriate time to check on you."
e) hope this goes without saying, have a talkback on the ready sending to all channels and use it so everyone on stage can hear you as you run through everything.
f) If you don't take control, then they will dump on you like they did and then chaos ensues.
During the show, make sure you make eye contact with every player at least once during song break and give them the thumbs up/down or just ask if you can.
Save your settings (thumb drive, paper, whatever, keep notes of what they had for setup so next time you can set them up/ dial it in and everyone is happy for a quick soundcheck.
Make sure they have confidence in you and you in them. Make them comfortable that they will be taken care of. Do NOT take your eyes off the band during their set. Make sure they know they can count on you when someone new starts off a song and if they need more of themselves you should be already on their vocal knob ready to adjust. these are the kind of things that make you different/better.
Spend some time getting to know the people in the band if they are present during setup. You should have googled the band, got their names, and listened to their top songs the day before the gig.
6) Pre-show when you are alone, besides checking everything (including hearing all wedges) you should dial-in wedges with only the vocal mic in front of it. Having a digital board and pad makes this easier to understand. get a feel for what is unit gain for your vocal mics (like an audix om-7 needs a LOT more gain for unity than a om-3, let alone a 58.) get a feel for how much to turn up aux send to monitors to get a reasonable level, and how far can you push it.
7) Ring out monitors to learn each mic, placement in the room, what frequencies in the room are problematic (what is a problem in FOH might be different than on stage depending on venue.)
8) If you will have IEM mixes, you should have an IEM for yourself so you can hear it HOW they hear it. This isn't always possible, so use headphones on the output of a belt pack before the show. Keep in mind, that some transmitters have a headphone jack, but they rarely sound like post-compassion.
my hands hurt, I may add more. Hope this helps, and perhaps do some searching on youtube for monitor engineers and learn, watch them work. Some great info can be found on Dave Rat's youtube channel, and I am sure there are plenty of others.
Best of Luck
Rob