What's Really Going on Here?

I took a weekend off to watch my son play at a tournament.  I've made some observations that got me thinking about how and what we evaluate on the field to make decisions to manage the game.
I observed a game in the morning in which I noticed something odd about the officials match control.  In this particular game, the white team was clearly more physical, but less skilled than the other team.  The green team had far more skill with the ball, but seemed to be slightly less physical.  The temperature of the game was clearly going up slowly but surely over the time that I watched.  White was getting increasingly physical, even grabbing at the back of no less than 3 players on breakaways.  What surprised me is there was neither an "Advantage!" from the referee nor a whistle.  White obviously noticed they could get away with more, so they increased the aggressiveness of their challenges.  At one point, I observed a scissors tackle on against a green midfielder that was shielding a ball into touch.  No foul.  Suddenly, the green midfielder turned on the white player and went nose to nose with him!  I heard him say something like "that better not happen again."  The center barely made a move toward the players to get things under control.  At that point, the AR motioned the player over.  I was close enough to hear him admonish the green player for his behavior.
Let's examine what was happening here.  By my observation, I see white playing increasingly aggressive, testing the limits with the referee.  The referee seemingly does nothing to show white exactly where the line is, so they keep testing.  At some point, we reach the green team's limit on what they will tolerate, so they lash out.  The referees punish the green team.
I am of the opinion that what one sees on the field is merely one way to evaluate what is happening.  The apparent frustration of one team should be noticed.  Perhaps the frustration is because we, as officials, are missing something that is happening.  It could be green is just getting outplayed, or it could be that white is indeed playing unfairly.  In the latter case, it is just a matter of time before green takes justice into their own hands as they see that the referee is either unwilling or incapable of dealing with the problem.  This is one of the reasons we see major incidents in some games.  In the game described above, the only caution issued in the game was for an "f-bomb" uttered by white after blowing a decent opportunity on goal.  We have to have our priorities right.  This is a U-15 game.  Where I referee, you are going to hear language on the field that is not directed at anyone.  Worry about the big things, like the temperature of the game.
The federation has a directive that is aimed at this sort of thing.  It is titled Game Management Model - Foul Selection and Recognition.  It discusses the idea of promoting game flow, versus the need to call fouls and control the game.  Check it out.