You weren't cowardly and into your whiteness. You were simply a victim of what keeps many white people from being as open about others as they probably wish to be. Prejudiced white people are more-than-ready to punish and shun those who step off the all-white reservation, and fear of that punishment works to keep most white people in line. Imagine what they, not to mention the world, could be, if they only let go of their fear.

You weren't silent. You were sensible. You can't take any chance of escalating an encounter that could become nasty, especially as your children were there to witness it.