Our Eucalyptus Cloud in production (version 4.0.1) was successfully built. It was built but for many many months we had no applications to run on it so we didn't know about scaling and I/O throughput. We had plenty of memory and ram but the problem was we had a single Dell Powerconnect 2724 switch that was VLAN'd into two, one for the public traffic, one for the private node controller/CC network.

When running multi-node Map-Reduce jobs this is not a good idea as the mappers and reducers have to talk back to each other we flooded the switch which was also processing public traffic. So to fix this - we simply broke the private traffic off the VLAN and onto its own switch and let me tell you... massive improvement. *Logically

This reminds me of a quote I heard from Steven Murawkski...

"Temporary is permanent." </blockquote>

Which is exactly what happened here.

Lesson learned.