Assuring quality in the Ganeti code-base

Iustin Pop

June 2012


Ganeti is…

The problem

How to ensure that the code is correct? How to do it without proprietary tools?

Many possibilities:

… but they do not work in isolation

Ganeti development life-cycle

Except for the design, each step needs to use automated methods in order to assure a consistent quality across the code base: people are creative, but not consistent; computers are “dumb”, but persistent.

Each phase can introduce/find bugs in multiple parts of the software:

Feature requirements/Design phase

Programming language

Code style


Code reviews

Continuous build

Integration tests

Integration tests #2

Canary process

Production use

Common issues

  1. dynamic typing in Python coupled with incomplete test coverage means some paths are exercised only in some corner cases
    • after seeing the Nth type error in error reporting code path, we've added "simulation modes" to a few operation which behave as if most errors happened (to test error reporting)
  2. data validation issues
    • require validation at Ganeti/system level, or between Ganeti components?
    • how to degrade gracefully in have of real-world issues?
    • implemented ht mini-type system for Python, used also for data validation:

      ht.TListOf(ht.TElemOf([1, 2, 3]))
      ht.TAnd(ht.TNotNone, ht.TString)
      ht.TDictOf(ht.TElemOf(["a", "b", "c"]), ht.TInt)

And many others…


Thank you for listening!