[SDL] [GSoC] Automated Test Suite Monthly Report

[René Dudfield]

On Thu, Jul 2, 2009 at 6:40 PM, Edgar Simo<bobbens at gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
>
> As you may know, I'm doing the GSoC project of creating an automatic
> test suite to find and detect regressions in SDL automatically [1]. This
> will be a benefit in the future as regressions and bugs will be easier
> and faster to detect.
>
> As you may know I've had final exams until this Monday (included), which
> have prevented me from coding as much as I would have liked to. That
> however is now over so I can code freely until the completion of the
> project.
>

sweet!

> Some information of the design, after looking at the libraries that
> Pierre Phaneuf suggested [2], I've decided to write up a very simple one
> similar to CuTest [3]. I've called the simple testsuite maker SDL_AT for
> SDL Autoomatic Testing, it currently consists of a few functions and
> basically divides tests into what I call suites and cases. You create
> multiple testsuites each with multiple testcases and it'll report what
> goes wrong/right. Currently it's very simple, but if needed it can be
> easily expanded to write success/fails into a database or create output
> already formatted for posting to the bug tracker [4].
>

Cool.  Using a test suite with a build bot is *very* useful for
writing cross platform code, and having it tested automatically.  Then
people writing code on linux can have there changes tested by the
build bot automatically.    (NOTE: this also encourages people to
write tests for their code... so they can have the buildbot test it
for them).



> The code is currently located in my branch [5] specifically in
> SDL/test/automated [6]. It contains only a few tests for:
>
> 1) RWops
> 2) Surface - Only primitives and blending primitves at the moment.
>
> Things on rendering tests. As per Sam's indication, I am considering
> that all implementations are going to draw exactly the same pixel per
> pixel and am testing against prerendered images considering that the
> current Linux implementation does indeed work well (please inform me if
> it doesn't).

Note, that SDL can draw differently depending on the video mode set.
ie, 24bit, or 32bit.  So it won't render exactly the same in all
situations because of this.


> To avoid possible disturbances caused by loading external
> files, the images are stored in C structs so the values can be accessed
> directly with no need of parsing nor loading.
>

Consider for some tests reusing the data available in the current examples?


> Following systems I will add (most likely in this order):
>
> 1) Surface - Finish blitting and others.
> 2) Rect - Collision testing.
> 3) Pixel Formatting - Conversions and the like.
> 4) Video - Rendering tests (not sure if it makes sense since it's most
> likely OpenGL and OpenGL isn't pixel perfect)
>
> The problem which was mentioned before is that there's a lot of stuff
> I'd like to be able to write test suites for, but I'm not entirely sure
> if it's possible to do without writing platform specific tests and sort
> of duplicated the code in SDL for that platform. An example would be the
> entire event system (would need user interaction most likely). I'm not
> really sure what the best way to approach this would be.
>

For pygame(python bindings for SDL) we found that it is impossible to
test some subsystems automatically.  Since they require input/output.

Interactive tests is what we called a set of tests that require human
interaction.  They just take a Y/N answers to questions so people can
run through them very quickly.  By default the 'interactive' tests
aren't run of course - and we try to make the interactive tests only
when it's impossible to test otherwise.

Also note, that SDL wasn't designed to be testable(but is fairly
testable)... however now is the chance for you to say 'this particular
API is not testable... please consider these changes so it's possible
for me to automatically test it'.

You can 'mock' out some parts, and test most of the API (the 1.2 API anyway).

You might find the python tests useful for you in some cases, to see
how we tested particular parts.  Or if you have trouble testing one
part, please feel free to ask how we did it - we might be able to give
some insight.


cheers,