[SDL] Stack overflow when initializing video subsystem

[Urs Dönni]

Thanks for your answer.
I did play around with SDL_WINDOWID, in particular I did not set it
before I set the video mode. With that the application starts (with
SDL in a seperate window), but I only see a black screen when using
OpenGL (it works on someone else's computer though).

I'll keep looking what I find out about that matter, I'll let you know
if I have any more information, problems or solutions.

Regards,
Urs

On 3/29/07, Ryan C. Gordon <icculus at icculus.org> wrote:
>
> > That's where the problems start:
> > WinMessage (from sdl_sysevents.c) gets called with msg=0x00000081.
> > This message seems to be unknown, so the function HandleMessage gets
> > called. This calls DIB_HandleMessage (sdl_dibevents.c), which also
> > doesn't know the message code, and calls CallWindowProc again - which,
> > after some calls in user32.dll and so on, returns to WinMessage. So I
> > get a stack overflow because WinMessage calls DIB_HandleMessage which
> > then goes back to WinMessage.
>
> 0x81 is WM_NCCREATE, and it's a standard message, so everyone using SDL
> would be hosed with infinite recursion if there wasn't some other issue.
>
> CallWindowProc() is only used if there's a window specified via
> SDL_WINDOWID environment variable hack...and it would cause bad, bad
> things to happen if that ID matched the SDL window itself somehow.
>
> SDL will query for the SDL_WINDOWID window's event handling procedure
> and call it...and SDL won't destroy an SDL_WINDOWID window in SDL_Quit()
> or a second call to SDL_SetVideoMode().
>
> So here's my hypothesis:
> 1) App is calling SDL_Init() and SDL_SetVideoMode().
> 2) App is then setting SDL_WINDOWID environment variable to the ID of
> the SDL_SetVideoMode() window.
> 3) App is then reinitializing window with a second call to
> SDL_SetVideoMode()...maybe a resize event or something.
> 4) SDL doesn't destroy the original window created by the first
> SDL_SetVideoMode, since it sees SDL_WINDOWID.
> 5) When SDL is supposed to create a new one, it reuses the old and
> queries for its win32 event handler, which is the usual SDL win32 event
> handler.
> 6) Window gets an event, passes through SDL, passes to the original
> event handler, which _is_ SDL, so it passes through again, again, again,
> until stack overflows.
>
> I'm a little foggy on the details of exactly how this scenario happens
> in part #4, but I think that's roughly what's happening. I don't think
> this is an SDL bug in this case. Let me know if this info points you to
> anything in your app, though, and we'll go forward from there.
>
> --ryan.
>
> _______________________________________________
> SDL mailing list
> SDL at lists.libsdl.org
> http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
>
>