experchange > delphi

skybuck2000 (01-07-19, 03:09 PM)
Hello,

There is some speculation (LOL) going on that Windows 10 OS Schedular has abug which affects threadripper 2 model 2990wx.

So I set out to create an application (in Delphi) which can create as many threads as the user wants and simply increment a counter on each thread.

Then this counter and delta since last read is display per thread.

(I created this new tool today in just one day, still in beta/testing phases).

Now that the tool is functioning I have started investigating the thread schedular of Windows 7 and also the way Delphi operates and it's debugger.

I just extended the tool so it allows "naming" of threads for debugging purposes, it works by calling a function which sets the thread name when it starts running.

What I am seeing I am not liking.

It's very weird.

Conceptually the code looks as follows:

// initialize settings

// start enable

// stop disable

*** start button routine ***

// disable start button

// create/start threads

// enable stop button

*** stop button routine ***

// disable stop button

// stop/destroy thread

// enable start button

*** display routine ***

// display counters in VCL/form/grid

Now I am seeing some weird results which is as follows:

1. The start button is pressed

2. The threads are created (BUT NOT REALLY ?).

3. The stop button is enabled before the all threads are created.

This allows the user to press the stop button, which then tries to destroy the threads.

But this is where it gets weird.

Not all threads were created yet ?!?!?!?!?!?!?!?!?!?!?!?!?!?!?!?!?!?!?

Especially since the threads immediately starts running, it uses an interlock increment to increase the counter.

and the display/main thread is trying to read all threads with interlock read, not sure if I implemented this read correctly but think so.

To me it seems like "beginthread" api of windows which Delphi uses in TThread class is not actually creating the threads immediately... I tried setting suspended to false... but they still seemed "queued".

Only after a while are these threads created... this seems weird to me... and allows execution to happen out of intended programmer order.... very strange.

Definetly a strange coding bug. I can probably solve it easy but this is not really my points (?).

This "queueing" behaviour is simply not documented ?!???

At least not here:



Is it documented anywhere ? Or have I stumbled upon something strange ?

My problem just to be exact is:

create 64 threads returns before all 64 threads are actually created.

So many many instructions later the threads are created...

meanwhile all kinds of other instructions are executed which should not really have been executed yet.... cause they may depend on these newer threadsbeing created/being there and so forth... so kinda weird...

Perhaps I have even stumbled upon a short coming/bug in windows schedular or thread creation methods/beginthread in combination with interlock increment and interlock read... which might be interferring/delaying thread creation ?

Perhaps blocking/locking this thread creation process.

Now this "process" is exaggarted by the debugger... cause it slows down thesystem/processor/application etc.

So I added a third button which calls the start and stop button as quick aspossible, so I only have to press this third button and it's indeed a problem.

The stop button becomes clickable/executionable before the threads are fully created resulting in a strange situation/bug where the program seems to hang (?)

No significant CPU activity happens, so it's a strange hang bug... probablybecause "destroythread" is being called on non-existing threads cause theynot created yet ?! How weird is that ?!

I checked another source of information:



Again no mention of this queueing behaviour that I am seeing.

With debugger running the process is much faster but it's still happening just at a much faster pace then a human would be able to recgonize, but it'sthere.

By quickly pressing button3, this problem can still be triggered which is kinda weird, tried to protect it with interlock even... I don't think this is supposed to happen... unless application process messages changed in delphi and is now actually multi threaded or something and running out of order, usually main thread runs code in sequence/serial... so kinda strange.

Or perhaps some VCL buttons and other code are making the main thread run beyond where it should be or something which would also be weird... I do so more than one thread starting though in debugger which is not a thread I created... kinda strange... 3 threads to be exact.

Anyway... my mom interrupted me two times already during this investigated via phone... but anyway =D

I tried to find some information about this, but nothing turns up... everybody seems to assume these threads are creates instantly... which simply does not seem to be the case.

So what I saw I didn't like and I had a bad feeling about it and still havea bad feeling about it and starting to get a little bit a bad feeling about.

Now maybe it's just a Delphi issue, but I kinda doubt that a little bit.

Anyway I suspect there might be a "spectre" exploit related to "multi-threading" but in combination with "create" and "destroy" of threads causing of creation/time delaying.

Or perhaps incorrect destruction of threads might be exploiteable but causing this condition to occur in every day software where no waits were deployable to wait for actually thread start(s).

But perhaps there is even more... perhaps in combination with "locks" or perhaps even windows schedular... or windows creation queue doing something weird.

Anyway I am not an expert in windows api and windows thread creation technology/queues and mechanism.

So if anybody out there knows some web links to/for futher information about it I am all ears ! ;)

I will continue my investigation into this windows 7 thread schedular... hope that won't take too long... and once that is done I will release the tool so it can be tested on threadripper 2 model 2990wx, hopefully somebody will run it then on that thing.

To confirm or deny these rumors of a windows schedular bug... pretty fascinating in itself.

I must warn for now... this "tool" seems to bottleneck the processor greatly because of all the interlock increments.... but for now I see no reason why not to do it this way... it's bascically about maxing out the processor.... but yeah... now I kinda see what this might be the wrong way to go aboutit.

It lags/locks the processor so badly that even the thread schedular is starving for cpu power to do it's thing... at least when the number of threads far exceeds actually cores.

(I think I saw something interesting but might be bug related in my own code... class var... is "block statement widescope" instead of just line wide/scope... so some variables turned into class vars unintentionally... scare delphi language feature... but anyway... now that I know this it's corrected and won't happen again hopefully... I like learning new Delphi and performance things =D)

Consider this a warning that something fishy might be going on... and that a possible thread-spectre might exist...

So far I have seen "hyper-threading" / "symetric multi-threading" spectre.... but no actual threading spectre yet ! =D

But who knows... a ghost might be in there too ! For now take it with a grain of salt ? But do investigate I would say ? ;)

So bottom line and number one question is:

1. Are threads speculativatly created ?! =DDDDDDDDDDDDDDDDDD

2. Are threads speculatively not created ?!?!?!?!?!?

I would assume no speculation occurs because I would assume beginthread is an api call which "triggers" a kernal call.. and switching a process threadover from user land to kernal land is a barrier for speculative execution and thus it does not happen... but ofcourse I am not sure what beginthread actually does ?!

Anyway I have seen enough that it worries me a little bit...

Bye for now,
Skybuck.
skybuck2000 (01-07-19, 03:24 PM)
Two more api's are involved which are called after beginthread:

pthread_mutex_init(FCreateSuspendedMutex, nil);
pthread_mutex_lock(FCreateSuspendedMutex);

these seem to be something different, perhaps not windows apis but something else not sure.

Perhaps these are responsible for the queuing and weird out of order execution of threads not being fully created until much later ?!?

Bye for now,
Skybuck.
skybuck2000 (01-07-19, 03:31 PM)
Seems like I also made a little mistake... assumption is mother of all fucks up.. or perhaps delphi find declaration was bugged don't know.

But the beginthread is actually not a WINDOWS API.

How deliciously confusing of Delphi LOL.

It does exist somewhere in windows but it's not what is being called.

This is being called:

function BeginThread(SecurityAttributes: Pointer; StackSize: LongWord;
ThreadFunc: TThreadFunc; Parameter: Pointer; CreationFlags: LongWord;
var ThreadId: TThreadID): THandle;
var
P: PThreadRec;
begin
if Assigned(SystemThreadFuncProc) then
P := PThreadRec(SystemThreadFuncProc(ThreadFunc, Parameter))
else
begin
New(P);
P.Func := ThreadFunc;
P.Parameter := Parameter;
end;

IsMultiThread := TRUE;

Result := CreateThread(SecurityAttributes, StackSize, @ThreadWrapper, P,
CreationFlags, ThreadID);

{ P variable is supposed to be freed by the ThreadWrapper routine.
If the call to CreateThread fails, then ThreadWrapper will not be called
and P will not get freed. Check for failure now and free P if required.
}
if Result = 0 then
Dispose(P);
end;

It's very familiar... maybe even a drop-in replacement for beginthread windows api.. no time to investigation that further for now...

Just need to deal with reality for now.

So the real windows api it is calling is:

//TODO: TThreadFunc vs TFNThreadStartRoutine
function CreateThread(SecurityAttributes: Pointer; StackSize: SIZE_T;
ThreadFunc: TThreadFunc; Parameter: Pointer;
CreationFlags: DWORD; var ThreadId: DWORD): THandle; stdcall;
external kernel name 'CreateThread';

kernel = kernel32.dll

Bye for now,
Skybuck.
skybuck2000 (01-07-19, 03:42 PM)


Nothing is mention about threads starting long *after* this call to CreateThread.

(It does mention threads can start running before the call returns).

I'm looking for the opposite...

I am not sure if what I am seeing is a debugger effect. It could be.

Though when I run without debugging... the effect seems to be there as well... though I am not entirely sure... the code is a bit sloppy... but theoretically... it should not be hanging... so it does seem to indicate something fishy is going on.

I did write the code to the best of my abilities when it comes to creating and destroying threads... at least "delphi-wise".

Perhaps delphi is doing something weird and I should fall back to just using windows apis only... to for example circumvent this weird pmutex crap.

Though if it is somewhat debugger related... maybe it's still there somewhere in the background monitoring something... or perhaps debug code... not yet sure if it happens in release mode...

This is kinda weird though.

I kinda want to get to this bottom of this... but for now it's too weird and I am getting tired have to go sleep soon...

I need more information about how windows work internally when it creates threads what actually happens ?

Also some behavioural description of these pmutex crappy things needs to be investigated as well and information about it's behaviour... does it list anything and then later re-execute/resume ?

Weird stuff.

Ofcourse it could also be something totally else... weird processor speculative execution behaviour for threads.

Perhaps it's speculating that some threads will never be created...

Perhaps it's a branch effect... perhaps it's waiting for a certain branch to be taken... but I am pretty sure those branches where taking...

But still... perhaps something is delaying thread creation on purpose... not sure...

Again weird stuff.

Do your own testing to find it I'd say shouldn't be too difficult to see this weird behaviour.

Would be nice to see some C/C++ code but it would have to be fully open source... specially if command prompt was used to write lines... mine uses a gui... which might be a bit to heavy handed to investigate processor and schedular behaviour.

Though ultimate command prompt is kinda a GUI too.... and might be serialized and such... don't want writelns to serialize my threads... that's the thing ;)

Bye for now,
Skybuck.
skybuck2000 (01-07-19, 03:56 PM)
Fortunately some information is available:



Scanning this quickly does show initial state and creation described as well:

Very interesting document for windows scheduling in general.

Perhaps there are some clues in there about what might be going in relation to what I am seeing.

BONUS POINTS if this document covers what happens when a DEBUGGER is involved ?! ;)

Will have to read this tomorrow more closely ;)

This is a larger document, may also contain some interesting information in relation to thread creation:



Also interesting document:



Not sure if it's pure windows or also some weird .net crap, but who knows ;)
Probably mostly windows stuff =D :P

This is kinda funny but could be a nice way to investigate the schedular algorithms:

A user mode schedular:



For a moment there I thought it was possible to hook back into "system schedular algorithm" but reading it closer it says "without system schedular" at first I read "with system schedular" lol... like some hook back or something but nope...

Still a bit interesting though for side curiosity project sometime or so ;) or maybe not at all.

Bye for now,
Skybuck.
Similar Threads