[PEAK] Reactor-driven microthreads

[Phillip J. Eby]

At 01:50 PM 12/31/03 -0500, Bob Ippolito wrote:

>On Dec 31, 2003, at 1:12 PM, Phillip J. Eby wrote:
>
>>At 12:12 PM 12/31/03 -0500, Bob Ippolito wrote:
>>>On Dec 31, 2003, at 12:22 AM, Phillip J. Eby wrote:
>>>
>>>>Hm.  This seems like the first really good use case I've seen for
>>>>having a macro facility in Python, since it would allow us to spell
>>>>the yield+errorcheck combination with less boilerplate.  Ah well.
>>>
>>>That, and "flattening" generators
>>>
>>>def someGenerator():
>>>     for something in someOtherGenerator():
>>>         yield something
>>>
>>>This gets really ugly after a while, but it happens a lot when you are
>>>doing "microthreads".
>>
>>Actually, the Thread class in my second post allows you to do:
>>
>>def someGenerator():
>>     yield someOtherGenerator(); resume()
>>
>>Thread objects maintain an external "stack" of nested generators, so you 
>>don't need to do this kind of passthru.
>
>That is the other way to do it, but I don't like that at all.  It's not a 
>general solution, I use this 'pattern' very often with or without a 
>'scheduler' on the outside.

I'm a bit confused.  Do you mean that you often need to yield the contents 
of another generator often, or that you use 
microthreading-without-a-scheduler often?


>I didn't write and I don't use twisted.flow so I have no idea, but I think 
>that the code that I posted in the mailing list is somewhat roughly 
>equivalent (in a same-concept-but-not-nearly-as-polished kind of way) to 
>yours though... except it requires you to yield a particular kind of 
>object from your iterable "microthread" and that object has your "resume" 
>method.  The only real thing about your implementation that I disagree 
>with is this global resume function.

 From my POV, 'resume()' is a macro that happens to be implemented as a 
function.  :)

I'd be interested in seeing the code you mention though; got an archive 
URL?  (I assume you're referring to the twisted mailing list.)


>Oh, and I disagree with the fact that you say "generators provide an ideal 
>basis".  Ideally, you'd be using Stackless Python.

No, ideally Python would be "stackless" in that sense.  The fact that 
there's a separate Stackless Python is what makes it less than 
ideal.  :)  I meant, "an ideal basis, given the limitations of Python 
2.2".  The "given current Python" part is second nature to me because I 
expect to be supporting CPython 2.2 in production through at least 
mid-2004, so it doesn't occur to me to contemplate using anything else.  :)


>In any case, I use code like this quite often with and without Twisted and 
>agree that it helps in quite a few situations but in others you really 
>ought to be using an event driven model (when what you're doing isn't 
>nearly as linear, which does happen).

Certainly anything whose "one obvious way to do it" is a state machine 
should be implemented as a state machine.  But the specific examples I've 
been talking about are things whose "obvious way to do it" is a 
thread.  And even when writing a state machine, it's quite likely that an 
"obvious way to do it" is to write a Thread that acts as the dispatch loop!


>   A facility like yours will surely lower the barrier to Twisted 
> programming because as you said most programmers are far better 
> linear/iterative thinkers.  The real problem is, of course, that Python 
> sucks at this concurrency which is why Twisted feels like the tail 
> wagging the dog.  I think that things could be made a LOT nicer with 
> macros, but I don't see a version of Python with macros happening anytime soon.

Nope.  On the whole, though, I think the API I've proposed to this point is 
pretty minimal overhead: yield+resume is a small price to pay for 
co-operative multitasking with ubiquitous "threads".