The behavior of stream seems somewhat odd; perhaps it shouldn't be Operation<X>

[iplaylf2]

The documentation at Effection Collections mentions that Stream corresponds to AsyncIterable, but in practice, AsyncIterable -> AsyncIterator does not require await during usage:

const delay = (x: number) => new Promise(r => setTimeout(r, x));

(async () => {
    const stream: AsyncIterable<number> = (async function* () {
        await delay(100)
        yield 1
        await delay(100)
        yield 2
        await delay(100)
        yield 3
    })();

    const subcription1 = stream[Symbol.asyncIterator]() // No `await` on the stream itself
    // const subcription2 = ...

    console.log(await subcription1.next())
    // ....
})()

typescript playground

The example in the Effection documentation is also confusing:

    let subscription1 = yield* channel;
    let subscription2 = yield* channel;

There are two different subscriptions created here, and it feels like merely using yield* to wait for an Operation<X> creates a new task, whereas await waiting for a Promise<X> does not.

Nowadays, for the same functionality, I think the following expression in Effection is better:

import { main, createChannel } from 'effection';

await main(function*() {
  let channel = createChannel();

  // the channel has no subscribers yet!
  yield* channel.send('too early');

  let subscription1 = channel.subscribe();
  let subscription2 = channel.subscribe();

  yield* channel.send('hello');
  yield* channel.send('world');

  console.log(yield* subscription1.next());
  //=> { done: false, value: "hello" }
  console.log(yield* subscription1.next());
  //=> { done: false, value: "world" }
  console.log(yield* subscription2.next());
  //=> { done: false, value: "hello" }
  console.log(yield* subscription2.next());
  //=> { done: false, value: "world" }
});

Hmm, what about Stream? Maybe the signature of Stream should be Stream<T> = () => Subscription<T>. This way, it would be analogous to AsyncIterable and AsyncIterator.

[cowboyd]

@iplaylf2 I think I see where you're coming from. There definitely are some differences between the Async/Await primitives and their Effection analogues, and that they can seem confusing and/or unnecessary when you are new to Effection. However, it is precisely in these differences that you will find the key to the guarantees that Effection provides you that Async/Await does not. Specifically:

It is impossible in Effection to create any effect whatsoever that is not bound to a scope.

This may seem restrictive at first, but it is actually a superpower.

Take for example the delay function you defined:

const delay = (x: number) => new Promise(r => setTimeout(r, x));

This eagerly creates the timeout effect every time it is called, and it can be called anywhere at any time:

const delay = (x: number) => new Promise(r => setTimeout(r, x));

delay(500);

(() => { delay(1000}; delay(400); })();

for (let i = 0; i < 1000; i++) {
  delay(i);
}

This will create over 1000 calls to setTimeout(), and there is no way to tell where they came from. By contrast, the Effection analogue of the Promise constructor is action() and it does not eagerly create context-free effects. The previous example using action would look like this:

const delay = (x: number) => action(r => { 
  let timeoutId = setTimout(r,x}; 
  return () => clearTimeout(timeoutId);
});

delay(500);

(() => { delay(1000}; delay(400); })();

for (let i =0; i < 1000; i++) {
  delay(i);
}

Which causes setTimeout() to be called exactly zero times. That's because unlike the Promise constructor which actively does a thing, an action is only a recipe for how to do the thing. And, the only way you can actually cause it to be invoked is by evaluating the action with yield*. When you do, that call to setTimeout is implicitly associated with the scope in which the action was evaluated, and so if that scope goes away for any reason whatsoever(return, error, or cancellation), the setTimeout effect will be uninstalled.

The same principle applies to subscriptions. Usually, whether it is a file handle, a web socket, a bucket storage download, or whatever, there is some "hot" state associated with a subscription. AsyncIterable will usually create that hot state eagerly in a context-free fashion, just like the Promise constructor.

 const subcription1 = stream[Symbol.asyncIterator]() // No `await` on the stream itself

In that code, it is unknown what state [Symbol.asyncIterator]() could allocate. It might open a TCP connection for example to start reading chunks, it might open a file. But there is no "owner" of that state. It is effectively global, and could very easily be leaked. In the same way as action(), activating an effect must be associated with a scope, so the subscription itself must have an owner, as well as each operation that is evaluated to get the next item.

I hope that helps explain why the differences are necessary. While there are analogues for Async/Await constructs in Effection, they are only that: analogues. They must be different it very distinct ways so that we can experience the guarantees that structured concurrency provides which Async/Await doesn't.

I'll leave one final difference which I think is important to grasp. In Effection, yield* is the analogue to await. However, yield* is much more than just a mechanism to await the settlement of a Task (although it is that). While await is limited strictly to resuming the current async function when a promise settles, yield* is a universal evaluator of any effect. In other words, while await can evaluate a single type of effect (promise settlement), yield* can evaluate an infinite number of effect types. An example of this is the context api. which has nothing to do with asynchrony (and is, in fact, synchronous).

[iplaylf2]

My previous understanding of Operation was fundamentally incorrect. If I had properly understood how generator functions can be used to represent () => Operation, I wouldn't have maintained this misunderstanding for so long.

When a generator function is directly called, it doesn't execute the code in its body immediately. Instead, it requires obtaining an Iterator and invoking the next() method to start executing the code. This is precisely how yield/yield* works - when you yield* an Iterable instance, it actually executes the code in the generator function that created it. Each yield* operation essentially creates a new Iterator and performs a complete iteration through it.

Effection uses the same syntax to handle Operations. The principle remains identical: every time you yield* an Operation instance, it means executing the code in the generator function that created that Operation instance.

Regarding the context aspect - I can't articulate it clearly yet, but I intuitively perceive yield* Operation as synchronously executing code within a scope. This scope represents exactly what I expect: a transparent and controlled "coroutine code block"... I'll reach out again if I encounter unresolved issues in this area.

In any case, thank you for helping me identify the root issue. The Stream concept is starting to make sense to me.

[cowboyd]

Great!

I intuitively perceive yield* Operation as synchronously executing code within a scope.

You're pretty much spot on, and you've hit on a very important point. Effection is fundamentally synchronous in nature, so if operations can resolve synchronously, they do. Among other things, this means that Effection streams are appropriate for handling DOM events, whereas async iterables are not.

[iplaylf2]

I confidently refactored my code, but the results still didn't meet expectations. Perhaps there's indeed something unreasonable with the Stream type definition.

I haven't used yield* and generator functions enough. Today I revisited related knowledge and finally distinguished between the Generator instances returned by generator functions and all Iterable instances that can be used with yield*.

First, what does yield* do:

yield* first gets the iterator from the operand by calling the latter's [Symbol.iterator]() method. Then, each time the next() method ...

MDN

In TypeScript, any object implementing the Iterable protocol can legally be used as the operand of yield*.

interface Iterable<T, TReturn = any, TNext = any> {
    [Symbol.iterator](): Iterator<T, TReturn, TNext>;
}

declare const foo: Iterable<unknown>

function* test() {
    yield* foo // passes TypeScript's check
}

However, generator functions return Generators that implement Iterable but also have methods like next. They are "stateful" - two yield* operations won't do the same thing:

function *foo(){
    return 2333
}

function* test(){
    const generator=foo()

    console.log(yield* generator) // prints 233
    console.log(yield* generator) // prints undefined
}

Array.from(test())

It's understandable that generators can slice execution through next methods and maintain state.

But a generator being stateful doesn't mean the operand of yield* must be stateful. We can easily construct "stateless" operands:

function* foo() {
    return 2333
}

function* test() {
    const generator = { [Symbol.iterator]: () => foo() }

    console.log(yield* generator) // prints 233
    console.log(yield* generator) // prints 233
}

Array.from(test())

As long as [Symbol.iterator] always returns a new iterator/generator.

Now back to the question: "Why do I think Stream's type definition is unreasonable?" @cowboyd

In Effection, we always use generator functions to create Operations. Although Operation's signature resembles Iterable, its construction method inherently makes it a Generator - unless it's not expressed through generator functions. TypeScript's automatic type inference always reminds us that these functions return Generator-type things.

We have to accept that Operations created via generator functions are "stateful" - yielding* the same Operation twice will do nothing on the second attempt and directly return undefined. What's frustrating is that TypeScript won't warn about this undefined result from the second yield* - issues only surface during execution.

But in Effection's documentation, we see channel usage like:

  let channel = createChannel();

  // the channel has no subscribers yet!
  yield* channel.send('too early');

  let subscription1 = yield* channel;
  let subscription2 = yield* channel;

The channel (Stream-type instance) is yield* twice. Maybe it's a new Iterable subclass? But no - Stream is an implementation of the Operation generic:

type Stream<T, TReturn> = Operation<Subscription<T, TReturn>>;

Thus in Effection, we must constantly remember which Operations created via generator functions can only be yielded once, and which are "stateless" Operations constructed differently that can be yielded multiple times. Without checking implementations, we can't distinguish them - not a good development experience. (Even distinguishing between Operation and Stream isn't enough, as Stream is more specific.)

My thoughts:

1. Generator functions are too convenient for creating Operations to abandon
2. But there's no technical reason for generator functions to return "stateless" Iterables

Maybe Effection could introduce a subtype XXX for "stateless" Iterables that can be yield* multiple times. For example:

Stream<T, TReturn> = XXX<Subscription<T, TReturn>>

This way, we could identify it without knowing implementations.

Ideally, Stream (XXX) shouldn't be a subtype of Operation, preventing assignment of Stream instances to Operation-type variables.

[cowboyd]

This is definitely a gotcha. And in cases where you want to re-use the same operation more than once, you need to wrap your generator function in an Iterable object. This is what the action() constructor does for example: https://github.com/joshamaju/effection/blob/effect-benchmark/lib/action.ts#L49

In fact, in version 2, we had an explicit check for the case you describe, and if it happened, we would raise a DoubleEvalError https://github.com/thefrontside/effection/blob/v2/packages/core/src/controller/iterator-controller.ts#L23-L24 Perhaps we should bring that back with a link to this discussion or perhaps a section in the FAQ explaining this whole phenomenon?

Ideally, Stream (XXX) shouldn't be a subtype of Operation, preventing assignment of Stream instances to Operation-type variables.

We did this initially in v3 where the stream interface was:

export interface Stream<T,TDone> {
  subscribe(): Operation<Subscription<T, TDone>>;
}

However, we found that this made authoring streams more cumbersome; especially streams that were composites of other streams. There was an awful lot of subscribe() wrapping / delegation that just felt like ceremony and since we rarely consumed them via the subscription we opted to make the tradeoff of making streams easier to write #859

Was it the right decision? Maybe. Maybe not, but the upshot is that defining a stream as a resource is almost always the right thing to do.