[Feature Request]: Enable testability of middleware pipelines

[mattstermiller]

Is your request related to a problem you have?

I am excited about using kafka flow for event processing in our organization. However, I am trying to write tests for middleware pipelines and facing enormous difficulty in doing so. To illustrate the kind of test I am trying to write, I have a consumer that sets up a middleware pipeline like this:

// in consumer setup:
consumer.Topic( ... )
    // more stuff
    .AddMiddlewares(SetupPipeline);
    
public static void SetupPipeline(IConsumerMiddlewareConfigurationBuilder middlewares)
    => middlewares.AddEventNameFilter([PatientFlagCreatedEvent, PatientFlagRemovedEvent])
        .AddSingleTypeProtobufDeserializer<PatientFlagEventP>()
        .AddValidationAndLogErrors((PatientFlagEventP evt) => evt.Validate())
        .RetryForever(
            retry => retry.HandleAnyException()
                .WithExponentialBackoff()
        )
        .AddTypedHandlers(h => h.AddHandler<PatientFlagEventHandler>())
        .AddSuccessLogging<PatientFlagEventHandler>();

And then I am trying to write a test where I call SetupPipeline and get something that I can invoke the middleware pipeline by passing IMessageContext values.

Describe the solution you'd like

I would like to have a utility similar to this:

MiddlewareRunner BuildMiddleware(IServiceCollection serviceCollection, Action<IConsumerMiddlewareConfigurationBuilder> buildMiddleware)

...where MiddlewareRunner is something like this:

Task Run(IMessageContext messageContext)

I tried building this myself, but there are so many interfaces involved (maybe all of them) where the implementation is internal to KafkaFlow, even IMessageContext itself and IConsumerContext, etc. I see no feasible way of achieving this without abusing reflection to instantiate the internal types.

Alternatively, a simple in-memory consumer implementation that I could pass raw kafka messages to might also be desirable.

Besides this, the only way I see to test a middleware pipeline is to write a full integration test with an instance of Kafka, schema registry, etc. This seems really heavy-handed. I wish I could write simple, fast tests to verify that a combination of middlewares work as expected without getting the entire stack involved.

Are you able to help bring it to life and contribute with a Pull Request?

Yes

Additional context

No response