Applying the pipeline pattern

Patterns

pattern (noun): a reliable sample of traits, acts, tendencies, or other observable characteristics

• Instruction pipelines
• Unix pipes
• Collection pipeline (Martin Fowler)
• Pipes and filters (Enterprise Integration Patterns)

UNIX

We should have some ways of coupling programs like garden hose--screw in another segment when it becomes when it becomes necessary to massage data in another way.

M.D.McIlroy

October 11, 1964

Pipelines

• Processing items of the same type
• Small tools/services
• Fixed sequence of operations
• Well established communication interfaces
• Buffers

Advantages

• No cyclic dependencies
• Parallelism
• Laziness
• Immutability
• Debugging

Characteristics

UNIX pipe

• Processing item: lines
• Small tools
• Communication interface: text
• Buffer
• Lazy and parallel
• Immutable

Functional programming

"90% of functional programming is list processing"

map
filter
reduce

Loop with Java streams

employees.stream().
          filter(e -> e.getAge() > 30).
          map(Employee::getSalary).
          reduce(Integer::sum)

Java Streams

• Lazy
• Short-circuiting
• Traversible only once
• Effectively final

Java pipes

• Processing item: objects
• Small methods
• Communication interface: method invocation
• Lazy and parallel
• Immutable

Reactive streams

asynchronous stream processing with non-blocking back pressure

Source(List(1,2,3)).
  filter(_ % 2 == 0).
  map(_ * 2).
  runFold(0)(_ + _).
  foreach(println)

Akka streams

Reactive pipeline

• Processing item: message
• Small services
• Communication interface: message passing
• Buffers: actor inboxes
• Lazy, asynchronous and parallel
• Immutable

RxJava filter

RxJava map

RxJava reduce

SentimentMetrics Library

• Fetching message from provider
• Analyzing message
  • Language
  • Gender of user
  • Sentiment
• Saving to DB
• Automation rules
• Displaying to customer

SentimentMetrics, take 2

• Service
  • Drawbacks
    • (de)serialization
    • network overhead
  • Advantages
    • redundancy
    • failover
    • throughput
    • flexibility in deployment
    • upgradeability

Sentiment pipeline

• Processing item: message
• Small services
• Communication: Redis
• Buffers: Redis list/Actors
• Lazy, asynchronous and parallel
• Immutable

Sentiment services

Conclusion

• Small specialized components
• Eliminate shared global mutable state
• Good design principles are much longer-lived than buzzwords

Stream-centric microservices

Services are composed using Unix-like pipelines (the Web meets Unix = true loose-coupling)

Juval Löwy

Links

• http://doc.cat-v.org/unix/pipes/
• http://martinfowler.com/articles/collection-pipeline/
• http://www.enterpriseintegrationpatterns.com/patterns/messaging/PipesAndFilters.html
• http://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data
• https://medium.com/@skamille/microservices-real-architectural-patterns-68bd83bbb6cd
• https://en.wikipedia.org/wiki/Microservices
• http://david-peter.de/cube-composer/