Skip to the content.

Pipeline training pattern

Usecase

Architecture

The pipeline training pattern is a advancement of the batch training pattern. Each job is defined as separate resources, server, container, or worker, to make it possible to deploy the job resource independently to run and retry the job flexibly. Since the job will be deployed as a separate resource, it will be executed after the dependent job. The result of the former job will be succeeded to the latter as its input data. For sake of fault tolerance, the processed data can be stored in DWH. You don’t have to run the job right after the former job completion. It is possible to run the time-consuming job frequently, and other jobs not so often.
Note that the pattern may make the job workflow and resource management complex. While it increases independency of the job, you need to select resource and define jobs.

Diagram

diagram

Pros

Cons

Needs consideration