DO NOT MERGE - Pipeline performance test project #4154
Conversation
Signed-off-by: Laura Couto <[email protected]>
Signed-off-by: Laura Couto <[email protected]>
Signed-off-by: Laura Couto <[email protected]>
Signed-off-by: Laura Couto <[email protected]>
Signed-off-by: Laura Couto <[email protected]>
lrcouto
requested review from
noklam,
ankatiyar,
ElenaKhaustova and
astrojuanlu
There was a problem hiding this comment.
Thanks! I am not able to run the pipeline with missing data so I just quickly review it on a high level.
Can you add some description do the PR explaining how to use this pipeline test. I see that most of the pipeline here are mocking with sleep, why did you end up going with this implementation?
For example, if I want to answer the question, does Kedro run too slow when it needs to connect to a database, what command should I run?
| @@ -0,0 +1,20 @@ | |||
| # What is this for? | |||
|
|
|||
| This folder should be used to store configuration files used by Kedro or by separate tools. | |||
There was a problem hiding this comment.
Is there any specific configuration needed to be documented? Otherwise I think we can remove this from our project
| @@ -0,0 +1,98 @@ | |||
| # performance-test | |||
There was a problem hiding this comment.
Maybe it's more helpful to document how this project should be used, otherwise I suggest removing it as these template doesn't add much information for us.
| def register_pipelines(self) -> Dict[str, Pipeline]: | ||
| from performance_test.pipelines.expense_analysis import ( | ||
| pipeline as expense_analysis_pipeline, | ||
| ) | ||
|
|
||
| return { | ||
| "__default__": expense_analysis_pipeline.create_pipeline(), | ||
| "expense_analysis": expense_analysis_pipeline.create_pipeline(), | ||
| } |
There was a problem hiding this comment.
Does this belongs to pipeline_registry.py?
Description
Kedro project made to simulate delays and latency in specific points of a Kedro pipeline. Pass the desired delays in seconds using the
--paramsflag. For example:kedro run --params=hook_delay=5,dataset_load_delay=5,file_save_delay=5Development notes
Developer Certificate of Origin
We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a
Signed-off-byline in the commit message. See our wiki for guidance.If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.
Checklist
RELEASE.mdfile