Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
P
Pypelines
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package registry
Container registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
HaissLab
Data Management
Pypelines
Commits
a0fb2912
Commit
a0fb2912
authored
1 year ago
by
Timothe Jost
Browse files
Options
Downloads
Patches
Plain Diff
doc for stepmethod
parent
6b1b9c6f
No related branches found
No related tags found
No related merge requests found
Pipeline
#120638
passed
1 year ago
Stage: test
Changes
2
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
src/pypelines/__init__.py
+1
-1
1 addition, 1 deletion
src/pypelines/__init__.py
src/pypelines/steps.py
+32
-2
32 additions, 2 deletions
src/pypelines/steps.py
with
33 additions
and
3 deletions
src/pypelines/__init__.py
+
1
−
1
View file @
a0fb2912
__version__
=
"
0.0.1
8
"
__version__
=
"
0.0.1
9
"
from
.
import
loggs
from
.pipes
import
*
...
...
This diff is collapsed.
Click to expand it.
src/pypelines/steps.py
+
32
−
2
View file @
a0fb2912
...
...
@@ -15,6 +15,28 @@ if TYPE_CHECKING:
def
stepmethod
(
requires
=
[],
version
=
None
,
do_dispatch
=
True
,
on_save_callbacks
=
[]):
"""
Wrapper to attach some attributes to a method of a pipeline
'
s pipe. These methods are necessary to trigger the
pipeline creation mechanism on that step_method after the pipe has been fully defined.
Args:
requires (list, optional): single string or list of strings corresponding to other pipeline steps needed.
The other pipeline steps must belong to the same pipeline than the one of the step_method. Defaults to [].
version (_type_, optional): version of the step method. Changing this from none to a text or number will
result in previous saved outputs to be reprocessed upon generation check, so that you can more easily
control what needs to be reprocessed if you change an important computation step, and the minimum needed,
but no less, will be automatically reprocessed. Defaults to None.
do_dispatch (bool, optional): Wether to perform dispatch mechanism on calls of load, save and generate,
or not, for this step_method. Defaults to True.
on_save_callbacks (list, optional): The save callbacks can be a single callable, or a list of callables.
Additionnaly, independently if you supply a singloe of multiple callables,
they can be a tuple of (callable, named_argument_dict) instead of a simple callable.
The arguments in the dict will override arguments that would have been passed by the generation mechanism,
such as session, extra and pipeline. Defaults to [].
Returns:
callable : the callable with extra attributes attached
"""
# This allows method to register class methods inheriting of BasePipe as steps.
# It basically just step an "is_step" stamp on the method that are defined as steps.
# This stamp will later be used in the metaclass __new__ to set additionnal usefull attributes to those methods
...
...
@@ -337,9 +359,17 @@ class BaseStep:
# TODO an option could be added to catch, display and store exceptions tracebacks,
# while allowing the pipeline to continue,
# in case the callbacks are not absolutely necessary for the pipeline process. (ex, generate plots)
for
callback
in
self
.
callbacks
:
for
callback_data
in
self
.
callbacks
:
arguments
=
{
"
session
"
:
session
,
"
extra
"
:
extra
,
"
pipeline
"
:
self
.
pipeline
}
if
isinstance
(
callback_data
,
tuple
):
callback
=
callback_data
[
0
]
overriding_arguments
=
callback_data
[
1
]
else
:
callback
=
callback_data
overriding_arguments
=
{}
arguments
.
update
(
overriding_arguments
)
try
:
callback
(
session
=
session
,
extra
=
extra
,
pipeline
=
self
.
pipeline
)
callback
(
**
arguments
)
except
Exception
as
e
:
import
traceback
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment