Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
P
Pypelines
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
HaissLab
Data Management
Pypelines
Commits
5b9acb8a
Commit
5b9acb8a
authored
1 year ago
by
Timothe Jost
Browse files
Options
Downloads
Patches
Plain Diff
some additions to make multisession guess a bit smarter
parent
d090b883
No related branches found
Branches containing commit
No related tags found
No related merge requests found
Pipeline
#129977
passed
1 year ago
Stage: test
Changes
3
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
src/pypelines/__init__.py
+1
-1
1 addition, 1 deletion
src/pypelines/__init__.py
src/pypelines/pipes.py
+20
-2
20 additions, 2 deletions
src/pypelines/pipes.py
src/pypelines/steps.py
+5
-1
5 additions, 1 deletion
src/pypelines/steps.py
with
26 additions
and
4 deletions
src/pypelines/__init__.py
+
1
−
1
View file @
5b9acb8a
__version__
=
"
0.0.5
6
"
__version__
=
"
0.0.5
7
"
from
.
import
loggs
from
.pipes
import
*
...
...
This diff is collapsed.
Click to expand it.
src/pypelines/pipes.py
+
20
−
2
View file @
5b9acb8a
...
...
@@ -6,6 +6,8 @@ from .disk import BaseDiskObject
from
functools
import
wraps
import
inspect
,
hashlib
from
pandas
import
DataFrame
from
abc
import
ABCMeta
,
abstractmethod
from
typing
import
Callable
,
Type
,
Iterable
,
Protocol
,
TYPE_CHECKING
,
Literal
,
Dict
...
...
@@ -208,7 +210,23 @@ class BasePipe(metaclass=ABCMeta):
list
(
self
.
steps
.
values
()),
key
=
lambda
item
:
item
.
get_level
(
selfish
=
True
),
reverse
=
reverse
)
highest_step
=
None
if
isinstance
(
session
,
DataFrame
):
# if multisession, we assume we are trying to just load sessions
# that all have reached the same level of requirements. (otherwise, use generate)
# because of that, we use only the first session in the lot to search the highest loadable step
search_on_session
=
session
.
iloc
[
0
]
else
:
search_on_session
=
session
for
step
in
ordered_steps
:
if
step
.
get_disk_object
(
session
,
extra
).
is_matching
():
return
step
.
load
(
session
,
extra
)
if
step
.
get_disk_object
(
search_on_session
,
extra
).
is_matching
():
highest_step
=
step
if
highest_step
is
not
None
:
# if we found one : it is not None
# we use the load wrapper, wich will dispatch to multissession or not automatically,
# depending on session type (Series or DataFrame)
return
highest_step
.
load
(
session
,
extra
)
raise
ValueError
(
f
"
Could not find a
{
self
}
object to load for the session
{
session
.
alias
}
with extra
{
extra
}
"
)
This diff is collapsed.
Click to expand it.
src/pypelines/steps.py
+
5
−
1
View file @
5b9acb8a
from
functools
import
wraps
,
partial
,
update_wrapper
from
.loggs
import
loggedmethod
,
NAMELENGTH
from
.arguments
import
autoload_arguments
import
logging
,
inspect
import
logging
,
inspect
from
pandas
import
DataFrame
from
dataclasses
import
dataclass
from
types
import
MethodType
...
...
@@ -258,6 +259,9 @@ class BaseStep:
ValueError: If the disk object does not match and has a status message.
"""
# print("extra in load wrapper : ", extra)
if
isinstance
(
session
,
DataFrame
):
return
self
.
multisession
.
load
(
sessions
=
session
,
extras
=
extra
)
if
extra
is
None
:
extra
=
self
.
get_default_extra
()
# print("extra in load wrapper after None : ", extra)
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment