Quentin Guilloteau
d5bb59468c
analysis workflow
2024-09-12 14:07:43 +02:00
Quentin Guilloteau
a4d9ede97d
add support for --build-arg flag
2024-09-11 10:51:38 +02:00
Quentin Guilloteau
ce977ceb9e
now working on g5k
2024-08-29 18:16:12 +02:00
Quentin Guilloteau
9289b10b84
start splitting the workflow into several pieces
2024-08-29 13:10:24 +02:00
Quentin Guilloteau
a62d338bd0
moving some config around
2024-08-29 12:58:50 +02:00
Quentin Guilloteau
31e2ff0ca5
seperate the nix environments into several files
2024-08-29 11:41:29 +02:00
Quentin Guilloteau
d8d2a2e8e2
remove wrapper ecg
2024-08-29 11:23:54 +02:00
Quentin Guilloteau
025a16b62c
starting cleaning snakefile
2024-08-28 14:35:13 +02:00
9c6ce2d150
Since analysis and aggregation rules had directories in their input, Snakemake complained because of these missing directories, even if he was supposed to create them. So I removed the directories from the inputs of the rules, and used global variables instead (and for some, just wrote the directory path directly in the shell command).
...
Fixed the missing option specifier for the analysis wrapper for the softenv analysis. Also fixed the way arguments are parsed by the wrapper.
2024-08-26 14:40:43 +02:00
97447e59a1
The results of the get_analysis_dates function being used as inputs in some Snakemake rules, this causes Snakemake to overwrite older analysis, probably because the files used to generate these older analysis have been modified. To avoid overwriting older analysis, we only specify today's analysis in the input, and use a bash wrapper script to both fetch the older analysis by itself (by giving it the folder where to look), and run the analysis with the fetched files. So I removed get_analysis_dates and replaced it with the wrapper script every time it was used.
...
I also removed the older unfinished analysis wrapper that I forgot it existed...
2024-08-23 16:14:46 +02:00
f2cf7e3498
Handling of the case where no artifact configuration files have been found, or they have all been blacklisted.
2024-08-22 18:37:44 +02:00
b8212061fd
The date of the day is now always added to the output of get_analysis_dates, because we obviously want to generate the analysis of the day, even if there are older analysis in the folder already...
2024-08-22 18:21:00 +02:00
ce826c35a6
Fixed aggregation by separating it into multiple rules. Fixed calls to get_analysis_dates. Fixed plot headers and plot script in plotting rules, which are now working ( close #44 ). Rule all has been modified according to these changes. Removed global analysis rule.
...
List of artifacts to build now generated using all available blacklists, not just the last one (ensures that we don't miss some blacklisted artifacts). Fixed update blacklist rule, because of grep's output. Removed default blacklist.
2024-08-22 17:55:05 +02:00
2d5b043f8f
Fixed analysis and aggregate rules to get all ECG/analysis outputs for all possible dates. Written a function to get all available outputs for this purpose. Defined variables for arrays used multiple times. Simplified aggregate rule, but needs fix, because cannot have a list of shell commands apparently. Modified plotting rules according to those changes.
2024-08-21 21:15:09 +02:00
9c6fe6700b
The updating of the blacklist has been reworked to be functional: it is not relying on a symlink anymore (this symlink has been removed). Instead, the last blacklist is fetched by the get_artifacts_to_build function. Modified Snakefile according to the change in the name of a build status. Created a default config file for Snakemake, allowing custom configurations to be written to "config.yaml".
2024-08-20 18:55:12 +02:00
72a638cd17
Added conference date to Nickel contract + artifacts ( close #29 ). Changes localhost to 127.0.0.1 in test artifact, because localhost doesn't work in a Nix environment.
2024-08-20 17:44:41 +02:00
0a8b0c85c6
Fixed analysis part in Snakemake again, needs testing. Moved test and template artifacts back to parent folder. Changes analysis output dir.
2024-08-19 19:02:23 +02:00
ece5749b7a
Fixed analysis in Snakefile. Started working on plotting part.
2024-08-19 16:37:51 +02:00
f6f20cb0d6
Integrated the whole analysis part to the Snakefile, but needs testing ( #44 ).
2024-08-19 15:21:28 +02:00
e943be4efe
Integrated softenv analysis to workflow. Changed input and output cmd options in analysis scripts to take multiple files at once. Moved test and template artifacts in an excluded folder.
2024-08-19 14:59:08 +02:00
1e5dd77cd8
Working on the integration of the analysis to the snakemake workflow.
2024-08-16 17:41:20 +02:00
0bbed9d0f5
Adding missing changes from the previous commit because I wasn't in the root folder...
2024-08-15 12:23:34 +02:00
74228cd1c5
Now optional with a default value in the Nickel contract: version, comment, git_packages, misc_packages, python_venvs ( close #30 ).
2024-08-06 11:44:38 +02:00
114d0e5816
Added support for other virtualization techs in the Nickel contract ( close #28 ). Added Nix dependency for Snakemake diagram generation.
2024-07-30 18:06:16 +02:00
ff2fc2d365
Added version number to the Nickel contracts and artifacts ( close #27 ).
2024-07-29 16:39:30 +02:00
e8b25b74a7
Added build status analysis. Now logging build status even if build is successful, to make analysis easier (as mentionned in #26 ). Added a new error type.
2024-07-26 19:00:25 +02:00
48d3da9c7d
Removed image name from the config file, now generated from the artifact URL ( close #24 ). Fixed the "--no-cache" option when using Docker cache.
2024-07-24 18:58:50 +02:00
d2768d5373
Added initial support for Python venvs, close #21 . Disabled the Docker build cache by default, and added an option to reenable it, close #23 . Removed conditions on the existance of git_packages and misc_packages, because the workflow enforces their existance with the Nickel artifact contract.
2024-07-24 18:05:31 +02:00
4329e333ea
Added shi_welder, renamed OAR timeout error.
2024-07-24 10:13:53 +02:00
687b777912
Merge remote-tracking branch 'origin/workflow_config'
2024-07-23 17:26:03 +02:00
Quentin Guilloteau
944b1bf6f9
checkpoint in case of reaching walltime
2024-07-21 17:32:04 +02:00
Quentin Guilloteau
566eb778f2
fix execution on grid5000 from namespace
2024-07-21 16:13:52 +02:00
Quentin Guilloteau
318955865c
typo
2024-07-21 16:12:40 +02:00
Quentin Guilloteau
8ed548c435
first try for g5k execution
2024-07-20 15:41:56 +02:00
cc605943a9
Added a comment section for artifacts configuration. Moved hints to the Nickel contract.
2024-07-19 17:05:46 +02:00
Quentin Guilloteau
9769f6b533
Add config for snakemake
2024-07-19 16:33:27 +02:00
a7bf9cee94
Written a doc for the output. Removed the "type" attribute from the Nickel contract, closing #17 .
2024-07-19 16:18:49 +02:00
d791064183
Added Nickel constraint on values.
2024-07-17 14:56:38 +02:00
1aaa77cba3
Fixed Nickel contract and check files. Fixed input/output files in Snakefile.
2024-07-16 17:35:09 +02:00
69e447ab0a
Fixed Nickel config files. Added artifact hash log on Snakefile + modified some variables names. Switched from 'tgz' to 'tar' for archive type.
2024-07-16 16:47:17 +02:00
Quentin Guilloteau
4432a93123
integrate nickel in workflow
2024-07-16 13:59:44 +02:00
Quentin Guilloteau
28126f8b3e
basic workflow for ecg
2024-07-11 15:17:16 +02:00
Quentin Guilloteau
3737dd29e6
add start of workflow to extract non blacklisted artifacts
2024-07-11 13:37:07 +02:00
