Quentin Guilloteau
a62d338bd0
moving some config around
2024-08-29 12:58:50 +02:00
Quentin Guilloteau
d8d2a2e8e2
remove wrapper ecg
2024-08-29 11:23:54 +02:00
Quentin Guilloteau
025a16b62c
starting cleaning snakefile
2024-08-28 14:35:13 +02:00
9c6ce2d150
Since analysis and aggregation rules had directories in their input, Snakemake complained because of these missing directories, even if he was supposed to create them. So I removed the directories from the inputs of the rules, and used global variables instead (and for some, just wrote the directory path directly in the shell command).
...
Fixed the missing option specifier for the analysis wrapper for the softenv analysis. Also fixed the way arguments are parsed by the wrapper.
2024-08-26 14:40:43 +02:00
97447e59a1
The results of the get_analysis_dates function being used as inputs in some Snakemake rules, this causes Snakemake to overwrite older analysis, probably because the files used to generate these older analysis have been modified. To avoid overwriting older analysis, we only specify today's analysis in the input, and use a bash wrapper script to both fetch the older analysis by itself (by giving it the folder where to look), and run the analysis with the fetched files. So I removed get_analysis_dates and replaced it with the wrapper script every time it was used.
...
I also removed the older unfinished analysis wrapper that I forgot it existed...
2024-08-23 16:14:46 +02:00
ce826c35a6
Fixed aggregation by separating it into multiple rules. Fixed calls to get_analysis_dates. Fixed plot headers and plot script in plotting rules, which are now working ( close #44 ). Rule all has been modified according to these changes. Removed global analysis rule.
...
List of artifacts to build now generated using all available blacklists, not just the last one (ensures that we don't miss some blacklisted artifacts). Fixed update blacklist rule, because of grep's output. Removed default blacklist.
2024-08-22 17:55:05 +02:00
2d5b043f8f
Fixed analysis and aggregate rules to get all ECG/analysis outputs for all possible dates. Written a function to get all available outputs for this purpose. Defined variables for arrays used multiple times. Simplified aggregate rule, but needs fix, because cannot have a list of shell commands apparently. Modified plotting rules according to those changes.
2024-08-21 21:15:09 +02:00
9c6fe6700b
The updating of the blacklist has been reworked to be functional: it is not relying on a symlink anymore (this symlink has been removed). Instead, the last blacklist is fetched by the get_artifacts_to_build function. Modified Snakefile according to the change in the name of a build status. Created a default config file for Snakemake, allowing custom configurations to be written to "config.yaml".
2024-08-20 18:55:12 +02:00
0a8b0c85c6
Fixed analysis part in Snakemake again, needs testing. Moved test and template artifacts back to parent folder. Changes analysis output dir.
2024-08-19 19:02:23 +02:00
ece5749b7a
Fixed analysis in Snakefile. Started working on plotting part.
2024-08-19 16:37:51 +02:00
f6f20cb0d6
Integrated the whole analysis part to the Snakefile, but needs testing ( #44 ).
2024-08-19 15:21:28 +02:00
e943be4efe
Integrated softenv analysis to workflow. Changed input and output cmd options in analysis scripts to take multiple files at once. Moved test and template artifacts in an excluded folder.
2024-08-19 14:59:08 +02:00
1e5dd77cd8
Working on the integration of the analysis to the snakemake workflow.
2024-08-16 17:41:20 +02:00
0bbed9d0f5
Adding missing changes from the previous commit because I wasn't in the root folder...
2024-08-15 12:23:34 +02:00
e8b25b74a7
Added build status analysis. Now logging build status even if build is successful, to make analysis easier (as mentionned in #26 ). Added a new error type.
2024-07-26 19:00:25 +02:00
Quentin Guilloteau
944b1bf6f9
checkpoint in case of reaching walltime
2024-07-21 17:32:04 +02:00
Quentin Guilloteau
566eb778f2
fix execution on grid5000 from namespace
2024-07-21 16:13:52 +02:00
Quentin Guilloteau
8ed548c435
first try for g5k execution
2024-07-20 15:41:56 +02:00
Quentin Guilloteau
9769f6b533
Add config for snakemake
2024-07-19 16:33:27 +02:00
1aaa77cba3
Fixed Nickel contract and check files. Fixed input/output files in Snakefile.
2024-07-16 17:35:09 +02:00
69e447ab0a
Fixed Nickel config files. Added artifact hash log on Snakefile + modified some variables names. Switched from 'tgz' to 'tar' for archive type.
2024-07-16 16:47:17 +02:00
Quentin Guilloteau
4432a93123
integrate nickel in workflow
2024-07-16 13:59:44 +02:00
Quentin Guilloteau
28126f8b3e
basic workflow for ecg
2024-07-11 15:17:16 +02:00
