348a1adc2c
Updated protocol with some details and current state of the workflow.
2024-08-26 15:12:08 +02:00
9c6ce2d150
Since analysis and aggregation rules had directories in their input, Snakemake complained because of these missing directories, even if he was supposed to create them. So I removed the directories from the inputs of the rules, and used global variables instead (and for some, just wrote the directory path directly in the shell command).
...
Fixed the missing option specifier for the analysis wrapper for the softenv analysis. Also fixed the way arguments are parsed by the wrapper.
2024-08-26 14:40:43 +02:00
97447e59a1
The results of the get_analysis_dates function being used as inputs in some Snakemake rules, this causes Snakemake to overwrite older analysis, probably because the files used to generate these older analysis have been modified. To avoid overwriting older analysis, we only specify today's analysis in the input, and use a bash wrapper script to both fetch the older analysis by itself (by giving it the folder where to look), and run the analysis with the fetched files. So I removed get_analysis_dates and replaced it with the wrapper script every time it was used.
...
I also removed the older unfinished analysis wrapper that I forgot it existed...
2024-08-23 16:14:46 +02:00
514e186c3d
Updated doc with details on analysis outputs. Moved test scripts to a dedicated directory.
2024-08-22 19:19:40 +02:00
0c281e1051
Added missing Python venv to test artifact.
2024-08-22 19:00:36 +02:00
be05d8d70a
Deleted the Git package from the configuration file of the test artifact, because it is a Git repo that is generated when the container is being built, so version number will always change.
2024-08-22 18:40:08 +02:00
f2cf7e3498
Handling of the case where no artifact configuration files have been found, or they have all been blacklisted.
2024-08-22 18:37:44 +02:00
b8212061fd
The date of the day is now always added to the output of get_analysis_dates, because we obviously want to generate the analysis of the day, even if there are older analysis in the folder already...
2024-08-22 18:21:00 +02:00
ce826c35a6
Fixed aggregation by separating it into multiple rules. Fixed calls to get_analysis_dates. Fixed plot headers and plot script in plotting rules, which are now working ( close #44 ). Rule all has been modified according to these changes. Removed global analysis rule.
...
List of artifacts to build now generated using all available blacklists, not just the last one (ensures that we don't miss some blacklisted artifacts). Fixed update blacklist rule, because of grep's output. Removed default blacklist.
2024-08-22 17:55:05 +02:00
2d5b043f8f
Fixed analysis and aggregate rules to get all ECG/analysis outputs for all possible dates. Written a function to get all available outputs for this purpose. Defined variables for arrays used multiple times. Simplified aggregate rule, but needs fix, because cannot have a list of shell commands apparently. Modified plotting rules according to those changes.
2024-08-21 21:15:09 +02:00
e3d01ae34e
Updated doc with plotting script's changes.
2024-08-21 17:07:10 +02:00
3eba87d47a
Added an option to set the output path of the plotting script. Changed name and order of the options of the plotting script.
2024-08-21 15:33:49 +02:00
52c629b9b8
Comments on Snakemake config files.
2024-08-20 18:59:31 +02:00
9c6fe6700b
The updating of the blacklist has been reworked to be functional: it is not relying on a symlink anymore (this symlink has been removed). Instead, the last blacklist is fetched by the get_artifacts_to_build function. Modified Snakefile according to the change in the name of a build status. Created a default config file for Snakemake, allowing custom configurations to be written to "config.yaml".
2024-08-20 18:55:12 +02:00
513b21754c
Changed build status "package_unavailable" to "package_install_failed" to cover more errors. Added an error message for this status. Outputs of ECG are now always created at the beginning to avoid issues with Snakemake (maybe not a good idea).
2024-08-20 18:47:32 +02:00
72a638cd17
Added conference date to Nickel contract + artifacts ( close #29 ). Changes localhost to 127.0.0.1 in test artifact, because localhost doesn't work in a Nix environment.
2024-08-20 17:44:41 +02:00
37176178f1
Restored blacklist.
2024-08-19 19:09:41 +02:00
0a8b0c85c6
Fixed analysis part in Snakemake again, needs testing. Moved test and template artifacts back to parent folder. Changes analysis output dir.
2024-08-19 19:02:23 +02:00
ece5749b7a
Fixed analysis in Snakefile. Started working on plotting part.
2024-08-19 16:37:51 +02:00
f6f20cb0d6
Integrated the whole analysis part to the Snakefile, but needs testing ( #44 ).
2024-08-19 15:21:28 +02:00
e943be4efe
Integrated softenv analysis to workflow. Changed input and output cmd options in analysis scripts to take multiple files at once. Moved test and template artifacts in an excluded folder.
2024-08-19 14:59:08 +02:00
1e5dd77cd8
Working on the integration of the analysis to the snakemake workflow.
2024-08-16 17:41:20 +02:00
744b3b0d35
Added doc for running the Snakemake workflow.
2024-08-15 12:32:39 +02:00
0bbed9d0f5
Adding missing changes from the previous commit because I wasn't in the root folder...
2024-08-15 12:23:34 +02:00
cb1a62217c
The handling of the output of ECG and of ECG's own errors has been transfered to a new wrapper script that calls ECG. Now, ECG doesn't take an output log file as parameter, rather just write everything to the terminal. It is then retrieved by the wrapper scripts that writes the ouput to the given output log file, and logs any script crash. README has been updated according to these changes, and the verbose and log path options of ECG have been removed. Close #25 .
...
Started integrating the analysis to the workflow, not done yet (#44 ).
Removed legacy line and bar plot scripts.
2024-08-15 12:16:53 +02:00
aad9318e6e
Updated README with plot script and protocol.
2024-08-14 12:58:55 +02:00
13780c9f39
Finished bar plots. Merged bar and line plots into a single plot script.
2024-08-14 12:51:52 +02:00
b7a84c5a78
Improved the formatting of dates in R plots. Switched to a functionnal approach in the line plot. Broke the bar plot while trying to improve the formatting of dates :( Added the tidyverse R package to the Nix flake.
2024-08-09 18:31:17 +02:00
5aae58d680
Initial script for bar plotting. Continued line plot script (started in the last commit, but forgot to mention it).
2024-08-07 20:50:42 +02:00
e2903ffac1
Removed the headers from the output of the analysis, because they will have to be combined. So I made the columns deterministic. Added the supported package sources to the doc. Updated the protocol with the information from the doc + typos.
2024-08-07 19:51:21 +02:00
c816cbde2c
Updated doc.
2024-08-07 17:32:46 +02:00
a5e32f9224
Added timestamp to each row of each analysis' result. Package changes analysis now specifies if a package source has no packages that changed.
2024-08-07 17:31:35 +02:00
3d6b6d1ade
Updated doc with more details.
2024-08-07 16:51:19 +02:00
56c3682124
Finished the package changes analysis.
2024-08-07 12:26:36 +02:00
4b91a6cb5d
Modified analysis according to the fact that logs for multiple executions of ECG will not be appended to the same file, rather written to a new file every time. Added a column for the artifact name in the artifact hash log for that reason. Updated README. Added sed as dependency for Nix. Started writing package changes analysis.
2024-08-07 11:22:54 +02:00
eae7c40d59
Added timestamp to every row in package list ( close #38 ).
2024-08-06 16:59:36 +02:00
58dd52e403
Separated the analysis into 3 different scripts.
2024-08-06 16:50:07 +02:00
74228cd1c5
Now optional with a default value in the Nickel contract: version, comment, git_packages, misc_packages, python_venvs ( close #30 ).
2024-08-06 11:44:38 +02:00
ac6c100c6b
Fixed venv pkglist ( close #21 ).
2024-08-05 18:22:38 +02:00
ab72ee398c
output_analysis now takes multiple files and not just a single directory as argument for the analysis input ( close #36 ).
2024-08-05 17:19:45 +02:00
cd9a9000d5
Adding a column with the name of the artifact in each row of the package list ( close #37 ).
2024-08-05 17:04:42 +02:00
ea6d78dac2
Switched from an empty hash to indicate download failure to a hash equal to "-1". Fixed a bug where the cache folder would not be removed in case artifact download failed.
2024-08-05 16:29:07 +02:00
Quentin Guilloteau
199d1dc282
minor improvement to protocol
2024-08-02 17:49:55 +02:00
Quentin Guilloteau
43ceead6b0
Continue writing the protocol
2024-08-02 11:04:30 +02:00
Quentin Guilloteau
a24091b390
start writing protocol
2024-08-01 17:49:41 +02:00
114d0e5816
Added support for other virtualization techs in the Nickel contract ( close #28 ). Added Nix dependency for Snakemake diagram generation.
2024-07-30 18:06:16 +02:00
ff2fc2d365
Added version number to the Nickel contracts and artifacts ( close #27 ).
2024-07-29 16:39:30 +02:00
14ca1da160
Now handling the script's own crashes: saving to log and build status file ( close #25 ).
2024-07-29 15:17:22 +02:00
e8b25b74a7
Added build status analysis. Now logging build status even if build is successful, to make analysis easier (as mentionned in #26 ). Added a new error type.
2024-07-26 19:00:25 +02:00
4db323fb03
Implemented artifact hash analysis. Switched to a table of tables instead of a single table to be able to identify multiple artifacts for the artifact analysis.
2024-07-26 17:01:59 +02:00
