GuilloteauQ/study-docker-repro-longevity
1
0
Fork 0
Code Issues 6 Pull Requests Actions Projects Releases Wiki Activity

Commit Graph

  • b6e52dd92f 20241104 cleaning Quentin Guilloteau 2024-11-04 10:49:59 +0100
  • 34f66bcf2a fix --build-arg missing space Quentin Guilloteau 2024-10-01 17:51:32 +0200
  • ce34a87b8f First measures europar24 Quentin Guilloteau 2024-10-01 13:49:23 +0200
  • d5bb59468c analysis workflow Quentin Guilloteau 2024-09-12 14:07:43 +0200
  • 95fa82edaa fix the issue with the unsafe git repository Quentin Guilloteau 2024-09-12 14:07:19 +0200
  • db4e4f3baf system g5k by default Quentin Guilloteau 2024-09-11 10:51:59 +0200
  • a4d9ede97d add support for --build-arg flag Quentin Guilloteau 2024-09-11 10:51:38 +0200
  • 2a0b4373ac [europar24] all artifacts with dockerfile 🎉 Quentin Guilloteau 2024-09-11 09:42:38 +0200
  • 709a67caf5 Fix missing brace Quentin Guilloteau 2024-09-09 23:46:09 +0200
  • 14a5bd5c5e merge Quentin Guilloteau 2024-09-09 23:18:24 +0200
  • 18194b8f1a continue protocol Quentin Guilloteau 2024-09-09 23:17:24 +0200
  • ce977ceb9e now working on g5k Quentin Guilloteau 2024-08-29 18:16:12 +0200
  • 9289b10b84 start splitting the workflow into several pieces Quentin Guilloteau 2024-08-29 13:10:24 +0200
  • a62d338bd0 moving some config around Quentin Guilloteau 2024-08-29 12:58:50 +0200
  • 31e2ff0ca5 seperate the nix environments into several files Quentin Guilloteau 2024-08-29 11:41:29 +0200
  • d8d2a2e8e2 remove wrapper ecg Quentin Guilloteau 2024-08-29 11:23:54 +0200
  • 20c7238581 fix typo in ecg Quentin Guilloteau 2024-08-29 10:34:05 +0200
  • 742d6008f5 package ecg with nix Quentin Guilloteau 2024-08-29 10:32:54 +0200
  • 817ec821c5 big cleaning of ecg. probably does not work right now: to be tested Quentin Guilloteau 2024-08-29 10:22:51 +0200
  • 258cd64e40 removing useless lines from ecg Quentin Guilloteau 2024-08-28 14:37:00 +0200
  • 025a16b62c starting cleaning snakefile Quentin Guilloteau 2024-08-28 14:35:13 +0200
  • d57f8b019e Added some EuroPar 2024 artifacts. main antux18 2024-08-26 16:29:20 +0200
  • 348a1adc2c Updated protocol with some details and current state of the workflow. antux18 2024-08-26 15:12:08 +0200
  • 9c6ce2d150 Since analysis and aggregation rules had directories in their input, Snakemake complained because of these missing directories, even if he was supposed to create them. So I removed the directories from the inputs of the rules, and used global variables instead (and for some, just wrote the directory path directly in the shell command). Fixed the missing option specifier for the analysis wrapper for the softenv analysis. Also fixed the way arguments are parsed by the wrapper. antux18 2024-08-26 14:40:43 +0200
  • 97447e59a1 The results of the get_analysis_dates function being used as inputs in some Snakemake rules, this causes Snakemake to overwrite older analysis, probably because the files used to generate these older analysis have been modified. To avoid overwriting older analysis, we only specify today's analysis in the input, and use a bash wrapper script to both fetch the older analysis by itself (by giving it the folder where to look), and run the analysis with the fetched files. So I removed get_analysis_dates and replaced it with the wrapper script every time it was used. I also removed the older unfinished analysis wrapper that I forgot it existed... antux18 2024-08-23 16:14:46 +0200
  • 514e186c3d Updated doc with details on analysis outputs. Moved test scripts to a dedicated directory. antux18 2024-08-22 19:19:40 +0200
  • 0c281e1051 Added missing Python venv to test artifact. antux18 2024-08-22 19:00:36 +0200
  • be05d8d70a Deleted the Git package from the configuration file of the test artifact, because it is a Git repo that is generated when the container is being built, so version number will always change. antux18 2024-08-22 18:40:08 +0200
  • f2cf7e3498 Handling of the case where no artifact configuration files have been found, or they have all been blacklisted. antux18 2024-08-22 18:37:44 +0200
  • b8212061fd The date of the day is now always added to the output of get_analysis_dates, because we obviously want to generate the analysis of the day, even if there are older analysis in the folder already... antux18 2024-08-22 18:21:00 +0200
  • ce826c35a6 Fixed aggregation by separating it into multiple rules. Fixed calls to get_analysis_dates. Fixed plot headers and plot script in plotting rules, which are now working (close #44). Rule all has been modified according to these changes. Removed global analysis rule. List of artifacts to build now generated using all available blacklists, not just the last one (ensures that we don't miss some blacklisted artifacts). Fixed update blacklist rule, because of grep's output. Removed default blacklist. antux18 2024-08-22 17:55:05 +0200
  • 2d5b043f8f Fixed analysis and aggregate rules to get all ECG/analysis outputs for all possible dates. Written a function to get all available outputs for this purpose. Defined variables for arrays used multiple times. Simplified aggregate rule, but needs fix, because cannot have a list of shell commands apparently. Modified plotting rules according to those changes. antux18 2024-08-21 21:15:09 +0200
  • e3d01ae34e Updated doc with plotting script's changes. antux18 2024-08-21 17:07:10 +0200
  • 3eba87d47a Added an option to set the output path of the plotting script. Changed name and order of the options of the plotting script. antux18 2024-08-21 15:33:49 +0200
  • 52c629b9b8 Comments on Snakemake config files. antux18 2024-08-20 18:59:31 +0200
  • 9c6fe6700b The updating of the blacklist has been reworked to be functional: it is not relying on a symlink anymore (this symlink has been removed). Instead, the last blacklist is fetched by the get_artifacts_to_build function. Modified Snakefile according to the change in the name of a build status. Created a default config file for Snakemake, allowing custom configurations to be written to "config.yaml". antux18 2024-08-20 18:55:12 +0200
  • 513b21754c Changed build status "package_unavailable" to "package_install_failed" to cover more errors. Added an error message for this status. Outputs of ECG are now always created at the beginning to avoid issues with Snakemake (maybe not a good idea). antux18 2024-08-20 18:47:32 +0200
  • 72a638cd17 Added conference date to Nickel contract + artifacts (close #29). Changes localhost to 127.0.0.1 in test artifact, because localhost doesn't work in a Nix environment. antux18 2024-08-20 17:44:41 +0200
  • 37176178f1 Restored blacklist. antux18 2024-08-19 19:09:41 +0200
  • 0a8b0c85c6 Fixed analysis part in Snakemake again, needs testing. Moved test and template artifacts back to parent folder. Changes analysis output dir. antux18 2024-08-19 19:02:23 +0200
  • ece5749b7a Fixed analysis in Snakefile. Started working on plotting part. antux18 2024-08-19 16:37:51 +0200
  • f6f20cb0d6 Integrated the whole analysis part to the Snakefile, but needs testing (#44). antux18 2024-08-19 15:21:28 +0200
  • e943be4efe Integrated softenv analysis to workflow. Changed input and output cmd options in analysis scripts to take multiple files at once. Moved test and template artifacts in an excluded folder. antux18 2024-08-19 14:59:08 +0200
  • 1e5dd77cd8 Working on the integration of the analysis to the snakemake workflow. antux18 2024-08-16 17:41:20 +0200
  • 744b3b0d35 Added doc for running the Snakemake workflow. antux18 2024-08-15 12:32:39 +0200
  • 0bbed9d0f5 Adding missing changes from the previous commit because I wasn't in the root folder... antux18 2024-08-15 12:23:34 +0200
  • cb1a62217c The handling of the output of ECG and of ECG's own errors has been transfered to a new wrapper script that calls ECG. Now, ECG doesn't take an output log file as parameter, rather just write everything to the terminal. It is then retrieved by the wrapper scripts that writes the ouput to the given output log file, and logs any script crash. README has been updated according to these changes, and the verbose and log path options of ECG have been removed. Close #25. Started integrating the analysis to the workflow, not done yet (#44). Removed legacy line and bar plot scripts. antux18 2024-08-15 12:16:53 +0200
  • aad9318e6e Updated README with plot script and protocol. antux18 2024-08-14 12:58:55 +0200
  • 13780c9f39 Finished bar plots. Merged bar and line plots into a single plot script. antux18 2024-08-14 12:51:52 +0200
  • b7a84c5a78 Improved the formatting of dates in R plots. Switched to a functionnal approach in the line plot. Broke the bar plot while trying to improve the formatting of dates :( Added the tidyverse R package to the Nix flake. antux18 2024-08-09 18:31:17 +0200
  • 5aae58d680 Initial script for bar plotting. Continued line plot script (started in the last commit, but forgot to mention it). antux18 2024-08-07 20:50:42 +0200
  • e2903ffac1 Removed the headers from the output of the analysis, because they will have to be combined. So I made the columns deterministic. Added the supported package sources to the doc. Updated the protocol with the information from the doc + typos. antux18 2024-08-07 19:51:21 +0200
  • c816cbde2c Updated doc. antux18 2024-08-07 17:32:46 +0200
  • a5e32f9224 Added timestamp to each row of each analysis' result. Package changes analysis now specifies if a package source has no packages that changed. antux18 2024-08-07 17:31:35 +0200
  • 3d6b6d1ade Updated doc with more details. antux18 2024-08-07 16:51:19 +0200
  • 56c3682124 Finished the package changes analysis. antux18 2024-08-07 12:26:36 +0200
  • 4b91a6cb5d Modified analysis according to the fact that logs for multiple executions of ECG will not be appended to the same file, rather written to a new file every time. Added a column for the artifact name in the artifact hash log for that reason. Updated README. Added sed as dependency for Nix. Started writing package changes analysis. antux18 2024-08-07 11:22:54 +0200
  • eae7c40d59 Added timestamp to every row in package list (close #38). antux18 2024-08-06 16:59:36 +0200
  • 58dd52e403 Separated the analysis into 3 different scripts. antux18 2024-08-06 16:50:07 +0200
  • 74228cd1c5 Now optional with a default value in the Nickel contract: version, comment, git_packages, misc_packages, python_venvs (close #30). antux18 2024-08-06 11:44:38 +0200
  • ac6c100c6b Fixed venv pkglist (close #21). antux18 2024-08-05 18:22:38 +0200
  • ab72ee398c output_analysis now takes multiple files and not just a single directory as argument for the analysis input (close #36). antux18 2024-08-05 17:19:45 +0200
  • cd9a9000d5 Adding a column with the name of the artifact in each row of the package list (close #37). antux18 2024-08-05 17:04:42 +0200
  • ea6d78dac2 Switched from an empty hash to indicate download failure to a hash equal to "-1". Fixed a bug where the cache folder would not be removed in case artifact download failed. antux18 2024-08-05 16:29:07 +0200
  • 199d1dc282 minor improvement to protocol Quentin Guilloteau 2024-08-02 17:49:55 +0200
  • 43ceead6b0 Continue writing the protocol Quentin Guilloteau 2024-08-02 11:04:30 +0200
  • a24091b390 start writing protocol Quentin Guilloteau 2024-08-01 17:49:41 +0200
  • 114d0e5816 Added support for other virtualization techs in the Nickel contract (close #28). Added Nix dependency for Snakemake diagram generation. antux18 2024-07-30 18:06:16 +0200
  • ff2fc2d365 Added version number to the Nickel contracts and artifacts (close #27). antux18 2024-07-29 16:37:44 +0200
  • 14ca1da160 Now handling the script's own crashes: saving to log and build status file (close #25). antux18 2024-07-29 15:17:22 +0200
  • e8b25b74a7 Added build status analysis. Now logging build status even if build is successful, to make analysis easier (as mentionned in #26). Added a new error type. antux18 2024-07-26 19:00:25 +0200
  • 4db323fb03 Implemented artifact hash analysis. Switched to a table of tables instead of a single table to be able to identify multiple artifacts for the artifact analysis. antux18 2024-07-26 17:01:59 +0200
  • f25f5e356d Handling the case where artifact/misc packages cannot be downloaded. antux18 2024-07-26 13:28:26 +0200
  • 8574c5e238 Softenv analysis written. antux18 2024-07-26 12:58:00 +0200
  • 3c648c1a09 Written skeleton for output analysis (#26). Renamed some arguments of ECG. antux18 2024-07-25 18:03:14 +0200
  • e118cf3da1 Added the convention on config file name on the doc. antux18 2024-07-25 09:32:52 +0200
  • 1d925ca364 Now using the name of the config file (without ext) as image name, meaning the name of the config file should comply with Docker image naming rules (should be the case). antux18 2024-07-25 09:27:30 +0200
  • 48d3da9c7d Removed image name from the config file, now generated from the artifact URL (close #24). Fixed the "--no-cache" option when using Docker cache. antux18 2024-07-24 18:58:50 +0200
  • d2768d5373 Added initial support for Python venvs, close #21. Disabled the Docker build cache by default, and added an option to reenable it, close #23. Removed conditions on the existance of git_packages and misc_packages, because the workflow enforces their existance with the Nickel artifact contract. antux18 2024-07-24 18:05:31 +0200
  • aab563fbfe Fixed shi_welder (wrong path to Git repos). antux18 2024-07-24 16:17:11 +0200
  • 8687bb03f4 Switching from pip freeze to pip list because of inconsistent output mentionned in #18, close #22. Adding albab_k9db config. antux18 2024-07-24 11:34:05 +0200
  • 4329e333ea Added shi_welder, renamed OAR timeout error. antux18 2024-07-24 10:13:53 +0200
  • cc50349cf5 Almost solved the removing of bash login messages from outputs (#18) but still needs some work ("dace" pip package incorrectly displayed for deinsum). antux18 2024-07-23 21:04:34 +0200
  • 5ba63ad50e Fixed merge. antux18 2024-07-23 17:27:10 +0200
  • 687b777912 Merge remote-tracking branch 'origin/workflow_config' antux18 2024-07-23 17:26:03 +0200
  • 35f8b4c369 Merge branch 'main' of https://forge.chapril.org/GuilloteauQ/study-docker-repro-longevity antux18 2024-07-23 09:38:24 +0200
  • 557f766a02 Removing global variables, closing #19. antux18 2024-07-23 09:38:03 +0200
  • 944b1bf6f9 checkpoint in case of reaching walltime workflow_config Quentin Guilloteau 2024-07-21 17:32:04 +0200
  • ef257e6026 add prefix to config Quentin Guilloteau 2024-07-21 16:14:58 +0200
  • 566eb778f2 fix execution on grid5000 from namespace Quentin Guilloteau 2024-07-21 16:13:52 +0200
  • 318955865c typo Quentin Guilloteau 2024-07-21 16:12:40 +0200
  • dd6eb9bcbb now using tmp dir to decompress the artifact Quentin Guilloteau 2024-07-21 16:11:55 +0200
  • 8ed548c435 first try for g5k execution Quentin Guilloteau 2024-07-20 15:41:56 +0200
  • cc605943a9 Added a comment section for artifacts configuration. Moved hints to the Nickel contract. antux18 2024-07-19 17:05:46 +0200
  • d7749369b4 move artifacts_nickel and artifacts_json to artifacts/nickel and artifacts/json Quentin Guilloteau 2024-07-19 16:38:18 +0200
  • 9769f6b533 Add config for snakemake Quentin Guilloteau 2024-07-19 16:33:27 +0200
  • 9a29c526fb Small doc fixes. antux18 2024-07-19 16:23:39 +0200
  • a7bf9cee94 Written a doc for the output. Removed the "type" attribute from the Nickel contract, closing #17. antux18 2024-07-19 16:18:49 +0200
  • 1861f44f1e Removed the "type" attribute in the misc_package description. antux18 2024-07-19 15:28:58 +0200
  • cb98727554 Added artifacts_json to git ignore and deleting, closing #15. Deleted old YAML files. Added hints on the config template. antux18 2024-07-19 15:23:00 +0200