CHANGES.txt 5.83 KB
Newer Older
Olivier Sallou's avatar
Olivier Sallou committed
1
3.1.0:
2
  ## Needs database upgrade
3
  If using biomaj-watcher, must use version >= 3.1.0
4
  Feature #67,#66,#61 switch to micro service architecture. Still works in local monolithic install
Olivier Sallou's avatar
Olivier Sallou committed
5 6
  Fix some configuration parameter loading when not defined in config
  Fix HTTP parsing parameters loading
Olivier Sallou's avatar
Olivier Sallou committed
7
  Fix download_or_copy to copy files in last production release if available instead of downloading files again
Olivier Sallou's avatar
Olivier Sallou committed
8
  Manage user migration for micro services
9 10 11
  Feature #74 add influxdb statistics
  Feature #65 add a release info file at the root of the bank which can be used by other services to know the latest release available
  Feature #25 experimental support of rsync protocol
12
  Add rate limiting for download with micro services
13
  Limit email size to 2Mb, log file may be truncated
14 15
3.0.20:
  Fix #55: Added support for https and directhttps
16
  Add possibility to define files to download from a local file with remote.list parameter
Olivier Sallou's avatar
Olivier Sallou committed
17
  Fix visibility modification (bug deleted the bank properties field)
18
  Fix #65 Add release file in bank dir after update
19
  Add md5 or sha256 checksum checks if files are downloaded and available
Olivier Sallou's avatar
Olivier Sallou committed
20 21
3.0.19:
  Fix missing README.md in package
22
  Fix #53 avoid duplicates in pending databases
23 24
3.0.18:
  Add migration method to update schema when needed
25
  Manage HTTP month format to support text format (Jan, Feb, ...) and int format (01, 02, ...)
26
  New optional bank property http.parse.file.date.format to extract date in HTTP protocol following python date regexp format (http://www.tutorialspoint.com/python/time_strptime.htm)
27
      Example: %d-%b-%Y %H:%M
28 29 30 31
3.0.17:
  Fix #47: save_as error with directhttp protocol
  Fix #45: error with pending releases when release has dots in value
  typo/pylint fixes
Olivier Sallou's avatar
Olivier Sallou committed
32 33
3.0.16:
  Do not use config values, trust database values #39
Olivier Sallou's avatar
Olivier Sallou committed
34
  Fix #42: Add optional release.separator to name the bank directory bankname_release (underscore as default)
35 36
3.0.15:
  Fix #37: remote local files history from db and put it in cache.dir
37 38
  Feature #38: add optional keep.old.sessions parameter to keep all sessions in database, even for removed releases
  Feature #28: add optional release.format parameter to specify the date format of a release
39 40
3.0.14:
  Fix in method set_owner
Olivier Sallou's avatar
Olivier Sallou committed
41
  Force release to be a str
42
  Fix #32: fix --from-task issue when calling a meta process
Olivier Sallou's avatar
Olivier Sallou committed
43 44
  Fix #34: remove release from pending when doing cleanup of old sessions
  Remove logs on some operations
Olivier Sallou's avatar
Olivier Sallou committed
45
  Add --status-ko option to list bank in error state
46
  Fix #36 manage workflows over by error or unfinished
Olivier Sallou's avatar
Olivier Sallou committed
47 48
3.0.13:
  Fix #27: Thread lock issue during download
Olivier Sallou's avatar
Olivier Sallou committed
49
           New optional attribute in bank properties: timeout.download
Olivier Sallou's avatar
Olivier Sallou committed
50
  HTTP protocol fix (deepcopy error)
51
3.0.12:
52
  Fix index deletion on bank removal
Olivier Sallou's avatar
Olivier Sallou committed
53 54 55
  Fix lock errors on dir creation for multi-threads,
      pre-create directroy structure in offline directory
  Fix #26: save error when too many files in bank
Olivier Sallou's avatar
Olivier Sallou committed
56
3.0.11:
Olivier Sallou's avatar
Olivier Sallou committed
57 58 59
  Fix in session management with pre and rm processes
  Fix #23: Check workflow step name passed to
           --stop-after/--start-after/--from-task
Olivier Sallou's avatar
Olivier Sallou committed
60
  Fix #24: deprecated delete_by_query method in elasticsearch
Olivier Sallou's avatar
Olivier Sallou committed
61
  Add some controls on base directories
Olivier Sallou's avatar
Olivier Sallou committed
62

63 64
3.0.10:
  Change dir to process.dir to find processes in subdirs
65
  If all files found in offline dir, continue workflow with no download
66 67
  Remove extra log files for bank dependencies (computed banks)
  Fix computed bank update when sub banks are not updated
68
  Fix #15 when remote reverts to a previous release
69 70
  Feature #16: get possibility not to download files (for computed banks for
               example). Set protocol='none' in bank properties.
Olivier Sallou's avatar
Olivier Sallou committed
71
  Fix on --check with some protocols
Olivier Sallou's avatar
Olivier Sallou committed
72
  Fix #21 release.file not supported for directhttp protocol
73 74 75
  Feature #22: add localrelease and remoterelease bank properties to use the
               remote release as an expression in other properties
               => remote.dir = xx/yy/%(remoterelease)s/zz
Olivier Sallou's avatar
Olivier Sallou committed
76 77 78 79
  Feature #17,#20: detect remote modifications even if release is the same
               new parameter release.control (true, false) to force a check
               even if remote release (file controlled or date) is the same.
  Fix on 'multi' protocol
80
  Fix on "save_as" regexp when remote.files starts with a ^ character.
81

Olivier Sallou's avatar
Olivier Sallou committed
82
3.0.9:
Olivier Sallou's avatar
Olivier Sallou committed
83 84 85 86 87 88
  Fix thread synchro issue:
      during download some download threads could be alive while main thread continues worflow
      the fix prevents using Ctrl-C during download
  Workflow fix:
      if subtask of workflow fails, fail main task
osallou authored 14 hours ago
89 90 91
3.0.8:
  do not test index if elasticsearch is not up
  minor fixes
Olivier Sallou's avatar
Olivier Sallou committed
92 93
  add http proxy support
  pylint fixes
94
  retry uncompress once in case of failure (#13)
Olivier Sallou's avatar
Olivier Sallou committed
95
3.0.7:
Olivier Sallou's avatar
Olivier Sallou committed
96 97
  Reindent code, pep8 fixes
  Various fixes on var names and OrderedDict suport for Python < 2.7
Olivier Sallou's avatar
Olivier Sallou committed
98 99
  Merge config files to be able to reference global.properties variables in bank
    property file in format %(xx)s
100
  Use ConfigParser instead of SafeConfigParser that will be deprecated
101
3.0.6:
Olivier Sallou's avatar
Olivier Sallou committed
102
  Add option --remove-pending to remove all pending sessions and directories
103
  Add process env variables logdir and logfile
104
  Fix Unicode issue with old versions of PyCurl.
Olivier Sallou's avatar
Olivier Sallou committed
105 106 107
3.0.5:
  Fix removal workflow during an update workflow, removedrelease was current
  release.
Olivier Sallou's avatar
Olivier Sallou committed
108
  Fix shebang of biomaj-cli, and python 2/3 compat issue
Olivier Sallou's avatar
Olivier Sallou committed
109
3.0.4:
Olivier Sallou's avatar
Olivier Sallou committed
110
  Update code to make it Python 3 compatible
Olivier Sallou's avatar
Olivier Sallou committed
111
  Use ldap3 library (pure Python and p2,3 compatible) instead of python-ldap
Olivier Sallou's avatar
Olivier Sallou committed
112 113 114 115 116 117 118
  get possiblity to save downloaded files for ftp and http without keeping full
  directory structure:
    remote.files can include groups to save file without directory structure,
     or partial directories only, examples:
     remote.files = genomes/fasta/.*\.gz => save files in offline directory, keeping remote structure offlinedir/genomes/fasta/
     remote.files = genomes/fasta/(.*\.gz) => save files in offline directory offlinedir/
     remote.files = genomes/(fasta)/(.*\.gz) => save files in offline directory offlinedir/fasta