Collection_Data_Types.rst 17.6 KB
Newer Older
1
2
3
.. sectnum::
   :start: 5
   
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
4
5
6
7
8
.. _Collection_Data_types:

*********************
Collection Data Types
*********************
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
9

10
Exercises
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
11
12
=========

13
Exercise
14
15
--------

16
17
| Draw the representation in memory of the following expressions.
| what is the data type of each object?
18
19
20
21
22
23
24
25

::   

   x = [1, 2, 3, 4]
   y = x[1]
   y = 3.14
   x[1] = 'foo'
   
26
27
28
29
30
.. figure:: _static/figs/list_1.png
   :width: 400px
   :alt: set
   :figclass: align-center
   
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
::

   x = [1, 2, 3, 4]
   x += [5, 6]

.. figure:: _static/figs/augmented_assignment_list.png  
   :width: 400px
   :alt: set
   :figclass: align-center 

::

   >>> x = [1, 2, 3, 4]
   >>> id(x)
   139950507563632
   >>> x += [5,6]
   >>> id(x)
   139950507563632
   
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
50
51
With mutable object like ``list`` when we mutate the object the state of the object is modified.
But the reference to the object is still unchanged.
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
52
So in this example we have two ways to access to the list [1,2] if we modify the state of the list itself.
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
53
54
but not the references to this object, then the 2 variables x and y still reference the list containing
[1,2,3,4]. 
55

56
57
58
59
60
61
62
compare with the exercise on string and integers:

Since list are mutable, when ``+=`` is used the original list object is modified, so no rebinding of *x* is necessary.
We can observe this using *id()* which give the memory adress of an object. This adress does not change after the
``+=`` operation.

.. note::
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
63
   even the results is the same there is a subtelty to use augmented operator.
64
65
66
   in ``a operator= b`` python looks up ``a`` ’s value only once, so it is potentially faster
   than the ``a = a operator b``.

67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98

compare ::

   x = 3
   y = x
   y += 3
   x = ?
   y = ?
   
   
.. figure:: _static/figs/augmented_assignment_int2.png  
   :width: 400px
   :alt: augmented_assignment
   :figclass: align-center 

   
and ::

   x = [1,2]
   y = x
   y += [3,4]
   x = ?
   y = ?  


.. figure:: _static/figs/augmented_assignment_list2.png  
   :width: 400px
   :alt: list extend
   :figclass: align-center 



99
Exercise
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
--------

wihout using python shell, what is the results of the following statements:  
 
.. note:: 
   sum is a function which return the sum of each elements of a list.
      
::
 
   x = [1, 2, 3, 4]
   x[3] = -4 # what is the value of x now ?
   y = sum(x)/len(x) #what is the value of y ? why ?
   
   y = 0

because sum(x) is an integer, len(x) is also an integer so in python2.x the result is an integer, 
all the digits after the periods are discarded.
In python3 we will obtain the expected result (see :ref:``) 
   
   
120
Exercise
121
122
123
124
--------

How to compute safely the average of a list? ::

Bertrand  NÉRON's avatar
Bertrand NÉRON committed
125
   float(sum(l)) / float(len(l))
126

127
Exercise
128
129
--------

130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
Draw the representation in memory of the following expressions. ::

   x = [1, ['a','b','c'], 3, 4]
   y = x[1]
   y[2] = 'z'
   # what is the value of x ?
   
.. figure:: _static/figs/list_2-1.png
   :width: 400px
   :alt: set
   :figclass: align-center
   

.. container:: clearer

    .. image :: _static/figs/spacer.png
       
 When we execute *y = x[1]*, we create ``y`` wich reference the list ``['a', 'b', 'c']``.
 This list has 2 references on it: ``y`` and ``x[1]`` .
   
   
.. figure:: _static/figs/list_2-2.png
   :width: 400px
   :alt: set
   :figclass: align-center
 
   
.. container:: clearer

    .. image :: _static/figs/spacer.png
       
   
 This object is a list so it is a mutable object.
 So we can access **and** modify it by the two ways ``y`` or ``x[1]`` ::
 
   x = [1, ['a','b','z'], 3, 4]
    
167
168
169
170
171
172
173
174
175
Exercise
--------

from the list l = [1, 2, 3, 4, 5, 6, 7, 8, 9] generate 2 lists l1 containing all odd values, and l2 all even values.::

   l = [1, 2, 3, 4, 5, 6, 7, 8, 9]
   l1 = l[::2]
   l2 = l[1::2]

176
    
177
Exercise
178
179
--------
   
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
generate a list containing all codons.
   
pseudocode:
"""""""""""

| *function all_codons()*
|     *all_codons <- empty list*
|     *let varying the first base*
|     *for each first base let varying the second base*
|     *for each combination first base, second base let varying the third base*
|     *add the concatenation base 1 base 2 base 3 to all_codons*
|     *return all_codons*

first implementation:
"""""""""""""""""""""
.. literalinclude:: _static/code/codons.py
   :linenos:
   :language: python

::

   python -i codons.py 
   >>> codons = all_codons()
   
:download:`codons.py <_static/code/codons.py>` .  

second implementation:
""""""""""""""""""""""

Mathematically speaking the generation of all codons can be the cartesiens product 
between 3 vectors 'acgt'. 
In python there is a function to do that in ``itertools module``: `https://docs.python.org/2/library/itertools.html#itertools.product <product>`_


.. literalinclude:: _static/code/codons_itertools.py
   :linenos:
   :language: python

::

   python -i codons.py 
   >>> codons = all_codons()
222
   
223
224
:download:`codons_itertools.py <_static/code/codons_itertools.py>` .

225
               
226
Exercise
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
227
228
229
230
231
232
233
234
235
236
--------

From a list return a new list without any duplicate, regardless of the order of items. 
For example: ::

   >>> l = [5,2,3,2,2,3,5,1]
   >>> uniqify(l)
   >>> [1,2,3,5] #is one of the solutions 


237
238
pseudocode:
"""""""""""
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
239

240
241
242
243
244
| *function uniqify(l)*
|     *uniq <- empty list*
|     *for each element of l*
|        *add element if is not in uniq*
|     *return uniq*
245

246
247
implementation:
"""""""""""""""
248

249
250
251
.. literalinclude:: _static/code/uniqify.py
   :linenos:
   :language: python
252

253
::
254

255
256
257
   >>> l=[1,2,3,2,3,4,5,1,2,3,3,2,7,8,9]
   >>> uniqify(l)
   [1, 2, 3, 4, 5, 7, 8, 9]
258

259
:download:`codons_itertools.py <_static/code/codons_itertools.py>` .
260

261
262
second implementation:
""""""""""""""""""""""
263

264
265
266
267
268
The problem with the first implementation come from the line 4.
Remember that the membership operator uses a linear search for list, which can be slow for very large collections.
If we plan to use ``uniqify`` with large list we should find a better algorithm.
In the specification we can read that uniqify can work *regardless the order of the resulting list*.
So we can use the specifycity of set ::
269

270
271
 
   >>> list(set(l))
272

273

274
275
Exercise
--------
276

277
We need to compute the occurence of all kmers of a given lenght present in a sequence.
278

279
Below we propose 2 algorithms. 
280

281
282
pseudo code 1
"""""""""""""
283

284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
|   *function get_kmer_occurences(seq, kmer_len)*
|      *all_kmers <- generate all possible kmer of kmer_len*
|      *occurences <- 0* 
|      *for each kmer in all_kmers*
|         *count occurence of kmer*
|         *store occurence*
     
pseudo code 2
"""""""""""""

|  *function get_kmer_occurences(seq, kmer_len)*
|     *all_kmers <- empty*
|     *from i = 0 to sequence length - kmer_len*
|        *kmer <- kmer startin at pos i im sequence*
|        *increase by of occurence of kmer*
 
300

301
.. note::
302

303
304
305
306
307
308
309
310
311
312
313
314
315
316
   Computer scientists typically measure an algorithm’s efficiency in terms of its worst-case running time, 
   which is the largest amount of time an algorithm can take given the most difficult input of a fixed size. 
   The advantage to considering the worst case running time is that we are guaranteed that our algorithm 
   will never behave worse than our worst-case estimate.
   
   Big-O notation compactly describes the running time of an algorithm. 
   For example, if your algorithm for sorting an array of n numbers takes roughly n2 operations for the most difficult dataset, 
   then we say that the running time of your algorithm is O(n2). In reality, depending on your implementation, it may be use any number of operations, 
   such as 1.5n2, n2 + n + 2, or 0.5n2 + 1; all these algorithms are O(n2) because big-O notation only cares about the term that grows the fastest with 
   respect to the size of the input. This is because as n grows very large, the difference in behavior between two O(n2) functions, 
   like 999 · n2 and n2 + 3n + 9999999, is negligible when compared to the behavior of functions from different classes, 
   say O(n2) and O(n6). Of course, we would prefer an algorithm requiring 1/2 · n2 steps to an algorithm requiring 1000 · n2 steps.

   When we write that the running time of an algorithm is O(n2), we technically mean that it does not grow faster than a function with a 
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
317
   leading term of c · n2, for some constant c. Formally, a function f(n) is Big-O of function g(n), or O(g(n)), when f(n) <= c · g(n) for some 
318
319
320
321
   constant c and sufficiently large n.

   For more on Big-O notation, see A `http://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/Beginner's <Guide to Big-O Notation>`_.
   
322

323
Compare the pseudocode of each of them and implement the fastest one. ::
324

325
326
327
328
329
330
331
332
333
   """gtcagaccttcctcctcagaagctcacagaaaaacacgctttctgaaagattccacactcaatgccaaaatataccacag
      gaaaattttgcaaggctcacggatttccagtgcaccactggctaaccaagtaggagcacctcttctactgccatgaaagg
      aaaccttcaaaccctaccactgagccattaactaccatcctgtttaagatctgaaaaacatgaagactgtattgctcctg
      atttgtcttctaggatctgctttcaccactccaaccgatccattgaactaccaatttggggcccatggacagaaaactgc
      agagaagcataaatatactcattctgaaatgccagaggaagagaacacagggtttgtaaacaaaggtgatgtgctgtctg
      gccacaggaccataaaagcagaggtaccggtactggatacacagaaggatgagccctgggcttccagaagacaaggacaa
      ggtgatggtgagcatcaaacaaaaaacagcctgaggagcattaacttccttactctgcacagtaatccagggttggcttc
      tgataaccaggaaagcaactctggcagcagcagggaacagcacagctctgagcaccaccagcccaggaggcacaggaaac
      acggcaacatggctggccagtgggctctgagaggagaaagtccagtggatgctcttggtctggttcgtgagcgcaacaca"""
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
334
335


336
In the first alogrithm. 
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
337

338
339
340
| we first compute all kmers we generate 4\ :sup:`kmer length`
| then we count the occurence of each kmer in the sequence
| so for each kmer we read all the sequence so the algorith is in O( 4\ :sup:`kmer length` * ``sequence length``) 
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
341

342
343
| In the secon algorithm we read the sequence only once 
| So the algorithm is in O(sequence length)
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
344
345


346
Compute the 6 mers occurences of the sequence above, and print each 6mer and it's occurence one per line.
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
347

348
349
350
.. literalinclude:: _static/code/kmer.py
   :linenos:
   :language: python
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
351

352
::
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
353

354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
   >>> s = """"gtcagaccttcctcctcagaagctcacagaaaaacacgctttctgaaagattccacactcaatgccaaaatataccacag
   ... gaaaattttgcaaggctcacggatttccagtgcaccactggctaaccaagtaggagcacctcttctactgccatgaaagg
   ... aaaccttcaaaccctaccactgagccattaactaccatcctgtttaagatctgaaaaacatgaagactgtattgctcctg
   ... atttgtcttctaggatctgctttcaccactccaaccgatccattgaactaccaatttggggcccatggacagaaaactgc
   ... agagaagcataaatatactcattctgaaatgccagaggaagagaacacagggtttgtaaacaaaggtgatgtgctgtctg
   ... gccacaggaccataaaagcagaggtaccggtactggatacacagaaggatgagccctgggcttccagaagacaaggacaa
   ... ggtgatggtgagcatcaaacaaaaaacagcctgaggagcattaacttccttactctgcacagtaatccagggttggcttc
   ... tgataaccaggaaagcaactctggcagcagcagggaacagcacagctctgagcaccaccagcccaggaggcacaggaaac
   ... acggcaacatggctggccagtgggctctgagaggagaaagtccagtggatgctcttggtctggttcgtgagcgcaacaca"""
   >>> s = s.replace('\n', '')
   >>> kmers = get_kmer_occurences(s, 6)
   >>> for kmer in kmers:
   >>>   print kmer[0], '..', kmer[1]
   gcagag .. 2
   aacttc .. 1
   gcaact .. 1
   aaatat .. 2
   
   
:download:`kmer.py <_static/code/kmer.py>` .
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
374
375
376


bonus:
377
""""""
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
378

379
Print the kmers by ordered by occurences.
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
380

381
382
| see `https://docs.python.org/2/library/stdtypes.html#mutable-sequence-types <sort>`_
| see `https://docs.python.org/2/library/operator.html#operator.itemgetter <operator.itemgetter>`_
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
383
384


385
386
387
.. literalinclude:: _static/code/kmer_2.py
   :linenos:
   :language: python
388

389
::
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
390

391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
   >>> s = """"gtcagaccttcctcctcagaagctcacagaaaaacacgctttctgaaagattccacactcaatgccaaaatataccacag
   ... gaaaattttgcaaggctcacggatttccagtgcaccactggctaaccaagtaggagcacctcttctactgccatgaaagg
   ... aaaccttcaaaccctaccactgagccattaactaccatcctgtttaagatctgaaaaacatgaagactgtattgctcctg
   ... atttgtcttctaggatctgctttcaccactccaaccgatccattgaactaccaatttggggcccatggacagaaaactgc
   ... agagaagcataaatatactcattctgaaatgccagaggaagagaacacagggtttgtaaacaaaggtgatgtgctgtctg
   ... gccacaggaccataaaagcagaggtaccggtactggatacacagaaggatgagccctgggcttccagaagacaaggacaa
   ... ggtgatggtgagcatcaaacaaaaaacagcctgaggagcattaacttccttactctgcacagtaatccagggttggcttc
   ... tgataaccaggaaagcaactctggcagcagcagggaacagcacagctctgagcaccaccagcccaggaggcacaggaaac
   ... acggcaacatggctggccagtgggctctgagaggagaaagtccagtggatgctcttggtctggttcgtgagcgcaacaca"""
   >>> s = s.replace('\n', '')
   >>> kmers = get_kmer_occurences(s, 6)
   >>> for kmer, occ in kmers:
   >>>   print kmer, '..', occ
   cacagg .. 4
   aggaaa .. 4
   ttctga .. 3
   ccagtg .. 3
   
   
:download:`kmer_2.py <_static/code/kmer_2.py>` .
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
411
412


413
414
415
Exercise
--------

416
417
418
419
420
| Write a function which take a sequence as parameter and return it's reversed complement.
| Write the pseudocode before to propose an implementation.

pseudocode:
"""""""""""
421

422
423
424
425
426
427
| *function reverse_comp(sequence)*
|     *complement <- establish a correpondance and each base and its complement*
|     *rev_seq <- revert the sequence*
|     *rev_comp <- empty*
|     *for each nt of rev_seq*
|        *concatenate nt complement to rev_comp*
428
|     *return rev_comp*
429

430
431
432
.. literalinclude:: _static/code/rev_comp2.py
   :linenos:
   :language: python
433

434
435
436
437
438
::
   >>> from rev_comp import rev_comp
   >>>
   >>> seq = 'acggcaacatggctggccagtgggctctgagaggagaaagtccagtggatgctcttggtctggttcgtgagcgcaacaca'
   >>> print rev_comp(seq)
439
   tgtgttgcgctcacgaaccagaccaagagcatccactggactttctcctctcagagcccactggccagccatgttgccgt
440
441
442
   
:download:`rev_comp.py <_static/code/rev_comp.py>` .

443
  
444
445
446
447
448
449
450
451
452
453
454
455
456
457
other solution
""""""""""""""

python provide an interresting method for our problem. 
The ``translate`` method work on string and need a parameter which is a object
that can do the correspondance between characters in old string a the new one.
``maketrans`` is a function in module ``string`` that allow us to build this object.
``maketrans`` take 2 arguments, two strings, the first string contains the characters
to change, the second string the corresponding characters in the new string.
Thus the two strings **must** have the same lenght. The correspondance between
the characters to change and their new values is made in funtion of thier position.
the first character of the first string will be replaced by the first character of the second string,
the second character of the first string will be replaced by the second character of the second string, on so on.   
So we can write the reverse complement without loop.
458
459
460
461
   
.. literalinclude:: _static/code/rev_comp2.py
   :linenos:
   :language: python
462
463

::
464
465
466
467
468
469
470
   >>> from rev_comp2 import rev_comp
   >>>
   >>> seq = 'acggcaacatggctggccagtgggctctgagaggagaaagtccagtggatgctcttggtctggttcgtgagcgcaacaca'
   >>> print rev_comp(seq)
   tgtgttgcgctcacgaaccagaccaagagcatccactggactttctcctctcagagcccactggccagccatgttgccgt
   
:download:`rev_comp2.py <_static/code/rev_comp2.py>` .
471

472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
Exercise
--------

let the following enzymes collection: ::
 
   import collections
   RestrictEnzyme = collections.namedtuple("RestrictEnzyme", "name comment sequence cut end")

   ecor1 = RestrictEnzyme("EcoRI", "Ecoli restriction enzime I", "gaattc", 1, "sticky")
   ecor5 = RestrictEnzyme("EcoRV", "Ecoli restriction enzime V", "gatatc", 3, "blunt")
   bamh1 = RestrictEnzyme("BamHI", "type II restriction endonuclease from Bacillus amyloliquefaciens ", "ggatcc", 1, "sticky")
   hind3 = RestrictEnzyme("HindIII", "type II site-specific nuclease from Haemophilus influenzae", "aagctt", 1 , "sticky")
   taq1 = RestrictEnzyme("TaqI", "Thermus aquaticus", "tcga", 1 , "sticky")
   not1 = RestrictEnzyme("NotI", "Nocardia otitidis", "gcggccgc", 2 , "sticky")
   sau3a1 = RestrictEnzyme("Sau3aI", "Staphylococcus aureus", "gatc", 0 , "sticky")
   hae3 = RestrictEnzyme("HaeIII", "Haemophilus aegyptius", "ggcc", 2 , "blunt")
   sma1 =  RestrictEnzyme("SmaI", "Serratia marcescens", "cccggg", 3 , "blunt")

and the 2 dna fragments: ::

   dna_1 = """tcgcgcaacgtcgcctacatctcaagattcagcgccgagatccccgggggttgagcgatccccgtcagttggcgtgaattcag
   cagcagcgcaccccgggcgtagaattccagttgcagataatagctgatttagttaacttggatcacagaagcttccaga
   ccaccgtatggatcccaacgcactgttacggatccaattcgtacgtttggggtgatttgattcccgctgcctgccagg"""

   dna_2 = """gagcatgagcggaattctgcatagcgcaagaatgcggccgcttagagcgatgctgccctaaactctatgcagcgggcgtgagg
   attcagtggcttcagaattcctcccgggagaagctgaatagtgaaacgattgaggtgttgtggtgaaccgagtaag
   agcagcttaaatcggagagaattccatttactggccagggtaagagttttggtaaatatatagtgatatctggcttg"""

| which enzymes cut the dna_1 ?
|                  the dna_2 ?
|                  the dna_1 but not the dna_2?


#. Write a function *seq_one_line* which take a multi lines sequence and return a sequence in one line.
#. Write a function *enz_filter* which take a sequence and a list of enzymes and return a new list containing 
   the enzymes which are a binding site in the sequence
#. use the functions above to compute the enzymes which cut the dna_1 
   apply the same functions to compute the enzymes which cut the dna_2
   compute the difference between the enzymes which cut the dna_1 and enzymes which cut the dna_2
   
512
.. literalinclude:: _static/code/enzyme_1.py
513
514
515
516
517
518
519
520
521
522
523
524
525
   :linenos:
   :language: python

::
   from enzyme_1 import *
   
   enzymes = [ecor1, ecor5, bamh1, hind3, taq1, not1, sau3a1, hae3, sma1]
   dna_1 = one_line(dna_1)
   dans_2 = one_line(dna_2)
   enz_1 = enz_filter(enzymes, dna_1)
   enz_2 = enz_filter(enzymes, dna_2) 
   enz1_only = set(enz_1) - set(enz_2)

526
:download:`enzymes_1.py <_static/code/enzyme_1.py>` .
527
528
529
530
531
532
533
534
535
536

with this algorithm we find if an enzyme cut the dna but we cannot find all cuts in the dna for an enzyme. ::

   enzymes = [ecor1, ecor5, bamh1, hind3, taq1, not1, sau3a1, hae3, sma1]
   digest_1 = []
   for enz in enzymes:
      print enz.name, dna_1.count(enz.sequence)

the latter algorithm display the number of occurence of each enzyme, But we cannot determine the position of every sites.
We will see how to do this later.
537
538


Bertrand  NÉRON's avatar
Bertrand NÉRON committed
539
      
540
Exercise
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
--------

given the following dict : ::

   d = {1 : 'a', 2 : 'b', 3 : 'c' , 4 : 'd'}
   
We want obtain a new dict with the keys and the values inverted so we will obtain: ::

   inverted_d  {'a': 1, 'c': 3, 'b': 2, 'd': 4}

solution ::

   inverted_d = {}
   for key in d.keys():
       inverted_d[d[key]] = key
       
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
557
558
559
560
561
562
solution ::

   inverted_d = {}
   for key, value in d.items():
       inverted_d[value] = key
              
Bertrand  NÉRON's avatar
Bertrand NÉRON committed
563
564
solution ::

565
566
   inverted_d = {v : k for k, v in d.items()}