Files
obikmer/scripts/__pycache__/pseudo_alignment_nesting.cpython-314.pyc
T

171 lines
28 KiB
Plaintext
Raw Normal View History

+
”UÌj!Bãól€Rt^RIt^RIt^RItRtRtRtRtRt Rt
Rt ] R 8Xd
] !4R#R#)
aw
Family overlap and clade-closure ("nesting") of a SNP pseudo-alignment.
Each alignment column is one family; a genome "carries a call" at a column
when its character is not an absence symbol (default '-' and '?').
Outputs (prefix given by -o):
<prefix>_family_overlap.csv genome x genome matrix: number of columns where
both genomes carry a call (diagonal = 0, same
convention as `obikmer phylo --family-overlap`)
<prefix>_nesting.csv per clade-size stratum: how closed the families
are under the tree
Nesting measure: for a family carried by the genome set S, let C(S) be the
smallest tree clade containing S. completeness = |S| / |C(S)|. A value of 1
means the family is clade-closed (every genome of the clade carries it); low
values mean dropout or patchy conservation inside the clade.
The tree is rooted as written in the Newick file unless --outgroup is given
(rooted at the neighbour of that tip, which then becomes a child of the root).
Optional lift test (--lift-test): do families covering a rare pair (A,B) also
cover an abundant pair (A,C) more often than the margins alone imply? For each
genome A, B ranges over its --lift-frac rarest partners and C over its most
abundant ones (ranked on pair overlap). Statistics, on a random column subsample:
T = mean over triples of O_ABC / O_AB = P(C | A,B)
L = sum n_A * O_ABC / sum O_AB * O_AC (lift; 1 if B,C independent given A)
Null: curveball swap randomisation of the genome x family presence matrix, which
preserves every genome's family count and every family's size exactly. Each null
replicate re-selects the rare/abundant partners from its own overlaps, so the
selection procedure is identical to the observed one.
Optional quartet test (--quartet-test): does flattening pair overlaps also flatten
quartet coverage? --quartets N genome quartets are drawn uniformly; for each, the
number of columns where all four genomes carry a call is counted exactly. Reported
per stratum of the quartet's depth (size of the smallest tree clade containing the
four genomes): count distribution, share of uncovered quartets, and the ratio of the
4-way count to the smallest of the six pair overlaps (1 = perfect nesting; the pair
overlap is an upper bound of the 4-way count).
Usage
-----
pseudo_alignment_nesting.py ALIGNMENT.fasta TREEFILE -o prefix
[--outgroup LABEL] [--absent '-?'] [--chunk 131072]
[--quartet-test [--quartets 20000]]
[--lift-test [--lift-columns 200000] [--lift-frac 0.1]
[--null-reps 20] [--trades 5000] [--seed N]]
NcóRaaa a
€\P!R\R7oVFpRS\V4&K ..Ruoo
o VVV V
3RlpRp\ VR4;_uu_4pVF„pVP R4'dLVe V!V4SP
VR,P4^,P44.pKeVP
VP44K† RRR4Ve V!V4S\P!S
4S 3# +'giL5;i) zCReturn (labels, packed presence bits per genome, alignment length).é©ÚdtypeTNcóh<€\P!RPV4\PR7pSf
\ V4oM>\ V4S8wd/\
P !SR, R\ V4 RS 24SP\P!SV,(44R#)órNz : length z != éÿÿÿÿ) ÚnpÚ
frombufferÚjoinÚuint8ÚlenÚsysÚexitÚappendÚpackbits)ÚpartsÚseqÚ
absent_lutÚlabelsÚlengthÚpackeds& €€€€ÚO/Users/coissac/Sync/travail/__MOI__/obikmer/scripts/pseudo_alignment_nesting.pyÚflushÚ"read_fasta_presence.<locals>.flush?szø€ä�mŠm˜CŸH™H U›O´2·8±8Ô<ˆØ Š>ܘ“X‰FÜ
�‹X˜Ô
Ü �HŠH˜˜r�
�| 9¬S°«X¨J°d¸6¸(ÐCÔ DØ�
‰
”b—k’k :¨c¥?Ð"2Ó3Ö4rÚrbó>ºéNN) r ÚzerosÚboolÚordÚopenÚ
startswithrÚsplitÚdecodeÚstripÚvstack) ÚpathÚabsentÚchrrÚfhÚlinerrrrs && @@@@rÚread_fasta_presencer-8sðû€ä—’˜#¤TÔ*€JÛˆØ"ˆ
”3�r“7Óñà  TЀFˆF�F÷5ð5ð
€EÜ
ˆd�D× Ô ˜RÛˆDØ�‰˜t×$Ò$ØÒ$Ù˜%”LØ—
‘
˜d 2�hŸn™nÓ.¨qÕ1×8Ñ8Ó:Ô;Ø’à— ‘ ˜TŸZ™Z›\Ö*ñ÷
ð
ÒÙ
ˆeŒ Ø ”2—9’9˜VÓ$ fÐ ,Ð,÷
× ús
ÁB DÄ D& có–a
a a €...uo o o
.Rr!^\V4rCV
V V 3RlpW48EdžW,pVR9d V^,
pK!VR8XdUV'd VR ,pMV!4pVPV4RpV^,
pV!4pVS V&VPV4K|VR8Xd@VP4V!4pVR ,S V&VPV4V^,
pKÂVR8XdHVP4VR ,pV^,
pTp W”8dW ,R9d V ^,
p KT pEKVR8XdSV^,p W”8dW ,R 9d V ^,
p K\W^,V 4S
Vf
VR ,MT&RpT pEKiVR
8XdM3Tp W”8dW ,R9d V ^,
p KWV S VR ,&T pEK¤S S S
3#) zJParse Newick into (names, parent, length) arrays; internal labels ignored.NcóŽ<€SPR4SPR4SPR4\S4^,
#)Nçr)rr
)rÚnamesÚparents€€€rÚnew_nodeÚparse_newick.<locals>.new_node]s5ø€Ø
� ‰ �TÔØ�
‰
�bÔØ�
‰
�cÔÜ�5‹z˜A�~Ðrz
Ú(Ú,Ú)z:,);Ú:z,);Ú;r)r
rÚpopÚfloat)
ÚtextÚstackÚcurÚiÚnr3ÚcÚnodeÚchildÚjrr1r2s
& @@@rÚ parse_newickrEWs°ú€à  BЀEˆ6�6Ø�Tˆ3Ø Œc�$‹i€q÷ð �%Ø �GˆØ � Œ>Ø
��FŠAØ
�#ŒXߨ˜R•y‘á“z�Ø— ‘ ˜TÔ"؈CØ
��FˆAÙ“JˆEØ ˆF�5‰MØ �L‰L˜Ö Ø
�#ŒXØ �I‰IŒKÙ“JˆEØ! "�IˆF�5‰MØ �L‰L˜Ô Ø
��FŠAØ
�#ŒXØ �I‰IŒKؘ•)ˆCØ
��FˆA؈AØ”%˜D�G¨6Ô1Ø�Q•’Ø‹AØ
�#ŒXØ�A•ˆAØ”%˜D�G¨5Ô0Ø�Q•’Ü8=¸dÀqÅ5ȸmÓ8LˆF ¢ �5˜–9°Ñ 5؈CØ‹AØ
�#ŒXØ àˆAØ”%˜D�G¨6Ô1Ø�Q•’Ø# a˜yˆE�%˜•)Ñ Ø‹AØ �&˜&Ð Ð rc ó°€VUu.uFp.NK pp\V4FLwrgV^8¼gKWV,PWrV,34WW,PWbV,34KN \R\V444pVe(VPV4p WY,^,^,p.VR//rËp
V.V0ríV
'd°V
P 4pWV,UUu.uFwppWþ9gKVV3NK pppVF(wppVP V4W¶,V,W¿&K* VUUu.uFwrôVNK uppWÆ&V
P
\WÆ,44V
PV4K·V
Uu.uFqlV,'dKVNK pp\V4UUu/uFwpqfVbK
ppp\V4p/p\V
4FgpWÆ,'gVV,VV,^,3M9VWÆ,^,,^,VWÆ,R,,^,3VV&Ki \P!VV3\PR7p\P!VV34pV
FÇpWÆ,p\\V44F¤p\V^,\V44F�pVVV,,VVV,,uwppwppVV,^,VV,^,,
pV;VVV1VV13&VVV1VV13&W¶,;VVV1VV13&VVV1VV13&Kƒ K¦ KÉ \P!VUu.uF qkV,NK
up4p V R,V R,,^V,,
p!VUu.uF q`V,NK
upVV!3#uupiuuppiuuppiuupiuuppiuupiuupi)zHDFS leaf order plus LCA-clade-size and patristic matrices in that order.c3ó<"€TFwrV^8gKVx€K R#5i)éN©)Ú.0ÚvÚps& rÚ <genexpr>Ú&leaf_order_matrices.<locals>.<genexpr>™sé€Ð9Ñ/‘d�a°1°q±5—’Ó/ùs‚ ’
r0rr©ºNNNN©NrP)Ú enumeraterÚnextÚindexr:ÚaddÚextendÚreversedr
r ÚonesÚint32rÚrangeÚarray)"r1r2rÚoutgroupÚ_ÚadjrKrLÚrootÚtipÚorderÚdepthÚchildrenr=ÚseenÚuÚwÚkidsÚleavesÚrÚrankr@ÚspanÚcladeÚ lca_depthÚaÚbÚa0Úa1Úb0Úb1ÚsizeÚ
leaf_depthÚ patristics"&&&& rÚleaf_order_matricesrw’s_€áÓ
‘u�!‹2‘u€CÐ
ܘ&Ö!‰ˆØ �Ž6Ø �F�M‰M˜1 Q�i˜.Ô )Ø �F�M‰M˜1 Q�i˜.Ö )ñ"ô Ñ9œi¨Ô/Ó9Ó 9€DØÒØ�k‰k˜(Ó#ˆØ�x˜�{˜1�~ˆà $¨ ¨b�(€EØ�&˜4˜&ˆ4ß
Ø �I‰I‹KˆØ#&¦6Ô;¡6™4˜1˜a¨Q©]”��A“¡6ˆÑ;Û‰DˆAˆqØ �H‰H�QŒKØ•x !•|ˆE‹Hññ&*Ô*¡T™T˜Q“q¡TÒ*ˆ‰ Ø
� ‰ ”X˜h�kÓ*Ô+Ø
� ‰ �QŽáÓ
2™�A¨q§k¡k�aˆa™€FÐ
2Ü& vÔ.Ô /Ñ.‘T�Q˜ˆqŠDÑ.€DÑ /Ü ˆF‹ €AØ
€DÜ
�eŽ_ˆØ08· ´ �4˜•7˜D �G a�KÑ(Ø �•˜Q•Õ  Õ # T¨(­+°b­/Õ%:¸1Õ%=ðB?ˆˆQ‹ñô
�GŠG�Q˜�F¤"§(¡(Ô +€EÜ—’˜!˜Q˜Ó €IÛ
ˆØ�{ˆÜ”s˜4“yÖ!ˆAܘ1˜q�5¤# d£)Ö,�Ø%)¨$¨q­'¥]°D¸¸a½µMÐ"‘��R™(˜2˜rؘA•w˜q•z D¨¥G¨A¥JÕ.�Ø<@Ð@��b˜�e˜R ˜U�lÑ# e¨B¨r¨E°2°b°5¨LÑ&9ØDIÅHÐL� ˜"˜R˜%  B ˜,Ñ'¨)°B°r°E¸2¸b¸5°LÓ*Aó -ó"ñô—’©VÓ4©V¨ Ÿ8˜8©VÑ4Ó5€Jؘ7Õ# j°Õ&9Õ9¸AÀ ½MÕI€IÙ$Ó %™f˜�!�HˆH™fÑ % u¨iÐ 7Ð7ùòU ùó<ùó+ùò3ùÛ /ùò"5ùâ %s5… N2Ã" N7Ã2N7Ä0
N=Å:OÆOÆ%OÍOÎOcóz€VP^,pVP\P4pW3P,p\ ^\
W^,
,44pR;p;rx^p \V4EFªp
\P!WJ,RR7U u.uF
q»V
8wgK V NK p p V RVWÅ)RríW
,p\VP44pV^8XdKlW=,RV3,W>,RV3,P,pWJV
3,P\P4WJV3,P\P4ppVV\VP44,,
pV\VR,VR,,P44,
pV^8„pV\VV,VVR3,, P44,
pV \VP44V,,
p EK­ Wg, W‰, 3#uup i)zN(L, T) for the rare/abundant partner triples of every genome (see module doc).r0Ústable)ÚkindNrPrOrQ)
ÚshapeÚastyper Úfloat32ÚTÚmaxÚroundrZÚargsortr;ÚsumÚfloat64Úint)ÚpresÚfracr@ÚpfÚoverlapÚkÚnumÚdenÚ ratio_sumÚn_ratiornrDraÚrareÚabundantÚcolsÚn_aÚo_abcÚo_abÚo_acÚoks&& rÚ
lift_statsr–ÁsÀ€à �
‰
�1�
€AØ
�‰”R—Z‘ZÓ €BØ—4‘4�i€GÜ ˆAŒu�T �U•^Ó$Ó%€AØÐ€CЈ#Ø€GÜ
�1�XˆÜŸJšJ w¥z¸ÕAÓLÑA�qÈ!ÁV—�ÑAˆÐLؘr ˜ E¨"¨# JˆhØ�wˆÜ�D—H‘H“JÓˆØ �!Œ8Ù Ø•˜˜D˜Õ! B¥L°°D°Õ$9×$;Ñ$;Õ;ˆØ ˜WÕ%×,Ñ,¬R¯Z©ZÓ8¸'ÀXÀ+Õ:N×:UÑ:UÔVX×V`ÑV`Ó:aˆdˆØ ˆs”U˜5Ÿ9™9›;Ó'Õ'Õ'ˆØ Œu�d˜7•m d¨7¥mÕ3×8Ñ8Ó:Ó;Õ;ˆØ
�A‰XˆØ”U˜E "�I¨¨R°¨X­Õ6×;Ñ;Ó=Ó>Õ>ˆ Ø”3�r—v‘v“x“= 1Õ$Õ$‹ñð �9�iÕ)Ð )Ð)ùòMs ÂH8Â*H8có`€VP^,p\V4F‹pVPV^RR7wrVW,W,r‡\P!Wx, 4p \ Wx(,P
44p
VPV 4p RWy&RW‰&RWyRV
&RW‰V
R&K� R#)zGIn-place curveball trades: preserve every row sum and every column sum.F)rtÚreplaceTN)r{rZÚchoicer Ú flatnonzeror„r‚Ú permutation) r…ÚtradesÚrngr@r]r?rDrnroÚdiffr‰s &&& rÚ curveballrŸÚs˜€à �
‰
�1�
€AÜ
�6Ž]ˆØ�z‰z˜! !¨UˆzÓ3‰ˆØ�w˜�ˆ1Ü�~Š~˜a�eÓ$ˆÜ ��R•— ‘ “Ó ˆØ�‰˜tÓ$ˆØˆ‰Øˆ‰Øˆˆr�ˆ(‰ ؈ˆqˆrˆ(‹ órc
ó„€VP^,p\P!VPWG34^^R7R,p\P!V\P
R7p Rp