IL Feedback #883
Comments
If I understand correctly, "לכולם" is a surface word that corresponds to two syntactic words "ל" and "כולם". (the join=right is because comma follows). A similar case form IT sample would be This gets converted to CoNLL-U like: and to vert like Note that vert (exactly for cases like this) have multivalued attributes on norm, lemma etc. Not ideal, but best we can do with vertical files. |
random person check דורון אביטל
I have checked random person: https://github.com/GiliGoldin/ParlaMint/blob/8040ae5cd6579b7e4f414517a766ce5ce8b93f74/Samples/ParlaMint-IL/ParlaMint-IL-listPerson.xml#L241-L254 <person xml:id="person.18990">
<persName>
<forename>דורון</forename>
<surname>אביטל</surname>
</persName>
<sex value="M"/>
<birth when="1959-01-22">
<placeName>ישראל</placeName>
</birth>
<affiliation ref="#org.122" role="member" from="2011-03-18" to="2013-02-05"/>
<affiliation ref="#ParlaMint-IL-KNESS" role="member" from="2011-03-18" to="2013-02-05"/>
<affiliation ref="#ParlaMint-IL-GOV" role="member" from="2011-03-18" to="2013-02-05"/>
<affiliation ref="#ParlaMint-IL-GOV" role="minister" from="2011-03-18" to="2013-02-05"/>
</person>His parliamentary group status at the time of membership: <relation name="opposition" active="#org.122" passive="#ParlaMint-IL-GOV" from="2009-03-31" to="2012-05-08"/>
<relation name="coalition" mutual="#org.122" from="2012-05-08" to="2012-07-17"/>
<relation name="opposition" active="#org.122" passive="#ParlaMint-IL-GOV" from="2012-07-17" to="2013-02-05"/>There are some weirds:
Not sure if you understand the concept of members of the government in ParlaMint. It seems that all parliament members who are affiliated with the parliamentary group in the coalition are members of the government. |
|
INVALID @matyaskopp fault:
|
|
Sorry I don't understand. What is the repetition? Shouldn't there be a
meeting for each term? It's written in Hebrew and in English. How should it
be then?
…On Mon, Nov 25, 2024 at 8:36 AM Matyáš Kopp ***@***.***> wrote:
<meeting> element in teiCorpus
- non-unique meeting element in teiCorpus
<meeting> element should be unique within the file, there are repetitions
in a corpus root file:
https://github.com/GiliGoldin/ParlaMint/blob/8040ae5cd6579b7e4f414517a766ce5ce8b93f74/Samples/ParlaMint-IL/ParlaMint-IL.xml#L13-L36
<meeting n="14"
corresp="#ParlaMint-IL-KNESS"
ana="#parla.uni #parla.term #period_14"
xml:lang="he">הכנסת ה-14</meeting>
<meeting n="14"
corresp="#ParlaMint-IL-KNESS"
ana="#parla.uni #parla.term #period_14"
xml:lang="en">14th Knesset</meeting>
<meeting n="18"
corresp="#ParlaMint-IL-KNESS"
ana="#parla.uni #parla.term #period_18"
xml:lang="he">הכנסת ה-18</meeting>
<meeting n="18"
corresp="#ParlaMint-IL-KNESS"
ana="#parla.uni #parla.term #period_18"
xml:lang="en">18th Knesset</meeting>
<meeting n="24"
corresp="#ParlaMint-IL-KNESS"
ana="#parla.uni #parla.term #period_24"
xml:lang="he">הכנסת ה-24</meeting>
<meeting n="24"
corresp="#ParlaMint-IL-KNESS"
ana="#parla.uni #parla.term #period_24"
xml:lang="en">24th Knesset</meeting>
—
Reply to this email directly, view it on GitHub
<#883 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAUPHHRALKQI4WGVG4P2EL2CLANJAVCNFSM6AAAAABSJLAHHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJWHE4DCOBSGI>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
Of course, you are right. |
extra filesremove from repository:
|
|
You are right, it seems that I inserted the time of his faction membership
as the time in the government instead of the time in the coalition. I will
fix this.
I did assign each coalition member as a government member and as a
minister. I see now that this is a mistake, I will remove the minister role
since I don't have the information regarding the roles of the ministers and
government positions. In our corpus we do consider all the people in the
coalition to be government members.
…On Sun, Nov 24, 2024, 23:49 Matyáš Kopp ***@***.***> wrote:
random person check דורון אביטל
- government affiliation of דורון אביטל (
https://he.wikipedia.org/wiki/%D7%93%D7%95%D7%A8%D7%95%D7%9F_%D7%90%D7%91%D7%99%D7%98%D7%9C
)
I have checked random person:
https://github.com/GiliGoldin/ParlaMint/blob/8040ae5cd6579b7e4f414517a766ce5ce8b93f74/Samples/ParlaMint-IL/ParlaMint-IL-listPerson.xml#L241-L254
<person xml:id="person.18990">
<persName>
<forename>דורון</forename>
<surname>אביטל</surname>
</persName>
<sex value="M"/>
<birth when="1959-01-22">
<placeName>ישראל</placeName>
</birth>
<affiliation ref="#org.122" role="member" from="2011-03-18" to="2013-02-05"/>
<affiliation ref="#ParlaMint-IL-KNESS" role="member" from="2011-03-18" to="2013-02-05"/>
<affiliation ref="#ParlaMint-IL-GOV" role="member" from="2011-03-18" to="2013-02-05"/>
<affiliation ref="#ParlaMint-IL-GOV" role="minister" from="2011-03-18" to="2013-02-05"/>
</person>
His parliamentary group status at the time of membership:
<relation name="opposition" active="#org.122" passive="#ParlaMint-IL-GOV" from="2009-03-31" to="2012-05-08"/>
<relation name="coalition" mutual="#org.122" from="2012-05-08" to="2012-07-17"/>
<relation name="opposition" active="#org.122" passive="#ParlaMint-IL-GOV" from="2012-07-17" to="2013-02-05"/>
There are some weirds:
- he is at the same time a member of government and in the opposition
- the government membership has the same timespan as parliament
membership (in Czechia, it takes some time(weeks-months) to become a
minister after becoming a parliament member)
- wiki does not say he was a minister
Not sure if you understand the concept of members of the government in
ParlaMint. It seems that all parliament members who are affiliated with the
parliamentary group in the coalition are members of the government.
https://clarin-eric.github.io/ParlaMint/#sec-affiliation
A member of government is someone who has some position in government (not
everyone from the coalition)
—
Reply to this email directly, view it on GitHub
<#883 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACAUPHBMBXSEFDNHDW7AJ7T2CJCWBAVCNFSM6AAAAABSJLAHHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJWGI2TEMJQGE>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
<langUsage>
<language ident="he">עברית</language>
<language ident="en">אנגלית</language>
<language ident="he">Hebrew</language>
<language ident="en">English</language>
</langUsage>should be: <langUsage>
<language ident="he" xml:lang="he">עברית</language>
<language ident="en" xml:lang="he">אנגלית</language>
<language ident="he" xml:lang="en">Hebrew</language>
<language ident="en" xml:lang="en">English</language>
</langUsage> |
There are still some taxonomies which are IL-specific or not linked: I guess they can be removed |
|
Thanks for the great progress; I have ticked what has been resolved so far. If anything is unclear, please ask.
Well, you made more changes than just removing ministers and fixing the beginnings of timespans in 199b869; see Netanyahu: Some remove seem to be correct (e.g. Netanyahu was not in government with Bennett) - I hope you are aware of these changes - it was a bugfix, not accidental removal. The government beginnings seem to be okay (if the start of the coalition is the start of the government), but now you have most probably time spans without government because you have shifted only beginnings (old government still works after new MPs make parliamentary oath). We have a script for enriching tei with tsv data: |
Url should contain the proper source of the transcription (if available), so everyone can see the source that you have transformed to corpus.
The sources can be found online but I don't have this specific URL information since we didn't process the files directly from the website, we received them in email directly from the Knesset archivists. |
Okay, it's a shame. You can add it to your checklist for improving your source corpus. |
I made sure to use the coalition dates rather than the faction membership dates. This caused all the mentioned changes which are correct now. The start of the coalition membership is the start of the government membership, not the parliamentary oath, but yes the end will be the end of the coalition membership. |
This was fixed according to what TomazErjavec suggested.
This was fixed
This was fixed. There are no IL-specifix taxonomies anymore
This was fixed
Those are factions that are made of only one independent MP, but they are considered as a regular faction/political party in the parliament. I don't see why to save them differently. |
@GiliGoldin, it was fixed only partially, see my previous comment: #883 (comment)
|
If this reflects the reality in Knesset, then do it this way - I am ok with it. |
join attribute
There are too many joins, so the raw TEI and annotated (TEI.ana) versions are different and then you can compare folders (I use meld):
|
|
@GiliGoldin, you removed your comment before I could react, so there are probably still some doubts. I can give you an example https://github.com/GiliGoldin/ParlaMint/blob/4571733fe48a9d200c92fd1ba7b02807bfc7ccfb/Samples/ParlaMint-IL/2009/ParlaMint-IL_2009-10-21-18ptv139208.ana.xml#L484-L519 on how should this sentence be encoded. <s xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0">
<w lemma="איפה"
pos="ADV"
msd="UPosTag=ADV|PronType=Int"
xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t1">איפה</w>
<w xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t2-3" join="right">המכינה<w xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t2"
norm="ה"
lemma="ה"
pos="DET"
msd="UPosTag=DET|Definite=Def|PronType=Art"/>
<w xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t3"
norm="מכינה"
lemma="מכינה"
pos="NOUN"
msd="UPosTag=NOUN|Gender=Fem|Number=Sing"/>
</w>
<w lemma="הוקם"
pos="VERB"
msd="UPosTag=VERB|Gender=Fem|HebBinyan=HUFAL|Number=Sing|Person=3|Tense=Fut|Voice=Pass"
xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t4">תוקם</w>
<pc xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t5"
msd="UPosTag=PUNCT"
join="right">?</pc>
<linkGrp targFunc="head argument" type="UD-SYN"><!-- SKIPPING --></linkGrp>
</s>It should be: <s xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0">
<w lemma="איפה"
pos="ADV"
msd="UPosTag=ADV|PronType=Int"
xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t1">איפה</w>
<!--
(ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t2-3) removing join="right"
because the token(ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t4)
on the right(=following) in this file is not joined
-->
<w xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t2-3">המכינה<w xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t2"
norm="ה"
lemma="ה"
pos="DET"
msd="UPosTag=DET|Definite=Def|PronType=Art"/>
<w xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t3"
norm="מכינה"
lemma="מכינה"
pos="NOUN"
msd="UPosTag=NOUN|Gender=Fem|Number=Sing"/>
</w>
<!--
(ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t4) added join="right"
because the punctation(ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t5) is joined with this token
-->
<w lemma="הוקם"
join="right"
pos="VERB"
msd="UPosTag=VERB|Gender=Fem|HebBinyan=HUFAL|Number=Sing|Person=3|Tense=Fut|Voice=Pass"
xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t4">תוקם</w>
<!--
(ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t5) removing join="right"
because the sentence is at the end of the sentence
-->
<pc xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u2.p0.s0.t5"
msd="UPosTag=PUNCT"
join="right">?</pc>
<linkGrp targFunc="head argument" type="UD-SYN"><!-- SKIPPING --></linkGrp>
</s>I am sorry if I have written it unambiguously in #882. I hope this example helps |
|
Yes, I removed the comment since I noticed more problems that needed fixing. |
The idea is to store source spacing, not to create a typographically correct one. But the current state is much better - it does not break the text, so we can leave it as it is. |
|
I have spotted one easy-fix join issue: Make sure that the last token in a sentence does not contain the <s xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u10.p0.s1">
<!-- SKIPPING -->
<pc xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u10.p0.s1.t12"
msd="UPosTag=PUNCT"
join="right">-</pc>
<pc xml:id="ParlaMint-IL_2009-10-21-18ptv139208.u10.p0.s1.t13"
msd="UPosTag=PUNCT"
join="right">-</pc> <!-- REMOVE THIS JOIN -->
<linkGrp targFunc="head argument" type="UD-SYN"><!-- SKIPPING --></linkGrp>
</s>I believe your pipeline will be ready to run on all data when you fix this. @GiliGoldin, thanks for the exceptional work! @TomazErjavec, just for your update, ParlaMint-IL sample is close to being ready. |
That's great, thank you so much! |
|
Hi, the full data is located here: Happy holidays! |
|
@GiliGoldin, thanks for letting us know. Will have a look and try to process it soon. |
Thanks for the great work on the corpora!
Please do not be scared of a long task list (everyone received it). I hope it will help you improve your corpus. I am ready to help and discuss any ambiguities or doubts, so do not hesitate to ask.
Are component filenames really unique
The filenames (file IDs
/TEI/@id) must be unique. I am not sure if multiple plenary/committee meetings can be held on the same day.maintitle unique and also in Hebrew
The text value of the main title in component files has to be unique within the corpus and there also should be Hebrew translation:
https://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/2021/ParlaMint-IL_2021-12-21.xml#L9
so instead of
reference corpus, you can place date and some more info that makes it unique (because you encode committees too, I believe the date is not enough):<meeting>element in plenarysparla.termparla.session?parla.meetingparla.sittingValues from the
<meeting>elements are used in corcondancers for filtering transcriptions, so the correct encoding is really important. See documentation: https://clarin-eric.github.io/ParlaMint/#exa-titleStmtCompand also the taxonomy:
I believe this plenary hearing file: https://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/2009/ParlaMint-IL_2009-03-12.xml#L12
should be encoded this way
<meeting>element in committeesparla.termparla.meetingparla.sittinghttps://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/2021/ParlaMint-IL_2021-12-21.xml#L12
can be encoded this way:
It is a pity you do not have committee organizations and texts so they can be linked. ParlaMint-BE has committee meetings too (but no
<org>anization). On the other hand CZ and HU have organizations but not corresponding texts. It would be great to have one corpus that has both :-)<meeting>element in teiCorpusparla.termThere should be a list of terms in the
<meeting>elements in corpus root files, like this:ParlaMint/Samples/ParlaMint-AT/ParlaMint-AT.xml
Lines 10 to 17 in f9a0b6a
annotation of the file
TEI/@ana#parla.sittingintoTEI/@anaAdd
#parla.sittingintoTEI/@anaif one file corresponds to one sitting or the#parla.meetingvalue can be used if sitting is one to one to meeting.bibliography
idno URL- texts available online, but the source is a different corpus that does not preserve this informationhttps://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/2009/ParlaMint-IL_2009-03-12.xml#L52-L53
should contain correct single day
when="2009-03-12"- the day of making text public or meeting date.Url should contain the proper source of the transcription (if available), so everyone can see the source that you have transformed to corpus.
settingDesc
<setting>Take a look at examples from other corpora:
ParlaMint/Samples/ParlaMint-AT/2022/ParlaMint-AT_2022-10-12-027-XXVII-NRSITZ-00178.xml
Lines 111 to 119 in 9946040
ParlaMint/Samples/ParlaMint-CZ/2023/ParlaMint-CZ_2023-07-26-ps2021-071-07-000-000.xml
Lines 106 to 114 in 9946040
ParlaMint/Samples/ParlaMint-SI/2022/ParlaMint-SI_2022-04-06-SDZ8-Izredna-99.xml
Lines 95 to 101 in 9946040
ID format
u/@idseg/@ids/@idw/@idandpc/@idI know that ID value is just for technical purposes, but consider changing them in the way most corpora do it, something like
{file_id}.u{utteranceN}.p{paragraphN}.s{sentenceN}.w{tokenN}(CZech style of creating ids).changed ids in annotated version
For technical reasons, we want to preserve utterances and segment ids in annotated versions (they would be equal). When you annotate the corpus, you are only enriching it, not changing existing content.
https://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/2009/ParlaMint-IL_2009-10-21.xml#L97-L100
vs annotated version:
https://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/2009/ParlaMint-IL_2009-10-21.ana.xml#L103-L106
syntactic vs orthographic words
https://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/2009/ParlaMint-IL_2009-10-21.ana.xml#L107-L130
annotation with udpipe for easier illustration:
it should be encoded this way:
Or the ways documented here: https://clarin-eric.github.io/ParlaMint/#sec-ana-norm
Not sure... "לכולם" does not have a lemma....
@TomazErjavec please help me here. We need to be able to convert it into conllu and vert. It would also be great if it would be possible to search it as one word for users...
named entities
I have found only single-word named entities which were adjected, like this:
taxonomies
there are two types of taxonomies
ParlaMint-taxonomy-prefix where no changes are allowed, only translation is required (except UD-SYN)ParlaMint-IL-taxonomy-You have changed the content of the common taxonomies and also the filenames, so the taxonomies do not match the ParlaMint ones.
You can initialize common taxonomies with this command. Run it in the repository root folder:
it creates taxonomies in
Sample/ParlaMint-ILand place placeholders where the translations should appear (it overwrites existing ones if the filename is equal)If you have the correct filename and IDs, you can use this sequence to prefill your translations:
languages
<langUsage>https://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/ParlaMint-IL.ana.xml#L143-L146
there should be both(
@ident) languages information stored in both(@xml:lang) languages, like this:ParlaMint/Samples/ParlaMint-CZ/ParlaMint-CZ.ana.xml
Lines 147 to 152 in f9a0b6a
invalit label content
org/listEvent/event/labelcontenthttps://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/ParlaMint-IL-listOrg.xml#L420-L427
This appears in multiple organizations; the above is just a sample.
abbreviated form is longer than full
org/orgNamehttps://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/ParlaMint-IL-listOrg.xml#L647-L648
This appears in multiple organizations; the above is just a sample.
independent MP forms parliamentary group
https://github.com/GiliGoldin/ParlaMint/blob/27a4fa70319f58c2dfeaf5e8bae00eff0f10fc8a/Samples/ParlaMint-IL/ParlaMint-IL-listOrg.xml#L2045-L2051
approx 30 occurrences.
This solution allows to affiliate with political orientation an independent MP, but it is really strange. Probably we have to find a better solution. (@TomazErjavec ??)
only member affiliations
corpus contains only
memberroles. is there a possibility to add various roles? See https://clarin-eric.github.io/ParlaMint/#sec-affiliationunknown person name
it is not necessary to fill in both forename and surname if unknown. If the person is completely unknown, then he/she shouldn't have a person record in listPerson (you can also skip
@whoattribute in utterance)The text was updated successfully, but these errors were encountered: