Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLDR-17014 No code fallbacks for language paths #4254

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
+73 −40
Draft

CLDR-17014 No code fallbacks for language paths #4254

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f846a3c
CLDR-17014 No code fallbacks for language paths
btangmu Dec 21, 2024
7efc8d8
CLDR-17014 Revise some tests so they pass
btangmu Dec 22, 2024
cdb5251
CLDR-17014 Move code for language extra paths to new ExtraPaths.java
btangmu Dec 23, 2024
aaf5e63
CLDR-17014 Debugging and minor optimization
btangmu Dec 23, 2024
e3f2f32
CLDR-17014 Fix testLSR using root.fullIterable()
btangmu Dec 23, 2024
518b994
CLDR-17014 Fix populateLanguagePaths bug
btangmu Dec 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion common/testData/localeIdentifiers/localeDisplayName.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2368,7 +2368,7 @@ zh-Hans-fonipa; zh (Hans, FONIPA)

en-MM; en (MM)
es; es
es-419; es_419
es-419; es (419)
es-Cyrl-MX; es (Cyrl, MX)
hi-Latn; hi (Latn)
nl-BE; nl (BE)
Expand Down
40 changes: 40 additions & 0 deletions tools/cldr-code/src/main/java/org/unicode/cldr/util/CLDRFile.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3148,6 +3148,10 @@ public Set<String> getRawExtraPaths() {
private List<String> getRawExtraPathsPrivate() {
Set<String> toAddTo = new HashSet<>();
SupplementalDataInfo supplementalData = CLDRConfig.getInstance().getSupplementalDataInfo();

// languages
getLanguageExtraPaths(toAddTo);

// units
PluralInfo plurals = supplementalData.getPlurals(PluralType.cardinal, getLocaleID());
if (plurals == null && DEBUG) {
Expand Down Expand Up @@ -3339,6 +3343,42 @@ private List<String> getRawExtraPathsPrivate() {
return toAddTo.stream().map(String::intern).collect(Collectors.toList());
}

private void getLanguageExtraPaths(Set<String> toAddTo) {
Set<String> codes =
StandardCodes.make().getSurveyToolDisplayCodes(NameType.LANGUAGE.getNameName());
codes.remove(XMLSource.ROOT_ID);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is wrong, my mistake -- it actually modifies the set in StandardCodes! Need to make a copy of the set. We might want to prevent this kind of bug by making getSurveyToolDisplayCodes return an unmodifiable set, maybe using Set.copyOf.

We currently have this, which doesn't work: CLDRLanguageCodes = CldrUtility.protectCollection(CLDRLanguageCodes);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by my last commit -- we still should look into Set.copyOf as an alternative to protectCollection

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: In general, use ImmutableSet.copyOf instead of Set.copyOf. The latter changes the order in the set, which may be important (TreeSet or LinkedHashSets get messed up, and it doesn't hurt for HashSet).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we need to fix CldrUtility.protectCollection(CLDRLanguageCodes); — if it does't work then a lot of other things could go wrong.

String[] extraCodes = {
"ar_001", "de_AT", "de_CH", "en_AU", "en_CA", "en_GB", "en_US", "es_419", "es_ES",
"es_MX", "fa_AF", "fr_CA", "fr_CH", "frc", "hi_Latn", "lou", "nds_NL", "nl_BE",
"pt_BR", "pt_PT", "ro_MD", "sw_CD", "zh_Hans", "zh_Hant"
};
codes.addAll(List.of(extraCodes));
for (Iterator<String> codeIt = codes.iterator(); codeIt.hasNext(); ) {
String code = codeIt.next();
String fullpath = NameType.LANGUAGE.getKeyPath(code);
toAddTo.add(fullpath);
}
toAddTo.add(languageAltPath("en_GB", "short"));
toAddTo.add(languageAltPath("en_US", "short"));
toAddTo.add(languageAltPath("az", "short"));
toAddTo.add(languageAltPath("ckb", "menu"));
toAddTo.add(languageAltPath("ckb", "variant"));
toAddTo.add(languageAltPath("hi_Latn", "variant"));
toAddTo.add(languageAltPath("yue", "menu"));
toAddTo.add(languageAltPath("zh", "menu"));
toAddTo.add(languageAltPath("zh_Hans", "long"));
toAddTo.add(languageAltPath("zh_Hant", "long"));
}

private String languageAltPath(String code, String alt) {
String fullpath = NameType.LANGUAGE.getKeyPath(code);
// Insert the @alt= string after the last occurrence of "]"
StringBuffer fullpathBuf = new StringBuffer(fullpath);
return fullpathBuf
.insert(fullpathBuf.lastIndexOf("]") + 1, "[@alt=\"" + alt + "\"]")
.toString();
}

private void addPluralCounts(
Collection<String> toAddTo,
final Set<Count> pluralCounts,
Expand Down
31 changes: 1 addition & 30 deletions tools/cldr-code/src/main/java/org/unicode/cldr/util/XMLSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1604,7 +1604,6 @@ public String getLocaleID() {
Map<String, String> zone_countries = sc.getZoneToCountry();
List<NameType> nameTypeList =
List.of(
NameType.LANGUAGE,
NameType.SCRIPT,
NameType.TERRITORY,
NameType.VARIANT,
Expand All @@ -1628,36 +1627,10 @@ public String getLocaleID() {
if (s != null && s.size() == 1) continue;
}
value = TimezoneFormatter.getFallbackName(value);
} else if (nameType == NameType.LANGUAGE) {
if (ROOT_ID.equals(value)) {
continue;
}
}
addFallbackCode(nameType, code, value);
}
}

String[] extraCodes = {
"ar_001", "de_AT", "de_CH", "en_AU", "en_CA", "en_GB", "en_US", "es_419", "es_ES",
"es_MX", "fa_AF", "fr_CA", "fr_CH", "frc", "hi_Latn", "lou", "nds_NL", "nl_BE",
"pt_BR", "pt_PT", "ro_MD", "sw_CD", "zh_Hans", "zh_Hant"
};
for (String extraCode : extraCodes) {
addFallbackCode(NameType.LANGUAGE, extraCode, extraCode);
}

addFallbackCode(NameType.LANGUAGE, "en_GB", "en_GB", "short");
addFallbackCode(NameType.LANGUAGE, "en_US", "en_US", "short");
addFallbackCode(NameType.LANGUAGE, "az", "az", "short");

addFallbackCode(NameType.LANGUAGE, "ckb", "ckb", "menu");
addFallbackCode(NameType.LANGUAGE, "ckb", "ckb", "variant");
addFallbackCode(NameType.LANGUAGE, "hi_Latn", "hi_Latn", "variant");
addFallbackCode(NameType.LANGUAGE, "yue", "yue", "menu");
addFallbackCode(NameType.LANGUAGE, "zh", "zh", "menu");
addFallbackCode(NameType.LANGUAGE, "zh_Hans", "zh", "long");
addFallbackCode(NameType.LANGUAGE, "zh_Hant", "zh", "long");

addFallbackCode(NameType.SCRIPT, "Hans", "Hans", "stand-alone");
addFallbackCode(NameType.SCRIPT, "Hant", "Hant", "stand-alone");

Expand Down Expand Up @@ -1742,9 +1715,7 @@ private static void addFallbackCode(
NameType nameType, String code, String value, String alt) {
String fullpath = nameType.getKeyPath(code);
String distinguishingPath = addFallbackCodeToConstructedItems(fullpath, value, alt);
if (nameType == NameType.LANGUAGE
|| nameType == NameType.SCRIPT
|| nameType == NameType.TERRITORY) {
if (nameType == NameType.SCRIPT || nameType == NameType.TERRITORY) {
allowDuplicates.put(distinguishingPath, code);
}
}
Expand Down
2 changes: 2 additions & 0 deletions tools/cldr-code/src/test/java/org/unicode/cldr/unittest/TestCLDRFile.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,8 @@
path.contains("/metazone")
|| path.contains("/timeZoneNames")
|| path.contains("/gender")
|| path.startsWith(
"//ldml/localeDisplayNames/languages/language")
|| path.startsWith("//ldml/numbers/currencies/currency")
|| path.startsWith("//ldml/personNames/sampleName")
|| path.contains("/availableFormats")
Expand Down Expand Up @@ -914,7 +916,7 @@
}
String value = swissHighGerman.getStringValue(xpath);
if (value != null && value.indexOf('ß') >= 0) {
warnln("«" + value + "» contains ß at " + xpath);

Check warning on line 919 in tools/cldr-code/src/test/java/org/unicode/cldr/unittest/TestCLDRFile.java

View workflow job for this annotation

GitHub Actions / build 

 (TestCLDRFile.java:919)  Warning: «Deko | Emotion | Gefühl | Herz | Herzdekoration | Liebe | lila | violett | weiß» contains ß at //ldml/annotations/annotation[@cp="💟"]

Check warning on line 919 in tools/cldr-code/src/test/java/org/unicode/cldr/unittest/TestCLDRFile.java

View workflow job for this annotation

GitHub Actions / build 

 (TestCLDRFile.java:919)  Warning: «Emotion | Gefühl | Herz | Herzen | Hochzeitstag | Jahrestag | kreisen | Liebe | süß» contains ß at //ldml/annotations/annotation[@cp="💞"]

Check warning on line 919 in tools/cldr-code/src/test/java/org/unicode/cldr/unittest/TestCLDRFile.java

View workflow job for this annotation

GitHub Actions / build 

 (TestCLDRFile.java:919)  Warning: «alt | älter | erwachsen | Großeltern | Mensch | Person | weise» contains ß at //ldml/annotations/annotation[@cp="🧓"]

Check warning on line 919 in tools/cldr-code/src/test/java/org/unicode/cldr/unittest/TestCLDRFile.java

View workflow job for this annotation

GitHub Actions / build 

 (TestCLDRFile.java:919)  Warning: «aquamarin | Emotion | gefallen | gefällt | Gefühl | hdl | hellblau | Herz | Liebe | mag | mögen | niedlich | süß | türkis | zyan» contains ß at //ldml/annotations/annotation[@cp="🩵"]

Check warning on line 919 in tools/cldr-code/src/test/java/org/unicode/cldr/unittest/TestCLDRFile.java

View workflow job for this annotation

GitHub Actions / build 

 (TestCLDRFile.java:919)  Warning: «Gesicht | keine Ahnung | Smiley | traurig | verwirrt | verwundert | weiß nicht» contains ß at //ldml/annotations/annotation[@cp="😕"]
}
}
}
Expand Down
2 changes: 0 additions & 2 deletions tools/cldr-code/src/test/java/org/unicode/cldr/util/TestCLDRFile.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,8 +270,6 @@ public void testGetPaths() {
List<LocaleInheritanceInfo> pwf = f.getPathsWhereFound(p);
assertEquals(
List.of(
new LocaleInheritanceInfo(
XMLSource.CODE_FALLBACK_ID, GERMAN, Reason.constructed),
new LocaleInheritanceInfo(
XMLSource.ROOT_ID,
"//ldml/localeDisplayNames/localeDisplayPattern/localePattern",
Expand Down
Loading