To whom it may concern
Hi Dear,
I have created an arabic scripture that contains tashkeels.
When I search any word without giving tashkeel, app searches nothing.
I want app to ignore tashkeel while searching any word.
Please work on this option for next release.
Humble regards.
Raja Sand
Pakistan
I think the developers would need more information about what characters you would like to be ignored in the search. From my experience it probably would be good to ignore the U+0640 ARABIC TATWEEL, used for kashida justification. But do you have other characters in mind as well? The developers would need information on specific code points, which can be found in the Unicode standard:
Dear,
Following is the list of Tashkeel/Harakat, that I want to get ignored while searching any word.
Thanks Jeff that does make it clearer.
Raja, the PDF you point to lists many code points. Are you saying you want each one included in the list to be ignored? From out point of view it would be clearer if you just list the code point you want SAB to ignore.
Dear Ian,
Really became much happy to see that you are trying to resolve the issue. Thanks in anticipation.
Yes! I want all these characters to be ignored by SAB while searching any word.
Actually these are called Harkat/tashkeel. These are not the main characters, These character are applied with other character to change its meaning.
The same word with different harakat (one of these) changes the meaning completely. So, if any does not know the exact word, that came in text with harkat, he just write simple word without any harkat, then he must get his/her word/words with different harakats.
Hope now the question is clear.
I am wondering whether one of the Search options (under Features) was designed to do something like this (except that it currently only specifies accents and tones). Perhaps it could be extended to other groups of characters that should be ignore when searching.
In my opinion, the tatweel should always be ignored. This character is simply used as a line filler, extending the baseline to make a word longer and better fill the line. Note that this character isnât even mentioned in the PDF Raja pointed to, as it is a different kind of thing.
I think the characters in Rajaâs PDF should not always be ignored, but I think adding the âMatch accents and tonesâ option to the app is a great way to handle this issue. So Raja - you would need to add this option to your app configuration, as shown in mjpennyâs screen shot above. I would recommend checking both of the boxes that he circled in red. It wonât work yet for you, but once this change is made in SAB, the user can check this box on the search page in the app to ignore the harkat/tashkeel when he is doing a search.
And to the developers: I assume the tatweel (always) and the other characters (only if the option is checked) would be removed from the search text in addition to the text being searched? I could imagine a case where someone copies a word with tatweels and pastes it into the search box, but if you are removing tatweels from the text being searched, it wouldnât be found unless you also removed the tatweels from the search text. Just a little sanity check⊠
All of the harakat characters in that document have an Mn category (Nonspacing_Mark) and that is the same category for other combining marks like acute and grave. So, Iâd hope that turning off âMatch accents and tonesâ might be the solution to the problem.
Also, I do know that in some cases tatweel/kashida is used as a mark holder (for example: tatweel plus combining hamza). In any case, the combining mark you would like to be ignored so maybe itâs okay to ignore tatweel as well.
Agreed!
It will better to add an option in search option menu either to ignore harkat/tashkeel or to search with harkat/tashkeel.
Raja,
I have reread your posts in TeamHelp and here.
You were asked the question âWhat are your settings for Features / Search?â That is the settings found that are like the picture below from SAB?
We need to know an answer to that question.
Have you tried different settings?
What differences do you see?
I have tried all these setting one by one, but all in vain.
After trying all possible options, I put my question in front of you.
Thanks for that clarification.
Although we have not yet got to that, an APP is planned for an Arabic script language including vowels. This is in the future, but it is good I see this communication. I can not yet say anything about the FEATURES/SEARCH OPTIONS but I can say something about the vowels & signs ( tashkeel) that should be ignored for this upcoming APP. First I give you a link to a very nice and detailed overview of all Arabic Unicode characters (vowels, letters, digits, signs):
https://www.key-shortcut.com/en/writing-systems/ïșïșŰą-arabic-alphabet/
On the right side of the page you can choose COMPACT or COMPLETE ⊠bot are very helpful.
When you choose COMPACT you will see the Unicodes.
In my case U+063B up to U+0657 should be ignored.
in the SEARCH section of SAB
Everything that is named LETTER does not belong to TASHKEEL, same for DIGITS and LIGATURES. This for your information.
Nice to hear from you.
U+063B to U+064A are the main letters, these must be included in search option.
But the range, to be ignored, is from
U+618, U+64B to U+658, U+670
Hi Dear
@mcquayi @jeff_heath @mjpenny @Lorna @joop
Please any progress regarding the issueâŠ
waiting for helpâŠ
Sorry, I agree with Raja about the range to be ignored:
U+0618, U+064B to U+0658, U+0670
I made a mistake in my message. Sorry!
I propose the following new feature in SAB. Under Features > Search, we add 3 more options:
[ ] 'Match vowel markings' is selected by default
[ ] Show 'Match vowel markings' option on Search page
Vowel markings to ignore:
[empty text box]
Notes:
U+0618,U+064B,U+064C,U+064D,U+064E,U+064F,U+0650,U+0651,U+0652,U+0653,U+0654,U+0655,U+0656,U+0657,U+0658,U+0670
U+0618, U+064B - U-0658, U+0670
s/[ŰÙ-ÙÙ°]//
U+0618 - U+061A, U+064B - U+0652
Would this satisfy everyone? Are there better ways of doing this?
Good Idea!!!
It will be a great feature and I think it is the resolution of the said issueâŠ
Furthre, our developers know more and very well.
I disagree. For one thing, I think you are just picking the characters that are important to you. For example, you say to exclude U+0618 ARABIC SMALL FATHA, but what about U+0619 ARABIC SMALL DAMMA? It is an almost identical character, and it may not be important for you to ignore, but it could be important for the next guy. And I also disagree with adding more check boxes in the app. Your âMatch vowel markingsâ is really the same thing as the âMatch accents and tonesâ option that exists already. You just want to change it to fit your specific need.
So I propose something much simpler. I propose that when âMatch accents and tonesâ is not checked (i.e. you want to ignore accents and tones), it automatically ignores everything that is added to a base character (which includes Arabic vowels and combining diacritics in the U+0300 block), as defined in the Unicode standard. These characters all have a General_Category in the Unicode Character Database of:
Mn Nonspacing_Mark a nonspacing combining mark (zero advance width)
FYI - the Unicode Character Database is found here:
https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt
And this is a description of how to read it:
https://www.unicode.org/reports/tr44/
What is currently ignored by the code if âMatch accents and tonesâ is not checked? Obviously apps currently created with the App Builders donât ignore all of these characters marked Mn. If they did, then we wouldnât be having this discussion, because all of the Arabic vowels are marked Mn, and could be easily ignored with the options currently available.
So I would propose that if the âMatch accents and tonesâ box is not selected, then app searches should ignore anything marked Mn in the database. This should be easy to do programmatically. (Programmer notes: In Python, I use if unicodedata.category(letter) == 'Mn'. And you would need to remove Mn characters from both the search word and text searched, before you do the comparison. I imagine ignoring all Mn characters would actually make the code simpler than it currently is, since currently it ignores just some select subset.)
I donât think itâs a problem for Arabic users to have to figure out that ignoring accents and tones includes vowels. But if you are insistent that the checkbox should say something about vowels, then we could add a field "Custom text for âMatch accents and tonesâ " - and you could put your vowel text there. But its function really is replacing accents and tones, and I think Arabic users should be able to figure that out.
If we follow this proposal to ignore all the Mn characters, I think the field at the bottom needs to change. Right now it says, âSpecify the replacements and diacritics to remove if not matching accents and tones.â For one thing, I think it should appear above the input buttons fields, right under the âMatch accents and tonesâ checkboxes. And I have trouble understanding the double negative. I think it should try to explain the combining marks first, then invite additional removals or replacements, something like this:
Ignoring accents and tones will automatically ignore nonspacing combining marks (as defined in the Unicode standard), things like combining diacritics and Arabic script combining vowels. Specify here (separated by spaces) any additional characters to ignore, or replacements to make in the search text. E.g. âĂ©>e Ăš>eâ would allow those two composed characters to be found with a simple âeâ in the search string. The string â_ -â or â\u005F \u002Dâ would ignore underscores and hyphens in the search. (Note that you can use \uXXXX to define a character by its Unicode hex code.)
I think making these changes will probably serve the needs of 99% of users. The cases that arenât served are when a user doesnât want to ignore certain characters marked Mn, but does want to ignore others. Because with the above, if you choose to ignore, all of the Mn characters will be ignored. I personally canât think of a case when this would be needed. But I think 99% of those 1% of cases could be served by adding another simple field (below the field with the long explanation) that has this label:
Enter here characters to ignore or replacements to always use in the search, even when âMatch accents and tonesâ is selected.
I would think in this case you wouldnât even want to show the âMatch accents and tonesâ option on the Search page.
So thatâs my proposal. I donât think it would be very much work for the developers to implement (especially if we didnât add the second field, but just kept the original field, changed its label and moved it up on the configuration page). And I think it makes more sense than making even more specific search criteria basically just for Arabic.
Previously we talked about always ignoring U+0640 ARABIC TATWEEL. I think that should still be done. But other characters would be ignored based on their character type, and whether the box is checked or not.
Just FYI - there are 96 Arabic characters that are marked Mn in the Unicode standard. The best way to find them is to look for the dotted circle âbaseâ under combining characters on the 3 Arabic Unicode code pages (links below). But you can also search for â;Mn;â in the Unicode Character Database.