[sword-devel] CzeCSP xref encoding error

[Karl Kleinpaste]

I got a Xiphos crash report this morning from someone using CzeCSP. The
error is that there are xref encoding glitches that Sword fails to
parse, which cause Xiphos to crash when trying to process them. Oops.

The immediate glitch is fixed in Xiphos so as to avoid the crash. The
underlying module problem is that there are many xrefs which have what I
believe is an improper target.

Good example:
<note n="o308" osisID="Rom.7.11!o308" osisRef="Rom.7.11"
type="crossReference">Gn 3,20</note>

Bad example:
<note n="o305" osisID="Rom.7.8!o305" osisRef="Rom.7.8"
type="crossReference">v. 11</note>

The xref target is not in BCV format. getEntryAttributes() returns
nothingness for "v. 11", and in Xiphos' older code that eventually
crawls down into a plain C interface to GTK, this propagates as NULL.

So as I said, the crash is fixed. But I believe the example above is
bad due to this target error. Am I correct?

$ mod2imp CzeCSP | grep '>v\.' | wc
2434 117012 1539356

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/bf3433f6/attachment.html>

[Karl Kleinpaste]

mod2imp CzeCSP |
while read spec ; do
echo "$spec"
spec=${spec%%:*}
spec=${spec##\$\$\$}
read content
echo "$content" | sed -e 's/>v\.[^0-9]*\([0-9]\)/>'"$spec"':\1/g'
done > csp.imp.new
imp2vs csp.imp.new -z -o . -b 4 -v German -l cs
mkdir ~/.sword/modules/texts/ztext/czecsp/.HOLD
mv ~/.sword/modules/texts/ztext/czecsp/[on]t.???
~/.sword/modules/texts/ztext/czecsp/.HOLD
mv [on]t.??? ~/.sword/modules/texts/ztext/czecsp

This corrects all "v." xrefs to use their in-chapter BCV reference, and
now Xiphos' new warning for bad xrefs is no longer being tripped. The
.HOLD usage is to keep the original content on hand, just in case.

That said, other references seem still to be wrong, e.g. the good
example previously mentioned has an xref target of "Gen 3,20" which is
not quite proper either; getEntryAttributes() apparently decides this
means Rom 7:20.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/1a1d084f/attachment-0001.html>

[DM Smith]

That said, other references seem still to be wrong, e.g. the good example previously mentioned has an xref target of "Gen 3,20" which is not quite proper either; getEntryAttributes() apparently decides this means Rom 7:20.

I'm currently looking at a similar problem in encoding a Spanish text. Trying to figure out how to get it to work.

In the following it is using Gn for Genesis. It also uses a ',' as a chapter verse separator.

<note n="o308" osisID="Rom.7.11!o308" osisRef="Rom.7.11" type="crossReference">Gn 3,20</note>

I'm not sure whether one of the other or both is the problem.

This is defined in the locale for cs, which is the proper locale ref for Czech. I'm finding GN in just a few of the locales.
Could that be part of the problem? Using the wrong locale or GN not being in the English/default?

Apparently that it comes up with Rom 7:20 because Rom 7:11 is the basis for the parsing of "Gn 3,20" and it is parsing the 20 as a verse belonging to the basis.

Here is an example of the problem I'm seeing:

I'm using the following to parse this:
LocaleMgr::getSystemLocaleMgr()->setDefaultLocaleName("es");
string osisID = "Gen.1.1"; // The basis for the reference
VerseKey key = osisID.c_str();
string inRef = "Sal. 102:25; Isa. 40:21; Juan 1:1, 2; Heb. 1:10";
outRef = VerseKey::convertToOSIS(inRef.c_str(), &key);

The output is:
<note type='crossReference' osisID='Gen.1.1.xref.A' n='A'>
<reference osisRef="Isa.40.21">Sal. 102:25; Isa. 40:21</reference>;
<reference osisRef="Isa.40.2">Juan 1:1, 2</reference>;
<reference osisRef="Heb.1.10">Heb. 1:10</reference>
</note>

Seemingly it skips Sal. 102:25; and gets Isa. 40:20, which it then uses as the basis for the next reference, where it skips Juan 1:1 and takes 2 as part of Isa 40.

In both es.conf and es-utf8.conf we have the following:
SAL=Ps
JUAN=John

Don't know why it can't see Sal and Juan.

In Him,
DM

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/5dd21ad1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4145 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/5dd21ad1/attachment-0001.p7s>

[DM Smith]

Both are bad. See below.

I got a Xiphos crash report this morning from someone using CzeCSP. The error is that there are xref encoding glitches that Sword fails to parse, which cause Xiphos to crash when trying to process them. Oops.
The immediate glitch is fixed in Xiphos so as to avoid the crash. The underlying module problem is that there are many xrefs which have what I believe is an improper target.
<note n="o308" osisID="Rom.7.11!o308" osisRef="Rom.7.11" type="crossReference">Gn 3,20</note>

This should be:
<note n="o308" osisID="Rom.7.11!o308" osisRef="Rom.7.11" type="crossReference"><reference osisRef="Gen.3.20">Gn 3,20</reference></note>

<note n="o305" osisID="Rom.7.8!o305" osisRef="Rom.7.8" type="crossReference">v. 11</note>

I think this should parse if given the basis of the current verse Romans 7.8. (Not that it does.) IIRC, the parser had special code to recognize v as a prefix for a verse number, but it may have been for 'v11' or 'v 11'.

But it should be:
<note n="o305" osisID="Rom.7.8!o305" osisRef="Rom.7.8" type="crossReference"><reference osisRef="Rom.7.11">v. 11</reference></note>

On a different note, the n="o305" is not appropriate. OSIS defines n as the value that should be shown for the footnote marker. Some front ends do this. It will look ugly.

The xref target is not in BCV format. getEntryAttributes() returns nothingness for "v. 11", and in Xiphos' older code that eventually crawls down into a plain C interface to GTK, this propagates as NULL.
So as I said, the crash is fixed. But I believe the example above is bad due to this target error. Am I correct?

IMHO: Yes it is bad.

-- DM

$ mod2imp CzeCSP | grep '>v\.' | wc
2434 117012 1539356
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/56947870/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4145 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/56947870/attachment.p7s>

[Karl Kleinpaste]

Post by DM Smith
IIRC, the parser had special code to recognize v as a prefix for a
verse number, but it may have been for 'v11' or 'v 11'.

If it matters, most of those "v." xrefs used a Unicode space, "v.[weird
space here]11." Perhaps the parser is unhappy with that?

[Matěj Cepl]

Post by Karl Kleinpaste
I got a Xiphos crash report this morning from someone using CzeCSP. The
error is that there are xref encoding glitches that Sword fails to
parse, which cause Xiphos to crash when trying to process them. Oops.

I am the author of CzeCSP conversion into OSIS XML format. The problem
is that I don?t have in my source texts anything to hang on other than a
free text. I have absolutely no clue who to do the parsing in XSLT
(which is what I used for XML-to-XML conversion) and I will have a
really hard time to parse this free text in Python (but I may try
eventually; it has been
https://luther.ceplovi.cz/bugzilla/show_bug.cgi?id=180 for some time).

However, I don't see anything in
http://img.forministry.com/7/7B/7BB51FB8-84B3-4FF3-939ED473FA90A632/DOC/OSIS2_1UserManual_06March2006_-_with_O%27Donnell_edits.PDF which would indicate that content of <note> should be anything else than a plain text, so I am afraid it is a bug libsword if it requires something else than it has right to expect.

Post by Karl Kleinpaste
So as I said, the crash is fixed. But I believe the example above is
bad due to this target error. Am I correct?
$ mod2imp CzeCSP | grep '>v\.' | wc
2434 117012 1539356

Of course it just documents how difficult it is to require from all
modules to produce perfect parseable code in the conent of <note>.

Blessings,

Mat?j
--
http://www.ceplovi.cz/matej/, Jabber: mcepl at ceplovi.cz
GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC

Two things fill the heart with renewed and increasing awe and
reverence the more often and the more steadily that they are
meditated on: the starry skies above me and the moral law inside
me.
-- Immanuel Kant: Critique of Practical Reason
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/59f7f444/attachment.sig>

[Karl Kleinpaste]

it has been https://luther.ceplovi.cz/bugzilla/show_bug.cgi?id=180 for some time

In that bug report:

/While OSIS allows for more than one reference in an osisRef, SWORD
does not handle the linking for it, but only the first listed./

I've noticed this before, and been annoyed by it. When there is a long
xref list, I consider it vastly preferable that the entire set be one
refList, as returned by getEntryAttributes. I have a module containing
this as a single xref:

Exod 15:2; 17:16; Pss 68:5, 19; 77:12; 89:9; 94:7, 12; 102:19; 104:35;
105:45; 106:1, 48; 111:1; 112:1; 113:1, 9; 115:17, 18; 116:19; 117:2;
118:5, 14, 17-19; 122:4; 130:3; 135:1, 3, 4, 21; 146:1, 10; 147:1, 20;
148:1, 14; 149:1, 9; 150:1, 6; Isa 12:2; 26:4; 38:11

(NETnote, Song 8:6#24.)

I really don't want every single item in that nightmare individually
clickable; I want the whole thing as one unit, shipped off to a verse
list UI element of some kind (Xiphos does it in the sidebar), where I
can peruse the individual items as I wish.

Why doesn't Sword allow this in the OSIS case?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/ae5b01a0/attachment.html>

[DM Smith]

Post by Matěj Cepl

Post by Karl Kleinpaste
I got a Xiphos crash report this morning from someone using CzeCSP. The
error is that there are xref encoding glitches that Sword fails to
parse, which cause Xiphos to crash when trying to process them. Oops.

I am the author of CzeCSP conversion into OSIS XML format. The problem
is that I don?t have in my source texts anything to hang on other than a
free text. I have absolutely no clue who to do the parsing in XSLT
(which is what I used for XML-to-XML conversion) and I will have a
really hard time to parse this free text in Python (but I may try
eventually; it has been
https://luther.ceplovi.cz/bugzilla/show_bug.cgi?id=180 for some time).

In your xslt it is possible to call Java code. Here is how I do it using Xalan and JSword:
<xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0"
xmlns:jsword="http://xml.apache.org/xalan/java"
extension-element-prefixes="jsword">
...
<!-- Create a versification from which verse numbers are understood -->
<xsl:variable name="v11nf" select="jsword:org.crosswire.jsword.versification.system.Versifications.instance()"/>
<!-- Create a global key factory from which OSIS ids will be generated -->
<xsl:variable name="keyf" select="jsword:org.crosswire.jsword.passage.PassageKeyFactory.instance()"/>
<!-- Create a global number shaper that can transform 0-9 into other number systems. -->
<xsl:variable name="shaper" select="jsword:org.crosswire.common.icu.NumberShaper.new()"/>
...
<xsl:template match="verse" mode="print-notes">
<xsl:if test=".//note[not(@type) or not(@type = 'x-strongsMarkup')]">
<xsl:variable name="versification" select="jsword:getVersification($v11nf, $v11n)"/>
<xsl:variable name="passage" select="jsword:getValidKey($keyf, $versification, @osisID)"/>
<a href="#{substring-before(concat(@osisID, ' '), ' ')}">
<xsl:value-of select="jsword:getName($passage)"/>
</a>
<xsl:apply-templates select=".//note" mode="print-notes" />
<div><xsl:text>&#160;</xsl:text></div>
</xsl:if>
</xsl:template>
...
</xsl:stylesheet>

I invoke xalan.jar (or what is embedded in java), with jsword jar in the classpath, to do the transformation.

The syntax is particular to xalan. There is a different syntax for saxon. Haven't tried it though.

There are Java bindings for the SWORD library. Troy can give more on this. They might dig deep enough to expose the SWORD parser in all of it's glory.

Post by Matěj Cepl
However, I don't see anything in
http://img.forministry.com/7/7B/7BB51FB8-84B3-4FF3-939ED473FA90A632/DOC/OSIS2_1UserManual_06March2006_-_with_O%27Donnell_edits.PDF which would indicate that content of <note> should be anything else than a plain text, so I am afraid it is a bug libsword if it requires something else than it has right to expect.

A proper reference in OSIS is wrapped in <reference osisRef="the OSIS reference">the text reference</reference>
A cross reference not is <note type="crossReference">one or more cross references</note>

It is these two things together that produce the reference.

Basically, the reference element says to ignore the text reference and use the osis reference instead.

Post by Matěj Cepl

Post by Karl Kleinpaste
So as I said, the crash is fixed. But I believe the example above is
bad due to this target error. Am I correct?
$ mod2imp CzeCSP | grep '>v\.' | wc
2434 117012 1539356

Of course it just documents how difficult it is to require from all
modules to produce perfect parseable code in the conent of <note>.
Blessings,
Mat?j
--
http://www.ceplovi.cz/matej/, Jabber: mcepl at ceplovi.cz
GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC
Two things fill the heart with renewed and increasing awe and
reverence the more often and the more steadily that they are
meditated on: the starry skies above me and the moral law inside
me.
-- Immanuel Kant: Critique of Practical Reason
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/40ff912e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4145 bytes
Desc: not available
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/40ff912e/attachment.p7s>

[Greg Hellings]

Post by Karl Kleinpaste
I got a Xiphos crash report this morning from someone using CzeCSP. The
error is that there are xref encoding glitches that Sword fails to
parse, which cause Xiphos to crash when trying to process them. Oops.
I am the author of CzeCSP conversion into OSIS XML format. The problem
is that I don?t have in my source texts anything to hang on other than a
free text. I have absolutely no clue who to do the parsing in XSLT
(which is what I used for XML-to-XML conversion) and I will have a
really hard time to parse this free text in Python (but I may try
eventually; it has been
https://luther.ceplovi.cz/bugzilla/show_bug.cgi?id=180 for some time).
<xsl:stylesheet
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
version="1.0"
xmlns:jsword="http://xml.apache.org/xalan/java"
extension-element-prefixes="jsword">
...
<!-- Create a versification from which verse numbers are understood -->
<xsl:variable name="v11nf"
select="jsword:org.crosswire.jsword.versification.system.Versifications.instance()"/>
<!-- Create a global key factory from which OSIS ids will be generated -->
<xsl:variable name="keyf"
select="jsword:org.crosswire.jsword.passage.PassageKeyFactory.instance()"/>
<!-- Create a global number shaper that can transform 0-9 into other number systems. -->
<xsl:variable name="shaper"
select="jsword:org.crosswire.common.icu.NumberShaper.new()"/>
...
<xsl:template match="verse" mode="print-notes">
<xsl:variable name="versification"
select="jsword:getVersification($v11nf, $v11n)"/>
<xsl:variable name="passage" select="jsword:getValidKey($keyf,
<xsl:value-of select="jsword:getName($passage)"/>
</a>
<xsl:apply-templates select=".//note" mode="print-notes" />
<div><xsl:text>&#160;</xsl:text></div>
</xsl:if>
</xsl:template>
...
</xsl:stylesheet>
I invoke xalan.jar (or what is embedded in java), with jsword jar in the
classpath, to do the transformation.
The syntax is particular to xalan. There is a different syntax for saxon.
Haven't tried it though.
There are Java bindings for the SWORD library. Troy can give more on this.
They might dig deep enough to expose the SWORD parser in all of it's glory.

If not, it is also possible to do this with Python and the full swig
bindings using something like xsltproc.py which can be found around places
where the lxml and libxslt Python bindings are located.

--Greg

Post by Karl Kleinpaste
However, I don't see anything in
http://img.forministry.com/7/7B/7BB51FB8-84B3-4FF3-939ED473FA90A632/DOC/OSIS2_1UserManual_06March2006_-_with_O%27Donnell_edits.PDFwhich would indicate that content of <note> should be anything else than a
plain text, so I am afraid it is a bug libsword if it requires something
else than it has right to expect.
A proper reference in OSIS is wrapped in <reference osisRef="the OSIS
reference">the text reference</reference>
A cross reference not is <note type="crossReference">one or more cross references</note>
It is these two things together that produce the reference.
Basically, the reference element says to ignore the text reference and use
the osis reference instead.
So as I said, the crash is fixed. But I believe the example above is
bad due to this target error. Am I correct?
$ mod2imp CzeCSP | grep '>v\.' | wc
2434 117012 1539356
Of course it just documents how difficult it is to require from all
modules to produce perfect parseable code in the conent of <note>.
Blessings,
Mat?j
--
http://www.ceplovi.cz/matej/, Jabber: mcepl at ceplovi.cz
GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC
Two things fill the heart with renewed and increasing awe and
reverence the more often and the more steadily that they are
meditated on: the starry skies above me and the moral law inside
me.
-- Immanuel Kant: Critique of Practical Reason
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page
_______________________________________________
sword-devel mailing list: sword-devel at crosswire.org
http://www.crosswire.org/mailman/listinfo/sword-devel
Instructions to unsubscribe/change your settings at above page

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/0b506995/attachment-0001.html>

[Matěj Cepl]

Wow! Thanks a lot! I have added comment
https://luther.ceplovi.cz/bugzilla/show_bug.cgi?id=180#c2 to my bug and
I will have to digest it sometime. I was using Saxon because of its
incredible speed (I went from 5 minutes with xsltproc to 25sec with
Saxon9), so I will check how to invoke jSword from Saxon, but it should
be possible as well (at least according to
http://www.saxonica.com/documentation/#!extensibility and
http://www.cafeconleche.org/books/xmljava/chapters/ch17s03.html).

BTW, my XSLT script is
http://luther.ceplovi.cz/git/czekms-csp_bible.git/tree/CEP2OSIS.xsl if
you have any comments on it (apparently your XSLT abilities are much
higher than mine).

Thanks a lot,

Mat?j
--
http://www.ceplovi.cz/matej/, Jabber: mcepl at ceplovi.cz
GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC

The state is the great fictitious entity by which everyone seeks
to live at the expense of everyone else.
-- Frederick Bastiat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/02a17bb4/attachment.sig>

[Matěj Cepl]

Post by Matěj Cepl
BTW, my XSLT script is
http://luther.ceplovi.cz/git/czekms-csp_bible.git/tree/CEP2OSIS.xsl if
you have any comments on it (apparently your XSLT abilities are much
higher than mine).

So, I guess it will be just a bit simpler, I need just to modify this
template:

<!-- refers to other place in Bible -->
<xsl:template match="odkazo">
<xsl:variable name="N" select="@n"/>
<xsl:variable name="refID" ><xsl:call-template
name="getRefID" /></xsl:variable>
<xsl:element name="note">
<xsl:attribute name="type">crossReference</xsl:attribute>
<xsl:attribute name="osisRef"><xsl:value-of
select="$refID" /></xsl:attribute>
<xsl:attribute name="osisID"><xsl:value-of
select="concat($refID,'!', at n)" /></xsl:attribute>
<xsl:attribute name="n"><xsl:value-of
select="./@n" /></xsl:attribute>
<xsl:apply-templates select="//defpozno[@n=$N]"
mode="collect-footnote" />
</xsl:element>
</xsl:template>

Interesting.

Mat?j
--
http://www.ceplovi.cz/matej/, Jabber: mcepl at ceplovi.cz
GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC

If you are not paying for it, you're not the customer; you're the
product being sold.
-- blue_beetle (2010-08-26)
http://www.metafilter.com/95152/Userdriven-discontent#3256046
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/c11ee624/attachment.sig>

[Matěj Cepl]

Post by Matěj Cepl

Post by Matěj Cepl
BTW, my XSLT script is
http://luther.ceplovi.cz/git/czekms-csp_bible.git/tree/CEP2OSIS.xsl if
you have any comments on it (apparently your XSLT abilities are much
higher than mine).

So, I guess it will be just a bit simpler, I need just to modify this
<!-- refers to other place in Bible -->
<xsl:template match="odkazo">

Err, this one as well:

<!-- refers to a text note -->
<xsl:template match="odkaz">
<xsl:variable name="refID" ><xsl:call-template
name="getRefID" /></xsl:variable>
<xsl:variable name="N" select="./@n"/>
<xsl:element name="note">
<xsl:attribute name="type">study</xsl:attribute>
<xsl:attribute name="osisRef"><xsl:value-of
select="$refID" /></xsl:attribute>
<xsl:attribute name="osisID"><xsl:value-of
select="concat($refID,'!', at n)" /></xsl:attribute>
<xsl:attribute name="n"><xsl:value-of
select="./@n" /></xsl:attribute>
<xsl:apply-templates select="//defpozn[@n=$N]"
mode="collect-footnote" />
</xsl:element>
</xsl:template>
--
http://www.ceplovi.cz/matej/, Jabber: mcepl at ceplovi.cz
GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC

My life has been full of terrible misfortunes most of which never
happened.
-- Michel de Montaigne
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
URL: <http://www.crosswire.org/pipermail/sword-devel/attachments/20140410/0167e266/attachment.sig>